A Plug-in Method for Representation Factorization

In this work, we focus on decomposing the latent representations in GANs or learned feature representations in deep auto-encoders into semantically controllable factors in a semi-supervised manner, without modifying the original trained models. Specifically, we propose a Factors Decomposer-Entangler Network (FDEN) that learns to decompose a latent representation into mutually independent factors. Given a latent representation, the proposed framework draws a set of interpretable factors, each aligned to independent factors of variations by maximizing their total correlation in an information-theoretic means. As a plug-in method, we have applied our proposed FDEN to the existing networks of Adversarially Learned Inference and Pioneer Network and conducted computer vision tasks of image-to-image translation in semantic ways, e.g., changing styles while keeping an identify of a subject, and object classification in a few-shot learning scheme. We have also validated the effectiveness of our method with various ablation studies in qualitative, quantitative, and statistical examination.