Disentangling Visual Embeddings with Minimal Distributional Assumptions

Interest in understanding and factorizing embedding spaces learned by deep encoders is growing. Concept discovery methods search the embedding spaces for interpretable latent components like object shape or color and disentangle them into individual axes in the embedding space. Yet, the applicability of modern disentanglement learning techniques or independent component analysis (ICA) is limited when it comes to vision tasks: They either require training a model of the complex image-generating process or their rigid stochastic independence assumptions on the component distribution are violated in practice. In this work, we identify components in encoder embedding spaces without distributional assumptions and without training a generator. Instead, we utilize functional compositionality properties of image-generating processes. We derive two novel post-hoc component discovery methods and prove theoretical identifiability guarantees. We study them in realistic visual disentanglement tasks with correlated components and violated functional assumptions. Our approaches stably maintain superior performance against 300+ state-of-the-art disentanglement and component analysis models.