Minimum Description Length of a Spectrum Variational Autoencoder: A Theory

Deep neural networks trained through end-to-end learning have achieved remarkable success across various domains in the past decade. However, the end-to-end learning strategy faces two fundamental limitations: the struggle to form explainable representations in a self-supervised manner, and the inability to compress information rigorously following the Minimum Description Length (MDL) principle. In this paper, we establish a novel theory connecting these two challenges. We design the Spectrum VAE, a novel deep learning architecture whose minimum description length (MDL) can be rigorously evaluated. Then, we introduce the concept of latent dimension combinations, or what we term spiking patterns, and demonstrate that the observed spiking patterns should be as few as possible based on the training data in order for the Spectrum VAE to achieve the MDL. Finally, our theory demonstrates that when the MDL is achieved with respect to the given data distribution, the model will naturally produce explainable latent representations of the data. That is, explainable representations of the data, or understanding the data, can be achieved in a self-supervised manner simply by making the deep neural network obey the MDL principle. In our opinion, this reveals an even more profound principle: Understanding means to represent the acquired information by as small an amount of information as possible. This work is entirely theoretical and aims at inspiring future research to realize self-supervised explainable AI simply by obeying the MDL principle.