Learning Generative ConvNet with Continuous Latent Factors by Alternating Back-Propagation

The supervised learning of the discriminative convolutional neural network (ConvNet) is powered by back-propagation on the parameters. In this paper, we show that the unsupervised learning of a popular top-down generative ConvNet model with latent continuous factors can be accomplished by a learning algorithm that consists of alternatively performing back-propagation on both the latent factors and the parameters. The model is a non-linear generalization of factor analysis, where the high-dimensional observed data, is assumed to be the noisy version of a vector generated by a non-linear transformation of a low-dimensional vector of continuous latent factors. Furthermore, it is assumed that these latent factors follow known independent distributions, such as standard normal distributions, and the non-linear transformation is assumed to be parametrized by a top-down ConvNet, which is capable of approximating the highly non-linear mapping from the latent factors to the image. We explore a simple and natural learning algorithm for this model that alternates between the following two steps: (1) inferring the latent factors by Langevin dynamics or gradient descent, and (2) updating the parameters of the ConvNet by gradient descent. Step (1) is based on the gradient of the reconstruction error with respect to the latent factors, which is available by back-propagation. We call this step inferential back-propagation. Step (2) is based on the gradient of the reconstruction error with respect to the parameters, and is also obtained by back-propagation. We refer to this step as learning back-propagation. The code for inferential back-propagation is actually part of the code for learning back-propagation, and thus the inferential back-propagation is actually a by-product of the learning back-propagation. We show that our algorithm can learn realistic generative models of images and sounds.