A Closed-Form EM Algorithm for Sparse Coding

We define and discuss the first sparse coding algorithm based on closed-form EM updates and continuous latent variables. The underlying generative model consists of a flexibly parameterized `spike-and-slab' prior and a standard Gaussian noise model. Closed-form solutions for E- and M-step equations are derived by generalizing probabilistic PCA. The resulting EM algorithm infers all model parameters including the level of sparsity, and it takes all modes of a potentially multi-modal posterior into account. The computational cost of the algorithm scales exponentially with the number of hidden dimensions. However, with current computational resources, efficient training of model parameters is still possible for medium-scale problems (involving up to 20 hidden dimensions). Thus the model can be applied to the typical range of source separation tasks. In numerical experiments on artificial data we verify likelihood maximization and show that the derived algorithm recovers the sparse directions of standard sparse coding distributions. On source separation benchmarks comprising realistic data we show that the algorithm is competitive with other recent methods.