Variational Inference in Nonconjugate Models

Mean-field variational inference is widely used for approximate posterior inference in many probabilistic models. When the model is conditionally conjugate, variational updates are in closed-form. However, many models of interest are nonconjuate and practitioners may face the challenges of deriving the corresponding variational updates. In this paper, we develop and study two generic variational strategies for nonconjugate models---Laplace variational inference and delta method variational inference---which place minimal conditions on the model. These strategies extend and unify existing methods that were derived for specific models. We illustrate our approach on the correlated topic models, Bayesian logistic regression, and hierarchical Bayesian logistic regression. Our experimental results show that our methods work well on real-world datasets.