Message Passing Stein Variational Gradient Descent

Stein variational gradient descent (SVGD) is a remarkable recent Bayesian inference method, which has stronger approximating ability than traditional variational inference methods, and is more effective than Monte Carlo methods with the same particle size. However, we observed that SVGD still manifests particle degeneracy as the dimension increases: particles tend to collapse on local modes. We take an initial step towards understanding this phenomenon by analyzing the repulsive force and find that there exists a negative correlation between the repulsive force and the dimensionality which should be blamed for this phenomenon. We also propose Message Passing SVGD (MP-SVGD) to solve this problem. By leveraging the conditional independence structure of probabilistic graphical models (PGMs), MP-SVGD converts the original high dimensional global inference problem into a set of local ones over the Markov blanket with lower dimensions. Experimental results show its advantages of exploring structural information over SVGD and particle efficiency and approximation flexibility over other inference methods on graphical models.