Bayesian Inference for Gaussian Process Classifiers with Annealing and Exact-Approximate MCMC

Kernel methods have revolutionized the fields of pattern recognition and machine learning. The importance of achieving a sound quantification of uncertainty in predictions by characterizing the posterior distribution over kernel parameters exactly has been demonstrated in several applications. This paper focuses on Markov chain Monte Carlo (MCMC) based inference of covariance (kernel) parameters for Gaussian process classifiers. Recently, the exact-approximate MCMC approach has been proposed as a practical way to efficiently infer covariance parameters in Gaussian process classifiers exactly. In this approach, an unbiased estimate of the marginal likelihood obtained by importance sampling replaces the actual marginal likelihood in the Hastings ratio. This paper presents the application of annealed importance sampling to obtain a low-variance unbiased estimate of the marginal likelihood. This paper empirically demonstrates that annealed importance sampling reduces the variance of the estimate of the marginal likelihood exponentially in the number of data compared to importance sampling, while the computational cost scales only polynomially. The results on real data demonstrate that employing annealed importance sampling in the exact-approximate MCMC approach represents a step forward in the development of fully automated exact inference engines for Gaussian process classifiers.