Differentially Private Variational Inference for Non-conjugate Models

As collecting huge amounts of personal data from individuals has been established as a standard nowadays, it is really important to use these data in a conscientious way. For example, when performing inference using these data, one has to make sure individuals' identities or the privacy of the data are not compromised. Differential privacy is a powerful framework that introduces stochasticity into the computation to guarantee that it is difficult to breach the privacy using the output of the computation. Differentially private versions of many important machine learning methods have been proposed, but still there is a long way to pave towards an efficient unified approach applicable to handle many models. In this paper, we propose a differentially private variational inference method with a very wide applicability. The variational inference is based on stochastic gradient ascent and can handle non-conjugate models as well as conjugate ones. Differential privacy is achieved by perturbing the gradients. We explore ways to make the algorithm more efficient through privacy amplification from subsampling and through clipping the gradients to limit the amount of information they leak. We explore the effect of different parameter combinations in logistic regression problems where the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees.