We develop an omnibus framework for regularized logistic regression by simulation- based inference, exploiting two important results on scale mixtures of normals. By carefully choosing a hierarchical model for the likelihood by one type of mixture, and how regularization may be implemented by another, we obtain subtly different MCMC schemes with varying efficiency depending on the data type (binary v. binomial, say) and the desired estimator (maximum likelihood, maximum a posteriori, posterior mean, etc.). Advantages of this umbrella approach include flexibility, computational efficiency, application in p >> n settings, uncertainty estimates, variable selection, and an ability to assess the optimal degree of regularization in a fully Bayesian setup. We compare the statistical and algorithmic efficiency of each of our proposed methods against each other, and against modern alternatives on synthetic and real data.
View on arXiv