Scalable Bayesian variable selection and model averaging under block orthogonal design

We show how to carry out fully Bayesian variable selection and model averaging in linear models when both the number of observations and covariates are large. We work under the assumption that the Gram matrix is block-diagonal. Apart from orthogonal regression and various contexts where this is satisfied by design, this framework may serve in future work as a basis for computational approximations to more general design matrices with clusters of correlated predictors. Our approach returns the most probable model of any given size without resorting to numerical integration, posterior probabilities for any number of models by evaluating a single one-dimensional integral that can be computed upfront, and other quantities of interest such as variable inclusion probabilities and model averaged regression estimates by carrying out an adaptive, deterministic one-dimensional numerical integration. This integration and model search are done using novel schemes we introduce in this article. We do not require Markov Chain Monte Carlo. The overall computational cost scales linearly with the number of blocks, which can be processed in parallel, and exponentially with the block size, rendering it most adequate in situations where predictors are organized in many moderately-sized blocks.