Regularity Properties of High-dimensional Covariate Matrices

Regularity properties such as the incoherence condition, the restricted isometry property, compatibility, restricted eigenvalue and $\ell_q$ sensitivity of covariate matrices play a pivotal role in high-dimensional regression and compressed sensing. Yet, like computing the spark of a matrix, we first show that it is NP-hard to check the conditions involving all submatrices of a given size. This motivates us to investigate what classes of design matrices satisfy these conditions. We demonstrate that the most general property, $\ell_q$ sensitivity, holds with high probability for covariate matrices sampled from populations with a suitably regular covariance matrix. The probability lower bound and sample size required depend on the tail behavior of the random noise. We examine this for three important cases, bounded, sub-Gaussian, and finite moment random noises. We further show that $\ell_q$ sensitivity is preserved under natural operations on the data. Our work is particularly important for many statistical applications, in which the covariates are observational and correlated and can be thought of as fixed in advance.