Eigenvalue Distributions of Variance Components Estimators in High-Dimensional Random Effects Models

We study the spectra of MANOVA estimators and other quadratic estimators for variance component covariance matrices in multivariate random effects models. When the dimensionality of the observations is large and comparable to both the number of samples and the number of realizations of each random effect, the empirical eigenvalue distributions of such estimators are well-approximated by deterministic laws. The Stieltjes transforms of these laws may be characterized by systems of fixed-point equations, which are numerically solvable by a simple iterative procedure. Our derivation of this characterization uses operator-valued free probability theory, and we establish a general asymptotic freeness result for families of rectangular orthogonally-invariant random matrices, which is of independent interest. Our work is motivated by the estimation of components of covariance between multiple phenotypic traits in quantitative genetics, and we specialize our results to common experimental designs that arise in this application.