Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model

Since the seminal work of Stein (1956) it has been understood that the empirical covariance matrix can be improved by shrinkage of the empirical eigenvalues. In this paper, we consider a proportional-growth asymptotic framework with $n$ observations and $p_n$ variables having limit $p_n/n \to \gamma \in (0,1]$. We assume the population covariance matrix $\Sigma$ follows the popular spiked covariance model, in which several eigenvalues are significantly larger than all the others, which all equal $1$. Factoring the empirical covariance matrix $S$ as $S = V \Lambda V'$ with $V$ orthogonal and $\Lambda$ diagonal, we consider shrinkers of the form $\hat{\Sigma} = \eta(S) = V \eta(\Lambda) V'$ where $\eta(\Lambda)_{ii} = \eta(\Lambda_{ii})$ is a scalar nonlinearity that operates individually on the diagonal entries of $\Lambda$. Many loss functions for covariance estimation have been considered in previous work. We organize and amplify the list, and study 26 loss functions, including Stein, Entropy, Divergence, Fr\'{e}chet, Bhattacharya/Matusita, Frobenius Norm, Operator Norm, Nuclear Norm and Condition Number losses. For each of these loss functions, and each suitable fixed nonlinearity $\eta$, there is a strictly positive asymptotic loss which we evaluate precisely. For each of these 26 loss functions, there is a unique admissible shrinker dominating all other shrinkers; it takes the form $\hat{\Sigma}^* = V \eta^*(\Lambda) V'$ for a certain loss-dependent scalar nonlinearity $\eta^*$ which we characterize. For 17 of these loss functions, we derive a simple analytical expression for the optimal nonlinearity $\eta^*$; in all cases we tabulate the optimal nonlinearity and provide software to evaluate it numerically on a computer. We also tabulate the asymptotic slope, and, where relevant, the asymptotic shift of the optimal nonlinearity.