Stability of cross-validation and minmax-optimal number of folds

In this paper, we analyze the properties of cross-validation from the perspective of the stability, that is, the probablistic maximal difference between the training error and the test error of the cross-validated model. In both the i.i.d.\ and non-i.i.d.\ cases, we derive the probablistic upper bounds of the one-round and average test error, referred to as the one-round/convoluted Rademacher-bounds, as the measure of cross-validation stability. We show that the convoluted Rademacher-bounds quantify the stability of the out-of-sample performance of the cross-validated model in terms of its training error, the sample sizes, number of folds $K$, the `heaviness' in the tails of the loss distribution (encoded as Orlicz-$\Psi_\nu$ norms) and the Rademacher complexity of the model class $\Lambda$. Using the convoluted Rademacher-bounds, we also define the minmax-optimal number of folds, at which the performance of the cross-validated model on new-coming samples is most stable, for cross-validation. The minmax-optimal number of folds also reveals that, given sample size, stability maximization (or upper bound minimization) may help to quantify optimality in hyper-parameter tuning or other learning tasks with large variation.