Information-based inference in sloppy and singular models

A central problem in statistics is model selection: the choice between competing models of a stochastic process whose observables are corrupted by noise. In information-based inference, model selection is performed by maximizing the estimated predictive performance. We propose a frequen- tist information criterion (FIC) which extends the applicability of information-based inference to the analysis of singular and sloppy models. In these scenarios, the Akaike information criterion (AIC) can result in significant under or over-estimates of the predictive complexity. Two important mechanisms for this failure are examined: an implicit multiple testing problem and the presence of unidentifiable parameters. FIC rectifies this failure by applying a frequentist approximation to compute the com- plexity. For regular models in the large-sample-size limit, AIC and FIC are equal, but in general the complexity exhibits a sample-size dependent scaling. In the context of singular models, FIC can ex- hibit Bayesian information criterion-like or Hannan-Quinn-like scalings with sample size. FIC does not depend on ad hoc prior distributions or exogenous regularization and can be applied when struc- tured data complicates the use of cross-validatation.