Empirical risk minimization and complexity of dynamical models

A dynamical model consists of a continuous self-map $T: \mathcal{X} \to \mathcal{X}$ of a compact state space $\mathcal{X}$ and a continuous observation function $f: \mathcal{X} \to \mathbb{R}$. Dynamical models give rise to real-valued sequences when $f$ is applied to repeated iterates of $T$ beginning at some initial state. This paper considers the fitting of a parametrized family of dynamical models to an observed real-valued stochastic process using empirical risk minimization. More precisely, at each finite time, one selects a model yielding values that minimize the average per-symbol loss with the observed process. The limiting behavior of the minimum risk parameters is studied in a general setting where the observed process need not be generated by a dynamical model in the family and minimal conditions are placed on the loss function. We establish a general convergence theorem for minimum risk estimators and ergodic observations, showing that the limiting parameter set is characterized by the projection of the observed process onto the set of processes associated with the dynamical family, where the projection is taken with respect to a joining-based distortion between stationary processes that is a generalization of the $\overline{d}$-distance. We then turn our attention to observations generated from an ergodic process plus an i.i.d. noise process and study conditions under which empirical risk minimization can effectively separate the signal from the noise. The key, necessary condition in the latter results is that the family of dynamical models has limited complexity, which is quantified through a notion of entropy for families of infinite sequences. Close connections between entropy and limiting average mean widths for stationary processes are established.