Online Piecewise Linear Regression via Infinite Depth Context Trees

We investigate the problem of sequential piecewise linear regression from a competitive framework. For an arbitrary and unknown data length $n$, we first introduce a method to partition the regressor space. Particularly, we present a recursive method that divides the regressor space into $O(n)$ disjoint regions (partitions) that can result in approximately $1.5^n$ different piecewise linear models on the regressor space. For each partition, we introduce a universal linear regressor whose performance is nearly as well as the best linear regressor whose parameters are set non-causally. We then use an infinite depth context tree to represent all piecewise linear models and introduce a universal algorithm to achieve the performance of the best piecewise linear model that can be selected in hindsight. In this sense, the introduced algorithm is twice-universal such that it sequentially achieves the performance of the best model that uses the optimal regression parameters. We then show that the performance of the introduced algorithm is asymptotically the same as the optimal twice differentiable function that is selected in hindsight. Our algorithm achieves this performance only with a computational complexity upper bounded by $O(n)$. Furthermore, we present a low complexity, namely $O(\log(n))$, implementation of the algorithm achieving the same upper bound under certain regularity conditions. We provide the explicit description of the algorithm as well as the upper bounds on the regret with respect to the best nonlinear and piecewise linear models, and demonstrate the performance of the algorithm through simulations.