Support Vector Regression, Smooth Splines, and Time Series Prediction

Delay coordinates and support vector regression are among the techniques commonly used for time series prediction. We show that the combination of these two techniques leads to systematic error that obstructs convergence. A preliminary step of spline smoothing restores convergence and leads to predictions that are consistently more accurate, typically by about a factor of $2$ or so. Since the algorithm without spline smoothing is not convergent, the improvement in accuracy can even be as high as a factor of $100$. Assuming local isotropy, the systematic error in the absence of spline smoothing is estimated to be $d\sigma^{2}L/2$, where $d$ is the embedding dimension, $\sigma^{2}$ is the variance of Gaussian noise in the signal, and $L$ is a global bound on the Hessian of the exact predictor. The smooth spline, although very effective, is shown not to have even first order accuracy, unless the noise is unusually mild. The lack of order of accuracy implies that attempts to take advantage of invariance in time to enhance fidelity of learning are unlikely to be successful.