Locally Regularized Readouts for Echo-state Networks

Echo state network (ESN) is viewed as a temporal non-orthogonal expansion with pseudo-random parameters. Such expansions naturally give rise to regressors of various relevance to a teacher output. This study shows using the orthogonal forward regression (OFR) algorithm that often only certain amount of the generated echo-regressors truly explain variance of the teacher output. Remaining echo-regressors increase the explained variance of the teacher signal only marginally and often causes numerical ill-conditioning. It is therefore desirable to identify and drop echo-regressors which do not improve the model quality and prevent ill-conditioning. In this study we present locally regularized linear readout which together with OFR analysis is capable of identifying significant echo-regressors and penalizing (or removing) the undesired non-significant echo-regressors. Locally regularized linear readout presented here is studied not only as a technique that improves robustness and accuracy of an ESN but also as a technique that enables better evaluation of ESN pseudo-random parameters and ESN dimensionality. Moreover, linear readouts have limitations in terms of their flexibility which might be sometimes insufficient for a task at hand. For such cases, we present locally regularized radial basis function (RBF) readout. It is a flexible and parsimonious readout with excellent generalization abilities and is viable alternative to a relevance vector machine (RVM) or a feed-forward neural network based readouts.