Using Multilevel Circulant Matrix Approximation to Speed Up Kernel Logistic Regression

Kernel logistic regression (KLR) is a conventional nonlinear classifier in machine learning. With the explosive growth of data size, the storage and computation of large dense kernel matrices is a major challenge in scaling KLR to massive datasets. This paper proposes an approximate Newton method for efficiently solving large-scale KLR problems by exploiting the storage and computing advantages of multilevel circulant matrix(MCM). We reduce the storage space of the Newton method to $O(n)$ by approximating the kernel matrix $\boldsymbol K$ with the MCM $\boldsymbol K_{\boldsymbol{q}}$, where $n$ is the number of training instances and $\boldsymbol{q}$ is the order of the MCM $\boldsymbol K_{\boldsymbol{q}}$. And by approximating the coefficient matrix of the Newton equation as an MCM, we can reduce the computational complexity in the Newton direction to $O(n \log n)$. Our method can run in linearithmic time at each iteration by using the multidimensional fast Fourier transform (mFFT). Experimental results on some large-scale binary and multi-classification problems show that our method enables KLR to scale to large scale problems with less memory consumption and less time without sacrificing test accuracy.