Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives

Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: originally designed for strongly convex objectives, is also very robust under non-strongly convex or sum-of-non-convex settings. If $f(x)$ is a sum of smooth, convex functions but $f$ is not strongly convex (such as Lasso or logistic regression), we propose a variant SVRG++ that makes a novel choice of growing epoch length on top of SVRG. SVRG++ is a direct, faster variant of SVRG in this setting. If $f(x)$ is a sum of non-convex functions but $f$ is strongly convex, we show that the convergence of SVRG linearly depends on the non-convexity parameter of the summands. This improves the best known result in this setting, and gives better running time for stochastic PCA.