Fast Stochastic Variance Reduced ADMM for Stochastic Composition Optimization

In this paper, we consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for the strongly convex and Lipschitz smooth objectives, and a convergence rate of $O( \log S/S)$, which improves upon the known $O(S^{-4/9})$ rate when the objective is convex and Lipschitz smooth. Moreover, it processes a rate of $O(1/\sqrt{S})$ when the objective is convex but without Lipschitz smoothness. We also conduct experiments and show that it outperforms existing algorithms.