Stochastic Alternating Direction Method of Multipliers with Variance Reduction for Nonconvex Optimization

In this work, we study the stochastic alternating direction method of multipliers (ADMM) method for optimizing nonconvex problems, and propose two classes of nonconvex stochastic ADMM with variance reduction. The first class is the nonconvex stochastic variance reduced gradient ADMM (SVRG-ADMM), which uses a multi-stage strategy to progressively reduce the variance of stochastic gradients. The second is the nonconvex stochastic average gradient ADMM (SAGA-ADMM), which additionally uses the old gradients estimated in the previous iteration. Theoretically, we analyze convergence of the SVRG-ADMM and SAGA-ADMM, and prove that they enjoy the iteration complexity bound of $O(1/\epsilon)$ to reach an $\epsilon$-stationary solution. In particular, we provide a general framework to analyze convergence and iteration complexity of the nonconvex stochastic ADMM with variance reduction. In addition, we prove that the simple stochastic ADMM (S-ADMM), in which the variance of the stochastic gradients is free, is divergent under some conditions. Finally, the experimental results on some real datasets back up our theoretical results. To the best of our knowledge, this is the first study of iteration complexity of the stochastic ADMM for the noncovex problems.