Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems

In this paper we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high-level the method is an inexact proximal point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochastic projected subgradient method. The primary contribution of this paper is a simple proof that the proposed algorithm converges at the same rate as the stochastic gradient method for smooth nonconvex problems. This result validates the use of stochastic subgradient methods in nonsmooth, nonconvex optimization as is common when optimizing neural networks.