Homotopy Smoothing for Non-Smooth Problems with Lower Complexity than $O(1/ε)$

In this paper, we develop a novel {\bf ho}moto{\bf p}y {\bf s}moothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. Such kind of non-smooth optimization problems arise in many applications, e.g., machine learning, image processing, statistics, cone programming, etc. The best known iteration complexity for solving such non-smooth optimization problems is $O(1/\epsilon)$ without any assumption on the strong convexity. In this work, we will show that the proposed HOPS achieves a lower iteration complexity of $\tilde O(1/\epsilon^{1-\theta})$ with $\theta\in(0,1]$ capturing the local sharpness of the objective function around the optimal solutions. To the best of our knowledge, this is the lowest iteration complexity achieved so far for the considered non-smooth optimization problems without strong convexity assumption. The HOPS algorithm uses Nesterov's smoothing trick and Nesterov's accelerated gradient method and runs in stages, which gradually decreases the smoothing parameter until it yields a sufficiently good approximation of the original function. Experimental results verify the effectiveness of HOPS in comparison with Nesterov's smoothing algorithm and the primal-dual style of first-order methods.