Randomized Block Frank-Wolfe for Convergent Large-Scale Learning

Owing to their low-complexity iterations, Frank-Wolfe (FW) solvers are well suited for various large-scale learning tasks. When block-separable constraints are also present, randomized FW has been shown to further reduce complexity by updating only a fraction of coordinate blocks per iteration. In this context, the present work develops feasibility-ensuring step sizes, and provably convergent randomized block Frank-Wolfe (RB-FW) solvers that are flexible in selecting the number of blocks to update per iteration. Convergence rates of RB-FW are established through computational bounds on a primal sub-optimality measure, and on the duality gap. Different from existing convergence analysis, which only applies to a step-size sequence that does not generally lead to feasible iterates, the analysis here includes two classes of step-size sequences that not only guarantee feasibility of the iterates, but also enhance flexibility in choosing decay rates. The novel convergence results are markedly broadened to encompass also nonconvex objectives, and further assert that RB-FW with exact line-search reaches a stationary point at rate $\mathcal{O}(1/\sqrt{t})$. Performance of RB-FW with different step sizes and number of blocks is demonstrated in two applications, namely charging of electrical vehicles and structural support vector machines. Simulated tests demonstrate the impressive performance improvement of RB-FW relative to existing randomized single-block FW methods.