Using their interpretation as fixed-point iterations, we review first order gradient methods for minimizing convex objective functions. Due to their conceptual and algorithmic simplicity, first order gradient methods are widely used in machine learning methods involving massive datasets. In particular, stochastic first order methods are considered the de-facto standard for training deep neural networks. By studying these methods within fixed-point theory provides us with powerful tools to study the convergence properties of a wide range of gradient methods. In particular, first order methods using inexact or noisy gradients, such as in stochastic gradient descent, can be studied using well-known results on inexact fixed-point iterations. Moreover, as illustrated clearly in this paper, the fixed-point picture allows an elegant derivation of accelerations for basic gradient methods. In particular, we show how gradient descent can be accelerated by an fixed- point preserving transformation of an operator associated with the objective function.
View on arXiv