Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance

6 June 2024

Papers citing "Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance"

11 / 11 papers shown

Title
Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation Robert M. Gower Guillaume Garrigos Nicolas Loizou Dimitris Oikonomou Konstantin Mishchenko Fabian Schaipp 31 0 0 02 Apr 2025
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes Igor Sokolov Peter Richtárik 59 1 0 22 Dec 2024
An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes Antonio Orvieto Lin Xiao 22 1 0 05 Jul 2024
Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad Sayantan Choudhury N. Tupitsa Nicolas Loizou Samuel Horváth Martin Takáč Eduard A. Gorbunov 18 1 0 05 Mar 2024
SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms Farshed Abdukhakimov Chulu Xiang Dmitry Kamzolov Robert Mansel Gower Martin Takáč 19 2 0 28 Dec 2023
Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization Karan N. Chadha Gary Cheng John C. Duchi 29 14 0 07 Jan 2021
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 56 119 0 05 Mar 2020
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 73 126 0 16 Oct 2018
L4: Practical loss-based stepsize adaptation for deep learning Michal Rolínek Georg Martius ODL 21 63 0 14 Feb 2018
Densely Connected Convolutional Networks Gao Huang Zhuang Liu L. V. D. van der Maaten Kilian Q. Weinberger PINN 3DV 236 35,884 0 25 Aug 2016
Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes Ohad Shamir Tong Zhang 99 570 0 08 Dec 2012