Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates

24 May 2019

Papers citing "Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates"

45 / 45 papers shown

Title
Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent Hikaru Umeda Hideaki Iiduka 72 2 0 17 Feb 2025
Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models Matteo Lapucci Davide Pucci 37 1 0 06 Aug 2024
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance Dimitris Oikonomou Nicolas Loizou 55 4 0 06 Jun 2024
Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation Aaron Mishkin Mert Pilanci Mark Schmidt 66 1 0 03 Apr 2024
Faster Convergence for Transformer Fine-tuning with Line Search Methods Philip Kenneweg Leonardo Galli Tristan Kenneweg Barbara Hammer ODL 51 2 0 27 Mar 2024
Level Set Teleportation: An Optimization Perspective Aaron Mishkin A. Bietti Robert Mansel Gower 45 1 0 05 Mar 2024
An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning Mohammad Salehi Subhadip Mukherjee Lindon Roberts Matthias Joachim Ehrhardt 31 5 0 19 Aug 2023
First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities Aleksandr Beznosikov S. Samsonov Marina Sheshukova Alexander Gasnikov A. Naumov Eric Moulines 54 14 0 25 May 2023
Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning Achraf Bahamou D. Goldfarb ODL 36 0 0 23 May 2023
MoMo: Momentum Models for Adaptive Learning Rates Fabian Schaipp Ruben Ohana Michael Eickenberg Aaron Defazio Robert Mansel Gower 37 10 0 12 May 2023
Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions S. Choudhury Eduard A. Gorbunov Nicolas Loizou 29 13 0 27 Feb 2023
DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule Maor Ivgi Oliver Hinder Y. Carmon ODL 37 57 0 08 Feb 2023
Target-based Surrogates for Stochastic Optimization J. Lavington Sharan Vaswani Reza Babanezhad Mark Schmidt Nicolas Le Roux 60 5 0 06 Feb 2023
Improved Policy Optimization for Online Imitation Learning J. Lavington Sharan Vaswani Mark Schmidt OffRL 28 6 0 29 Jul 2022
Learning Optimal Flows for Non-Equilibrium Importance Sampling Yu Cao Eric Vanden-Eijnden 21 3 0 20 Jun 2022
Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method Aaron Defazio Baoyu Zhou Lin Xiao ODL 27 5 0 14 Jun 2022
Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization Junchi Yang Xiang Li Niao He ODL 45 22 0 01 Jun 2022
Making SGD Parameter-Free Y. Carmon Oliver Hinder 30 43 0 04 May 2022
Latency Optimization for Blockchain-Empowered Federated Learning in Multi-Server Edge Computing Dinh C. Nguyen Seyyedali Hosseinalipour David J. Love P. Pathirana Christopher G. Brinton 36 47 0 18 Mar 2022
Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems Quentin Le Lidec Fabian Schramm Louis Montaut Cordelia Schmid Ivan Laptev Justin Carpentier 38 24 0 08 Mar 2022
Amortized Proximal Optimization Juhan Bae Paul Vicol Jeff Z. HaoChen Roger C. Grosse ODL 33 14 0 28 Feb 2022
A Stochastic Bundle Method for Interpolating Networks Alasdair Paren Leonard Berrada Rudra P. K. Poudel M. P. Kumar 26 4 0 29 Jan 2022
Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Points Armin Eftekhari 24 1 0 25 Dec 2021
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize Ryan DÓrazio Nicolas Loizou I. Laradji Ioannis Mitliagkas 39 30 0 28 Oct 2021
Iterative Teaching by Label Synthesis Weiyang Liu Zhen Liu Hanchen Wang Liam Paull Bernhard Schölkopf Adrian Weller 50 16 0 27 Oct 2021
Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent Sharan Vaswani Benjamin Dubois-Taine Reza Babanezhad 53 11 0 21 Oct 2021
KOALA: A Kalman Optimization Algorithm with Loss Adaptivity A. Davtyan Sepehr Sameni L. Cerkezi Givi Meishvili Adam Bielski Paolo Favaro ODL 58 2 0 07 Jul 2021
Stochastic Polyak Stepsize with a Moving Target Robert Mansel Gower Aaron Defazio Michael G. Rabbat 32 17 0 22 Jun 2021
Understanding approximate and unrolled dictionary learning for pattern recovery Benoit Malézieux Thomas Moreau M. Kowalski MU 22 10 0 11 Jun 2021
SVRG Meets AdaGrad: Painless Variance Reduction Benjamin Dubois-Taine Sharan Vaswani Reza Babanezhad Mark Schmidt Simon Lacoste-Julien 23 18 0 18 Feb 2021
On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization Abolfazl Hashemi Anish Acharya Rudrajit Das H. Vikalo Sujay Sanghavi Inderjit Dhillon 20 7 0 20 Nov 2020
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering Ricky T. Q. Chen Dami Choi Lukas Balles David Duvenaud Philipp Hennig ODL 46 6 0 09 Nov 2020
A straightforward line search approach on the expected empirical loss for stochastic deep learning problems Max Mutschler A. Zell 38 0 0 02 Oct 2020
Stochastic Adaptive Line Search for Differentially Private Optimization Chen Chen Jaewoo Lee 22 14 0 18 Aug 2020
A Weakly Supervised Consistency-based Learning Method for COVID-19 Segmentation in CT Images I. Laradji Pau Rodríguez López Oscar Manas Keegan Lensink M. Law Lironne Kurzman William Parker David Vazquez Derek Nowrouzezahrai 23 84 0 04 Jul 2020
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt Frank Schneider Philipp Hennig ODL 47 162 0 03 Jul 2020
Adversarial Example Games A. Bose Gauthier Gidel Hugo Berrard Andre Cianflone Pascal Vincent Simon Lacoste-Julien William L. Hamilton AAML GAN 38 51 0 01 Jul 2020
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation Robert Mansel Gower Othmane Sebbouh Nicolas Loizou 27 74 0 18 Jun 2020
Statistical Adaptive Stochastic Gradient Methods Pengchuan Zhang Hunter Lang Qiang Liu Lin Xiao ODL 15 11 0 25 Feb 2020
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence Nicolas Loizou Sharan Vaswani I. Laradji Simon Lacoste-Julien 29 181 0 24 Feb 2020
Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization Samuel Horváth Lihua Lei Peter Richtárik Michael I. Jordan 57 30 0 13 Feb 2020
Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities Zhongruo Wang Krishnakumar Balasubramanian Shiqian Ma Meisam Razaviyayn 21 25 0 22 Jan 2020
Adaptive Stochastic Optimization Frank E. Curtis K. Scheinberg ODL 19 29 0 18 Jan 2020
L4: Practical loss-based stepsize adaptation for deep learning Michal Rolínek Georg Martius ODL 49 63 0 14 Feb 2018
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark Schmidt 139 1,205 0 16 Aug 2016