SGD: General Analysis and Improved Rates

27 January 2019

Papers citing "SGD: General Analysis and Improved Rates"

43 / 43 papers shown

Title
Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization Dmitry Kovalev 52 0 0 16 Mar 2025
Random Reshuffling for Stochastic Gradient Langevin Dynamics Luke Shaw Peter A. Whalley 75 3 0 28 Jan 2025
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance Dimitris Oikonomou Nicolas Loizou 45 4 0 06 Jun 2024
Demystifying SGD with Doubly Stochastic Gradients Kyurae Kim Joohwan Ko Yian Ma Jacob R. Gardner 48 0 0 03 Jun 2024
An Inexact Halpern Iteration with Application to Distributionally Robust Optimization Ling Liang Kim-Chuan Toh Jia Jie Zhu 16 4 0 08 Feb 2024
AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size P. Ostroukhov Aigerim Zhumabayeva Chulu Xiang Alexander Gasnikov Martin Takáč Dmitry Kamzolov ODL 33 2 0 07 Feb 2024
Provably Scalable Black-Box Variational Inference with Structured Variational Families Joohwan Ko Kyurae Kim W. Kim Jacob R. Gardner BDL 17 1 0 19 Jan 2024
Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications Jie Hu Vishwaraj Doshi Do Young Eun 25 4 0 17 Jan 2024
On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations Antoine Godichon-Baggioni Nicklas Werge ODL 26 3 0 29 Nov 2023
Demystifying the Myths and Legends of Nonconvex Convergence of SGD Aritra Dutta El Houcine Bergou Soumia Boucherouite Nicklas Werge M. Kandemir Xin Li 18 0 0 19 Oct 2023
When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluation Chao Huang Geng Tian Ming Tang FedML 22 4 0 23 Aug 2023
Provable convergence guarantees for black-box variational inference Justin Domke Guillaume Garrigos Robert Mansel Gower 13 18 0 04 Jun 2023
Statistical Inference with Stochastic Gradient Methods under $φ$ -mixing Data Ruiqi Liu X. Chen Zuofeng Shang FedML 12 6 0 24 Feb 2023
Distributed Stochastic Optimization under a General Variance Condition Kun-Yen Huang Xiao Li Shin-Yi Pu FedML 14 5 0 30 Jan 2023
Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes Konstantin Mishchenko Slavomír Hanzely Peter Richtárik FedML 14 5 0 17 Jan 2023
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey Aleksandr Beznosikov Boris Polyak Eduard A. Gorbunov D. Kovalev Alexander Gasnikov 32 31 0 29 Aug 2022
Empirical Study of Overfitting in Deep FNN Prediction Models for Breast Cancer Metastasis Chuhan Xu Pablo Coen-Pirani Xia Jiang AI4CE 13 1 0 03 Aug 2022
Towards Understanding Sharpness-Aware Minimization Maksym Andriushchenko Nicolas Flammarion AAML 19 131 0 13 Jun 2022
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning Liam Collins Hamed Hassani Aryan Mokhtari Sanjay Shakkottai FedML 19 74 0 27 May 2022
Federated Random Reshuffling with Compression and Variance Reduction Grigory Malinovsky Peter Richtárik FedML 10 10 0 08 May 2022
Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD Konstantinos E. Nikolakakis Farzin Haddadpour Amin Karbasi Dionysios S. Kalogerias 22 17 0 26 Apr 2022
BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling Cheng Wan Youjie Li Ang Li Namjae Kim Yingyan Lin GNN 27 75 0 21 Mar 2022
Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods Aleksandr Beznosikov Eduard A. Gorbunov Hugo Berard Nicolas Loizou 17 47 0 15 Feb 2022
Towards Federated Learning on Time-Evolving Heterogeneous Data Yongxin Guo Tao R. Lin Xiaoying Tang FedML 6 30 0 25 Dec 2021
Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum J. Kim Panos Toulis Anastasios Kyrillidis 10 8 0 11 Nov 2021
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize Ryan DÓrazio Nicolas Loizou I. Laradji Ioannis Mitliagkas 20 30 0 28 Oct 2021
Decentralized Composite Optimization with Compression Yao Li Xiaorui Liu Jiliang Tang Ming Yan Kun Yuan 13 9 0 10 Aug 2021
Towards Biologically Plausible Convolutional Networks Roman Pogodin Yash Mehta Timothy Lillicrap P. Latham 11 22 0 22 Jun 2021
Stochastic Polyak Stepsize with a Moving Target Robert Mansel Gower Aaron Defazio Michael G. Rabbat 13 17 0 22 Jun 2021
Variance Reduced Training with Stratified Sampling for Forecasting Models Yucheng Lu Youngsuk Park Lifan Chen Bernie Wang Christopher De Sa Dean Phillips Foster AI4TS 20 17 0 02 Mar 2021
AEGD: Adaptive Gradient Descent with Energy Hailiang Liu Xuping Tian ODL 20 11 0 10 Oct 2020
DBS: Dynamic Batch Size For Distributed Deep Neural Network Training Qing Ye Yuhao Zhou Mingjia Shi Yanan Sun Jiancheng Lv 9 11 0 23 Jul 2020
On stochastic mirror descent with interacting particles: convergence properties and variance reduction Anastasia Borovykh N. Kantas P. Parpas G. Pavliotis 13 11 0 15 Jul 2020
Stochastic Hamiltonian Gradient Methods for Smooth Games Nicolas Loizou Hugo Berard Alexia Jolicoeur-Martineau Pascal Vincent Simon Lacoste-Julien Ioannis Mitliagkas 17 51 0 08 Jul 2020
Random Reshuffling: Simple Analysis with Vast Improvements Konstantin Mishchenko Ahmed Khaled Peter Richtárik 15 129 0 10 Jun 2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates Anastasia Koloskova Nicolas Loizou Sadra Boreiri Martin Jaggi Sebastian U. Stich FedML 16 489 0 23 Mar 2020
Gradient tracking and variance reduction for decentralized optimization and machine learning Ran Xin S. Kar U. Khan 9 10 0 13 Feb 2020
Variance Reduced Coordinate Descent with Acceleration: New Method With a Surprising Application to Finite-Sum Problems Filip Hanzely D. Kovalev Peter Richtárik 24 17 0 11 Feb 2020
Better Theory for SGD in the Nonconvex World Ahmed Khaled Peter Richtárik 11 178 0 09 Feb 2020
Unified Optimal Analysis of the (Stochastic) Gradient Method Sebastian U. Stich 12 112 0 09 Jul 2019
Characterization of Convex Objective Functions and Optimal Expected Convergence Rates for SGD Marten van Dijk Lam M. Nguyen Phuong Ha Nguyen Dzung Phan 14 6 0 09 Oct 2018
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark W. Schmidt 119 1,190 0 16 Aug 2016
Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes Ohad Shamir Tong Zhang 99 570 0 08 Dec 2012