Momentum-Based Variance Reduction in Non-Convex SGD

24 May 2019

Papers citing "Momentum-Based Variance Reduction in Non-Convex SGD"

50 / 77 papers shown

Title
Efficient Curvature-Aware Hypergradient Approximation for Bilevel Optimization Youran Dong Junfeng Yang Wei-Ting Yao Jin Zhang 130 0 0 04 May 2025
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling Daniil Medyakov Gleb Molodtsov S. Chezhegov Alexey Rebrikov Aleksandr Beznosikov 103 0 0 21 Feb 2025
FedTLU: Federated Learning with Targeted Layer Updates Jong-Ik Park Carlee Joe-Wong FedML 84 0 0 28 Jan 2025
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML T. Dahan Kfir Y. Levy 65 0 0 17 Jan 2025
On the Performance Analysis of Momentum Method: A Frequency Domain Perspective Xianliang Li Jun Luo Zhiwei Zheng Hanxiao Wang Li Luo Lingkun Wen Linlong Wu Sheng Xu 72 0 0 29 Nov 2024
Extended convexity and smoothness and their applications in deep learning Binchuan Qi Wei Gong Li Li 61 0 0 08 Oct 2024
DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction Xinwei Zhang Zhiqi Bu Borja Balle Mingyi Hong Meisam Razaviyayn Vahab Mirrokni 76 2 0 04 Oct 2024
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 43 2 0 28 Sep 2024
On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy Saber Malekmohammadi G. Farnadi 24 2 0 26 Sep 2024
Gradient-Free Method for Heavily Constrained Nonconvex Optimization Wanli Shi Hongchang Gao Bin Gu 16 5 0 31 Aug 2024
SOREL: A Stochastic Algorithm for Spectral Risks Minimization Yuze Ge Rujun Jiang 38 0 0 19 Jul 2024
Private Zeroth-Order Nonsmooth Nonconvex Optimization Qinzi Zhang Hoang Tran Ashok Cutkosky 40 4 0 27 Jun 2024
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum H. Cai Sulaiman A. Alghunaim Ali H.Sayed 43 1 0 18 Jun 2024
Random Scaling and Momentum for Non-smooth Non-convex Optimization Qinzi Zhang Ashok Cutkosky 37 4 0 16 May 2024
Robust Decentralized Learning with Local Updates and Gradient Tracking Sajjad Ghiasvand Amirhossein Reisizadeh Mahnoosh Alizadeh Ramtin Pedarsani 39 3 0 02 May 2024
Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs Swetha Ganesh Washim Uddin Mondal Vaneet Aggarwal 44 3 0 02 Apr 2024
Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries Swetha Ganesh Jiayu Chen Gugan Thoppe Vaneet Aggarwal FedML 64 1 0 15 Mar 2024
Non-Convex Stochastic Composite Optimization with Polyak Momentum Yuan Gao Anton Rodomanov Sebastian U. Stich 31 6 0 05 Mar 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 16 39 0 05 Feb 2024
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis Jie Hao Xiaochuan Gong Mingrui Liu 30 7 0 17 Jan 2024
FedDRO: Federated Compositional Optimization for Distributionally Robust Learning Prashant Khanduri Chengyin Li Rafi Ibn Sultan Yao Qiang Joerg Kliewer Dongxiao Zhu 36 0 0 21 Nov 2023
A Coefficient Makes SVRG Effective Yida Yin Zhiqiu Xu Zhiyuan Li Trevor Darrell Zhuang Liu 33 1 0 09 Nov 2023
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization Zhen Qin Zhishuai Liu Pan Xu 20 1 0 24 Oct 2023
Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates Ahmad Rammal Kaja Gruntkowska Nikita Fedin Eduard A. Gorbunov Peter Richtárik 45 5 0 15 Oct 2023
Variance-reduced accelerated methods for decentralized stochastic double-regularized nonconvex strongly-concave minimax problems Gabriel Mancino-Ball Yangyang Xu 20 8 0 14 Jul 2023
Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality Ziyang Wei Wanrong Zhu Wei Biao Wu 27 3 0 13 Jul 2023
Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation Davoud Ataee Tarzanagh Mingchen Li Pranay Sharma Samet Oymak 24 0 0 02 Jun 2023
Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features Aleksandr Beznosikov David Dobre Gauthier Gidel 25 5 0 23 Apr 2023
On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level Feihu Huang 27 18 0 07 Mar 2023
Enhanced Adaptive Gradient Algorithms for Nonconvex-PL Minimax Optimization Feihu Huang Chunyu Xuan Xinrui Wang Siqi Zhang Songcan Chen 28 7 0 07 Mar 2023
A One-Sample Decentralized Proximal Algorithm for Non-Convex Stochastic Composite Optimization Tesi Xiao Xuxing Chen Krishnakumar Balasubramanian Saeed Ghadimi 36 10 0 20 Feb 2023
Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks Arun Ganesh Daogao Liu Sewoong Oh Abhradeep Thakurta ODL 27 12 0 20 Feb 2023
Breaking the Lower Bound with (Little) Structure: Acceleration in Non-Convex Stochastic Optimization with Heavy-Tailed Noise Zijian Liu Jiawei Zhang Zhengyuan Zhou 32 12 0 14 Feb 2023
Near-Optimal Non-Convex Stochastic Optimization under Generalized Smoothness Zijian Liu Srikanth Jagabathula Zhengyuan Zhou 16 4 0 13 Feb 2023
Stochastic Dimension-reduced Second-order Methods for Policy Optimization Jinsong Liu Chen Xie Qinwen Deng Dongdong Ge Yi-Li Ye 24 1 0 28 Jan 2023
Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization Le‐Yu Chen Jing Xu Luo Luo 31 15 0 16 Jan 2023
Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction Bowen Lei Dongkuan Xu Ruqi Zhang Shuren He Bani Mallick 29 6 0 09 Jan 2023
Stochastic Inexact Augmented Lagrangian Method for Nonconvex Expectation Constrained Optimization Zichong Li Pinzhuo Chen Sijia Liu Songtao Lu Yangyang Xu 27 17 0 19 Dec 2022
Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis Quoc Tran-Dinh Marten van Dijk 34 0 0 19 Dec 2022
Variance-Reduced Conservative Policy Iteration Naman Agarwal Brian Bullins Karan Singh 24 3 0 12 Dec 2022
Decentralized Stochastic Gradient Descent Ascent for Finite-Sum Minimax Problems Hongchang Gao 24 16 0 06 Dec 2022
An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization Le‐Yu Chen Haishan Ye Luo Luo 65 5 0 05 Dec 2022
Adaptive Stochastic Optimisation of Nonconvex Composite Objectives Weijia Shao F. Sivrikaya S. Albayrak 13 0 0 21 Nov 2022
Fast Adaptive Federated Bilevel Optimization Feihu Huang FedML 20 7 0 02 Nov 2022
Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models Shujian Zhang Chengyue Gong Xingchao Liu RALM 43 6 0 02 Nov 2022
Momentum Aggregation for Private Non-convex ERM Hoang Tran Ashok Cutkosky 26 14 0 12 Oct 2022
SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities Aleksandr Beznosikov Alexander Gasnikov 34 2 0 12 Oct 2022
SENDER: SEmi-Nonlinear Deep Efficient Reconstructor for Extraction Canonical, Meta, and Sub Functional Connectivity in the Human Brain Wei Zhang Yun-Jian Bao 9 0 0 12 Sep 2022
Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization Wei Jiang Gang Li Yibo Wang Lijun Zhang Tianbao Yang 32 16 0 18 Jul 2022
Multi-block Min-max Bilevel Optimization with Applications in Multi-task Deep AUC Maximization Quanqi Hu Yongjian Zhong Tianbao Yang 25 16 0 01 Jun 2022