signSGD: Compressed Optimisation for Non-Convex Problems

13 February 2018

Jeremy Bernstein

Yu-Xiang Wang

Kamyar Azizzadenesheli

Papers citing "signSGD: Compressed Optimisation for Non-Convex Problems"

50 / 148 papers shown

Title
Convergence Analysis of Asynchronous Federated Learning with Gradient Compression for Non-Convex Optimization Diying Yang Yingwei Hou Danyang Xiao Weigang Wu FedML 39 0 0 28 Apr 2025
AlphaGrad: Non-Linear Gradient Normalization Optimizer Soham Sane ODL 48 0 0 22 Apr 2025
Accelerated Distributed Optimization with Compression and Error Feedback Yuan Gao Anton Rodomanov Jeremy Rack Sebastian U. Stich 43 0 0 11 Mar 2025
FOCUS: First Order Concentrated Updating Scheme Yizhou Liu Ziming Liu Jeff Gore ODL 108 1 0 21 Jan 2025
Grams: Gradient Descent with Adaptive Momentum Scaling Yang Cao Xiaoyu Li Zhao-quan Song ODL 85 2 0 22 Dec 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 64 0 0 26 Nov 2024
Cautious Optimizers: Improving Training with One Line of Code Kaizhao Liang Lizhang Chen B. Liu Qiang Liu ODL 106 5 0 25 Nov 2024
Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis Zhijie Chen Qiaobo Li A. Banerjee FedML 30 0 0 11 Nov 2024
Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees Aleksandar Armacki Shuhua Yu Pranay Sharma Gauri Joshi Dragana Bajović D. Jakovetić S. Kar 55 2 0 17 Oct 2024
From Gradient Clipping to Normalization for Heavy Tailed SGD Florian Hübler Ilyas Fatkhullin Niao He 40 5 0 17 Oct 2024
DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs Yingsong Luo Ling Chen MQ 21 0 0 16 Oct 2024
The Optimization Landscape of SGD Across the Feature Learning Strength Alexander B. Atanasov Alexandru Meterez James B. Simon C. Pehlevan 43 2 0 06 Oct 2024
Faster Acceleration for Steepest Descent Site Bai Brian Bullins ODL 31 0 0 28 Sep 2024
Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models Hui-Po Wang Mario Fritz 33 3 0 26 Sep 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 37 2 0 25 Sep 2024
Advancing Hybrid Defense for Byzantine Attacks in Federated Learning Kai Yue Richeng Jin Chau-Wai Wong H. Dai AAML 31 0 0 10 Sep 2024
Deconstructing What Makes a Good Optimizer for Language Models Rosie Zhao Depen Morwani David Brandfonbrener Nikhil Vyas Sham Kakade 42 17 0 10 Jul 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 59 3 0 24 May 2024
Leakage-Resilient and Carbon-Neutral Aggregation Featuring the Federated AI-enabled Critical Infrastructure Zehang Deng Ruoxi Sun Minhui Xue Sheng Wen S. Çamtepe Surya Nepal Yang Xiang 35 1 0 24 May 2024
Comparisons Are All You Need for Optimizing Smooth Functions Chenyi Zhang Tongyang Li AAML 24 1 0 19 May 2024
Flattened one-bit stochastic gradient descent: compressed distributed optimization with controlled variance A. Stollenwerk Laurent Jacques FedML 18 0 0 17 May 2024
KDk: A Defense Mechanism Against Label Inference Attacks in Vertical Federated Learning Marco Arazzi S. Nicolazzo Antonino Nocera FedML AAML 31 3 0 18 Apr 2024
Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks Matteo Tucat Anirbit Mukherjee Procheta Sen Mingfei Sun Omar Rivasplata MLT 31 1 0 12 Apr 2024
$Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization$ Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization Shuo Xie Zhiyuan Li OffRL 32 12 0 05 Apr 2024
Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance Qi Zhang Yi Zhou Shaofeng Zou 27 3 0 01 Apr 2024
SignSGD with Federated Voting Chanho Park H. Vincent Poor Namyoon Lee FedML 33 1 0 25 Mar 2024
Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functions Siyuan Zhang Nachuan Xiao Xin Liu 61 1 0 18 Mar 2024
Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes Xiaoxin Su Yipeng Zhou Laizhong Cui John C. S. Lui Jiangchuan Liu FedML 27 1 0 06 Feb 2024
RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation Weiying Xie Zixuan Wang Jitao Ma Daixun Li Yunsong Li 30 0 0 29 Dec 2023
Blind Federated Learning via Over-the-Air q-QAM Saeed Razavikia José Hélio da Cruz Júnior Carlo Fischione 35 3 0 07 Nov 2023
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing Terence Jie Chua Wen-li Yu Junfeng Zhao Kwok-Yan Lam FedML 24 5 0 26 Oct 2023
Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts Lizhang Chen Bo Liu Kaizhao Liang Qian Liu ODL 19 15 0 09 Oct 2023
Communication-Efficient Federated Learning via Regularized Sparse Random Networks Mohamad Mestoukirdi Omid Esrafilian David Gesbert Qianrui Li N. Gresset FedML 15 0 0 19 Sep 2023
Distributed Extra-gradient with Optimal Complexity and Communication Guarantees Ali Ramezani-Kebrya Kimon Antonakopoulos Igor Krawczuk Justin Deschenaux V. Cevher 32 2 0 17 Aug 2023
Private Federated Learning with Dynamic Power Control via Non-Coherent Over-the-Air Computation Anbang Zhang Shuaishuai Guo Shuai Liu 16 2 0 05 Aug 2023
Compressed Private Aggregation for Scalable and Robust Federated Learning over Massive Networks Natalie Lang Nir Shlezinger Rafael G. L. DÓliveira S. E. Rouayheb FedML 65 4 0 01 Aug 2023
Samplable Anonymous Aggregation for Private Federated Data Analysis Kunal Talwar Shan Wang Audra McMillan Vojta Jina Vitaly Feldman ... Congzheng Song Karl Tarbe Sebastian Vogt L. Winstrom Shundong Zhou FedML 30 13 0 27 Jul 2023
Communication-Efficient Split Learning via Adaptive Feature-Wise Compression Yong-Nam Oh Jaeho Lee Christopher G. Brinton Yo-Seb Jeon MQ 28 7 0 20 Jul 2023
Accelerating Distributed ML Training via Selective Synchronization S. Tyagi Martin Swany FedML 24 3 0 16 Jul 2023
Multiplicative update rules for accelerating deep learning training and increasing robustness Manos Kirtas Nikolaos Passalis Anastasios Tefas AAML OOD 26 2 0 14 Jul 2023
FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout Jingjing Xue Min Liu Sheng Sun Yuwei Wang Hui Jiang Xue Jiang 13 7 0 14 Jul 2023
An Efficient Virtual Data Generation Method for Reducing Communication in Federated Learning Cheng Yang Xue Yang Dongxian Wu Xiaohu Tang FedML 21 0 0 21 Jun 2023
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training Hong Liu Zhiyuan Li David Leo Wright Hall Percy Liang Tengyu Ma VLM 18 128 0 23 May 2023
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression Yutong He Xinmeng Huang Yiming Chen W. Yin Kun Yuan 26 7 0 12 May 2023
Computing Functions Over-the-Air Using Digital Modulations Saeed Razavikia J. M. B. D. Silva Carlo Fischione 39 13 0 01 Mar 2023
Mask-guided BERT for Few Shot Text Classification Wenxiong Liao Zheng Liu Haixing Dai Zihao Wu Yiyang Zhang ... Dajiang Zhu Tianming Liu Sheng R. Li Xiang Li Hongmin Cai VLM 36 39 0 21 Feb 2023
Breaking the Communication-Privacy-Accuracy Tradeoff with $f$ -Differential Privacy Richeng Jin Z. Su C. Zhong Zhaoyang Zhang Tony Q. S. Quek H. Dai FedML 19 2 0 19 Feb 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 50 350 0 13 Feb 2023
PolarAir: A Compressed Sensing Scheme for Over-the-Air Federated Learning Michail Gkagkos Krishna R. Narayanan J. Chamberland C. Georghiades 30 0 0 24 Jan 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 8 25 0 24 Jan 2023