Rethinking gradient sparsification as total error minimization

2 August 2021

Papers citing "Rethinking gradient sparsification as total error minimization"

35 / 35 papers shown

Title
Accelerated Distributed Optimization with Compression and Error Feedback Yuan Gao Anton Rodomanov Jeremy Rack Sebastian U. Stich 38 0 0 11 Mar 2025
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation Shih-yang Liu Huck Yang Nai Chit Fung Nai Chit Fung Hongxu Yin ... Jan Kautz Yu-Chun Wang Pavlo Molchanov Min-Hung Chen Min-Hung Chen MQ 29 0 0 28 Oct 2024
Novel Gradient Sparsification Algorithm via Bayesian Inference Ali Bereyhi B. Liang G. Boudreau Ali Afana 28 2 0 23 Sep 2024
Sparse Incremental Aggregation in Multi-Hop Federated Learning Sourav Mukherjee N. Razmi Armin Dekorsy P. Popovski Bho Matthiesen FedML 19 1 0 25 Jul 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 19 3 0 28 May 2024
FedMPQ: Secure and Communication-Efficient Federated Learning with Multi-codebook Product Quantization Xu Yang Jiapeng Zhang Qifeng Zhang Zhuo Tang MQ 22 0 0 21 Apr 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 26 6 0 09 Apr 2024
SignSGD with Federated Voting Chanho Park H. Vincent Poor Namyoon Lee FedML 27 1 0 25 Mar 2024
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning Daegun Yoon Sangyoon Oh 30 0 0 21 Feb 2024
Communication-Efficient Distributed Learning with Local Immediate Error Compensation Yifei Cheng Li Shen Linli Xu Xun Qian Shiwei Wu Yiming Zhou Tie Zhang Dacheng Tao Enhong Chen 32 0 0 19 Feb 2024
Demystifying the Myths and Legends of Nonconvex Convergence of SGD Aritra Dutta El Houcine Bergou Soumia Boucherouite Nicklas Werge M. Kandemir Xin Li 13 0 0 19 Oct 2023
Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates Ahmad Rammal Kaja Gruntkowska Nikita Fedin Eduard A. Gorbunov Peter Richtárik 32 5 0 15 Oct 2023
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training Daegun Yoon Sangyoon Oh 19 1 0 02 Oct 2023
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification Daegun Yoon Sangyoon Oh 15 1 0 07 Jul 2023
Communication Resources Constrained Hierarchical Federated Learning for End-to-End Autonomous Driving Weihua Kou Shuai Wang Guangxu Zhu Bin Luo Yingxian Chen Derrick Wing Kwan Ng Yik-Chung Wu 11 12 0 28 Jun 2023
Leveraging The Edge-to-Cloud Continuum for Scalable Machine Learning on Decentralized Data A. Abdelmoniem 22 1 0 19 Jun 2023
A Guide Through the Zoo of Biased SGD Yury Demidovich Grigory Malinovsky Igor Sokolov Peter Richtárik 25 22 0 25 May 2023
Momentum Provably Improves Error Feedback! Ilyas Fatkhullin A. Tyurin Peter Richtárik 8 19 0 24 May 2023
Unified analysis of SGD-type methods Eduard A. Gorbunov 19 2 0 29 Mar 2023
Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence Yuhao Zhou Mingjia Shi Yuanxi Li Qing Ye Yanan Sun Jiancheng Lv 8 3 0 27 Feb 2023
Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity Richeng Jin Xiaofan He C. Zhong Zhaoyang Zhang Tony Q. S. Quek H. Dai FedML 16 1 0 19 Feb 2023
L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning Mohammadreza Alimohammadi I. Markov Elias Frantar Dan Alistarh 22 5 0 31 Oct 2022
Personalized Federated Learning with Communication Compression El Houcine Bergou Konstantin Burlachenko Aritra Dutta Peter Richtárik FedML 69 9 0 12 Sep 2022
Towards Energy-Aware Federated Learning on Battery-Powered Clients Amna Arouj A. Abdelmoniem 16 26 0 09 Aug 2022
Towards Efficient Communications in Federated Learning: A Contemporary Survey Zihao Zhao Yuzhu Mao Yang Liu Linqi Song Ouyang Ye Xinlei Chen Wenbo Ding FedML 41 59 0 02 Aug 2022
Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression Xinmeng Huang Yiming Chen W. Yin Kun Yuan 14 25 0 08 Jun 2022
Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation Rustem Islamov Xun Qian Slavomír Hanzely M. Safaryan Peter Richtárik 22 16 0 07 Jun 2022
Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top Eduard A. Gorbunov Samuel Horváth Peter Richtárik Gauthier Gidel AAML 13 0 0 01 Jun 2022
FedShuffle: Recipes for Better Use of Local Work in Federated Learning Samuel Horváth Maziar Sanjabi Lin Xiao Peter Richtárik Michael G. Rabbat FedML 25 21 0 27 Apr 2022
Distributed Methods with Absolute Compression and Error Compensation Marina Danilova Eduard A. Gorbunov 11 5 0 04 Mar 2022
OLIVE: Oblivious Federated Learning on Trusted Execution Environment against the risk of sparsification Fumiyuki Kato Yang Cao Masatoshi Yoshikawa FedML 6 5 0 15 Feb 2022
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 49 73 0 26 Jan 2021
Linearly Converging Error Compensated SGD Eduard A. Gorbunov D. Kovalev Dmitry Makarenko Peter Richtárik 158 77 0 23 Oct 2020
Detached Error Feedback for Distributed SGD with Random Sparsification An Xu Heng-Chiao Huang 23 9 0 11 Apr 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019