A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks

14 January 2019

Papers citing "A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks"

50 / 66 papers shown

Title
Exploring Scaling Laws for Local SGD in Large Language Model Training Qiaozhi He Xiaomin Zhuang Zhihua Wu 20 4 0 20 Sep 2024
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning Zichen Tang Junlin Huang Rudan Yan Yuxin Wang Zhenheng Tang S. Shi Amelie Chi Zhou Xiaowen Chu FedML 50 2 0 27 Aug 2024
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression Wenchen Han S. Vargaftik Michael Mitzenmacher Brad Karp Ran Ben-Basat 33 2 0 01 Jul 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 29 6 0 09 Apr 2024
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning Daegun Yoon Sangyoon Oh 38 0 0 21 Feb 2024
Accelerating Distributed Deep Learning using Lossless Homomorphic Compression Haoyu Li Yuchen Xu Jiayi Chen Rohit Dwivedula Wenfei Wu Keqiang He Aditya Akella Daehyeok Kim FedML AI4CE 15 4 0 12 Feb 2024
Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning Guangfeng Yan Tan Li Yuanzhang Xiao Hanxu Hou Linqi Song MQ 19 0 0 02 Feb 2024
Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learning Guangfeng Yan Tan Li Tian-Shing Lan Kui Wu Linqi Song 19 6 0 12 Dec 2023
FedCode: Communication-Efficient Federated Learning via Transferring Codebooks Saeed Khalilian Gourtani Vasileios Tsouvalas T. Ozcelebi N. Meratnia FedML 26 5 0 15 Nov 2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression Lin Meng Yuzhong Sun Weimin Li 23 1 0 08 Nov 2023
Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt Zihao Zhao Zhenpeng Shi Yang Liu Wenbo Ding FedML 29 1 0 04 Oct 2023
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training Daegun Yoon Sangyoon Oh 21 1 0 02 Oct 2023
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity Pengyun Yue Hanzheng Zhao Cong Fang Di He Liwei Wang Zhouchen Lin Song-Chun Zhu 26 1 0 23 Sep 2023
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs Zhenheng Tang Yuxin Wang Xin He Longteng Zhang Xinglin Pan ... Rongfei Zeng Kaiyong Zhao S. Shi Bingsheng He Xiaowen Chu 28 29 0 03 Sep 2023
A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency Jiawei Shao Zijian Li Wenqiang Sun Tailin Zhou Yuchang Sun Lumin Liu Zehong Lin Yuyi Mao Jun Zhang FedML 26 22 0 20 Jul 2023
Accelerating Distributed ML Training via Selective Synchronization S. Tyagi Martin Swany FedML 19 3 0 16 Jul 2023
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification Daegun Yoon Sangyoon Oh 20 1 0 07 Jul 2023
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning Lin Zhang Longteng Zhang S. Shi X. Chu Bo-wen Li OffRL 15 7 0 15 Jun 2023
Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning Guangfeng Yan Tan Li Kui Wu Linqi Song 19 12 0 26 Apr 2023
SparDL: Distributed Deep Learning Training with Efficient Sparse Communication Minjun Zhao Yichen Yin Yuren Mao Qing Liu Lu Chen Yunjun Gao 11 1 0 03 Apr 2023
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost Jinfan Chen Shigang Li Ran Guo Jinhui Yuan Torsten Hoefler 15 2 0 17 Jan 2023
Top-k data selection via distributed sample quantile inference Xu Zhang M. Vasconcelos 20 1 0 01 Dec 2022
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment Daegun Yoon Sangyoon Oh 18 0 0 18 Sep 2022
Towards Efficient Communications in Federated Learning: A Contemporary Survey Zihao Zhao Yuzhu Mao Yang Liu Linqi Song Ouyang Ye Xinlei Chen Wenbo Ding FedML 43 59 0 02 Aug 2022
Byzantine Fault Tolerance in Distributed Machine Learning : a Survey Djamila Bouhata Hamouma Moumen Moumen Hamouma Ahcène Bounceur AI4CE 23 7 0 05 May 2022
Near-Optimal Sparse Allreduce for Distributed Deep Learning Shigang Li Torsten Hoefler 18 50 0 19 Jan 2022
AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descent Nhuong V. Nguyen Song Han 8 2 0 27 Dec 2021
Collaborative Learning over Wireless Networks: An Introductory Overview Emre Ozfatura Deniz Gunduz H. Vincent Poor 17 11 0 07 Dec 2021
FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update Junyu Luo Jianlei Yang Xucheng Ye Xin Guo Weisheng Zhao FedML 18 14 0 20 Aug 2021
A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration Lingfei Dai Boyu Diao Chao Li Yongjun Xu 28 5 0 13 Aug 2021
BenchENAS: A Benchmarking Platform for Evolutionary Neural Architecture Search Xiangning Xie Yuqiao Liu Yanan Sun Gary G. Yen Bing Xue Mengjie Zhang 46 16 0 09 Aug 2021
ErrorCompensatedX: error compensation for variance reduced algorithms Hanlin Tang Yao Li Ji Liu Ming Yan 14 9 0 04 Aug 2021
Neural Distributed Source Coding Jay Whang Alliot Nagle Anish Acharya Hyeji Kim A. Dimakis 30 20 0 05 Jun 2021
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training Chia-Yu Chen Jiamin Ni Songtao Lu Xiaodong Cui Pin-Yu Chen ... Naigang Wang Swagath Venkataramani Vijayalakshmi Srinivasan Wei Zhang K. Gopalakrishnan 27 66 0 21 Apr 2021
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed Conglong Li A. A. Awan Hanlin Tang Samyam Rajbhandari Yuxiong He 32 33 0 13 Apr 2021
On the Utility of Gradient Compression in Distributed Training Systems Saurabh Agarwal Hongyi Wang Shivaram Venkataraman Dimitris Papailiopoulos 12 46 0 28 Feb 2021
Peering Beyond the Gradient Veil with Distributed Auto Differentiation Bradley T. Baker Aashis Khanal Vince D. Calhoun Barak A. Pearlmutter Sergey Plis 18 1 0 18 Feb 2021
Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach Junshan Zhang Na Li M. Dedeoglu FedML 26 41 0 16 Feb 2021
Distributed Online Learning for Joint Regret with Communication Constraints Dirk van der Hoeven Hédi Hadiji T. Erven 15 5 0 15 Feb 2021
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed Hanlin Tang Shaoduo Gan A. A. Awan Samyam Rajbhandari Conglong Li Xiangru Lian Ji Liu Ce Zhang Yuxiong He AI4CE 27 84 0 04 Feb 2021
FEDZIP: A Compression Framework for Communication-Efficient Federated Learning Amirhossein Malekijoo Mohammad Javad Fadaeieslam Hanieh Malekijou Morteza Homayounfar F. Alizadeh-Shabdiz Reza Rawassizadeh FedML 16 52 0 02 Feb 2021
Time-Correlated Sparsification for Communication-Efficient Federated Learning Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 34 47 0 21 Jan 2021
Bayesian Federated Learning over Wireless Networks Seunghoon Lee Chanhoo Park Songnam Hong Yonina C. Eldar Namyoon Lee 18 23 0 31 Dec 2020
Distributed Sparse SGD with Majority Voting Kerem Ozfatura Emre Ozfatura Deniz Gunduz FedML 33 4 0 12 Nov 2020
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification Saurabh Agarwal Hongyi Wang Kangwook Lee Shivaram Venkataraman Dimitris Papailiopoulos 26 25 0 29 Oct 2020
Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes Marten van Dijk Nhuong V. Nguyen Toan N. Nguyen Lam M. Nguyen Quoc Tran-Dinh Phuong Ha Nguyen FedML 34 10 0 27 Oct 2020
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters S. Shi Xianhao Zhou Shutao Song Xingyao Wang Zilin Zhu ... Chenyang Guo Bo Yang Zhibo Chen Yongjian Wu X. Chu GNN 13 55 0 20 Oct 2020
PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks Qing Ye Y. Han Yanan Sun Jiancheng Lv 18 3 0 06 Sep 2020
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm Hanlin Tang Shaoduo Gan Samyam Rajbhandari Xiangru Lian Ji Liu Yuxiong He Ce Zhang 20 8 0 26 Aug 2020
DBS: Dynamic Batch Size For Distributed Deep Neural Network Training Qing Ye Yuhao Zhou Mingjia Shi Yanan Sun Jiancheng Lv 14 11 0 23 Jul 2020

A Distributed Synchronous SGD Algorithm with Global Top-kkk Sparsification for Low Bandwidth Networks

Papers citing "A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks"

A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks