AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

7 December 2017

Papers citing "AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training"

50 / 65 papers shown

Title
FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning Chaimaa Medjadji Sadi Alawadi Feras M. Awaysheh Guilain Leduc Sylvain Kubler Yves Le Traon FedML MQ 224 0 0 05 Nov 2025
Novel Gradient Sparsification Algorithm via Bayesian InferenceInternational Workshop on Machine Learning for Signal Processing (MLSP), 2024 Ali Bereyhi B. Liang G. Boudreau Ali Afana 185 5 0 23 Sep 2024
I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey Noah Lewis J. L. Bez Suren Byna 403 4 0 16 Apr 2024
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real SystemInternational Symposium on High-Performance Computer Architecture (HPCA), 2024 Hongsun Jang Jaeyong Song Jaewon Jung Jaeyoung Park Youngsok Kim Jinho Lee 135 27 0 11 Mar 2024
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning Daegun Yoon Sangyoon Oh 109 1 0 21 Feb 2024
Communication-Efficient Distributed Learning with Local Immediate Error Compensation Yifei Cheng Li Shen Linli Xu Xun Qian Shiwei Wu Yiming Zhou Tie Zhang Dacheng Tao Enhong Chen 168 1 0 19 Feb 2024
Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications Hongda Shen Eren Kurshan AAML 144 3 0 28 Dec 2023
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications Grant Wilkins Sheng Di Jon C. Calhoun Zilinghan Li Kibaek Kim Robert Underwood Richard Mortier Franck Cappello FedML 206 9 0 20 Dec 2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression Lin Meng Yuzhong Sun Weimin Li 186 4 0 08 Nov 2023
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN TrainingInternational Conference on High Performance Computing (HiPC), 2023 Daegun Yoon Sangyoon Oh 173 2 0 02 Oct 2023
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient SparsificationInternational Conference on Parallel Processing (ICPP), 2023 Daegun Yoon Sangyoon Oh 204 1 0 07 Jul 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication CompressionInternational Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2023 Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 238 37 0 24 Jan 2023
L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning Mohammadreza Alimohammadi I. Markov Elias Frantar Dan Alistarh 195 4 0 31 Oct 2022
Approximate Computing and the Efficient Machine Learning Expedition J. Henkel Hai Helen Li A. Raghunathan M. Tahoori Swagath Venkataramani Xiaoxuan Yang Georgios Zervakis 172 23 0 02 Oct 2022
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment Daegun Yoon Sangyoon Oh 131 0 0 18 Sep 2022
Reconciling Security and Communication Efficiency in Federated LearningIEEE Data Engineering Bulletin (DEB), 2022 Karthik Prasad Sayan Ghosh Graham Cormode Ilya Mironov Ashkan Yousefpour Pierre Stock FedML 144 11 0 26 Jul 2022
sqSGD: Locally Private and Communication Efficient Federated Learning Yan Feng Tao Xiong Ruofan Wu Lingjuan Lv Leilei Shi FedML 162 2 0 21 Jun 2022
Rate-Distortion Theoretic Bounds on Generalization Error for Distributed LearningNeural Information Processing Systems (NeurIPS), 2022 Romain Chor Abdellatif Zaidi Milad Sefidgaran FedML 257 17 0 06 Jun 2022
ByteComp: Revisiting Gradient Compression in Distributed Training Zhuang Wang Yanghua Peng Yibo Zhu T. Ng 168 2 0 28 May 2022
Efficient Direct-Connect Topologies for Collective CommunicationsSymposium on Networked Systems Design and Implementation (NSDI), 2022 Liangyu Zhao Siddharth Pal Tapan Chugh Weiyang Wang Jason Fantl P. Basu J. Khoury Arvind Krishnamurthy 373 14 0 07 Feb 2022
TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training JobsSymposium on Networked Systems Design and Implementation (NSDI), 2022 Weiyang Wang Moein Khazraee Zhizhen Zhong M. Ghobadi Zhihao Jia Dheevatsa Mudigere Ying Zhang A. Kewitsch 395 140 0 01 Feb 2022
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction Keshav Santhanam Omar Khattab Jon Saad-Falcon Christopher Potts Matei A. Zaharia 354 548 0 02 Dec 2021
Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters Adarsh Kumar Kausik Subramanian Shivaram Venkataraman Aditya Akella 105 6 0 20 Nov 2021
CGX: Adaptive System Support for Communication-Efficient Deep Learning I. Markov Hamidreza Ramezanikebrya Dan Alistarh GNN 241 5 0 16 Nov 2021
Resource-Efficient Federated LearningEuropean Conference on Computer Systems (EuroSys), 2021 A. Abdelmoniem Atal Narayan Sahu Marco Canini Suhaib A. Fahmy FedML 205 68 0 01 Nov 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 257 18 0 01 Nov 2021
Revealing and Protecting Labels in Distributed TrainingNeural Information Processing Systems (NeurIPS), 2021 Trung D. Q. Dang Om Thakkar Swaroop Indra Ramaswamy Rajiv Mathews Peter Chin Franccoise Beaufays 98 29 0 31 Oct 2021
A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration Lingfei Dai Boyu Diao Chao Li Yongjun Xu 158 5 0 13 Aug 2021
CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay CompensationInternational Conference on Parallel Processing (ICPP), 2021 Enda Yu Dezun Dong Yemao Xu Shuo Ouyang Xiangke Liao 108 5 0 21 Jun 2021
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed TrainingNeural Information Processing Systems (NeurIPS), 2021 Chia-Yu Chen Jiamin Ni Songtao Lu Xiaodong Cui Pin-Yu Chen ... Naigang Wang Swagath Venkataramani Vijayalakshmi Srinivasan Wei Zhang K. Gopalakrishnan 163 73 0 21 Apr 2021
MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training Zhuang Wang X. Wu T. Ng GNN 102 4 0 28 Mar 2021
Pufferfish: Communication-efficient Models At No Extra CostConference on Machine Learning and Systems (MLSys), 2021 Hongyi Wang Saurabh Agarwal Dimitris Papailiopoulos 129 67 0 05 Mar 2021
On the Impact of Device and Behavioral Heterogeneity in Federated Learning A. Abdelmoniem Chen-Yu Ho Pantelis Papageorgiou Muhammad Bilal Marco Canini FedML 144 18 0 15 Feb 2021
An Efficient Statistical-based Gradient Compression Technique for Distributed Training SystemsConference on Machine Learning and Systems (MLSys), 2021 A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 178 91 0 26 Jan 2021
DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication SchedulingIEEE Journal on Selected Areas in Communications (JSAC), 2021 Shangming Cai Dongsheng Wang Haixia Wang Yongqiang Lyu Guangquan Xu Xi Zheng A. Vasilakos 201 8 0 20 Jan 2021
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification Saurabh Agarwal Hongyi Wang Kangwook Lee Shivaram Venkataraman Dimitris Papailiopoulos 167 26 0 29 Oct 2020
Fairness-aware Agnostic Federated LearningSDM (SDM), 2020 Wei Du Depeng Xu Xintao Wu Hanghang Tong FedML 182 149 0 10 Oct 2020
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt Frank Schneider Philipp Hennig ODL 675 186 0 03 Jul 2020
Is Network the Bottleneck of Distributed Training? Zhen Zhang Chaokun Chang Yanghua Peng Yida Wang R. Arora Xin Jin 219 88 0 17 Jun 2020
Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data Chengxu Yang Qipeng Wang Mengwei Xu Shangguang Wang Kaigui Bian Yunxin Liu Xuanzhe Liu 138 24 0 12 Jun 2020
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang Shaoshuai Shi Wei Wang Yue Liu Xiaowen Chu 214 54 0 10 Mar 2020
Communication optimization strategies for distributed deep neural network training: A survey Shuo Ouyang Dezun Dong Yemao Xu Liquan Xiao 280 12 0 06 Mar 2020
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer SelectionIEEE International Conference on Distributed Computing Systems (ICDCS), 2020 Zhenheng Tang Shaoshuai Shi Xiaowen Chu FedML 142 63 0 22 Feb 2020
Communication-Efficient Edge AI: Algorithms and SystemsIEEE Communications Surveys and Tutorials (COMST), 2020 Yuanming Shi Kai Yang Tao Jiang Jun Zhang Khaled B. Letaief GNN 167 385 0 22 Feb 2020
Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning Yining Qi Zhihao Qu Song Guo Xin Gao Ruixuan Li Baoliu Ye FedML 120 9 0 22 Jan 2020
Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning ApproachIEEE International Conference on Distributed Computing Systems (ICDCS), 2020 Pengchao Han Maroun Touma K. Leung FedML 250 210 0 14 Jan 2020
Understanding Top-k Sparsification in Distributed Deep Learning Shaoshuai Shi Xiaowen Chu Ka Chun Cheung Simon See 302 115 0 20 Nov 2019
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence GuaranteesEuropean Conference on Artificial Intelligence (ECAI), 2019 Shaoshuai Shi Zhenheng Tang Qiang-qiang Wang Kaiyong Zhao Xiaowen Chu 209 27 0 20 Nov 2019
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep LearningAAAI Conference on Artificial Intelligence (AAAI), 2019 Aritra Dutta El Houcine Bergou A. Abdelmoniem Chen-Yu Ho Atal Narayan Sahu Marco Canini Panos Kalnis 154 85 0 19 Nov 2019
Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning XINYAN DAI Xiao Yan Kaiwen Zhou Han Yang K. K. Ng James Cheng Yu Fan FedML 132 49 0 12 Nov 2019