8-Bit Approximations for Parallelism in Deep Learning

14 November 2015

Tim Dettmers

Papers citing "8-Bit Approximations for Parallelism in Deep Learning"

24 / 24 papers shown

Title
Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics Cong Xu Wenbin Liang Mo Yu Anan Liu Kaipeng Zhang Lizhuang Ma Yufei Guo Jun Wang Wentao Zhang MQ 57 0 0 01 May 2025
FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients Leming Shen Qiang Yang Kaiyan Cui Yuanqing Zheng Xiao-Yong Wei Jianwei Liu Jinsong Han FedML 79 11 0 28 Feb 2025
Irrational Complex Rotations Empower Low-bit Optimizers Zhen Tian Wayne Xin Zhao Zhicheng Dou MQ 46 0 0 22 Jan 2025
No Need to Talk: Asynchronous Mixture of Language Models Anastasiia Filippova Angelos Katharopoulos David Grangier Ronan Collobert MoE 44 0 0 04 Oct 2024
4-bit Shampoo for Memory-Efficient Network Training Sike Wang Jia Li Pan Zhou Hua Huang MQ 44 6 0 28 May 2024
Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models Alan Chan Ben Bucknall Herbie Bradley David M. Krueger 16 6 0 22 Dec 2023
Better Question-Answering Models on a Budget Yudhanjaya Wijeratne Ishan Marikar ALM 25 0 0 24 Apr 2023
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient Max Ryabinin Tim Dettmers Michael Diskin Alexander Borzunov MoE 35 31 0 27 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 35 5 0 06 Jan 2023
Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression Xiaoyun Li Ping Li FedML 36 4 0 25 Nov 2022
Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters Yang Xiang Zhihua Wu Weibao Gong Siyu Ding Xianjie Mo ... Yue Yu Ge Li Yu Sun Yanjun Ma Dianhai Yu 24 5 0 19 May 2022
FastSGD: A Fast Compressed SGD Framework for Distributed Machine Learning Keyu Yang Lu Chen Zhihao Zeng Yunjun Gao 28 9 0 08 Dec 2021
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 34 276 0 06 Oct 2021
Rethinking gradient sparsification as total error minimization Atal Narayan Sahu Aritra Dutta A. Abdelmoniem Trambak Banerjee Marco Canini Panos Kalnis 45 56 0 02 Aug 2021
On the Utility of Gradient Compression in Distributed Training Systems Saurabh Agarwal Hongyi Wang Shivaram Venkataraman Dimitris Papailiopoulos 38 46 0 28 Feb 2021
Pruning and Quantization for Deep Neural Network Acceleration: A Survey Tailin Liang C. Glossner Lei Wang Shaobo Shi Xiaotong Zhang MQ 150 676 0 24 Jan 2021
An Overview of Neural Network Compression James OÑeill AI4CE 45 98 0 05 Jun 2020
A Review of Privacy-preserving Federated Learning for the Internet-of-Things Christopher Briggs Zhong Fan Péter András 28 15 0 24 Apr 2020
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts Max Ryabinin Anton I. Gusev FedML 27 48 0 10 Feb 2020
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning Aritra Dutta El Houcine Bergou A. Abdelmoniem Chen-Yu Ho Atal Narayan Sahu Marco Canini Panos Kalnis 33 77 0 19 Nov 2019
$λ$ -NIC: Interactive Serverless Compute on Programmable SmartNICs Sean Choi M. Shahbaz B. Prabhakar M. Rosenblum 28 47 0 26 Sep 2019
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches Md. Zahangir Alom T. Taha C. Yakopcic Stefan Westberg P. Sidike Mst Shamima Nasrin B. Van Essen A. Awwal V. Asari VLM 29 874 0 03 Mar 2018
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis Tal Ben-Nun Torsten Hoefler GNN 33 703 0 26 Feb 2018
On Scale-out Deep Learning Training for Cloud and HPC Srinivas Sridharan K. Vaidyanathan Dhiraj D. Kalamkar Dipankar Das Mikhail E. Smorkalov ... Dheevatsa Mudigere Naveen Mellempudi Sasikanth Avancha Bharat Kaul Pradeep Dubey BDL 26 30 0 24 Jan 2018