Elixir: Train a Large Language Model on a Small GPU Cluster

v1v2v3 (latest)

Elixir: Train a Large Language Model on a Small GPU Cluster

10 December 2022

Yang You

ArXiv (abs)PDF HTML Github (40917★)

Papers citing "Elixir: Train a Large Language Model on a Small GPU Cluster"

6 / 6 papers shown

Title
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Yang Liu 315 29 0 29 Jul 2024
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 192 3 0 12 Jun 2024
2BP: 2-Stage Backpropagation Christopher Rae Joseph K. L. Lee James Richings MoE MQ 99 0 0 28 May 2024
A Comparative Analysis of Distributed Training Strategies for GPT-2 Ishan Patwardhan Shubham Gandhi Om M. Khare Amit Joshi Suraj Sawant 256 3 0 24 May 2024
Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a Single GPU Changyue Liao Mo Sun Zihan Yang Kaiqi Chen Binhang Yuan Leilei Gan Zeke Wang 135 2 0 11 Mar 2024
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models Yuliang Liu Shenggui Li Jiarui Fang Yan Shao Boyuan Yao Yang You OffRL 182 10 0 06 Feb 2023