Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
Single GPU

Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a Single GPU

11 March 2024

Changyue Liao

Mo Sun

Zihan Yang

Zeke Wang

Papers citing "Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a Single GPU"

3 / 3 papers shown

Title
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 399 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019