Optimizing Distributed Training on Frontier for Large Language Models

Optimizing Distributed Training on Frontier for Large Language Models

20 December 2023

Feiyi Wang

Prasanna Balaprakash

Papers citing "Optimizing Distributed Training on Frontier for Large Language Models"

11 / 11 papers shown

Title
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers Siddharth Singh Prajwal Singhania Aditya K. Ranjan John Kirchenbauer Jonas Geiping ... Abhimanyu Hans Manli Shu Aditya Tomar Tom Goldstein A. Bhatele 92 2 0 12 Feb 2025
RedPajama: an Open Dataset for Training Large Language Models Maurice Weber Daniel Y. Fu Quentin Anthony Yonatan Oren S. Adams ... Tri Dao Percy Liang Christopher Ré Irina Rish Ce Zhang 94 52 0 19 Nov 2024
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation Rambod Azimi Rishav Rishav M. Teichmann Samira Ebrahimi Kahou ALM 18 0 0 28 Oct 2024
Comprehensive Performance Modeling and System Design Insights for Foundation Models Shashank Subramanian Ermal Rrapaj Peter Harrington Smeet Chheda S. Farrell Brian Austin Samuel Williams N. Wright W. Bhimji 23 0 0 30 Sep 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 69 7 0 29 Jul 2024
Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars Wesley Brewer Aditya Kashi Sajal Dash A. Tsaris Junqi Yin Mallikarjun Shankar Feiyi Wang 30 0 0 24 Jun 2024
AI-coupled HPC Workflow Applications, Middleware and Performance Wes Brewer Ana Gainaru Frédéric Suter Feiyi Wang M. Emani S. Jha 25 10 0 20 Jun 2024
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability Xiao Wang A. Tsaris Siyan Liu Jong Youl Choi Ming Fan Wei Zhang Ju Yin M. Ashfaq Dan Lu Prasanna Balaprakash 14 7 0 23 Apr 2024
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models Haochen Tan Zhijiang Guo Zhan Shi Lu Xu Zhili Liu ... Xiaoguang Li Yasheng Wang Lifeng Shang Qun Liu Linqi Song 22 12 0 26 Jan 2024
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 94 0 14 Jul 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 4,424 0 23 Jan 2020