A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

22 May 2023

Papers citing "A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs"

5 / 5 papers shown

Title
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He S. Feizi A. Bhatele 32 13 0 04 Jun 2024
Nemotron-4 15B Technical Report Jupinder Parmar Shrimai Prabhumoye Joseph Jennings M. Patwary Sandeep Subramanian ... Ashwath Aithal Oleksii Kuchaiev M. Shoeybi Jonathan Cohen Bryan Catanzaro 31 22 0 26 Feb 2024
Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control Zhigang Wang Xu Zhang Ning Wang Chuanfei Xu Jie Nie Zhiqiang Wei Yu Gu Ge Yu 11 0 0 21 Jan 2024
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019