Demystifying the Communication Characteristics for Distributed Transformer Models

19 August 2024

Mustafa Abduljabbar

Hari Subramoni

Papers citing "Demystifying the Communication Characteristics for Distributed Transformer Models"

3 / 3 papers shown

Title
GPU-centric Communication Schemes for HPC and ML Applications Naveen Namashivayam GNN 25 0 0 31 Mar 2025
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019