Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large
Language Model Training

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training

5 June 2024

Xu Han

Cheng Yang

Zhiyuan Liu

Chuan Shi

Papers citing "Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training"

7 / 7 papers shown

Title
How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks Yusen Wu Junwu Xiong Xiaotie Deng LLMAG 36 0 0 04 Apr 2025
PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization Xinyi Wan Penghui Qi Guangxing Huang Jialin Li Min Lin 34 0 0 03 Mar 2025
APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs Yuxiang Huang Mingye Li Xu Han Chaojun Xiao Weilin Zhao Sun Ao Hao Zhou Jie Zhou Zhiyuan Liu Maosong Sun 39 0 0 17 Feb 2025
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 59 7 0 29 Jul 2024
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 94 0 14 Jul 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 155 399 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019