PatrickStar: Parallel Training of Pre-trained Models via Chunk-based
Memory Management

PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management

12 August 2021

Yang You

Papers citing "PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management"

5 / 5 papers shown

Title
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 28 0 0 12 Jun 2024
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent Xiaonan Nie Yi Liu Fangcheng Fu J. Xue Dian Jiao Xupeng Miao Yangyu Tao Bin Cui MoE 24 16 0 06 Mar 2023
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models Yuliang Liu Shenggui Li Jiarui Fang Yan Shao Boyuan Yao Yang You OffRL 19 7 0 06 Feb 2023
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,817 0 17 Sep 2019