Colossal-Auto: Unified Automation of Parallelization and Activation
Checkpoint for Large-scale Models

Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models

6 February 2023

Yang You

Papers citing "Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models"

5 / 5 papers shown

Title
Towards Green AI: Current status and future research Christian Clemm Lutz Stobbe Kishan Wimalawarne Jan Druschke 38 1 0 01 May 2024
UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming Hao Lin Ke Wu Jie Li Jun Yu Li Wu-Jun Li 24 1 0 31 Jul 2023
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 130 0 14 Jul 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 399 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019