Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory

27 November 2019

Papers citing "Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory"

8 / 8 papers shown

Title
GPU Memory Usage Optimization for Backward Propagation in Deep Network Training Ding-Yong Hong Tzu-Hsien Tsai Ning Wang Pangfeng Liu Jan-Jan Wu 39 0 0 18 Feb 2025
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 28 0 0 12 Jun 2024
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models Yuliang Liu Shenggui Li Jiarui Fang Yan Shao Boyuan Yao Yang You OffRL 19 7 0 06 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 20 47 0 02 Feb 2023
Survey on Large Scale Neural Network Training Julia Gusak Daria Cherniuk Alena Shilova A. Katrutsa Daniel Bershatsky ... Lionel Eyraud-Dubois Oleg Shlyazhko Denis Dimitrov Ivan V. Oseledets Olivier Beaumont 22 10 0 21 Feb 2022
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management Jiarui Fang Zilin Zhu Shenggui Li Hui Su Yang Yu Jie Zhou Yang You VLM 26 24 0 12 Aug 2021
A Study of Checkpointing in Large Scale Training of Deep Neural Networks Elvis Rojas A. Kahira Esteban Meneses L. Bautista-Gomez Rosa M. Badia 11 22 0 01 Dec 2020
Dynamic Tensor Rematerialization Marisa Kirisame Steven Lyubomirsky Altan Haan Jennifer Brennan Mike He Jared Roesch Tianqi Chen Zachary Tatlock 11 93 0 17 Jun 2020