SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

13 January 2018

Papers citing "SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks"

31 / 31 papers shown

Title
Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting Chunlin Tian Li Li Kahou Tam Yebo Wu Chengzhong Xu FedML 24 1 0 12 Oct 2024
AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments Cheng Fang Sicong Liu Zimu Zhou Bin Guo Jiaqi Tang Ke Ma Zhiwen Yu TTA 31 1 0 10 Oct 2024
Optimizing Large Model Training through Overlapped Activation Recomputation Ping Chen Wenjie Zhang Shuibing He Yingjie Gu Zhuwei Peng ... Yi Zheng Zhefeng Wang Yanlong Yin Gang Chen Gang Chen 35 5 0 13 Jun 2024
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 30 0 0 12 Jun 2024
Partitioned Neural Network Training via Synthetic Intermediate Labels C. V. Karadag Nezih Topaloglu 34 1 0 17 Mar 2024
QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources Zhikai Li Xiaoxuan Liu Banghua Zhu Zhen Dong Qingyi Gu Kurt Keutzer MQ 32 7 0 11 Oct 2023
Full Parameter Fine-tuning for Large Language Models with Limited Resources Kai Lv Yuqing Yang Tengxiao Liu Qi-jie Gao Qipeng Guo Xipeng Qiu 45 126 0 16 Jun 2023
An Evaluation of Memory Optimization Methods for Training Neural Networks Xiaoxuan Liu Siddharth Jha Alvin Cheung 23 0 0 26 Mar 2023
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent Xiaonan Nie Yi Liu Fangcheng Fu J. Xue Dian Jiao Xupeng Miao Yangyu Tao Bin Cui MoE 24 16 0 06 Mar 2023
Mimose: An Input-Aware Checkpointing Planner for Efficient Training on GPU Jian-He Liao Mingzhen Li Qingxiao Sun Jiwei Hao F. Yu ... Ye Tao Zicheng Zhang Hailong Yang Zhongzhi Luan D. Qian 23 4 0 06 Sep 2022
Do Residual Neural Networks discretize Neural Ordinary Differential Equations? Michael E. Sander Pierre Ablin Gabriel Peyré 32 25 0 29 May 2022
SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures Yunjae Lee Jin-Won Chung Minsoo Rhu GNN 29 48 0 10 May 2022
Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards Youngeun Kwon Minsoo Rhu 21 27 0 10 May 2022
Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers Youjie Li Amar Phanishayee D. Murray Jakub Tarnawski N. Kim 11 19 0 02 Feb 2022
Understanding Performance Problems in Deep Learning Systems Junming Cao Bihuan Chen Chao Sun Longjie Hu Shuai Wu Xin Peng 30 27 0 03 Dec 2021
COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression Sian Jin Chengming Zhang Xintong Jiang Yunhe Feng Hui Guan Guanpeng Li S. Song Dingwen Tao 25 23 0 18 Nov 2021
Sinkformers: Transformers with Doubly Stochastic Attention Michael E. Sander Pierre Ablin Mathieu Blondel Gabriel Peyré 29 76 0 22 Oct 2021
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training Jianfei Chen Lianmin Zheng Z. Yao Dequan Wang Ion Stoica Michael W. Mahoney Joseph E. Gonzalez MQ 19 74 0 29 Apr 2021
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning Samyam Rajbhandari Olatunji Ruwase Jeff Rasley Shaden Smith Yuxiong He GNN 32 367 0 16 Apr 2021
Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture S. Min Kun Wu Sitao Huang Mert Hidayetouglu Jinjun Xiong Eiman Ebrahimi Deming Chen Wen-mei W. Hwu GNN 10 67 0 04 Mar 2021
Momentum Residual Neural Networks Michael E. Sander Pierre Ablin Mathieu Blondel Gabriel Peyré 24 56 0 15 Feb 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 168 414 0 18 Jan 2021
A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression Sian Jin Guanpeng Li S. Song Dingwen Tao AI4CE 29 12 0 18 Nov 2020
Block-term Tensor Neural Networks Jinmian Ye Guangxi Li Di Chen Haiqin Yang Shandian Zhe Zenglin Xu 24 30 0 10 Oct 2020
Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA M. Wahib Haoyu Zhang Truong Thao Nguyen Aleksandr Drozd Jens Domke Lingqi Zhang Ryousei Takano Satoshi Matsuoka OODD 34 23 0 26 Aug 2020
The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism Yosuke Oyama N. Maruyama Nikoli Dryden Erin McCarthy P. Harrington J. Balewski Satoshi Matsuoka Peter Nugent B. Van Essen 3DV AI4CE 24 37 0 25 Jul 2020
The Ramifications of Making Deep Neural Networks Compact N. Jha Sparsh Mittal Govardhan Mattela 16 14 0 26 Jun 2020
Profiling based Out-of-core Hybrid Method for Large Neural Networks Yuki Ito Haruki Imai Tung D. Le Yasushi Negishi K. Kawachiya R. Matsumiya Toshio Endo 14 9 0 11 Jul 2019
Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism Nikoli Dryden N. Maruyama Tom Benson Tim Moon M. Snir B. Van Essen 18 49 0 15 Mar 2019
AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks Jinrong Guo Wantao Liu Wang Wang Q. Lu Songlin Hu Jizhong Han Ruixuan Li 11 9 0 21 Jan 2019
Wireless Network Intelligence at the Edge Jihong Park S. Samarakoon M. Bennis Mérouane Debbah 19 518 0 07 Dec 2018