Training Deep Nets with Sublinear Memory Cost

21 April 2016

Papers citing "Training Deep Nets with Sublinear Memory Cost"

50 / 197 papers shown

Title
HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddle Guoxia Wang Xiaomin Fang Zhihua Wu Yiqun Liu Yang Xue Yingfei Xiang Dianhai Yu Fan Wang Yanjun Ma 28 31 0 12 Jul 2022
Transforming PageRank into an Infinite-Depth Graph Neural Network Andreas Roth Thomas Liebig GNN 34 14 0 01 Jul 2022
RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network Vitaliy Chiley Vithursan Thangarasa Abhay Gupta Anshul Samar Joel Hestness D. DeCoste 48 8 0 28 Jun 2022
Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities Zejiang Shen Kyle Lo L. Yu N. Dahlberg Margo Schlanger Doug Downey ELM AILaw 37 43 0 22 Jun 2022
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning Yi-Lin Sung Jaemin Cho Mohit Bansal VLM 21 234 0 13 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 58 2,023 0 27 May 2022
ETAD: Training Action Detection End to End on a Laptop Shuming Liu Mengmeng Xu Chen Zhao Xu Zhao Bernard Ghanem 44 6 0 14 May 2022
Reducing Activation Recomputation in Large Transformer Models V. Korthikanti Jared Casper Sangkug Lym Lawrence C. McAfee M. Andersch M. Shoeybi Bryan Catanzaro AI4CE 27 256 0 10 May 2022
TALLFormer: Temporal Action Localization with a Long-memory Transformer Feng Cheng Gedas Bertasius ViT 24 91 0 04 Apr 2022
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection Junjie Huang Guan Huang 27 332 0 31 Mar 2022
Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions David Bieber Rishab Goel Daniel Zheng Hugo Larochelle Daniel Tarlow 16 15 0 07 Mar 2022
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection Hao Zhang Feng Li Shilong Liu Lei Zhang Hang Su Jun Zhu L. Ni H. Shum ViT 54 1,373 0 07 Mar 2022
R-GCN: The R Could Stand for Random Vic Degraeve Gilles Vandewiele F. Ongenae Sofie Van Hoecke GNN 23 13 0 04 Mar 2022
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours Shenggan Cheng Xuanlei Zhao Guangyang Lu Bin-Rui Li Zhongming Yu Tian Zheng R. Wu Xiwen Zhang Jian Peng Yang You AI4CE 19 30 0 02 Mar 2022
DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training Joya Chen Kai Xu Yuhui Wang Yifei Cheng Angela Yao 19 7 0 28 Feb 2022
Survey on Large Scale Neural Network Training Julia Gusak Daria Cherniuk Alena Shilova A. Katrutsa Daniel Bershatsky ... Lionel Eyraud-Dubois Oleg Shlyazhko Denis Dimitrov Ivan V. Oseledets Olivier Beaumont 22 10 0 21 Feb 2022
Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision Priya Goyal Quentin Duval Isaac Seessel Mathilde Caron Ishan Misra Levent Sagun Armand Joulin Piotr Bojanowski VLM SSL 26 110 0 16 Feb 2022
Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers Youjie Li Amar Phanishayee D. Murray Jakub Tarnawski N. Kim 11 19 0 02 Feb 2022
Tutorial on amortized optimization Brandon Amos OffRL 75 43 0 01 Feb 2022
ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation Han Zhang Weichong Yin Yewei Fang Lanxin Li Boqiang Duan Zhihua Wu Yu Sun Hao Tian Hua-Hong Wu Haifeng Wang 27 58 0 31 Dec 2021
Ultrasound Speckle Suppression and Denoising using MRI-derived Normalizing Flow Priors Vincent van de Schaft Ruud J. G. van Sloun OOD MedIm 16 6 0 24 Dec 2021
Efficient Large Scale Language Modeling with Mixtures of Experts Mikel Artetxe Shruti Bhosale Naman Goyal Todor Mihaylov Myle Ott ... Jeff Wang Luke Zettlemoyer Mona T. Diab Zornitsa Kozareva Ves Stoyanov MoE 54 188 0 20 Dec 2021
Long Context Question Answering via Supervised Contrastive Learning Avi Caciularu Ido Dagan Jacob Goldberger Arman Cohan RALM 19 23 0 16 Dec 2021
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 18 139 0 10 Dec 2021
Mesa: A Memory-saving Training Framework for Transformers Zizheng Pan Peng Chen Haoyu He Jing Liu Jianfei Cai Bohan Zhuang 23 20 0 22 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng-Wei Zhang Li Dong Furu Wei B. Guo ViT 52 1,746 0 18 Nov 2021
COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression Sian Jin Chengming Zhang Xintong Jiang Yunhe Feng Hui Guan Guanpeng Li S. Song Dingwen Tao 23 23 0 18 Nov 2021
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale Arun Babu Changhan Wang Andros Tjandra Kushal Lakhotia Qiantong Xu ... Yatharth Saraf J. Pino Alexei Baevski Alexis Conneau Michael Auli SSL 23 656 0 17 Nov 2021
Gradients are Not All You Need Luke Metz C. Freeman S. Schoenholz Tal Kachman 28 93 0 10 Nov 2021
FILIP: Fine-grained Interactive Language-Image Pre-Training Lewei Yao Runhu Huang Lu Hou Guansong Lu Minzhe Niu Hang Xu Xiaodan Liang Zhenguo Li Xin Jiang Chunjing Xu VLM CLIP 28 614 0 09 Nov 2021
BitTrain: Sparse Bitmap Compression for Memory-Efficient Training on the Edge Abdelrahman I. Hosny Marina Neseem Sherief Reda MQ 33 4 0 29 Oct 2021
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training Yongbin Li Hongxin Liu Zhengda Bian Boxiang Wang Haichen Huang Fan Cui Chuan-Qing Wang Yang You GNN 19 143 0 28 Oct 2021
Hydra: A System for Large Multi-Model Deep Learning Kabir Nagrecha Arun Kumar MoE AI4CE 38 5 0 16 Oct 2021
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding Junlong Li Yiheng Xu Lei Cui Furu Wei VLM 3DGS 23 59 0 16 Oct 2021
Partial Variable Training for Efficient On-Device Federated Learning Tien-Ju Yang Dhruv Guliani F. Beaufays Giovanni Motta FedML 19 25 0 11 Oct 2021
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning Shaohua Wu Xudong Zhao Tong Yu Rongguo Zhang C. Shen ... Feng Li Hong Zhu Jiangang Luo Liang Xu Xuanwei Zhang ALM 18 59 0 10 Oct 2021
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 17 269 0 06 Oct 2021
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management Jiarui Fang Zilin Zhu Shenggui Li Hui Su Yang Yu Jie Zhou Yang You VLM 29 24 0 12 Aug 2021
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 131 0 14 Jul 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 184 411 0 14 Jul 2021
Feature Alignment as a Generative Process T. S. Farias Jonas Maziero DiffM BDL 16 1 0 23 Jun 2021
CPM-2: Large-scale Cost-effective Pre-trained Language Models Zhengyan Zhang Yuxian Gu Xu Han Shengqi Chen Chaojun Xiao ... Minlie Huang Wentao Han Yang Liu Xiaoyan Zhu Maosong Sun MoE 26 86 0 20 Jun 2021
SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models Zaccharie Ramzi Florian Mannel Shaojie Bai Jean-Luc Starck P. Ciuciu Thomas Moreau 29 28 0 01 Jun 2021
Doc2Dict: Information Extraction as Text Generation Benjamin Townsend Eamon Ito-Fisher Lily Zhang Madison May 28 7 0 16 May 2021
GSPMD: General and Scalable Parallelization for ML Computation Graphs Yuanzhong Xu HyoukJoong Lee Dehao Chen Blake A. Hechtman Yanping Huang ... Noam M. Shazeer Shibo Wang Tao Wang Yonghui Wu Zhifeng Chen MoE 28 127 0 10 May 2021
Poolingformer: Long Document Modeling with Pooling Attention Hang Zhang Yeyun Gong Yelong Shen Weisheng Li Jiancheng Lv Nan Duan Weizhu Chen 35 98 0 10 May 2021
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers Pradeep Dasigi Kyle Lo Iz Beltagy Arman Cohan Noah A. Smith Matt Gardner RALM 31 279 0 07 May 2021
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning Samyam Rajbhandari Olatunji Ruwase Jeff Rasley Shaden Smith Yuxiong He GNN 30 366 0 16 Apr 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan M. Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 11 645 0 09 Apr 2021
No frame left behind: Full Video Action Recognition X. Liu S. Pintea F. Karimi Nejadasl O. Booij J. C. V. Gemert 19 40 0 29 Mar 2021