Training Deep Nets with Sublinear Memory Cost

21 April 2016

Papers citing "Training Deep Nets with Sublinear Memory Cost"

50 / 193 papers shown

Title
You Only Look One Step: Accelerating Backpropagation in Diffusion Sampling with Gradient Shortcuts Hongkun Dou Zeyu Li Xingyu Jiang H. Li Lijun Yang Wen Yao Yue Deng DiffM 38 0 0 12 May 2025
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients Yezhen Wang Zhouhao Yang Brian K Chen Fanyi Pu Bo-wen Li Tianyu Gao Kenji Kawaguchi 41 0 0 03 May 2025
RayZer: A Self-supervised Large View Synthesis Model Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao ... Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos 65 0 0 01 May 2025
Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User X. Wang Chunxuan Xia Junyi Li Fanzhe Meng Lei Huang Jinpeng Wang Wayne Xin Zhao Ji-Rong Wen 63 0 0 29 Apr 2025
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model Team Seawead Ceyuan Yang Zhijie Lin Yang Zhao Shanchuan Lin ... Zuquan Song Zhenheng Yang Jiashi Feng Jianchao Yang Lu Jiang DiffM 83 1 0 11 Apr 2025
Kimi-VL Technical Report Kimi Team Angang Du B. Yin Bowei Xing Bowen Qu ... Zhiqi Huang Zihao Huang Zijia Zhao Z. Chen Zongyu Lin MLLM VLM MoE 198 1 0 10 Apr 2025
MaskAttn-UNet: A Mask Attention-Driven Framework for Universal Low-Resolution Image Segmentation Anzhe Cheng Chenzhong Yin Yu Chang Heng Ping Shixuan Li Shahin Nazarian Paul Bogdan SSeg 86 0 0 11 Mar 2025
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Anh Tong Thanh Nguyen-Tang Dongeun Lee Duc Nguyen Toan M. Tran David Hall Cheongwoong Kang Jaesik Choi 35 0 0 03 Mar 2025
GPU Memory Usage Optimization for Backward Propagation in Deep Network Training Ding-Yong Hong Tzu-Hsien Tsai Ning Wang Pangfeng Liu Jan-Jan Wu 44 0 0 18 Feb 2025
GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysis Angelos Zavras Dimitrios Michail Xiao Xiang Zhu Begum Demir Ioannis Papoutsis VLM 86 0 0 13 Feb 2025
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers Siddharth Singh Prajwal Singhania Aditya K. Ranjan John Kirchenbauer Jonas Geiping ... Abhimanyu Hans Manli Shu Aditya Tomar Tom Goldstein A. Bhatele 96 2 0 12 Feb 2025
Memory-Efficient Fine-Tuning of Transformers via Token Selection Antoine Simoulin Namyong Park Xiaoyi Liu Grey Yang 110 0 0 31 Jan 2025
Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders Rui Chen Jianfeng Zhang Yixun Liang Guan Luo Weiyu Li Jiarui Liu Xiu Li Xiaoxiao Long Jiashi Feng P. Tan 71 11 0 23 Dec 2024
GME: Improving Universal Multimodal Retrieval by Multimodal LLMs Xin Zhang Yanzhao Zhang Wen Xie Mingxin Li Ziqi Dai Dingkun Long Pengjun Xie Meishan Zhang Wenjie Li M. Zhang 116 7 0 22 Dec 2024
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance Chu Myaet Thwal Ye Lin Tun Minh N. H. Nguyen Eui-nam Huh Choong Seon Hong VLM 74 0 0 05 Dec 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 83 1 0 26 Nov 2024
Large Language Model with Region-guided Referring and Grounding for CT Report Generation Z. Chen Yequan Bie Haibo Jin Hao Chen 179 0 0 23 Nov 2024
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 195 3 0 20 Nov 2024
Navigating Extremes: Dynamic Sparsity in Large Output Spaces Nasib Ullah Erik Schultheis Mike Lasby Yani Andrew Ioannou Rohit Babbar 33 0 0 05 Nov 2024
CompAct: Compressed Activations for Memory-Efficient LLM Training Yara Shamshoum Nitzan Hodos Yuval Sieradzki Assaf Schuster MQ VLM 44 0 0 20 Oct 2024
Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting Chunlin Tian Li Li Kahou Tam Yebo Wu Chengzhong Xu FedML 24 1 0 12 Oct 2024
AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments Cheng Fang Sicong Liu Zimu Zhou Bin Guo Jiaqi Tang Ke Ma Zhiwen Yu TTA 31 1 0 10 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 52 4 0 09 Oct 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 42 12 0 11 Sep 2024
Efficient fine-tuning of 37-level GraphCast with the Canadian global deterministic analysis Christopher Subich 36 1 0 26 Aug 2024
Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval Guangyuan Ma Yongliang Ma Xing Wu Zhenpeng Su Ming Zhou Songlin Hu OOD 39 2 0 20 Aug 2024
Understanding the Performance and Estimating the Cost of LLM Fine-Tuning Yuchen Xia Jiho Kim Yuhan Chen Haojie Ye Souvik Kundu Cong Hao Nishil Talati MoE 35 20 0 08 Aug 2024
ADBM: Adversarial diffusion bridge model for reliable adversarial purification Xiao-Li Li Wenxuan Sun Huanran Chen Qiongxiu Li Yining Liu Yingzhe He Jie Shi Xiaolin Hu AAML 58 7 0 01 Aug 2024
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun Xinhao Li Karan Dalal Jiarui Xu Arjun Vikram ... Xinlei Chen Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin 58 93 0 05 Jul 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 37 1 0 24 Jun 2024
Adding Conditional Control to Diffusion Models with Reinforcement Learning Yulai Zhao Masatoshi Uehara Gabriele Scalia Tommaso Biancalani Sergey Levine Ehsan Hajiramezanali Ehsan Hajiramezanali AI4CE 54 3 0 17 Jun 2024
Optimizing Large Model Training through Overlapped Activation Recomputation Ping Chen Wenjie Zhang Shuibing He Yingjie Gu Zhuwei Peng ... Yi Zheng Zhefeng Wang Yanlong Yin Gang Chen Gang Chen 33 5 0 13 Jun 2024
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 28 0 0 12 Jun 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 32 3 0 28 May 2024
CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems Abbas Ghaddar David Alfonso-Hermelo Philippe Langlais Mehdi Rezagholizadeh Boxing Chen Prasanna Parthasarathi 34 0 0 24 May 2024
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data Sachin Mehta Maxwell Horton Fartash Faghri Mohammad Hossein Sekhavat Mahyar Najibi Mehrdad Farajtabar Oncel Tuzel Mohammad Rastegari VLM CLIP 38 6 0 24 Apr 2024
Simple and Scalable Strategies to Continually Pre-train Large Language Models Adam Ibrahim Benjamin Thérien Kshitij Gupta Mats L. Richter Quentin Anthony Timothée Lesort Eugene Belilovsky Irina Rish KELM CLL 44 51 0 13 Mar 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 43 173 0 06 Mar 2024
GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence Kundan Krishna S. Ramprasad Prakhar Gupta Byron C. Wallace Zachary Chase Lipton Jeffrey P. Bigham HILM KELM SyDa 52 9 0 19 Feb 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 13 39 0 05 Feb 2024
Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness Samaneh Shafee A. Bessani Pedro M. Ferreira 26 19 0 26 Jan 2024
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference Xuanlei Zhao Shenggan Cheng Guangyang Lu Jiarui Fang Hao Zhou Bin Jia Ziming Liu Yang You MQ 17 3 0 19 Jan 2024
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture M. A. D. L. Balaguer Vinamra Benara Renato Luiz de Freitas Cunha Roberto de M. Estevao Filho Todd Hendry ... Morris Sharp B. Silva Swati Sharma Vijay Aski Ranveer Chandra FaML 30 81 0 16 Jan 2024
Unicron: Economizing Self-Healing LLM Training at Scale Tao He Xue Li Zhibin Wang Kun Qian Jingbo Xu Wenyuan Yu Jingren Zhou 14 14 0 30 Dec 2023
Optimizing Diffusion Noise Can Serve As Universal Motion Priors Korrawe Karunratanakul Konpat Preechakul Emre Aksan Thabo Beeler Supasorn Suwajanakorn Siyu Tang DiffM 29 37 0 19 Dec 2023
Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply Chengting Yu Fengzhao Zhang Hanzhi Ma Aili Wang Er-ping Li 22 1 0 12 Dec 2023
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 35 31 0 08 Dec 2023
Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices Beibei Zhang Hongwei Zhu Feng Gao Zhihui Yang Xiaoyang Sean Wang 27 1 0 07 Dec 2023
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction Lei Guan Dongsheng Li Jiye Liang Wenjian Wang Wenjian Wang Xicheng Lu 20 1 0 01 Dec 2023
End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames Shuming Liu Chen-Da Liu-Zhang Chen Zhao Bernard Ghanem 33 25 0 28 Nov 2023