Title
Understanding Stragglers in Large Model Training Using What-if Analysis Jinkun Lin Ziheng Jiang Zuquan Song Sida Zhao Menghan Yu ... Shuguang Wang Haibin Lin Xin Liu Aurojit Panda Jinyang Li 22 0 0 09 May 2025
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling Xiao Wang Jong Youl Choi Takuya Kurihaya Isaac Lyngaas Hong-Jun Yoon ... Dali Wang Peter Thornton Prasanna Balaprakash M. Ashfaq Dan Lu 28 0 0 07 May 2025
Galvatron: An Automatic Distributed System for Efficient Foundation Model Training Xinyi Liu Y. Wang Shenhan Zhu Fangcheng Fu Qingshuo Liu Guangming Lin Bin Cui GNN 88 0 0 30 Apr 2025
MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core Dennis Liu Zijie Yan Xin Yao Tong Liu V. Korthikanti ... Jiajie Yao Chandler Zhou David Wu Xipeng Li J. Yang MoE 56 0 0 21 Apr 2025
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model Team Seawead Ceyuan Yang Zhijie Lin Yang Zhao Shanchuan Lin ... Zuquan Song Zhenheng Yang Jiashi Feng Jianchao Yang Lu Jiang DiffM 81 1 0 11 Apr 2025
AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation Tuhin Chakrabarty Philippe Laban C. Wu 32 1 0 10 Apr 2025
SEA-LION: Southeast Asian Languages in One Network Raymond Ng Thanh Ngan Nguyen Yuli Huang Ngee Chia Tai Wai Yi Leong ... David Ong Tat-Wee B. Liu William-Chandra Tjhi Erik Cambria Leslie Teo 36 11 0 08 Apr 2025
TAGC: Optimizing Gradient Communication in Distributed Transformer Training Igor Polyakov Alexey Dukhanov Egor Spirin 39 0 0 08 Apr 2025
Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training Yijie Zheng Bangjun Xiao Lei Shi Xiaoyang Li Faming Wu Tianyu Li Xuefeng Xiao Y. Zhang Y. Wang Shouda Liu MLLM MoE 67 1 0 31 Mar 2025
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control Nvidia Hassan Abu Alhaija Jose M. Alvarez Maciej Bala Tiffany Cai ... Yuchong Ye Xiaodong Yang X. Yang Xiaohui Zeng Yu Zeng VGen 90 1 0 18 Mar 2025
Florenz: Scaling Laws for Systematic Generalization in Vision-Language Models Julian Spravil Sebastian Houben Sven Behnke VLM 68 0 0 12 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao X. Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 72 0 0 09 Mar 2025
EuroBERT: Scaling Multilingual Encoders for European Languages Nicolas Boizard Hippolyte Gisserot-Boukhlef Duarte M. Alves André F. T. Martins Ayoub Hammal ... Maxime Peyrard Nuno M. Guerreiro Patrick Fernandes Ricardo Rei Pierre Colombo 93 1 0 07 Mar 2025
Mixtera: A Data Plane for Foundation Model Training Maximilian Böther Xiaozhe Yao Tolga Kerimoglu Ana Klimovic Viktor Gsteiger Ana Klimovic MoE 81 0 0 27 Feb 2025
NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation Zhiyuan Liu Yanchen Luo Han Huang Enzhi Zhang Sihang Li Junfeng Fang Yaorui Shi X. Wang Kenji Kawaguchi Tat-Seng Chua 100 3 0 18 Feb 2025
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale Fan Zhou Zengzhi Wang Qian Liu Junlong Li Pengfei Liu ALM 100 15 0 17 Feb 2025
GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He Peng Ye Yuchen Ren Yuan Yuan Lei Chen AI4TS AI4CE 120 0 0 13 Feb 2025
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers Siddharth Singh Prajwal Singhania Aditya K. Ranjan John Kirchenbauer Jonas Geiping ... Abhimanyu Hans Manli Shu Aditya Tomar Tom Goldstein A. Bhatele 94 2 0 12 Feb 2025
360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation Hamed Firooz Maziar Sanjabi Adrian Englhardt Aman Gupta Ben Levine ... Xiaoling Zhai Ya Xu Yu Wang Yun Dai Yun Dai ALM 42 3 0 27 Jan 2025
Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training Jonghytun Lee Yongqin Wang Rachit Rajat M. Annavaram 50 0 0 20 Jan 2025
LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models Hieu Man Nghia Trung Ngo Viet Dac Lai Ryan Rossi Franck Dernoncourt T. Nguyen 115 0 0 01 Jan 2025
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism Tim Tsz-Kit Lau Weijian Li Chenwei Xu Han Liu Mladen Kolar 109 0 0 30 Dec 2024
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 145 3 0 20 Nov 2024
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 28 4 0 24 Oct 2024
Liger Kernel: Efficient Triton Kernels for LLM Training Pin-Lun Hsu Yun Dai Vignesh Kothapalli Qingquan Song Shao Tang Siyu Zhu Steven Shimizu Shivam Sahni Haowen Ning Yanning Chen 39 26 0 14 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 100 0 0 07 Oct 2024
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis Ulyana Piterbarg Lerrel Pinto Rob Fergus SyDa 37 2 0 03 Oct 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 64 1 0 16 Sep 2024
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer Jinghan Yao Sam Ade Jacobs Masahiro Tanaka Olatunji Ruwase A. Shafi D. Panda 28 2 0 30 Aug 2024
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining Dongyang Liu Shitian Zhao Le Zhuo Weifeng Lin Yu Qiao Xinyue Li Qi Qin Yu Qiao Hongsheng Li Peng Gao MLLM 62 48 0 05 Aug 2024
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts Xi Victoria Lin Akshat Shrivastava Liang Luo Srinivasan Iyer Mike Lewis Gargi Gosh Luke Zettlemoyer Armen Aghajanyan MoE 38 20 0 31 Jul 2024
Small Molecule Optimization with Large Language Models Philipp Guevorguian Menua Bedrosian Tigran Fahradyan Gayane Chilingaryan Hrant Khachatrian Armen Aghajanyan 27 1 0 26 Jul 2024
DistML.js: Installation-free Distributed Deep Learning Framework for Web Browsers Masatoshi Hidaka Tomohiro Hashimoto Yuto Nishizawa Tatsuya Harada 24 2 0 01 Jul 2024
ProTrain: Efficient LLM Training via Memory-Aware Techniques Hanmei Yang Jin Zhou Yao Fu Xiaoqun Wang Ramine Roane Hui Guan Tongping Liu VLM 28 0 0 12 Jun 2024
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Peize Sun Yi Jiang Shoufa Chen Shilong Zhang Bingyue Peng Ping Luo Zehuan Yuan VLM 60 220 0 10 Jun 2024
Performance-Aligned LLMs for Generating Fast Code Daniel Nichols Pranav Polasam Harshitha Menon Aniruddha Marathe T. Gamblin A. Bhatele 27 8 0 29 Apr 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 68 2 0 03 Apr 2024
Partitioned Neural Network Training via Synthetic Intermediate Labels C. V. Karadag Nezih Topaloglu 29 0 0 17 Mar 2024
DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation Sunghyeon Woo Baeseong Park Byeongwook Kim Minjung Jo S. Kwon Dongsuk Jeon Dongsoo Lee 57 2 0 27 Feb 2024
LLM4SecHW: Leveraging Domain Specific Large Language Model for Hardware Debugging Weimin Fu Kaichen Yang R. Dutta Xiaolong Guo Gang Qu 19 23 0 28 Jan 2024
FlexModel: A Framework for Interpretability of Distributed Large Language Models Matthew Choi Muhammad Adil Asif John Willes David Emerson AI4CE ALM 22 1 0 05 Dec 2023
PrivateLoRA For Efficient Privacy Preserving LLM Yiming Wang Yu Lin Xiaodong Zeng Guannan Zhang 32 11 0 23 Nov 2023
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model Avamarie Brueggeman Andrea Madotto Zhaojiang Lin Tushar Nagarajan Matt Smith ... Peyman Heidari Yue Liu Kavya Srinet Babak Damavandi Anuj Kumar MLLM 29 92 0 27 Sep 2023
Benchmarks for Detecting Measurement Tampering Fabien Roger Ryan Greenblatt Max Nadeau Buck Shlegeris Nate Thomas 26 2 0 29 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 88 10,947 0 18 Jul 2023
Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training Shengwei Li Zhiquan Lai Yanqi Hao Weijie Liu Ke-shi Ge Xiaoge Deng Dongsheng Li KaiCheng Lu 11 10 0 25 May 2023
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021