Title
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo Zachary B. Charles Gabriel Teston Lucio Dery Keith Rush Nova Fallen Zachary Garrett Arthur Szlam Arthur Douillard 459 6 0 12 Mar 2025
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model Loubna Ben Allal Anton Lozhkov Elie Bakouch Gabriel Martín Blázquez Guilherme Penedo ... Cyril Zakka Mathieu Morlon Colin Raffel Leandro von Werra Thomas Wolf MoE 164 46 0 04 Feb 2025
How Does Critical Batch Size Scale in Pre-training? Hanlin Zhang Depen Morwani Nikhil Vyas Jingfeng Wu Difan Zou Udaya Ghai Dean Phillips Foster Sham Kakade 189 18 0 29 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 620 1 0 07 Oct 2024
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws Nikhil Sardana Jacob P. Portes Sasha Doubov Jonathan Frankle LRM 415 88 0 31 Dec 2023