Broken Neural Scaling Laws

Broken Neural Scaling Laws

26 October 2022

Ethan Caballero

David M. Krueger

Papers citing "Broken Neural Scaling Laws"

18 / 18 papers shown

Title
Position: Enough of Scaling LLMs! Lets Focus on Downscaling Ayan Sengupta Yash Goel Tanmoy Chakraborty 34 0 0 02 May 2025
Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective Yuko Nakagi Keigo Tada Sota Yoshino Shinji Nishimoto Yu Takagi LRM 37 0 0 28 Feb 2025
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 46 4 0 30 Sep 2024
Scaling Laws in Linear Regression: Compute, Parameters, and Data Licong Lin Jingfeng Wu Sham Kakade Peter L. Bartlett Jason D. Lee LRM 20 15 0 12 Jun 2024
From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks Jacob Russin Sam Whitman McGrath Danielle J. Williams Lotem Elber-Dorozko AI4CE 59 2 0 24 May 2024
In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization Herilalaina Rakotoarison Steven Adriaensen Neeratyoy Mallik Samir Garibov Eddie Bergman Frank Hutter AI4CE 19 8 0 25 Apr 2024
YaART: Yet Another ART Rendering Technology Sergey Kastryulin Artem Konev Alexander Shishenya Eugene Lyapustin Artem Khurshudov ... Dmitrii Kornilov Mikhail Romanov Artem Babenko Sergei Ovcharenko Valentin Khrulkov EGVM 22 1 0 08 Apr 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 37 58 0 25 Mar 2024
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws Nikhil Sardana Jacob P. Portes Sasha Doubov Jonathan Frankle LRM 220 64 0 31 Dec 2023
LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models Ahmad Faiz S. Kaneda Ruhan Wang Rita Osi Parteek Sharma Fan Chen Lei Jiang 10 55 0 25 Sep 2023
Uncovering Neural Scaling Laws in Molecular Representation Learning Dingshuo Chen Yanqiao Zhu Jieyu Zhang Yuanqi Du Zhixun Li Qiang Liu Shu Wu Liang Wang 21 15 0 15 Sep 2023
Scaling laws for single-agent reinforcement learning Jacob Hilton Jie Tang John Schulman 11 20 0 31 Jan 2023
Revisiting Neural Scaling Laws in Language and Vision Ibrahim M. Alabdulmohsin Behnam Neyshabur Xiaohua Zhai 148 101 0 13 Sep 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Truthful AI: Developing and governing AI that does not lie Owain Evans Owen Cotton-Barratt Lukas Finnveden Adam Bales Avital Balwit Peter Wills Luca Righetti William Saunders HILM 220 107 0 13 Oct 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 239 2,554 0 04 May 2021
E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials Simon L. Batzner Albert Musaelian Lixin Sun Mario Geiger J. Mailoa M. Kornbluth N. Molinari Tess E. Smidt Boris Kozinsky 188 1,218 0 08 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 4,424 0 23 Jan 2020