A Solvable Model of Neural Scaling Laws

30 October 2022

Papers citing "A Solvable Model of Neural Scaling Laws"

50 / 51 papers shown

Title
Superposition Yields Robust Neural Scaling Yizhou Liu Ziming Liu Jeff Gore MILM 10 0 0 15 May 2025
Learning curves theory for hierarchically compositional data with power-law distributed features Francesco Cagnetta Hyunmo Kang M. Wyart 24 0 0 11 May 2025
Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures Francesco Cagnetta Alessandro Favero Antonio Sclocchi M. Wyart 21 0 0 11 May 2025
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules Kairong Luo Haodong Wen Shengding Hu Zhenbo Sun Zhiyuan Liu Maosong Sun Kaifeng Lyu Wenguang Chen CLL 59 1 0 17 Mar 2025
Uncertainty Quantification From Scaling Laws in Deep Neural Networks Ibrahim Elsharkawy Yonatan Kahn Benjamin Hooberman UQCV 45 0 0 07 Mar 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 33 0 0 28 Feb 2025
Physics of Skill Learning Ziming Liu Yizhou Liu Eric J. Michaud Jeff Gore Max Tegmark 41 0 0 21 Jan 2025
Loss-to-Loss Prediction: Scaling Laws for All Datasets David Brandfonbrener Nikhil Anand Nikhil Vyas Eran Malach Sham Kakade 77 3 0 19 Nov 2024
Scaling Laws for Pre-training Agents and World Models Tim Pearce Tabish Rashid Dave Bignell Raluca Georgescu Sam Devlin Katja Hofmann LM&Ro 37 7 0 07 Nov 2024
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws M. E. Ildiz Halil Alperen Gozeten Ege Onur Taga Marco Mondelli Samet Oymak 54 2 0 24 Oct 2024
A Simple Model of Inference Scaling Laws Noam Levi LRM 32 6 0 21 Oct 2024
Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra Roman Worschech B. Rosenow 36 0 0 11 Oct 2024
Strong Model Collapse Elvis Dohmatob Yunzhen Feng Arjun Subramonian Julia Kempe 26 9 0 07 Oct 2024
How Feature Learning Can Improve Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 49 12 0 26 Sep 2024
Unified Neural Network Scaling Laws and Scale-time Equivalence Akhilan Boopathy Ila Fiete 30 0 0 09 Sep 2024
Risk and cross validation in ridge regression with correlated samples Alexander B. Atanasov Jacob A. Zavatone-Veth C. Pehlevan 27 4 0 08 Aug 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 50 19 0 27 Jun 2024
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations Rylan Schaeffer Victor Lecomte Dhruv Pai Andres Carranza Berivan Isik ... Yann LeCun SueYeon Chung Andrey Gromov Ravid Shwartz-Ziv Sanmi Koyejo 41 5 0 13 Jun 2024
Scaling Laws in Linear Regression: Compute, Parameters, and Data Licong Lin Jingfeng Wu Sham Kakade Peter L. Bartlett Jason D. Lee LRM 33 15 0 12 Jun 2024
Reconciling Kaplan and Chinchilla Scaling Laws Tim Pearce Jinyeop Song 32 7 0 12 Jun 2024
Hardness of Learning Neural Networks under the Manifold Hypothesis B. Kiani Jason Wang Melanie Weber 34 2 0 03 Jun 2024
Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Nets Khen Cohen Noam Levi Yaron Oz BDL 23 1 0 28 May 2024
Bayesian RG Flow in Neural Network Field Theories Jessica N. Howard Marc S. Klinger Anindita Maiti A. G. Stapleton 60 1 0 27 May 2024
Model Collapse Demystified: The Case of Regression Elvis Dohmatob Yunzhen Feng Julia Kempe 32 32 0 12 Feb 2024
A Dynamical Model of Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 44 36 0 02 Feb 2024
Scaling Laws in Jet Classification Joshua D. Batson Yonatan Kahn 16 2 0 04 Dec 2023
More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory James B. Simon Dhruva Karkada Nikhil Ghosh Mikhail Belkin AI4CE BDL 26 13 0 24 Nov 2023
The Universal Statistical Structure and Scaling Laws of Chaos and Turbulence Noam Levi Yaron Oz AI4CE 21 1 0 02 Nov 2023
Scaling Laws for Associative Memories Vivien A. Cabannes Elvis Dohmatob A. Bietti 11 19 0 04 Oct 2023
The semantic landscape paradigm for neural networks Shreyas Gokhale 21 2 0 18 Jul 2023
Spectral-Bias and Kernel-Task Alignment in Physically Informed Neural Networks Inbar Seroussi Asaf Miron Z. Ringel PINN 32 0 0 12 Jul 2023
Large Language Models Michael R Douglas LLMAG LM&MA 33 555 0 11 Jul 2023
The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets Noam Levi Yaron Oz 27 4 0 26 Jun 2023
Scaling MLPs: A Tale of Inductive Bias Gregor Bachmann Sotiris Anagnostidis Thomas Hofmann 29 38 0 23 Jun 2023
A Rainbow in Deep Network Black Boxes Florentin Guth Brice Ménard G. Rochette S. Mallat 14 10 0 29 May 2023
Structures of Neural Network Effective Theories cCaugin Ararat Tianji Cai Cem Tekin Zhengkang Zhang 47 7 0 03 May 2023
Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations Tengfei Xu Dachuan Liu Peng Hao Bo Wang 28 4 0 09 Apr 2023
Inductive biases in deep learning models for weather prediction Jannik Thümmel Matthias Karlbauer S. Otte C. Zarfl Georg Martius ... Thomas Scholten Ulrich Friedrich V. Wulfmeyer B. Goswami Martin Volker Butz AI4CE 31 4 0 06 Apr 2023
The Quantization Model of Neural Scaling Eric J. Michaud Ziming Liu Uzay Girit Max Tegmark MILM 22 77 0 23 Mar 2023
Unifying Grokking and Double Descent Peter W. Battaglia David Raposo Kelsey 30 31 0 10 Mar 2023
Learning curves for deep structured Gaussian feature models Jacob A. Zavatone-Veth C. Pehlevan MLT 10 11 0 01 Mar 2023
A Simplistic Model of Neural Scaling Laws: Multiperiodic Santa Fe Processes L. Debowski MILM 8 11 0 17 Feb 2023
Over-parametrization via Lifting for Low-rank Matrix Sensing: Conversion of Spurious Solutions to Strict Saddle Points Ziye Ma Igor Molybog Javad Lavaei Somayeh Sojoudi 11 3 0 15 Feb 2023
Bayesian Interpolation with Deep Linear Networks Boris Hanin Alexander Zlokapa 34 25 0 29 Dec 2022
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes Alexander B. Atanasov Blake Bordelon Sabarish Sainathan C. Pehlevan 14 26 0 23 Dec 2022
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions Hugo Cui Bruno Loureiro Florent Krzakala Lenka Zdeborová 38 10 0 29 Jan 2022
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 17 245 0 12 Feb 2021
Learning Curve Theory Marcus Hutter 128 58 0 08 Feb 2021
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? Tal Linzen 216 188 0 03 May 2020
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar C. Pehlevan 131 200 0 07 Feb 2020