A Tale of Tails: Model Collapse as a Change of Scaling Laws

10 February 2024

Papers citing "A Tale of Tails: Model Collapse as a Change of Scaling Laws"

47 / 47 papers shown

Title
Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data? Grgur Kovač Jérémy Perez Rémy Portelas Peter Ford Dominey Pierre-Yves Oudeyer 33 0 0 04 Apr 2025
Position: Model Collapse Does Not Mean What You Think Rylan Schaeffer Joshua Kazdan Alvan Caleb Arulandu Sanmi Koyejo 51 0 0 05 Mar 2025
LLM as a Broken Telephone: Iterative Generation Distorts Information Amr Mohamed Mingmeng Geng Michalis Vazirgiannis Guokan Shang 57 1 0 27 Feb 2025
A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops Shi Fu Yingjie Wang Yuzhu Chen Xinmei Tian Dacheng Tao 48 1 0 26 Feb 2025
Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling Yiwen Ding Zhiheng Xi Wei He Zhuoyuan Li Yitao Zhai Xiaowei Shi Xunliang Cai Tao Gui Qi Zhang Xuanjing Huang LRM 59 3 0 24 Feb 2025
Machine-generated text detection prevents language model collapse George Drayson Emine Yilmaz Vasileios Lampos DeLMO 57 0 0 21 Feb 2025
GiFT: Gibbs Fine-Tuning for Code Generation Haochen Li Wanjin Feng Xin Zhou Zhiqi Shen SyDa 68 1 0 17 Feb 2025
Escaping Collapse: The Strength of Weak Data for Large Language Model Training Kareem Amin Sara Babakniya Alex Bie Weiwei Kong Umar Syed Sergei Vassilvitskii 68 1 0 13 Feb 2025
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges Nayoung Lee Ziyang Cai Avi Schwarzschild Kangwook Lee Dimitris Papailiopoulos ReLM VLM LRM AI4CE 73 4 0 03 Feb 2025
Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping Pu Yang Yunzhen Feng Ziyuan Chen Yuhang Wu Zhuoyuan Li DiffM 96 0 0 31 Jan 2025
Diverse Preference Optimization Jack Lanchantin Angelica Chen S. Dhuliawala Ping Yu Jason Weston Sainbayar Sukhbaatar Ilia Kulikov 88 4 0 30 Jan 2025
Rate of Model Collapse in Recursive Training A. Suresh A. Thangaraj Aditya Nanda Kishore Khandavally SyDa 25 5 0 23 Dec 2024
Loss-to-Loss Prediction: Scaling Laws for All Datasets David Brandfonbrener Nikhil Anand Nikhil Vyas Eran Malach Sham Kakade 77 3 0 19 Nov 2024
Zipfian Whitening Sho Yokoi Han Bao Hiroto Kurita Hidetoshi Shimodaira 27 0 0 01 Nov 2024
Universality of the $π^2/6$ Pathway in Avoiding Model Collapse Apratim Dey D. Donoho 50 5 0 30 Oct 2024
Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models Wenda Li Huijie Zhang Qing Qu WIGM 41 0 0 28 Oct 2024
Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification Hsun-Yu Kuo Yin-Hsiang Liao Yu-Chieh Chao Wei-Yun Ma Pu-Jen Cheng SyDa 43 2 0 28 Oct 2024
Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World Joshua Kazdan Rylan Schaeffer Apratim Dey Matthias Gerstgrasser Rafael Rafailov D. Donoho Sanmi Koyejo 45 11 0 22 Oct 2024
Bias Amplification: Large Language Models as Increasingly Biased Media Ze Wang Zekun Wu Jeremy Zhang Navya Jain Xin Guan Skylar Lu Saloni Gupta Adriano Soares Koshiyama 37 0 0 19 Oct 2024
Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR Christoph Minixhofer Ondˇrej Klejch Peter Bell 19 0 0 16 Oct 2024
Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra Roman Worschech B. Rosenow 36 0 0 11 Oct 2024
Emergent properties with repeated examples Francois Charton Julia Kempe AIMat 18 2 0 09 Oct 2024
Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning Zhengyu Hu Yichuan Li Zhengyu Chen J. Wang Han Liu Kyumin Lee Kaize Ding GNN 101 1 0 09 Oct 2024
O1 Replication Journey: A Strategic Progress Report -- Part 1 Yiwei Qin Xuefeng Li Haoyang Zou Yixiu Liu Shijie Xia ... Yixin Ye Weizhe Yuan Hector Liu Y. Li Pengfei Liu VLM 37 67 0 08 Oct 2024
Strong Model Collapse Elvis Dohmatob Yunzhen Feng Arjun Subramonian Julia Kempe 26 9 0 07 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 46 4 0 30 Sep 2024
Self-Improving Diffusion Models with Synthetic Data Sina Alemohammad Ahmed Imtiaz Humayun S. Agarwal John Collomosse Richard G. Baraniuk 23 10 0 29 Aug 2024
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions Luca Pappalardo Emanuele Ferragina Salvatore Citraro Giuliano Cornacchia M. Nanni ... D. Gambetta Giovanni Mauro Virginia Morini Valentina Pansanella D. Pedreschi 39 8 0 29 Jun 2024
Scaling Synthetic Data Creation with 1,000,000,000 Personas Tao Ge Xin Chan Dian Yu Haitao Mi Dong Yu Dong Yu SyDa 106 89 0 28 Jun 2024
How Stable is Stable Diffusion under Recursive InPainting (RIP)? Javier Conde Miguel González Gonzalo Martínez Fernando Moral Elena Merino-Gómez Pedro Reviriego DiffM 21 3 0 27 Jun 2024
Understanding Hallucinations in Diffusion Models through Mode Interpolation Sumukh K. Aithal Pratyush Maini Zachary Chase Lipton J. Zico Kolter DiffM 40 18 0 13 Jun 2024
Scaling Laws in Linear Regression: Compute, Parameters, and Data Licong Lin Jingfeng Wu Sham Kakade Peter L. Bartlett Jason D. Lee LRM 28 15 0 12 Jun 2024
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement Yunzhen Feng Elvis Dohmatob Pu Yang Francois Charton Julia Kempe 39 17 0 11 Jun 2024
Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images Krishnakant Singh Thanush Navaratnam Jannik Holmer Simone Schaub-Meyer Stefan Roth DiffM 33 18 0 30 May 2024
Large Language Models Can Self-Improve At Web Agent Tasks Ajay Patel M. Hofmarcher Claudiu Leoveanu-Condrei Marius-Constantin Dinu Chris Callison-Burch Sepp Hochreiter LLMAG 21 23 0 30 May 2024
Infinite-Dimensional Feature Interaction Chenhui Xu Fuxun Yu Maoliang Li Zihao Zheng Zirui Xu Jinjun Xiong Xiang Chen 27 1 0 22 May 2024
Age-Dependent Analysis and Stochastic Generation of Child-Directed Speech Okko Rasanen Daniil Kocharov 25 0 0 13 May 2024
How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse M. Seddik Suei-Wen Chen Soufiane Hayou Pierre Youssef Merouane Debbah 26 29 0 07 Apr 2024
AI and the Problem of Knowledge Collapse Andrew J. Peterson 33 16 0 04 Apr 2024
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents Corby Rosset Ho-Lam Chung Guanghui Qin Ethan C. Chau Zhuo Feng Ahmed Hassan Awadallah Jennifer Neville Nikhil Rao 22 10 0 27 Feb 2024
Model Collapse Demystified: The Case of Regression Elvis Dohmatob Yunzhen Feng Julia Kempe 32 32 0 12 Feb 2024
Human-AI Coevolution D. Pedreschi Luca Pappalardo Emanuele Ferragina R. Baeza-Yates Albert-László Barabási ... P. Lukowicz A. Passarella Alex Pentland John Shawe-Taylor Alessandro Vespignani 17 14 0 23 Jun 2023
Will we run out of data? Limits of LLM scaling based on human-generated data Pablo Villalobos A. Ho J. Sevilla T. Besiroglu Lennart Heim Marius Hobbhahn ALM 28 106 0 26 Oct 2022
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 253 4,735 0 24 Feb 2021
Learning Curve Theory Marcus Hutter 128 56 0 08 Feb 2021
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar C. Pehlevan 131 199 0 07 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020