The Curse of Recursion: Training on Generated Data Makes Models Forget

27 May 2023

Papers citing "The Curse of Recursion: Training on Generated Data Makes Models Forget"

39 / 39 papers shown

Title
LLM Watermarking Using Mixtures and Statistical-to-Computational Gaps Pedro Abdalla Roman Vershynin WaLM 35 0 0 02 May 2025
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation Haris Riaz Sourav Sanjukta Bhabesh Vinayak Arannil Miguel Ballesteros Graham Horwood SyDa 50 0 0 17 Apr 2025
Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data? Grgur Kovač Jérémy Perez Rémy Portelas Peter Ford Dominey Pierre-Yves Oudeyer 33 0 0 04 Apr 2025
Swift Hydra: Self-Reinforcing Generative Framework for Anomaly Detection with Multiple Mamba Models Nguyen H K. Do Truc Nguyen Malik Hassanaly Raed Alharbi Jung Taek Seo My T. Thai 54 0 0 09 Mar 2025
Position: Model Collapse Does Not Mean What You Think Rylan Schaeffer Joshua Kazdan Alvan Caleb Arulandu Sanmi Koyejo 60 0 0 05 Mar 2025
LLM as a Broken Telephone: Iterative Generation Distorts Information Amr Mohamed Mingmeng Geng Michalis Vazirgiannis Guokan Shang 69 1 0 27 Feb 2025
Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling Yiwen Ding Zhiheng Xi Wei He Zhuoyuan Li Yitao Zhai Xiaowei Shi Xunliang Cai Tao Gui Qi Zhang Xuanjing Huang LRM 69 3 0 24 Feb 2025
Machine-generated text detection prevents language model collapse George Drayson Emine Yilmaz Vasileios Lampos DeLMO 62 0 0 21 Feb 2025
Preference Optimization for Reasoning with Pseudo Feedback Fangkai Jiao Geyang Guo Xingxing Zhang Nancy F. Chen Shafiq R. Joty Furu Wei LRM 99 9 0 17 Feb 2025
ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classification Y. Meena Vaibhav Singh Ayush Maheshwari Amrith Krishna Ganesh Ramakrishnan AI4TS 97 0 0 09 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 115 3 0 06 Feb 2025
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges Nayoung Lee Ziyang Cai Avi Schwarzschild Kangwook Lee Dimitris Papailiopoulos ReLM VLM LRM AI4CE 77 4 0 03 Feb 2025
Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping Pu Yang Yunzhen Feng Ziyuan Chen Yuhang Wu Zhuoyuan Li DiffM 101 0 0 31 Jan 2025
Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop Ekaterina Artemova Akim Tsvigun Dominik Schlechtweg Natalia Fedorova Konstantin Chernyshev Sergei Tilga Boris Obmoroshev SyDa VLM 122 0 0 28 Jan 2025
Aligning Instruction Tuning with Pre-training Yiming Liang Tianyu Zheng Xinrun Du Ge Zhang J. Liu ... Zhaoxiang Zhang Wenhao Huang Jiajun Zhang Xiang Yue Jiajun Zhang 86 1 0 16 Jan 2025
Improving Object Detection by Modifying Synthetic Data with Explainable AI Nitish Mital Simon Malzard Richard Walters Celso M. De Melo Raghuveer Rao Victoria Nockles 74 0 0 02 Dec 2024
A Review of Fairness and A Practical Guide to Selecting Context-Appropriate Fairness Metrics in Machine Learning Caleb J. S. Barr Olivia Erdelyi Paul D. Docherty Randolph C. Grace FaML 68 0 0 10 Nov 2024
Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification Hsun-Yu Kuo Yin-Hsiang Liao Yu-Chieh Chao Wei-Yun Ma Pu-Jen Cheng SyDa 45 2 0 28 Oct 2024
Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World Joshua Kazdan Rylan Schaeffer Apratim Dey Matthias Gerstgrasser Rafael Rafailov D. Donoho Sanmi Koyejo 50 11 0 22 Oct 2024
Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning Xiaochuan Li Zichun Yu Chenyan Xiong SyDa 31 1 0 18 Oct 2024
Understanding Generative AI Content with Embedding Models Max Vargas Reilly Cannon A. Engel Anand D. Sarwate Tony Chiang 52 3 0 19 Aug 2024
Learning to Rewrite: Generalized LLM-Generated Text Detection Wei Hao Ran Li Weiliang Zhao Junfeng Yang Chengzhi Mao DeLMO 54 3 0 08 Aug 2024
Scaling Synthetic Data Creation with 1,000,000,000 Personas Tao Ge Xin Chan Dian Yu Haitao Mi Dong Yu Dong Yu SyDa 111 92 0 28 Jun 2024
Linguistic Collapse: Neural Collapse in (Large) Language Models Robert Wu V. Papyan 46 12 0 28 May 2024
A social path to human-like artificial intelligence Edgar A. Duénez-Guzmán Suzanne Sadedin Jane X. Wang Kevin R. McKee Joel Z. Leibo GNN 28 28 0 22 May 2024
Securing the Future of GenAI: Policy and Technology Mihai Christodorescu Craven S. Feizi Neil Zhenqiang Gong Mia Hoffmann ... Jessica Newman Emelia Probasco Yanjun Qi Khawaja Shams Turek SILM 46 3 0 21 May 2024
A Survey on Self-Evolution of Large Language Models Zhengwei Tao Ting-En Lin Xiancai Chen Hangyu Li Yuchuan Wu Yongbin Li Zhi Jin Fei Huang Dacheng Tao Jingren Zhou LRM LM&Ro 51 22 0 22 Apr 2024
Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? Shayne Longpre Robert Mahari Naana Obeng-Marnu William Brannon Tobin South Katy Gero Sandy Pentland Jad Kabbara 56 5 0 19 Apr 2024
How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse M. Seddik Suei-Wen Chen Soufiane Hayou Pierre Youssef Merouane Debbah 42 30 0 07 Apr 2024
Synth $^2$ : Boosting Visual-Language Models with Synthetic Captions and Image Embeddings Sahand Sharifzadeh Christos Kaplanis Shreya Pathak D. Kumaran Anastasija Ilić Jovana Mitrović Charles Blundell Andrea Banino VLM 39 9 0 12 Mar 2024
TinyGSM: achieving >80% on GSM8k with small language models Bingbin Liu Sébastien Bubeck Ronen Eldan Janardhan Kulkarni Yuanzhi Li Anh Nguyen Rachel A. Ward Yi Zhang ALM 19 47 0 14 Dec 2023
Nepotistically Trained Generative-AI Models Collapse Matyáš Boháček Hany Farid 49 17 0 20 Nov 2023
Synthetically Enhanced: Unveiling Synthetic Data's Potential in Medical Imaging Research Bardia Khosravi Frank Li Theo Dapamede Pouria Rouzrokh Cooper Gamble ... C. Wyles Andrew B. Sellergren S. Purkayastha Bradley J. Erickson J. Gichoya MedIm 27 17 0 15 Nov 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 93 10,977 0 18 Jul 2023
Mini-Giants: "Small" Language Models and Open Source Win-Win Zhengping Zhou Lezhi Li Xinxi Chen Andy Li SyDa ALM MoE 24 6 0 17 Jul 2023
Evade ChatGPT Detectors via A Single Space Shuyang Cai Wanyun Cui DeLMO 33 15 0 05 Jul 2023
When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions Weiming Zhuang Chen Chen Lingjuan Lyu C. L. P. Chen Yaochu Jin Lingjuan Lyu AIFin AI4CE 99 85 0 27 Jun 2023
The Variance-Gamma Distribution: A Review Adrian Fischer Robert E. Gaunt A. Sarantsev 11 17 0 09 Mar 2023
Will we run out of data? Limits of LLM scaling based on human-generated data Pablo Villalobos A. Ho J. Sevilla T. Besiroglu Lennart Heim Marius Hobbhahn ALM 33 108 0 26 Oct 2022