Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

25 March 2024

Jiasheng Ye

Tianxiang Sun

Xipeng Qiu

Papers citing "Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance"

3 / 53 papers shown

Title
Learning Curve Theory Marcus Hutter 128 56 0 08 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020