Zyda: A 1.3T Dataset for Open Language Modeling

Zyda: A 1.3T Dataset for Open Language Modeling

4 June 2024

Jonathan Pilault

James Whittington

Quentin Anthony

Papers citing "Zyda: A 1.3T Dataset for Open Language Modeling"

5 / 5 papers shown

Title
Zyda-2: a 5 Trillion Token High-Quality Dataset Yury Tokpanov Paolo Glorioso Quentin Anthony Beren Millidge 31 3 0 09 Nov 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 123 415 0 13 Mar 2024
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020