Oasis: Data Curation and Assessment System for Pretraining of Large
Language Models

Oasis: Data Curation and Assessment System for Pretraining of Large Language Models

21 November 2023

Kang Liu

Jun Zhao

Papers citing "Oasis: Data Curation and Assessment System for Pretraining of Large Language Models"

9 / 9 papers shown

Title
What's New in My Data? Novelty Exploration via Contrastive Generation Masaru Isonuma Ivan Titov 16 0 0 18 Oct 2024
DCA-Bench: A Benchmark for Dataset Curation Agents Benhao Huang Yingzhuo Yu Jin Huang Xingjian Zhang Jiaqi Ma 23 1 0 11 Jun 2024
On Catastrophic Inheritance of Large Foundation Models Hao Chen Bhiksha Raj Xing Xie Jindong Wang AI4CE 48 12 0 02 Feb 2024
Data Portraits: Recording Foundation Model Training Data Marc Marone Benjamin Van Durme 132 30 0 06 Mar 2023
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing Jason Alan Fries Leon Weber Natasha Seelam Gabriel Altay Debajyoti Datta ... Minh Chien Vu Trishala Neeraj Jonas Golde Albert Villanova del Moral Benjamin Beilharz LM&MA 93 45 0 30 Jun 2022
Challenges in Detoxifying Language Models Johannes Welbl Amelia Glaese J. Uesato Sumanth Dathathri John F. J. Mellor Lisa Anne Hendricks Kirsty Anderson Pushmeet Kohli Ben Coppin Po-Sen Huang LM&MA 242 191 0 15 Sep 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 4,424 0 23 Jan 2020