SlimPajama-DC: Understanding Data Combinations for LLM Training

SlimPajama-DC: Understanding Data Combinations for LLM Training

19 September 2023

Natalia Vassilieva

Papers citing "SlimPajama-DC: Understanding Data Combinations for LLM Training"

14 / 14 papers shown

Title
Mixtera: A Data Plane for Foundation Model Training Maximilian Böther Xiaozhe Yao Tolga Kerimoglu Ana Klimovic Viktor Gsteiger Ana Klimovic MoE 78 0 0 27 Feb 2025
FED: Fast and Efficient Dataset Deduplication Framework with GPU Acceleration Youngjun Son Chaewon Kim Jaejin Lee 45 0 0 02 Jan 2025
Mastering the Craft of Data Synthesis for CodeLLMs Meng Chen Philip Arthur Qianyu Feng Cong Duy Vu Hoang Yu-Heng Hong ... Mark Johnson K. K. Don Dharmasiri Long Duong Yuan-Fang Li SyDa 46 1 0 16 Oct 2024
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 43 7 0 11 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 35 1 0 02 Oct 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 51 9 0 24 Jul 2024
SaulLM-7B: A pioneering Large Language Model for Law Pierre Colombo T. Pires Malik Boudiaf Dominic Culver Rui Melo ... Andre F. T. Martins Fabrizio Esposito Vera Lúcia Raposo Sofia Morgado Michael Desa ELM AILaw 39 63 0 06 Mar 2024
Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy P. Schoenegger Indre Tuminauskaite Peter S. Park Rafael Valdece Sousa Bastos P. Tetlock 29 24 0 29 Feb 2024
Balanced Data Sampling for Language Model Training with Clustering Yunfan Shao Linyang Li Zhaoye Fei Hang Yan Dahua Lin Xipeng Qiu 29 8 0 22 Feb 2024
OLMo: Accelerating the Science of Language Models Dirk Groeneveld Iz Beltagy Pete Walsh Akshita Bhagia Rodney Michael Kinney ... Jesse Dodge Kyle Lo Luca Soldaini Noah A. Smith Hanna Hajishirzi OSLM 130 349 0 01 Feb 2024
Calibrated Language Models Must Hallucinate Adam Tauman Kalai Santosh Vempala HILM 13 75 0 24 Nov 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 242 690 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 273 2,878 0 15 Sep 2016