Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

8 January 2025

Papers citing "Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum"

5 / 5 papers shown

Title
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu-Xi Cheng MoE 61 1 0 07 Mar 2025
Fewer Truncations Improve Language Modeling Hantian Ding Zijian Wang Giovanni Paolini Varun Kumar Anoop Deoras Dan Roth Stefano Soatto 56 13 0 16 Apr 2024
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 55 77 0 19 Sep 2023
Sequence Length is a Domain: Length-based Overfitting in Transformer Models Dusan Varis Ondrej Bojar 49 56 0 15 Sep 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020