Structured Packing in LLM Training Improves Long Context Utilization

Structured Packing in LLM Training Improves Long Context Utilization

28 December 2023

Konrad Staniszewski

Szymon Tworkowski

Sebastian Jaszczur

Henryk Michalewski

Lukasz Kuciñski

Papers citing "Structured Packing in LLM Training Improves Long Context Utilization"

4 / 4 papers shown

Title
Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs Jiancheng Dong Lei Jiang Wei Jin Lu Cheng 36 1 0 18 Aug 2024
Analysing The Impact of Sequence Composition on Language Model Pre-Training Yu Zhao Yuanbin Qu Konrad Staniszewski Szymon Tworkowski Wei Liu Piotr Milo's Yuxiang Wu Pasquale Minervini 29 13 0 21 Feb 2024
Pre-Training to Learn in Context Yuxian Gu Li Dong Furu Wei Minlie Huang CLIP LRM ReLM 108 37 0 16 May 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,881 0 04 Mar 2022