Training LLMs over Neurally Compressed Text

4 April 2024

Papers citing "Training LLMs over Neurally Compressed Text"

7 / 7 papers shown

Title
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance Omer Goldman Avi Caciularu Matan Eyal Kris Cao Idan Szpektor Reut Tsarfaty 38 22 0 10 Mar 2024
Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models Luke Vilnis Yury Zemlyanskiy Patrick C. Murray Alexandre Passos Sumit Sanghai 49 9 0 18 Oct 2022
Sequence Length is a Domain: Length-based Overfitting in Transformer Models Dusan Varis Ondrej Bojar 49 56 0 15 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 242 690 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020