Title
Don't be lazy: CompleteP enables compute-efficient deep transformers Nolan Dey Bin Claire Zhang Lorenzo Noci Mufan Bill Li Blake Bordelon Shane Bergsma C. Pehlevan Boris Hanin Joel Hestness 37 0 0 02 May 2025
AeroLite: Tag-Guided Lightweight Generation of Aerial Image Captions Xing Zi Tengjun Ni Xianjing Fan Xian Tao Jun Li Ali Braytee Mukesh Prasad 23 0 0 13 Apr 2025
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions E. Liu Amanda Bertsch Lintang Sutawika Lindia Tjuatja Patrick Fernandes ... S. Carolin (Haas) Lawrence Aditi Raghunathan Kiril Gashteovski Graham Neubig 67 0 0 05 Mar 2025
Crystal: Illuminating LLM Abilities on Language and Code Tianhua Tao Junbo Li Bowen Tan Hongyi Wang William Marshall ... Joel Hestness Natalia Vassilieva Zhiqiang Shen Eric P. Xing Zhengzhong Liu 40 4 0 06 Nov 2024
How Does Critical Batch Size Scale in Pre-training? Hanlin Zhang Depen Morwani Nikhil Vyas Jingfeng Wu Difan Zou Udaya Ghai Dean Phillips Foster Sham Kakade 64 8 0 29 Oct 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 51 9 0 24 Jul 2024
Sparse maximal update parameterization: A holistic approach to sparse training dynamics Nolan Dey Shane Bergsma Joel Hestness 19 5 0 24 May 2024
Does Transformer Interpretability Transfer to RNNs? Gonccalo Paulo Thomas Marshall Nora Belrose 41 6 0 09 Apr 2024
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining Jacob P. Portes Alex Trott Sam Havens Daniel King Abhinav Venigalla Moin Nadeem Nikhil Sardana D. Khudia Jonathan Frankle 13 16 0 29 Dec 2023
I Know You Did Not Write That! A Sampling Based Watermarking Method for Identifying Machine Generated Text Kaan Efe Keles Ömer Kaan Gürbüz Mucahid Kutlu WaLM 10 1 0 29 Nov 2023
Towards the Law of Capacity Gap in Distilling Language Models Chen Zhang Dawei Song Zheyu Ye Yan Gao ELM 18 20 0 13 Nov 2023
Position Interpolation Improves ALiBi Extrapolation Faisal Al-Khateeb Nolan Dey Daria Soboleva Joel Hestness LLMSV 15 5 0 18 Oct 2023
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 225 103 0 27 Oct 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 242 690 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
The Woman Worked as a Babysitter: On Biases in Language Generation Emily Sheng Kai-Wei Chang Premkumar Natarajan Nanyun Peng 206 607 0 03 Sep 2019