Title
Towards Optimal Learning of Language Models Yuxian Gu Li Dong Y. Hao Qingxiu Dong Minlie Huang Furu Wei 36 7 0 27 Feb 2024
An Integrated Data Processing Framework for Pretraining Foundation Models Yiding Sun Feng Wang Yutao Zhu Wayne Xin Zhao Jiaxin Mao 29 4 0 26 Feb 2024
Balanced Data Sampling for Language Model Training with Clustering Yunfan Shao Linyang Li Zhaoye Fei Hang Yan Dahua Lin Xipeng Qiu 24 8 0 22 Feb 2024
Analysing The Impact of Sequence Composition on Language Model Pre-Training Yu Zhao Yuanbin Qu Konrad Staniszewski Szymon Tworkowski Wei Liu Piotr Milo's Yuxiang Wu Pasquale Minervini 21 13 0 21 Feb 2024
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling Pratyush Maini Skyler Seto Richard He Bai David Grangier Yizhe Zhang Navdeep Jaitly SyDa 33 54 0 29 Jan 2024
DsDm: Model-Aware Dataset Selection with Datamodels Logan Engstrom Axel Feldmann A. Madry OODD 8 45 0 23 Jan 2024
LLM360: Towards Fully Transparent Open-Source LLMs Zhengzhong Liu Aurick Qiao W. Neiswanger Hongyi Wang Bowen Tan ... Zhiting Hu Mark Schulze Preslav Nakov Timothy Baldwin Eric P. Xing 36 68 0 11 Dec 2023
Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding Talfan Evans Shreya Pathak Hamza Merzic Jonathan Schwarz Ryutaro Tanno Olivier J. Hénaff 8 16 0 08 Dec 2023
Efficient Online Data Mixing For Language Model Pre-Training Alon Albalak Liangming Pan Colin Raffel W. Wang 17 32 0 05 Dec 2023
REDUCR: Robust Data Downsampling Using Class Priority Reweighting William Bankes George Hughes Ilija Bogunovic Zi Wang 13 3 0 01 Dec 2023
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models Zeming Chen Alejandro Hernández Cano Angelika Romanou Antoine Bonnet Kyle Matoba ... Axel Marmet Syrielle Montariol Mary-Anne Hartley Martin Jaggi Antoine Bosselut LM&MA AI4MH MedIm 10 174 0 27 Nov 2023
Towards the Law of Capacity Gap in Distilling Language Models Chen Zhang Dawei Song Zheyu Ye Yan Gao ELM 10 20 0 13 Nov 2023
Training CLIP models on Data from Scientific Papers Calvin Metzger VLM CLIP 11 1 0 08 Nov 2023
Self-Influence Guided Data Reweighting for Language Model Pre-training Megh Thakkar Tolga Bolukbasi Sriram Ganapathy Shikhar Vashishth Sarath Chandar Partha P. Talukdar MILM 20 10 0 02 Nov 2023
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models Zhenpeng Su Xing Wu Xue Bai Zijia Lin Hui Chen Guiguang Ding Wei Zhou Songlin Hu 16 5 0 30 Oct 2023
MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications Yizhe Yang Huashan Sun Jiawei Li Runheng Liu Yinghao Li Yuhang Liu Heyan Huang Yang Gao ALM LRM 8 8 0 24 Oct 2023
DoGE: Domain Reweighting with Generalization Estimation Simin Fan Matteo Pagliardini Martin Jaggi 11 29 0 23 Oct 2023
Irreducible Curriculum for Language Model Pretraining Simin Fan Martin Jaggi 8 2 0 23 Oct 2023
Llemma: An Open Language Model For Mathematics Zhangir Azerbayev Hailey Schoelkopf Keiran Paster Marco Dos Santos Stephen Marcus McAleer Albert Q. Jiang Jia Deng Stella Biderman Sean Welleck CLL 24 204 0 16 Oct 2023
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning Mengzhou Xia Tianyu Gao Zhiyuan Zeng Danqi Chen 24 262 0 10 Oct 2023
FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamics Yupei Du Albert Gatt Dong Nguyen 9 1 0 10 Oct 2023
SlimPajama-DC: Understanding Data Combinations for LLM Training Zhiqiang Shen Tianhua Tao Liqun Ma W. Neiswanger Zhengzhong Liu ... Bowen Tan Joel Hestness Natalia Vassilieva Daria Soboleva Eric P. Xing 17 44 0 19 Sep 2023
Context is Environment Sharut Gupta Stefanie Jegelka David Lopez-Paz Kartik Ahuja 17 0 0 18 Sep 2023
Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen Yilun Huang Zhijian Ma Hesen Chen Xuchen Pan ... Zhaoyang Liu Jinyang Gao Yaliang Li Bolin Ding Jingren Zhou SyDa VLM 12 29 0 05 Sep 2023
Sparks of Large Audio Models: A Survey and Outlook S. Latif Moazzam Shoukat Fahad Shamshad Muhammad Usama Yi Ren ... Wenwu Wang Xulong Zhang Roberto Togneri Erik Cambria Björn W. Schuller LM&MA AuLLM 27 36 0 24 Aug 2023
D4: Improving LLM Pretraining via Document De-Duplication and Diversification Kushal Tirumala Daniel Simig Armen Aghajanyan Ari S. Morcos SyDa 6 103 0 23 Aug 2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore Sewon Min Suchin Gururangan Eric Wallace Hannaneh Hajishirzi Noah A. Smith Luke Zettlemoyer AILaw 16 63 0 08 Aug 2023
Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models Mayee F. Chen Nicholas Roberts Kush S. Bhatia Jue Wang Ce Zhang Frederic Sala Christopher Ré SyDa 18 50 0 26 Jul 2023
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets Seonghyeon Ye Doyoung Kim Sungdong Kim Hyeonbin Hwang Seungone Kim Yongrae Jo James Thorne Juho Kim Minjoon Seo ALM 27 96 0 20 Jul 2023
Deceptive Alignment Monitoring Andres Carranza Dhruv Pai Rylan Schaeffer Arnuv Tandon Oluwasanmi Koyejo 29 4 0 20 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 13 41 0 12 Jul 2023
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding Yanzhe Zhang Ruiyi Zhang Jiuxiang Gu Yufan Zhou Nedim Lipka Diyi Yang Tongfei Sun VLM MLLM 20 217 0 29 Jun 2023
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurenccon Lucile Saulnier Léo Tronchon Stas Bekman Amanpreet Singh ... Siddharth Karamcheti Alexander M. Rush Douwe Kiela Matthieu Cord Victor Sanh 14 137 0 21 Jun 2023
Efficient Stochastic Approximation of Minimax Excess Risk Optimization Lijun Zhang Haomin Bai W. Tu Ping Yang Yao Hu 10 3 0 31 May 2023
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity Shayne Longpre Gregory Yauney Emily Reif Katherine Lee Adam Roberts ... Denny Zhou Jason W. Wei Kevin Robinson David M. Mimno Daphne Ippolito 16 145 0 22 May 2023
Automatic Document Selection for Efficient Encoder Pretraining Yukun Feng Patrick Xia Benjamin Van Durme João Sedoc 44 7 0 20 Oct 2022
mGPT: Few-Shot Learners Go Multilingual Oleh Shliazhko Alena Fenogenova Maria Tikhonova Vladislav Mikhailov Anastasia Kozlova Tatiana Shavrina 12 146 0 15 Apr 2022
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 236 626 0 21 Apr 2021
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training Krishnateja Killamsetty D. Sivasubramanian Ganesh Ramakrishnan A. De Rishabh K. Iyer OOD 78 184 0 27 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 242 1,977 0 31 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 264 5,290 0 05 Nov 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 279 39,083 0 01 Sep 2014