Title
Scalable Meta-Learning via Mixed-Mode Differentiation Iurii Kemaev Dan A. Calian Luisa M. Zintgraf Gregory Farquhar H. V. Hasselt 79 1 0 01 May 2025
Optimizing ML Training with Metagradient Descent Logan Engstrom Andrew Ilyas Benjamin Chen Axel Feldmann William Moses Aleksander Madry 62 4 0 17 Mar 2025
Predictive Data Selection: The Data That Predicts Is the Data That Teaches Kashun Shum Yuanmin Huang Hongjian Zou Qi Ding Yixuan Liao Xiao Chen Qian Liu Junxian He 94 3 0 02 Mar 2025
Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining Daouda Sow Herbert Woisetschläger Saikiran Bulusu Shiqiang Wang Hans-Arno Jacobsen Yingbin Liang 100 4 0 10 Feb 2025
SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection Han Shen Pin-Yu Chen Payel Das Tianyi Chen ALM 67 18 0 09 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 82 6 0 09 Oct 2024
Data, Data Everywhere: A Guide for Pretraining Dataset Construction Jupinder Parmar Shrimai Prabhumoye Joseph Jennings Bo Liu Aastha Jhunjhunwala Zhilin Wang M. Patwary Mohammad Shoeybi Bryan Catanzaro 78 7 0 08 Jul 2024
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting Rui Pan Dylan Zhang Hanning Zhang Xingyuan Pan Minrui Xu Jipeng Zhang Renjie Pi Xiaoyu Wang Tong Zhang 77 9 0 28 Jun 2024
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale Guilherme Penedo Hynek Kydlícek Loubna Ben Allal Anton Lozhkov Margaret Mitchell Colin Raffel Leandro von Werra Thomas Wolf 85 223 0 25 Jun 2024
Data Shapley in One Training Run Jiachen T. Wang Prateek Mittal Dawn Song Ruoxi Jia TDI 78 11 0 16 Jun 2024
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research Luca Soldaini Rodney Michael Kinney Akshita Bhagia Dustin Schwenk David Atkinson ... Hanna Hajishirzi Iz Beltagy Dirk Groeneveld Jesse Dodge Kyle Lo 56 255 0 31 Jan 2024
Meta-Learning Online Adaptation of Language Models Nathan J. Hu E. Mitchell Christopher D. Manning Chelsea Finn KELM 45 36 0 24 May 2023
LIMA: Less Is More for Alignment Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun ... Susan Zhang Gargi Ghosh M. Lewis Luke Zettlemoyer Omer Levy ALM 71 819 0 18 May 2023
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining Sang Michael Xie Hieu H. Pham Xuanyi Dong Nan Du Hanxiao Liu Yifeng Lu Percy Liang Quoc V. Le Tengyu Ma Adams Wei Yu MoMe MoE 86 195 0 17 May 2023
Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt Sören Mindermann J. Brauner Muhammed Razzak Mrinank Sharma Andreas Kirsch ... Benedikt Höltgen Aidan Gomez Adrien Morisot Sebastian Farquhar Y. Gal 72 157 0 14 Jun 2022
Scaling Language Models: Methods, Analysis & Insights from Training Gopher Jack W. Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann ... Jeff Stanway L. Bennett Demis Hassabis Koray Kavukcuoglu G. Irving 48 1,303 0 08 Dec 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 223 9,946 0 17 Jun 2021
Podracer architectures for scalable Reinforcement Learning Matteo Hessel M. Kroiss Aidan Clark Iurii Kemaev John Quan Thomas Keck Fabio Viola H. V. Hasselt 31 39 0 13 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 666 28,659 0 26 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 376 2,051 0 31 Dec 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 74 1,724 0 26 Nov 2019
Optimizing Data Usage via Differentiable Rewards Xinyi Wang Hieu H. Pham Paul Michel Antonios Anastasopoulos J. Carbonell Graham Neubig 44 61 0 22 Nov 2019
Optimizing Millions of Hyperparameters by Implicit Differentiation Jonathan Lorraine Paul Vicol David Duvenaud DD 98 409 0 06 Nov 2019
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data Guillaume Wenzek Marie-Anne Lachaux Alexis Conneau Vishrav Chaudhary Francisco Guzmán Armand Joulin Edouard Grave 67 649 0 01 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 258 19,824 0 23 Oct 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 64 2,373 0 19 May 2019
Data Shapley: Equitable Valuation of Data for Machine Learning Amirata Ghorbani James Zou TDI FedML 59 771 0 05 Apr 2019
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge Alon Talmor Jonathan Herzig Nicholas Lourie Jonathan Berant RALM 115 1,677 0 02 Nov 2018
DARTS: Differentiable Architecture Search Hanxiao Liu Karen Simonyan Yiming Yang 159 4,326 0 24 Jun 2018
Meta-Gradient Reinforcement Learning Zhongwen Xu H. V. Hasselt David Silver 89 324 0 24 May 2018
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 64 2,474 0 14 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 427 129,831 0 12 Jun 2017
Understanding Black-box Predictions via Influence Functions Pang Wei Koh Percy Liang TDI 134 2,854 0 14 Mar 2017
Evolution Strategies as a Scalable Alternative to Reinforcement Learning Tim Salimans Jonathan Ho Xi Chen Szymon Sidor Ilya Sutskever 65 1,523 0 10 Mar 2017
Bag of Tricks for Efficient Text Classification Armand Joulin Edouard Grave Piotr Bojanowski Tomas Mikolov VLM 85 4,596 0 06 Jul 2016
Hyperparameter optimization with approximate gradient Fabian Pedregosa 85 449 0 07 Feb 2016
Gradient-based Hyperparameter Optimization through Reversible Learning D. Maclaurin David Duvenaud Ryan P. Adams DD 163 941 0 11 Feb 2015
On the difficulty of training Recurrent Neural Networks Razvan Pascanu Tomas Mikolov Yoshua Bengio ODL 114 5,318 0 21 Nov 2012