Farewell to Aimless Large-scale Pretraining: Influential Subset
Selection for Language Model

Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model

22 May 2023

Jie Zhou

Papers citing "Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model"

10 / 10 papers shown

Title
Most Influential Subset Selection: Challenges, Promises, and Beyond Yuzheng Hu Pingbang Hu Han Zhao Jiaqi W. Ma TDI 136 2 0 10 Jan 2025
In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models Ayrton San Joaquin Bin Wang Zhengyuan Liu Nicholas Asher Brian Lim Philippe Muller Nancy Chen 24 0 0 07 Aug 2024
REQUAL-LM: Reliability and Equity through Aggregation in Large Language Models Sana Ebrahimi N. Shahbazi Abolfazl Asudeh 24 1 0 17 Apr 2024
Orthogonal Subspace Learning for Language Model Continual Learning Xiao Wang Tianze Chen Qiming Ge Han Xia Rong Bao Rui Zheng Qi Zhang Tao Gui Xuanjing Huang CLL 112 85 0 22 Oct 2023
Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models Mayee F. Chen Nicholas Roberts Kush S. Bhatia Jue Wang Ce Zhang Frederic Sala Christopher Ré SyDa 23 50 0 26 Jul 2023
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework Xingcheng Yao Yanan Zheng Xiaocong Yang Zhilin Yang 24 44 0 07 Nov 2021
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training Krishnateja Killamsetty D. Sivasubramanian Ganesh Ramakrishnan A. De Rishabh K. Iyer OOD 83 184 0 27 Feb 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,798 0 14 Dec 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 234 578 0 12 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020