MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models

10 June 2024

Chenyan Xiong

Papers citing "MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models"

25 / 25 papers shown

Title
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection Kai Hua Steven Wu Ge Zhang Ke Shen LRM 18 0 0 12 May 2025
QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining Fengze Liu Weidong Zhou Binbin Liu Zhimiao Yu Yifan Zhang ... Yifeng Yu Bingni Zhang Xiaohuan Zhou Taifeng Wang Yong Cao 55 0 0 23 Apr 2025
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models Xinlin Zhuang Jiahui Peng Ren Ma Y. Wang Tianyi Bai Xingjian Wei Jiantao Qiu Chi Zhang Ying Qian Conghui He 39 0 0 19 Apr 2025
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training Shizhe Diao Yu Yang Y. Fu Xin Dong Dan Su ... Hongxu Yin M. Patwary Yingyan Jan Kautz Pavlo Molchanov 33 0 0 17 Apr 2025
From Fairness to Truthfulness: Rethinking Data Valuation Design Dongyang Fan Tyler J. Rotello Sai Praneeth Karimireddy TDI 46 0 0 07 Apr 2025
Large-Scale Data Selection for Instruction Tuning Hamish Ivison Muru Zhang Faeze Brahman Pang Wei Koh Pradeep Dasigi ALM 71 1 0 03 Mar 2025
Predictive Data Selection: The Data That Predicts Is the Data That Teaches Kashun Shum Y. Huang Hongjian Zou Qi Ding Yixuan Liao X. Chen Qian Liu Junxian He 57 2 0 02 Mar 2025
ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation Yanzhou Pan Huawei Lin Yide Ran Jiamin Chen Xiaodong Yu Weijie Zhao Denghui Zhang Zhaozhuo Xu 40 0 0 02 Mar 2025
MergeIT: From Selection to Merging for Efficient Instruction Tuning Hongyi Cai Yuqian Fu Hongming Fu Bo Zhao MoMe 47 0 0 25 Feb 2025
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale Fan Zhou Zengzhi Wang Qian Liu Junlong Li Pengfei Liu ALM 97 14 0 17 Feb 2025
FRAMES: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy Xuemiao Zhang Feiyu Duan Liangyu Xu Yongwei Zhou Sirui Wang Rongxiang Weng J. Wang Xunliang Cai 60 0 0 08 Feb 2025
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities Qirun Dai Dylan Zhang Jiaqi W. Ma Hao Peng TDI 52 1 0 21 Jan 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 85 11 0 31 Dec 2024
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging Clara Na Ian H. Magnusson A. Jha Tom Sherborne Emma Strubell Jesse Dodge Pradeep Dasigi MoMe 36 4 0 21 Oct 2024
Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning Xiaochuan Li Zichun Yu Chenyan Xiong SyDa 27 1 0 18 Oct 2024
Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining Tianyi Bai Ling Yang Zhen Hao Wong Jiahui Peng Xinlin Zhuang ... Lijun Wu Jiantao Qiu Wentao Zhang Binhang Yuan Conghui He LLMAG 23 4 0 10 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 40 4 0 09 Oct 2024
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling David Grangier Simin Fan Skyler Seto Pierre Ablin 34 3 0 30 Sep 2024
Harnessing Diversity for Important Data Selection in Pretraining Large Language Models Chi Zhang Huaping Zhong Kuan Zhang Chengliang Chai Rui Wang ... Lei Cao Ju Fan Ye Yuan Guoren Wang Conghui He TDI 28 4 0 25 Sep 2024
Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement Simon Yu Liangyu Chen Sara Ahmadian Marzieh Fadaee 29 6 0 17 Sep 2024
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min-Bin Lin MoE 64 34 1 01 Jul 2024
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining Yu Meng Chenyan Xiong Payal Bajaj Saurabh Tiwary Paul N. Bennett Jiawei Han Xia Song 114 201 0 16 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 223 4,424 0 23 Jan 2020