Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic

10 April 2024

J. Zico Kolter

Papers citing "Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic"

35 / 35 papers shown

Title
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning Yixuan Even Xu Yash Savani Fei Fang Zico Kolter OffRL 24 1 0 18 Apr 2025
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training Shizhe Diao Yu Yang Y. Fu Xin Dong Dan Su ... Hongxu Yin M. Patwary Yingyan Jan Kautz Pavlo Molchanov 33 0 0 17 Apr 2025
DataDecide: How to Predict Best Pretraining Data with Small Experiments Ian H. Magnusson Nguyen Tai Ben Bogin David Heineman Jena D. Hwang ... Dirk Groeneveld Oyvind Tafjord Noah A. Smith Pang Wei Koh Jesse Dodge ALM 30 0 0 15 Apr 2025
PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity Mustafa Burak Gurbuz Xingyu Zheng C. Dovrolis OOD 38 0 0 07 Apr 2025
Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework Thomson Yen Andrew Siah Haozhe Chen Tianyi Peng Daniel Guetta Hongseok Namkoong 48 0 0 26 Mar 2025
Squeeze Out Tokens from Sample for Finer-Grained Data Governance Weixiong Lin Chen Ju Haicheng Wang Shengchao Hu Shuai Xiao ... Yuheng Jiao Mingshuai Yao Jinsong Lan Qingwen Liu Ying Chen 48 0 0 18 Mar 2025
Concept-as-Tree: Synthetic Data is All You Need for VLM Personalization Ruichuan An Kai Zeng Ming Lu Sihan Yang Renrui Zhang Huitong Ji Qizhe Zhang Y. Luo Hao Liang Wentao Zhang 59 0 0 17 Mar 2025
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules Kairong Luo Haodong Wen Shengding Hu Zhenbo Sun Zhiyuan Liu Maosong Sun Kaifeng Lyu Wenguang Chen CLL 52 0 0 17 Mar 2025
Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining Mikey Shechter Yair Carmon CLIP 42 0 0 11 Mar 2025
Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality Alex Fang Hadi Pouransari Matt Jordan Alexander Toshev Vaishaal Shankar Ludwig Schmidt Tom Gunter 69 0 0 10 Mar 2025
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions E. Liu Amanda Bertsch Lintang Sutawika Lindia Tjuatja Patrick Fernandes ... S. Carolin (Haas) Lawrence Aditi Raghunathan Kiril Gashteovski Graham Neubig 60 0 0 05 Mar 2025
(Mis)Fitting: A Survey of Scaling Laws Margaret Li Sneha Kudugunta Luke Zettlemoyer 69 2 0 26 Feb 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 105 23 0 24 Feb 2025
The interplay between domain specialization and model size Roseval Malaquias Junior Ramon Pires Thales Sales Almeida Kenzo Sakiyama R. Romero R. Nogueira 41 0 0 03 Jan 2025
Compute-Constrained Data Selection Junjie Oscar Yin Alexander M. Rush 37 0 0 21 Oct 2024
Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws Yiding Jiang Allan Zhou Zhili Feng Sadhika Malladi J. Zico Kolter 33 15 0 15 Oct 2024
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 43 7 0 11 Oct 2024
Unsupervised Data Validation Methods for Efficient Model Training Yurii Paniv 24 1 0 10 Oct 2024
SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation Maying Shen Nadine Chang Sifei Liu Jose M. Alvarez 24 0 0 20 Sep 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 48 5 0 11 Jul 2024
Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions Yu-Guan Hsieh Cheng-Yu Hsieh Shih-Ying Yeh Louis Béthune Hadi Pour Ansari Pavan Kumar Anasosalu Vasu Chun-Liang Li Ranjay Krishna Oncel Tuzel Marco Cuturi 58 4 0 09 Jul 2024
Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning Yijun Dong Hoang Phan Xiang Pan Qi Lei 37 4 0 08 Jul 2024
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min-Bin Lin MoE 55 34 1 01 Jul 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 48 19 0 27 Jun 2024
Data curation via joint example selection further accelerates multimodal learning Talfan Evans Nikhil Parthasarathy Hamza Merzic Olivier J. Hénaff 23 12 0 25 Jun 2024
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models Zichun Yu Spandan Das Chenyan Xiong 23 24 0 10 Jun 2024
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models Zachary Ankner Cody Blakeney Kartik K. Sreenivasan Max Marion Matthew L. Leavitt Mansheej Paul 30 23 0 30 May 2024
CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning Yiping Wang Yifang Chen Wendan Yan Alex Fang Wenjing Zhou Kevin G. Jamieson S. Du 32 7 0 29 May 2024
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations Alexander Hägele Elie Bakouch Atli Kosson Loubna Ben Allal Leandro von Werra Martin Jaggi 36 33 0 28 May 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping-Chia Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 47 31 0 26 May 2024
Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp Rachel Hong William Agnew Tadayoshi Kohno Jamie Morgenstern 27 9 0 13 May 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 37 58 0 25 Mar 2024
Language models scale reliably with over-training and on downstream tasks S. Gadre Georgios Smyrnis Vaishaal Shankar Suchin Gururangan Mitchell Wortsman ... Y. Carmon Achal Dave Reinhard Heckel Niklas Muennighoff Ludwig Schmidt ALM ELM LRM 91 40 0 13 Mar 2024
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 293 3,683 0 11 Feb 2021