SemDeDup: Data-efficient learning at web-scale through semantic deduplication

16 March 2023

Papers citing "SemDeDup: Data-efficient learning at web-scale through semantic deduplication"

34 / 34 papers shown

Title
R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training Albert Ge Tzu-Heng Huang John Cooper Avi Trost Ziyi Chu Satya Sai Srinath Namburi GNVV Ziyang Cai Kendall Park Nicholas Roberts Frederic Sala 53 0 0 01 May 2025
QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining Fengze Liu Weidong Zhou Binbin Liu Zhimiao Yu Yifan Zhang ... Yifeng Yu Bingni Zhang Xiaohuan Zhou Taifeng Wang Yong Cao 55 0 0 23 Apr 2025
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models Xinlin Zhuang Jiahui Peng Ren Ma Y. Wang Tianyi Bai Xingjian Wei Jiantao Qiu Chi Zhang Ying Qian Conghui He 39 0 0 19 Apr 2025
ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selection Xiaoxuan Zhu Zhouhong Gu Baiqian Wu Suhang Zheng Tao Wang Tianyu Li Hongwei Feng Yanghua Xiao 40 0 0 01 Apr 2025
PinLanding: Content-First Keyword Landing Page Generation via Multi-Modal AI for Web-Scale Discovery Faye Zhang Jasmine Wan Qianyu Cheng Jinfeng Rao 33 0 0 01 Mar 2025
ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval Guanqi Zhan Yuanpei Liu Kai Han Weidi Xie Andrew Zisserman VLM 132 0 0 21 Feb 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 88 12 0 31 Dec 2024
Demystifying CLIP Data Hu Xu Saining Xie Xiaoqing Ellen Tan Po-Yao (Bernie) Huang Russell Howes Vasu Sharma Shang-Wen Li Gargi Ghosh Luke Zettlemoyer Christoph Feichtenhofer VLM CLIP 34 108 0 31 Dec 2024
Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding Zilin Du Haoxin Li Jianfei Yu Boyang Li 126 0 0 01 Dec 2024
LSHBloom: Memory-efficient, Extreme-scale Document Deduplication A. Khan Robert Underwood Carlo Siebenschuh Y. Babuji Aswathy Ajith Kyle Hippe Ozan Gokdemir Alexander Brace Kyle Chard Ian T. Foster 38 0 0 06 Nov 2024
Improving Model Evaluation using SMART Filtering of Benchmark Datasets Vipul Gupta Candace Ross David Pantoja R. Passonneau Megan Ung Adina Williams 70 1 0 26 Oct 2024
ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment Elyas Obbad Iddah Mlauzi Brando Miranda Rylan Schaeffer Kamal Obbad Suhana Bedi Sanmi Koyejo CVBM 48 0 0 23 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 48 4 0 09 Oct 2024
Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation Kun Yuan V. Srivastav Nassir Navab N. Padoy 44 7 0 30 Sep 2024
Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review Neha Prakriya Jui-Nan Yen Cho-Jui Hsieh Jason Cong KELM AI4CE LRM 31 1 0 10 Sep 2024
Improving Pretraining Data Using Perplexity Correlations Tristan Thrush Christopher Potts Tatsunori Hashimoto 32 17 0 09 Sep 2024
Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions Yu-Guan Hsieh Cheng-Yu Hsieh Shih-Ying Yeh Louis Béthune Hadi Pour Ansari Pavan Kumar Anasosalu Vasu Chun-Liang Li Ranjay Krishna Oncel Tuzel Marco Cuturi 58 4 0 09 Jul 2024
Concept-skill Transferability-based Data Selection for Large Vision-Language Models Jaewoo Lee Boyang Li Sung Ju Hwang VLM 33 8 0 16 Jun 2024
SAVA: Scalable Learning-Agnostic Data Valuation Samuel Kessler Tam Le Vu Nguyen TDI 51 0 0 03 Jun 2024
The Mosaic Memory of Large Language Models Igor Shilov Matthieu Meeus Yves-Alexandre de Montjoye 39 3 0 24 May 2024
Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores Chantal Shaib Joe Barrow Jiuding Sun Alexa F. Siu Byron C. Wallace A. Nenkova 66 31 0 01 Mar 2024
Balanced Data Sampling for Language Model Training with Clustering Yunfan Shao Linyang Li Zhaoye Fei Hang Yan Dahua Lin Xipeng Qiu 29 8 0 22 Feb 2024
Analysing The Impact of Sequence Composition on Language Model Pre-Training Yu Zhao Yuanbin Qu Konrad Staniszewski Szymon Tworkowski Wei Liu Piotr Milo's Yuxiang Wu Pasquale Minervini 34 14 0 21 Feb 2024
Effective pruning of web-scale datasets based on complexity of concept clusters Amro Abbas E. Rusak Kushal Tirumala Wieland Brendel Kamalika Chaudhuri Ari S. Morcos VLM CLIP 21 22 0 09 Jan 2024
Mitigating Open-Vocabulary Caption Hallucinations Assaf Ben-Kish Moran Yanuka Morris Alper Raja Giryes Hadar Averbuch-Elor MLLM VLM 16 6 0 06 Dec 2023
Farzi Data: Autoregressive Data Distillation Noveen Sachdeva Zexue He Wang-Cheng Kang Jianmo Ni D. Cheng Julian McAuley DD 17 3 0 15 Oct 2023
Self-Supervised Pre-Training Boosts Semantic Scene Segmentation on LiDAR Data Mariona Carós Ariadna Just Santi Seguí Jordi Vitrià SSL 3DPC 11 3 0 05 Sep 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 13 41 0 12 Jul 2023
The MiniPile Challenge for Data-Efficient Language Models Jean Kaddour MoE ALM 24 41 0 17 Apr 2023
Neural networks learn to magnify areas near decision boundaries Jacob A. Zavatone-Veth Sheng Yang Julian Rubinfien C. Pehlevan MLT AI4CE 18 6 0 26 Jan 2023
Data Distillation: A Survey Noveen Sachdeva Julian McAuley DD 30 73 0 11 Jan 2023
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 590 0 14 Jul 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 303 5,773 0 29 Apr 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020