Detecting Pretraining Data from Large Language Models

25 October 2023

Weijia Shi

Luke Zettlemoyer

Papers citing "Detecting Pretraining Data from Large Language Models"

41 / 41 papers shown

Title
WaterDrum: Watermarking for Data-centric Unlearning Metric Xinyang Lu Xinyuan Niu Gregory Kang Ruey Lau Bui Thi Cam Nhung Rachael Hwee Ling Sim Fanyu Wen Chuan-Sheng Foo S. Ng Bryan Kian Hsiang Low MU 55 0 0 08 May 2025
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation Vaidehi Patil Yi-Lin Sung Peter Hase Jie Peng Tianlong Chen Mohit Bansal AAML MU 79 3 0 01 May 2025
Beyond Public Access in LLM Pre-Training Data Sruly Rosenblat Tim O'Reilly Ilan Strauss MLAU 55 0 0 24 Apr 2025
Single-pass Detection of Jailbreaking Input in Large Language Models Leyla Naz Candogan Yongtao Wu Elias Abad Rocamora Grigorios G. Chrysos V. Cevher AAML 47 0 0 24 Feb 2025
The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text Matthieu Meeus Lukas Wutschitz Santiago Zanella Béguelin Shruti Tople Reza Shokri 75 0 0 24 Feb 2025
MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations Kaixuan Huang Jiacheng Guo Zihao Li X. Ji Jiawei Ge ... Yangsibo Huang Chi Jin Xinyun Chen Chiyuan Zhang Mengdi Wang AAML LRM 91 7 0 10 Feb 2025
Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection Ali Naseh Niloofar Mireshghallah 51 0 0 20 Jan 2025
Are Large Language Models Memorizing Bug Benchmarks? Daniel Ramos Claudia Mamede Kush Jain Paulo Canelas Catarina Gamboa Claire Le Goues PILM ELM 94 6 0 20 Nov 2024
Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination D. Song Sicheng Lai Shunian Chen Lichao Sun Benyou Wang 83 0 0 06 Nov 2024
Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset Yingzi Ma Jiongxiao Wang Fei-Yue Wang Siyuan Ma Jiazhao Li ... B. Li Yejin Choi M. Chen Chaowei Xiao Chaowei Xiao MU 52 6 0 05 Nov 2024
On Memorization of Large Language Models in Logical Reasoning Chulin Xie Yangsibo Huang Chiyuan Zhang Da Yu Xinyun Chen Bill Yuchen Lin Bo Li Badih Ghazi Ravi Kumar LRM 45 20 0 30 Oct 2024
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models Jinghan Jia Jiancheng Liu Yihua Zhang Parikshit Ram Nathalie Baracaldo Sijia Liu MU 35 2 0 23 Oct 2024
Reconstruction of Differentially Private Text Sanitization via Large Language Models Shuchao Pang Zhigang Lu H. Wang Peng Fu Yongbin Zhou Minhui Xue AAML 51 4 0 16 Oct 2024
Detecting Training Data of Large Language Models via Expectation Maximization Gyuwan Kim Yang Li Evangelia Spiliopoulou Jie Ma Miguel Ballesteros William Yang Wang MIALM 90 3 2 10 Oct 2024
Fine-tuning can Help Detect Pretraining Data from Large Language Models H. Zhang Songxin Zhang Bingyi Jing Hongxin Wei 34 0 0 09 Oct 2024
Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning Chongyu Fan Jiancheng Liu Licong Lin Jinghan Jia Ruiqi Zhang Song Mei Sijia Liu MU 43 15 0 09 Oct 2024
AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text Ximing Lu Melanie Sclar Skyler Hallinan Niloofar Mireshghallah Jiacheng Liu ... Allyson Ettinger Liwei Jiang Khyathi Raghavi Chandu Nouha Dziri Yejin Choi DeLMO 46 11 0 05 Oct 2024
Position: LLM Unlearning Benchmarks are Weak Measures of Progress Pratiksha Thaker Shengyuan Hu Neil Kale Yash Maurya Zhiwei Steven Wu Virginia Smith MU 45 10 0 03 Oct 2024
Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data Jie Zhang Debeshee Das Gautam Kamath Florian Tramèr MIALM MIACV 223 16 1 29 Sep 2024
Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data Atilla Akkus Mingjie Li Junjie Chu Junjie Chu Michael Backes Sinem Sav Sinem Sav SILM SyDa 30 1 0 12 Sep 2024
Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directions Shumaila Javaid R. A. Khalil Nasir Saeed Bin He Mohamed-Slim Alouini 32 9 0 05 Jul 2024
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon USVSN Sai Prashanth Alvin Deng Kyle O'Brien Jyothir S V Mohammad Aflah Khan ... Jacob Ray Fuehne Stella Biderman Tracy Ke Katherine Lee Naomi Saphra 55 12 0 25 Jun 2024
Blind Baselines Beat Membership Inference Attacks for Foundation Models Debeshee Das Jie Zhang Florian Tramèr MIALM 72 28 1 23 Jun 2024
Fantastic Copyrighted Beasts and How (Not) to Generate Them Luxi He Yangsibo Huang Weijia Shi Tinghao Xie Haotian Liu Yue Wang Luke Zettlemoyer Chiyuan Zhang Danqi Chen Peter Henderson 41 9 0 20 Jun 2024
The Mosaic Memory of Large Language Models Igor Shilov Matthieu Meeus Yves-Alexandre de Montjoye 39 3 0 24 May 2024
Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models Jiaqi Li Qianshan Wei Chuanyi Zhang Guilin Qi Miaozeng Du Yongrui Chen Sheng Bi Fan Liu VLM MU 67 12 0 21 May 2024
Building a Large Japanese Web Corpus for Large Language Models Naoaki Okazaki Kakeru Hattori Hirai Shota Hiroki Iida Masanari Ohi Kazuki Fujii Taishi Nakamura Mengsay Loem Rio Yokota Sakae Mizuki 47 6 0 27 Apr 2024
AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees William Fleshman Aleem Khan Marc Marone Benjamin Van Durme CLL KELM 42 3 0 12 Apr 2024
Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models Jingyang Zhang Jingwei Sun Eric C. Yeats Ouyang Yang Martin Kuo Jianyi Zhang Hao Frank Yang Hai Li 29 41 0 03 Apr 2024
Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs Aly M. Kassem Omar Mahmoud Niloofar Mireshghallah Hyunwoo J. Kim Yulia Tsvetkov Yejin Choi Sherif Saad Santu Rana 47 18 0 05 Mar 2024
Watermarking Makes Language Models Radioactive Tom Sander Pierre Fernandez Alain Durmus Matthijs Douze Teddy Furon WaLM 29 11 0 22 Feb 2024
Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space Leo Schwinn David Dobre Sophie Xhonneux Gauthier Gidel Stephan Gunnemann AAML 47 36 0 14 Feb 2024
Black-Box Access is Insufficient for Rigorous AI Audits Stephen Casper Carson Ezell Charlotte Siegmann Noam Kolt Taylor Lynn Curtis ... Michael Gerovitch David Bau Max Tegmark David M. Krueger Dylan Hadfield-Menell AAML 13 76 0 25 Jan 2024
TOFU: A Task of Fictitious Unlearning for LLMs Pratyush Maini Zhili Feng Avi Schwarzschild Zachary Chase Lipton J. Zico Kolter MU CLL 38 141 0 11 Jan 2024
Investigating Data Contamination for Pre-training Language Models Minhao Jiang Ken Ziyu Liu Ming Zhong Rylan Schaeffer Siru Ouyang Jiawei Han Sanmi Koyejo 23 62 0 11 Jan 2024
TinyGSM: achieving >80% on GSM8k with small language models Bingbin Liu Sébastien Bubeck Ronen Eldan Janardhan Kulkarni Yuanzhi Li Anh Nguyen Rachel A. Ward Yi Zhang ALM 19 47 0 14 Dec 2023
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples Shuo Yang Wei-Lin Chiang Lianmin Zheng Joseph E. Gonzalez Ion Stoica ALM 17 110 0 08 Nov 2023
Who's Harry Potter? Approximate Unlearning in LLMs Ronen Eldan M. Russinovich MU MoMe 101 172 0 03 Oct 2023
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4 Kent K. Chang Mackenzie Cramer Sandeep Soni David Bamman RALM 140 109 0 28 Apr 2023
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,808 0 14 Dec 2020