Differentially Private Optimization on Large Model at Small Cost

30 September 2022

Zhiqi Bu

Yu-Xiang Wang

Sheng Zha

George Karypis

ArXiv PDF HTML

Papers citing "Differentially Private Optimization on Large Model at Small Cost"

46 / 46 papers shown

Title
Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks? Hao Du Shang Liu Yang Cao AAML 45 0 0 28 Apr 2025
AdvSGM: Differentially Private Graph Learning via Adversarial Skip-gram Model Sen Zhang Qingqing Ye Haibo Hu Jianliang Xu 42 0 0 27 Mar 2025
Generating Synthetic Data with Formal Privacy Guarantees: State of the Art and the Road Ahead Viktor Schlegel Anil A Bharath Zilong Zhao Kevin Yee 66 0 0 26 Mar 2025
Empirical Privacy Variance Yuzheng Hu Fan Wu Ruicheng Xian Yuhang Liu Lydia Zakynthinou Pritish Kamath Chiyuan Zhang David A. Forsyth 62 0 0 16 Mar 2025
Climate And Resource Awareness is Imperative to Achieving Sustainable AI (and Preventing a Global AI Arms Race) Pedram Bakhtiarifard Pınar Tözün Christian Igel Raghavendra Selvan 37 0 0 27 Feb 2025
Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training Toan Tran Ruixuan Liu Li Xiong MU 41 0 0 27 Feb 2025
DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models Yanming Liu Xinyue Peng Yuwei Zhang Xiaolan Ke Songhang Deng ... Sheng Cheng Xun Wang Jianwei Yin Tianyu Du Xuhong Zhang 70 0 0 21 Feb 2025
Structure-Preference Enabled Graph Embedding Generation under Differential Privacy Sen Zhang Qingqing Ye Haibo Hu 34 0 0 08 Jan 2025
Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions Hao Du Shang Liu Lele Zheng Yang Cao Atsuyoshi Nakamura Lei Chen AAML 109 3 0 21 Dec 2024
DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction Xinwei Zhang Zhiqi Bu Borja Balle Mingyi Hong Meisam Razaviyayn Vahab Mirrokni 74 2 0 04 Oct 2024
LLM-PBE: Assessing Data Privacy in Large Language Models Qinbin Li Junyuan Hong Chulin Xie Jeffrey Tan Rachel Xin ... Dan Hendrycks Zhangyang Wang Bo Li Bingsheng He Dawn Song ELM PILM 36 12 0 23 Aug 2024
Differentially Private Block-wise Gradient Shuffle for Deep Learning Zilong Zhang FedML 19 0 0 31 Jul 2024
Weights Shuffling for Improving DPSGD in Transformer-based Models Jungang Yang Zhe Ji Liyao Xiang 22 0 0 22 Jul 2024
Towards Efficient and Scalable Training of Differentially Private Deep Learning Sebastian Rodriguez Beltran Marlon Tobaben Niki Loppi Antti Honkela 16 0 0 25 Jun 2024
Data Shapley in One Training Run Jiachen T. Wang Prateek Mittal Dawn Song Ruoxi Jia TDI 27 7 0 16 Jun 2024
Efficient Differentially Private Fine-Tuning of Diffusion Models Jing Liu Andrew Lowy T. Koike-Akino K. Parsons Ye Wang 21 0 0 07 Jun 2024
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs Charlie Hou Akshat Shrivastava Hongyuan Zhan Rylan Conway Trang Le Adithya Sagar Giulia Fanti Daniel Lazar 24 8 0 05 Jun 2024
LMO-DP: Optimizing the Randomization Mechanism for Differentially Private Fine-Tuning (Large) Language Models Qin Yang Meisam Mohammady Han Wang Ali Payani Ashish Kundu Kai Shu Yan Yan Yuan Hong 20 0 0 29 May 2024
Delving into Differentially Private Transformer Youlong Ding Xueyang Wu Yining Meng Yonggang Luo Hao Wang Weike Pan 16 5 0 28 May 2024
A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures Thanh Tam Nguyen T. T. Huynh Zhao Ren Thanh Toan Nguyen Phi Le Nguyen Hongzhi Yin Quoc Viet Hung Nguyen 53 8 0 31 Mar 2024
Differentially Private Representation Learning via Image Captioning Tom Sander Yaodong Yu Maziar Sanjabi Alain Durmus Yi-An Ma Kamalika Chaudhuri Chuan Guo 48 3 0 04 Mar 2024
On the Convergence of Differentially-Private Fine-tuning: To Linearly Probe or to Fully Fine-tune? Shuqi Ke Charlie Hou Giulia Fanti Sewoong Oh 34 4 0 29 Feb 2024
Pre-training Differentially Private Models with Limited Public Data Zhiqi Bu Xinwei Zhang Mingyi Hong Sheng Zha George Karypis 77 3 0 28 Feb 2024
Privacy-Preserving Instructions for Aligning Large Language Models Da Yu Peter Kairouz Sewoong Oh Zheng Xu 32 17 0 21 Feb 2024
Purifying Large Language Models by Ensembling a Small Language Model Tianlin Li Qian Liu Tianyu Pang Chao Du Qing-Wu Guo Yang Liu Min-Bin Lin 38 16 0 19 Feb 2024
Dataset Condensation Driven Machine Unlearning Junaid Iqbal Khan DD 31 1 0 31 Jan 2024
Private Fine-tuning of Large Language Models with Zeroth-order Optimization Xinyu Tang Ashwinee Panda Milad Nasr Saeed Mahloujifar Prateek Mittal 44 18 0 09 Jan 2024
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift Pratiksha Thaker Amrith Rajagopal Setlur Zhiwei Steven Wu Virginia Smith 20 2 0 24 Dec 2023
Zero redundancy distributed learning with differential privacy Zhiqi Bu Justin Chiu Ruixuan Liu Sheng Zha George Karypis 38 9 0 20 Nov 2023
On the accuracy and efficiency of group-wise clipping in differentially private optimization Zhiqi Bu Ruixuan Liu Yu-Xiang Wang Sheng Zha George Karypis VLM 17 4 0 30 Oct 2023
DPZero: Private Fine-Tuning of Language Models without Backpropagation Liang Zhang Bingcong Li K. K. Thekumparampil Sewoong Oh Niao He 28 11 0 14 Oct 2023
Tangent Transformers for Composition, Privacy and Removal Tian Yu Liu Aditya Golatkar Stefano Soatto 21 8 0 16 Jul 2023
Differentially Private Image Classification by Learning Priors from Random Processes Xinyu Tang Ashwinee Panda Vikash Sehwag Prateek Mittal 10 20 0 08 Jun 2023
DPFormer: Learning Differentially Private Transformer on Long-Tailed Data Youlong Ding Xueyang Wu Hongya Wang Weike Pan 13 0 0 28 May 2023
DP-SGD Without Clipping: The Lipschitz Neural Network Way Louis Bethune Thomas Massena Thibaut Boissin Yannick Prudent Corentin Friedrich Franck Mamalet A. Bellet M. Serrurier David Vigouroux 27 9 0 25 May 2023
Privacy-Preserving In-Context Learning for Large Language Models Tong Wu Ashwinee Panda Jiachen T. Wang Prateek Mittal 46 29 0 02 May 2023
Why Is Public Pretraining Necessary for Private Model Training? Arun Ganesh Mahdi Haghifam Milad Nasr Sewoong Oh Thomas Steinke Om Thakkar Abhradeep Thakurta Lun Wang 11 36 0 19 Feb 2023
Differentially Private Natural Language Models: Recent Advances and Future Directions Lijie Hu Ivan Habernal Lei Shen Di Wang AAML 13 18 0 22 Jan 2023
Differentially Private Image Classification from Features Harsh Mehta Walid Krichene Abhradeep Thakurta Alexey Kurakin Ashok Cutkosky 24 7 0 24 Nov 2022
Differentially Private Bias-Term Fine-tuning of Foundation Models Zhiqi Bu Yu-Xiang Wang Sheng Zha George Karypis 10 46 0 30 Sep 2022
Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy Zhiqi Bu J. Mao Shiyun Xu 131 47 0 21 May 2022
Differentially Private Fine-tuning of Language Models Da Yu Saurabh Naik A. Backurs Sivakanth Gopi Huseyin A. Inan ... Y. Lee Andre Manoel Lukas Wutschitz Sergey Yekhanin Huishuai Zhang 134 344 0 13 Oct 2021
Opacus: User-Friendly Differential Privacy Library in PyTorch Ashkan Yousefpour I. Shilov Alexandre Sablayrolles Davide Testuggine Karthik Prasad ... Sayan Gosh Akash Bharadwaj Jessica Zhao Graham Cormode Ilya Mironov VLM 144 347 0 25 Sep 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,808 0 14 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018
Efficient Per-Example Gradient Computations Ian Goodfellow 160 73 0 07 Oct 2015