Private Fine-tuning of Large Language Models with Zeroth-order Optimization

9 January 2024

Papers citing "Private Fine-tuning of Large Language Models with Zeroth-order Optimization"

26 / 26 papers shown

Title
Forward Learning with Differential Privacy Mingqian Feng Zeliang Zhang Jinyang Jiang Yijie Peng Chenliang Xu 39 0 0 01 Apr 2025
Privacy Auditing of Large Language Models Ashwinee Panda Xinyu Tang Milad Nasr Christopher A. Choquette-Choo Prateek Mittal PILM 54 5 0 09 Mar 2025
Towards hyperparameter-free optimization with differential privacy Zhiqi Bu Ruixuan Liu 24 1 0 02 Mar 2025
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning Raghav Singhal Kaustubh Ponkshe Rohit Vartak Lav R. Varshney Praneeth Vepakomma FedML 64 0 0 24 Feb 2025
Private Text Generation by Seeding Large Language Model Prompts Supriya Nagesh Justin Y. Chen Nina Mishra Tal Wagner SyDa SILM 52 1 0 20 Feb 2025
ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization Keisuke Sugiura Hiroki Matsutani MQ 36 1 0 08 Jan 2025
Balls-and-Bins Sampling for DP-SGD Lynn Chua Badih Ghazi Charlie Harrison Ethan Leeman Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chiyuan Zhang 78 3 0 21 Dec 2024
Scalable DP-SGD: Shuffling vs. Poisson Subsampling Lynn Chua Badih Ghazi Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chiyuan Zhang 28 5 0 06 Nov 2024
Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning Fengyu Gao Ruida Zhou T. Wang Cong Shen Jing Yang 21 2 0 15 Oct 2024
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs Charlie Hou Akshat Shrivastava Hongyuan Zhan Rylan Conway Trang Le Adithya Sagar Giulia Fanti Daniel Lazar 24 3 0 05 Jun 2024
AI Risk Management Should Incorporate Both Safety and Security Xiangyu Qi Yangsibo Huang Yi Zeng Edoardo Debenedetti Jonas Geiping ... Chaowei Xiao Bo-wen Li Dawn Song Peter Henderson Prateek Mittal AAML 43 9 0 29 May 2024
Privacy-preserving Fine-tuning of Large Language Models through Flatness Tiejin Chen Longchao Da Huixue Zhou Pingzhi Li Kaixiong Zhou Tianlong Chen Hua Wei 24 3 0 07 Mar 2024
Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer Yanjun Zhao Sizhe Dang Haishan Ye Guang Dai Yi Qian Ivor W.Tsang 56 8 0 23 Feb 2024
Privacy-Preserving Instructions for Aligning Large Language Models Da Yu Peter Kairouz Sewoong Oh Zheng Xu 32 10 0 21 Feb 2024
Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning Zhicheng Liu Jian Lou W. Bao Y. Hu Baochun Li Z. Qin K. Ren 21 7 0 12 Feb 2024
Zero redundancy distributed learning with differential privacy Zhiqi Bu Justin Chiu Ruixuan Liu Sheng Zha George Karypis 27 4 0 20 Nov 2023
DPZero: Private Fine-Tuning of Language Models without Backpropagation Liang Zhang Bingcong Li K. K. Thekumparampil Sewoong Oh Niao He 20 7 0 14 Oct 2023
Privacy-Preserving In-Context Learning for Large Language Models Tong Wu Ashwinee Panda Jiachen T. Wang Prateek Mittal 36 29 0 02 May 2023
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy Natalia Ponomareva Hussein Hazimeh Alexey Kurakin Zheng Xu Carson E. Denison H. B. McMahan Sergei Vassilvitskii Steve Chien Abhradeep Thakurta 83 165 0 01 Mar 2023
Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining Florian Tramèr Gautam Kamath Nicholas Carlini SILM 28 57 0 13 Dec 2022
Differentially Private Fine-tuning of Language Models Da Yu Saurabh Naik A. Backurs Sivakanth Gopi Huseyin A. Inan ... Y. Lee Andre Manoel Lukas Wutschitz Sergey Yekhanin Huishuai Zhang 128 258 0 13 Oct 2021
Opacus: User-Friendly Differential Privacy Library in PyTorch Ashkan Yousefpour I. Shilov Alexandre Sablayrolles Davide Testuggine Karthik Prasad ... Sayan Gosh Akash Bharadwaj Jessica Zhao Graham Cormode Ilya Mironov VLM 131 268 0 25 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 275 3,784 0 18 Apr 2021
Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning Da Yu Huishuai Zhang Wei Chen Tie-Yan Liu FedML SILM 81 110 0 25 Feb 2021
Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao Adam Fisch Danqi Chen 238 1,898 0 31 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018