DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training

3 October 2023

Konstantinos Parasyris

Papers citing "DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training"

35 / 35 papers shown

Title
Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles Amir Ali Farzin Yuen-Man Pun Iman Shames 19 0 0 04 May 2025
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients Yezhen Wang Zhouhao Yang Brian K Chen Fanyi Pu Bo-wen Li Tianyu Gao Kenji Kawaguchi 34 0 0 03 May 2025
Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search Nuojin Cheng Alireza Doostan Stephen Becker 34 0 0 30 Apr 2025
TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks Mohammad Maheri Hamed Haddadi Alex Davidson 63 0 0 27 Apr 2025
Efficient Personalization of Quantized Diffusion Model without Backpropagation H. Seo Wongi Jeong Kyungryeol Lee Se Young Chun DiffM MQ 73 0 0 19 Mar 2025
ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory Liangyu Wang Jie Ren Hang Xu Junxiao Wang Huanyi Xie David E. Keyes Di Wang 58 0 0 16 Mar 2025
SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization Shubhankar Borse K. Bhardwaj Mohammad Reza Karimi Dastjerdi Hyojin Park Shreya Kadambi ... Prathamesh Mandke Ankita Nayak Harris Teague Munawar Hayat Fatih Porikli DiffM 73 1 0 27 Feb 2025
LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM Yehonathan Refael Iftach Arbel Ofir Lindenbaum Tom Tirer 64 0 0 26 Feb 2025
MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models Zhen Zhang Y. Yang Kai Zhen Nathan Susanj Athanasios Mouchtaris Siegfried Kunzmann Zheng Zhang 48 0 0 17 Feb 2025
Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks Yequan Zhao Xinling Yu Xian Xiao Z. Chen Z. Liu G. Kurczveil R. Beausoleil S. Liu Z. Zhang 45 0 0 17 Feb 2025
ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization Keisuke Sugiura Hiroki Matsutani MQ 36 1 0 08 Jan 2025
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 81 1 0 26 Nov 2024
Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach Yequan Zhao Hai Li Ian Young Zheng-Wei Zhang MQ 24 2 0 07 Nov 2024
Stepping Forward on the Last Mile Chen Feng Shaojie Zhuo Xiaopeng Zhang R. Ramakrishnan Zhaocong Yuan Andrew Zou Li 25 0 0 06 Nov 2024
FLOPS: Forward Learning with OPtimal Sampling Tao Ren Zishi Zhang Jinyang Jiang Guanghao Li Zeliang Zhang Mingqian Feng Yijie Peng 30 1 0 08 Oct 2024
On Unsupervised Prompt Learning for Classification with Black-box Language Models Zhen-Yu Zhang Jiandong Zhang Huaxiu Yao Gang Niu Masashi Sugiyama 21 2 0 04 Oct 2024
Unifying back-propagation and forward-forward algorithms through model predictive control Lianhai Ren Qianxiao Li 19 1 0 29 Sep 2024
DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization Pucheng Dang Xing Hu Dong Li Rui Zhang Qi Guo Kaidi Xu DiffM 31 4 0 18 Aug 2024
Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks Mingqing Xiao Qingyan Meng Zongpeng Zhang D.K. He Zhouchen Lin 19 0 0 17 Jul 2024
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Wentao Guo Jikai Long Yimeng Zeng Zirui Liu Xinyu Yang ... Osbert Bastani Christopher De Sa Xiaodong Yu Beidi Chen Zhaozhuo Xu 26 14 0 05 Jun 2024
Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient Hao Di Haishan Ye Yueling Zhang Xiangyu Chang Guang Dai Ivor W. Tsang 21 1 0 28 May 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 54 3 0 24 May 2024
Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models Yimeng Zhang Xin Chen Jinghan Jia Yihua Zhang Chongyu Fan Jiancheng Liu Mingyi Hong Ke Ding Sijia Liu DiffM 29 51 0 24 May 2024
The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection Mohammad Jafari Yimeng Zhang Yihua Zhang Sijia Liu 25 2 0 18 Mar 2024
Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning Yong Liu Zirui Zhu Chaoyu Gong Minhao Cheng Cho-Jui Hsieh Yang You MoE 27 16 0 24 Feb 2024
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark Yihua Zhang Pingzhi Li Junyuan Hong Jiaxiang Li Yimeng Zhang ... Wotao Yin Mingyi Hong Zhangyang Wang Sijia Liu Tianlong Chen 17 12 0 18 Feb 2024
ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs Yuhan Li Peisong Wang Zhixun Li Jeffrey Xu Yu Jia Li 19 38 0 17 Feb 2024
Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning Zhicheng Liu Jian Lou W. Bao Y. Hu Baochun Li Z. Qin K. Ren 29 7 0 12 Feb 2024
To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now Yimeng Zhang Jinghan Jia Xin Chen Aochuan Chen Yihua Zhang Jiancheng Liu Ke Ding Sijia Liu DiffM 17 79 0 18 Oct 2023
DPZero: Private Fine-Tuning of Language Models without Backpropagation Liang Zhang Bingcong Li K. K. Thekumparampil Sewoong Oh Niao He 25 11 0 14 Oct 2023
Scaling Forward Gradient With Local Losses Mengye Ren Simon Kornblith Renjie Liao Geoffrey E. Hinton 64 49 0 07 Oct 2022
Optimization without Backpropagation Gabriel Belouze 14 7 0 13 Sep 2022
SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks Timoleon Moraitis Dmitry Toichkin Adrien Journé Yansong Chua Qinghai Guo AAML BDL 65 17 0 12 Jul 2021
Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm Bin Gu Guodong Liu Yanfu Zhang Xiang Geng Heng-Chiao Huang 31 18 0 17 Feb 2021
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 172 1,018 0 06 Mar 2020