Pruning Neural Networks at Initialization: Why are We Missing the Mark?

18 September 2020

Jonathan Frankle

Gintare Karolina Dziugaite

Daniel M. Roy

Michael Carbin

ArXiv PDF HTML

Papers citing "Pruning Neural Networks at Initialization: Why are We Missing the Mark?"

44 / 44 papers shown

Title
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries Chris Kolb T. Weber Bernd Bischl David Rügamer 109 0 0 04 Feb 2025
DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecasting Hao Wu Haomin Wen Guibin Zhang Yutong Xia Kai Wang Yuxuan Liang Yu Zheng Kun Wang 62 2 0 17 Jan 2025
Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning Andy Li A. Durrant Milan Markovic Lu Yin Georgios Leontidis Tianlong Chen Lu Yin Georgios Leontidis 75 0 0 20 Nov 2024
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training Elia Cunegatti Leonardo Lucio Custode Giovanni Iacca 47 0 0 11 Nov 2024
Mask in the Mirror: Implicit Sparsification Tom Jacobs R. Burkholz 45 3 0 19 Aug 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 43 7 0 02 Jul 2024
Survival of the Fittest Representation: A Case Study with Modular Addition Xiaoman Delores Ding Zifan Carl Guo Eric J. Michaud Ziming Liu Max Tegmark 48 3 0 27 May 2024
Rapid Deployment of DNNs for Edge Computing via Structured Pruning at Initialization Bailey J. Eccles Leon Wong Blesson Varghese 33 2 0 22 Apr 2024
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning Chu Myaet Thwal Minh N. H. Nguyen Ye Lin Tun Seongjin Kim My T. Thai Choong Seon Hong 51 5 0 22 Jan 2024
Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI Kai Huang Wei Gao 15 35 0 21 Dec 2023
Neural Networks at a Fraction with Pruned Quaternions Sahel Mohammad Iqbal Subhankar Mishra 28 4 0 13 Aug 2023
NTK-SAP: Improving neural network pruning by aligning training dynamics Yite Wang Dawei Li Ruoyu Sun 28 19 0 06 Apr 2023
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! Shiwei Liu Tianlong Chen Zhenyu (Allen) Zhang Xuxi Chen Tianjin Huang Ajay Jaiswal Zhangyang Wang 29 29 0 03 Mar 2023
Balanced Training for Sparse GANs Yite Wang Jing Wu N. Hovakimyan Ruoyu Sun 34 9 0 28 Feb 2023
Considering Layerwise Importance in the Lottery Ticket Hypothesis Benjamin Vandersmissen José Oramas 15 1 0 22 Feb 2023
Quantum Ridgelet Transform: Winning Lottery Ticket of Neural Networks with Quantum Computation H. Yamasaki Sathyawageeswar Subramanian Satoshi Hayakawa Sho Sonoda MLT 30 4 0 27 Jan 2023
Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning Huan Wang Can Qin Yue Bai Yun Fu 32 20 0 12 Jan 2023
Can We Find Strong Lottery Tickets in Generative Models? Sangyeop Yeo Yoojin Jang Jy-yong Sohn Dongyoon Han Jaejun Yoo 13 6 0 16 Dec 2022
AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models S. Kwon Jeonghoon Kim Jeongin Bae Kang Min Yoo Jin-Hwa Kim Baeseong Park Byeongwook Kim Jung-Woo Ha Nako Sung Dongsoo Lee MQ 23 30 0 08 Oct 2022
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? Mansheej Paul F. Chen Brett W. Larsen Jonathan Frankle Surya Ganguli Gintare Karolina Dziugaite UQCV 25 38 0 06 Oct 2022
What to Prune and What Not to Prune at Initialization Maha Haroon 13 0 0 06 Sep 2022
Complexity-Driven CNN Compression for Resource-constrained Edge AI Muhammad Zawish Steven Davy L. Abraham 28 16 0 26 Aug 2022
PrUE: Distilling Knowledge from Sparse Teacher Networks Shaopu Wang Xiaojun Chen Mengzhen Kou Jinqiao Shi 8 2 0 03 Jul 2022
Sparse Double Descent: Where Network Pruning Aggravates Overfitting Zhengqi He Zeke Xie Quanzhi Zhu Zengchang Qin 69 27 0 17 Jun 2022
Convolutional and Residual Networks Provably Contain Lottery Tickets R. Burkholz UQCV MLT 35 13 0 04 May 2022
Most Activation Functions Can Win the Lottery Without Excessive Depth R. Burkholz MLT 63 18 0 04 May 2022
LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification Sharath Girish Kamal Gupta Saurabh Singh Abhinav Shrivastava 28 11 0 06 Apr 2022
The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks Xin Yu Thiago Serra Srikumar Ramalingam Shandian Zhe 36 48 0 09 Mar 2022
Structured Pruning is All You Need for Pruning CNNs at Initialization Yaohui Cai Weizhe Hua Hongzheng Chen G. E. Suh Christopher De Sa Zhiru Zhang CVBM 39 14 0 04 Mar 2022
Rare Gems: Finding Lottery Tickets at Initialization Kartik K. Sreenivasan Jy-yong Sohn Liu Yang Matthew Grinde Alliot Nagle Hongyi Wang Eric P. Xing Kangwook Lee Dimitris Papailiopoulos 19 42 0 24 Feb 2022
Meta-Learning Sparse Implicit Neural Representations Jaehoon Lee Jihoon Tack Namhoon Lee Jinwoo Shin 22 44 0 27 Oct 2021
Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning Soufiane Hayou Bo He Gintare Karolina Dziugaite 20 2 0 22 Oct 2021
Sparse Training via Boosting Pruning Plasticity with Neuroregeneration Shiwei Liu Tianlong Chen Xiaohan Chen Zahra Atashgahi Lu Yin Huanyu Kou Li Shen Mykola Pechenizkiy Zhangyang Wang D. Mocanu 34 111 0 19 Jun 2021
Can Subnetwork Structure be the Key to Out-of-Distribution Generalization? Dinghuai Zhang Kartik Ahuja Yilun Xu Yisen Wang Aaron Courville OOD 20 95 0 05 Jun 2021
Generalization bounds via distillation Daniel J. Hsu Ziwei Ji Matus Telgarsky Lan Wang FedML 13 32 0 12 Apr 2021
Recent Advances on Neural Network Pruning at Initialization Huan Wang Can Qin Yue Bai Yulun Zhang Yun Fu CVBM 31 64 0 11 Mar 2021
Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch Aojun Zhou Yukun Ma Junnan Zhu Jianbo Liu Zhijie Zhang Kun Yuan Wenxiu Sun Hongsheng Li 50 239 0 08 Feb 2021
Rethinking Weight Decay For Efficient Neural Network Pruning Hugo Tessier Vincent Gripon Mathieu Léonardon M. Arzel T. Hannagan David Bertrand 26 25 0 20 Nov 2020
Are wider nets better given the same number of parameters? A. Golubeva Behnam Neyshabur Guy Gur-Ari 16 44 0 27 Oct 2020
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot Jingtong Su Yihang Chen Tianle Cai Tianhao Wu Ruiqi Gao Liwei Wang J. Lee 6 85 0 22 Sep 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 188 1,027 0 06 Mar 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 224 382 0 05 Mar 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 159 234 0 04 Mar 2020
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 119 577 0 27 Feb 2015