Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers

6 February 2023

Papers citing "Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers"

32 / 32 papers shown

Title
E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation Boqian Wu Q. Xiao Shiwei Liu Lu Yin Mykola Pechenizkiy D. Mocanu M. V. Keulen Elena Mocanu MedIm 53 4 0 20 Feb 2025
Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity Alessandro Pierro Steven Abreu Jonathan Timcheck Philipp Stratmann Andreas Wild S. Shrestha 65 0 0 03 Feb 2025
Navigating Extremes: Dynamic Sparsity in Large Output Spaces Nasib Ullah Erik Schultheis Mike Lasby Yani Andrew Ioannou Rohit Babbar 33 0 0 05 Nov 2024
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness Boqian Wu Q. Xiao Shunxin Wang N. Strisciuglio Mykola Pechenizkiy M. V. Keulen D. Mocanu Elena Mocanu OOD 3DH 52 0 0 03 Oct 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 34 17 0 01 Jul 2024
Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization A. Nowak Lukasz Gniecki Filip Szatkowski Jacek Tabor 22 0 0 03 Jun 2024
Importance Estimation with Random Gradient for Neural Network Pruning Suman Sapkota Binod Bhattarai 30 1 0 31 Oct 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 23 78 0 08 Oct 2023
A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness Ze Peng Lei Qi Yinghuan Shi Yang Gao 20 3 0 06 Sep 2023
Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates Murat Onur Yildirim Elif Ceren Gok Yildirim Ghada Sokar D. Mocanu Joaquin Vanschoren CLL 20 7 0 28 Aug 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 13 41 0 12 Jul 2023
Neural Network Pruning for Real-time Polyp Segmentation Suman Sapkota Pranav Poudel Sudarshan Regmi Bibek Panthi Binod Bhattarai MedIm 26 0 0 22 Jun 2023
Vision Transformers for Mobile Applications: A Short Survey Nahid Alam Steven Kolawole S. Sethi Nishant Bansali Karina Nguyen ViT 16 3 0 30 May 2023
Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers Zahra Atashgahi Mykola Pechenizkiy Raymond N. J. Veldhuis D. Mocanu AI4TS AI4CE 24 1 0 28 May 2023
Visual Tuning Bruce X. B. Yu Jianlong Chang Haixin Wang Lin Liu Shijie Wang ... Lingxi Xie Haojie Li Zhouchen Lin Qi Tian Chang Wen Chen VLM 39 38 0 10 May 2023
JaxPruner: A concise library for sparsity research Jooyoung Lee Wonpyo Park Nicole Mitchell Jonathan Pilault J. Obando-Ceron ... Hong-Seok Kim Yann N. Dauphin Karolina Dziugaite P. S. Castro Utku Evci 31 14 0 27 Apr 2023
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! Shiwei Liu Tianlong Chen Zhenyu (Allen) Zhang Xuxi Chen Tianjin Huang Ajay Jaiswal Zhangyang Wang 24 29 0 03 Mar 2023
PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation Ningxin Zheng Huiqiang Jiang Quan Zhang Zhenhua Han Yuqing Yang ... Fan Yang Chengruidong Zhang Lili Qiu Mao Yang Lidong Zhou 32 26 0 26 Jan 2023
Minimalistic Unsupervised Learning with the Sparse Manifold Transform Yubei Chen Zeyu Yun Y. Ma Bruno A. Olshausen Yann LeCun 34 8 0 30 Sep 2022
On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence Y. Ma Doris Y. Tsao H. Shum 59 75 0 11 Jul 2022
On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks Hongru Yang Zhangyang Wang MLT 22 8 0 27 Mar 2022
Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance Shiwei Liu Yuesong Tian Tianlong Chen Li Shen 22 8 0 05 Mar 2022
Sparsity Winning Twice: Better Robust Generalization from More Efficient Training Tianlong Chen Zhenyu (Allen) Zhang Pengju Wang Santosh Balachandra Haoyu Ma Zehao Wang Zhangyang Wang OOD AAML 77 46 0 20 Feb 2022
Powerpropagation: A sparsity inducing weight reparameterisation Jonathan Richard Schwarz Siddhant M. Jayakumar Razvan Pascanu P. Latham Yee Whye Teh 87 54 0 01 Oct 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 642 0 21 Apr 2021
Truly Sparse Neural Networks at Scale Selima Curci D. Mocanu Mykola Pechenizkiy 18 19 0 02 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 139 684 0 31 Jan 2021
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 148 376 0 23 Jul 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 178 1,027 0 06 Mar 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 222 382 0 05 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Y. Gal Zoubin Ghahramani UQCV BDL 247 9,109 0 06 Jun 2015