Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch

8 February 2021

Papers citing "Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch"

50 / 145 papers shown

Title
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 29 27 0 05 Feb 2024
Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods Bo-Kyeong Kim Geonmin Kim Tae-Ho Kim Thibault Castells Shinkook Choi Junho Shin Hyoung-Kyu Song 62 30 0 05 Feb 2024
The LLM Surgeon Tycho F. A. van der Ouderaa Markus Nagel M. V. Baalen Yuki Markus Asano Tijmen Blankevoort 26 14 0 28 Dec 2023
Model-Based Control with Sparse Neural Dynamics Ziang Liu Genggeng Zhou Jeff He Tobia Marcucci Fei-Fei Li Jiajun Wu Yunzhu Li AI4CE 35 17 0 20 Dec 2023
Fluctuation-based Adaptive Structured Pruning for Large Language Models Yongqi An Xu Zhao Tao Yu Ming Tang Jinqiao Wang 31 42 0 19 Dec 2023
USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models Shaojin Ding David Qiu David Rim Yanzhang He Oleg Rybakov ... Tara N. Sainath Zhonglin Han Jian Li Amir Yazdanbakhsh Shivani Agrawal MQ 26 9 0 13 Dec 2023
MaxQ: Multi-Axis Query for N:M Sparsity Network Jingyang Xiang Siqi Li Junhao Chen Zhuangzhi Chen Tianxin Huang Linpeng Peng Yong-Jin Liu 16 0 0 12 Dec 2023
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Tianyu Ding Tianyi Chen Haidong Zhu Jiachen Jiang Yiqi Zhong Jinxin Zhou Guangzhi Wang Zhihui Zhu Ilya Zharkov Luming Liang 27 22 0 01 Dec 2023
E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity Yun Li Lin Niu Xipeng Zhang Kai Liu Jianchen Zhu Zhanhui Kang MoE 32 11 0 24 Oct 2023
Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloads Hongxiang Fan Stylianos I. Venieris Alexandros Kouris Nicholas D. Lane 15 7 0 17 Oct 2023
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs Yu-xin Zhang Lirui Zhao Mingbao Lin Yunyun Sun Yiwu Yao Xingjia Han Jared Tanner Shiwei Liu Rongrong Ji SyDa 37 40 0 13 Oct 2023
SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Acceleration Jingyang Xiang Siqi Li Jun Chen Shipeng Bai Yukai Ma Guang Dai Yong-Jin Liu 24 1 0 10 Oct 2023
Compressing LLMs: The Truth is Rarely Pure and Never Simple Ajay Jaiswal Zhe Gan Xianzhi Du Bowen Zhang Zhangyang Wang Yinfei Yang MQ 36 45 0 02 Oct 2023
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs Lu Yin Ajay Jaiswal Shiwei Liu Souvik Kundu Zhangyang Wang 22 7 0 29 Sep 2023
Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design Chao Fang Wei Sun Aojun Zhou Zhongfeng Wang 11 3 0 22 Sep 2023
Scaling Laws for Sparsely-Connected Foundation Models Elias Frantar C. Riquelme N. Houlsby Dan Alistarh Utku Evci 18 35 0 15 Sep 2023
Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges Fei Dou Jin Ye Geng Yuan Qin Lu Wei Niu ... Hongyue Sun Yunli Shao Changying Li Tianming Liu Wenzhan Song AI4CE 21 29 0 14 Sep 2023
Bandwidth-efficient Inference for Neural Image Compression Shanzhi Yin Tongda Xu Yongsheng Liang Yuanyuan Wang Yanghao Li Yan Wang Jingjing Liu 23 1 0 06 Sep 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 23 3 0 07 Aug 2023
Rosko: Row Skipping Outer Products for Sparse Matrix Multiplication Kernels Vikas Natesh Andrew Sabot H. T. Kung Mark Ting 17 0 0 08 Jul 2023
Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer Peng Mi Li Shen Tianhe Ren Yiyi Zhou Tianshuo Xu Xiaoshuai Sun Tongliang Liu Rongrong Ji Dacheng Tao AAML 33 2 0 30 Jun 2023
An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs Haihao Shen Hengyu Meng Bo Dong Zhe Wang Ofir Zafrir ... Hanwen Chang Qun Gao Zi. Wang Guy Boudoukh Moshe Wasserblat MoE 31 4 0 28 Jun 2023
Efficient Online Processing with Deep Neural Networks Lukas Hedegaard 18 0 0 23 Jun 2023
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training A. Nowak Bram Grooten D. Mocanu Jacek Tabor 21 9 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 56 353 0 20 Jun 2023
Spatial Re-parameterization for N:M Sparsity Yu-xin Zhang Mingbao Lin Yunshan Zhong Mengzhao Chen Fei Chao Rongrong Ji 39 2 0 09 Jun 2023
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter Ajay Jaiswal Shiwei Liu Tianlong Chen Zhangyang Wang VLM 21 33 0 06 Jun 2023
Dynamic Sparsity Is Channel-Level Sparsity Learner Lu Yin Gen Li Meng Fang Lijuan Shen Tianjin Huang Zhangyang Wang Vlado Menkovski Xiaolong Ma Mykola Pechenizkiy Shiwei Liu 19 20 0 30 May 2023
HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity Yannan Nellie Wu Po-An Tsai Saurav Muralidharan A. Parashar Vivienne Sze J. Emer 21 23 0 22 May 2023
PDP: Parameter-free Differentiable Pruning is All You Need Minsik Cho Saurabh N. Adya Devang Naik VLM 15 10 0 18 May 2023
Dynamic Sparse Training with Structured Sparsity Mike Lasby A. Golubeva Utku Evci Mihai Nica Yani Andrew Ioannou 29 19 0 03 May 2023
JaxPruner: A concise library for sparsity research Jooyoung Lee Wonpyo Park Nicole Mitchell Jonathan Pilault J. Obando-Ceron ... Hong-Seok Kim Yann N. Dauphin Karolina Dziugaite P. S. Castro Utku Evci 33 14 0 27 Apr 2023
STen: Productive and Efficient Sparsity in PyTorch Andrei Ivanov Nikoli Dryden Tal Ben-Nun Saleh Ashkboos Torsten Hoefler 30 4 0 15 Apr 2023
Training Large Language Models Efficiently with Sparsity and Dataflow V. Srinivasan Darshan Gandhi Urmish Thakker R. Prabhakar MoE 28 6 0 11 Apr 2023
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models Vithursan Thangarasa Abhay Gupta William Marshall Tianda Li Kevin Leong D. DeCoste Sean Lie Shreyas Saxena MoE AI4CE 16 18 0 18 Mar 2023
VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUs Geonhwa Jeong S. Damani A. Bambhaniya Eric Qin C. Hughes S. Subramoney Hyesoon Kim T. Krishna MoE 19 24 0 17 Feb 2023
Workload-Balanced Pruning for Sparse Spiking Neural Networks Ruokai Yin Youngeun Kim Yuhang Li Abhishek Moitra Nitin Satpute Anna Hambitzer Priyadarshini Panda 23 18 0 13 Feb 2023
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning Bram Grooten Ghada Sokar Shibhansh Dohare Elena Mocanu Matthew E. Taylor Mykola Pechenizkiy D. Mocanu 25 11 0 13 Feb 2023
Bi-directional Masks for Efficient N:M Sparse Training Yu-xin Zhang Yiting Luo Mingbao Lin Yunshan Zhong Jingjing Xie Fei Chao Rongrong Ji 44 15 0 13 Feb 2023
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers Shiwei Liu Zhangyang Wang 30 30 0 06 Feb 2023
STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition Yucheng Lu Shivani Agrawal Suvinay Subramanian Oleg Rybakov Chris De Sa Amir Yazdanbakhsh 16 16 0 02 Feb 2023
When Layers Play the Lottery, all Tickets Win at Initialization Artur Jordão George Correa de Araujo H. Maia Hélio Pedrini 13 3 0 25 Jan 2023
Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning Huan Wang Can Qin Yue Bai Yun Fu 32 20 0 12 Jan 2023
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Elias Frantar Dan Alistarh VLM 30 624 0 02 Jan 2023
Where to Pay Attention in Sparse Training for Feature Selection? Ghada Sokar Zahra Atashgahi Mykola Pechenizkiy D. Mocanu 25 17 0 26 Nov 2022
SNIPER Training: Single-Shot Sparse Training for Text-to-Speech Perry Lam Huayun Zhang Nancy F. Chen Berrak Sisman Dorien Herremans VLM 14 0 0 14 Nov 2022
ClassPruning: Speed Up Image Restoration Networks by Dynamic N:M Pruning Yang Zhou Yuda Song Hui Qian Xin Du VLM 28 1 0 10 Nov 2022
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 32 292 0 09 Nov 2022
Soft Masking for Cost-Constrained Channel Pruning Ryan Humble Maying Shen J. Latorre Eric Darve1 J. Álvarez 14 13 0 04 Nov 2022
EAPruning: Evolutionary Pruning for Vision Transformers and CNNs Qingyuan Li Bo-Wen Zhang Xiangxiang Chu ViT VLM 27 3 0 01 Oct 2022