SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

28 January 2025

Mohammad Mozaffari

Amir Yazdanbakhsh

Papers citing "SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs"

12 / 12 papers shown

Title
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity Daniel Haziza Timothy Chou Dhruv Choudhary Luca Wehrstedt Francisco Massa Jiecao Yu Geonhwa Jeong Supriya Rao Patrick Labatut Jesse Cai 42 0 0 20 Mar 2025
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation Z. Liu Ruijie Zhang Z. Wang Zi Yang Paul Hovland Bogdan Nicolae Franck Cappello Z. Zhang 42 0 0 16 Feb 2025
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 29 7 0 08 Oct 2024
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Stephen Zhang V. Papyan VLM 43 1 0 20 Sep 2024
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization Haoran You Yipin Guo Yichao Fu Wei Zhou Huihong Shi Xiaofan Zhang Souvik Kundu Amir Yazdanbakhsh Y. Lin KELM 44 7 0 10 Jun 2024
Accelerating Transformer Pre-training with 2:4 Sparsity Yuezhou Hu Kang Zhao Weiyu Huang Jianfei Chen Jun Zhu 57 7 0 02 Apr 2024
Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers A. Bambhaniya Amir Yazdanbakhsh Suvinay Subramanian Sheng-Chun Kao Shivani Agrawal Utku Evci Tushar Krishna 49 16 0 07 Feb 2024
Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask Sheng-Chun Kao Amir Yazdanbakhsh Suvinay Subramanian Shivani Agrawal Utku Evci T. Krishna 48 12 0 15 Sep 2022
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 44 110 0 16 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 139 684 0 31 Jan 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 248 1,986 0 31 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018