Communities
Connect sessions
AI calendar
Organizations
Join Slack
Contact Sales
Search
Open menu
Home
Papers
2004.03844
Cited By
v1
v2
v3 (latest)
On the Effect of Dropping Layers of Pre-trained Transformer Models
Computer Speech and Language (CSL), 2020
8 April 2020
Hassan Sajjad
Fahim Dalvi
Nadir Durrani
Preslav Nakov
Re-assign community
ArXiv (abs)
PDF
HTML
HuggingFace (1 upvotes)
Papers citing
"On the Effect of Dropping Layers of Pre-trained Transformer Models"
50 / 56 papers shown
Iterative Layer Pruning for Efficient Translation Inference
Yasmin Moslem
Muhammad Hazim Al Farouq
John D. Kelleher
157
2
0
26 Oct 2025
QLENS: Towards A Quantum Perspective of Language Transformers
Aditya Gupta
Kirandeep Kaur
Manya Chadha
Chirag Shah
AI4CE
167
0
0
13 Oct 2025
Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting
E. Tsunoo
Hayato Futami
Yosuke Kashiwagi
Siddhant Arora
Shinji Watanabe
135
0
0
01 Oct 2025
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction
Zhexiong Liu
Diane Litman
KELM
205
2
0
30 Sep 2025
TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning
Seohyun Lee
Wenzhi Fang
Dong-Jun Han
Seyyedali Hosseinalipour
Christopher G. Brinton
161
1
0
30 Sep 2025
Efficient Large Language Models with Zero-Shot Adjustable Acceleration
Sajjad Kachuee
M. Sharifkhani
237
0
0
01 Sep 2025
On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility View
Tao Guo
Junxiao Wang
Fushuo Huo
Laizhong Cui
Song Guo
Jie Gui
Dacheng Tao
134
0
0
22 Aug 2025
FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment
Manning Zhu
Songtao Guo
Pengzhan Zhou
Yansong Ning
Chang Han
Dewen Qiao
193
0
0
18 Aug 2025
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
Bingjie Zhang
Hongkang Li
Changlong Shi
Guowei Rong
He Zhao
Dongsheng Wang
Dandan Guo
Meng Wang
MoMe
334
1
0
10 Jun 2025
FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts
Xinyi Wang
Lirong Gao
Haobo Wang
Yiming Zhang
Junbo Zhao
MoE
277
1
0
31 May 2025
LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs
Journal of Information Security and Applications (JISA), 2025
Luis Ibanez-Lissen
Lorena Gonzalez-Manzano
José Maria De Fuentes
Nicolas Anciaux
164
3
0
30 May 2025
Efficient Speech Translation through Model Compression and Knowledge Distillation
International Workshop on Spoken Language Translation (IWSLT), 2025
Yasmin Moslem
265
2
0
26 May 2025
RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
Yi-Lin Sung
Prateek Yadav
Jialu Li
Jaehong Yoon
Joey Tianyi Zhou
MQ
373
2
0
03 Mar 2025
How Redundant Is the Transformer Stack in Speech Representation Models?
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024
Teresa Dorszewski
Albert Kjøller Jacobsen
Lenka Tětková
Lars Kai Hansen
512
3
0
20 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers
Neha Verma
Kenton W. Murray
Kevin Duh
AI4CE
422
0
0
10 Jan 2025
TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs
Lanxiang Hu
Tajana Rosing
Hao Zhang
358
2
0
15 Dec 2024
CULL-MT: Compression Using Language and Layer pruning for Machine Translation
Pedram Rostami
M. Dousti
331
3
0
10 Nov 2024
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024
Kai Yao
P. Gao
Lichun Li
Yuan Zhao
Xiaofeng Wang
Wei Wang
Jianke Zhu
196
8
0
15 Oct 2024
Persistent Topological Features in Large Language Models
Yuri Gardinazzi
Giada Panerai
Karthik Viswanathan
A. Ansuini
Alberto Cazzaniga
Matteo Biagetti
617
8
0
14 Oct 2024
Resource Allocation and Secure Wireless Communication in the Large Model-based Mobile Edge Computing System
Zefan Wang
Yitong Wang
Jun Zhao
242
2
0
29 Jun 2024
The Remarkable Robustness of LLMs: Stages of Inference?
Vedang Lad
Wes Gurnee
Max Tegmark
Max Tegmark
647
113
0
27 Jun 2024
Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent
Lin Wang
Zhichao Wang
Xiaoying Tang
267
2
0
17 Jun 2024
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Yilong Chen
Linhao Zhang
Junyuan Shang
Ying Tai
Tingwen Liu
Shuohuan Wang
Yu Sun
289
11
0
03 Jun 2024
S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs
Wei Zhong
Manasa Bharadwaj
430
10
0
30 May 2024
FedPFT: Federated Proxy Fine-Tuning of Foundation Models
Zhaopeng Peng
Xiaoliang Fan
Yufan Chen
Zheng Wang
Shirui Pan
Chenglu Wen
Ruisheng Zhang
Cheng-i Wang
306
19
0
17 Apr 2024
The Unreasonable Ineffectiveness of the Deeper Layers
Andrey Gromov
Kushal Tirumala
Hassan Shapourian
Paolo Glorioso
Daniel A. Roberts
657
192
0
26 Mar 2024
Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers
Shuzhou Yuan
Ercong Nie
Bolei Ma
Michael Farber
435
5
0
18 Feb 2024
Graph Neural Networks for Antisocial Behavior Detection on Twitter
Martina Toshevska
S. Kalajdziski
Sonja Gievska
149
2
0
28 Dec 2023
CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023
Kaiyan Zhang
Ning Ding
Biqing Qi
Xuekai Zhu
Xinwei Long
Bowen Zhou
315
5
0
24 Oct 2023
Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks
Zixuan Ke
Bing Liu
Wenhan Xiong
Asli Celikyilmaz
Haoran Li
CLL
298
11
0
13 Oct 2023
Can pruning make Large Language Models more efficient?
Sia Gholami
Marwan Omar
350
21
0
06 Oct 2023
ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
International Conference on Learning Representations (ICLR), 2023
Yi-Lin Sung
Jaehong Yoon
Mohit Bansal
VLM
357
22
0
04 Oct 2023
CoMFLP: Correlation Measure based Fast Search on ASR Layer Pruning
Interspeech (Interspeech), 2023
W. Liu
Zhiyuan Peng
Tan Lee
273
2
0
21 Sep 2023
Multilingual Text Representation
Fahim Faisal
261
1
0
02 Sep 2023
Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models
International Conference on Learning Representations (ICLR), 2023
Seungcheol Park
Ho-Jin Choi
U. Kang
VLM
349
13
0
07 Aug 2023
Deep Model Compression Also Helps Models Capture Ambiguity
Annual Meeting of the Association for Computational Linguistics (ACL), 2023
Hancheol Park
Jong C. Park
368
2
0
12 Jun 2023
PruMUX: Augmenting Data Multiplexing with Model Compression
Annual Meeting of the Association for Computational Linguistics (ACL), 2023
Yushan Su
Vishvak Murahari
Karthik Narasimhan
Keqin Li
317
3
0
24 May 2023
Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling
Y. Zhu
Xuebing Yang
Yuanyuan Wu
Wensheng Zhang
MedIm
350
4
0
15 May 2023
The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
Anastasiia Grishina
Max Hort
Leon Moonen
356
15
0
08 May 2023
Gradient-Free Structured Pruning with Unlabeled Data
International Conference on Machine Learning (ICML), 2023
Azade Nova
H. Dai
Dale Schuurmans
SyDa
371
38
0
07 Mar 2023
Offsite-Tuning: Transfer Learning without Full Model
Guangxuan Xiao
Ji Lin
Song Han
299
100
0
09 Feb 2023
Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers
Yuta Matsumoto
Benjamin Heinzerling
Masashi Yoshikawa
Kentaro Inui
AIFin
269
5
0
17 Jan 2023
On the Transformation of Latent Space in Fine-Tuned NLP Models
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022
Nadir Durrani
Hassan Sajjad
Fahim Dalvi
Firoj Alam
292
20
0
23 Oct 2022
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022
Shuo Xie
Jiahao Qiu
Ankita Pasad
Li Du
Qing Qu
Hongyuan Mei
276
16
0
18 Oct 2022
Efficient Methods for Natural Language Processing: A Survey
Transactions of the Association for Computational Linguistics (TACL), 2022
Marcos Vinícius Treviso
Ji-Ung Lee
Tianchu Ji
Betty van Aken
Qingqing Cao
...
Emma Strubell
Niranjan Balasubramanian
Leon Derczynski
Iryna Gurevych
Roy Schwartz
500
151
0
31 Aug 2022
Embedding Recycling for Language Models
Findings (Findings), 2022
Jon Saad-Falcon
Amanpreet Singh
Luca Soldaini
Mike DÁrcy
Arman Cohan
Doug Downey
KELM
231
5
0
11 Jul 2022
Discovering Salient Neurons in Deep NLP Models
Journal of machine learning research (JMLR), 2022
Nadir Durrani
Fahim Dalvi
Hassan Sajjad
KELM
MILM
367
20
0
27 Jun 2022
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency
Annual Meeting of the Association for Computational Linguistics (ACL), 2022
Yanyang Li
Fuli Luo
Runxin Xu
Songfang Huang
Fei Huang
Liwei Wang
194
3
0
06 Apr 2022
A Fast Post-Training Pruning Framework for Transformers
Neural Information Processing Systems (NeurIPS), 2022
Woosuk Kwon
Sehoon Kim
Michael W. Mahoney
Joseph Hassoun
Kurt Keutzer
A. Gholami
292
213
0
29 Mar 2022
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
Knowledge Discovery and Data Mining (KDD), 2022
Ruixuan Liu
Fangzhao Wu
Chuhan Wu
Yanlin Wang
Lingjuan Lyu
Hong Chen
Xing Xie
FedML
258
102
0
16 Feb 2022
1
2
Next
Page 1 of 2
