Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

23 May 2019

Papers citing "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned"

50 / 169 papers shown

Title
Sparse Interventions in Language Models with Differentiable Masking Nicola De Cao Leon Schmid Dieuwke Hupkes Ivan Titov 30 27 0 13 Dec 2021
Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View Di Jin Elena Sergeeva W. Weng Geeticka Chauhan Peter Szolovits OOD 31 55 0 05 Dec 2021
Interpreting Deep Learning Models in Natural Language Processing: A Review Xiaofei Sun Diyi Yang Xiaoya Li Tianwei Zhang Yuxian Meng Han Qiu Guoyin Wang Eduard H. Hovy Jiwei Li 17 44 0 20 Oct 2021
On the Pitfalls of Analyzing Individual Neurons in Language Models Omer Antverg Yonatan Belinkov MILM 22 49 0 14 Oct 2021
Global Vision Transformer Pruning with Hessian-Aware Saliency Huanrui Yang Hongxu Yin Maying Shen Pavlo Molchanov Hai Helen Li Jan Kautz ViT 30 38 0 10 Oct 2021
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 19 117 0 05 Oct 2021
On the Prunability of Attention Heads in Multilingual BERT Aakriti Budhraja Madhura Pande Pratyush Kumar Mitesh M. Khapra 42 4 0 26 Sep 2021
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models Goro Kobayashi Tatsuki Kuribayashi Sho Yokoi Kentaro Inui 158 46 0 15 Sep 2021
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders Han He Jinho D. Choi 43 87 0 14 Sep 2021
The Grammar-Learning Trajectories of Neural Language Models Leshem Choshen Guy Hacohen D. Weinshall Omri Abend 27 28 0 13 Sep 2021
Document-level Entity-based Extraction as Template Generation Kung-Hsiang Huang Sam Tang Nanyun Peng 14 54 0 10 Sep 2021
Bag of Tricks for Optimizing Transformer Efficiency Ye Lin Yanyang Li Tong Xiao Jingbo Zhu 21 6 0 09 Sep 2021
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems Potsawee Manakul Mark J. F. Gales 13 5 0 08 Sep 2021
Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience G. Chrysostomou Nikolaos Aletras 24 16 0 31 Aug 2021
T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLP Raymond Li Wen Xiao Lanjun Wang Hyeju Jang Giuseppe Carenini ViT 20 23 0 31 Aug 2021
Layer-wise Model Pruning based on Mutual Information Chun Fan Jiwei Li Xiang Ao Fei Wu Yuxian Meng Xiaofei Sun 38 19 0 28 Aug 2021
Differentiable Subset Pruning of Transformer Heads Jiaoda Li Ryan Cotterell Mrinmaya Sachan 37 53 0 10 Aug 2021
A Dynamic Head Importance Computation Mechanism for Neural Machine Translation Akshay Goindani Manish Shrivastava 11 4 0 03 Aug 2021
Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives Ben Saunders Necati Cihan Camgöz Richard Bowden SLR 25 50 0 23 Jul 2021
A Primer on Pretrained Multilingual Language Models Sumanth Doddapaneni Gowtham Ramesh Mitesh M. Khapra Anoop Kunchukuttan Pratyush Kumar LRM 43 73 0 01 Jul 2021
AutoFormer: Searching Transformers for Visual Recognition Minghao Chen Houwen Peng Jianlong Fu Haibin Ling ViT 36 259 0 01 Jul 2021
The MultiBERTs: BERT Reproductions for Robustness Analysis Thibault Sellam Steve Yadlowsky Jason W. Wei Naomi Saphra Alexander DÁmour ... Iulia Turc Jacob Eisenstein Dipanjan Das Ian Tenney Ellie Pavlick 22 93 0 30 Jun 2021
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering Ahjeong Seo Gi-Cheon Kang J. Park Byoung-Tak Zhang 13 53 0 19 Jun 2021
What Context Features Can Transformer Language Models Use? J. O'Connor Jacob Andreas KELM 21 75 0 15 Jun 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 37 813 0 14 Jun 2021
On Compositional Generalization of Neural Machine Translation Yafu Li Yongjing Yin Yulong Chen Yue Zhang 148 44 0 31 May 2021
LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond Daniel Loureiro A. Jorge Jose Camacho-Collados 33 26 0 26 May 2021
Rationalization through Concepts Diego Antognini Boi Faltings FAtt 13 19 0 11 May 2021
Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses Aina Garí Soler Marianna Apidianaki MILM 201 68 0 29 Apr 2021
Code Structure Guided Transformer for Source Code Summarization Shuzheng Gao Cuiyun Gao Yulan He Jichuan Zeng L. Nie Xin Xia Michael R. Lyu 15 96 0 19 Apr 2021
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation Mozhdeh Gheini Xiang Ren Jonathan May LRM 20 105 0 18 Apr 2021
Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm Dongkuan Xu Ian En-Hsu Yen Jinxi Zhao Zhibin Xiao VLM AAML 28 56 0 18 Apr 2021
Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey Danielle Saunders AI4CE 11 85 0 14 Apr 2021
DirectProbe: Studying Representations without Classifiers Yichu Zhou Vivek Srikumar 27 27 0 13 Apr 2021
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation Shuhao Gu Yang Feng Wanying Xie CLL AI4CE 25 27 0 25 Mar 2021
Structured Co-reference Graph Attention for Video-grounded Dialogue Junyeong Kim Sunjae Yoon Dahyun Kim Chang-Dong Yoo 18 26 0 24 Mar 2021
The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures Sushant Singh A. Mahmood AI4TS 55 92 0 23 Mar 2021
Learning Calibrated-Guidance for Object Detection in Aerial Images Zongqi Wei Dong Liang Dong-Ming Zhang Liyan Zhang Qixiang Geng Mingqiang Wei Huiyu Zhou 22 35 0 21 Mar 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021
Transformer Feed-Forward Layers Are Key-Value Memories Mor Geva R. Schuster Jonathan Berant Omer Levy KELM 22 741 0 29 Dec 2020
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade Lei Li Yankai Lin Deli Chen Shuhuai Ren Peng Li Jie Zhou Xu Sun 26 51 0 29 Dec 2020
Multi-Head Self-Attention with Role-Guided Masks Dongsheng Wang Casper Hansen Lucas Chaves Lima Christian B. Hansen Maria Maistro J. Simonsen Christina Lioma 21 1 0 22 Dec 2020
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning Hanrui Wang Zhekai Zhang Song Han 20 373 0 17 Dec 2020
Positional Artefacts Propagate Through Masked Language Model Embeddings Ziyang Luo Artur Kulmizev Xiaoxi Mao 22 41 0 09 Nov 2020
Rethinking embedding coupling in pre-trained language models Hyung Won Chung Thibault Févry Henry Tsai Melvin Johnson Sebastian Ruder 93 142 0 24 Oct 2020
Pretrained Transformers for Text Ranking: BERT and Beyond Jimmy J. Lin Rodrigo Nogueira Andrew Yates VLM 219 608 0 13 Oct 2020
SMYRF: Efficient Attention using Asymmetric Clustering Giannis Daras Nikita Kitaev Augustus Odena A. Dimakis 23 44 0 11 Oct 2020
On the Sub-Layer Functionalities of Transformer Decoder Yilin Yang Longyue Wang Shuming Shi Prasad Tadepalli Stefan Lee Zhaopeng Tu 24 27 0 06 Oct 2020
TernaryBERT: Distillation-aware Ultra-low Bit BERT Wei Zhang Lu Hou Yichun Yin Lifeng Shang Xiao Chen Xin Jiang Qun Liu MQ 22 208 0 27 Sep 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 74 1,101 0 14 Sep 2020