SparseBERT: Rethinking the Importance Analysis in Self-attention

SparseBERT: Rethinking the Importance Analysis in Self-attention

25 February 2021

Jiahui Gao

Hang Xu

Xiaodan Liang

Papers citing "SparseBERT: Rethinking the Importance Analysis in Self-attention"

13 / 13 papers shown

Title
Efficient Pretraining Length Scaling Bohong Wu Shen Yan Sijun Zhang Jianqiao Lu Yutao Zeng Ya Wang Xun Zhou 124 0 0 21 Apr 2025
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models Amit Dhurandhar Tejaswini Pedapati Ronny Luss Soham Dan Aurélie C. Lozano Payel Das Georgios Kollias 22 3 0 28 Feb 2024
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurélien Lucchi Thomas Hofmann 34 53 0 25 May 2023
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention Wenhao Li Xiaoyuan Yi Jinyi Hu Maosong Sun Xing Xie 21 0 0 14 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 30 3 0 27 Oct 2022
Revisiting Over-smoothing in BERT from the Perspective of Graph Han Shi Jiahui Gao Hang Xu Xiaodan Liang Zhenguo Li Lingpeng Kong Stephen M. S. Lee James T. Kwok 22 71 0 17 Feb 2022
Discourse-Aware Soft Prompting for Text Generation Marjan Ghazvininejad Vladimir Karpukhin Vera Gor Asli Celikyilmaz 23 6 0 10 Dec 2021
Transformer Acceleration with Dynamic Sparse Attention Liu Liu Zheng Qu Zhaodong Chen Yufei Ding Yuan Xie 19 20 0 21 Oct 2021
AutoBERT-Zero: Evolving BERT Backbone from Scratch Jiahui Gao Hang Xu Han Shi Xiaozhe Ren Philip L. H. Yu Xiaodan Liang Xin Jiang Zhenguo Li 19 37 0 15 Jul 2021
Few-Shot Segmentation via Cycle-Consistent Transformer Gengwei Zhang Guoliang Kang Yi Yang Yunchao Wei ViT 19 177 0 04 Jun 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 259 2,013 0 28 Jul 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,950 0 20 Apr 2018
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,743 0 26 Sep 2016