Fast Transformers with Clustered Attention

v1v2 (latest)

Fast Transformers with Clustered Attention

9 July 2020

Angelos Katharopoulos

Franccois Fleuret

ArXiv (abs)PDF HTML

Papers citing "Fast Transformers with Clustered Attention"

8 / 58 papers shown

Title
A Practical Survey on Faster and Lighter Transformers Quentin Fournier G. Caron Daniel Aloise 137 104 0 26 Mar 2021
Finetuning Pretrained Transformers into RNNs Jungo Kasai Hao Peng Yizhe Zhang Dani Yogatama Gabriel Ilharco Nikolaos Pappas Yi Mao Weizhu Chen Noah A. Smith 112 67 0 24 Mar 2021
Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention Yunyang Xiong Zhanpeng Zeng Rudrasis Chakraborty Mingxing Tan G. Fung Yin Li Vikas Singh 110 526 0 07 Feb 2021
RomeBERT: Robust Training of Multi-Exit BERT Shijie Geng Peng Gao Zuohui Fu Yongfeng Zhang 81 28 0 24 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir Fahad Shahbaz Khan M. Shah ViT 385 2,560 0 04 Jan 2021
End-to-End Object Detection with Adaptive Clustering Transformer Minghang Zheng Peng Gao Renrui Zhang Kunchang Li Xiaogang Wang Hongsheng Li Hao Dong ViT 188 199 0 18 Nov 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 230 1,133 0 14 Sep 2020
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding Shuohang Wang Luowei Zhou Zhe Gan Yen-Chun Chen Yuwei Fang S. Sun Yu Cheng Jingjing Liu 93 29 0 13 Sep 2020