Long Range Arena: A Benchmark for Efficient Transformers

8 November 2020

Papers citing "Long Range Arena: A Benchmark for Efficient Transformers"

50 / 134 papers shown

Title
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens Zhanpeng Zeng Cole Hawkins Min-Fong Hong Aston Zhang Nikolaos Pappas Vikas Singh Shuai Zheng 19 6 0 07 May 2023
The Role of Global and Local Context in Named Entity Recognition Arthur Amalvy Vincent Labatut Richard Dufour 30 4 0 04 May 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 19 1 0 17 Apr 2023
Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection Ashay Patel Petru-Daniel Tudosiu W. H. Pinaya G. Cook Vicky Goh Sebastien Ourselin M. Jorge Cardoso OOD ViT MedIm 18 11 0 14 Apr 2023
Scallop: A Language for Neurosymbolic Programming Ziyang Li Jiani Huang Mayur Naik ReLM LRM NAI 14 30 0 10 Apr 2023
Dialogue-Contextualized Re-ranking for Medical History-Taking Jian Zhu Ilya Valmianski Anitha Kannan 19 1 0 04 Apr 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 19 6 0 16 Feb 2023
Simple Hardware-Efficient Long Convolutions for Sequence Modeling Daniel Y. Fu Elliot L. Epstein Eric N. D. Nguyen A. Thomas Michael Zhang Tri Dao Atri Rudra Christopher Ré 11 51 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 19 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 17 8 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 17 9 0 02 Feb 2023
Byte Pair Encoding for Symbolic Music Nathan Fradet Nicolas Gutowski F. Chhel Jean-Pierre Briot 22 15 0 27 Jan 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 43 367 0 28 Dec 2022
JEMMA: An Extensible Java Dataset for ML4Code Applications Anjan Karmakar Miltiadis Allamanis Romain Robbes VLM 10 3 0 18 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 117 36 0 15 Dec 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 30 3 0 27 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 29 69 0 19 Oct 2022
Neural Attentive Circuits Nasim Rahaman M. Weiß Francesco Locatello C. Pal Yoshua Bengio Bernhard Schölkopf Erran L. Li Nicolas Ballas 19 6 0 14 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification Ilias Chalkidis Xiang Dai Manos Fergadiotis Prodromos Malakasiotis Desmond Elliott 30 33 0 11 Oct 2022
WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability Yufan Zhuang Zihan Wang Fangbo Tao Jingbo Shang ViT AI4TS 15 3 0 05 Oct 2022
Liquid Structural State-Space Models Ramin Hasani Mathias Lechner Tsun-Hsuan Wang Makram Chahine Alexander Amini Daniela Rus AI4TS 97 95 0 26 Sep 2022
Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design Hongxiang Fan Thomas C. P. Chau Stylianos I. Venieris Royson Lee Alexandros Kouris Wayne Luk Nicholas D. Lane Mohamed S. Abdelfattah 22 56 0 20 Sep 2022
Efficient Quantized Sparse Matrix Operations on Tensor Cores Shigang Li Kazuki Osawa Torsten Hoefler 69 31 0 14 Sep 2022
Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation Nadine Behrmann S. Golestaneh Zico Kolter Juergen Gall M. Noroozi 16 71 0 01 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences Peng Qi Guangtao Wang Jing Huang 8 0 0 03 Aug 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 21 9 0 01 Aug 2022
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks Nigamaa Nayakanti Rami Al-Rfou Aurick Zhou Kratarth Goel Khaled S. Refaat Benjamin Sapp AI4TS 40 234 0 12 Jul 2022
An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics Huan Yee Koh Jiaxin Ju Ming Liu Shirui Pan 73 122 0 03 Jul 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 26 231 0 27 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 14 294 0 23 Jun 2022
Long Range Graph Benchmark Vijay Prakash Dwivedi Ladislav Rampášek Mikhail Galkin Alipanah Parviz Guy Wolf A. Luu Dominique Beaini 24 193 0 16 Jun 2022
VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models Wangchunshu Zhou Yan Zeng Shizhe Diao Xinsong Zhang CoGe VLM 17 13 0 30 May 2022
Chefs' Random Tables: Non-Trigonometric Random Features Valerii Likhosherstov K. Choromanski Kumar Avinava Dubey Frederick Liu Tamás Sarlós Adrian Weller 18 17 0 30 May 2022
Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning Aniket Didolkar Kshitij Gupta Anirudh Goyal Nitesh B. Gundavarapu Alex Lamb Nan Rosemary Ke Yoshua Bengio AI4CE 110 17 0 30 May 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 56 2,017 0 27 May 2022
Recipe for a General, Powerful, Scalable Graph Transformer Ladislav Rampášek Mikhail Galkin Vijay Prakash Dwivedi A. Luu Guy Wolf Dominique Beaini 43 507 0 25 May 2022
Temporally Precise Action Spotting in Soccer Videos Using Dense Detection Anchors J. C. V. Soares Avijit Shah Topojoy Biswas 30 32 0 20 May 2022
Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention Tong Yu Ruslan Khalitov Lei Cheng Zhirong Yang MoE 16 10 0 22 Apr 2022
Usage of specific attention improves change point detection Anna Dmitrienko Evgenia Romanenkova Alexey Zaytsev 11 0 0 18 Apr 2022
Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models Phyllis Ang Bhuwan Dhingra Lisa Wu Wills 22 6 0 15 Apr 2022
A Call for Clarity in Beam Search: How It Works and When It Stops Jungo Kasai Keisuke Sakaguchi Ronan Le Bras Dragomir R. Radev Yejin Choi Noah A. Smith 24 6 0 11 Apr 2022
A sequence-to-sequence approach for document-level relation extraction John Giorgi Gary D. Bader Bo Wang 15 52 0 03 Apr 2022
TubeDETR: Spatio-Temporal Video Grounding with Transformers Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid ViT 12 94 0 30 Mar 2022
A Fast Transformer-based General-Purpose Lossless Compressor Yushun Mao Yufei Cui Tei-Wei Kuo C. Xue ViT AI4CE 11 28 0 30 Mar 2022
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection Xin Huang A. Khetan Rene Bidart Zohar S. Karnin 17 14 0 27 Mar 2022
Diagonal State Spaces are as Effective as Structured State Spaces Ankit Gupta Albert Gu Jonathan Berant 34 290 0 27 Mar 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 16 94 0 11 Mar 2022