Long Range Arena: A Benchmark for Efficient Transformers

8 November 2020

Papers citing "Long Range Arena: A Benchmark for Efficient Transformers"

34 / 134 papers shown

Title
Dynamic N:M Fine-grained Structured Sparse Attention Mechanism Zhaodong Chen Yuying Quan Zheng Qu L. Liu Yufei Ding Yuan Xie 16 21 0 28 Feb 2022
FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks Maksim Zubkov Daniil Gavrilov 15 0 0 23 Feb 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 21 211 0 17 Feb 2022
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 28 29 0 27 Jan 2022
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks Lei Cheng Ruslan Khalitov Tong Yu Zhirong Yang 20 32 0 06 Jan 2022
LongT5: Efficient Text-To-Text Transformer for Long Sequences Mandy Guo Joshua Ainslie David C. Uthus Santiago Ontanon Jianmo Ni Yun-hsuan Sung Yinfei Yang VLM 31 307 0 15 Dec 2021
Discourse-Aware Soft Prompting for Text Generation Marjan Ghazvininejad Vladimir Karpukhin Vera Gor Asli Celikyilmaz 23 6 0 10 Dec 2021
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 18 139 0 10 Dec 2021
Attention-Based Model and Deep Reinforcement Learning for Distribution of Event Processing Tasks A. Mazayev F. Al-Tam N. Correia 15 5 0 07 Dec 2021
Learning Query Expansion over the Nearest Neighbor Graph Benjamin Klein Lior Wolf 11 1 0 05 Dec 2021
Show Your Work: Scratchpads for Intermediate Computation with Language Models Maxwell Nye Anders Andreassen Guy Gur-Ari Henryk Michalewski Jacob Austin ... Aitor Lewkowycz Maarten Bosma D. Luan Charles Sutton Augustus Odena ReLM LRM 17 695 0 30 Nov 2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao-quan Song Atri Rudra Christopher Ré 22 75 0 30 Nov 2021
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 14 1,644 0 31 Oct 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 30 98 0 25 Oct 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 16 161 0 22 Oct 2021
Inductive Biases and Variable Creation in Self-Attention Mechanisms Benjamin L. Edelman Surbhi Goel Sham Kakade Cyril Zhang 22 115 0 19 Oct 2021
Metadata Shaping: Natural Language Annotations for the Tail Simran Arora Sen Wu Enci Liu Christopher Ré 17 0 0 16 Oct 2021
Bank transactions embeddings help to uncover current macroeconomics Maria Begicheva Alexey Zaytsev 11 4 0 14 Oct 2021
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline Xiangyang Liu Tianxiang Sun Junliang He Jiawen Wu Lingling Wu Xinyu Zhang Hao Jiang Zhao Cao Xuanjing Huang Xipeng Qiu ELM 19 46 0 13 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 19 3 0 06 Oct 2021
Classification of hierarchical text using geometric deep learning: the case of clinical trials corpus Sohrab Ferdowsi Nikolay Borissov J. Knafou P. Amini Douglas Teodoro 16 7 0 04 Oct 2021
LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation Jian-Yu Guan Zhuoer Feng Yamei Chen Ru He Xiaoxi Mao Changjie Fan Minlie Huang 31 31 0 30 Aug 2021
Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation Samuel Cahyawijaya 26 12 0 24 Aug 2021
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning Paul Pu Liang Yiwei Lyu Xiang Fan Zetian Wu Yun Cheng ... Peter Wu Michelle A. Lee Yuke Zhu Ruslan Salakhutdinov Louis-Philippe Morency VLM 21 157 0 15 Jul 2021
The Benchmark Lottery Mostafa Dehghani Yi Tay A. Gritsenko Zhe Zhao N. Houlsby Fernando Diaz Donald Metzler Oriol Vinyals 15 89 0 14 Jul 2021
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 23 57 0 13 Jul 2021
Rethinking Search: Making Domain Experts out of Dilettantes Donald Metzler Yi Tay Dara Bahri Marc Najork LRM 18 46 0 05 May 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 28 2,078 0 29 Mar 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 21 328 0 29 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 10 216 0 09 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 972 0 04 Mar 2021
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 63 1,097 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 2,009 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 238 578 0 12 Mar 2020