Reformer: The Efficient Transformer

13 January 2020

Papers citing "Reformer: The Efficient Transformer"

50 / 388 papers shown

Title
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 73 66 0 08 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 23 3 0 06 Oct 2021
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 24 117 0 05 Oct 2021
Classification of hierarchical text using geometric deep learning: the case of clinical trials corpus Sohrab Ferdowsi Nikolay Borissov J. Knafou P. Amini Douglas Teodoro 16 7 0 04 Oct 2021
UFO-ViT: High Performance Linear Vision Transformer without Softmax Jeonggeun Song ViT 114 20 0 29 Sep 2021
Digital Signal Processing Using Deep Neural Networks Brian Shevitski Y. Watkins Nicole Man Michael Girard AI4CE 18 4 0 21 Sep 2021
Do Long-Range Language Models Actually Use Long-Range Context? Simeng Sun Kalpesh Krishna Andrew Mattarella-Micke Mohit Iyyer RALM 25 80 0 19 Sep 2021
PnP-DETR: Towards Efficient Visual Analysis with Transformers Tao Wang Li Yuan Yunpeng Chen Jiashi Feng Shuicheng Yan ViT 22 82 0 15 Sep 2021
SHAPE: Shifted Absolute Position Embedding for Transformers Shun Kiyono Sosuke Kobayashi Jun Suzuki Kentaro Inui 233 45 0 13 Sep 2021
A Strong Baseline for Query Efficient Attacks in a Black Box Setting Rishabh Maheshwary Saket Maheshwary Vikram Pudi AAML 24 30 0 10 Sep 2021
Combining Transformers with Natural Language Explanations Federico Ruggeri Marco Lippi Paolo Torroni 17 1 0 02 Sep 2021
Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation Samuel Cahyawijaya 26 12 0 24 Aug 2021
Fastformer: Additive Attention Can Be All You Need Chuhan Wu Fangzhao Wu Tao Qi Yongfeng Huang Xing Xie 40 117 0 20 Aug 2021
Learning to Match Features with Seeded Graph Matching Network Hongkai Chen Zixin Luo Jiahui Zhang Lei Zhou Xuyang Bai Zeyu Hu Chiew-Lan Tai Long Quan 15 111 0 19 Aug 2021
Making Transformers Solve Compositional Tasks Santiago Ontañón Joshua Ainslie Vaclav Cvicek Zachary Kenneth Fisher 33 70 0 09 Aug 2021
Armour: Generalizable Compact Self-Attention for Vision Transformers Lingchuan Meng ViT 19 3 0 03 Aug 2021
A Unified Efficient Pyramid Transformer for Semantic Segmentation Fangrui Zhu Yi Zhu Li Zhang Chongruo Wu Yanwei Fu Mu Li ViT 29 29 0 29 Jul 2021
Image Fusion Transformer VS Vibashan Jeya Maria Jose Valanarasu Poojan Oza Vishal M. Patel ViT 30 116 0 19 Jul 2021
AutoBERT-Zero: Evolving BERT Backbone from Scratch Jiahui Gao Hang Xu Han Shi Xiaozhe Ren Philip L. H. Yu Xiaodan Liang Xin Jiang Zhenguo Li 19 37 0 15 Jul 2021
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 32 57 0 13 Jul 2021
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting Haixu Wu Jiehui Xu Jianmin Wang Mingsheng Long AI4TS 14 2,090 0 24 Jun 2021
Multi-head or Single-head? An Empirical Comparison for Transformer Training Liyuan Liu Jialu Liu Jiawei Han 21 32 0 17 Jun 2021
Large-Scale Chemical Language Representations Capture Molecular Structure and Properties Jerret Ross Brian M. Belgodere Vijil Chenthamarakshan Inkit Padhi Youssef Mroueh Payel Das AI4CE 21 272 0 17 Jun 2021
What Context Features Can Transformer Language Models Use? J. O'Connor Jacob Andreas KELM 21 75 0 15 Jun 2021
Training Graph Neural Networks with 1000 Layers Guohao Li Matthias Muller Bernard Ghanem V. Koltun GNN AI4CE 36 235 0 14 Jun 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 37 813 0 14 Jun 2021
GraphiT: Encoding Graph Structure in Transformers Grégoire Mialon Dexiong Chen Margot Selosse Julien Mairal 20 163 0 10 Jun 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 32 1,086 0 08 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 29 22 0 31 May 2021
Memory-Efficient Differentiable Transformer Architecture Search Yuekai Zhao Li Dong Yelong Shen Zhihua Zhang Furu Wei Weizhu Chen ViT 27 17 0 31 May 2021
Towards mental time travel: a hierarchical memory for reinforcement learning agents Andrew Kyle Lampinen Stephanie C. Y. Chan Andrea Banino Felix Hill 21 47 0 28 May 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li R. L. Jin ViT 43 105 0 28 May 2021
DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons Dawn Drain Colin B. Clement Guillermo Serrato Neel Sundaresan 17 31 0 19 May 2021
Poolingformer: Long Document Modeling with Pooling Attention Hang Zhang Yeyun Gong Yelong Shen Weisheng Li Jiancheng Lv Nan Duan Weizhu Chen 35 98 0 10 May 2021
T-EMDE: Sketching-based global similarity for cross-modal retrieval Barbara Rychalska Mikolaj Wieczorek Jacek Dąbrowski 25 0 0 10 May 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 48 1,221 0 22 Apr 2021
How to Train BERT with an Academic Budget Peter Izsak Moshe Berchansky Omer Levy 12 112 0 15 Apr 2021
Charged particle tracking via edge-classifying interaction networks G. Dezoort S. Thais Javier Mauricio Duarte Vesal Razavimaleki M. Atkinson I. Ojalvo Mark S. Neubauer P. Elmer 25 46 0 30 Mar 2021
Historical Inertia: A Neglected but Powerful Baseline for Long Sequence Time-series Forecasting Yue Cui Jiandong Xie Kai Zheng AI4TS 14 53 0 30 Mar 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 30 2,086 0 29 Mar 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 23 328 0 29 Mar 2021
The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures Sushant Singh A. Mahmood AI4TS 60 92 0 23 Mar 2021
Instance-level Image Retrieval using Reranking Transformers Fuwen Tan Jiangbo Yuan Vicente Ordonez ViT 23 89 0 22 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 25 126 0 19 Mar 2021
Generating Images with Sparse Representations C. Nash Jacob Menick Sander Dieleman Peter W. Battaglia 19 199 0 05 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 973 0 04 Mar 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 269 179 0 17 Feb 2021
Mind the Gap: Assessing Temporal Generalization in Neural Language Models Angeliki Lazaridou A. Kuncoro E. Gribovskaya Devang Agrawal Adam Liska ... Sebastian Ruder Dani Yogatama Kris Cao Susannah Young Phil Blunsom VLM 30 207 0 03 Feb 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 11 2,070 0 11 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021