Efficient Transformers: A Survey

14 September 2020

Papers citing "Efficient Transformers: A Survey"

50 / 633 papers shown

Title
CageViT: Convolutional Activation Guided Efficient Vision Transformer Hao Zheng Jinbao Wang Xiantong Zhen H. Chen Jingkuan Song Feng Zheng ViT 10 0 0 17 May 2023
DLUE: Benchmarking Document Language Understanding Ruoxi Xu Hongyu Lin Xinyan Guan Xianpei Han Yingfei Sun Le Sun ELM 8 0 0 16 May 2023
SKI to go Faster: Accelerating Toeplitz Neural Networks via Asymmetric Kernels Alexander Moreno Jonathan Mei Luke Walters 8 0 0 15 May 2023
Visual Tuning Bruce X. B. Yu Jianlong Chang Haixin Wang Lin Liu Shijie Wang ... Lingxi Xie Haojie Li Zhouchen Lin Qi Tian Chang Wen Chen VLM 39 37 0 10 May 2023
Code Execution with Pre-trained Language Models Chenxiao Liu Shuai Lu Weizhu Chen Daxin Jiang Alexey Svyatkovskiy Shengyu Fu Neel Sundaresan Nan Duan ELM 20 21 0 08 May 2023
Leveraging BERT Language Model for Arabic Long Document Classification Muhammad Al-Qurishi 6 0 0 04 May 2023
Unlimiformer: Long-Range Transformers with Unlimited Length Input Amanda Bertsch Uri Alon Graham Neubig Matthew R. Gormley RALM 94 122 0 02 May 2023
IMP: Iterative Matching and Pose Estimation with Adaptive Pooling Fei Xue Ignas Budvytis R. Cipolla 31 13 0 28 Apr 2023
SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish Dmytro Kalpakchi Johan Boye SyDa 6 3 0 27 Apr 2023
Learning to Compress Prompts with Gist Tokens Jesse Mu Xiang Lisa Li Noah D. Goodman VLM 14 204 0 17 Apr 2023
Neural Attention Forests: Transformer-Based Forest Improvement A. Konstantinov Lev V. Utkin A. Lukashin Vladimir Mulukha 14 4 0 12 Apr 2023
SELFormer: Molecular Representation Learning via SELFIES Language Models Atakan Yüksel Erva Ulusoy Atabey Ünlü Tunca Dogan 25 54 0 10 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 24 39 0 07 Apr 2023
EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation Y. Shi H. Cai Amin Ansari Fatih Porikli MDE 83 16 0 06 Apr 2023
Inductive biases in deep learning models for weather prediction Jannik Thümmel Matthias Karlbauer S. Otte C. Zarfl Georg Martius ... Thomas Scholten Ulrich Friedrich V. Wulfmeyer B. Goswami Martin Volker Butz AI4CE 31 4 0 06 Apr 2023
Multiscale Attention via Wavelet Neural Operators for Vision Transformers Anahita Nekoozadeh M. Ahmadzadeh Zahra Mardani ViT 22 2 0 22 Mar 2023
Building artificial neural circuits for domain-general cognition: a primer on brain-inspired systems-level architecture Jascha Achterberg Danyal Akarca Moataz Assem Moritz P. Heimbach D. Astle John Duncan AI4CE 28 4 0 21 Mar 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 35 46 0 21 Mar 2023
Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers Jaehoon Yoo Semin Kim Doyup Lee Chiheon Kim Seunghoon Hong 21 3 0 20 Mar 2023
CerviFormer: A Pap-smear based cervical cancer classification method using cross attention and latent transformer Bhaswati Singha Deo M. Pal P. Panigrahi A. Pradhan MedIm 17 21 0 17 Mar 2023
BiFormer: Vision Transformer with Bi-Level Routing Attention Lei Zhu Xinjiang Wang Zhanghan Ke Wayne Zhang Rynson W. H. Lau 123 438 0 15 Mar 2023
HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism Yuguang Yang Y. Pan Jingjing Yin Jiangyu Han Lei Ma Heng Lu 15 8 0 15 Mar 2023
Learning Accurate Template Matching with Differentiable Coarse-to-Fine Correspondence Refinement Zhirui Gao Renjiao Yi Zheng Qin Yunfan Ye Chenyang Zhu Kaiping Xu 16 7 0 15 Mar 2023
X-Former: In-Memory Acceleration of Transformers S. Sridharan Jacob R. Stevens Kaushik Roy A. Raghunathan GNN 8 33 0 13 Mar 2023
Graph Neural Network contextual embedding for Deep Learning on Tabular Data Mario Villaizán-Vallelado Matteo Salvatori B. Carro Antonio J. Sánchez-Esguevillas AI4CE LMTD 17 14 0 11 Mar 2023
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference Haiyang Huang Newsha Ardalani Anna Y. Sun Liu Ke Hsien-Hsin S. Lee Anjali Sridhar Shruti Bhosale Carole-Jean Wu Benjamin C. Lee MoE 65 21 0 10 Mar 2023
Exphormer: Sparse Transformers for Graphs Hamed Shirzad A. Velingker B. Venkatachalam Danica J. Sutherland A. Sinop 11 96 0 10 Mar 2023
TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders Mingyue Cheng Qi Liu Zhiding Liu Haotong Zhang Rujiao Zhang Enhong Chen AI4TS 29 42 0 01 Mar 2023
AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers Shikhar Tuli N. Jha 12 31 0 28 Feb 2023
A Survey on Long Text Modeling with Transformers Zican Dong Tianyi Tang Lunyi Li Wayne Xin Zhao VLM 19 52 0 28 Feb 2023
GNOT: A General Neural Operator Transformer for Operator Learning Zhongkai Hao Zhengyi Wang Hang Su Chengyang Ying Yinpeng Dong Songming Liu Ze Cheng Jian Song Jun Zhu AI4CE 11 157 0 28 Feb 2023
TBFormer: Two-Branch Transformer for Image Forgery Localization Yaqi Liu Binbin Lv Xin Jin Xiaoyue Chen Xiaokun Zhang ViT 18 25 0 25 Feb 2023
Deep Learning for Video-Text Retrieval: a Review Cunjuan Zhu Qi Jia Wei-Neng Chen Yanming Guo Yu Liu 19 14 0 24 Feb 2023
Natural Language Processing in the Legal Domain Daniel Martin Katz D. Hartung Lauritz Gerlach Abhik Jana M. Bommarito ELM AILaw 4 53 0 23 Feb 2023
FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification Mingyue Cheng Qi Liu Zhiding Liu Zhi Li Yucong Luo Enhong Chen AI4TS 11 31 0 20 Feb 2023
Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey S. Dubey Satish Kumar Singh ViT 18 32 0 17 Feb 2023
Role of Bias Terms in Dot-Product Attention Mahdi Namazifar Devamanyu Hazarika Dilek Z. Hakkani-Tür 13 3 0 16 Feb 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 19 6 0 16 Feb 2023
Lightweight Transformers for Clinical Natural Language Processing Omid Rohanian Mohammadmahdi Nouriborji Hannah Jauncey Samaneh Kouchaki Isaric Clinical Characterisation Group Lei A. Clifton L. Merson David A. Clifton MedIm LM&MA 8 12 0 09 Feb 2023
KDEformer: Accelerating Transformers via Kernel Density Estimation A. Zandieh Insu Han Majid Daliri Amin Karbasi 23 37 0 05 Feb 2023
PSST! Prosodic Speech Segmentation with Transformers Nathan Roll C. Graham Simon Todd VLM 15 5 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 15 9 0 02 Feb 2023
The Power of External Memory in Increasing Predictive Model Capacity Cenk Baykal D. Cutler Nishanth Dikkala Nikhil Ghosh Rina Panigrahy Xin Wang KELM 11 0 0 31 Jan 2023
Alternating Updates for Efficient Transformers Cenk Baykal D. Cutler Nishanth Dikkala Nikhil Ghosh Rina Panigrahy Xin Wang MoE 27 5 0 30 Jan 2023
Exploring Attention Map Reuse for Efficient Transformer Neural Networks Kyuhong Shim Jungwook Choi Wonyong Sung ViT 12 3 0 29 Jan 2023
Predicting Visit Cost of Obstructive Sleep Apnea using Electronic Healthcare Records with Transformer Zhaoyang Chen Lina Siltala-Li Mikko Lassila Pekka Malo Eeva Vilkkumaa T. Saaresranta A. Virkki 8 4 0 28 Jan 2023
On the Connection Between MPNN and Graph Transformer Chen Cai Truong Son-Hy Rose Yu Yusu Wang 21 50 0 27 Jan 2023
A Survey on Transformers in Reinforcement Learning Wenzhe Li Hao Luo Zichuan Lin Chongjie Zhang Zongqing Lu Deheng Ye OffRL MU AI4CE 22 55 0 08 Jan 2023
Robust representations of oil wells' intervals via sparse attention mechanism Alina Rogulina N. Baramiia Valerii Kornilov Sergey Petrakov Alexey Zaytsev AI4TS OOD 6 1 0 29 Dec 2022
Cramming: Training a Language Model on a Single GPU in One Day Jonas Geiping Tom Goldstein MoE 28 83 0 28 Dec 2022