ETC: Encoding Long and Structured Inputs in Transformers

ETC: Encoding Long and Structured Inputs in Transformers

17 April 2020

Joshua Ainslie

Santiago Ontanon

Zachary Kenneth Fisher

Sumit Sanghai

Papers citing "ETC: Encoding Long and Structured Inputs in Transformers"

17 / 17 papers shown

Title
Scaling Transformer to 1M tokens and beyond with RMT Aydar Bulatov Yuri Kuratov Yermek Kapushev Mikhail Burtsev LRM 14 87 0 19 Apr 2023
MUX-PLMs: Data Multiplexing for High-throughput Language Models Vishvak Murahari A. Deshpande Carlos E. Jimenez Izhak Shafran Mingqiu Wang Yuan Cao Karthik Narasimhan MoE 13 5 0 24 Feb 2023
Recurrent Memory Transformer Aydar Bulatov Yuri Kuratov Mikhail Burtsev CLL 11 101 0 14 Jul 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 26 231 0 27 Jun 2022
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation Qihang Yu Huiyu Wang Dahun Kim Siyuan Qiao Maxwell D. Collins Yukun Zhu Hartwig Adam Alan Yuille Liang-Chieh Chen ViT MedIm 32 89 0 17 Jun 2022
Fine-tuning Image Transformers using Learnable Memory Mark Sandler A. Zhmoginov Max Vladymyrov Andrew Jackson ViT 21 47 0 29 Mar 2022
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection Xin Huang A. Khetan Rene Bidart Zohar S. Karnin 17 14 0 27 Mar 2022
LongT5: Efficient Text-To-Text Transformer for Long Sequences Mandy Guo Joshua Ainslie David C. Uthus Santiago Ontanon Jianmo Ni Yun-hsuan Sung Yinfei Yang VLM 31 307 0 15 Dec 2021
Focal Self-attention for Local-Global Interactions in Vision Transformers Jianwei Yang Chunyuan Li Pengchuan Zhang Xiyang Dai Bin Xiao Lu Yuan Jianfeng Gao ViT 42 428 0 01 Jul 2021
Thinking Like Transformers Gail Weiss Yoav Goldberg Eran Yahav AI4CE 24 126 0 13 Jun 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 23 328 0 29 Mar 2021
Code Summarization with Structure-induced Transformer Hongqiu Wu Hai Zhao Min Zhang 30 84 0 29 Dec 2020
Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval Akari Asai Eunsol Choi RALM 37 51 0 22 Oct 2020
Open Question Answering over Tables and Text Wenhu Chen Ming-Wei Chang Eva Schlinger W. Wang William W. Cohen LMTD RALM 31 193 0 20 Oct 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 74 1,101 0 14 Sep 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 579 0 12 Mar 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,956 0 20 Apr 2018