Simple Local Attentions Remain Competitive for Long-Context Tasks

Simple Local Attentions Remain Competitive for Long-Context Tasks

14 December 2021

Diana Liskovich

Yashar Mehdad

Papers citing "Simple Local Attentions Remain Competitive for Long-Context Tasks"

13 / 13 papers shown

Title
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 60 5 0 04 Nov 2024
Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis Zhiyu Jin Xuli Shen Bin Li Xiangyang Xue 18 36 0 14 Jun 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 27 9 0 24 May 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 19 18 0 09 Feb 2023
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 24 115 0 20 Dec 2022
How Far are We from Robust Long Abstractive Summarization? Huan Yee Koh Jiaxin Ju He Zhang Ming Liu Shirui Pan HILM 23 39 0 30 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification Ilias Chalkidis Xiang Dai Manos Fergadiotis Prodromos Malakasiotis Desmond Elliott 30 33 0 11 Oct 2022
Adapting Pretrained Text-to-Text Models for Long Text Sequences Wenhan Xiong Anchit Gupta Shubham Toshniwal Yashar Mehdad Wen-tau Yih RALM VLM 49 30 0 21 Sep 2022
Shortformer: Better Language Modeling using Shorter Inputs Ofir Press Noah A. Smith M. Lewis 219 88 0 31 Dec 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 63 1,097 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 2,009 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 238 578 0 12 Mar 2020