FlashAttention: Fast and Memory-Efficient Exact Attention with
IO-Awareness

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

27 May 2022

Christopher Ré

Papers citing "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"

18 / 1,418 papers shown

Title
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 20 105 0 31 Aug 2022
Boosting Distributed Training Performance of the Unpadded BERT Model Jinle Zeng Min Li Zhihua Wu Jiaqi Liu Yuang Liu Dianhai Yu Yanjun Ma 6 11 0 17 Aug 2022
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment Qiang Chen Xiaokang Chen Jian Wang Shan Zhang Kun Yao Haocheng Feng Junyu Han Errui Ding Gang Zeng Jingdong Wang ViT 23 71 0 26 Jul 2022
DETRs with Hybrid Matching Ding Jia Yuhui Yuan Hao He Xiao-pei Wu Haojun Yu Weihong Lin Lei-huan Sun Chao Zhang Hanhua Hu 16 179 0 26 Jul 2022
Efficient High-Resolution Deep Learning: A Survey Arian Bakhtiarnia Qi Zhang Alexandros Iosifidis MedIm 11 17 0 26 Jul 2022
Vision Transformers: From Semantic Segmentation to Dense Prediction Li Zhang Jiachen Lu Sixiao Zheng Xinxuan Zhao Xiatian Zhu Yanwei Fu Tao Xiang Jianfeng Feng Philip H. S. Torr ViT 16 7 0 19 Jul 2022
Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding Leonid Boytsov David Akinpelu Tianyi Lin Fangwei Gao Yutian Zhao Jeffrey Huang Nipun Katyal Eric Nyberg 23 9 0 04 Jul 2022
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks Tuan Dinh Yuchen Zeng Ruisu Zhang Ziqian Lin Michael Gira Shashank Rajput Jy-yong Sohn Dimitris Papailiopoulos Kangwook Lee LMTD 26 123 0 14 Jun 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 26 518 0 13 Jun 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 71 220 0 21 Feb 2022
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 13 137 0 10 Dec 2021
An Empirical Study: Extensive Deep Temporal Point Process Haitao Lin Cheng Tan Lirong Wu Zhangyang Gao Stan. Z. Li AI4TS 6 12 0 19 Oct 2021
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren H. Dai Zihang Dai Mengjiao Yang J. Leskovec Dale Schuurmans Bo Dai 73 66 0 12 Jul 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 260 165 0 17 Feb 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 228 502 0 12 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019
Neural Legal Judgment Prediction in English Ilias Chalkidis Ion Androutsopoulos Nikolaos Aletras AILaw ELM 98 279 0 05 Jun 2019