FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

11 July 2024

Papers citing "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision"

32 / 82 papers shown

Title
NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs Nursena Köprücü Destiny Okpekpe Antonio Orvieto Mamba 23 1 0 31 Oct 2024
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Y. Lu Kurt Keutzer Jianfei Chen Song Han MQ 55 9 0 25 Oct 2024
POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference Aditya K Kamath Ramya Prabhu Jayashree Mohan Simon Peter R. Ramjee Ashish Panwar 46 9 0 23 Oct 2024
Scaling Diffusion Language Models via Adaptation from Autoregressive Models Shansan Gong Shivam Agarwal Yizhe Zhang Jiacheng Ye Lin Zheng ... Peilin Zhao W. Bi Jiawei Han Hao Peng Lingpeng Kong AI4CE 65 14 0 23 Oct 2024
FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs Haoran Lin Xianzhi Yu Kang Zhao Lu Hou Zongyuan Zhan ... Cheng Qian Ying Zhang Yinfei Pan Yu Yang Weiguo Liu LRM 11 1 0 22 Oct 2024
Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song Yufa Zhou 52 15 0 12 Oct 2024
Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM Haiyue Ma Jian Liu Ronny Krashinsky 16 0 0 10 Oct 2024
Differential Transformer Tianzhu Ye Li Dong Yuqing Xia Yutao Sun Yi Zhu Gao Huang Furu Wei 42 0 0 07 Oct 2024
UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation Zixuan Li Jing Xiong Fanghua Ye Chuanyang Zheng Xun Wu ... Xiaodan Liang Chengming Li Zhenan Sun Lingpeng Kong Ngai Wong RALM UQLM 27 2 0 03 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 80 18 0 03 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 73 1 0 02 Oct 2024
Scrambled text: training Language Models to correct OCR errors using synthetic data Jonathan Bourne SyDa 34 2 0 29 Sep 2024
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction Zhenmei Shi Yifei Ming Xuan-Phi Nguyen Yingyu Liang Shafiq Joty 73 27 0 25 Sep 2024
INT-FlashAttention: Enabling Flash Attention for INT8 Quantization Shimao Chen Zirui Liu Zhiying Wu Ce Zheng Peizhuang Cong Zihan Jiang Yuhan Wu Lei Su Tong Yang MQ VLM 39 3 0 25 Sep 2024
Efficiently Dispatching Flash Attention For Partially Filled Attention Masks Agniv Sharma Jonas Geiping 14 0 0 23 Sep 2024
A Tighter Complexity Analysis of SparseGPT Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 63 21 0 22 Aug 2024
LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference Zhiwen Mo Lei Wang Jianyu Wei Zhichen Zeng Shijie Cao ... Naifeng Jing Ting Cao Jilong Xue Fan Yang Mao Yang 51 4 0 12 Aug 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 69 7 0 29 Jul 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 31 0 25 Jul 2024
Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget Vikash Sehwag Xianghao Kong Jingtao Li Michael Spranger Lingjuan Lyu DiffM 32 8 0 22 Jul 2024
LLM Inference Serving: Survey of Recent Advances and Opportunities Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 73 15 0 17 Jul 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 38 2 0 21 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 54 0 11 Jun 2024
Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers Rya Sanovar Srikant Bharadwaj Renée St. Amant Victor Rühle Saravan Rajmohan 49 6 0 17 May 2024
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention Ramya Prabhu Ajay Nayak Jayashree Mohan R. Ramjee Ashish Panwar VLM 50 24 0 07 May 2024
Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level Ali Hassani Wen-mei W. Hwu Humphrey Shi 23 2 0 07 Mar 2024
Massive Activations in Large Language Models Mingjie Sun Xinlei Chen J. Zico Kolter Zhuang Liu 60 64 0 27 Feb 2024
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks Albert Tseng Jerry Chee Qingyao Sun Volodymyr Kuleshov Christopher De Sa MQ 123 91 0 06 Feb 2024
ReAct: Synergizing Reasoning and Acting in Language Models Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan Yuan Cao LLMAG ReLM LRM 223 2,413 0 06 Oct 2022
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman M. Shoeybi Michael Siu Hao Wu BDL VLM MQ 65 119 0 12 Sep 2022
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 228 578 0 12 Mar 2020