FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

17 July 2023

Papers citing "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning"

50 / 161 papers shown

Title
OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning Zhaochen Su Linjie Li Mingyang Song Yunzhuo Hao Zhengyuan Yang ... Guanjie Chen Jiawei Gu Juntao Li Xiaoye Qu Yu Cheng OffRL LRM 13 0 0 13 May 2025
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling Xiao Wang Jong Youl Choi Takuya Kurihaya Isaac Lyngaas Hong-Jun Yoon ... Dali Wang Peter Thornton Prasanna Balaprakash M. Ashfaq Dan Lu 26 0 0 07 May 2025
No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves D. Jiang Mengmeng Wang Liuzhuozheng Li Lei Zhang Haoyu Wang Wei Wei Guang Dai Yanning Zhang Jingdong Wang DiffM 44 0 0 05 May 2025
Phantora: Live GPU Cluster Simulation for Machine Learning System Performance Estimation Jianxing Qin Jingrong Chen Xinhao Kong Yongji Wu Liang Luo Z. Wang Ying Zhang Tingjun Chen Alvin R. Lebeck Danyang Zhuo 48 0 0 02 May 2025
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding Bradley McDanel S. Zhang Y. Hu Zining Liu MoE 47 0 0 02 May 2025
RayZer: A Self-supervised Large View Synthesis Model Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao ... Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos 62 0 0 01 May 2025
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook Muyi Bao Shuchang Lyu Zhaoyang Xu Huiyu Zhou Jinchang Ren Shiming Xiang X. Li Guangliang Cheng Mamba 72 0 0 01 May 2025
GPU Performance Portability needs Autotuning Burkhard Ringlein Thomas Parnell Radu Stoica 61 0 0 30 Apr 2025
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax Zayd Muhammad Kawakibi Zuhri Erland Hilman Fuadi Alham Fikri Aji 31 0 0 29 Apr 2025
Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User X. Wang Chunxuan Xia Junyi Li Fanzhe Meng Lei Huang Jinpeng Wang Wayne Xin Zhao Ji-Rong Wen 60 0 0 29 Apr 2025
LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields Zhengqin Li Dilin Wang Ka Chen Zhaoyang Lv Thu Nguyen-Phuoc ... Yufeng Zhu Carl S. Marshall Yufeng Ren Richard A. Newcombe Zhao Dong 3DV 83 0 0 28 Apr 2025
LR-IAD:Mask-Free Industrial Anomaly Detection with Logical Reasoning Peijian Zeng Feiyan Pang Zhanbo Wang Aimin Yang 68 0 0 28 Apr 2025
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation Yi Lu Wanxu Zhao Xin Zhou Chenxin An C. Wang ... Jun Zhao Tao Ji Tao Gui Qi Zhang Xuanjing Huang 39 0 0 26 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 51 0 0 24 Apr 2025
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark Hanlei Zhang Zhuohang Li Yeshuang Zhu Hua Xu Peiwu Wang Haige Zhu Jie Zhou Jinchao Zhang 25 0 0 23 Apr 2025
Efficient Pretraining Length Scaling Bohong Wu Shen Yan Sijun Zhang Jianqiao Lu Yutao Zeng Ya Wang Xun Zhou 61 0 0 21 Apr 2025
How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos? Rahul Thapa Andrew Li Qingyang Wu B. He Yuki Sahashi ... Angela Zhang Ben Athiwaratkun S. Song David Ouyang James Y. Zou LM&MA 45 0 0 19 Apr 2025
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models Xinlin Zhuang Jiahui Peng Ren Ma Y. Wang Tianyi Bai Xingjian Wei Jiantao Qiu Chi Zhang Ying Qian Conghui He 39 0 0 19 Apr 2025
CSPLADE: Learned Sparse Retrieval with Causal Language Models Zhichao Xu Aosong Feng Yijun Tian Haibo Ding Lin Leee Cheong RALM 40 0 0 15 Apr 2025
APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay Akshara Prabhakar Z. Liu Weiran Yao Jianguo Zhang Ming Zhu ... Juan Carlos Niebles Shelby Heinecke H. Wang S. Caiming Xiong VGen 79 2 0 04 Apr 2025
Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training Zhijun Wang Jiahuan Li Hao Zhou Rongxiang Weng J. Wang Xin Huang Xue Han Junlan Feng Chao Deng Shujian Huang LRM 48 1 0 02 Apr 2025
FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning Hang Guo Yawei Li Taolin Zhang J. Wang Tao Dai Shu-Tao Xia Luca Benini 67 1 0 30 Mar 2025
Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities Raman Dutt Harleen Hanspal Guoxuan Xia Petru-Daniel Tudosiu Alexander Black Yongxin Yang Steven G. McDonagh Sarah Parisot MoE 38 0 0 28 Mar 2025
Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence Yijiong Yu LRM AIMat 90 1 0 26 Mar 2025
Strong Baseline: Multi-UAV Tracking via YOLOv12 with BoT-SORT-ReID Yu-Hsi Chen 39 0 0 21 Mar 2025
ML-Triton, A Multi-Level Compilation and Language Extension to Triton GPU Programming Dewei Wang Wei Zhu Liyang Ling Ettore Tiotto Quintin Wang Whitney Tsang Julian Opperman Jacky Deng 41 0 0 19 Mar 2025
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels M. Beck Korbinian Poppel Phillip Lippe Sepp Hochreiter 59 1 0 18 Mar 2025
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models Teng Wang Zhangyi Jiang Zhenqi He Wenhan Yang Yanan Zheng Zeyu Li Zifan He Shenyang Tong Hailei Gong LRM 90 1 0 16 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu-Xi Cheng MoE 61 1 0 07 Mar 2025
EuroBERT: Scaling Multilingual Encoders for European Languages Nicolas Boizard Hippolyte Gisserot-Boukhlef Duarte M. Alves André F. T. Martins Ayoub Hammal ... Maxime Peyrard Nuno M. Guerreiro Patrick Fernandes Ricardo Rei Pierre Colombo 67 1 0 07 Mar 2025
Predicting Team Performance from Communications in Simulated Search-and-Rescue Ali Jalal-Kamali Nikolos Gurney David Pynadath AI4TS 108 0 0 05 Mar 2025
END: Early Noise Dropping for Efficient and Effective Context Denoising Hongye Jin Pei Chen Jingfeng Yang Z. Wang Meng-Long Jiang ... X. Zhang Zheng Li Tianyi Liu Huasheng Li Bing Yin 72 0 0 26 Feb 2025
AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms Feiyang Chen Yu Cheng Lei Wang Yuqing Xia Ziming Miao ... Fan Yang J. Xue Zhi Yang M. Yang H. Chen 71 1 0 24 Feb 2025
UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings Layba Fiaz Munief Hassan Tahir Sana Shams Sarmad Hussain 49 0 0 24 Feb 2025
Vision-LSTM: xLSTM as Generic Vision Backbone Benedikt Alkin M. Beck Korbinian Poppel Sepp Hochreiter Johannes Brandstetter VLM 56 39 0 24 Feb 2025
DeepInteraction++: Multi-Modality Interaction for Autonomous Driving Zeyu Yang Nan Song Wei Li Xiatian Zhu L. Zhang Philip H. S. Torr 69 4 0 24 Feb 2025
Fine-Tuning Qwen 2.5 3B for Realistic Movie Dialogue Generation Kartik Gupta VGen 46 0 0 22 Feb 2025
Surface Vision Mamba: Leveraging Bidirectional State Space Model for Efficient Spherical Manifold Representation Rongzhao He Weihao Zheng Leilei Zhao Ying Wang Dalin Zhu Dan Wu Bin Hu Mamba 89 0 0 21 Feb 2025
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks Thomas Schmied Thomas Adler Vihang Patil M. Beck Korbinian Poppel Johannes Brandstetter G. Klambauer Razvan Pascanu Sepp Hochreiter 70 4 0 21 Feb 2025
Neural Attention Search Difan Deng Marius Lindauer 85 0 0 21 Feb 2025
NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation Zhiyuan Liu Yanchen Luo Han Huang Enzhi Zhang Sihang Li Junfeng Fang Yaorui Shi X. Wang Kenji Kawaguchi Tat-Seng Chua 100 3 0 18 Feb 2025
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale Fan Zhou Zengzhi Wang Qian Liu Junlong Li Pengfei Liu ALM 100 14 0 17 Feb 2025
Low-Rank Thinning Annabelle Michael Carrell Albert Gong Abhishek Shetty Raaz Dwivedi Lester W. Mackey 53 0 0 17 Feb 2025
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment Jingcheng Deng Zhongtao Jiang Liang Pang Liwei Chen Kun Xu Zihao Wei Huawei Shen Xueqi Cheng 49 1 0 17 Feb 2025
Strada-LLM: Graph LLM for traffic prediction Seyed Mohamad Moghadas Yangxintong Lyu Bruno Cornelis Alexandre Alahi Adrian Munteanu AI4TS 56 0 0 17 Feb 2025
KernelBench: Can LLMs Write Efficient GPU Kernels? Anne Ouyang Simon Guo Simran Arora Alex L. Zhang William Hu Christopher Ré Azalia Mirhoseini ALM 38 1 0 14 Feb 2025
GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He Peng Ye Yuchen Ren Yuan Yuan Lei Chen AI4TS AI4CE 88 0 0 13 Feb 2025
LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models Tzu-Tao Chang Shivaram Venkataraman VLM 99 0 0 04 Feb 2025
Twilight: Adaptive Attention Sparsity with Hierarchical Top- $p$ Pruning C. Lin Jiaming Tang Shuo Yang Hanshuo Wang Tian Tang Boyu Tian Ion Stoica Song Han Mingyu Gao 90 2 0 04 Feb 2025
Longer Attention Span: Increasing Transformer Context Length with Sparse Graph Processing Techniques Nathaniel Tomczak Sanmukh Kuppannagari 89 0 0 31 Jan 2025