SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

17 December 2020

Song Han

Papers citing "SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning"

50 / 160 papers shown

Title
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Z. Qiu Z. Wang Bo Zheng Zeyu Huang Kaiyue Wen ... Fei Huang Suozhi Huang Dayiheng Liu Jingren Zhou Junyang Lin MoE 23 0 0 10 May 2025
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax Zayd Muhammad Kawakibi Zuhri Erland Hilman Fuadi Alham Fikri Aji 31 0 0 29 Apr 2025
Efficient Pretraining Length Scaling Bohong Wu Shen Yan Sijun Zhang Jianqiao Lu Yutao Zeng Ya Wang Xun Zhou 121 0 0 21 Apr 2025
TC-MGC: Text-Conditioned Multi-Grained Contrastive Learning for Text-Video Retrieval Xiaolun Jing Genke Yang Jian Chu 26 0 0 07 Apr 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 34 0 0 06 Apr 2025
Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization Minsu Kim Seongmin Hong RyeoWook Ko S. Choi Hunjong Lee Junsoo Kim J. Kim Jongse Park 57 0 0 24 Mar 2025
AxBERT: An Interpretable Chinese Spelling Correction Method Driven by Associative Knowledge Network Fanyu Wang Hangyu Zhu Zhenping Xie 40 0 0 04 Mar 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 166 0 0 03 Mar 2025
CipherPrune: Efficient and Scalable Private Transformer Inference Yancheng Zhang J. Xue Mengxin Zheng Mimi Xie Mingzhe Zhang Lei Jiang Qian Lou 53 2 0 24 Feb 2025
PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System Yintao He Haiyu Mao Christina Giannoula Mohammad Sadrosadati Juan Gómez Luna Huawei Li Xiaowei Li Ying Wang O. Mutlu 41 5 0 21 Feb 2025
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding Konstantin Berestizshevsky Renzo Andri Lukas Cavigelli 80 1 0 12 Feb 2025
Ditto: Accelerating Diffusion Model via Temporal Value Similarity Sungbin Kim Hyunwuk Lee Wonho Cho Mincheol Park Won Woo Ro 56 1 0 20 Jan 2025
MPCache: MPC-Friendly KV Cache Eviction for Efficient Private Large Language Model Inference Wenxuan Zeng Ye Dong Jinjin Zhou Junming Ma Jin Tan Runsheng Wang Meng Li 49 0 0 12 Jan 2025
EXION: Exploiting Inter- and Intra-Iteration Output Sparsity for Diffusion Models Jaehoon Heo Adiwena Putra Jieon Yoon Sungwoong Yune Hangyeol Lee Ji-Hoon Kim Joo-Young Kim DiffM 55 1 0 10 Jan 2025
Multimodal joint prediction of traffic spatial-temporal data with graph sparse attention mechanism and bidirectional temporal convolutional network Dongran Zhang Jiangnan Yan K. Polat A. Alhudhaif Jun Li AI4TS 29 11 0 31 Dec 2024
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Haozhao Wang Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 119 1 0 18 Dec 2024
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning Yiwu Zhong Zhuoming Liu Yin Li Liwei Wang 82 2 0 04 Dec 2024
SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors M. Rakka J. Li Guohao Dai A. Eltawil M. Fouda Fadi J. Kurdahi 65 1 0 26 Nov 2024
MixPE: Quantization and Hardware Co-design for Efficient LLM Inference Yu Zhang M. Wang Lancheng Zou Wulong Liu Hui-Ling Zhen M. Yuan Bei Yu MQ 74 1 0 25 Nov 2024
BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference Junqi Zhao Zhijin Fang Shu Li Shaohui Yang Shichao He 37 2 0 30 Oct 2024
MoDification: Mixture of Depths Made Easy C. Zhang M. Zhong Qimeng Wang Xuantao Lu Zheyu Ye ... Yan Gao Yao Hu Kehai Chen Min Zhang Dawei Song VLM MoE 35 2 0 18 Oct 2024
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs Yizhao Gao Zhichen Zeng Dayou Du Shijie Cao Hayden Kwok-Hay So ... Junjie Lai Mao Yang Ting Cao Fan Yang M. Yang 50 18 0 17 Oct 2024
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Yanyue Xie Zhi Zhang Ding Zhou Cong Xie Ziang Song Xin Liu Yanzhi Wang Xue Lin An Xu LLMAG 38 3 0 15 Oct 2024
In-context KV-Cache Eviction for LLMs via Attention-Gate Zihao Zeng Bokai Lin Tianqi Hou Hao Zhang Zhijie Deng 38 1 0 15 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 31 7 0 08 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 57 15 0 06 Oct 2024
AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization Yifan Tan Haoze Wang Chao Yan Yangdong Deng MQ 26 2 0 25 Sep 2024
FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAs Ehsan Kabir Md. Arafat Kabir Austin R. J. Downey Jason D. Bakos David Andrews Miaoqing Huang GNN 26 0 0 21 Sep 2024
ProTEA: Programmable Transformer Encoder Acceleration on FPGA Ehsan Kabir Jason D. Bakos David Andrews Miaoqing Huang 14 0 0 21 Sep 2024
Recall: Empowering Multimodal Embedding for Edge Devices Dongqi Cai Shangguang Wang Chen Peng Zeling Zhang Mengwei Xu 27 3 0 09 Sep 2024
An Analog and Digital Hybrid Attention Accelerator for Transformers with Charge-based In-memory Computing Ashkan Moradifirouzabadi Divya Sri Dodla Mingu Kang 14 0 0 08 Sep 2024
Hardware Acceleration of LLMs: A comprehensive survey and comparison Nikoletta Koilia C. Kachris 50 5 0 05 Sep 2024
Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching Sungmin Yun Kwanhee Kyung Juhwan Cho Jaewan Choi Jongmin Kim Byeongho Kim Sukhan Lee Kyomin Sohn Jung Ho Ahn MoE 36 5 0 02 Sep 2024
TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computing Abhishek Moitra Abhiroop Bhattacharjee Youngeun Kim Priyadarshini Panda ViT 32 2 0 22 Aug 2024
Practical token pruning for foundation models in few-shot conversational virtual assistant systems Haode Qi Cheng Qian Jian Ni Pratyush Singh Reza Fazeli Gengyu Wang Zhongzheng Shu Eric Wayne Juergen Bross 20 0 0 21 Aug 2024
Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture Yu Feng Weikai Lin Zihan Liu Jingwen Leng Minyi Guo Han Zhao Xiaofeng Hou Jieru Zhao Yuhao Zhu 34 3 0 13 Aug 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 38 9 0 10 Aug 2024
A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder Hyun Rae Jo Dong Kun Shin 32 4 0 30 Jul 2024
Sparse Refinement for Efficient High-Resolution Semantic Segmentation Zhijian Liu Zhuoyang Zhang Samir Khaki Shang Yang Haotian Tang Chenfeng Xu Kurt Keutzer Song Han SSeg 46 1 0 26 Jul 2024
Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimation Junyoung Park Myeonggu Kang Yunki Han Yang-Gon Kim Jaekang Shin Lee-Sup Kim 19 0 0 21 Jul 2024
TorchGT: A Holistic System for Large-scale Graph Transformer Training Mengdie Zhang Jie Sun Qi Hu Peng Sun Zeke Wang Yonggang Wen Tianwei Zhang GNN 39 2 0 19 Jul 2024
Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference Ghadeer Jaradat M. Tolba Ghada Alsuhli Hani Saleh Mahmoud Al-Qutayri Thanos Stouraitis Baker Mohammad 37 0 0 17 Jul 2024
Characterizing Prompt Compression Methods for Long Context Inference Siddharth Jha Lutfi Eren Erdogan Sehoon Kim Kurt Keutzer A. Gholami 26 5 0 11 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Huiqiang Jiang Yucheng Li Chengruidong Zhang Qianhui Wu Xufang Luo ... Amir H. Abdi Dongsheng Li Chin-Yew Lin Yuqing Yang L. Qiu 67 82 0 02 Jul 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 43 7 0 02 Jul 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 44 17 0 21 Jun 2024
Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference Donghyeon Joo Ramyad Hadidi S. Feizi Bahar Asgari MQ 29 0 0 17 Jun 2024
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization Jungi Lee Wonbeom Lee Jaewoong Sim MQ 29 14 0 16 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 8 0 04 Jun 2024