Scatterbrain: Unifying Sparse and Low-rank Attention Approximation

28 October 2021

Papers citing "Scatterbrain: Unifying Sparse and Low-rank Attention Approximation"

24 / 24 papers shown

Title
Low-Rank Thinning Annabelle Michael Carrell Albert Gong Abhishek Shetty Raaz Dwivedi Lester W. Mackey 56 0 0 17 Feb 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 67 5 0 28 Jan 2025
Error Correction Code Transformer: From Non-Unified to Unified Yongli Yan Jieao Zhu Tianyue Zheng Jiaqi He Linglong Dai 18 1 0 04 Oct 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 67 14 0 30 Jul 2024
LORTSAR: Low-Rank Transformer for Skeleton-based Action Recognition Soroush Oraki Harry Zhuang Jie Liang 46 1 0 19 Jul 2024
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Jiayi Yao Hanchen Li Yuhan Liu Siddhant Ray Yihua Cheng Qizheng Zhang Kuntai Du Shan Lu Junchen Jiang 42 14 0 26 May 2024
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models Dennis Wu Jerry Yao-Chieh Hu Teng-Yun Hsiao Han Liu 40 28 0 04 Apr 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 36 6 0 28 Feb 2024
How to Protect Copyright Data in Optimization of Large Language Models? T. Chu Zhao-quan Song Chiwun Yang 28 29 0 23 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 35 3 0 18 Aug 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 30 9 0 26 May 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 24 39 0 07 Apr 2023
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Vithursan Thangarasa Shreyas Saxena Abhay Gupta Sean Lie 21 3 0 21 Mar 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 19 18 0 09 Feb 2023
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention Jyotikrishna Dass Shang Wu Huihong Shi Chaojian Li Zhifan Ye Zhongfeng Wang Yingyan Lin 15 49 0 09 Nov 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022
Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design Hongxiang Fan Thomas C. P. Chau Stylianos I. Venieris Royson Lee Alexandros Kouris Wayne Luk Nicholas D. Lane Mohamed S. Abdelfattah 22 56 0 20 Sep 2022
Sublinear Time Algorithm for Online Weighted Bipartite Matching Han Hu Zhao-quan Song Runzhou Tao Zhaozhuo Xu Junze Yin Danyang Zhuo 13 7 0 05 Aug 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 56 2,017 0 27 May 2022
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao-quan Song Atri Rudra Christopher Ré 22 75 0 30 Nov 2021
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 66 1,097 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 251 2,009 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 238 578 0 12 Mar 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018