The Hidden Attention of Mamba Models

The Hidden Attention of Mamba Models

3 March 2024

Itamar Zimerman

Lior Wolf

Papers citing "The Hidden Attention of Mamba Models"

18 / 18 papers shown

Title
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism Aviv Bick Eric P. Xing Albert Gu RALM 81 0 0 22 Apr 2025
From S4 to Mamba: A Comprehensive Survey on Structured State Space Models Shriyank Somvanshi Md Monzurul Islam Mahmuda Sultana Mimi Sazzad Bin Bashar Polock Gaurab Chhetri Subasish Das Mamba AI4TS 40 0 0 22 Mar 2025
MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations Hongyu Ke Jack Morris K. Oguchi Xiaofei Cao Yongkang Liu Haoxin Wang Yi Ding Mamba 71 0 0 18 Mar 2025
A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers Ashish Parmanand Pandey Alan John Varghese Sarang Patil Mengjia Xu Mamba 90 0 0 15 Dec 2024
UmambaTSF: A U-shaped Multi-Scale Long-Term Time Series Forecasting Method Using Mamba Li Wu Wenbin Pei Jiulong Jiao Qiang Zhang Mamba AI4TS 20 2 0 15 Oct 2024
Lambda-Skip Connections: the architectural component that prevents Rank Collapse Federico Arangath Joseph Jerome Sieber M. Zeilinger Carmen Amo Alonso 33 0 0 14 Oct 2024
Cross-attention Inspired Selective State Space Models for Target Sound Extraction Donghang Wu Yiwen Wang Xihong Wu T. Qu Mamba 26 3 0 07 Sep 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 67 13 0 20 Jun 2024
MambaLRP: Explaining Selective State Space Sequence Models F. Jafari G. Montavon Klaus-Robert Müller Oliver Eberle Mamba 47 9 0 11 Jun 2024
TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification Md. Atik Ahamed Qiang Cheng Mamba 55 1 0 06 Jun 2024
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks Jerome Sieber Carmen Amo Alonso A. Didier M. Zeilinger Antonio Orvieto AAML 42 7 0 24 May 2024
Graph Mamba: Towards Learning on Graphs with State Space Models Ali Behrouz Farnoosh Hashemi AI4CE 104 57 0 13 Feb 2024
Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data Shufan Li Harkanwar Singh Aditya Grover Mamba 78 56 0 08 Feb 2024
VM-UNet: Vision Mamba UNet for Medical Image Segmentation Jiacheng Ruan Suncheng Xiang Mamba 67 241 0 04 Feb 2024
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation Jun Ma Feifei Li Bo Wang Mamba 77 314 0 09 Jan 2024
Structured State Space Models for In-Context Reinforcement Learning Chris Xiaoxuan Lu Yannick Schroecker Albert Gu Emilio Parisotto Jakob N. Foerster Satinder Singh Feryal M. P. Behbahani AI4TS 84 80 0 07 Mar 2023
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 71 220 0 21 Feb 2022
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020