DeciMamba: Exploring the Length Extrapolation Potential of Mamba

DeciMamba: Exploring the Length Extrapolation Potential of Mamba

20 June 2024

Itamar Zimerman

Shady Abu Hussein

Papers citing "DeciMamba: Exploring the Length Extrapolation Potential of Mamba"

9 / 9 papers shown

Title
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Z. Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 26 0 0 06 May 2025
LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement Zhifan Ye Kejing Xia Yonggan Fu Xin Dong Jihoon Hong Xiangchi Yuan Shizhe Diao Jan Kautz Pavlo Molchanov Yingyan Lin Mamba 17 2 0 22 Apr 2025
A Novel Mamba-based Sequential Recommendation Method Jun Yuan Mamba 34 0 0 10 Apr 2025
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 70 13 0 20 Nov 2024
Teaching VLMs to Localize Specific Objects from In-context Examples Sivan Doveh Nimrod Shabtay Wei Lin Eli Schwartz Hilde Kuehne ... Leonid Karlinsky James Glass Assaf Arbelle S. Ullman Muhammad Jehanzeb Mirza VLM 84 1 0 20 Nov 2024
Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling Yingfa Chen Xinrong Zhang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun Mamba 24 0 0 09 Oct 2024
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models Soham De Samuel L. Smith Anushan Fernando Aleksandar Botev George-Christian Muraru ... David Budden Yee Whye Teh Razvan Pascanu Nando de Freitas Çağlar Gülçehre Mamba 51 116 0 29 Feb 2024
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 476 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 234 1,508 0 31 Dec 2020