Diagonal State Spaces are as Effective as Structured State Spaces

27 March 2022

Papers citing "Diagonal State Spaces are as Effective as Structured State Spaces"

50 / 226 papers shown

Title
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling Yair Schiff Chia-Hsiang Kao Aaron Gokaslan Tri Dao Albert Gu Volodymyr Kuleshov Mamba 16 78 0 05 Mar 2024
MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection Tianxiang Chen Zi Ye Zhentao Tan Tao Gong Yue-bo Wu Qi Chu Bin Liu Nenghai Yu Jieping Ye Mamba 48 43 0 04 Mar 2024
The Hidden Attention of Mamba Models Ameen Ali Itamar Zimerman Lior Wolf Mamba 26 57 0 03 Mar 2024
Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation Haruka Ezoe Kazuhiro Sato 13 0 0 25 Feb 2024
Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning Chi-Sheng Chen Guan-Ying Chen Dong Zhou Di Jiang Daishi Chen Mamba 28 11 0 24 Feb 2024
State Space Models for Event Cameras Nikola Zubić Mathias Gehrig Davide Scaramuzza 28 35 0 23 Feb 2024
Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers Markus Hiller Krista A. Ehinger Tom Drummond 28 0 0 19 Feb 2024
PointMamba: A Simple State Space Model for Point Cloud Analysis Dingkang Liang Xin Zhou Wei Xu Xingkui Zhu Zhikang Zou Xiaoqing Ye Xinyu Wang Xiang Bai 79 87 0 16 Feb 2024
On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era Matteo Tiezzi Michele Casoni Alessandro Betti Tommaso Guidi Marco Gori S. Melacci 16 9 0 12 Feb 2024
Scalable Diffusion Models with State Space Backbone Zhengcong Fei Mingyuan Fan Changqian Yu Junshi Huang 59 33 0 08 Feb 2024
Is Mamba Capable of In-Context Learning? Riccardo Grazzi Julien N. Siems Simon Schrodi Thomas Brox Frank Hutter 18 20 0 05 Feb 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 21 26 0 05 Feb 2024
MambaByte: Token-free Selective State Space Model Junxiong Wang Tushaar Gangavarapu Jing Nathan Yan Alexander M. Rush Mamba 12 34 0 24 Jan 2024
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model Lianghui Zhu Bencheng Liao Qian Zhang Xinlong Wang Wenyu Liu Xinggang Wang Mamba 32 699 0 17 Jan 2024
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts Maciej Pióro Kamil Ciebiera Krystian Król Jan Ludziejewski Michał Krutul Jakub Krajewski Szymon Antoniak Piotr Miłoś Marek Cygan Sebastian Jaszczur MoE Mamba 15 54 0 08 Jan 2024
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng E. Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 19 74 0 17 Dec 2023
Learning Long Sequences in Spiking Neural Networks Matei Ioan Stan Oliver Rhodes 24 10 0 14 Dec 2023
Spectral State Space Models Naman Agarwal Daniel Suo Xinyi Chen Elad Hazan 17 11 0 11 Dec 2023
Gated Linear Attention Transformers with Hardware-Efficient Training Songlin Yang Bailin Wang Yikang Shen Rameswar Panda Yoon Kim 34 138 0 11 Dec 2023
Structured state-space models are deep Wiener models Fabio Bonassi Carl R. Andersson Per Mattsson Thomas B. Schon 25 3 0 11 Dec 2023
Recurrent Distance Filtering for Graph Representation Learning Yuhui Ding Antonio Orvieto Bobby He Thomas Hofmann GNN 19 6 0 03 Dec 2023
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Tianyu Ding Tianyi Chen Haidong Zhu Jiachen Jiang Yiqi Zhong Jinxin Zhou Guangzhi Wang Zhihui Zhu Ilya Zharkov Luming Liang 25 21 0 01 Dec 2023
Diffusion Models Without Attention Jing Nathan Yan Jiatao Gu Alexander M. Rush 14 60 0 30 Nov 2023
On the Long Range Abilities of Transformers Itamar Zimerman Lior Wolf 8 4 0 28 Nov 2023
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity Zhen Qin Yiran Zhong 18 6 0 15 Nov 2023
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores Daniel Y. Fu Hermann Kumbong Eric N. D. Nguyen Christopher Ré VLM 20 28 0 10 Nov 2023
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability Jishnu Ray Chowdhury Cornelia Caragea 19 5 0 08 Nov 2023
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions Stefano Massaroli Michael Poli Daniel Y. Fu Hermann Kumbong Rom N. Parnichkun ... Atri Rudra Ce Zhang Christopher Ré Stefano Ermon Yoshua Bengio 21 12 0 28 Oct 2023
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer Qingru Zhang Dhananjay Ram Cole Hawkins Sheng Zha Tuo Zhao 19 15 0 19 Oct 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 11 40 0 04 Oct 2023
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors Ido Amos Jonathan Berant Ankit Gupta 14 24 0 04 Oct 2023
Multi-Dimensional Hyena for Spatial Inductive Bias Itamar Zimerman Lior Wolf ViT 17 4 0 24 Sep 2023
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models Zican Dong Tianyi Tang Junyi Li Wayne Xin Zhao Ji-Rong Wen RALM ALM 10 34 0 23 Sep 2023
Augmenting conformers with structured state-space sequence models for online speech recognition Haozhe Shan Albert Gu Zhong Meng Weiran Wang Krzysztof Choromanski Tara N. Sainath RALM 11 4 0 15 Sep 2023
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks Ting-Han Fan Ta-Chung Chi Alexander I. Rudnicky LRM 14 5 0 14 Sep 2023
Gated recurrent neural networks discover attention Nicolas Zucchet Seijin Kobayashi Yassir Akram J. Oswald Maxime Larcher Angelika Steger João Sacramento 18 8 0 04 Sep 2023
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer Zhen Qin Dong Li Weigao Sun Weixuan Sun Xuyang Shen ... Yunshen Wei Baohong Lv Xiao Luo Yu Qiao Yiran Zhong 30 15 0 27 Jul 2023
Efficient Beam Tree Recursion Jishnu Ray Chowdhury Cornelia Caragea 6 3 0 20 Jul 2023
Facing Off World Model Backbones: RNNs, Transformers, and S4 Fei Deng Junyeong Park Sungjin Ahn 17 24 0 05 Jul 2023
Efficient Dynamics Modeling in Interactive Environments with Koopman Theory Arnab Kumar Mondal Siba Smarak Panigrahi Sai Rajeswar K. Siddiqi Siamak Ravanbakhsh 13 0 0 20 Jun 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 43 13 0 19 Jun 2023
Block-State Transformers Mahan Fathi Jonathan Pilault Orhan Firat C. Pal Pierre-Luc Bacon Ross Goroshin 15 17 0 15 Jun 2023
2-D SSM: A General Spatial Layer for Visual Transformers Ethan Baron Itamar Zimerman Lior Wolf 18 14 0 11 Jun 2023
Decision S4: Efficient Sequence-Based RL via State Spaces Layers Shmuel Bar-David Itamar Zimerman Eliya Nachmani Lior Wolf OffRL 21 22 0 08 Jun 2023
Exploring the Promise and Limits of Real-Time Recurrent Learning Kazuki Irie Anand Gopalakrishnan Jürgen Schmidhuber 14 15 0 30 May 2023
A Quantitative Review on Language Model Efficiency Research Meng-Long Jiang Hy Dang Lingbo Tong 17 0 0 28 May 2023
Revisiting Structured Variational Autoencoders Yixiu Zhao Scott W. Linderman BDL DRL 17 8 0 25 May 2023
Online learning of long-range dependencies Nicolas Zucchet Robert Meier Simon Schug Asier Mujika João Sacramento CLL 33 18 0 25 May 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 19 9 0 24 May 2023
Toeplitz Neural Network for Sequence Modeling Zhen Qin Xiaodong Han Weixuan Sun Bowen He Dong Li Dongxu Li Yuchao Dai Lingpeng Kong Yiran Zhong AI4TS ViT 22 40 0 08 May 2023