On Pruning State-Space LLMs

Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation.

@article{ghattas2025_2502.18886,
  title={ On Pruning State-Space LLMs },
  author={ Tamer Ghattas and Michael Hassid and Roy Schwartz },
  journal={arXiv preprint arXiv:2502.18886},
  year={ 2025 }
}