Choose Your Model Size: Any Compression by a Single Gradient Descent

The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off from a single stochastic gradient descent run. To ensure parameter efficiency, we use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty. The resulting pruning order gives rise to a global parameter ranking that allows us to materialize models of any target size. Importantly, the compressed models exhibit strong predictive downstream performance without the need for costly fine-tuning. We evaluate ACIP on a large selection of open-weight LLMs and tasks, and demonstrate state-of-the-art results compared to existing factorisation-based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.

@article{genzel2025_2502.01717,
  title={ Choose Your Model Size: Any Compression by a Single Gradient Descent },
  author={ Martin Genzel and Patrick Putzky and Pengfei Zhao and Sebastian Schulze and Mattes Mollenhauer and Robert Seidel and Stefan Dietzel and Thomas Wollmann },
  journal={arXiv preprint arXiv:2502.01717},
  year={ 2025 }
}