Paths and Ambient Spaces in Neural Loss Landscapes

Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.

@article{dold2025_2503.03382,
  title={ Paths and Ambient Spaces in Neural Loss Landscapes },
  author={ Daniel Dold and Julius Kobialka and Nicolai Palm and Emanuel Sommer and David Rügamer and Oliver Dürr },
  journal={arXiv preprint arXiv:2503.03382},
  year={ 2025 }
}