Non-identifiability distinguishes Neural Networks among Parametric Models

One of the enduring problems surrounding neural networks is to identify the factors that differentiate them from traditional statistical models. We prove a pair of results which distinguish feedforward neural networks among parametric models at the population level, for regression tasks. Firstly, we prove that for any pair of random variables $(X,Y)$, neural networks always learn a nontrivial relationship between $X$ and $Y$, if one exists. Secondly, we prove that for reasonable smooth parametric models, under local and global identifiability conditions, there exists a nontrivial $(X,Y)$ pair for which the parametric model learns the constant predictor $\mathbb{E}[Y]$. Together, our results suggest that a lack of identifiability distinguishes neural networks among the class of smooth parametric models.

@article{chatterjee2025_2504.18017,
  title={ Non-identifiability distinguishes Neural Networks among Parametric Models },
  author={ Sourav Chatterjee and Timothy Sudijono },
  journal={arXiv preprint arXiv:2504.18017},
  year={ 2025 }
}