A New Perspective on Uncertainty Quantification of Deep Ensembles

We present a novel perspective on uncertainty quantification in deep learning, based on the bias-variance decomposition of ensembles. We argue that the existing uncertainty estimation methods are suboptimal in estimating the prediction error of an ensemble, and propose a new two-stage procedure instead, where in the first stage we estimate the unknown function using an ensemble, and in the second stage we fit a separate neural net to the errors of the first model. We argue that this has several advantages, among which better control over regularization such as early stopping, and a more accurate approximation to the aleatoric uncertainty. We extensively test our method on both synthetic as well as real world MRI data, and find that our method significantly improves uncertainty estimates compared to various alternatives such as dropout and deep ensemble based methods.