We systematically explore a natural analogy between Bayesian statistics and thermal physics in which sample size corresponds to inverse temperature. We discover that some canonical thermodynamic quantities already correspond to well-established statistical quantities. Motivated by physical insight into thermal physics, we define two novel statistical quantities: a learning capacity and Gibbs entropy. The definition of the learning capacity leads to a critical insight: The well-known mechanism of failure of the equipartition theorem in statistical mechanics is the mechanism for anomalously-predictive or sloppy models in statistics. This correspondence between the learning and heat capacities provides new insight into the mechanism of machine learning. The correspondence also suggests a solution to a long-standing difficulty in Bayesian statistics: the definition of an objective prior. We propose that the Gibbs entropy provides a natural generalization of the principle of indifference that defines objectivity. This approach unifies the disparate Bayesian, frequentist and information-based paradigms of statistics by achieving coherent inference between these competing formulations.
View on arXiv