STATISTICAL-INFERENCE, OCCAMS RAZOR, AND STATISTICAL-MECHANICS ON THESPACE OF PROBABILITY-DISTRIBUTIONS

The task of parametric model selection is cast in terms of a statistic al mechanics on the space of probability distributions. Using the tech niques of low-temperature expansions, I arrive at a systematic series for the Bayesian posterior probability of a model family that signific antly extends known results in the literature. In particular, I arrive at a precise understanding of how Occam's razor, the principle that s impler models should be preferred until the data justify more complex models, is automatically embodied by probability theory. These results require a measure on the space of model parameters and I derive and d iscuss an interpretation of Jeffreys' prior distribution as a uniform prior over the distributions indexed by a family. Finally, I derive a theoretical index of the complexity of a parametric family relative to some true distribution that I call the razor of the model. The form o f the razor immediately suggests several interesting questions in the theory of learning that can be studied using the techniques of statist ical mechanics.