STATISTICAL PHYSICS, MIXTURES OF DISTRIBUTIONS, AND THE EM ALGORITHM

We show that there are strong relationships between approaches to optm ization and learning based on statistical physics or mixtures of exper ts. In particular, the EM algorithm can be interpreted as converging e ither to a local maximum of the mixtures model or to a saddle point so lution to the statistical physics system. An advantage of the statisti cal physics approach is that it naturally gives rise to a heuristic co ntinuation method, deterministic annealing, for finding good solutions .