Mixture of experts architectures for neural networks as a special case of conditional expectation formula.

Recently a new interesting architecture of neural networks called "mixture of experts" has been proposed as a tool of real multivariate approximation or prediction. We show that the underlying problem is closely related to ap proximating the joint probability density of involved variables by finite m ixture. Particularly, assuming normal mixtures, we can explicitly write the conditional expectation formula which can be interpreted as a mixture-of-e xperts network. In this way the related optimization problem can be reduced to standard estimation of normal mixtures by means of EM algorithm. The re sulting prediction is optimal in the sense of minimum dispersion if the ass umed mixture model is true. It is shown that some of the recently published results can be obtained by specifying the normal components of mixtures in a special form.