Unsupervised and supervised learning: Mutual information between parameters and observations

We study the mutual information between parameter and data for a family of supervised and unsupervised learning tasks. The parameter is a possibly, bu t not necessarily, high-dimensional vector. We derive exact bounds and asym ptotic behaviors for the mutual information as a function of the data size and of some properties of the probability of the data given the parameter. We compare these exact results with the predictions of replica calculations . We briefly discuss the universal properties of the mutual information as a function of data size. [S1063-651X(99)00403-1].