Unsupervised and supervised learning: Mutual information between parameters and observations

Citation
D. Herschkowitz et Jp. Nadal, Unsupervised and supervised learning: Mutual information between parameters and observations, PHYS REV E, 59(3), 1999, pp. 3344-3360
Citations number
21
Categorie Soggetti
Physics
Journal title
PHYSICAL REVIEW E
ISSN journal
1063651X → ACNP
Volume
59
Issue
3
Year of publication
1999
Part
B
Pages
3344 - 3360
Database
ISI
SICI code
1063-651X(199903)59:3<3344:UASLMI>2.0.ZU;2-F
Abstract
We study the mutual information between parameter and data for a family of supervised and unsupervised learning tasks. The parameter is a possibly, bu t not necessarily, high-dimensional vector. We derive exact bounds and asym ptotic behaviors for the mutual information as a function of the data size and of some properties of the probability of the data given the parameter. We compare these exact results with the predictions of replica calculations . We briefly discuss the universal properties of the mutual information as a function of data size. [S1063-651X(99)00403-1].