STATISTICAL-MECHANICS OF ENSEMBLE LEARNING

Within the context of learning a rule from examples, we study the gene ral characteristics of learning, with ensembles. The generalization pe rformance achieved by a simple model ensemble of linear students is ca lculated exactly in the thermodynamic limit of a large number of input components and shows a surprisingly rich behavior. Our main findings are the following. For learning in large ensembles, it is advantageous to use underregularized students, which actually overfit the training data. Globally optimal generalization performance can be obtained by choosing the training set sizes of the students optimally. For smaller ensembles, optimization of the ensemble weights can yield significant improvements in ensemble generalization performance, in particular if the individual students are subject to noise in the training process. Choosing students with a wide range of regularization parameters make s this improvement robust against changes in the unknown level of corr uption of the training data.