ONLINE LEARNING IN SOFT COMMITTEE MACHINES

The problem of on-line learning in two-layer neural networks is studie d within the framework of statistical mechanics. A fully connected com mittee machine with K hidden units is trained by gradient descent to p erform a task defined by a teacher committee machine with M hidden uni ts acting on randomly drawn inputs: The approach, based on a direct av eraging over the activation of the hidden units, results in a set of f irst-order differential equations that describes the dynamical evoluti on of the overlaps among the various hidden units and allows for a com putation of the generalization error. The equations of motion are obta ined analytically for general K and M and provide a powerful tool used here to study a variety of realizable, overrealizable, and unrealizab le learning scenarios and to analyze the role of the learning rate in controlling the evolution and convergence of the learning process.