FUNCTIONAL OPTIMIZATION OF ONLINE ALGORITHMS IN MULTILAYER NEURAL NETWORKS

We study the online dynamics of learning in fully connected soft commi ttee machines in the student-teacher scenario. The locally optimal mod ulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximize the. aver age generalization error decay per example. Simulations results for th e resulting algorithm are presented for a few cases. The symmetric pha se plateaux are found to be vastly reduced in comparison to those foun d when online,backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given.