LEARNING WITH REGULARIZERS IN MULTILAYER NEURAL NETWORKS

We study the effect of regularization in an on-line gradient-descent l earning scenario for a general two-layer student network with an arbit rary number of hidden units. Training examples are randomly drawn inpu t vectors labeled by a two-layer teacher network with an arbitrary num ber of hidden units that may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical ev olution of the order parameters and generalization error in various ph ases of the learning process, in both noiseless and noisy scenarios.