ROLE OF BIASES IN ONLINE LEARNING OF 2-LAYER NETWORKS

The influence of biases on the learning dynamics of a two-layer neural network, a normalized soft-committee machine, is studied for on-line gradient descent learning. Within a statistical mechanics framework, n umerical studies show that the inclusion of adjustable biases dramatic ally alters the learning dynamics found previously. The symmetric phas e that has often been predominant in the original model all but disapp ears for a nondegenerate bias task. The extended model furthermore exh ibits a much richer dynamical behavior, e.g., attractive suboptimal sy mmetric phases even for realizable cases and noiseless data.