GLOBALLY OPTIMAL ONLINE LEARNING RULES FOR MULTILAYER NEURAL NETWORKS

We present a method for determining the globally optimal on-line learn ing rule for a soft committee machine under a statistical mechanics fr amework. This rule maximizes the total reduction in generalization err or over the whole learning process. A simple example demonstrates that the locally optimal rule, which maximizes the rate of decrease in gen eralization error, may perform poorly in comparison.