GLOBALLY OPTIMAL LEARNING RATES FOR MULTILAYER NEURAL NETWORKS

A method for calculating the globally optimal learning rate in on-line gradient-descent training of multilayer neural networks is presented. The method is based on a variational approach which maximizes the dec rease in generalization error over a given time frame. We demonstrate the method by computing optimal learning rates in typical learning sce narios. The method can also be employed when different learning rates are allowed for different parameter vectors as well as to determine th e relevance of related training algorithms based on modifications to t he basic gradient-descent rule.