COMMENTS ON AN ACCELERATED LEARNING ALGORITHM FOR MULTILAYER PERCEPTRONS - OPTIMIZATION LAYER-BY-LAYER

This letter analyzes the performance of the neural network training me thod known as optimization layer by layer.(1) We show, from theoretica l considerations, that the amount of work required with OLL-Learning s cales as the third power of the network size, compared with the square of the network size for commonly used conjugate gradient training alg orithms. This theoretical estimate is confirmed through a practical ex ample. Thus, although OLL is shown to function very well for small neu ral networks (less than about 500 weights per layer), it is slower tha n CG for large neural networks. Second, we show that OLL does not alwa ys improve on the accuracy that can be obtained with CG. It seems that the final accuracy that can be obtained depends strongly on the initi al network weights.