R. Parisi et al., A GENERALIZED LEARNING-PARADIGM EXPLOITING THE STRUCTURE OF FEEDFORWARD NEURAL NETWORKS, IEEE transactions on neural networks, 7(6), 1996, pp. 1450-1460
In this paper a general class of fast learning algorithms for feedforw
ard neural networks is introduced and described, The approach exploits
the separability of each layer into linear and nonlinear blocks and c
onsists of two steps, The first step is the descent of the error funct
ional in the space of the outputs of the linear blocks (descent in the
neuron space), which can be performed using any preferred optimizatio
n strategy, In the second step, each linear block is optimized separat
ely by using a least squares (LS) criterion. To demonstrate the effect
iveness of the new approach, a detailed treatment of a gradient descen
t in the neuron space is conducted. The main properties of this approa
ch are the higher speed of convergence with respect to methods that em
ploy an ordinary gradient descent in the weight space backpropagation
(BP), better numerical conditioning, and lower computational cost comp
ared to techniques based on the Hessian matrix, The numerical stabilit
y is assured by the use of robust LS linear system solvers, operating
directly on the input data of each layer. Experimental results obtaine
d in three problems are described, which confirm the effectiveness of
the new method.