B. Srinivasan et al., BACK-PROPAGATION THROUGH ADJOINTS FOR THE IDENTIFICATION OF NONLINEARDYNAMIC-SYSTEMS USING RECURRENT NEURAL MODELS, IEEE transactions on neural networks, 5(2), 1994, pp. 213-228
In this paper, back propagation is reinvestigated for an efficient eva
luation of the gradient in arbitrary interconnections of recurrent sub
systems. It is shown that the error has to be back-propagated through
the adjoint model of the system and that the gradient can only be obta
ined after a delay. A faster version, accelerated back propagation, th
at eliminates this delay, is also developed. Various schemes including
the sensitivity method are studied to update the weights of the netwo
rk using these gradients. Motivated by the Lyapunov approach and the a
djoint model, the predictive back propagation and its variant, targete
d back propagation, are proposed. A further refinement, predictive bac
k propagation with filtering is then developed, where the states of th
e model are also updated. The convergence of this scheme is assured. I
t is shown that it is sufficient to back propagate as many time steps
as the order of the system for convergence. As a preamble, convergence
of on-line batch and sample-wise updates in feedforward models is ana
lyzed using the Lyapunov approach.