EXPERIMENTAL STUDIES OF MEMORY SURFACES AND LEARNING SURFACES IN RECURRENT NEURAL NETWORKS

The following approach is considered to be important for an insight in to the ''learning'' in the layered network. The adjustable parameter v ector w satisfying the required input-output relation of the network a lso is considered. Then the ''memory surface'' in w space formed by th e set of solutions for the nonlinear equation for w should be investig ated. It is shown also that the idea of the ''memory surface'' can be extended to the learning of the recurrent network (abbreviated as RNN) with a supervisor. This paper shows the following property by an expe riment. When the evaluation function E is defined by the square integr al of the error, then the behavior of RNN agrees essentially with the results obtained for the case of the layered network in the following four points: (1) there exists the memory surface in RNN in the same wa y as in the layered network; (2) the supervised learning of RNN amount s geometrically to the search in the intersection region of multiple ' 'memory surfaces'' for the global or local minimum of E on the learnin g surface formed in w-E space with w as the variable; (3) the feature of the learning surface of RNN is that the space is composed of a long ''valley'' and a flat ''hill'' with smaller gradient; and (4) the lea rning process by the successive search based on the gradient vector of E tends to progress near the memory surface along the valley.