On chaos and neural networks: The backpropagation paradigm

In training feed-forward neural networks using the backpropagation algorith m, a sensitivity to the values of the parameters of the algorithm has been observed. In particular, it has been observed that this sensitivity with re spect to the values of the parameters, such as the learning rate, plays an important role in the final outcome. In this tutorial paper, we will look a t neural networks from a dynamical systems point of view and examine its pr operties. To this purpose, we collect results regarding chaos theory as wel l as the backpropagation algorithm and establish a relationship between the m. We study in detail as an example the learning of the exclusive OR, an el ementary Boolean function. The following conclusions hold for our XOR neura l network: no chaos appears for learning rates lower than 5, when chaos occ urs, it disappears as learning progresses. For non-chaotic learning rates, the network learns faster than for other learning rates for which chaos occ urs.