THE DYNAMIC UNIVERSALITY OF SIGMOIDAL NEURAL NETWORKS

We investigate the computational power of recurrent neural networks th at apply the sigmoid activation function sigma(x)=[2/(1+e(-x))]-1. The se networks are extensively used in automatic learning of non-linear d ynamical behavior. We show that in the noiseless model, there exists a universal architecture that can be used to compute any recursive (Tur ing) function. This is the first result of its kind for the sigmoid ac tivation function; previous techniques only applied to linearized and truncated version of this function. The significance of our result, be sides the proving technique itself, lies in the popularity of the sigm oidal function both in engineering applications of artificial neural n etworks and in biological modelling. Our techniques can be applied to a much more general class of ''sigmoidal-like'' activation functions, suggesting that Turing universality is a relatively common properly of recurrent neural network models. (C) 1996 Academic Press, Inc.