WHY SOME FEEDFORWARD NETWORKS CANNOT LEARN SOME POLYNOMIALS

It seems natural to test feedforward networks on deterministic functio ns. Yet, some simple functions, notably polynomials, present some diff icult problems for approximation by feedforward networks. The estimate d parameters become unbounded and fail to follow any unique pattern. F urthermore, as the fit to the specified functions becomes closer, nume rical problems may develop in the algorithm. This paper explains why t hese problems occur for polynomials of order less than or equal to the number of hidden units of a feedforward network. We show that other e xamples occur for functions mathematically related to the network's sq uashing function. These difficulties do not indicate problems with the training algorithm, but occur as an inherent consequence of the role of the connection weights in feedforward networks.