OVERTRAINING, REGULARIZATION AND SEARCHING FOR A MINIMUM, WITH APPLICATION TO NEURAL NETWORKS

In this paper we discuss the role of criterion minimization as a means for parameter estimation. Most traditional methods, such as maximum l ikelihood and prediction error identification are based on these princ iples. However, somewhat surprisingly, it turns out that it is not alw ays 'optimal' to try to find the absolute minimum point of the criteri on. The reason is that 'stopped minimization' (where the iterations ha ve been terminated before the absolute minimum has been reached) has m ore or less identical properties as using regularization (adding a par ametric penalty term). Regularization is known to have beneficial effe cts on the variance of the parameter estimates and it reduces the 'var iance contribution' of the misfit. This also explains the concept of ' overtraining' in neural nets. How does one know when to terminate the iterations then? A useful criterion would be to stop iterations when t he criterion function applied to a validation data set no longer decre ases. However, in this paper, we show that applying this technique ext ensively may lead to the fact that the resulting estimate is an unregu larized estimate for the total data set: estimation + validation data.