OVERTRAINING IN NEURAL NETWORKS THAT INTERPRET CLINICAL-DATA

Backpropagation neural networks are a computer-based pattern-recogniti on method that has been applied to the interpretation of clinical data . Unlike rule-based pattern recognition, backpropagation networks lear n by being repetitively trained with examples of the patterns to be di fferentiated. We describe and analyze the phenomenon Of overtraining i n backpropagation networks. Overtraining refers to the reduction in ge neralization ability that can occur as networks are trained. The clini cal application we used was the differentiation of giant cell arteriti s (GCA) from other forms of vasculitis (OTH) based on results for 807 patients (593 OTH, 214 GCA) and eight clinical predictor variables. Th e 807 cases were randomly assigned to either a training set with 404 c ases or to a cross-validation set with the remaining 403 cases. The cr oss-validation set was used to monitor generalization during training. Results were obtained for eight networks, each derived from a differe nt random assignment of the 807 cases. Training error monotonically de creased during training. In contrast, the cross-validation error usual ly reached a minimum early in training while the training error was st ill decreasing. Training beyond the minimum cross-validation error was associated with an increased cross-validation error. The shape of the cross-validation error curve and the point during training correspond ing to the minimum cross-validation error varied with the composition of the data sets and the training conditions. The study indicates that training error is not a reliable indicator of a network's ability to generalize. To find the point during training when a network generaliz es best, one must monitor cross-validation error separately.