Empirical comparison of various methods for training feed-forward neural networks for salinity forecasting

Feed-forward artificial neural networks (ANNs) are being used increasingly to model water resources variables. In this technical note, six methods for optimizing the connection weights of feedforward ANNs are investigated in terms of generalization ability, parsimony, and training speed. These inclu de the generalized delta (GD) rule, the normalized cumulative delta (NCD) r ule, the delta-bar-delta (DBD) algorithm, the extended-delta-bar-delta (EDB D) algorithm, the QuickProp (QP) algorithm, and the MaxProp (MP) algorithm. Each of these algorithms is applied to a particular case study, the foreca sting of salinity in the River Murray at Murray Bridge, South Australia. Th irty models are developed for each algorithm, starting from different posit ions in weight space. The results obtained indicate that the generalization ability of the first-order methods investigated (i.e., GD, NCD, DBD, and E DBD) is better than that of the second-order algorithms (i.e., QP and MP). When the prediction errors are averaged over the 30 trials carried out, the performance of the first-order methods in which the size of the steps take n in weight space is automatically adjusted in response to changes in the e rror surface (i.e., DBD and EDBD) is better than that obtained when predete rmined step sizes are used (i.e., GD and NCD). However, the reverse applies when the best forecasts of the 30 trials are considered. The results obtai ned indicate that the EDBD algorithm is the most parsimonious and the MP al gorithm is the least parsimonious. It was found that any impact different l earning rules have on training speed is masked by the effect of epoch size and the number of hidden nodes required for optimal model performance.