A DYNAMICAL SYSTEM PERSPECTIVE OF STRUCTURAL LEARNING WITH FORGETTING

Structural learning with forgetting is an established method of using Laplace regularization to generate skeletal artificial neural networks . In this paper we develop a continuous dynamical system model of regu larization in which the associated regularization parameter is general ized to be a time-varying function. Analytic results are obtained for a Laplace regularizer and a quadratic error surface by solving a diffe rent linear system in each region of the weight space. This model also enables a comparison of Laplace and Gaussian regularization. Both of these regularizers have a greater effect in weight space directions wh ich are less important for minimization of a quadratic error function. However, for the Gaussian regularizer, the regularization parameter m odifies the associated linear system eigenvalues, in contrast to its f unction as a control input in the Laplace case. This difference provid es additional evidence for the superiority of the Laplace over the Gau ssian regularizer.