A DYNAMIC NEURAL-NETWORK ARCHITECTURE BY SEQUENTIAL PARTITIONING OF THE INPUT SPACE

We present a sequential approach to training multilayer perceptrons fo r pattern classification applications. The network is presented with e ach item of data only once and its architecture is dynamically adjuste d during training. At the arrival of each example, a decision whether to increase the complexity of the network, or simply train the existin g nodes is made based on three heuristic criteria. These criteria meas ure the position of the new item of data in the input space with respe ct to the information currently stored in the network. During the trai ning process, each layer is assumed to be an independent entity with i ts particular input space. By adding nodes to each layer, the algorith m is effectively adding a hyperplane to the input space, hence adding a partition in the input space for that layer. When existing nodes are sufficient to accommodate the incoming input, the corresponding hidde n nodes will be trained accordingly. Each hidden unit in the network i s trained in closed form by means of a recursive least-squares (RLS) a lgorithm. A local covariance matrix of the data is maintained at each node and the closed form solution is recursively updated. The three cr iteria are computed from these covariance matrices to keep low computa tional cost. The performance of the algorithm is illustrated on two pr oblems. The first problem is the two-dimensional Peterson and Barney v owel data. The second problem is a 33-dimensional data derived from a vision system for classifying wheat grains. The sequential nature of t he algorithm has an efficient hardware implementation in the form of s ystolic arrays, and the incremental training idea has better biologica l plausibility compared with iterative methods.