A recurrent neural network that learns to count

Parallel distributed processing (PDP) architectures demonstrate a potential ly radical alternative to the traditional theories of language processing t hat are based on serial computational models. However, learning complex str uctural relationships in temporal data presents a serious challenge to PDP systems. For example, automata theory dictates that processing strings from a context-free language (CFL) requires a stack or counter memory device. W hile some PDP models have been hand-crafted to emulate such a device, it is not clear how a neural network might develop such a device when learning a CFL. This research employs standard backpropagation training techniques fo r a recurrent neural network (RNN) in the task of learning to predict the n ext character in a simple deterministic CFL (DCFL). We show that an RNN can learn to recognize the structure of a simple DCFL. We use dynamical system s theory to identify how network states reflect that structure by building counters in phase space. The work is an empirical investigation which is co mplementary to theoretical analyses of network capabilities, yet original i n its specific configuration of dynamics involved. The application of dynam ical systems theory helps us relate the simulation results to theoretical r esults, and the learning task enables us to highlight some issues for under standing dynamical systems that process language with counters.