Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution

Catastrophic forgetting is a major problem for sequential learning in neura l networks. One very general solution to this problem, known as 'pseudorehe arsal', works well in practice for nonlinear networks but has not been anal ysed before. This paper formalizes pseudorehearsal in linear networks. We s how that the method can fail in low dimensions but is guaranteed to Succeed in high dimensions under fairly general conditions. In this case an optima l version of the method is equivalent to a simple modification of the 'delt a rule'.