Matrix momentum for practical natural gradient learning

An on-line learning rule, based on the introduction of a matrix momentum te rm, is presented, aimed at alleviating the computational costs of standard natural gradient learning. The new rule, natural gradient matrix momentum, is analysed in the case of two-layer feed-forward neural network learning v ia methods of statistical physics. It appears to provide a practical algori thm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity.