IMPROVED BINARY CLASSIFICATION PERFORMANCE USING AN INFORMATION-THEORETIC CRITERION

Feedforward neural networks trained to solve classification problems d efine an approximation of the conditional probabilities P(C-i\x) if th e output units correspond to categories C-i. The present paper shows t hat if a least mean squared error cost function is minimised during tr aining phase, the resulting approximation of the P(C-i\x)s is poor in the ranges of the input variable x where the conditional probabilities take on very low values. The use of the Kullback-Leibler distance mea sure is proposed to overcome this limitation; a cost function derived from this information theoretic measure is defined and a computational ly light training procedure is derived in the case of binary classific ation problems. The effectiveness of the proposed procedure is verifie d by means of comparative experiments.