Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure

It sometimes happens (for instance in case control studies) that a classifi er is trained on a data set that does not reflect the true a priori probabi lities of the target classes on real-world data. This may have a negative e ffect on the classification accuracy obtained on the real-world data set, e specially when the classifier's decisions are based on the a posteriori pro babilities of class membership. Indeed, in this case, the trained classifie r provides estimates of the a posteriori probabilities that are not valid f or this real-world data set (they rely on the a priori probabilities of the training set). Applying the classifier as is (without correcting its outpu ts with respect to these new conditions) on this new data set may thus be s uboptimal. In this note, we present a simple iterative procedure for adjust ing the outputs of the trained classifier with respect to these new a prior i probabilities without having to refit the model, even when these probabil ities are not known in advance. As a by-product, estimates of the new a pri ori probabilities are also obtained. This iterative algorithm is a straight forward instance of the expectation-maximization (EM) algorithm and is show n to maximize the likelihood of the new data. Thereafter, we discuss a stat istical test that can be applied to decide if the a priori class probabilit ies have changed from the training set to the real-world data. The procedur e is illustrated on different classification problems involving a multilaye r neural network, and comparisons with a standard procedure for a priori pr obability estimation are provided. Our original method, based on the EM alg orithm, is shown to be superior to the standard one for a priori probabilit y estimation. Experimental results also indicate that the classifier with a djusted outputs always performs better than the original one in terms of cl assification accuracy, when the a priori probability conditions differ from the training set to the real-world data. The gain in classification accura cy can be significant.