Comparing classifiers when the misallocation costs are uncertain

Receiver Operating Characteristic (ROC) curves are popular ways of summaris ing the performance of two class classification rules. In fact, however, th ey are extremely inconvenient. If the relative severity of the two differen t kinds of misclassification is known, then an awkward projection operation is required to deduce the overall loss. At the other extreme, when the rel ative severity is unknown, the area. under an ROC curve is often used as an index of performance. However, this essentially assumes that nothing whats oever is known about the relative severity - a situation which is very rare in real problems. We present an alternative plot which is more revealing t han an ROC plot and we describe a comparative index which allows one to tak e advantage of anything that may be known about the relative severity of th e two kinds of misclassification. (C) 1999 Pattern Recognition Society. Pub lished by Elsevier Science Ltd. All rights reserved.