Receiver operating characteristic analysis for intelligent medical systems- A new approach for finding confidence intervals

Intelligent systems are increasingly being deployed in medicine and healthc are, but there is a need for a robust and objective methodology for evaluat ing such systems. Potentially, receiver operating,characteristic (ROC) anal ysis could form a basis for the objective evaluation of intelligent medical systems. However, it has several weaknesses when applied to the types of d ata used to evaluate intelligent medical systems. First, small data sets ar e often used, which are unsatisfactory with existing methods. Second, many existing ROC methods use parametric assumptions which may not always be val id for the test cases selected. Third, system evaluations are often more co ncerned with particular, clinically meaningful, points on the curve, rather than on global indexes such as the more commonly used area under the curve . A novel, robust and accurate method is proposed, derived from first princip les, which calculates the probability density function (pdf) for each point on a ROC curve for any given sample size. Confidence intervals are produce d as contours on the pdf. The theoretical work has been validated by Monte Carlo simulations. It has also been applied to two real-world examples of R OC analysis, taken from the literature (classification of mammograms and di fferential diagnosis of pancreatic diseases), to investigate the confidence surfaces produced for real cases, and to illustrate how analysis of system performance can be enhanced. We illustrate the impact of sample size on sy stem performance from analysis of ROC pdf's and 95% confidence boundaries. This work establishes an important new method for generating pdf's, and pro vides an accurate and robust method of producing confidence intervals for R OC curves for the small sample sizes typical of intelligent medical systems . It is conjectured that, potentially, the method could be extended to dete rmine risks associated with the deployment of intelligent medical systems i n clinical practice.