A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation

We propose a principled formulation of the ROC curve that is constrained in a realistic way by the mechanism of probability summation. The constrained and conventional ROC formulations were fitted to 150 separate sets of rati ng data taken from previous observer studies of 250 or 529 chest radio grap hs. A total of 20 different readers had used either discrete or continuous rating scales to evaluate those chest cases for likelihood of separate spec ified abnormalities: interstitial disease, pulmonary nodule, pneumothorax, alveolar infiltrate, or rib fracture. Both ROC formulations were fitted sep arately to every set of rating data using maximum-likelihood statistical pr ocedures that specified each ROC curve by normally distributed latent varia bles with two scaling parameters, and estimated the area below the ROC curv e (A(Z)) with its standard error. The conventional and constrained binormal formulations usually fitted ROC curves that were nearly indistinguishable in form and in A(Z). But when fitted to asymmetric rating data that contain ed few false-positive cases, the conventional ROC curves often rose steeply , then flattened and extrapolated into an unrealistic upward "hook" at the higher false-positive rates. For those sets of rating data, the constrained ROC curves (without hooks) estimated larger values for A(Z) with smaller s tandard errors. The constrained ROC formulation describes observers' rating s of cases at least as well as the conventional ROC, and always guarantees a realistic fitted curve for observer performance. Its estimated parameters are easy to interpret, and may also be used to predict observer accuracy i n localizing the image abnormalities. (C) 2001 American Association of Phys icists in Medicine.