Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiology

Rationale and Objectives. The purpose of this study was to test whether the satisfaction of search (SOS) effect in chest radiology could be demonstrat ed with proper receiver operating characteristic (ROC) curves and with join t detection and localization ROC curves. Material and Methods. Data from an earlier ROC study of SOS in chest radiol ogy were analyzed with three proper ROC models and one ROC model for joint detection and localization. Fits of the models were compared on the basis o f likelihood-ratio chi-squared statistics (G(2)). To examine further the va lidity of the SOS effect in chest radiology, the authors also replicated th e earlier study with a new sample of readers, analyzing the new data with t he same methods. Results. The proper contaminated binormal model fit the data better than th e other two proper ROC models. Contaminated binormal analysis of the earlie r and the replication experiment demonstrated an SOS effect: a reduction in area under the ROC curve for detection of the native abnormalities with th e addition of nodules. Similarly, joint ROC analysis producing curves that appropriately cross the chance line gave similar results. Conclusion. Preventing inappropriate chance line crossing reduces measureme nt error and provides more powerful statistical tests. Results of both expe riments showed that the SOS effect in chest radiology can be demonstrated w ith ROC methods that avoid inappropriate crossing of the chance line.