Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis

Rationale and Objectives. Solutions have previously been presented to the p roblem of estimating the components of variance in the general linear model used for multivariate receiver operating characteristic (ROC) analysis. Th e case where the variance components do not change across the modalities un der comparison was first treated, followed by the case where they are permi tted to change. No analysis of uncertainties in these estimates has been pr esented previously. Materials and Methods, For the case where the variance components do not ch ange across modalities, the "jackknife-after-bootstrap" resampling procedur e can be used together with conventional linear propagation of variance to solve for the uncertainties in estimates of the components. For the case wh ere the components are permitted to change across modalities, a slight elab oration of this procedure is presented. Results. The approach was validated by Monte Carlo simulations, where uncer tainties in estimates of the variance components calculated by the jackknif e-after-bootstrap procedure were found to converge in the mean to the Monte Carlo results over many independent trials. The method is exemplified with data from a study of readers-with and without the aid of a computer-assist modality-given the task of discriminating benign from malignant masses in mammography. Conclusion. The present approach is relevant to a broad class of problems w here estimates of multiple contributions to the variance observed in ROC as sessment of diagnostic modalities are desired, in particular, for the asses sment of multiple-reader studies of computer-aided diagnosis in radiology w here the variance components may change across reading modalities (eg, unai ded vs computer-aided reading).