Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis

Rationale and Objectives. The purpose of this study was to develop an alter native approach to random-effects, receiver operating characteristic analys is inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berba um, and Metz (DBM) approach that yields additional information on the varia nce structure of the problem. Materials and Methods. Six population experiments were designed to determin e the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogou s bootstrap experiments may be substituted for the population experiments t o estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the cor responding multiple-bootstrap estimates and those obtained with the DBM app roach. Confidence intervals on the difference of ROC parameters for competi ng diagnostic modalities were estimated, and corresponding comparisons were made. Results. For mean values, the agreement of present estimates of variance st ructures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many var iance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM metho d. When reader variability was large, however, mean confidence intervals fr om this study were tighter than those with the DBM method and closer to pop ulation results. Conclusion. The jackknife approach of DBM provides a linear approximation t o receiver-operating-characteristic statistics that are intrinsically nonli near. The multiple-bootstrap technique of this study, however, provides a m ore general, nonparametric, maximum-likelihood approach. It also yields est imates of the variance structure previously unavailable.