DORFMAN-BERBAUM-METZ METHOD FOR STATISTICAL-ANALYSIS OF MULTIREADER, MULTIMODALITY RECEIVER OPERATING CHARACTERISTIC DATA - VALIDATION WITHCOMPUTER-SIMULATION

Rationale and Objectives. The authors examined the relationship betwee n the critical P value (alpha) and the empirical type I error rate whe n using the Dorfman-Berbaum-Metz (DBM) method for analysis of variance in multireader, multimodality receiver operating characteristic (ROC) data. Methods. The authors developed a linear mixed-effect model to g enerate continuous, normally distributed random decision variables con taining multiple sources (components) of variation. A range of magnitu des for these variance components was used to simulate experiments in which multiple readers (three or five) read images obtained with two m odalities from the same set of cases with no re-reading. Three binorma l population ROC curves, with areas of 0.962, 0.855, and 0.702, were i ncluded. Case-sample sizes ranged from 50 to 400, and either 50% or 10 % of cases were actually positive. For each experiment, 2,000 data set s were analyzed by the computer program, and the proportion of 2,000 m odality differences that was found to be statistically significant at an alpha level of .05 was tabulated. Results. The test for modality di fference performed well for the low and intermediate ROC curves, even with small case samples. For the high ROC curve, the small-sample resu lts were conservative. No relationship between observed type I error r ate and the magnitude of data correlation was evident. Conclusion. For typical ROC curves, the DBM method is robust in testing for modality effects in the null case, given a sufficient sample size. Instructions for obtaining a free copy of the software are given.