Dd. Dorfman et al., MONTE-CARLO VALIDATION OF A MULTIREADER METHOD FOR RECEIVER OPERATINGCHARACTERISTIC DISCRETE RATING DATA - FACTORIAL EXPERIMENTAL-DESIGN, Academic radiology, 5(9), 1998, pp. 591-602
Rational and Objectives. The authors conducted a series of null-case M
onte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) meth
od for comparing modalities with multireader receiver operating charac
teristic (ROC) discrete rating data. Materials and Methods. Monte Carl
o simulations were performed by using discrete ratings on fully crosse
d factorial designs with two modalities and three, five, and 10 hypoth
etical readers. The null hypothesis was true for all simulations. The
population ROC areas, latent variable structures, case sample sizes, a
nd normal/abnormal case sample ratios used in another study were used
in these simulations. Results, For equal allocation ratios and small (
A(z) = 0.702) and moderate (A(z) =0.855) ROC areas, the empirical type
I error rate closely matched the nominal a level. For very large ROC
areas (A(z) = 0.961), however, the empirical type I error rate was som
ewhat smaller than the nominal alpha level. This conservatism increase
d with decreasing case sample size and asymmetric normal/abnormal case
allocation ratio, The empirical type I error rate was sometimes sligh
tly larger than the nominal alpha level with many cases and few reader
s, where there was large residual, relatively small treatment-by-case
interaction and relatively large treatment-by-reader interaction, Conc
lusion. The results suggest that the DBM method provides trustworthy a
lpha levels with discrete ratings when the ROC area is not too large a
nd case and reader sample sizes are not too small. In other situations
, the test tends to be somewhat conservative or slightly liberal.