MONTE-CARLO VALIDATION OF A MULTIREADER METHOD FOR RECEIVER OPERATINGCHARACTERISTIC DISCRETE RATING DATA - FACTORIAL EXPERIMENTAL-DESIGN

Rational and Objectives. The authors conducted a series of null-case M onte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) meth od for comparing modalities with multireader receiver operating charac teristic (ROC) discrete rating data. Materials and Methods. Monte Carl o simulations were performed by using discrete ratings on fully crosse d factorial designs with two modalities and three, five, and 10 hypoth etical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, a nd normal/abnormal case sample ratios used in another study were used in these simulations. Results, For equal allocation ratios and small ( A(z) = 0.702) and moderate (A(z) =0.855) ROC areas, the empirical type I error rate closely matched the nominal a level. For very large ROC areas (A(z) = 0.961), however, the empirical type I error rate was som ewhat smaller than the nominal alpha level. This conservatism increase d with decreasing case sample size and asymmetric normal/abnormal case allocation ratio, The empirical type I error rate was sometimes sligh tly larger than the nominal alpha level with many cases and few reader s, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction, Conc lusion. The results suggest that the DBM method provides trustworthy a lpha levels with discrete ratings when the ROC area is not too large a nd case and reader sample sizes are not too small. In other situations , the test tends to be somewhat conservative or slightly liberal.