Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies

Rationale and Objectives. The authors attempted to assess experimentally th e magnitude of reader variability and the correlations and interactions amo ng cases, readers, and modalities during observer performance studies and t heir possible effects on study design and sample size. Materials and Methods, Published data from 32 selected receiver operating c haracteristic (ROC) studies were reviewed to compare the magnitude of the v ariance component from readers with the variance component from modality. E stimates of correlation and interactions among cases, readers, and modaliti es were also computed directly from ROC data ascertained during two large s tudies performed in our laboratory. Each of these two studies included 529 cases and six readers, but one study used eight modalities and the other ni ne. Results. Published results indicate that reader variability is task depende nt and larger (P < .05) than modality variability in detection of interstit ial disease. Measured correlations between modalities for the same reader w ere task dependent and ranged from 0.35 to 0.59. Modality-by-reader and mod ality-by-case interactions often are not important factors. The random erro r term was greater than the modality-by-reader interaction in 11 of 20 comp arisons and greater than the modality-by-case interaction in eight of 20 co mparisons. Conclusion. Use of the same cases interpreted with different modes is justi fiable in many situations because of the high variability From readers. Thi s comprehensive review of existing ROC studies resulted in parameter assess ments that can be used to better estimate sample-size requirements in multi reader ROC studies.