The "differential diagnosis" for multiple diseases: Comparison with the binary-truth state experiment in two empirical studies

Rationale and Objectives. In practice readers must often choose between mul tiple diagnoses. For assessing reader accuracy in these settings, Obuchowsk i et al have proposed the "differential diagnosis" method, which derives al l pairwise estimates of accuracy for the various diagnoses, along with summ ary measures of accuracy. The current study assessed the correspondence bet ween the differential diagnosis method and conventional binary-truth state experiments. Materials and Methods. Two empirical studies were conducted at two institut ions with different readers and diagnostic tests. Readers used the differen tial diagnosis format to interpret a set of cases. In subsequent readings t hey interpreted the cases in binary-truth state experiments. Spearman rank correlation coefficients and the percentages of agreement in scores were co mputed, and the areas under the receiver operating characteristic curves we re estimated and compared. Results. The between-format Spearman rank correlation coefficients were 0.6 97-0.718 and 0.750-0.780 for the two studies; the between-reader correlatio ns were 0.417 and 0.792, respectively. The percentages of agreement between formats for the two studies were 50.0%-51.7% and 72.9%-78.8%; the percenta ges of agreement between readers were 45.0% and 80%, respectively. In the f irst study there were several significant differences in the areas under re ceiver operating characteristic curves; in the second study these differenc es were small. Conclusion. The differences observed between the two formats can be attribu ted to within-reader variability and inherent differences in the questions posed to readers in the multiple-diagnoses versus binary-truth state readin g sessions. The differential diagnosis format is useful for estimating accu racy when there are multiple possible diagnoses.