Reliability and accuracy in reporting CIN in 14 laboratories - Developing new indices of diagnostic variability in an interlaboratory study

OBJECTIVE: To report the results of the second phase of a pilot study of an Italian national external quality evaluation program (EQA) aimed at assess ing whether participation in the first phase had increased the reliability and accuracy of diagnoses and to what extent. STUDY DESIGN: In the second phase, two sets of cervical smears (similar to the ones examined in the first phase) were circulated among the 14 particip ating cytologic laboratories throughout Italy. Responses were recorded on a standardized form. Participants were asked to judge the adequacy of each s mear and to formulate a diagnosis. They weve also asked to recommend manage ment of the patient on the basis of the smear report and to evaluate the de gree of diagnostic difficulty of each slide. The results were discussed in workshops, and it was possible to reach a consensus diagnosis on 37 of 40 s mears. In the statistical analysis, new indices of diagnostic variability w ere developed and calculated; a gross index of agreement, unweighted and we ighted kappas, analysis of exchangeability, sensitivity and specificity wer e also estimated. RESULTS: The results of the second phase are similar to those of the first phase and no substantial improvement in accuracy and little reduction in va riability were observed. The interventions carried out in this study (discu ssion between representatives gf laboratories of diagnostic differences and reassessment of the most controversial slides) were aimed at increasing co nsensus among the participating pathologists but were insufficient to chang e the diagnostic routines in their laboratories. CONCLUSION: It may be advisable to promote two kinds of interlaboratory qua lity programs with two separate but integrated components: (1) a cove compo nent with slides having clear-cut diagnoses, and (2) a continuing education component.