Variation between experienced observers in the interpretation of accident and emergency radiographs

Skill mix and role extension initiatives have highlighted the difficulty of establishing quality standards for the accuracy of plain film reporting. A n acceptable performance might be one which is indistinguishable from that of a group of experienced consultant radiologists. In order to assess the f easibility of setting such a standard, the variation between experienced ob servers must first be established. This study examines the variation found between three observers with the three major types of plain film examinatio n. 402 plain film examinations (205 skeletal, 100 chest and 97 abdominal) p erformed on accident and emergency patients were reported retrospectively a nd independently by three experienced radiologists. The clinical data suppl ied on the request cards were available to the readers. Each examination wa s categorized by each reader as being normal, as showing significant abnorm ality relevant to the current clinical problem, or as showing insignificant or irrelevant abnormality. Concordance between all three readers was found in 51%, 61% and 74% of abdominal, chest and skeletal radiographs, respecti vely. Weighted kappa values confirmed that the level of agreement between p airs of observers was higher with skeletal radiographs (K-w = 0.76-0.77) th an with chest (K-w,= 0.63-0.68), or abdominal (K-w = 0.50-0.78) examination s. However, the frequency of major disagreements (at least one reader repor ting "normal" and one reporting "relevant abnormality") was similar for abd ominal (11%), chest (12%) and skeletal (10%) radiographs. When the reports were reclassified into only two groups-either significantly abnormal or not -pairs of observers disagreed on 9-10% of skeletal, 11-19% of chest and 8-1 8% of abdominal cases. The average incidence of errors per observer was est imated to be in the range 3-6%. The magnitude of interobserver variation in plain film reporting is considerable, and must be taken into account when designing assessment techniques and setting quality standards for this acti vity.