The case for comprehensive quality indicator reliability assessment

To demonstrate the importance of evaluating overall quality indicator relia bility, in addition to component or variable level reliability, a compariso n of interrater agreement on four chart-abstracted pneumonia-related proces ses of care was conducted. The hospital medical records of 356 Medicare pat ients' recent discharges for pneumonia were independently abstracted by dif ferent abstractors. Kappa, prevalence and bias-adjusted kappa. P-pos, P-neg , and the Bias Index were used to assess reliability of composite quality i ndicators and their components. The adjusted kappas for the data elements u sed to determine eligibility to receive as well as to derive the pneumonia- related processes of care ranged from 0.68 to 1.0. The adjusted kappa assoc iated with overall eligibility to receive the pneumonia-related processes o f care was 0.63. The kappa statistics for determining if processes of care were provided ranged from 0.56 to 0.83 and increased to 0.65 and 0.85 upon adjustment for the prevalence effect. Kappas for the composite quality indi cators were lower, but improved with adjustment for the prevalence effect. The composite quality indicator with the highest adjusted kappa value was o xygenation assessment (0.93); the composite quality indicator with the lowe st adjusted kappa value was antibiotic administration within 8 hours of hos pital arrival (0.74). This study establishes the reliability of pneumonia i ndicators and underscores the need for reliability assessment at the qualit y indicator level, as well as at the component level. (C) 2001 Elsevier Sci ence Inc. All rights reserved.