How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record

PURPOSE: Despite widespread reliance on chart abstraction for quality measu rement, concerns persist about its reliability and validity. We prospective ly evaluated the validity of chart abstraction by directly comparing it wit h the gold standard of reports by standardized patients. SUBJECTS AND METHODS: Twenty randomly selected general internal medicine re sidents and attending faculty physicians at the primary care clinics of two Veterans Affairs Medical Centers blindly evaluated and treated actor-patie nts (standardized patients) who had one of four common diseases: diabetes, chronic obstructive pulmonary disease, coronary artery disease, or low back pain. Charts from the visits were abstracted using explicit quality criter ia; standardized patients completed a checklist containing the same criteri a. For each physician, quality was measured for two different cases of the four conditions (a total of 160 physician-patient encounters). We compared chart abstraction with standardized;patient reports for four aspects of the encounter: taking the history, examining the patient, making the diagnosis , and prescribing appropriate treat-ment. The sensitivity and specificity o f chart abstraction were calculated. RESULTS: The mean (+/- SD) chart abstraction score was 54% +/- 9%, substant ially less than the mean score on the standardized-patient checklist of 68% +/- 9% (P <0.001). This finding was similar for all four conditions and at both sites. "False positives"- chart-recorded necessary care actions not r eported by the standardized patients-resulted in a specificity of only 81%. The overall sensitivity of chart abstraction for necessary care was only 7 0%. CONCLUSIONS: Chart abstraction underestimates the quality of care for commo n outpatient general medical conditions when compared with standardized-pat ient reports. The medical record is neither sensitive nor specific. Quality measurements derived from chart abstraction may have important shortcoming s, particularly as the basis for drawing policy conclusions or making manag ement decisions. (C) 2000 by Excerpta Medica, Inc.