Comparison of vignettes, standardized patients, and chart abstraction - A prospective validation study of 3 methods for measuring quality

Context Better health care quality is a universal goal, yet measuring quali ty has proven to be difficult and problematic. A central problem has been i solating physician practices from other effects of the health care system, Objective To validate clinical vignettes as a method for measuring the comp etence of physicians and the quality of their actual practice. Design Prospective trial conducted in 1997 comparing 3 methods for measurin g the quality of care for 4 common outpatient conditions: (1) structured re ports by standardized patients (SPs), trained actors who presented unannoun ced to physicians' clinics (the gold standard); (2) abstraction of medical records for those same visits; and (3) physicians' responses to clinical vi gnettes that exactly corresponded to the SPs' presentations. Setting Outpatient primary care clinics at 2 Veterans Affairs medical cente rs. Participants Ninety-eight (97%) of 101 general internal medicine staff phys icians, faculty, and second- and third-year residents consented to be rando mized for the study. From this group, 10 physicians at each site were rando mly selected for inclusion. Main Outcome Measures A total of 160 quality scores (8 cases x 20 physician s) were generated for each method using identical explicit criteria based o n national guidelines and local expert panels, Scores were defined as the p ercentage of process criteria correctly met and were compared among the 3 m ethods. Results The quality of care, as measured by all 3 methods, ranged from 76.2 % (SPs) to 71.0% (vignettes) to 65.6% (chart abstraction). Measuring qualit y using vignettes consistently produced scores closer to the gold standard of SP scores than using chart abstraction, This pattern was robust when the scores were disaggregated by the 4 conditions (P<.001 to <.05), by case co mplexity (P<.001), by site (P<.001), and by level of physician training (P values from <.001 to <.05). The pattern persisted, although less dominantly , when we assessed the component domains of the clinical encounter-history, physical examination, diagnosis, and treatment, Vignettes were responsive to expected directions of variation in quality between sites and levels of training. The vignette responses did not appear to be sensitive to physicia ns' having seen an SP presenting with the same case. Conclusions Our data indicate that quality of health care can be measured i n an outpatient setting by using clinical vignettes. Vignettes appear to be a valid and comprehensive method that directly focuses on the process of c are provided in actual clinical practice. Vignettes show promise as an inex pensive case-mix adjusted method for measuring the quality of care provided by a group of physicians.