Cardiac surgery report cards: Comprehensive review and statistical critique

Public report cards and confidential, collaborative peer education represen t distinctly different approaches to cardiac surgery quality assessment and improvement. This review discusses the controversies regarding their metho dology and relative effectiveness. Report cards have been the more commonly used approach, typically as a resu lt of state legislation. They are based on the presumption that publication of outcomes effectively motivates providers, and that market forces will r eward higher quality. Numerous studies have challenged the validity of thes e hypotheses. Furthermore, although states with report cards have reported significant decreases in risk-adjusted mortality, it is unclear whether thi s improvement resulted from public disclosure or, rather, from the developm ent of internal quality programs by hospitals. An additional confounding fa ctor is the nationwide decline in heart surgery mortality, including states without quality monitoring. Finally, report cards may engender negative be haviors such as high-risk case avoidance and "gaming" of the reporting syst em, especially if individual surgeon results are published. The alternative approach, continuous quality improvement, may provide an op portunity to enhance performance and reduce interprovider variability while avoiding the unintended negative consequences of report cards. This collab orative method, which uses exchange visits between programs and determinati on of best practice, has been highly effective in northern New England and in the Veterans Affairs Administration. However, despite their potential ad vantages, quality programs based solely on confidential continuous quality improvement do not address the issue of public accountability. For this rea son, some states may continue to mandate report cards. In such instances, i t is imperative that appropriate statistical techniques and report formats are used, and that professional organizations simultaneously implement cont inuous quality improvement programs. The statistical methodology underlying current report cards is flawed, and does not justify the degree of accuracy presented to the public. All existi ng risk-adjustment methods have substantial inherent imprecision, and this is compounded when the results of such patient-level models are aggregated and used inappropriately to assess provider performance. Specific problems include sample size differences, clustering of observations, multiple compa risons, and failure to account for the random component of interprovider va riability. We advocate the use of hierarchical or multilevel statistical mo dels to address these concerns, as well as report formats that emphasize th e statistical uncertainty of the results. (C) 2001 by The Society of Thorac ic Surgeons.