A comparison of a Bayesian vs. a frequentist method for profiling hospitalperformance

The objective of this study was to compare the classification of hospitals as outcomes outliers using a commonly implemented frequentist statistical a pproach vs. an implementation of Bayesian hierarchical statistical models, using 30-day hospital-level mortality rates for a cohort of acute myocardia l infarction patients as a test case. For the frequentist approach, a logis tic regression model was constructed to predict mortality. For each hospita l, a risk-adjusted mortality rate was computed. Those hospitals whose 95% c onfidence interval, around the risk-adjusted mortality rate, excludes the m ean mortality rate were classified as outliers. With the Bayesian hierarchi cal models, three factors could vary: the profile of the typical patient (l ow, medium or high risk), the extent to which the mortality rate for the ty pical patient departed from average, and the probability that the mortality rate was indeed different by the specified amount. The agreement between t he two methods was compared for different patient profiles, threshold diffe rences from the average and probabilities. Only marginal agreement was show n between the Bayesian and frequentist approaches. In only five of the 27 c omparisons was the kappa statistic at least 0.40. The remaining 22 comparis ons demonstrated only marginal agreement between the two methods. Within th e Bayesian framework, hospital classification clearly depended on patient p rofile, threshold and probability of exceeding the threshold. These inconsi stencies raise questions about the validity of current methods for classify ing hospital performance, and suggest a need for urgent research into which methods are most meaningful to clinicians, managers and the general public .