GLOBAL JUDGMENTS VERSUS DECISION-MODEL-FACILITATED JUDGMENTS - ARE EXPERTS INTERNALLY CONSISTENT

A widely used method for evaluating the appropriateness of medical pro cedures and practices is the ''modified Delphi'' approach using expert panelists' global ratings. However, several difficulties in the assig nment of global ratings have led to a search for alternative methods, including the use of decision models. To examine the potential impact of using decision models with an expert panel, the authors compared a panel's global ratings for the appropriateness of carotid endarterecto my with the results or a decision-analytic model in which expert panel ists estimated probabilities and utilities that were used as inputs fo r the model. For 17 different patient scenarios, the nine expert panel ists showed variability in ''calibration'' between the two methods, wi th their expected utilities calculated from the model generally being higher than their global ratings. However, the correlation between the two methods was excellent. When the panel's median global utility was compared with the panel's median expected utility calculated from the model, the Spearman correlation coefficient was 0.88. This study demo nstrated that an expert panel's appropriateness ratings and their expe cted utilities were highly correlated. In addition, the panelists appe ared to be internally consistent in that their judgments about individ ual probabilities and utilities were correlated with their global judg ments. These results should encourage additional efforts to incorporat e decision models into the process of clinical guideline development. The authors believe that decision models can help improve a panel's ca pacity to understand and reconcile discordance, and increase their sat isfaction that the process reflects the best possible judgments.