THE COVARIANCE DECOMPOSITION OR THE PROBABILITY SCORE AND ITS USE IN EVALUATING PROGNOSTIC ESTIMATES

The probability score (PS) or Brier score has been used in a large num ber of studies in which physician judgment performance was assessed. H owever, the covariance decomposition of the PS has not previously been used to evaluate medical judgment. The authors introduce the techniqu e and demonstrate it by analyzing prognostic estimates of three groups : physicians, their patients, and the patients' decision-making surrog ates. The major components of the covariance decomposition-bias, slope , and scatter-are displayed in covariance graphs for each of the three groups. The decomposition reveals that whereas the physicians have th e best overall estimation performance, their bias and their scatter ar e not always superior to those of the other two groups. This is primar ily due to two factors. First, the physicians' prognostic estimates ar e pessimistic. Second, the patients place the large majority of their estimates in the most optimistic category, thereby achieving low scatt er. The authors suggest that the calculational simplicity of this deco mposition, its informativeness, and the intuitive nature of its compon ents make it a useful tool with which to analyze medical judgment.