Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited

In the validation of medical expert systems. agreement among different huma n specialists on a random sample of cases may be taken as a substitute to a missing gold standard. Distance measures between pairs of experts. extensi vely described in previous studies. do not take into account the influence of chance-expected agreement. A weighted kappa index, with three different weighting schemes. is proposed as an alternative to be applied in the gener al situation of N cases assessed by E experts about K possible diagnoses, e ach of. them qualified with one of G ordinal categories. A hierarchical clu ster analysis. applied to the kappa matrices generated. allows for the clas sification of the expert system among clinical specialists. providing a rel ative assessment of its diagnostic ability. The above methodology is applie d to the validation of two medical expert systems. PNEUMON-IA and RENOIR. ( C) 2000 Academic Press.