EVALUATING CONSENSUS AMONG PHYSICIANS IN MEDICAL KNOWLEDGE-BASE CONSTRUCTION

This study evaluates inter-author variability in knowledge base constr uction. Seven board-certified internists independently profiled ''acut e perinephric abscess'', using as reference material a set of 109 peer -reviewed articles. Each participant created a list of findings associ ated with the disease, estimated the predictive value and sensitivity of each finding, and assessed the pertinence of each article for makin g each judgment. Agreement in finding selection was significantly diff erent from chance: seven, six, and five participants selected the same finding 78.6, 9.8, and 1.6 times more often than predicted by chance. Findings with the highest sensitivity were most likely to be included by all participants. The selection of supporting evidence from the me dical literature was significantly related to each physician's agreeme nt with the majority. The study shows that, with appropriate guidance, physicians can reproducibly extract information from the medical lite rature, and thus established a foundation for multi-author knowledge b ase construction.