Evaluation of a method that supports pathology report coding

Objectives. The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the agreement between pathologists in coding an d the quality of a system that supports pathologists in coding pathology re ports were evaluated. Methods: Six sets of three pathologists each received a different set of 40 pathology reports. Five different SNOMED code lines accompanied each patho logy report. Three pathologists evaluated the correctness of each of these code lines. Kappa values and values for the reliability coefficients were d etermined to gain insight in the variance observed when coding pathology re ports. The system that is evaluated compares a newly entered report, repres ented as a multi-dimensional word vector, with reports in a library, repres ented in the some way. The reports in the library are already coded. The sy stem presents the code lines belonging to the five library reports most sim ilar to the newly entered one to the pathologist in this way supporting the pathologist in determining the correct codes. A high similarity between tw o reports is indicated by a large value of the inproduct of the vector of t he newly entered report and the vector of a report in the library. Results: Agreement between pathologists in coding was fair (average kappa o f 0.44). The reliability coefficient varied from 0.81 to 0.89 for the six s ets of pathology reports. The system gave correct suggestions in 50% of the reports. In another 30% it was helpful for the pathologists. Conclusions. On the basis of the level of the reliability coefficients it c ould be concluded that three pathologists are indeed sufficient for obtaini ng a gold standard for evaluating the system. The method used for comparing reports is not strong enough to allow fully automatic coding. It could be shown that the system induces a more uniform coding by pathologists. An eva luation of the incorrect suggestions of the system indicates that the perfo rmance of the system can still be improved.