Evaluation of the quality of information retrieval of clinical findings from a computerized patient database using a semantic terminological model

Objectives: To measure the strength of agreement between the concepts and r ecords retrieved from a computerized patient database, in response to physi cian-derived questions, using a semantic terminological model for clinical findings with those concepts and records excerpted clinically by manual ide ntification. The performance of the semantic terminological model is also c ompared with the more established retrieval methods of free-text search, IC D-10, and hierarchic retrieval. Design: A clinical database (Diabeta) of 106,000 patient problem record ent ries containing 2,625 unique concepts in an clinical academic department wa s used to compare semantic, free-text, ICD-10, and hierarchic data retrieva l against a gold standard in response to a battery of 47 clinical questions . Measurements: The performance of concept and record retrieval expressed as mean detection rate, positive predictive value, Yates corrected and Mantel- Haenszel chi-squared values, and Cohen kappa value, with significance estim ated using the Mann-Whitney test. Results: The semantic terminological model used to retrieve clinically usef ul concepts from a patient database performed well and better than other me thods, with a mean detection rate of 0.86, a positive predictive value of 0 .96, a Yates corrected chi-squared value of 1,537, a Mantel-Haenszel chi-sq uared value of 19,305 and a Cohen kappa of 0.88, Results for record retriev al were even better, with a mean record detection rate of 0.94, a positive predictive value of 0.99, a Yates corrected chi-squared value of 94,774, a Mantel-Haenszel chi-squared value of 1,550,356, and a Cohen kappa value of 0.94. The mean detection rate, Yates corrected chi-squared value, and Cohen kappa value for semantic retrieval were significantly better than for the other methods. Conclusion: The use of a semantic terminological model in this test scenari o provides an effective framework for representing clinical finding concept s and their relationships. Although currently incomplete, the model support s improved information retrieval from a patient database in response to cli nically relevant questions, when compared with alternative methods of analy sis.