A PERFORMANCE AND FAILURE ANALYSIS OF SAPHIRE WITH A MEDLINE TEST COLLECTION

Objective: Assess the performance of the SAPHIRE automated information retrieval system. Design: Comparative study of automated and human se arching of a MEDLINE test collection. Measurements: Recall and precisi on of SAPHIRE were compared with those attributes of novice physicians , expert physicians, and librarians for a test collection of 75 querie s and 2,334 citations. Failure analysis assessed the efficacy of the M etathesaurus as a concept vocabulary; the reasons for retrieval of non relevant articles and nonretrieval of relevant articles; and the effec t of changing the weighting formula for relevance ranking of retrieved articles. Results: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. Co nclusion: The current version of the Metathesaurus, as utilized by SAP HIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries, The most likely cause for retriev al of nonrelevant articles was the presence of some or all of the sear ch terms in the article, with frequencies high enough to lead to retri eval. The most likely cause for nonretrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hiera rchically related terms present instead. There were significant variat ions in performance when SAPHIRE's concept-weighing formulas were modi fied.