A COMPARATIVE APPROACH TO SYSTEM EVALUATION - DELEGATING CONTROL OF RETRIEVAL TESTS TO AN EXPERIMENTAL ONLINE SYSTEM

The paper describes the comparative approach to system evaluation used in this research project which delegated the administration of an onl ine retrieval test to an experimental online catalog to produce data f or evaluating the effectiveness of a new subject access design. The ca talog ushered users through a tightly controlled sequence of events an d performed and/or recorded all activity connected with data collectio n including searcher recruitment, questionnaire administration, and lo gging of users' searches and relevance assessments. In the absence of human intermediaries to monitor ongoing searches, system users were fr ee to complete as little or as much of the test administration as they wanted as well as enter queries and responses that were unsuitable to the search administration. This paper describes the methods that the researchers enlisted to sort out ''problem'' test administrations, for example, to identify out-of-scope queries, incomplete system administ rations, and suspect post-search questionnaire responses. It also cove rs how the researchers handled ''problem'' search administrations and what actions they would use to reduce or eliminate the occurrence of s uch administrations in future online retrieval tests that delegate con trol of retrieval tests to online systems.