Use of relevance criteria across stages of document evaluation: On the complementarity of experimental and naturalistic studies

This paper provides an overview of empirical investigations of people's use of relevance criteria. A laboratory experiment and a naturalistic study we re conducted to explore the patterns of movement in use of relevance criter ia during real-time document evaluation processes. The expectation was that the subjects would apply different frames of reference for relevance decis ion making as they moved from one stage of the document evaluation to anoth er, and that change would occur not only for particular criteria but also f or classes of criteria. The examination of such change was performed at two levels: a micro-level analysis, centering on the use of individual criteri a, and a macro-level analysis, concentrating on the use of criteria classes . General results of the two studies are included in this paper. Both proje cts provided information on criteria patterns, which challenge the conventi onal meaning-based approach to criteria classification. Building on these f indings, a classification scheme and criteria taxonomy that may enable gene ralization of criteria across different contexts is proposed.