Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI

We identify a problem with the process of research in the human-computer in teraction (HCI) community-an overemphasis on "radical invention" at the pri ce of achieving a common research focus. Without such a focus, it is diffic ult to build on previous work, to compare different interaction techniques objectively, and to make progress in developing theory. These problems at t he research level have implications for practice, too; as researchers we of ten are unable to give principled design advice to builders of new systems. We propose that the HCI community try to achieve a common focus around the notion of reference tasks. We offer arguments for the advantages of this a pproach as well as consider potential difficulties. We explain how referenc e tasks have been highly effective in focusing research into information re trieval and speech recognition. We discuss what factors have to be consider ed in selecting HCI reference tasks and present an example reference task ( for searching speech archives). This example illustrates the nature of refe rence tasks and points to the issues and problems involved in constructing and using them. We conclude with recommendations about what steps need to b e taken to execute the reference task research agenda This involves recomme ndations about both the technical research that needs to be done and change s in the way that the HCI research community operates. The technical resear ch involves identification of important user tasks by systematic requiremen ts gathering, definition and operationalization of reference tasks and eval uation metrics, and execution of task-based evaluation, along with judiciou s use of held trials. Perhaps more important, we have also suggested change s in community practice that HCI must adopt to make the reference tasks ide a work We must create forums for discussion of common tasks and methods by which people can compare systems and techniques. Only by doing this can the notion of reference tasks be integrated into the process of research and d evelopment, enabling the field to achieve the focus it desperately needs.