Assessing user competence: Conceptualization and measurement

Organizations today face great pressure to maximize the benefits from their investments in information technology (IT). They are challenged not just t o use IT, but to use it as effectively as possible. Understanding how to as sess the competence of users is critical in maximizing the effectiveness of IT use. Yet the user competence construct is largely absent from prominent technology acceptance and fit models, poorly conceptualized, and inconsist ently measured. We begin by presenting a conceptual model of the assessment of user competence to organize and clarify the diverse literature regardin g what user competence means and the problems of assessment. As an illustra tive study, we then report the findings from an experiment involving 66 par ticipants. The experiment was conducted to compare empirically two methods (paper and pencil tests versus self-report questionnaire), across two diffe rent types of software, or domains of knowledge (word processing versus spr eadsheet packages), and two different conceptualizations of competence (sof tware knowledge versus self-efficacy). The analysis shows statistical signi ficance in all three main effects. How user competence is measured, what is measured, what measurement context is employed: all influence the measurem ent outcome. Furthermore, significant interaction effects indicate that dif ferent combinations of measurement methods, conceptualization, and knowledg e domains produce different results. The concept of frame of reference, and its anchoring effect on subjects' responses, explains a number of these fi ndings. The study demonstrates the need for clarity in both defining what t ype of competence is being assessed and in drawing conclusions regarding co mpetence, based upon the types of measures used. Since the results suggest that definition and measurement of the user competence construct can change the ability score being captured, the existing information system (IS) mod els of usage must contain the concept of an ability rating. We conclude by discussing how user competence can be incorporated into the Task-Technology Fit model, as well as additional theoretical and practical implications of our research.