COGNITIVE DIFFERENCES IN PROFICIENT AND NONPROFICIENT ESSAY SCORERS

This article examines the behavioral differences of essay scorers who demonstrate different levels of proficiency for a psychometric scoring task. The authors compare three proficiency groups to identify differ ences in (a) essay features they consider, (b) their understandings of the scoring rubric, and (c) their decision-making procedures. Results indicate scorers with different levels of proficiency do not focus on different essay features when making evaluative decisions but their u nderstandings of the scoring criteria may vary Proficient scorers are more likely to focus on general features of an essay when making evalu ative decisions and to adopt values espoused by the scoring rubric tha n are less proficient scorers. Also, proficient scorers make evaluatio ns by reading the entire essay and then reviewing its content, whereas less proficient scorers may interrupt the reading process to monitor how well the essay satisfies the scoring criteria. Finally, the author s discuss implications for scorer selection and training.