'Mental model' comparison of automated and human scoring

'Mental models' used by automated scoring for the simulation divisions of t he computerized Architect Registration Examination are contrasted with thos e used by experienced human graders. Candidate solutions (N = 3613) receive d both automated and human holistic scores. Quantitative analyses suggest h igh correspondence between automated and human scores; thereby suggesting s imilar mental models are implemented. Solutions with discrepancies between automated and human scores were selected for qualitative analysis. The huma n graders were reconvened to review the human scores and to investigate the source of score discrepancies in light of rationales provided by the autom ated scoring process. After review, slightly more than half of the score di screpancies were reduced or eliminated. Six sources of discrepancy between original human scores and automated scores were identified: subjective crit eria; objective criteria; tolerances/weighting; details; examinee task inte rpretation; and unjustified. The tendency of the human graders to be compel led by automated score rationales varied by the nature of original score di screpancy. We determine that, while the automated scores are based on a men tal model consistent with that of expert graders, there remain some importa nt differences, both intentional and incidental, which distinguish between human and automated scoring. We conclude that automated scoring has the pot ential to enhance! the validity evidence of scores in addition to improving efficiency.