ON BAYESIAN-ANALYSIS OF MULTIRATER ORDINAL DATA - AN APPLICATION TO AUTOMATED ESSAY GRADING

A framework is proposed for the analysis of ordinal categorical data w hen ratings from several judges are available. I emphasize the tasks o f estimating latent trait characteristics of individual items, regress ing these latent traits on observed covariates, and comparing the perf ormance of raters. The model is illustrated in the design and evaluati on of an automated essay grader. This grader is based on a regression of variables, obtained from a grammar checker, on essay scores estimat ed from a panel of experts. The performance of the grader is evaluated relative to human graders, and implications on the reliability and re peatability of both automated and human raters is investigated.