EVALUATING RATER ACCURACY IN PERFORMANCE ASSESSMENTS

A new method for evaluating rater accuracy within the context of perfo rmance assessments is described Accuracy is defined as the match betwe en ratings obtained from operational raters and those obtained from an expert panel on a set of benchmark, exemplar, or anchor performances. An extended Rasch measurement model called the FACETS model is presen ted for examining rater accuracy. The FACETS model is illustrated with 373 benchmark papers rated by 20 operational raters and an expert pan el. The data are from the 1993 field test of the High School Graduatio n Writing Test in Georgia. The data suggest that there are statistical ly significant differences in rater accuracy; the data also suggest th at it is easier to be accurate on some benchmark papers than on others . A small example is presented to illustrate how the accuracy ordering of raters may not be invariant over different subsets of benchmarks u sed to evaluate accuracy.