LEAST-SQUARES MODELS TO CORRECT FOR RATER EFFECTS IN PERFORMANCE ASSESSMENT

This study describes three least squares models to control for rater e ffects in performance evaluation: ordinary least squares (OLS); weight ed least squares (WLS); and ordinary least squares, subsequent to appl ying a logistic transformation to observed ratings (LOG-OLS). The mode ls were applied to ratings obtained from four administrations of an or al examination required for certification in a medical specialty. For any single administration, there were 40 raters and approximately 115 candidates, and each candidate was rated by four raters. The results i ndicated that raters exhibited significant amounts of leniency error a nd that application of the least squares models would change the pass- fail status of approximately 7% to 9% of the candidates. Ratings adjus ted by the models demonstrated higher reliability and correlated sligh tly higher than observed ratings with the scores on a written examinat ion.