An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales

This laboratory research compared the reliability, validity, and accuracy o f a computerized adaptive rating scale (CARS) format and 2 relatively commo n and representative rating formats, The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled beha vioral statements to the rater to iteratively estimate a ratee's effectiven ess on 3 dimensions of contextual performance. Videotaped vignettes of 6 of fice workers were prepared, depicting prescripted levels of contextual perf ormance, and 112 subjects rated these vignettes using the CARS format and o ne or the other competing format. Results showed 23%-37% lower standard err ors of measurement for the CARS format. In addition, validity was significa ntly higher for the CARS format (d =.18), and Cronbach's accuracy coefficie nts showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.