DEVELOPMENT OF A SCORING ALGORITHM TO REPLACE EXPERT RATING FOR SCORING A COMPLEX PERFORMANCE-BASED ASSESSMENT

Performance assessments typically are scored by having experts rate in dividual performances. In contexts such as medical licensure, where th e examinee population is large and the pool of expert raters is limite d practically, this approach may be unworkable. This article describes an automated scoring algorithm for a computer simulation-based examin ation of physicians' patient-management skills. The algorithm is based on the policy used by clinicians in rating case performances. The res ults show that scores produced using this algorithm are highly correla ted to actual clinician ratings. These scores also are shown to be eff ective in discriminating between case performance judged to be passing or failing by an independent group of clinicians.