Challenges in using rater judgements in medical education

Changes in the healthcare environment are putting increasing pressure on me dical schools to make faculty accountable and to document the quality of th e medical education they provide. Faculty's ratings of students' performanc es and students' ratings of faculty's teaching are important elements in th ese efforts to document educational quality. This article discusses selecte d research related to factors affecting raters' judgements, analyses how ch anges in the health care environment are influencing such judgements, offer s some suggestions to moderate some of the effects and links these influenc es to the system that upholds professional standards. Ratings are known to have a positive bias (generosity error), provide limited discrimination and often fail to document serious deficits. The potential sources of these pr oblems relate to the mechanics of the rating task, the system used to obtai n ratings and factors affecting rater judgement. As managed care demands re duce the time faculty have for teaching, as system-wide disincentives to pr ovide negative ratings proliferate and as social engineering challenges, su ch as the Americans with Disabilities Act, impose differential standards fo r students, the natural tendency to avoid giving negative ratings becomes e ven harder to resist. Ultimately, these forces compromise the capability of faculty to uphold the standards of the profession. The author calls for a national effort to stem the erosion of those standards.