Reliability in evaluating letters of recommendation

To measure the interobserver reliability in evaluating letters of recommend ation for residency applicants, three letters were collected from each of t he application files of 58 residents at one program. The letters were rated by six faculty. Interobserver reliability, calculated using the kappa stat istic, was slight. These preliminary results show significant variability i n the interpretation of letters of recommendation.