A generalizability study of a new standardized rating form used to evaluate students' clinical clerkship performances

Purpose. To investigate the measurement characteristics of standardized cli nical evaluation forms (CEFs) used to assign grades for clerkship performan ce. Method. In 1996-97, the authors reviewed 5,168 CEFs completed for 175 stude nts in eight clerkships. Limiting their analysis to the three clerkships th at produced the most CEFs, the authors conducted a generalizability study t o determine the five variance components for each clerkship. A decision stu dy then calculated the generalizability coefficients and standard errors of measurement in each clerkship for varied numbers of raters and CEF items. Results. The generalizability study found large variance components attribu table to rater and rating context. The decision study found that, when thre e or more raters completed CEFs for a student, the generalizability coeffic ient and standard error of measurement reached levels acceptable for gradin g. Increasing the number of items on the CEF had no significant effect. Conclusion. The reliability of assigning students clerkship grades based on single CEFs is unacceptably low. However, CEFs can accurately measure stud ents' clerkship performances if completed by three or more raters.