INTERRATER CONCORDANCE FOR FACULTY GRADING OF STUDENT PERFORMANCES INA PROBLEM-BASED LEARNING COURSE

Purpose. To determine whether it is possible for faculty to arrive at consistent, non-idiosyncratic grades in a problem-based learning (PBL) course. Method. Integrated Case Studies and Medical Decision Making ( ICS) is the final course of the second year at the University of Pitts burgh School of Medicine. In ICS, 16 groups of nine students work in a PBL format over seven weeks. Each group is led by three faculty facil itators who, at the end of the course, independently give each student ratings for overall performance in the course and for each of seven p erformance categories. In 1993-94 and 1994-95, concordance in grades a mong the facilitators was determined by computing the intraclass corre lation coefficients [ICC (3,1)] for the overall scores, the seven perf ormance category scores, and all eight scores in aggregate. An ICC (3, 1) of greater than or equal to .1 was considered indicative of statist ically significant interrater concordance. An ICC (3,1)of greater than or equal to .7 was considered indicative of concordance of practical significance. Results. Because the facilitators occasionally did not r ate every student in every performance category, complete information was not available for all 32 groups. Statistically significant concord ance was achieved in the aggregate scores in 100% of 23 groups, and in the overall scores in 90% of 18 groups. In six of the seven performan ce categories, concordance was achieved in at least 75% of the groups (n = 16-20). Practically significant concordance was achieved in the a ggregate scores in 83% of 23 groups. Conclusion. The study results sho w that, given specific criteria by which to judge students' performanc es, it is possible to arrive at consistent, non-idiosyncratic grades f or students in PBL courses.