Evaluating surgical resident selection procedures

Purpose: The purposes of this study were to develop and assess a rating for m for selection of surgical residents, determine the criteria most importan t in selection, determine the reliability of the assessment form and proces s both within and across sites, and document differences in procedure and s tructure of resident selection processes across Canada. Methods: Twelve of 13 English-speaking orthopedic surgery training programs in Canada participated during the 1999 selection year. The critical incide nt technique was utilized to determine the criteria most important in selec tion. From these criteria a 10-item rating form was developed with each ite m on a 5-point scale. Sixty-six candidates were invited for interviews acro ss the country. Each interviewer completed one assessment form for each can didate, and independently ranked all candidates at the conclusion of all in terviews. Consensus final rank orders were then created for each residency program. Across all programs, pairwise program-by-program correlations for each assessment parameter were made. Results: The, internal consistency of assessment form ratings for each inte rviewer was moderately high (mean Cronbach's alpha = 0.71). A correlation b etween each item and the final rank order for each program revealed that th e items work ethic, interpersonal qualities, orthopedic experience, and ent husiasm correlated most highly with final candidate rank orders (r = 0.5, 0 .48, 0.48, 0.45, respectively). The interrater reliabilities (within panels ) and interpanel reliabilities (within programs) for the rank orders were 0 .67 and 0.63, respectively. Using the Spearman-Brown prophecy formula, it w as found that two panels with two interviewers on each panel are required t o obtain a stable measure of a given candidate (reliabilities of 0.80). The average pairwise program-by-program correlations were low for the final ca ndidate rank orders (0.14). Conclusions: A method was introduced to develop a standard, reliable candid ate assessment form to evaluate residency selection procedures. The assessm ent form ratings were found to be consistent within interviewers. Candidate assessments within programs (both between interviewers and between panels) were moderately reliable suggesting agreement within programs regarding th e relative quality of candidates, but there was very little agreement acros s programs. (C) 2001 Excerpta Medica, Inc. All rights reserved.