A comparison of physician examiners', standardized patients', and communication experts' ratings of international medical graduates' English proficiency

Purpose. To assess the quality of ratings of interviewing skills and oral E nglish proficiency provided on a clinical skills OSCE by physician examiner s, standardized patients (SPs), and communication skills experts. Method. In 1998, 73 candidates to the Ontario International Medical Graduat e (OIMG) Program completed a 29-station OSCE-type clinical skills selection examination. Physician examiners, SPs, and communication skills experts as sessed components of oral English proficiency and interview performance. Ba sed on these results, the frequency and generalizability of English-languag e flags, physician examiners' indications that spoken English skills were b ad enough to significantly impede communication with patients; the reliabil ity of the OIMG's Interview and Oral Performance Scales and generalizabilit y of overall interview and oral performance ratings; and comparisons of rep eated assessments by experts were calculated. Principal-components analysis was applied to the panels' ratings to determine a more economical expressi on of the language proficiency and interview communication skills results. Results. The mean number of English-language flags per candidate was 2.1, t he median was 1.0, and Cronbach's a of the ratings was 0.63. Means, SDs, an d alphas of the physician examiners' and SPs' ratings of the interview perf ormance scale were 9.15/10, 0.43, 0.36, and 9.30/10, 0.56, 0.50, respective ly. Corresponding values for overall interview performance ratings were 3.0 8/4, 0.30, 0.33, and 3.34/4, 0.32, 0.47. Means, SDs, and alphas of the phys ician examiners' and SPs' ratings of the oral performance scale were 8.54/1 0, 0.74, 0.78, and 8.74/10, 1.00, 0.76. Corresponding values for overall ra tings of oral performance were 3.85/5, 0.51, 0.68, and 4.08/5, 0.60, 0.68. For the two experts' ratings of two contiguous five-minute interview statio ns, internal consistencies were 0.88 and 0.78. For the two experts' ratings of standardized ren-minute interviews, internal consistencies were 0.81 an d 0.92. Correlations between the mean values of the experts' ratings of the ten- and five-minute stations were 0.45 and 0.51. Three factors emerged fr om the PCA, language proficiency, physician examiners' ratings of interview proficiency, and SPs' ratings of interview proficiency Conclusions. Consistency between the physician examiners' and SPs' ratings of English proficiency was observed; less agreement was observed in their r atings of interviewing skills, and little agreement was observed between th e experts' ratings. Communication skills results may be validly expressed b y three measures: one overall global rating of language proficiency provide d by physician examiners or SPs, and overall global ratings of interview pr oficiency provided separately by physician examiners and SPs.