SETTING STANDARDS AND DEFINING QUALITY OF PERFORMANCE IN THE VALIDATION OF A STANDARDIZED-PATIENT EXAMINATION FORMAT

Purpose. To evaluate whether written standards increase the reproducib ility of a physician-facilitated station in an objective structured cl inical examination (OSCE) designed to assess history, physical-examina tion, and communication skills. Method. The OSCE examination at the Un iversity of Texas Medical Branch-Galveston consists of ten eight-minut e stations. Six of these stations consist of three History, Physical e xamination, Problem-solving, and Plan (HPPP) station pairs. Each exist ing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evalua tion of third year medical students. Three pairs of faculty members we re used to determine interrater reliability by scoring videotapes of t hree HPPP stations' presentation and problem solving components. Facul ty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for signific ance using Fisher's R to Z transformation. The reproducibility and sta ndard error of measurement (SEM) were extrapolated for increasing amou nts of testing time. The HPPP component scores were also correlated wi th the written examination scores and preceptors' ratings. Data were o btained from the three HPPP stations used in the 1995-96 internal medi cine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater rel iability and reached statistical significance (p<.01) for one HPPP sta tion. Reproducibility fur the presentation and problem solving compone nts of the HPPP stations were >.80 after five hours of testing. The pr oblem solving component correlated at .37 and .19 with written examina tions and with ward grades, respectively. Conclusion. The data from th is study suggest that standards increase the reproducibility of presen tation and problem-solving components of an OSCE to a level as high as , or higher than, that associated with the history, physical examinati on, and communication components of traditional standardized-patient e xaminations.