Assessing the generalizability of OSCE measures across content domains

Purpose. To assess the degree to which OSCE measures generalize across mult iple administrations to the same students, Method. Students' scores from three OSCEs at one institution were correlate d to determine the generalizability of the scoring systems across course do mains. Results. Analysis revealed that while checklist scores showed quite low cor relations across examinations from different domains (ranging from 0.14 to 0.25), global process scores showed quite reasonable correlations (ranging from 0.30 to 0.44), with the correlations for global scores being significa ntly higher than those for checklist scores in all three comparisons. Conclusion. These data would seem to confirm the intuitions about each of t hese measures: the checklist scores are highly content-specific, while the global scores are evaluating a more broadly based set of skills. Implicatio ns for the use of these scales are discussed.