DIFFERENCES UNRELATED TO CLINICAL COMPETENCE IN THE RESULTS OF REPEATED MULTIPLE-STATION TESTS OF CLINICAL SKILLS

Purpose. To determine the extent to which systematic differences not r elated to group differences in clinical competence could be observed i n the results of six administrations of the same multiple-station test of clinical skills. Method. The same 16 stations were administered as part of the Ontario International Medical Graduates Program clinical skills screening examination on three different occasions, and on each occasion in two sessions. The interval between the first (fall 1986) and second (March 1990) administrations was four years, and that betwe en the second and third (July 1990) administrations, four months. Inte rnational medical graduates were the candidates in the first two admin istrations; the third administration was to a combined sample of fourt h-year clinical clerks and interns. A different item functioning (DIF) approach with sessions within administration as the grouping variable was used to describe the extent of differences unrelated to clinical competence in the results of the different test administrations. For t he generalizability and DIF calculations the designs were balanced by sampling down to 33 cases in each session. Results. Station DIF values varied considerably from station to station. DIF effects between sess ions with administrations were less than those between administrations , and were less between the two administrations separated by four mont hs than between the two administrations separated by four years. Concl usion. Hypotheses concerning the relative magnitudes of station DIF an d total-test aggregated values of DIE as a function of the time interv als between test occasions, were substantiated, demonstrating that the greater the time interval between test administrations, the greater t he magnitude of DIF.