Sequential testing in the assessment of clinical skills

Purpose. Educators who use standardized patient-based (SP-based) tests may save resources by using sequential testing. In this approach, students take a short screening test; only those who fail take a second test. This study investigated whether sequential testing increases efficiency with only a m inor decrease of validity. Method. In 1994-95, first- through fourth-year (Group 1) and sixth-year (Gr oup 2) medical students at the University of Maastricht took SP-based tests . Each test took two days. In a simulation experiment based on the data fro m those tests, the authors considered the first day as the screening test a nd the second day as the second test. They investigated efficiency and vali dity as a function of the cutoff score of the screening test. They develope d and evaluated a new method to determine the optimum cutoff score of the s creening test, a method based on minimization of the loss represented by th e (weighted) numbers of false positives and negatives in the screening test . Results. The negative predictive value (probability that a student would fa il the complete test if he or she had failed the screening test) was low (< 60%), while the positive predictive value was high (>96%). Accordingly, str ingent pass/fail cutoff scores in the screening test (75% for Group 1 and 8 0% for Group 2) produced optimum results. Using those cutoff values, only 2 6% and 11% of the students would have had to rake the complete test to get a "true" score, while only 0.2% and 0.0% of the students who passed the scr eening test went on to fail the complete test (false positives). Conclusions. In a sequential SP-based test, the pass/fail cutoff score of t he screening test should be stringent. This can considerably reduce testing time (30% to 40%), while keeping the percentage of false positives at an a cceptably low level of less than 0.2%. As an alternative to receiver operat or characteristic analysis, minimization of the loss function was found to be an appropriate method to determine the optimum cutoff value of the scree ning test.