Validity of surgical simulation for the assessment of operative skill

Background: Surgical simulators are being promoted as a means of assessing a surgeon's technical skills. Little evidence exists that simulator perform ance correlates with actual technical ability. This study was undertaken to determine the criterion and construct validity of currently available surg ical simulations in the evaluation of technical skill. Methods: Simulator assessment was carried out on 36 basic surgical trainees , 37 surgically naive first-year medical students and 16 experienced genera l surgical consultants. Some 26 trainees and 36 students underwent repeat a ssessment after 6 months. A previously validated, 19-point technical skill assessment form, based on direct observation of trainee performance in the operating theatre, was also completed by each trainee's supervising consult ant. Results: An insignificant or weak correlation was found between simulator p erformance and both duration of basic surgical experience and consultant as sessment of technical skill. Six months of basic surgical training led to a n improvement in performance, not seen in an untrained control group, in on ly one of the six simulations tested. Discrimination between surgically nai ve and experienced subjects was only demonstrated, in part, for four of the six tasks. Conclusion: The assessment of technical skill needs to be improved. Work is needed to establish the reliability and validity of currently available si mulation models before they are formally introduced for high-stakes assessm ent.