The reliability of logbook data of medical students: an estimation of interobserver agreement, sensitivity and specificity

Objective Logbooks are widely used in medical schools as an evaluation tool to assess students' progress towards objectives. To estimate whether stude nts fill in their logbooks reliably, we measured interobserver agreement by comparing doctors' data and students' data. Method Completed logbooks were collected at two subdivisions of the departm ent of Internal Medicine at the University Hospital of Groningen. The logbo ok contains 231 preprinted diseases. Doctors and students recorded the dise ases they had encountered. Interobserver agreement, expressed by the Jaccar d coefficient (J), was calculated for the complete set of diseases and for a subset of core diseases. To assess the kinds of errors which students mad e, sensitivity and specificity were determined. Results Logbook data of doctors and students are not fully consistent (mean J for the complete set of diseases was .23 and for the core diseases .36). The quality of the logbook data is high in the sense that students do not record many false identifications (mean specificity for the complete set of diseases and for the core diseases were .96 and .93, respectively); the qu ality is poor in the sense that students do not record all the diseases whi ch could be seen at the department (mean sensitivity for the complete set o f diseases is .36 and for the core diseases it is .51). Conclusion This study shows inconsistencies in recording diseases in a logb ook by students compared with doctors. In particular the diseases which are present at a department are under-reported by students. Supervision and fe edback are important mechanisms to optimize the students' use of (1) all di seases which could be encountered and (2) the logbook.