Assessment of practicing family physicians: Comparison of observation in amultiple station examination using standardized patients with observation of consultations in daily practice

Purpose. Looking for a valid, reliable, and feasible method to collect data on the performances of practicing family physicians, the authors compare t he measurement characteristics of a multiple-station examination (MSE) usin g standardized patients with those of a video assessment of regular consult ations in daily practice (practice video assessment, PVA). Method. In a cross-sectional study, consultations of 90 family physicians w ere videotaped both in an MSE and in their daily practices. Peer-observers used a validated instrument (MAAS-Global) to assess the physicians' communi cation with patients and their medical performances. The physicians were ra ndomly divided into two groups, comparable for demographic characteristics, and half underwent the assessments in reverse order to test for time-order effects. Content validity, criterion validity, reliability, and feasibilit y of the two methods were compared. Results. Content validity of the PVA was superior to that of the MSE, since the domain of general family practice care was better covered. Observed pa rticipants judged the videotaped practice consultations tu be "natural," wh ereas hardly any family physician, after reviewing the videotaped consultat ions of the MSE, recognized his or her usual working style. Specific criter ia made it possible to standardize real practice. Concerning criterion vali dity, only the medical-performance components of the two methods correlated . No correlation was found for the communication components. Real-practice performance proved to be less influenced by observation than was performanc e during the MSE. The reliabilities of the two methods, expected to be Lett er in the controlled MSE, were comparable. The administration of the PVA wa s more flexible, less costly, and better accepted by the family physicians than was that of the MSE. Conclusion. Assessment for quality improvement of family physicians' practi ces by video observation in daily practice is superior to video assessment in a simulated setting using standardized patients.