Evaluation of a computerized diagnostic decision support system for patients with pneumonia: Study design considerations

Planning the clinical evaluation of a computerized decision support system requires a strategy that encompasses the different aspects of the clinical problem, the technical difficulties of software and hardware integration an d implementation, the behavioral aspects of the targeted users, and the dis cipline of study design. Although clinical information systems are becoming more widely available, only a few decision support systems have been forma lly evaluated in clinical environments. Published accounts of difficulties associated with the clinical evaluation of decision support systems remain scarce. The authors report on a variety of behavioral, logistical, technica l, clinical, cost, and work flow issues that they had to address when choos ing a study design for a clinical trial for the evaluation of an integrated , real-time decision support system for the automatic identification of pat ients likely to have pneumonia in an emergency department. In the absence o f a true gold standard, they show how they created a credible, clinically a cceptable, and economical reference standard for the diagnosis of pneumonia , to determine the overall accuracy of the system. For the creation of a re ference standard, they describe the importance of recognizing verification bias and avoiding it. Finally, advantages and disadvantages of different st udy designs are explored with respect to the targeted users and the clinica l setting.