Design and pilot evaluation of a system to develop computer-based site-specific practice guidelines from decision models

Background. Local tailoring of clinical practice guidelines (CPGs) requires experts in medicine and evidence synthesis unavailable in many practice se ttings. The authors' computer-based system enables developers and users to create, disseminate, and tailor CPGs, using normative decision models (DMs) . Methods. ALCHEMIST, a web-based system, analyzes a DM, creates a CPG in t he form of an annotated algorithm, and displays for the guideline user the optimal strategy. ALCHEMIST'S interface enables remote users to tailor the guideline by changing underlying input variables and observing the new anno tated algorithm that is developed automatically. In a pilot evaluation of t he system, a DM was used to evaluate strategies for staging non-small-cell lung cancer. Subjects (n = 15) compared the automatically created CPG with published guidelines for this staging and critiqued both using a previously developed instrument to rate the CPGs' usability, accountability, and accu racy on a scale of 0 (worst) to 2 (best), with higher scores reflecting hig her quality. Results. The mean overall score for the ALCHEMIST GPG was 1.50 2, compared with the published-CPG score of 0.987 (p = 0.002). The ALCHEMIS T CPG scores for usability, accountability, and accuracy were 1.683, 1.393, and 1.430, respectively; the published CPG scores were 1.192, 0.941, and 0 .830 (each comparison p < 0.05). On a scale of 1 (worst) to 5 (best), users ' mean ratings of ALCHEMIST'S ease of use, usefulness of content, and prese ntation format were 4.76, 3.98, and 4.64, respectively. Conclusions. The re sults demonstrate the feasibility of a web-based system that automatically analyzes a DM and creates a CPG as an annotated algorithm, enabling remote users to develop site-specific CPGs. In the pilot evaluation, the ALCHEMIST guidelines met established criteria for quality and compared favorably wit h national CPGs. The high usability and usefulness ratings suggest that suc h systems can be a good tool for guideline development.