STATISTICAL PROCESS-CONTROL METHODS FOR EXPERT-SYSTEM PERFORMANCE MONITORING

The literature on the performance evaluation of medical expert systems is extensive, yet most of the techniques used in the early stages of system development are inappropriate for deployed expert systems. Beca use extensive clinical and informatics expertise and resources are req uired to perform evaluations, efficient yet effective methods of monit oring performance during the long-term maintenance phase of the expert system life cycle must be devised. Statistical process control techni ques provide a well-established methodology that can be used to define policies and procedures for continuous, concurrent performance evalua tion. Although the field of statistical process control has been devel oped for monitoring industrial processes, its tools, techniques, and t heory are easily transferred to the evaluation of expert systems. Stat istical process tools provide convenient visual methods and heuristic guidelines for detecting meaningful changes in expert system performan ce. The underlying statistical theory provides estimates of the detect ion capabilities of alternative evaluation strategies. This paper desc ribes a set of statistical process control tools that can be used to m onitor the performance of a number of deployed medical expert systems. It describes how p-charts are used in practice to monitor the GermWat cher expert system. The case volume and error rate of GermWatcher are then used to demonstrate how different inspection strategies would per form.