PERFORMANCE EVALUATION OF A LARGE DIAGNOSTIC EXPERT-SYSTEM USING A HEURISTIC TEST-CASE GENERATOR

Validating the performance of a knowledge-based system is a critical s tep in its commercialization process. Without exception, buyers of sys tems intended for serious purposes require a certain level of guarante es about system performance. This is particularly true for diagnostic systems. Yet, many problems exist in the validation process, especiall y as it applies to large knowledge-based systems. One of the biggest c hallenges facing the developer when validating the system's performanc e is knowing how much testing is sufficient to show that the system is valid. Exhaustive testing of the system is almost always impractical due to the many possible test cases that can be generated, many of whi ch are not useful. It would thus be highly desirable to have a means o f defining a representative set of test cases that, if executed correc tly by the system, would provide a high confidence in the system's val idity. This paper describes the experiences of the development ream in validating the performance of a large commercial diagnostic knowledge -based system. The description covers the procedure employed to carry out this task, as well as the heuristic technique used for generating the representative set of test cases. Copyright (C) 1996 Elsevier Scie nce Ltd