ALL-USES VS MUTATION TESTING - AN EXPERIMENTAL COMPARISON OF EFFECTIVENESS

The effectiveness of a test data adequacy criterion for a given progra m and specification is the probability that a test set satisfying the criterion will expose a fault. Experiments were performed to compare t he effectiveness of the mutation testing and all-uses test data adequa cy criteria at various coverage levels, for randomly generated test se ts. Large numbers of test sets were generated and executed, and for ea ch, the proportion of mutants killed or def-use associations covered w as measured. This data was used to estimate and compare the effectiven ess of the criteria. The results were mixed: at the highest coverage l evels considered, mutation was more effective than all-uses for five o f the nine subjects, all-uses was more effective than mutation for two subjects, and there was no clear winner for two subjects. However, mu tation testing was much more expensive than all-uses. The relationship between coverage and effectiveness for fixed-sized test sets was also explored and was found to be nonlinear and, in many cases, nonmonoton ic. (C) 1997 Elsevier Science Inc.