Modeling the effects of combining diverse software fault detection techniques

The software engineering literature contains many studies of the efficacy o f fault finding techniques. Few of these, however, consider what happens wh en several different techniques are used together. We show that the effecti veness of such multitechnique approaches depends upon quite subtle interpla y between their individual efficacies and dependence between them. The mode ling tool we use to study this problem is closely related to earlier work o n software design diversity. The earliest of these results showed that, und er quite plausible assumptions, it would be unreasonable even to expect sof tware versions that were developed "truly independently" to fail independen tly of one another. The key idea here was a "difficulty function" over the input space. Later work extended these ideas to introduce a notion of "forc ed" diversity, in which it became possible to obtain system failure behavio r better even than could be expected if the versions failed independently. In this paper, we show that many of these results for design diversity have counterparts in diverse fault detection in a single software version. We d efine measures of fault finding effectiveness and of diversity and show how these might be used to give guidance for the optimal application of differ ent fault finding procedures to a particular program. We show that the effe cts upon reliability of repeated applications of a particular fault finding procedure are not statistically independent-in fact, such an incorrect ass umption of independence will always give results that are too optimistic. F or diverse fault finding procedures, on the other hand, things are differen t: Here it is possible for effectiveness to be even greater than it would b e under an assumption of statistical independence. We show that diversity o f fault finding procedures is, in a precisely defined way, "a good thing" a nd should be applied as widely as possible. The new model and its results a re illustrated using some data from an experimental investigation into dive rse fault finding on a railway signalling application.