B. Littlewood et D. Wright, SOME CONSERVATIVE STOPPING RULES FOR THE OPERATIONAL TESTING OF SAFETY-CRITICAL SOFTWARE, IEEE transactions on software engineering, 23(11), 1997, pp. 673-683
Operational testing, which aims to generate sequences of test cases wi
th the same statistical properties as those that would be experienced
in real operational use, can be used to obtain quantitative measures o
f the reliability of software. In the case of safely critical software
it is common to demand that all known faults are removed. This means
that if there is a failure during the operational testing, the offendi
ng fault must be identified and removed. Thus an operational test for
safety critical software takes the form of a specified number of test
cases (or a specified period of working) that must be executed failure
-free. This paper addresses the problem of specifying the numbers of t
est cases (or time periods) required for a test, when the previous tes
t has terminated as a result of a failure. It has been proposed that,
after the obligatory fix of the offending fault, the software should b
e treated as if it were completely novel, and be required to pass exac
tly the same test as originally specified. The reasoning here claims t
o be conservative, inasmuch as no credit is given for any previous fai
lure-free operation prior to the failure that terminated the test. We
show that, in fact, this is not a conservative approach in all cases,
and propose instead some new Bayesian stopping rules. We show that the
degree of conservatism in stopping rules depends upon the precise way
in which the reliability requirement is expressed. We define a partic
ular form of conservatism that seems desirable on intuitive grounds, a
nd show that the stopping rules that exhibit this conservatism are als
o precisely the ones that seem preferable on other grounds.