Reliability of conformance tests

Conformance testing is considered from a statistical point of view. A s-con fidence interval is found for the reliability that an implementation of a s oftware package complies with specifications of a standard. Determination o f whether it complies depends on a conformance test, which is written direc tly from the standard. Although the conformance test is written directly fr om the standard it does not test all possible software parameter-settings t hat invoke the standard. Thus, statistical inference is necessary. A genera l s-confidence interval for the reliability is given when the specification requires that the implementation passes all the tests in the conformance t est suite. The conformance test is made of disjoint homogeneous partitions. The failure probability of the software is based on a weighted linear comb ination of the partition failure probabilities. An example is included.