WHEN TO STOP TESTING FOR LARGE SOFTWARE SYSTEMS WITH CHANGING CODE

Developers of large software systems must decide how long software sho uld be tested before releasing it. A common and usually unwarranted as sumption is that the code remains frozen during testing. We present a stochastic and economic framework to deal with systems that change as they are tested. The changes can occur because of the delivery of soft ware as it is developed, the way software is tested, the addition of f ixes, and so on. Specifically, we report the details of a real time tr ial of a large software system that had a substantial amount of code a dded during testing. We describe the methodology, give all of the rele vant details, and discuss the results obtained. We pay particular atte ntion to graphical methods that are easy to understand, and that provi de effective summaries of the testing process. Some of the plots found useful by the software testers include the Net Benefit Plot, which gi ves a running chart of the benefit; the Stopping Plot, which estimates the amount of additional time needed for testing; and diagnostic plot s. To encourage other researchers to try out different models, all of the relevant data are provided.