Prioritizing test cases for regression testing

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some perfor mance goal. Various goals are possible; one involves rate of fault detectio n-a measure of how quickly faults are detected within the testing process, An improved rate of fault detection during testing can provide faster feedb ack on the system under test and let software engineers begin correcting fa ults earlier than might otherwise be possible. One application of prioritiz ation techniques involves regression testing-the retesting of software foll owing modifications; in this context, prioritization techniques can take ad vantage of information gathered about the previous execution of test cases to obtain test case orderings. In this paper, we describe several technique s for using test execution information to prioritize test cases for regress ion testing, including: 1) techniques that order test cases based on their total coverage of code components, 2) techniques that order test cases base d on their coverage of code components not previously covered, and 3) techn iques that order test cases based on their estimated ability to reveal faul ts in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites fo r various programs and measured the rates of fault detection achieved by th e prioritized test suites, comparing those rates to the rates achieved by u ntreated, randomly ordered, and optimally ordered suites. Analysis of the d ata shows that each of the prioritization techniques studied improved the r ate of fault detection of test suites, and this improvement occurred even w ith the least expensive of those techniques. The data also shows, however, that considerable room remains for improvement. The studies highlight sever al cost-benefit trade-offs among the techniques studied, as well as several opportunities for future work.