Understanding and measuring the sources of variation in the prioritizationof regression test suites

Test case prioritization techniques let testers order their test cases so t hat those with higher priority, according to some criterion, are executed e arlier than those with lower priority. In previous work, we examined a vari ety of prioritization techniques to determine their ability to improve the rate of fault detection of test suites. Our studies showed that the rate of fault detection of test suites could be significantly improved by using mo re powerful prioritization techniques. In addition, they indicated that rat e of fault detection was closely associated with the target program. We als o observed a large quantity of unexplained variance, indicating that other factors must be affecting prioritization effectiveness. These observations motivate the following research questions: (1) Are there factors other than the target program and the prioritization technique that consistently affe ct the rate of fault detection of test suites? (2) What metrics are most re presentative of each factor? (3) Can the consideration of additional factor s lead to more efficient prioritization techniques? To address these questi ons, we performed a series of experiments exploring three factors: program structure, test suite composition, and change characteristics. This paper r eports the results and implications of those experiments.