Statistical analysis of computational tests of algorithms and heuristics

Statistical analysis is a powerful tool to apply when evaluating the perfor mance of computer implementations of algorithms and heuristics. Yet many co mputational studies in the literature do not use this tool to maximum effec tiveness. This paper examines the types of data that arise in computational comparisons and presents appropriate techniques for analyzing such data se ts. Case studies of computational tests from the open literature are re-eva luated using the proposed methods in order to illustrate the value of stati stical analysis for gaining insight into the behavior of the tested algorit hms.