SPEC AS A PERFORMANCE EVALUATION MEASURE

System Performance Evaluation Cooperative (SPEC) measures are commonly considered the most effective indicators for evaluating engineering w orkstation performance. Indeed, these measures have gradually become a de facto industry standard for CPU performance evaluation. But do the y truly provide added information value? The authors examine SPEC meas ures with the aim of learning their limitations and the areas in which caution should be exercised. They present results and evaluations of a comprehensive statistical analysis of SPEC rankings published in the SPEC Newsletter over a number of years. The major findings show that several programs in the benchmark suite contribute little additional i nformation to the measures: In fact, the potency profile of the system can be demonstrated by just a few representative programs. Moreover, the method of aggregating the results to create a single measure (or t wo measures) leads to distortion in presenting the overall system perf ormance. In evaluating the system, SPEC focuses on the CPU clock rate and thus may not reflect the performance of the entire system. This su ggests that the added value of the SPEC information is not very high. The authors thus advise that SPEC should be employed cautiously with f ull awareness of its shortcomings.