An empirical comparison of methodologies for obtaining results with specific accuracy and for run-length control in quantitative simulation

Any simulation in which simulated events are functions of(pseudo) random nu mbers is simply a statistical experiment. Thus any performance measures obt ained by simulation are estimates, and one must consider the precision of t he estimates before any constructive conclusions about the investigated sys tems or networks are made. To assist simulation practitioners in generating confidence intervals for point estimates, a number of output analysis and run-length control methodologies have been proposed. Given the diversity of these methods, a critical question to the simulation user is which of the various output analysis methods to select. This paper presents the results of an in-depth empirical comparison of methods of precision control, select ed by us as potential candidates for implementation in a software package f or fully automating data analysis and run-length control during simulation run time. The main measures considered were coverage (which is a measure of the accuracy of the method, defined as the probability that the confidence interval produced by the method contains the true value of the estimated p arameter), required run length, computational complexity and memory usage. Results were obtained from 20,000 benchmark experiments per method. Practic al implications are discussed.