Limitations of statistical design of experiments approaches in engineeringtesting

A hypothetical experiment and Monte Carlo simulations were used to examine the effectiveness of statistical design of experiments methods in identifyi ng from the experimental data the correct terms in postulated regression mo dels for a variety of experimental conditions. Two analysis of variance tec hniques (components of variance and pooled mean square error) combined with F-test statistics were investigated with first-order and second-order regr ession models. It was concluded that there are experimental conditions for which one or the other of the procedures results in model identification wi th high confidence, but there are also other conditions in which neither pr ocedure is successful. The ability of the statistical approaches to identif y the correct models varies so drastically, depending on experimental condi tions, that it seems unlikely that arbitrarily choosing a method and applyi ng it will lend to identification of the effects that are significant with a reasonable degree of confidence. It is concluded that before designing an d conducting an experiment, one should use simulations of the proposed expe riment with postulated truths in order to determine which statistical desig n of experiments approach, if any will identify the correct model from the experimental data with an acceptable degree of confidence. In addition, no significant change in the effectiveness of the methods in identifying the c orrect model was observed when systematic uncertainties of up to 10 percent in the independent variables and in the response were introduced into the simulations. An explanation is that the systematic errors in the simulation data caused a shift of the whole response surface up or down from the true value, without a significant change in shape. [S0098-2202(00)03102-3].