PRECIS OF STATISTICAL SIGNIFICANCE - RATIONALE, VALIDITY, AND UTILITY

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the fo llowing contrasts: (a) substantive hypotheses versus statistical hypot heses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiment s versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is th e hypothesis that errors are randomly distributed in data. Moreover, t he null hypothesis is never used as a categorical proposition. Statist ical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor res ponsible. The experimental conclusion is drawn with the inductive prin ciple underlying the experimental design. A chain of deductive argumen ts gives rise to the theoretical conclusion via the experimental concl usion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than r eal. The absolute size of the effect is not an index of evidential sup port for the substantive hypothesis. Nor is the effect size, by itself , informative as to the practical importance of the the research resul t. Being a conditional probability, statistical power cannot be the a priori probability of statistical significance. The validity of statis tical power is debatable because statistical significance is determine d with a single sampling distribution of the test statistic based on H -0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect siz e, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of t hem fulfill the nonstatistical functions expected of them by critics.