On judging the significance of differences by examining the overlap between confidence intervals

To judge whether the difference between two point estimates is statisticall y significant, data analysts often examine the overlap between the two asso ciated confidence intervals. We compare this technique to the standard meth od of testing significance under the common assumptions of consistency, asy mptotic normality, and asymptotic independence of the estimates. Rejection of the null hypothesis by the method of examining overlap implies rejection by the standard method, whereas failure to reject by the method of examini ng overlap does not imply failure to reject by the standard method. As a co nsequence, the method of examining overlap is more conservative (i.e., reje cts the null hypothesis less often) than the standard method when the null hypothesis is true, and it mistakenly fails to reject the null hypothesis m ore frequently than does the standard method when the null hypothesis is fa lse. Although the method of examining overlap is simple and especially conv enient when lists or graphs of confidence intervals have been presented, we conclude that it should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information n eeded to carry out a more appropriate procedure is unavailable.