Replicating effects and biases

One corm-non view is that a well-designed empirical study will reach conclu sions that can be found again and again if the study is replicated, whereas a poorly designed study is unlikely to replicate. In opposition to this vi ew, it is argued that a well-designed empirical study reaches conclusions t hat tend to replicate when correct and are less likely to replicate when in correct, whereas in a poorly designed study the conclusions tend to replica te even when incorrect. In observational studies of treatment effects, the same hidden bias may occur repeatedly in a series of studies, so the studie s reproduce the same distorted estimates of treatment effects. The purpose of this article is to point to strategies in research design that make it l ess likely that biased estimates will replicate, and to illustrate these st rategies with examples. To replicate effects without replicating biases, va ry the treatment assignment mechanism, so that the reasons subjects are spa red treatment are different in the original and replicated studies. Also, v ary the treatment envelope-that is, the ostensibly irrelevant features in w hich the treatment is packaged.