ASSIGNMENT METHODS IN EXPERIMENTATION - WHEN DO NONRANDOMIZED EXPERIMENTS APPROXIMATE ANSWERS FROM RANDOMIZED EXPERIMENTS

This meta-analysis compares effect size estimates from 51 randomized e xperiments to those from 47 nonrandomized experiments. These experimen ts were drawn from published and unpublished studies of Scholastic Apt itude Test coaching, ability grouping of students within classrooms, p resurgical education of patients to improve postsurgical outcome, and drug abuse prevention with juveniles. The raw results suggest that the two kinds of experiments yield very different answers. But when studi es are equated for crucial features (which is not always possible), no nrandomized experiments can yield a reasonably accurate effect size in comparison with randomized designs. Crucial design features include t he activity level of the intervention given the control group, pretest effect size, selection and attrition levels, and the accuracy of the effect-size estimation method. Implications of these results for the c onduct of meta-analysis and for the design of good nonrandomized exper iments are discussed.