Valid inference in random effects meta-analysis

The standard approach to inference for random effects meta-analysis relies on approximating the null distribution of a test statistic by a standard no rmal distribution. This approximation is asymptotic on k, the number of stu dies, and can be substantially in error in medical meta-analyses, which oft en have only a few studies. This paper proposes permutation and ad hoc meth ods for testing with the random effects model. Under the group permutation method, we randomly switch the treatment and control group labels in each t rial. This idea is similar to using a permutation distribution for a commun ity intervention trial where communities are randomized in pairs. The permu tation method theoretically controls the type I error rate for typical meta -analyses scenarios. We also suggest two ad hoc procedures. Our first sugge stion is to use a t-reference distribution with k - 1 degrees of freedom ra ther than a standard normal distribution for the usual random effects test statistic. We also investigate the use of a simple t-statistic on the repor ted treatment effects.