BIAS IN METAANALYSIS DETECTED BY A SIMPLE, GRAPHICAL TEST

Objective: Funnel plots (plots of effect estimates against sample size ) may be useful to detect bias in meta-analyses that were later contra dicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses ar e compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs cons isting of a meta-analysis and a single large trial (concordance of res ults was assumed if effects were in the same direction and the meta-an alytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading g eneral medicine journals 1993-6 and 38 meta-analyses from the second 1 996 issue of the Cochrane Database of Systematic Reviews. Main outcome measure: Degree of funnel plot asymmetry as measured by die intercept from regression of standard normal deviates against precision. Result s: Ln the eight pairs of meta-analysis and large trial that were ident ified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were f our concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry wa s present in three out of four discordant pairs but in none of concord ant pairs. In 14 (38%)journal meta-analyses and 5 (13%) Cochrane revie ws, funnel plot asymmetry indicated chat there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likel y presence of bias in meta-analyses, but as the capacity to detect bia s will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with con siderable caution.