Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?

Background The inclusion of only a subset of all available evidence in a me ta-analysis may introduce biases and threaten its validity; this is particu larly likely if the subset of included studies differ from those not includ ed, which may be the case for published and grey literature (unpublished st udies, with limited distribution). We set out to examine whether exclusion of grey literature, compared with its inclusion in meta-analysis, provides different estimates of the effectiveness of interventions assessed in rando mised trials. Methods From a random sample of 135 meta-analyses, we identified and retrie ved 33 publications that included both grey and published primary studies. The 33 publications contributed 41 separate meta-analyses from several dise ase areas. General characteristics of the meta-analyses and associated stud ies and outcome data at the trial level were collected. We explored the eff ects of the inclusion of grey literature on the quantitative results using logistic-regression analyses. Findings 33% of the meta-analyses were found to include some form of grey l iterature. The grey literature, when included, accounts for between 4.% and 75% of the studies in a meta-analysis. On average, published work, compare d with grey literature, yielded significantly larger estimates of the inter vention effect by 15% (ratio of odds ratios=1.15 [95% CI 1.04-1.28]). Exclu ding abstracts from the analysis further compounded the exaggeration (1.3 [ 1.0-1.60]). Interpretation The exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-a nalysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria.