What should be included in meta-analyses? An exploration of methodologicalissues using the ISPOT meta-analyses

Objective: To explore the impact of methodologic issues on the results of m eta-analyses. The following issues were examined: the type of literature se arch strategy used; inclusion or exclusion of non-peer-reviewed studies; th e inclusion or exclusion of non-English language publications; the effect o f trial quality; and the inclusion or exclusion of non-placebo-controlled s tudies. Methods: The International Study of Perioperative Transfusion (ISPOT) meta- analyses were used to evaluate each of the methodologic issues. The 10 meta -analyses consisted of technologies to reduce the need for perioperative re d blood cell transfusion. The number of trials for each of the meta-analyse s varied from 2 to 45. Both EMBASE and MEDLINE searches were conducted, inc luding the use of systematic search strategies. Results: MEDLINE identified the vast majority of trials. Alone, MEDLINE wou ld have missed 8 studies compared to 10 for EMBASE. Use of the systematic s earch strategies greatly reduced the number of articles to be reviewed comp ared to open searches. Type of publication, country of study origin, inclus ion of non-English publications, and trial quality had very little impact o n the estimates of effect. The use of placebo versus open-label control aff ected the magnitude of the odds ratio for two of the meta-analyses. The res ults of the two meta-analyses were not statistically significant if only pl acebo-controlled trials were included. Conclusions: While methodologic issues had very little impact on the ISPOT meta-analyses, further studies are needed in a variety of other clinical se ttings. Because MEDLINE, coupled with a review of the references in the ide ntified trials, identified the vast majority of trials, one needs to consid er the costs and benefits of searching EMBASE and the pursuance of unpublis hed and unindexed trials.