The hazards of scaring the quality of clinical trials for meta-analysis

Context Although it is widely recommended that clinical trials undergo some type of quality review, the number and variety of quality assessment scale s that exist make it unclear how to achieve the best assessment. Objective To determine whether the type of quality assessment scale used af fects the conclusions of meta-analytic studies. Design and Setting Meta-analysis of 17 trials comparing low-molecular-weigh t heparin (LMWH) with standard heparin for prevention of postoperative thro mbosis using 25 different scales to identify high-quality trials. The assoc iation between treatment effect and summary scores and the association with 3 key domains (concealment of treatment allocation, blinding of outcome as sessment, and handling of withdrawals) were examined in regression models. Main Outcome Measure Pooled relative risks of deep vein thrombosis with LMW H vs standard heparin in high-quality vs low-quality trials as determined b y 25 quality scales. Results Pooled relative risks from high-quality trials ranged from 0.63 (95 % confidence interval [CI], 0.44-0.90) to 0.90 (95% CI, 0.67-1.21) vs 0.52 (95% CI, 0.24-1.09) to 1.13 (95% CI, 0.70-1.82) for low-quality trials. For 6 scales, relative risks of high-quality trials were close to unity, indic ating that LMWH was not significantly superior to standard heparin, whereas low-quality trials showed better protection with LMWH (P<.05). Seven scale s showed the opposite: high quality trials showed an effect whereas low qua lity trials did not. For the remaining 12 scales, effect estimates were sim ilar in the 2 quality strata. In regression analysis, summary quality score s were not significantly associated with treatment effects. There was no si gnificant association of treatment effects with allocation concealment and handling of withdrawals. Open outcome assessment, however, influenced effec t size with the effect of LMWH, on average, being exaggerated by 35% (95% C I, 1%-57%; P=.046). Conclusions Our data indicate that the use of summary scores to identify tr ials of high quality is problematic. Relevant methodological aspects should be assessed individually and their influence on effect sizes explored.