An evaluation of homogeneity tests in meta-analyses in pain using simulations of individual patient data

In this paper we consider the validity and power of some commonly used stat istics for assessing the degree of homogeneity between trials in a meta-ana lysis. We show, using simulated individual patient data typical of that occ urring in randomized controlled trials in pain, that the most commonly used statistics do not give the expected levels of statistical significance (i. e. the proportion of trials giving a significant result is not equal to the proportion expected due to random chance) when used with truly homogeneous data. In addition, all such statistics are shown to have extremely low pow er to detect true heterogeneity even when that heterogeneity is very large. Since, in most practical situations, failure to detect heterogeneity does not allow us to say with any helpful degree of certainty that the data is t ruly homogeneous, we advocate the quantitative combination of results only where the trials contained in a meta-analysis can be shown to be clinically homogeneous. We propose as a definition of clinical homogeneity that all t rials have (i) fixed and clearly defined inclusion criteria and (ii) fixed and clearly defined outcomes or outcome measures. In pain relief, for examp le, the first of these would be satisfied by all patients having moderate o r severe pain, whilst the second would be satisfied by using at least 50% p ain relief as the successful outcome measure. (C) 2000 International Associ ation for the Study of Pain. Published by Elsevier Science B.V. All rights reserved.