THE UNPREDICTABILITY PARADOX - REVIEW OF EMPIRICAL COMPARISONS OF RANDOMIZED AND NONRANDOMIZED CLINICAL-TRIALS

Objective To summarise comparisons of randomised clinical trials and n on-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisa tion could not be separated from the effects of other methodological m anoeuvres. Design Systematic review Selection criteria Cohorts or meta -analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect Data source s Cochrane Review Methodology Database, Medline, SciSearch, bibliograp hies, hand searching of journals, personal communication with methodol ogists, and the reference lists of relevant articles. Main outcome mea sures Relation between randomisation and estimates of effect Results E leven studies that compared randomised controlled trials with non-rand omised controlled trials (eight for evaluations of the same interventi on and three across different interventions), two studies that compare d trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relati on between quality scores and estimates of treatment effects, were ide ntified. Failure to use random allocation and concealment of allocatio n were associated with relative increases in estimates of effects of 1 50% or more, relative decreases of up to 90%, inversion of the estimat ed effect and, in some cases, no difference. On average, failure to us e randomisation or adequate concealment of allocation resulted in larg er estimates of effect due to a poorer prognosis in non-randomly selec ted control groups compared with randomly selected control groups. Con clusions Failure to use adequately concealed random allocation can dis tort the apparent effects of care in either direction, causing the eff ects to seem either larger or smaller than they really are. The size o f these distortions can be as large as or larger than the size of the effects that are to be detected.