The randomized controlled clinical trial is an increasingly used metho
d in health services research. Analysis of methodology is needed to ac
celerate practical implementation of trial results, select trials for
meta-analysis, and improve trial quality in health services research.
The objectives of this study are to explore the methodology of health
services research trials, create and validate a streamlined quality ev
aluation tool, and identify frequent quality defects and confounding e
ffects on quality. The authors developed a quality questionnaire that
contained 20 evaluation criteria for health services research trials.
One hundred one trials from the Columbia Registry of Controlled Clinic
al Trials were evaluated using the new quality tool. The overall agree
ment between independent reviewers, Cohen's kappa, was 0.94 (+/-0.01).
Of a possible score of 100, the trials received an average score of 5
4.8 (+/-12.5). Five evaluation criteria indicated significant quality
deficiencies (sample size, description of case selection, data on poss
ible adverse effects, analysis of secondary effect variables, and retr
ospective analysis). The quality of study characteristics was signific
antly weaker than the quality of reporting characteristics (P < 0.001)
. The total average scores of Medline-indexed journals were better tha
n the non-Medline-indexed journals (P < 0.001). There was a positive c
orrelation between the overall quality and year of publication (R = 0.
21, P < 0.05). The authors conclude that the new quality evaluation to
ol leads to replicable results and there is an urgent need to improve
several study characteristics of clinical trials. In comparison to dru
g trials, site selection, randomization, and blinding often require di
fferent approaches in health services research.