Assessing the quality of randomized trials: Reliability of the Jadad scale

An instrument was developed and validated by Jadad, et al. to assess the qu ality of clinical trials using studies from the pain literature. Our study determined the reliability of the Jadad scale and the effect of blinding on interrater agreement in another group of primary studies. Four raters inde pendently assessed blinded and unblinded versions of 76 randomized trials. Interrater agreement was calculated among combinations of four raters for b linded and unblinded versions of the studies. A 4 x 2 x 2 repeated measures design was employed to evaluate the effect of blinding. The interrater agr eement for the Jadad scale was poor (kappa 0.37 to 0.39), but agreement imp roved substantially (kappa 0.53 to 0.59) with removal of the third item (an explanation of withdrawals). Blinding did not significantly affect the Jad ad scale scores. A more precise description of how to score the withdrawal item and careful conduct of a practice set of articles might improve interr ater agreement. In contrast with the conclusions reached by Jadad, we were unable to demonstrate a significant effect of blinding on the quality scare s. (C) Elsevier Science Inc. 1999.