Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis

PURPOSE: This study estimates the inter-rater and test-retest reliability o f Chalmers' quality scare scale in the context of bone mass loss and fractu re rate in postmenopausal women. METHODS: An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss an d fracture rate in postmenopausal women. Twenty articles were randomly sele cted and four raters independently assessed the quality of each article wit h Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names , journal, year of publication and source of funding. Raters were also aske d to assess all 20 articles one more time, two months after the first evalu ation. Intraclass (ICC) and test-retest correlation coefficients were calcu lated. RESULTS: The overall inter-rater ICC was 0.66 [0.55, 0.79](95%). The overal l test-retest reliability of Chalmers' scale was 0.81 [0.67, 0.98](95%) Whe n ratings were stratified according to articles' blinding status, blinded a ssessments generated a smaller inter-rater ICC than non blinded assessments : 0.30 [0.17, 0.53](95%) VS. 0.80 [0.71, 0.90](95%) In addition, analyzing sub-scales separately generated different estimates of reliability. CONCLUSIONS: This study shows that the reliability of the quality scale dev eloped by Chalmers substantially varies between sub-scales, and is highly d ependent on articles' blinding status. The possibility of bias in rating no n-blinded articles can not be ruled out. The reliability of the scale can a lso be dependent on the outcome studied. (C) 2000 Elsevier Science Inc. All rights reserved.