EVALUATING MEASUREMENT VARIABILITY IN CLINICAL INVESTIGATIONS - THE CASE OF ULTRASONIC ESTIMATION OF URINARY-BLADDER VOLUME

Objective To evaluate the quality of studies seeking to establish meas urement properties (reliability and validity) of ultrasonic estimation of urinary bladder volume. Design Online searching of the MEDLINE dat abase between 1966 and 1995, and scanning of bibliography of known stu dies on ultrasonic bladder volume estimation. Study selection and stud y quality assessment were performed independently by two reviewers. Ea ch article was evaluated for suitability of the reference standard, ad equacy of reported blinding of the observers and appropriateness of th e statistical index of concordance. The last two of these guidelines w ere applied to reliability studies (evaluating the relation among obse rved ultrasonic estimations), and all three guidelines were applied to validity studies (evaluating the relation of ultrasonic estimation wi th a definitive measurement). Population One hundred and twenty-five p articipants enrolled in the five reliability studies and 769 participa nts in the 27 validity studies selected for appraisal of their quality . Main outcome measure Rate of study compliance with preset criteria f or high quality. Results None of the studies complied with all of the criteria for high methodologic quality. In the five reliability studie s, investigators did not report adequate blinding of observers in thre e (60%) and an appropriate index of reliability was not used in any. A mong the 27 validity studies, there was a lack of a suitable reference standard in 6 (22%), an inadequate blinding in 25 (93%), and an inapp ropriate index of validity in all (100%). Conclusion Based on our guid elines for quality assessment, a large proportion of studies on measur ement properties was found to have inadequate methods, raising concern about the credibility of the reliability and validity estimates repor ted. These deficiencies highlight the lack of rigour employed in the d esign, conduct and analysis of reliability and validity studies, which has the potential for leading to patient mismanagement due to biases in the assessment of measurement variability in clinical investigation s.