ASSESSMENT OF THE VALIDITY AND RELIABILITY OF 3 SYSTEMS OF MEDICAL RECORD SCREENING FOR QUALITY OF CARE ASSESSMENT

OBJECTIVES, The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quali ty of care of patients undergoing cardiovascular procedures: (1) the U niform Clinical Data Set System (UCDSS), a software-based system devel oped by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) scr eening criteria. METHODS. A random sample of 451 medical records of in patients who underwent cardiovascular procedures were screened by trai ned nurses and were also examined by physicians masked to screening re sults. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. agreement between pairs of physician reviews about the quality of care was fair or slig ht (Kappa < 0.40). Agreement between nurses about presence of any scre ening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), an d fair for HMPS (0.41). The UCDSS was more sensitive to quality proble ms agreed on by two physicians than HCFA or HMPS (85%, 74% and 69%, re spectively), but was also less specific (48%, 57%, and 56%, respective ly). The UCDSS was much more labor-intensive with an average of 6.7 ho urs of nurse review per quality problem disclosed versus 2 hours requi red by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brou ght about substantial improvements in reliability of medical record sc reening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quali ty improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide int erventions.