Ma. Smith et al., PEER-REVIEW OF THE QUALITY OF CARE - RELIABILITY AND SOURCES OF VARIABILITY FOR OUTCOME AND PROCESS ASSESSMENTS, JAMA, the journal of the American Medical Association, 278(19), 1997, pp. 1573-1578
Context.-Peer assessments have traditionally been used to judge the qu
ality of care, but a major drawback has been poor interrater reliabili
ty. Objectives.-To compare the interrater reliability for outcome and
process assessments in a population of frail older adults and to ident
ify systematic sources of variability that contribute to poor reliabil
ity. Setting.-Eight sites participating in a managed care program that
integrates acute and long-term care for frail older adults. Patients.
A total of 313 frail older adults. Design.-Retrospective review of th
e medical record with 180 charts randomly assigned to 2 geriatricians,
2 geriatric nurse practitioners, or 1 geriatrician and 1 geriatric nu
rse practitioner and 133 charts randomly assigned to either a geriatri
cian or a geriatric nurse practitioner. Main Outcome Measures.-Interra
ter reliabilities for structured implicit judgments about process and
outcomes for overall care and care for each of 8 tracer conditions (eg
, arthritis). Results.-Outcome measures had higher interrater reliabil
ity than process measures. Five outcome measures achieved fair to good
reliability (more than 0.40), while none of the process measures achi
eved reliabilities more than 0.40. Three factors contributed to poorer
reliabilities for process measures: (1) an inability of reviewers to
differentiate among cases with respect to the quality of management, (
2) systematic bias from individual reviewers, and (3) systematic bias
related to the professional training of the reviewer (ie, physician or
nurse practitioner). Conclusions.-Peer assessments can play an import
ant role in characterizing the quality of care for complex patients wi
th multiple interrelated chronic conditions, but reliability can be po
or. Strategies to achieve adequate reliability for these assessments s
hould be applied. These strategies include emphasizing outcomes measur
ement, providing more structured assessments to identify true differen
ces in patient management, adjusting systematic bias resulting from th
e individual reviewer and their professional background, and averaging
scores from multiple reviewers. Future research on the reliability of
peer assessments should focus on improving the ability of process mea
sures to differentiate among cases with respect to the quality of mana
gement and on identifying additional sources of systematic bias for bo
th process and outcome measures, Explicit recognition of factors influ
encing reliability will strengthen efforts to develop sound measures f
or quality assurance.