Study objective: To measure the interrater and intrarater agreement of
existing emergency department triage systems. Methods: This 2-phase e
xperimental study of triage nurses and EMTs' ratings for 5 scripted pa
tient scenarios used in-person interviews and follow-up written survey
s. Results: Eighty-seven participant (>90% of those eligible) with 55
(63%) completed both phases of the study. Interrater agreement on tria
ge category was poor (kappa=.347 overall). Only 13 of 55 (24%) partici
pants rated the 5 cases the same severity in both phases; Kendall corr
elation (tau-B) comparing phases 1 and 2 varied from .145 to .554. Est
imates of admission probability varied widely. Estimates of the approp
riate time to physician evaluation (from immediate to 24 hours) was of
ten incongruous with severity ratings leg, 54% of those participants r
ating a case the lowest severity recommended evaluation within 8 hours
). There was good agreement on estimated need for an ED monitored bed
or diagnostic studies. Conclusion: Triage assessments (both interrater
and intrarater) by experienced personnel are inconsistent using these
5 standardized patient scenarios. These results challenge the reliabi
lity of current ED triage practice.