Performance of the score systems Acute Physiology and Chronic Health Evaluation II and III at an interdisciplinary intensive care unit, after customization

Background: Mortality predictions calculated using scoring scales are often not accurate in populations other than those in which the scales were deve loped because of differences in case-mix. The present study investigates th e effect of first-level customization, using a logistic regression techniqu e, on discrimination and calibration of the Acute Physiology and Chronic He alth Evaluation (APACHE) II and III scales. Method: Probabilities of hospital death for patients were estimated by appl ying APACHE II and III and comparing these with observed outcomes. Using th e split sample technique, a customized model to predict outcome was develop ed by logistic regression. The overall goodness-of-fit of the original and the customized models was assessed. Results: Of 3383 consecutive intensive care unit (ICU) admissions over 3 ye ars, 2795 patients could be analyzed, and were split randomly into developm ent and validation samples. The discriminative powers of APACHE II and III were unchanged by customization (areas under the receiver operating charact eristic [ROC] curve 0.82 and 0.85, respectively). Hosmer-Lemeshow goodness- of-fit tests showed good calibration for APACHE II, but insufficient calibr ation for APACHE III. Customization improved calibration for both models, w ith a good fit for APACHE III as well. However, fit was different for vario us subgroups. Conclusions: The overall goodness-of-fit of APACHE III mortality prediction was improved significantly by customization, but uniformity of fit in diff erent subgroups was not achieved. Therefore, application of the customized model provides no advantage, because differences in case-mix still limit co mparisons of quality of care.