Methods for estimating areas under receiver-operating characteristic curves: illustration with somatic-cell scores in subclinical intramammary infections

The aim of this study was to demonstrate receiver-operating characteristic (ROC) methodology in the context of bovine intramammary infection (IMI). Qu arter somatic cell scores (SCS) were available to evaluate quarter IMI, and the final IMI diagnosis was made from milk bacteriologic cultures. Data co nsisted of 11,453 quarter-milk samples collected on 2084 clinically healthy cows located in 154 Belgian herds. Bacteriological analyses showed 16.2%, 7.2%, and 11.9% of quarters infected with coagulase-positive Staphylococcus spp,, Streptococcus agalactiae, and coagulase-negative Staphylococcus spp. , respectively, The ROC curve indicated all the combinations of sensitivity and specificity that quarter SCS was able to provide as a test to identify quarter IMI. Among parametric, semi-parametric, and non-parametric methods to estimate area under ROC curves, the parametric method seemed the least appropriate for analyzing SCS in this study. With the non-parametric method , the total area under the ROC curves showed quarter SCS could identify qua rter IMI with an overall accuracy of 69%, 76%, and 59% for coagulase-positi ve Staphylococcus spp., S. agalactiae, and coagulase-negative Staphylococcu s spp., respectively. Parametric and nonparametric statistical tests showed that overall SCS diagnostic capability was significantly (p<0.01) differen t from chance and was different (p<0.01) across the three bacteria. However , the SCS thresholds yielding the highest percentage of quarters correctly classified as infected (for the observed prevalence and for equal costs ass igned to false-positive and false-negative results) were so high that they had no practical value. The major advantage of ROC analysis is the comprehe nsive description of the discrimination capacity of SCS for all possible ch oices of critical values. The major disadvantage is the dependency upon the gold standard used for the final diagnosis - but recent improvements of th e methodology will correct the problem. (C) 1999 Elsevier Science B.V. All rights reserved.