EVALUATING EXPERT KNOWLEDGE - PLANT-SPECIES RESPONSES TO CATTLE GRAZING AND FIRE

Expert judgment, standardized in a meaningful format, can be used to i dentify research/survey needs and to characterize areas of (dis)agreem ent in species responses, associated traits, and factors affecting res ponses. Feasible methods are needed to facilitate the evaluation of ex pertise in a complex domain characterized by moderate to low learnabil ity. Specific objectives for this study were 1) to evaluate agreement among experts on range plant species behavior and 2) to develop an agr eement-based classification method for plant species responses. Declar ative information at landscape scale was elicited from 7 role-suggeste d experts on expected responses to cattle grazing (none, moderate, or heavy) and fire (absent, applied in late summer or fall, or applied in late winter or spring) of 198 plant species from the Edwards Plateau (Texas). Trends were requested to be assessed in a 3-level ordinal sca le (decrease, unaffected, increase). Kappa statistics (pair-wise and m ulti-rater versions) and log-linear models were used to evaluate agree ment. A procedure based upon cumulative probability distributions of p ossible rating combinations was developed to classify plant species wh ile accounting for agreement, A total of 4,584 opinions (cattle grazin g: 2,959; fire: 1,625) was elicited and analyzed. Low to moderate agre ement was observed. Average pair-wise kappa statistics ranged from 0.0 7 to 0.39; multiple-rater kappa coefficients ranged from -0.17 to 0.53 . Log-linear analyses were consistent with those estimations: agreemen t beyond chance or baseline association between ratings (P < 0.05) was observed in 62 out of 114 possible pair-wise cases. Non-homogeneous m arginal distribution of opinion were an important source of disagreeme nt. Experts performed beyond chance expectations in all scenarios but agreement was better land pattern of agreement more consistent) when s cenarios were most familiar to the experts (e.g., heavy grazing and wi nter/spring burning). Almost 80% of species was classified beyond chan ce (P < 0.15) in grazing scenarios in contrast to only 40 to 60% in fi re scenarios. This resulted from less agreement among experts but also from apparent lack of knowledge. The procedure developed to classify plant species provides an objective criterion for evaluating agreement in an ordinal scale. Graphical representations facilitate understandi ng of relationships between the number of expert sources and their abi lity to distinguish among classes for a pre-defined confidence level.