GENERAL ESTIMATORS FOR THE RELIABILITY OF QUALITATIVE DATA

We study a proportional reduction in loss (PRL) measure for the reliab ility of categorical data and consider the general case in which each of N judges assigns a subject to one of K categories. This measure has been shown to be equivalent to a measure proposed by Perreault and Le igh for a special case when there are two equally competent judges, an d the correct category has a uniform prior distribution. We consider a general framework where the correct category is assumed to have an ar bitrary prior distribution, and where classification probabilities var y by correct category, judge, and category of classification. In this setting, we consider PRL reliability measures based on two estimators of the correct category-the empirical Bayes estimator and an estimator based on the judges' consensus choice. We also discuss four important special cases of the general model and study several types of lower b ounds for PRL reliability.