We study a proportional reduction in loss (PRL) measure for the reliab
ility of categorical data and consider the general case in which each
of N judges assigns a subject to one of K categories. This measure has
been shown to be equivalent to a measure proposed by Perreault and Le
igh for a special case when there are two equally competent judges, an
d the correct category has a uniform prior distribution. We consider a
general framework where the correct category is assumed to have an ar
bitrary prior distribution, and where classification probabilities var
y by correct category, judge, and category of classification. In this
setting, we consider PRL reliability measures based on two estimators
of the correct category-the empirical Bayes estimator and an estimator
based on the judges' consensus choice. We also discuss four important
special cases of the general model and study several types of lower b
ounds for PRL reliability.