Data based on qualitative judgments are prevalent in both academic res
earch in marketing and applied marketing research. Reliability measure
ment of qualitative data is important to determine the stability and q
uality of the data obtained. The authors assume a decision theoretic l
oss function, formally model the loss to the researcher of using wrong
judgments, and show how this produces a new, proportional reduction i
n loss (PRL) reliability measure that generalizes many existing quanti
tative and qualitative measures. Because the PRL measure is often cumb
ersome to compute directly, they provide reference tables that enable
the researcher to apply their approach easily. They then use this new
approach to explore several important practical issues in conducting m
arketing research with qualitative judgments. In particular, they addr
ess the issues of (1) how reliable qualitative data should be (extendi
ng directly from Nunnally's rule of thumb for Cronbach's alpha in quan
titative measurement), (2) how many judges are necessary given a known
proportion of agreement between judges, and (3) given a fixed number
of judges, what proportion of agreement must be obtained to ensure ade
quate reliability.