RELIABILITY-MEASURES FOR QUALITATIVE DATA - THEORY AND IMPLICATIONS

Data based on qualitative judgments are prevalent in both academic res earch in marketing and applied marketing research. Reliability measure ment of qualitative data is important to determine the stability and q uality of the data obtained. The authors assume a decision theoretic l oss function, formally model the loss to the researcher of using wrong judgments, and show how this produces a new, proportional reduction i n loss (PRL) reliability measure that generalizes many existing quanti tative and qualitative measures. Because the PRL measure is often cumb ersome to compute directly, they provide reference tables that enable the researcher to apply their approach easily. They then use this new approach to explore several important practical issues in conducting m arketing research with qualitative judgments. In particular, they addr ess the issues of (1) how reliable qualitative data should be (extendi ng directly from Nunnally's rule of thumb for Cronbach's alpha in quan titative measurement), (2) how many judges are necessary given a known proportion of agreement between judges, and (3) given a fixed number of judges, what proportion of agreement must be obtained to ensure ade quate reliability.