A MATRIX OF KAPPA-TYPE COEFFICIENTS TO ASSESS THE RELIABILITY OF NOMINAL SCALES

To investigate the reliability of nominal scales, Kraemer proposed a m easurement model from which kappa coefficients could be derived. More recently she suggested a matrix of coefficients as a comprehensive sum mary of reliability, contrasting this with use of a single summary kap pa statistic. The main diagonal of the matrix consists of binary kappa coefficients for each category which measure the reliability of each category relative to all others, while the off-diagonal elements are c orrelation coefficients for pairs of categories. The off-diagonal elem ents were suggested as measures of confusion between categories. Schou ten also suggested coefficients to assess confusion between pairs of c ategories, which might be used as alternative off-diagonal elements in a summary matrix. The two types of off-diagonal element will be compa red. It will be shown that Schouten's coefficients can be expressed in terms of the parameters of Kraemer's measurement model and that they are more easily interpreted as measures of confusion. First, the maxim um value for Schouten's coefficient is one. Secondly, for any pair of categories, Schouten's coefficient equals the proportionate reduction in the probability of classifying a subject in one category of the pai r having previously classified them in the other. Thirdly, where the c oefficient for a pair of categories is less than the summary kappa sta tistic, it will be shown that combining these two categories will incr ease the value of the summary kappa statistic. The methods of analysis are applied to data from a study of the reliability of psychiatric di agnosis and used to identify pairs of classifications between which th ere is substantial confusion. (C) 1998 John Wiley & Sons, Ltd.