Assessing and testing interrater agreement on a single target using multi-item rating scales

An index for assessing interrater agreement with respect to a single target using a multi-item rating scale is developed. The variance of rarer mean s cale scores is used as the numerator of the agreement index. Four variants of a disattenuated agreement index that vary in the random response term us ed as the denominator are examined. A variant of r'(wg( J )) using perfectl y collinear, uniformly distributed responses as the baseline is recommended as a useful index of disattenuated agreement. A test of the equality of va riances is recommended for deleting items with a level of interrater agreem ent significantly different from zero but tower than that of the remaining items.