Issues in measuring reliability - Correlation versus percentage of agreement

In many literacy studies, if is important to establish the reliability of i ndependent observers' judgements. Reliability most commonly is measured eit her by the percentage of agreement or the correlation between the observers ' judgments. This article argues that the percentage of agreement measure i s more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automa tically by the percentage of agreement measures and Ib) rates of chance agr eement are strongly influenced by the variability of the data, by "ceiling" and "floor" effects, and by the scoring of near agreement as perfect agree ment. For these reasons, the authors recommend that the field of literacy r esearch adopt correlation as the standard method for estimating the reliabi lity of observers' judgements.