Procedures are developed and compared for testing the equality of two depen
dent kappa statistics in the case of two raters and a dichotomous outcome v
ariable. Such problems may arise when each of a sample of subjects are rate
d under two distinct settings, and it is of interest to compare the observe
d levels of inter-observer and intra-observer agreement. The procedures com
pared are extensions of previously developed procedures for comparing kappa
statistics computed from independent samples. The results of a Monte Carlo
simulation show that adjusting for the dependency between samples tends to
be worthwhile only if the between-setting correlation is comparable in mag
nitude to the within-setting correlations, In this case, a goodness-of-fit
procedure that takes into account the dependency between samples is recomme
nded. Copyright (C) 2000 John Wiley & Sons, Ltd.