In general, intraclass correlation coefficients (ICC's) are designed t
o assess consistency or conformity between two or more quantitative me
asurements. They are claimed to handle a wide range of problems, inclu
ding questions of reliability, reproducibility and validity. It is sho
wn that care must be taken in choosing a suitable ICC with respect to
the underlying sampling theory. For this purpose a decision tree is de
veloped. It may be used to choose a coefficient which is appropriate f
or a specific study setting. We demonstrate that different ICC's may r
esult in quite different values for the same data set, even under the
same sampling theory. Other general limitations of ICC's are also addr
essed. Potential alternatives are presented and discussed, and some re
commendations are given for the use of an appropriate method.