Inter-rater and intra-rater reliability in the interpretation of MTI photoscreener photographs of native American preschool children

Purpose. To evaluate inter- and intra-rater reliability for the interpretat ion of MTI Photoscreener photographs taken in a population of Native Americ an preschool children with a high prevalence of astigmatism. Methods. Photo graphs of 369 children were rated by 11 nonexpert and 3 expert raters. Phot ographs for each child were scored as pass, refer, or retake. Nonexpert rat ers scored photos on two separate occasions, permitting analysis of intra-r ater reliability. Results. Analyses of pass/refer responses only: inter-rat er reliability was moderate to substantial among nonexpert raters and subst antial among expert raters. Intra-rater reliability among nonexperts was su bstantial. Analyses of all responses (pass, refer, and retake): inter-rater reliability for pass and refer scores was moderate among nonexperts and su bstantial among experts; for retake scores inter-rater reliability was slig ht for nonexperts and moderate for experts. Intra-rater reliability among n onexperts was substantial for pass and refer scores and moderate for retake scores. Conclusions. In this population with a high prevalence of astigmat ism, whether MTI photoscreening results are interpretable is much more vari able among and within raters than whether an interpretable photograph shoul d be scored as pass or refer. The level of agreement among raters in the cu rrent study was influenced by the experience of the raters. In addition, no nexpert raters were move likely to deem a photograph uninterpretable than e xpert raters.