Using portfolios in program evaluation: An investigation of interrater reliability

Portfolios and other open-ended assessments are increasingly incorporated I nto evaluations and testing programs. However, questions about the reliabil ity of such assessments continue to be raised. After reviewing forces that may be leading to increased interest in and use of portfolio assessment, we investigate the interrater reliability of a portfolio assessment used in a small-scale program evaluation. Three types of portfolio scores were inves tigated-analytic, combined analytic (formed by summing across analytic scor es), and holistic. The interrater reliability coefficient was highest for s ummed analytic scores (r = .86). Results indicate that at least three rater s are required to obtain acceptable levels of reliability for holistic and individual analytic scores.