Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytical scoring methods

Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores de rived from various holistic and analytical scoring strategies reported in t he literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the s imultaneous estimation of multiple sources of error variance to estimate th e reliability of test scores. Using GT, two studies were conducted to inves tigate the impact of the number of raters and the type of decision (relativ e vs. absolute) on the reliability of writing scores. The results of both s tudies indicated that the reliability coefficients for writing scores decli ne as (a) the number of raters is reduced and (b) when absolute decisions r ather than relative decisions are made.