Xh. Gao et Rl. Brennan, Variability of estimated variance components and related statistics in a performance assessment, APPL MEAS E, 14(2), 2001, pp. 191-203
Generalizability theory provides a conceptual and statistical framework for
estimating variance components and measurement precision. The theory has b
een widely used in evaluating technical qualities of performance assessment
s. However, estimates of variance components, measurement error variances,
and generalizability coefficients are likely to vary from one sample to ano
ther. This study empirically investigates sampling variability of estimated
variance components using data collected in several years for a listening
and writing performance assessment. This study also evaluates stability of
estimated measurement precision from year to year. The results indicated th
at the estimated variance components varied from one study to another, espe
cially when sample sizes were small. The estimated measurement error varian
ces and generalizability coefficients also changed from one year to another
. Measurement precision projected by a generalizability study may not be fu
lly realized in an actual decision study. The study points out the importan
ce of examining variability of estimated variance components and related st
atistics in performance assessments.