Variability of estimated variance components and related statistics in a performance assessment

Generalizability theory provides a conceptual and statistical framework for estimating variance components and measurement precision. The theory has b een widely used in evaluating technical qualities of performance assessment s. However, estimates of variance components, measurement error variances, and generalizability coefficients are likely to vary from one sample to ano ther. This study empirically investigates sampling variability of estimated variance components using data collected in several years for a listening and writing performance assessment. This study also evaluates stability of estimated measurement precision from year to year. The results indicated th at the estimated variance components varied from one study to another, espe cially when sample sizes were small. The estimated measurement error varian ces and generalizability coefficients also changed from one year to another . Measurement precision projected by a generalizability study may not be fu lly realized in an actual decision study. The study points out the importan ce of examining variability of estimated variance components and related st atistics in performance assessments.