The effect of errors in estimating ability on goodness-of-fit tests for IRT models

Assessing goodness of fit of item response theory models typically involves evaluating differences between observed and expected score response distri butions using a chi-square test statistic. When these methods are applied t o assessments that art shorter in length, uncertainty with which ability is estimated greatly affects the approximation to the null chi-square distrib ution. Results from a Monte Carlo study indicated serious departures betwee n null theoretical distributions and empirically derived sampling distribut ions for the chi-square statistic for tests with 8 and 16 constructed respo nse items. This article also describes a fit statistic that attempts to acc ount for the uncertainty in estimating ability and that could therefore be applied to testing situations in which ability is not precisely estimated. This method employs more information from the same distribution used to obt ain Bayesian point estimates of ability and reflects probabilities that exa minees have ability equal to a range of values rather than restricting expe ctations to single values.