Implications of test dimensionality for unidimensional IRT scoring: An investigation of a high-stakes testing program

Determining whether a test violates the assumption of unidimensionality is an important precursor to item response theory (IRT) analysis. However, a t est's unidimensionality or nonunidimensionality may be a matter of degree, and the implications of the degree of nonunidimensionality may depend on ho w the test is analyzed and how the results are to be used. This study exami ned the dimensionality of a high-stakes graduate training selection test an d the implications of the test's dimensionality for the IRT calibration and scoring of each section of the test. The dimensionality analyses suggested that, although the items within each of the sections were not completely h omogeneous, neither were they clearly measuring distinct constructs corresp onding to the content disciplines. The correlations between student scores based on item parameters that were estimated separately within discipline a nd then formed into weighted composites and scores based on item parameters that were estimated across discipline (within section) exceeded .99.