Prophesying the reliability of cognitively complex assessments

Traditional approaches, including the Spearman-Brown Prophecy Formula and g eneralizability theory, are unlikely to predict accurately the impact of te st length on the internal consistency of a cognitively complex assessment b ecause cognitively complex tasks are unlikely to be parallel or homogeneous . In addition, traditional approaches are limited to predicting impact of t est length on the internal consistency of an assessment. At times, a test s pecialist may wish to predict the impact of changes in the characteristics of the test takers on the internal consistency of an assessment. In this ar ticle, we present an approach that can be used to predict internal consiste ncy for cognitively complex assessments along 2 dimensions. First, we prese nt an approach to predict the internal consistency of a measurement procedu re after adding tasks that require the same test takers to use similar or d ifferent solution strategies than the original tasks. Second, we present an approach to predict the internal consistency of a measurement procedure af ter adding test takers who use different solution strategies to solve the s ame tasks than the original test takers. Finally, we use data from the 1992 National Assessment of Educational Progress mathematics assessment to illu strate estimating internal consistency after adding tasks or test takers.