SOME STATISTICAL AND LOGICAL CONSIDERATIONS WHEN RESCORING TESTS

When tests or portions of tests are scored subjectively by raters, a r escoring will yield a change in the ratings of some examinees. In a te st with a fixed passing score a rescoring will result in the change of some pass/fail decisions. The number of changes depends on: the relia bility of the rating system, the number of raters, the variability in examinee abilities, the proportion of examinees that initially pass, a nd the policy used to incorporate the rescore into the pass/fail decis ion. In this study, we provide a model that facilitates the evaluation of various rescoring strategies. We consider and compare the efficien cy of three rescoring strategies: (1) rescore everyone, (2) rescore fa ilures only, and (3) rescore within some range of the passing cutoff. These rescoring strategies are evaluated by direct simulation. Additio nally we consider the optimal allocation of rescores where the probabi lity someone asks to be rescored is inversely proportional to the dist ance from their initial score to the cutscore. This allocation is eval uated via numerical integration. A further generalization of the basic model is also considered in which a test is comprised of a mixture of objectively and subjectively scored items.