The effect of a 'don't know' option on test scores: number-right and formula scoring compared

Objectives In multiple-choice tests using a 'don't-know' option the number of correct minus incorrect answers was used as the test score (formula scor ing) in order to reduce the measurement error resulting from random guessin g. In the literature diverging results are reported when comparing formula scoring and number-fight scoring, the scoring method without the don't-know option. Design To investigate which method was most appropriate, both scoring metho ds were used in true-false tests (block tests) taken at the end of a second - and third-year educational module (block). The students were asked to ans wer each item initially by choosing from the response options true, false o r don't know, and secondly to replace all don't-know answers by a true-fals e answer. Setting Maastricht University, The Netherlands. Subjects Medical students. Results The correct scores for the don't-know answered items were found to be 4.5% and 5.9%, respectively, higher than expected with pure random guess work. This represents a source of bias with formula scoring, because studen ts who were less willing to guess obtained lower scores. The average differ ence in the correct minus incorrect score for the two scoring methods (2.5% , P < 0.001, and 3.4%, P < 0.001, respectively) indicates the size of the b ias (compare: the standard deviation of the score equals 11%). Test reliabi lity was higher with formula scoring (0.72 vs. 0.66 and 0.74 vs. 0.66), but the difference decreased when the test was restricted to items which were close to the core content of the block (0.81 vs. 0.77, resp. 0.75 vs. 0.70) . Conclusions In deciding what scoring method to use, less bias (number-right scoring) has to be weighed against higher reliability (formula scoring). A part from these psychometric reasons educational factors must be considered .