Psychometric and cognitive functioning of an under-determined computer-based response type for quantitative reasoning

We evaluated a computer-delivered response type for measuring quantitative skill. "Generating Examples" (GE) presents under-determined problems that c art have many right answers. We administered two GE tests that differed in the manipulation of specific item features hypothesized to affect difficult y. Analyses related to internal consistency reliability, external relations , and features contributing to item difficulty, adverse impact, and examine e perceptions, Results showed that GE scores were reasonably reliable but o nly moderately related to the GRE quantitative section, suggesting the two tests might be tapping somewhat different skills. Item features that increa sed difficulty included asking examinees to supply more than one correct an swer and to identify whether an item was solvable. Gender differences were similar to those found on the GRE quantitative and analytical test sections . Finally, examinees were divided on whether GE items were a fairer indicat or of ability than multiple-choice items, but still overwhelmingly preferre d to take the more conventional questions.