AN EMPIRICAL-STUDY ON THE NATURE OF TRICK TEST QUESTIONS

This study attempted to better define trick questions and see if stude nts could differentiate between trick and not-trick questions. Phase 1 elicited definitions of trick questions so as to identify essential c haracteristics. Seven components were found. Phase 2 obtained ratings to see which components of trick questions were considered to be most crucial. The intention of the item constructor and the fact that the q uestions had multiple correct answers received highest ratings from st udents. Phase 3 presented a collection of statistics items, some of wh ich were labeled on an a priori basis as being trick or not-trick. The analysis indicated that examinees were able to statistically differen tiate between trick and not-trick items, but the difference compared t o chance was small. Not-trick items were more successfully sorted than trick items, and trick items that were classified as intentional were sorted about as well as nonintentional items. Evidence seems to sugge st that the concept of trickiness is not as clear as some test constru ction textbook authors suggest.