QUAID: A questionnaire evaluation aid for survey methodologists

QUAID (question-understanding aid) is a software tool that assists survey m ethodologists, social scientists, and designers of questionnaires in improv ing the wording, syntax, and semantics of questions. The tool identifies po tential problems that respondents might have in comprehending the meaning o f questions on questionnaires. These problems can be scrutinized by researc hers when they revise questions to improve question comprehension and, ther eby, enhance the reliability and validity of answers. QUAID was designed to identify nine classes of problems, but only five of these problems are add ressed in this article: unfamiliar technical term, vague or imprecise relat ive term, vague or ambiguous noun phrase, complex syntax, and working memor y overload. We compared the output of QUAID with ratings of language expert s who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 505 questions on 11 surveys developed by the U.S. Censu s Bureau. Analyses of hit rates, false alarm rates, d' scores, recall score s, and precision scores revealed that QUAID was able to identify these five problems with questions, although improvements in QUAID's performance are anticipated in future research and development.