Assessing Validity and Test-Retest Reliability for "Pick K of N" Data

A mixed binomial error model is proposed which permits the evaluation of validity and test-retest reliability when individuals are asked to pick K of N objects. The approach explicitly incorporates the effect of guessing as well as the respondent's true discrimination ability, and recognizes that that ability will vary across respondents. So, given the choices made by a set of individuals, we are interested in estimating the distribution of true ability over the population. To evaluate the criterion-related validity each person's K choices (of N objects) are compared to an objective standard-the K "correct" choices. For the primary model considered here these data can be summarized by the distribution, over individuals, of the number of "matches" or agreements between a person's choices and the objective criterion. A similar procedure is used for evaluating test-retest reliability. The distribution for the number of matches is again used; but here the matches will refer to agreement between the individual's K choices on this occasion, and the K choices made by that individual on a previous occasion. The use of these reliability and validity results to address specific marketing questions, such as the segmentability of a market or the effect of advertising on perception and attitudes, is also discussed.