Unobserved heterogeneity as an alternative explanation for "reversal" effects in behavioral research

Behavioral researchers use analysis of variance (ANOVA) tests of difference s between treatment means or chi-square tests of differences between propor tions to provide support for empirical hypotheses about consumer behavior. These tests are typically conducted on data from "between-subjects" experim ents in which participants were randomly assigned to conditions. We show th at, despite using internally valid experimental designs such as this, aggre gation biases can arise in which the theoretically critical pattern holds i n the aggregate even though it holds for no (or few) individuals. First, we show that crossover interactions-often taken as strong evidence of moderat ing variables-can arise from the aggregation of two or more segments that d o not exhibit such interactions when considered separately. Second, we show that certain context effects that have been reported for choice problems c an result from the aggregation of two (or more) segments that do not exhibi t these effects when considered separately. Given these threats to the conc lusions drawn from experimental results, we describe the conditions under w hich unobserved heterogeneity can be ruled out as an alternative explanatio n based an one or more of the following: a priori considerations, derived p roperties, diagnostic statistics, and the results of latent class modeling. When these tests cannot rule out explanations based on unobserved heteroge neity, this is a serious problem for theorists who assume implicitly that t he same theoretical principle works equally for everyone, but for random er ror. The empirical data patterns revealed by our diagnostics can expose the weakness in the theory but not fix it. It remains for the researcher to do further work to understand the underlying constructs that drive heterogene ity effects and to revise theory accordingly.