Do we really 'know' what we think we know? A case study of seminal research and its subsequent overgeneralization

We show that the community of accounting researchers has not fully apprecia ted the sensitivity of research conclusions to (necessarily) subjective res earch design choices, and that this failure has led to the subsequent overg eneralization of early evidence. Our analysis is based on a case study that examines the effects of two basi c research design choices, and subsequent researchers' appreciation of the impact of those choices. To enhance our study's accessibility to a broad cr oss-section of the accounting research community, our case study focuses on an article that is covered in most accounting doctoral programs - the reci pient of the second American Accounting Association "Seminal Contribution t o Accounting Literature Award" Beaver's 1968 article on the information con tent of annual earnings announcements [Beaver, W. (1968). The information c ontent of annual earnings announcements. Empirical research in accounting; Selected studies, 1968. Supplement to Journal of Accounting Research, 6, 67 -92.] Our analysis reveals that the research design choices in Beaver's stu dy significantly affected the results. Beaver clearly explained these choic es, and was careful in drawing inferences, Nevertheless, subsequent researc hers interpreted Beaver's evidence too broadly. Our new empirical evidence suggests that the information content inference is much more fragile than i s generally suspected. We also present citation analyses showing that, cons istent with the Kuhnian view that adherents of a paradigm tend to ignore (l ater) anomalous evidence, subsequent researchers largely overlooked post-Be aver evidence that. was inconsistent with the paradigm's interpretation of Beaver's results. The paper concludes with a brief consideration of how cog nitive biases of individuals, combined with biases inherent in the review p rocess and the academy in general, foster an environment where the placemen t of the first research bricks affect the whole wall. While the case study focuses on an archival empirical financial accounting study, consideration of the effects of subjective research design choices and the documented def iciencies in the interpretation of research are relevant to empirical accou nting researchers in general and to critical theorists, as well. While accounting scholars 'think we understand the impact of research desig n choices, the evidence presented here suggests that we have collectively f ailed to apply this understanding in practice. Such failure delays the acqu isition of knowledge. This study provides a salient demonstration indicatin g that we must constantly remain on guard to recognize the subjective natur e of research design choices, to consider the likely effects of those choic es, and to be cautious in drawing generalizations. (C) 2000 Elsevier Scienc e Ltd. All rights reserved.