Searching for reliable relationships with statistics packages: An empirical example of the potential problems

Many social scientists appear to possess an overconfidence in the reliabili ty of research results from a single, small-sample, inferential study. In t his article, the authors speculate that "user-friendly" statistics packages have the potential to exacerbate statistical misinterpretation by providin g researchers with a tool to explore data easily and identify what is inter preted as "reliable" relationships. This article contains an empirical demo nstration of the potential problems that arise when a large number of stati stical tests are interpreted. Results show that statistically significant r esults may be unreliable. Also, a zero relationship can erroneously appear as a medium to large effect size relationship when a small sample is used ( e.g., n = 30). The authors suggest the need for multiple replications as th e criterion of a reliable finding.