Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement

Three alternative procedures to adjust significance levels for multiplicity are the traditional Bonferroni technique, a sequential Bonferroni techniqu e developed by Hochberg (1988), and a sequential approach for controlling t he false discovery rate proposed by Benjamini and Hochberg (1995). These pr ocedures are illustrated and compared using examples from the National Asse ssment of Educational Progress (NAEP). A prominent advantage of the Benjami ni and Hochberg (B-H) procedure, as demonstrated in these examples, is the greater invariance of statistical significance for given comparisons over a lternative family sizes. Simulation studies show that all three procedures maintain a false discovery rate bounded above, often grossly, by alpha (or alpha/2). For both uncorrelated and pairwise families of comparisons, the B -H technique is shown to have greater power than the Hochberg or Bonferroni procedures, and its power remains relatively stable as the number of compa risons becomes large, giving it an increasing advantage when many compariso ns are involved. We recommend that results from NAEP State Assessments be r eported using the B-H technique rather than the Bonferroni procedure.