Beta risk: an unrecognized risk of statistical error

Data collected in a medical study should, from a methodological point of vi ew, be considered as a sample taken from a larger population. The purpose o f the statistical analysis is to check whether the differences in the exper imental results observed in different subgroups are related to chance or no t. The risks of error must be known to assess the validity of the conclusio ns. The first order risk, also called the alpha risk, is the risk of announcing a wrongly positive conclusion, that is to conclude that there is a signifi cant difference that in reality does not exist. By convention, an alpha ris k of 5 p. 100 is generally accepted. This means that it is acceptable to an nounce a statistically positive test when no difference exists in at most 5 p. 100 of the cases. After recording and processing the data, the statisti cal analysis produces a Value called p that is the exact value of the first order risk in the given situation. If p is less than or equal to the alpha risk accepted before the study, it can be concluded that the observed diff erence is statistically significant at the chosen alpha level and that the p value represents the risk of first order risk in the given situation. If p is greater than the initially accepted alpha, the observed difference is not considered to be significant at the alpha level. But the assertion that two samples are equivalent, also involves a second o rder risk, also called the beta risk, that must be known. The beta risk is the risk of announcing wrongly negative results, that is to conclude that t wo samples are equivalent white in reality they are different. The number o f elements in each sample necessary to demonstrate a difference becomes gre ater as the size of the difference becomes smaller. The beta risk increases as the alpha risk decreases, the number of cases becomes smaller and the d ifference to detect becomes smaller. If a difference is not statistically s ignificant at the chosen alpha level, the beta risk of an erroneous conclus ion of equivalence is generally less than or equal to 20 p. 100. In most cases, the beta risk is not determined before the study but after, being calculated from the alpha risk, the sample size, and the non-signific ant difference observed. If the beta risk is found to be greater than 20 p. 100, no conclusion can be drawn and the study data are useless. It is ther efore preferable to define both the alpha and beta risk and the smallest cl inically pertinent difference, and to calculate the necessary sample size, before initiating the study. Let us take a numerical example where two different treatments, A and B, ar e given to two groups of 100 patients each. Treatment A reduced success in 70 cases and treatment B in 80 cases. The chi-squared test yields a p value of 0.10. The observed difference is thus not statistically significant at an alpha level of 5 p. 100. In this case, the calculated beta risk is 54 p. 100. With 200 patients and a beta risk of 20 p. 100, a difference of 20 p. 100 in the success rates between the two groups cannot be detected. If it is accepted that a difference of 10 p. 100 between the success rates is cli nically pertinent, to have an acceptable beta risk of 20 p. 100 and detect the difference, the study would have to include 500 patients instead of 200 . In conclusion, when a comparative study concludes that there is no signific ant difference between two groups, one cannot deduct that these two groups are identical unless the beta risk is less than 20 p. 100. If the beta risk is greater than 20 p. 100, or if it is not mentioned, one cannot conclude that the two groups are equivalent.