Statistical test to assess rank-order imaging studies

Rationale and Objectives. Rank-order experiments often provide a reasonable method of determining whether a large-scale receiver operating characteris tic study can be justified. The authors' purpose was to formalize a propose d method for analyzing rank-order imaging experiments and provide methods t hat can be used in determining sample sizes for both cases and raters. Materials and Methods. Simulations were conducted to determine the adequacy of the normal approximation of a statistic used to test the null hypothesi s of random ordering. For a multireader experiment, formulas are presented and guidelines are provided to enable investigators to determine the number of required readers (raters) and cases for a specific study. Results. When there are at least five ordered images per case, 10 cases are sufficient to test a random rank order. When there are only three or four images for a case, 20 cases are required. The authors constructed tables of statistical power for selected numbers of ordered images, numbers of cases , and degrees of trend, and they also provide an approximation for use in s ituations that are not tabled. Conclusion. The statistical methods for analyzing rank-order experiments an d estimating sample sizes for study planning are relatively simple to imple ment. The derived formulas for sample size estimation, when applied to typi cal imaging experiments, indicate that modest numbers of cases and readers are required for rank-order studies.