Reliability of the Amsterdam Clinical Challenge Scale (ACCS): a new instrument to assess the level of difficulty of patient cases in medical education

Introduction In problem-based medical curricula, consideration should be gi ven to the level of difficulty of patient cases used for training and asses sment. The Amsterdam Clinical Challenge Scale (ACCS) has been developed to assess the degree of difficulty of patient cases in a systematic and reprod ucible manner. To determine the reliability of the instrument two research questions were addressed: (1) How many judges are required, on the basis of the total score of the ACCS, to obtain a reliable estimate of the difficul ty of a single case? (2) How many cases and/or how many judges are needed t o reach an acceptable level of reliability of the total score of the ACCS? Method Four judges scored 36 patient scripts reflecting a wide range of pat ient problems encountered in general practice. Each script was scored four times. In the reliability analysis, the generalizability theory was applied . Results The results show that the judges did, indeed, use the whole range o f difficulty ratings. When the ACCS is applied to a single case, eight or m ore judges are needed to reach an acceptable level of reliability. When mor e cases are involved, fewer judges are needed; for 10 or more cases one jud ge will be sufficient. Conclusions Given the typical length, for example of an objective structure d clinical examination, the ACCS makes it possible to provide a reliable es timate of the level of difficulty of such a test with only a limited number of judges.