Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges

Introduction Progress testing is an assessment method that samples the comp lete domain of knowledge that is considered pertinent to undergraduate medi cal education. Because of the comprehensive nature of this test, it is very difficult to set a passing score. We obtained a progress test standard usi ng an Angoff procedure with recent graduates as judges. This paper reports on the reliability and credibility of this approach. Methods The Angoff procedure was applied to a sample of 146 progress test i tems. The items were judged by a panel of eight recently graduated students . Generalizability, theory was used to investigate the reliability as a fun ction of the number of items and judges. Credibility was judged by comparin g the pass/fail rates resulting from the standard arrived at by the Angoff procedure with those obtained using a relative and a fixed standard. Results The results indicate that an acceptable error score can be achieved , yielding a precision within one percentage on the scoring scale, by using 10 judges on a full-length progress test (i.e. 250 items). The pass/fail r ates associated with the Angoff standard came closest to those of the relat ive standard, which takes variations in test difficulty into account. A hig h correlation was found between item-Angoff estimates and the item P-values . Conclusion The results of this study suggest that the Angoff procedure, usi ng recently graduated students as judges, is an appropriate standard settin g method for a progress test.