USING AN EXTENDED ANGOFF PROCEDURE TO SET STANDARDS ON COMPLEX PERFORMANCE ASSESSMENTS

The number of assessments that are performance based is increasing rap idly, but to date there is not an established procedure for setting st andards on these assessments. The purposes of this article are to desc ribe several extensions to the Angoff procedure to accommodate the cha racteristics of a performance-based assessment and to present the resu lts of our research to apply this new procedure. The extensions includ ed a revised task for panelists, which involved the specification of ( a) expected scores for just barely certifiable candidates on polytomou sly scored exercises and (b) weights to reflect the relative importanc e of scoring dimensions at the exercise level and the exercises themse lves. The results obtained from the new procedure were mixed. On the o ne hand, panelists seemed to find the new procedure straightforward to apply and were able to reach a high level of agreement among themselv es about the standards. Confidence levels in the resulting standards w ere high. However, when given the choice, panelists definitely preferr ed a standard-setting procedure that was at least partly conjunctive, as opposed to the extended Angoff procedure that was completely compen satory in its nature. Suggestions for follow-up research are offered i n the article.