A Bayesian hierarchical model for multi-level repeated ordinal data: Analysis of oral practice examinations in a large anaesthesiology training program

Oral practice examinations (OPEs) are used in many anaesthesiology programm es to familiarize anaesthesiology residents with the format of the oral exa mination administered by the American Board of Anesthesiology. The OPE outc ome (final grade) consists of 'Definite Not Pass' , 'Probable Not Pass', 'P robable Pass' and 'Definite Pass'. In our study to assess the validity of t he OPE, residents took an average of two (ranging from one to six) OPEs, ea ch of which was evaluated by two board certified anaesthesiologists randoml y selected from a pool of 12. A key question of interest was to identify fa ctors, for example, the length of training, didactic experience and other c haracteristics, that most influence OPE outcome. In addition, we were inter ested in assessing the reliability of the final grade, that is, the covaria nce parameters are of interest as well. However, estimating variance compon ents in multi-level data with an unequal number of repeated ordinal outcome s presents several statistical challenges, such as how to estimate high dim ensional random effects parameters, especially for ordinal outcomes. We pro pose a Bayesian hierarchical proportional odds model for data with such com plexity. The flexibility of such a model allows us to make inference on the association of OPE outcomes with other factors and to estimate the varianc e components as well. Copyright (C) 1999 John Wiley & Sons, Ltd.