A LATENT-VARIABLE MODELING APPROACH TO ASSESSING INTERRATER RELIABILITY, TOPIC GENERALIZABILITY, AND VALIDITY OF A CONTENT ASSESSMENT SCORING RUBRIC

This article describes the limitations of certain statistical techniqu es and approaches for handling multiple criteria in assessing reliabil ity, concurrent validity, and generalizability. The article also sugge sts alternatives to using these approaches on performance assessment m easures. The application of the latent variable modeling approach to t he data revealed a significant improvement in the degree of interrater reliability, concurrent validity, and generalizability (over raters a nd topics) on a scoring rubric. The improvement in concurrent validity was particularly noticeable. However, it should be noted that one of the main limitations of this study was the use of a small number of su bjects, which could have affected the validity of some of the findings .