EFFECTS OF RATING STRATEGY ON INTERDIMENSIONAL VARIANCE, RELIABILITY,AND VALIDITY OF INTERVIEW RATINGS

This study experimentally manipulated various rating strategies to tes t effects on interdimensional variance, reliability, and validity of i nterview ratings. Undergraduate research participants (N = 180) rated 60 interview transcripts under 3 experimental conditions that represen ted different rating strategies. Interdimensional variance was higher, internal consistency reliability was lower, and interrater reliabilit y was higher in the rating condition that gave raters the least opport unity to form and use general impressions of interviewees. Rating stra tegy had no discernible effects on validity. To interpret these result s, the authors argued that the rating strategies used in this study di ffered in the degree to which they allowed raters to form idiosyncrati c general impressions of interviewees. These idiosyncratic halo effect s explain why rating strategies that raise internal consistency reliab ility simultaneously lower interrater reliability.