Multiple evaluations of grant proposals by independent assessors: Confirmatory factor analysis evaluations of reliability, validity, and structure

External assessor ratings of the quality of the research team and of the pr oposed research were evaluated for proposals submitted to the highly compet itive Australian Research Council large grants program. The structure of re sponses by multiple assessors was evaluated using a series of nested models that are variations of the traditional congeneric, tau-equivalent, and par allel measurement models. The estimated reliability based on four independe nt assessors was modest but comparable to other peer review research: .494 for proposal ratings, .634 for researcher ratings, and .704 for a total ass essment. Mean ratings (averaged over multiple assessors) of the proposal an d the research team were highly correlated (r = .85), suggesting a lack of differentiation and a substantial method/halo effect in ratings by the same assessor. Confirmatory factor analysis models supported this suggestion, r epresenting this method effect as correlated uniquenesses. Tests of whether proposal and researcher ratings reflected one or two latent variables, bas ed on the internal structure of the assessment ratings and relations with e xternal criteria (academic rank, prior funding, and publications), supporte d a one-factor solution.