Hw. Marsh et P. Bazeley, Multiple evaluations of grant proposals by independent assessors: Confirmatory factor analysis evaluations of reliability, validity, and structure, MULTIV BE R, 34(1), 1999, pp. 1-30
External assessor ratings of the quality of the research team and of the pr
oposed research were evaluated for proposals submitted to the highly compet
itive Australian Research Council large grants program. The structure of re
sponses by multiple assessors was evaluated using a series of nested models
that are variations of the traditional congeneric, tau-equivalent, and par
allel measurement models. The estimated reliability based on four independe
nt assessors was modest but comparable to other peer review research: .494
for proposal ratings, .634 for researcher ratings, and .704 for a total ass
essment. Mean ratings (averaged over multiple assessors) of the proposal an
d the research team were highly correlated (r = .85), suggesting a lack of
differentiation and a substantial method/halo effect in ratings by the same
assessor. Confirmatory factor analysis models supported this suggestion, r
epresenting this method effect as correlated uniquenesses. Tests of whether
proposal and researcher ratings reflected one or two latent variables, bas
ed on the internal structure of the assessment ratings and relations with e
xternal criteria (academic rank, prior funding, and publications), supporte
d a one-factor solution.