Multiple Evaluations of Grant Proposals by Independent Assessors: Confirmatory Factor Analysis Evaluations of Reliability, Validity, and Structure.

External assessor ratings of the quality of the research team and of the proposed research were evaluated for proposals submitted to the highly competitive Australian Research Council large grants program. The structure of the responses by multiple assessors was evaluated using a series of nested models that are variations of the traditional congeneric, tau-equivalent, and parallel measurement models. The estimated reliability based on four independent assessors was modest but comparable to other peer review research: .494 for proposal ratings, .634 for researcher ratings, and .704 for a total assessment. Mean ratings (averaged over multiple assessors) of the proposal and the research team were highly correlated (r = .85), suggesting a lack of differentiation and a substantial method/halo effect in ratings by the same assessor. Confirmatory factor analysis models supported this suggestion, representing this method effect as correlated uniquenesses. Tests of whether proposal and researcher ratings reflected one or two latent variables, based on the internal structure of the assessment ratings and relations with external criteria (academic rank, prior funding, and publications), supported a one-factor solution.