OBJECTIVE: Peer review is the gold standard for evaluating scientific quality. Compared with studies on inter-reviewer variability, research on panel evaluation is scarce. To appraise the reliability of panel evaluations in grant review, we compared scores by two expert panels reviewing the same grant proposals. Our main interest was to evaluate whether panel discussion improves reliability. METHODS: Thirty reviewers were randomly allocated to one of the two panels. Sixty-five grant proposals in the fields of clinical medicine and epidemiology were reviewed by both panels. All reviewers received 5-12 proposals. Each proposal was evaluated by two reviewers, using a six-point scale. The reliability of reviewer and panel scores was evaluated using Cohen's kappa with linear weighting. In addition, reliability was also evaluated for the panel mean scores (mean of reviewer scores was used as panel score). RESULTS: The proportion of large differences (at least two points) was 40% for reviewers in panel A, 36% for reviewers in panel B, 26% for the panel discussion scores, and 14% when the means of the two reviewer scores were used. The kappa for panel score after discussion was 0.23 (95% confidence interval: 0.08, 0.39). By using the mean of the reviewer scores, the panel coefficient was similarly 0.23 (0.00, 0.46). CONCLUSION: The reliability between panel scores was higher than between reviewer scores. The similar interpanel reliability, when using the final panel score or the mean value of reviewer scores, indicates that panel discussions per se did not improve the reliability of the evaluation.
OBJECTIVE: Peer review is the gold standard for evaluating scientific quality. Compared with studies on inter-reviewer variability, research on panel evaluation is scarce. To appraise the reliability of panel evaluations in grant review, we compared scores by two expert panels reviewing the same grant proposals. Our main interest was to evaluate whether panel discussion improves reliability. METHODS: Thirty reviewers were randomly allocated to one of the two panels. Sixty-five grant proposals in the fields of clinical medicine and epidemiology were reviewed by both panels. All reviewers received 5-12 proposals. Each proposal was evaluated by two reviewers, using a six-point scale. The reliability of reviewer and panel scores was evaluated using Cohen's kappa with linear weighting. In addition, reliability was also evaluated for the panel mean scores (mean of reviewer scores was used as panel score). RESULTS: The proportion of large differences (at least two points) was 40% for reviewers in panel A, 36% for reviewers in panel B, 26% for the panel discussion scores, and 14% when the means of the two reviewer scores were used. The kappa for panel score after discussion was 0.23 (95% confidence interval: 0.08, 0.39). By using the mean of the reviewer scores, the panel coefficient was similarly 0.23 (0.00, 0.46). CONCLUSION: The reliability between panel scores was higher than between reviewer scores. The similar interpanel reliability, when using the final panel score or the mean value of reviewer scores, indicates that panel discussions per se did not improve the reliability of the evaluation.
Authors: Elizabeth L Pier; Markus Brauer; Amarette Filut; Anna Kaatz; Joshua Raclaw; Mitchell J Nathan; Cecilia E Ford; Molly Carnes Journal: Proc Natl Acad Sci U S A Date: 2018-03-05 Impact factor: 11.205
Authors: Elizabeth L Pier; Joshua Raclaw; Anna Kaatz; Markus Brauer; Molly Carnes; Mitchell J Nathan; Cecilia E Ford Journal: Res Eval Date: 2017-02-14
Authors: Janelle E Letzen; Vani A Mathur; Mary R Janevic; Michael D Burton; Anna M Hood; Calia A Morais; Staja Q Booker; Claudia M Campbell; Edwin N Aroke; Burel R Goodin; Lisa C Campbell; Ericka N Merriwether Journal: J Pain Date: 2022-02-26 Impact factor: 5.383
Authors: Afton S Carpenter; Joanne H Sullivan; Arati Deshmukh; Scott R Glisson; Stephen A Gallo Journal: BMJ Open Date: 2015-09-08 Impact factor: 2.692