Robyn Tamblyn1, Nadyne Girard2, Christina J Qian2, James Hanley2. 1. Institute of Health Services and Policy Research (Tamblyn), Canadian Institutes of Health Research, Ottawa, Ont.; Departments of Medicine (Tamblyn) and Epidemiology, and Biostatistics (Girard, Qian, Hanley), Faculty of Medicine, McGill University, Montréal, Que. robyn.tamblyn@mcgill.ca. 2. Institute of Health Services and Policy Research (Tamblyn), Canadian Institutes of Health Research, Ottawa, Ont.; Departments of Medicine (Tamblyn) and Epidemiology, and Biostatistics (Girard, Qian, Hanley), Faculty of Medicine, McGill University, Montréal, Que.
Abstract
BACKGROUND: Peer review is used to determine what research is funded and published, yet little is known about its effectiveness, and it is suspected that there may be biases. We investigated the variability of peer review and factors influencing ratings of grant applications. METHODS: We evaluated all grant applications submitted to the Canadian Institutes of Health Research between 2012 and 2014. The contribution of application, principal applicant and reviewer characteristics to overall application score was assessed after adjusting for the applicant's scientific productivity. RESULTS: Among 11 624 applications, 66.2% of principal applicants were male and 64.1% were in a basic science domain. We found a significant nonlinear association between scientific productivity and final application score that differed by applicant gender and scientific domain, with higher scores associated with past funding success and h-index and lower scores associated with female applicants and those in the applied sciences. Significantly lower application scores were also associated with applicants who were older, evaluated by female reviewers only (v. male reviewers only, -0.05 points, 95% confidence interval [CI] -0.08 to -0.02) or reviewers in scientific domains different from the applicant's (-0.07 points, 95% CI -0.11 to -0.03). Significantly higher application scores were also associated with reviewer agreement in application score (0.23 points, 95% CI 0.20 to 0.26), the existence of reviewer conflicts (0.09 points, 95% CI 0.07 to 0.11), larger budget requests (0.01 points per $100 000, 95% CI 0.007 to 0.02), and resubmissions (0.15 points, 95% CI 0.14 to 0.17). In addition, reviewers with high expertise were more likely than those with less expertise to provide higher scores to applicants with higher past success rates (0.18 points, 95% CI 0.08 to 0.28). INTERPRETATION: There is evidence of bias in peer review of operating grants that is of sufficient magnitude to change application scores from fundable to nonfundable. This should be addressed by training and policy changes in research funding.
BACKGROUND: Peer review is used to determine what research is funded and published, yet little is known about its effectiveness, and it is suspected that there may be biases. We investigated the variability of peer review and factors influencing ratings of grant applications. METHODS: We evaluated all grant applications submitted to the Canadian Institutes of Health Research between 2012 and 2014. The contribution of application, principal applicant and reviewer characteristics to overall application score was assessed after adjusting for the applicant's scientific productivity. RESULTS: Among 11 624 applications, 66.2% of principal applicants were male and 64.1% were in a basic science domain. We found a significant nonlinear association between scientific productivity and final application score that differed by applicant gender and scientific domain, with higher scores associated with past funding success and h-index and lower scores associated with female applicants and those in the applied sciences. Significantly lower application scores were also associated with applicants who were older, evaluated by female reviewers only (v. male reviewers only, -0.05 points, 95% confidence interval [CI] -0.08 to -0.02) or reviewers in scientific domains different from the applicant's (-0.07 points, 95% CI -0.11 to -0.03). Significantly higher application scores were also associated with reviewer agreement in application score (0.23 points, 95% CI 0.20 to 0.26), the existence of reviewer conflicts (0.09 points, 95% CI 0.07 to 0.11), larger budget requests (0.01 points per $100 000, 95% CI 0.007 to 0.02), and resubmissions (0.15 points, 95% CI 0.14 to 0.17). In addition, reviewers with high expertise were more likely than those with less expertise to provide higher scores to applicants with higher past success rates (0.18 points, 95% CI 0.08 to 0.28). INTERPRETATION: There is evidence of bias in peer review of operating grants that is of sufficient magnitude to change application scores from fundable to nonfundable. This should be addressed by training and policy changes in research funding.
Authors: Jennifer Sheridan; Julia N Savoy; Anna Kaatz; You-Geon Lee; Amarette Filut; Molly Carnes Journal: J Womens Health (Larchmt) Date: 2017-04-04 Impact factor: 2.681
Authors: Matthew K Eblen; Robin M Wagner; Deepshikha RoyChowdhury; Katherine C Patel; Katrina Pearson Journal: PLoS One Date: 2016-06-01 Impact factor: 3.240
Authors: Djillali Annane; Nicolas Lerolle; Sylvain Meuris; Jean Sibilla; Keith M Olsen Journal: Intensive Care Med Date: 2018-11-13 Impact factor: 17.440
Authors: Sera Whitelaw; Lehana Thabane; Mamas A Mamas; Nosheen Reza; Khadijah Breathett; Pamela S Douglas; Harriette G C Van Spall Journal: J Am Coll Cardiol Date: 2020-10-27 Impact factor: 24.094
Authors: Nikola Komlenac; Marie Gustafsson Sendén; Petra Verdonk; Margarethe Hochleitner; Heidi Siller Journal: Adv Health Sci Educ Theory Pract Date: 2019-03-06 Impact factor: 3.853