Anna Kaatz1, Wairimu Magua, David R Zimmerman, Molly Carnes. 1. Dr. Kaatz is assistant scientist, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. Ms. Magua is a doctoral candidate, Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin. Dr. Zimmerman is professor, Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin. Dr. Carnes is director, Center for Women's Health Research, and professor, Departments of Medicine, Psychiatry, and Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin; and part-time physician, William S. Middleton Memorial Veterans Hospital Geriatric Research Education and Clinical Center, Madison, Wisconsin.
Abstract
PURPOSE: Career advancement in academic medicine often hinges on the ability to garner research funds. The National Institutes of Health's (NIH's) R01 award is the "gold standard" of an independent research program. Studies show inconsistencies in R01 reviewers' scoring and in award outcomes for certain applicant groups. Consistent with the NIH recommendation to examine potential bias in R01 peer review, the authors performed a text analysis of R01 reviewers' critiques. METHOD: The authors collected 454 critiques (262 from 91 unfunded and 192 from 67 funded applications) from 67 of 76 (88%) R01 investigators at the University of Wisconsin-Madison with initially unfunded applications subsequently funded between December 2007 and May 2009. To analyze critiques, the authors developed positive and negative grant application evaluation word categories and selected five existing categories relevant to grant review. They analyzed results with linear mixed-effects models for differences due to applicant and application characteristics. RESULTS: Critiques of funded applications contained more positive descriptors and superlatives and fewer negative evaluation words than critiques of unfunded applications. Experienced investigators' critiques contained more references to competence. Critiques showed differences due to applicant sex despite similar application scores or funding outcomes: more praise for applications from female investigators, greater reference to competence/ability for funded applications from female experienced investigators, and more negative evaluation words for applications from male investigators (all P<.05). CONCLUSIONS: Results suggest that text analysis is a promising tool for assessing consistency in R01 reviewers' judgments, and gender stereotypes may operate in R01 review.
PURPOSE: Career advancement in academic medicine often hinges on the ability to garner research funds. The National Institutes of Health's (NIH's) R01 award is the "gold standard" of an independent research program. Studies show inconsistencies in R01 reviewers' scoring and in award outcomes for certain applicant groups. Consistent with the NIH recommendation to examine potential bias in R01 peer review, the authors performed a text analysis of R01 reviewers' critiques. METHOD: The authors collected 454 critiques (262 from 91 unfunded and 192 from 67 funded applications) from 67 of 76 (88%) R01 investigators at the University of Wisconsin-Madison with initially unfunded applications subsequently funded between December 2007 and May 2009. To analyze critiques, the authors developed positive and negative grant application evaluation word categories and selected five existing categories relevant to grant review. They analyzed results with linear mixed-effects models for differences due to applicant and application characteristics. RESULTS: Critiques of funded applications contained more positive descriptors and superlatives and fewer negative evaluation words than critiques of unfunded applications. Experienced investigators' critiques contained more references to competence. Critiques showed differences due to applicant sex despite similar application scores or funding outcomes: more praise for applications from female investigators, greater reference to competence/ability for funded applications from female experienced investigators, and more negative evaluation words for applications from male investigators (all P<.05). CONCLUSIONS: Results suggest that text analysis is a promising tool for assessing consistency in R01 reviewers' judgments, and gender stereotypes may operate in R01 review.
Authors: Daniel M Blumenthal; Andrew R Olenski; Robert W Yeh; Doreen DeFaria Yeh; Amy Sarma; Ada C Stefanescu Schmidt; Malissa J Wood; Anupam B Jena Journal: Circulation Date: 2017-02-07 Impact factor: 29.690
Authors: Erica T Warner; René Carapinha; Griffin M Weber; Emorcia V Hill; Joan Y Reede Journal: J Womens Health (Larchmt) Date: 2017-08-03 Impact factor: 2.681
Authors: Elizabeth L Pier; Markus Brauer; Amarette Filut; Anna Kaatz; Joshua Raclaw; Mitchell J Nathan; Cecilia E Ford; Molly Carnes Journal: Proc Natl Acad Sci U S A Date: 2018-03-05 Impact factor: 11.205