Impact of an institutional grant award on early career investigator applicants and peer reviewers.

BACKGROUND: Obtaining research funding support is integral to a successful career in science. Training and practice in grant writing, as well as engagement in peer review of grant applications may help lead to successful research funding. However, there is little evidence on the impact of institutional programs on the career development of early career investigators (ECIs).
OBJECTIVES: Understand the impact of participation in an institutional research award program on the career development of ECIs.
METHODS: The Cardiovascular Research Institute of Vermont established an Early Career Research (ECR) award program in 2018. ECIs who participated as applicants or reviewers in the first 3 years of the program (2018-2020) were surveyed to understand the impact of the ECR award program on their grant writing and professional development.
RESULTS: Ninety-four percent of 17 applicants and 90% of 19 reviewers completed the survey. Ninety-two percent of funded and 75% of unfunded applicants, and 87% of reviewers reported that the program was beneficial to their professional development. Similarly, 85% of funded applicants, 75% of unfunded applicants, and 80% of reviewers reported improvement in their grant-writing skills. All respondents reported they would recommend the ECR award program to their peers.
CONCLUSIONS: This single-institution ECR award program had a positive impact on ECI's professional development and grant-writing skills and may lead to further extramural funding opportunities.

The impact of institutional awards on early career investigator (ECI) development is unknown.

This Early Career Research (ECR) award engaged ECIs as applicants and peer reviewers.

Survey results of applicants and peer reviewers showed improved grant‐writing skill.

This institutional ECR award positively impacted career development of ECIs.

INTRODUCTION

Early career investigators (ECI) need intensive training in scientific writing and peer‐review for successful career development. In the United States, recent funding rates for cardiovascular science ECIs (ie, trainees and junior faculty) applying for competitive national level grants are low. In 2020, the funding rate at the National Science Foundation for Biological Sciences was 36%, while this was 15% to 21% at the American Heart Association and 27% to 31% in 2017 at the National Institutes of Health (NIH).  The average age for receipt of first NIH R01 or equivalent grant (ie, independent funding) increased from 38 in 1980 to 42 to 45 as of 2013. Similarly, the proportion of younger (≤36 years of age) principal investigators with R01 funding decreased from 18% in 1983 to 3% in 2010.  While low success rates among ECIs can be attributed to many factors, including increasing competition and lack of early track records of funding, poorly written grant applications reduce the likelihood of success.

Institutional grant opportunities face similar challenges, where applications that lack clarity or conceptualization are frequently triaged. While the approach to grant writing can be taught in the classroom, grant‐writing skills are underemphasized in many university graduate program curricula. Crafting a successful grant application is a skill that requires practice and experience, and there remains an unmet need for ECI grant‐writing training. ,  We investigated the impact of an institutional cardiovascular science Early Career Research (ECR) award program on the scientific writing and professional development of ECIs.

METHODS

Setting

The Cardiovascular Research Institute (CVRI) at the University of Vermont (UVM) was established in 2002 with the aim to reduce the incidence, morbidity, and mortality of cardiovascular diseases through improving prevention, diagnosis, and treatment.  The CVRI Board of Directors is composed of faculty members with long‐standing track records of success in cardiovascular research and medicine. One objective of the CVRI is to foster the career development of ECIs. Toward this end, it established the Early Career Advisory Committee (ECAC) composed of medical and PhD students, postdoctoral trainees, clinical residents, fellows, and assistant professors (within 5 years of appointment) from different areas of cardiovascular science. ECAC members are selected by the CVRI Board of Directors using a competitive application process. A CVRI board member serves as a faculty advisor to the ECAC by assisting the committee in designing and implementing programs that will benefit the UVM ECI community.

In 2018, the ECAC initiated an annual ECR award to provide small pilot awards of up to $10,000 over 12 months for research or career development projects. The ECAC designed the program, selection criteria, and application materials with assistance of the faculty advisor and received approval and funding from the CVRI Board of Directors. Figure 1 describes the program methods. The CVRI administrator advertised the funding announcement, received applications, and conducted a peer review session. The ECAC faculty advisor provided a 90‐min training session on how to conduct peer review, following the NIH study section methods and application rating scale.  The faculty advisor and chair of the ECAC then assigned two peer reviewers to each application, respecting declared conflicts of interest. Peer reviewers were ECAC members, excluding those who applied for a grant or had conflict of interest. Peer reviewers provided written critiques, assigned scores, and presented these to the ECI peer review panel using NIH study section methods. The faculty advisor, who had substantial experience in NIH study sections and a long track record of grant funding, served as the Scientific Review Officer at the meeting and answered questions about the process but did not participate in the critiques or discussion.

FIGURE 1

Early career research award timeline. CVRI, Cardiovascular Research Institute; CV, curriculum vitae; ECAC, Early Career Advisory Committee

Program implementation

Eligibility

ECIs, including undergraduate, masters, PhD, and medical students; postdoctoral trainees; medical residents and fellows; and early career faculty and staff within 5 years of appointment (research assistant/specialist or assistant professor) were eligible to apply for the ECR award if conducting cardiovascular‐related research at UVM. Reapplication was encouraged for both funded and unfunded applicants.

Application

ECR award applications were accepted annually. Submission materials included a cover letter, research plan (2‐page maximum), NIH biosketch or current curriculum vitae, budget justification, and letter of support from a mentor or department chair. Letters from external collaborators were also accepted. Applicants were encouraged to outline career goals in the cover letter and clearly state how receipt of the ECR award will help further career advancement. All funded applicants were required to submit a project completion report at the end of the funding period and present research progress at CVRI events.

During the first year of the ECR award program, the full application was preceded by a letter of intent subject to an initial review process. Following internal review and feedback from applicants, the letter of intent step was removed in 2019, and applicants submitted full applications in response to the call for applications.

If animals or human subjects were involved in the research, awardees were required to receive institutional review board or institutional animal care and use committee approval before the start of the funding.

Program evaluation

A voluntary online survey was created using REDCap and distributed to all ECR award applicants, both funded and unfunded, from 2018 to 2019, such that data from funded applicants were collected 1 year after receipt of award. The survey link was also distributed to peer reviewers who participated in the program between 2018 and 2020. The primary aim was to describe the impact of an institutional ECR award experience on grant‐writing skills and professional development of both applicants and reviewers. Survey respondents reported demographics, current position, published work, and national level grants and perspectives on the ECR award experience. Survey questions were Likert‐style, multiple choice and free‐text (Appendix S1). A separate survey was distributed to all ECAC members before and after the peer review training session to ascertain perspectives about their experiences on the ECAC, including the impact of the grant review training session. Survey recipients were asked to rate their confidence in providing peer review for grants on a scale of 0 to 100.

According to the policy describing activities which constitute research at UVM, this work met criteria for operational improvement activities and was considered exempt from ethics review.

Statistical analysis

Results were expressed as median values with range. Comparisons between funded and unfunded applicant demographics were made with chi‐squared tests.

RESULTS

Applicant and reviewer demographics

The survey response rate for ECR applicants was 94% (16/17). From 2018 to 2019 the program funded nine applicants (53%), including one PhD student, four postdoctoral trainees, two research assistants, and two assistant professors from various disciplines, with projects ranging from study of brain vasculature to a small clinical trial. Two‐thirds of applications (both funded and unfunded) were basic science (56% funded, 63% unfunded). Applicants were balanced across sex and the funding rate was similar by sex (56% male, 44% female). Reporting of applicant age was optional and missing for 74% of reviewers and 24% of applicants, so we did not report this. All applicants self‐identified either as White, Asian, or Other. More senior applicants were funded at higher rates than more junior early career applicants (Table 1).

TABLE 1

Early Career Research (ECR) award applicant demographics

	2018–2019		P value
	Funded, n (%)	Unfunded, n (%)
Number of applicants	9	8
Sex			0.63
Male	5 (56)	3 (38)
Female	4 (44)	5 (62)
Race/Ethnicity			0.36
White	6 (67)	7 (88)
Asian	2 (22)	0 (0)
Black/African American	0 (0)	0 (0)
Hispanic/Latino	0 (0)	0 (0)
Other	1 (11)	1 (12)
Position at the time of application submission			0.05
Undergraduate student	0 (0)	1 (13)
Graduate student (Master/PhD)	1 (12)	4 (50)
Postdoctoral trainee	4 (44)	0 (0)
Clinical fellow	0 (0)	2 (25)
Research assistant/specialist	2 (22)	0 (0)
Assistant professor	2 (22)	1 (12)
Research types of application			0.62
Basic science	5 (56)	5 (63)
Clinical science	3 (34)	3 (37)
Epidemiologic science	1 (11)	0 (0)

The survey response rate for ECR reviewers was 90% (17/19), and their characteristics at the time of survey are shown in Table 2. Fifty‐nine percent were women, and 65% identified as White. Reviewers encompassed ECAC members with ECI faculty members (58%).

TABLE 2

ECR reviewer demographics at the time of survey (2018‐2020)

	Number (%)
Sex
Male	7 (41)
Female	10 (59)
Race/Ethnicity
White	11 (65)
Asian	3 (18)
Black/African American	0 (0)
Hispanic/Latino	2 (12)
Other	1 (5)
Current position
PhD student	1 (5)
Postdoctoral trainee	0 (0)
Medical student	1 (5)
Clinical fellow or resident	4 (24)
Assistant professor	9 (53)
Faculty scientist	1 (5)
Associate professor	1 (5)

Impact of ECR award on professional development

Most applicants (89% funded, 71% unfunded) reported that the ECR award was moderately or very helpful in enhancing their professional development (Figure 2A). Nearly all reviewers (94%) irrespective of the career stage (faculty vs trainees) also found the award program moderately or very helpful in their professional development (Figure 2B). We evaluated the impact of the grant review process on different aspects of reviewers’ professional development. Among reviewers, 94% reported the program was moderately or very helpful in improving their understanding of the grant review process, and 63% reported the program moderately or strongly improved their scientific writing skills (Figure 2C). Reviewers from each year (2018, 9; 2019, 11; and 2020, 9) reported an average 34% improvement in comfort with grant review as assessed by pre‐ and poststudy section surveys from 2018 to 2020 (Figure 3).

FIGURE 2

Professional development feedback of all participants involved in ECR award program. A and B, rating of how the ECR award program participation enhanced professional development (1, did not enhance; 2, mildly enhanced; 3, moderately enhanced; 4, strongly enhanced) for (A) applicants and (B) reviewers. C, rating of how ECR award program participation helped reviewers to understand the grant‐writing process and scientific writing skills (0, not applicable; 1, not helpful; 2, slightly helpful; 3, moderately helpful; 4, very helpful). The graphs present median rating scores and ranges; each dot represents an individual rating score

FIGURE 3

Change in reviewer comfort with providing peer review for grants over time. Peer reviewers were surveyed before (pre) and after (post) the grant review study section to ascertain subjective comfort with providing peer review for grant applications. Data are presented as medians; n, number of reviewers

Most reviewers (80%) and applicants (81% funded and 75% unfunded) reported that the ECR award program helped them in preparing national‐level grant applications (Figure 4A). Specifically, applicants identified the program to be moderately or very helpful in the domains of defining (94% funded, 92% unfunded), conceptualizing (92% funded, 89% unfunded), and refining the approach to a project (89% funded, 86% unfunded) (Figure 4B‐D).

FIGURE 4

Impact of ECR award program participation on scientific writing skills. A, overall rating of how ECR award program helped with writing future grant or fellowship applications based on categorical scale (0, not applicable; 1, not helpful; 2, slightly helpful; 3, moderately helpful; 4, very helpful). Specifically, respondents reported how the ECR award program helped funded and unfunded applicants (B) conceptualize a project (C) define a project, and (D) refine a project and approach (0, not applicable; 1, not helpful; 2, slightly helpful; 3, moderately helpful; 4, very helpful). The graphs present median rating scores and ranges; each dot represents an individual rating score

All applicants and reviewers reported they would recommend the ECR participation to their peers.

DISCUSSION

The CVRI of Vermont ECR award program was established to encourage and support ECIs in their research, improve grant writing and reviewing skills, and provide an active learning experience to train ECAC members on how to perform peer review of grants. Findings in this single‐institution study were that the experience was well received. Both applicants (irrespective of whether they received the award) and reviewers perceived a positive impact on professional development and grant preparation skills.

In contrast to a previous study of an institutional grant program where PhD students served as applicants and reviewers, our program included the entire ECI spectrum (undergraduates to assistant professors), and we did not find a notable difference in improvement of professional development and scientific writing skills between funded and unfunded applicants. Our data suggest that both funded and unfunded applicants found value in refining their research hypotheses and approaches, suggesting that the iterative process of grant authorship was a key learning experience. Similarly, ECI peer reviewers perceived this experience to be as beneficial for future grant preparation as the applicants, as demonstrated by perceived comfort with the grant review process and improvement in scientific writing skills. Additionally, our results demonstrated the improvement in comfort with providing peer review among ECI reviewers over time. There was an improved comfort level following the grant review training session each year, followed by a decline at the beginning of the academic year; this may be attributed to the departure of senior ECAC members following term completion and new members joining. Importantly, the comfort level in providing grant review increased cumulatively over the 3 years studied, suggesting retained confidence in grant review from repeated ECR peer review experience. While we have shown that application and participation in peer review for this institutional grant program is associated with perceived improvements in grant‐writing skills, there are other resources commonly available at academic institutions such as grant‐writing advisors, external grant‐writing courses, workshops by National Council of University Research Administrators. We believe the strength of the ECR grant program is the practical application of providing real‐world peer review in accordance with NIH protocols and/or writing a grant application.

This award funded nine research projects over 2 years, and the number of applications increased in the second year of the program; 100% of applicants and reviewers would recommend the ECR award program to their peers. While the ECR award attracted applicants from different career stages and racial backgrounds, there was an overall lack of diversity in applicants and reviewers as compared to the numbers of diverse students and trainees at UVM. UVM enrolled 10.6% to 13.4% students of color (American Indian or Alaska Native, Asian, Black/African‐American, and Native Hawaiian or Other Pacific Islander) annually at different career stages during 2012 to 2020; these numbers were further higher for UVM College of Medicine ranging from 25.3 to 33.7% annually.  We do not know if the distribution of race/ethnicity among the UVM cardiovascular research trainee population is similar to UVM‐wide diversity. However, as underrepresented groups might be less likely to apply for and receive grant funding, we hope our institutional award program can support these ECIs in similar fashion to national‐level grant‐writing training programs for underrepresented racial/ethnic groups. While the program could not control the diversity of applicants, the CVRI has made efforts to improve diversity across race, ethnicity and career stage for ECAC members by advertising the call for applications across the university to reach every possible applicant. Further, the ECAC faculty advisor discusses the role of unconscious bias in peer review with reviewers.

Some limitations of this study require consideration. Although the survey response rate was high, our results are limited by the relatively small sample size from a single institution and heterogeneity of respondents. We anticipate this will improve with a larger sample size assessed by future surveys regarding the ECR award program. This initial survey did not control for subjective reporting of perceived professional development and in the future would be strengthened by using validated tools to track improvement in scientific writing (ie, the 19‐question self‐assessment program). Additionally, we could not eliminate the possibility of biased responses from the reviewers and funded applicants due to their association with the CVRI and/or receipt of funding support. We did not collect information on native language, whether applicants had prior education outside the United States, or their prior level of training, and these factors may also contribute to interpretation and responses to the survey. A limitation of the ECR award peer review process is that reviewer critiques were not returned to the applicants. This was by design due to the small institutional size and concerns by ECAC members about reactions to review from their peers. For the next funding cycle, the ECAC decided to provide this feedback and hypothesize it will improve the program experience of applicants. We did not formally evaluate the effectiveness of the reviewer training session, although assessment of reviewer level of comfort in peer review before and after the session provided some evidence of this.

There is an unmet need for training in grant writing and peer review for ECIs. We are not aware of other reports of an ECR award program that focuses on critical aspects of a cardiovascular research career (scientific writing, peer review, and professional development) wherein ECIs are positively influenced by active participation in authorship and peer review. We also demonstrated the role of an institutional training and award program in improving peer review and scientific writing skills. We provide evidence that active learning through participation as a peer reviewer was beneficial to ECIs, suggesting there would be similar benefits at other universities, research institutions, and possibly funding agencies. The NIH recently established an Early Career Reviewer Program to help ECIs become competitive grant applicants by experiencing the peer review process. Though similar to our ECR award, the NIH program requests that the early career scientist be an assistant professor or equivalent role, whereas the diverse ECI cohort in this study included a spectrum of trainees. Further, long‐term study of ECR award participants and reviewers is needed to better understand the impact of the experience on career trajectory, including publications from ECR award‐funded work, national‐level funding, participation in national study sections, and attainment of academic positions.

AUTHOR CONTRIBUTIONS

AM, KW, ZL and JF designed the survey, distributed surveys, and collected and analyzed data. NO and MC conceptualized and designed the ECR award program. AM drafted the manuscript; AM, KW, ZL, JF, NO and MC edited and revised the manuscript. All authors approved the final version of the manuscript.

RELATIONSHIP DISCLOSURE

The authors declare no conflicts of interest.

Appendix S1

Click here for additional data file.