Development and Evaluation of a Five-Component Toolkit for Internal Medicine Residents Applying for Subspecialty Fellowships.

Introduction: Subspecialty fellowship is a common career path for internal medicine (IM) residents, but little is published on residency program curricula for guiding residents through the process of applying to subspecialty fellowships. We describe a toolkit to guide IM residents through this process.
Methods: We developed and implemented the Fellowship Application Toolkit for IM residents at the University of California, San Francisco, from 2018 to 2020. Educational strategies included live workshops, written resources, and one-to-one coaching, consisting of five components: fellowship application guidebook, Fellowship Application Information Night, alumni contact list, personal statement resources and coaches, and virtual interview workshop and mock interviews. Residents were surveyed both pre- and postintervention to evaluate these resources' use and efficacy.
Results: Survey response rates were 21 of 41 (51%) in 2018, 25 of 41 (61%) in 2019, and 24 of 43 (56%) in 2020. Most respondents indicated the resources were extremely or very effective, including 30 of 36 (83%) who used the guidebook, 31 of 37 (84%) who attended the Fellowship Application Information Night, 10 of 15 (67%) who used the alumni contact list, nine of 10 (90%) who used the personal statement resources, and 12 of 14 (86%) who attended the virtual interview workshop. Respondents strongly or somewhat agreed that the overall efficacy of the residency's fellowship advising improved from pre- to postintervention (four of 17 [24%] in 2018 vs. 17 of 21 [81%] in 2020, p < .001). Discussion: Our Fellowship Application Toolkit was effective at supporting IM residents applying to fellowships.

Educational Objectives

By the end of this activity, learners will be able to:

Describe the timeline and components of the fellowship application process.

Compose a personal statement using a defined template and specialty-specific examples.

Prepare for interviews and negotiate postinterview communication.

Connect with an alumni network at other institutions.

Introduction

The majority of internal medicine (IM) residents in the United States report subspecialty career plans.[1] During the 2021 ERAS application cycle, over eight thousand residents applied to subspecialty fellowships, a number that has increased over the past 5 years.[2,3] Despite this common career choice for IM residents, there is little published about how residency programs can best guide residents through the fellowship application process.

There are only a few published resources offering guidance for residents about how to independently prepare for fellowship interviews. For example, Bosslet and colleagues have provided an excellent structured approach to help residents prepare for the pulmonary/critical care medicine fellowship application process.[4] In addition, several resources list common interview questions and provide interview tips.[5-7] However, residents desire more formal guidance on the process, preferably incorporated into the residency program curriculum.[8]

There is limited data about how programs can best support residents who are applying for subspecialty fellowship. Some medical schools have designed programs to help medical students applying to residency,[9,10] but the fellowship application process has important distinctions, including increased postinterview communication, the expectation for a more defined career vision, and additional career and/or personal considerations. A study by Sinclair and colleagues described the introduction of a didactic session and an objective structured teaching exercise to help residents prepare for fellowship interviews.[11] However, no study to our knowledge has described the implementation of a comprehensive fellowship application series to help guide residents through each step of the fellowship application process (e.g., writing a personal statement, applying through ERAS, preparing for interviews, and postinterview communication).

We used the Kern model of curriculum development in medical education[12] to create the Fellowship Application Toolkit, which consists of educational workshops and resources to support IM residents applying to subspecialty fellowship. We evaluated the impact of this toolkit using pre- and postintervention surveys.

Methods

Setting and Participants

We developed the Fellowship Application Toolkit for residents at the University of California, San Francisco, IM residency program. We included all residents (categorical and primary care) as well as recent alumni who were applying to fellowship programs in the IM subspecialties. Workshops and resources were created and implemented over a 3-year period from 2018 through 2020.

Interventions/Curriculum Development

We used the Kern model of curriculum development in medical education[12] to create our toolkit. We first conducted a general needs assessment by reviewing the primary literature to identify existing fellowship application resources for residents. Next, we completed a targeted needs assessment by conducting two baseline focus groups of residents who had applied for subspecialty fellowship in the prior year (18 residents participated in total). Residents indicated dissatisfaction with the current level of support (“We get pretty much no guidance from the residency program right now,” “[I felt] completely on my own”) and a desire for more support from the residency program. They also provided input on curricular content, including the recommendation to create an overview fellowship advising document, an organized subspecialty-specific Q&A, an alumni database, and a repository of personal statements.

The goal of the Fellowship Application Toolkit was to support and guide residents though the fellowship application process. Our educational strategies were a mix of live workshops (in person or virtual), written resources, and one-to-one coaching and included the five components (Appendix A) listed below. We created a career advising Box folder on the residency website and stored all materials in this folder, including the PDF and PowerPoint presentations as well as video recordings of the virtual events for those who could not attend in real time. Implementation of the five components of the toolkit was as follows:

Fellowship application guidebook: We created a fellowship application guidebook with detailed logistics of the fellowship application process, including sections on resources, timeline, cost, short tracking, getting information on programs, letters of recommendation, personal statements, ERAS components, interviews, and postinterview communication (Appendix B). We solicited input from residents and fellowship program directors (PDs) during the development of the content. We also kept a running list of things to add based on resident feedback and updated the guidebook in January of each year to ensure it was up to date for each application cycle. For example, we added sample language about how to respond to certain postinterview communications based on resident request.

Fellowship Application Information Night: We hosted a 2-hour Fellowship Application Information Night as an in-person event in February 2019 and 2020 and as a virtual event in February 2021. The first hour was a general overview of the fellowship application process delivered via a PowerPoint presentation to the entire group (Appendix C), and the second hour included subspecialty-specific breakout rooms with the fellowship PD, associate PD, and/or fellows to discuss subspecialty-specific questions/experiences. Residents were asked to RSVP in advance and include which fellowship breakout room they planned to attend so that we could ensure that faculty from all fellowships of interest were present. Each room had at least one faculty member/fellow (range: one to five) and at least one resident (range: one to 15). Residents also submitted questions in advance. We shared this identical list of anonymous questions with the fellowship PDs and residents prior to the event and recommended that each room use this list of questions to start the conversation (Appendix D).

Alumni contact list: We emailed all alumni of our program who graduated in the prior 10 years and asked if they would be willing to be contacted by current residents to get fellowship and career advice. If so, we asked that they complete a Qualtrics form with their name, residency graduation year, fellowship specialty and location, current position, clinical interests, and email address. We used this information to create an alumni contact list in an Excel document, organized by subspecialty. We contacted alumni annually to update the list as needed and published an updated list in January or February of each year.

Personal statement resources: We developed personal statement resources comprising a personal statement template (structured guidance by paragraph; see personal statement section of Appendix B) and a repository of example personal statements representing different specialties and career goals. To collect the example personal statements, we emailed recent fellowship applicants and graduates and asked if they would be willing to share their personal statements anonymously (they could remove names of mentors or identifying details per their preference). We emailed these resources to residents in March and also offered the opportunity to sign up to get a personal statement coach. Then, we paired all interested residents with a personal statement coach (in a different subspecialty from the one in which the resident was applying, to preserve confidentiality) to review and give input on their personal statement between April and May of each year. Each coach was a faculty member who was already involved in the fellowship application process in their own specialty. Therefore, we did not require formal training for the coaches but offered to meet with them if additional training was desired.

Virtual interview workshop and mock interviews: We held a 2-hour Interview Information Night in July of each year that covered interview preparation, tips for virtual interviewing, postinterview communication, and a panel Q&A with recent fellowship applicants (Appendix E). After this event, we emailed residents to ask whether they were interested in participating in a mock interview and paired all interested residents with a faculty member (in a different subspecialty from the one in which the resident was applying) for mock interviews. Mock interviews were structured as 30-minute interviews, with 20 minutes for mock interview questions (at least one behavioral question) and 10 minutes for feedback. All faculty were provided with sample questions and an evaluation rubric (Appendix F). Sample interview questions were gathered by soliciting common interview questions from prior applicants and adapting questions from other sources.[13]

Outcomes Measured/Curriculum Evaluation

To evaluate the efficacy of the Fellowship Application Toolkit, we designed a 39-item questionnaire (Appendix G) with Qualtrics Survey Software using Artino and colleagues’ survey design process.[14] Item types included multiple-choice, 5-point Likert-scale, and open-ended questions. Survey validity was assessed using pilot testing. Residents were surveyed during the 2 weeks after Match Day both preintervention (2018) and postintervention (2019 and 2020) via electronic Qualtrics survey. Surveys were anonymous, and no incentives were provided. Both complete and incomplete responses were included in calculating the response rate.

Analysis of the Outcomes

Data were analyzed using Prism Software (GraphPad). We used descriptive statistics to summarize numeric responses. Comparisons between groups were made using the unpaired t test or two-sided Fisher exact test for small numbers. A p value of less than .05 was considered statistically significant. Qualitative survey data were analyzed using inductive content analysis to identify themes.[15] Laura A. Huppert and Jennifer M. Babik each coded the data, and differences were reconciled via iterative discussion.

IRB Statement

This work was deemed exempt by the University of California, San Francisco, Institutional Review Board.

Results

Survey response rates by year were as follows: 21 of 41 (51%) in 2018, 25 of 41 (61%) in 2019, and 24 of 43 (56%) in 2020. Demographic characteristics of the participants are shown in Table 1.

Table 1.

Demographic Information About the Survey Respondents

Feasibility

The five elements of the Fellowship Application Toolkit, including the approximate time and costs required for development and implementation, are shown in Appendix A. All events were developed and implemented by the two authors, with administrative support to help maintain the alumni contact list. Survey respondents reported their use of each resource: Thirty-six of 49 (74%) used the fellowship application guidebook, 37 of 49 (76%) attended the Fellowship Application Information Night, 15 of 49 (31%) used the alumni contact list, 10 of 24 (42%) used the personal statement resources, and 14 of 23 (61%) attended the virtual interview workshop (Figure 1A). Participants who did not use the resource were asked why—reasons varied by resource but most commonly were lack of knowledge of the resource or conflicting personal or residency obligations during the events (data not shown). Acceptability of each resource for the residents was measured explicitly via survey assessment, as described below. Fellowship PDs/associate PDs and personal statement coaches were surveyed and reported a high degree of satisfaction with the Fellowship Information Night and personal statement coaching, respectively (data not shown).

Figure 1.

Use and efficacy of the fellowship application advising resources. A: The percentage of survey respondents who used each fellowship application resource. B: The respondents who used each resource indicated the perceived efficacy of each resource. Responses to the right of the vertical baseline (0% axis) show the percentage of respondents who answered extremely effective or very effective. Responses to the left of the vertical baseline show the percentage of respondents who answered moderately effective, slightly effective, or not effective at all.

Perceived Efficacy of Each Resource

Of the respondents who used each resource, most found that they were extremely or very effective, including 30 of 36 (83%) who used the fellowship application guidebook, 31 of 37 (84%) who attended the Fellowship Application Information Night, 10 of 15 (67%) who used the alumni contact list, nine of 10 (90%) who used the personal statement resources, and 12 of 14 (86%) who attended the virtual interview workshop (Figure 1B). Free-text responses about the strengths and weaknesses of each resource were coded, and representative quotations are provided in Table 2.

Table 2.

Overall Strengths and Weaknesses of the Fellowship Application Advising Resources: Results From Content Analysis of Responses to Open-Ended Items

Perceived Efficacy of the Residency Program's Fellowship Application Advising

We assessed resident perception of the efficacy of the residency program's fellowship application advising both pre- and postintervention. Of note, the years considered pre- and postintervention differ based on the year the relevant resource was introduced, as noted below.

The perceived efficacy of the residency program's fellowship application advising improved from pre- to postintervention in most aspects of fellowship application advising (Figure 2A), with the following number of respondents indicating extremely or very effective advising for each domain:

Figure 2.

Efficacy of resident program advising and overall experience applying to fellowship pre- versus postintervention. A: Survey respondents were asked how effective the residency program was at providing advising about various aspects of the fellowship advising process. Responses to the right of the vertical baseline (0% axis) show the percentage of respondents who answered extremely effective or very effective. Responses to the left of the vertical baseline show the percentage of respondents who answered moderately effective, slightly effective, or not effective at all. Extremely effective and very effective were considered favorable responses, and the preintervention versus postintervention favorable responses were compared; statistical significance is indicated with an asterisk (∗), and p values are shown for each item. Of note, preintervention and postintervention years differ depending on the time that the relevant resource was introduced, as indicated. B: Survey respondents were asked about their overall experience applying to fellowships. Responses to the right of the vertical baseline (0% axis) show the percentage of respondents answering that they strongly agreed or somewhat agreed with the statement shown. Responses to the left of the vertical baseline show the percentage of respondents who answered that they neither agreed nor disagreed, somewhat disagreed, or strongly disagreed with the statement shown. Strongly agree and somewhat agree were considered favorable responses, and the preintervention (2018) versus postintervention (2020) favorable responses were compared; statistical significance is indicated with an asterisk (∗), and p values are shown for each item.

Timeline of the fellowship application process: six of 18 (33%) in 2018 versus 32 of 41 (78%) in 2019/2020, p = .002.

Writing a personal statement: six of 35 (17%) in 2018/2019 versus 14 of 21 (67%) in 2020, p < .001.

Providing subspecialty-specific advice about the fellowship application process: three of 16 (19%) in 2018 versus 18 of 32 (56%) in 2019/2020, p = .02.

Connecting with current or recent fellows: one of 14 (7%) in 2018 versus 19 of 35 (54%) in 2019/2020, p = .02.

Postinterview communication: five of 31 (16%) in 2018/2019 versus nine of 19 (47%) in 2020, p = .02.

Overall efficacy: four of 17 (24%) in 2018 versus 17 of 21 (81%) in 2020, p < .001.

Overall Experience Applying to Fellowship

We also evaluated the general experience applying to fellowship pre- and postintervention. Since resources were introduced in both 2019 and 2020, we considered 2020 as the postintervention time point. Between 2018 and 2020, there was a significant increase in the percentage of respondents who strongly or somewhat agreed that they felt supported by the residency program during the fellowship application process (seven of 15 [47%] in 2018 vs. 18 of 22 [82%] in 2020, p = .04) and those who indicated that they knew who they could turn to if they had questions about the fellowship advising process (six of 15 [40%] in 2018 vs. 20 of 22 [91%] in 2020, p = .002). There was no change in the number of respondents who agreed that applying to fellowship was stressful (13 of 15 [87%] in 2018 vs. 18 of 22 [82%] in 2020, p > 0.99; Figure 2B).

Discussion

We describe the development, implementation, and evaluation of the five-component Fellowship Application Toolkit for IM residents. Despite the importance of the fellowship application process to career trajectory, to our knowledge this is the first published description of a curriculum to guide residents through this critical phase in their training, as well as being one that can be adapted for use at other programs.

The resources in our toolkit were widely used and deemed to be efficacious. Specifically, over 60% of respondents used the fellowship application guidebook, attended the Fellowship Application Information Night, and participated in the virtual interview workshop. Four of the five resources were also rated as extremely or very effective by over 80% of survey respondents. Of the five components of the toolkit, the alumni contact list was utilized by the fewest number of respondents and also rated as the least effective. The main reason residents reported that they did not use it was because they already had contacts at other institutions. Interestingly, more respondents used the alumni contact list and felt it was effective during the 2020 virtual interview season than during the 2019 in-person application season (data not shown), so it may be particularly important if residents do not get to meet fellows in person on the interview trail. The free-text responses provided additional context about the use and efficacy of these events. For example, for the Fellowship Application Information Night, free-text responses highlighted specific strengths, such as the ability to submit anonymous questions, as well as specific recommendations for improvement, such as the desire to spend more time with fellowship PDs and a request to incorporate one-to-one time with fellowship PDs.

Not only were the individual resources deemed effective but, even more importantly, residents indicated that the overall efficacy of the residency program's fellowship advising improved after the introduction of these resources. Residents also felt more supported by the residency program and knew who they could turn to with questions postintervention. Prior studies indicated that residents desired more formal advising about the fellowship application process in their residency curriculum,[4] so our data demonstrate that these resources can achieve this goal.

Even though residents felt supported, most still felt that the fellowship application process was stressful both pre- and postintervention. Prior studies also found that IM residents reported stress associated with career planning[8] and subspecialty career choice.[16] Further research should evaluate why applying to fellowship is stressful and what countermeasures, if any, can be taken to mitigate the stress associated with this process.

In terms of feasibility, three of the five resources were introduced in 2019, and two additional resources were added in 2020, so it is possible to add events sequentially rather than trying to introduce them all in a single year. Some events were held in person and others virtually (especially in 2020 due to the COVID-19 pandemic), so both formats are possible. We found that the virtual format was ultimately easier to organize (we did not have to secure rooms, food, etc.) and also easier for both residents and faculty to attend after hours from home (or work). Costs were minimal, but there were notable investments of time required to create these resources. We are hopeful that by sharing the details of our curriculum development and evaluation, it will be easier to adapt and implement similar resources at other residency programs.

There are several limitations to this work. First, the survey response rate was between 51% and 61% per year and may have been biased towards residents who had a positive view of the resources. Second, we developed these resources for IM residents at a large university medical center, so considerations may be different in smaller programs without associated fellowship programs to provide direct support. Third, almost all residents at our program are US medical graduates who apply to (mostly) university fellowship programs; our resources and guidance were thus aimed at this specific population and may not be generalizable to all residency programs. Therefore, programs might consider a local needs assessment based on the experiences of their recent fellowship applicants to determine unique stressors or challenges that should be addressed. In the future, it will be important to determine best practices for fellowship application advising across all types of residency programs.

Conclusions

We developed the five-component Fellowship Application Toolkit for IM residents applying to subspecialty fellowships. This toolkit was highly utilized, was very effective, and led to improvements in the residents’ perceived level of support from the residency program. The toolkit can be adapted for use at other residency programs to help strengthen the advising for residents applying to subspecialty fellowships.

Elements of the Fellowship Application Toolkit.docx

Fellowship Application Guide.docx

Fellowship Application Information Night.pptx

Subspecialty Breakout Room Questions.docx

Preparing for Virtual Interviews.pptx

Mock Virtual Interview.docx

Survey Instrument.docx

All appendices are peer reviewed as integral parts of the Original Publication.