A Novel Curriculum to Prepare Internal Medicine Residents for Fellowship Interviews.

BACKGROUND: Applications to the Fellowship Match through the National Resident Matching Program (NRMP) Specialties Matching Service (SMS) are at an all-time high. Data regarding the preparedness of medical residents who go through the interview process is limited.
OBJECTIVE: To assess whether the implementation of an interview curriculum could improve medical resident preparedness for and performance during fellowship interviews.
METHODS: All third-year internal medicine residents (N = 18) at the Zucker School of Medicine at Hofstra/Northwell (Northwell) applying to subspecialty fellowship participated in an interview curriculum that comprised a didactic session and an Objective Structured Teaching Exercise (OSTE). Participants were surveyed on preparedness before and after the curriculum and medical residents and faculty were surveyed on medical resident performance after their OSTE and after their Northwell fellowship interview.
RESULTS: Out of the total possible number of participants, 16 (89%) were included in our analysis. Pre and post-test statistical differences in survey responses were evaluated using the Wilcoxon signed rank test. Medical resident preparedness and resident perceived performance increased in all measured categories, including overall preparedness (P = .001) and overall interview skills (P = .008). No significant change in faculty-rated resident performance was observed.
CONCLUSION: The development and institution of a formal interview curriculum improved medical resident preparedness and perceived performance. However, this significant improvement seen between medical resident pre and post surveys did not translate to improvement between faculty pre and post surveys. Future studies should look at fellowship match rates to objectively assess the impact of the curriculum.

Introduction

Applications to the Fellowship Match through the National Resident Matching Program (NRMP) Specialties Matching Service (SMS) are at an all-time high. In 2017, the NRMP reported 10 410 active applicants to fill 9766 positions.[1] This increase of 517 applicants from the prior year is due to the growing number of allopathic, osteopathic, and international applicants.

These statistics require internal medicine (IM) residents applying for fellowship to be fully prepared to be considered competitive candidates. In addition to a strong application with solid letters of recommendation and a commitment to research, competitive applicants need to be equipped to confidently answer the various types of questions they will be asked on their interview day.

The process by which medical residents apply for subspecialty fellowship is rigorous, often involving multiple interviews per day. According to a survey of almost 1200 fellowship directors, interactions with the faculty during the interview visit and interpersonal skills were the top 2 most cited factors when ranking applicants.[2] Data regarding the preparedness of medical residents who go through the interview process are limited. Furthermore, applicants receive little to no feedback after their interviews and are thus likely to repeat the same mistakes throughout the process. Verbal feedback from our system’s fellowship directors indicated that medical residents as a whole are unprepared to interview.

Although mock interviews have been described in medical[3,4] and pharmacy schools[5-7] as a way to prepare for residency interviews, to our knowledge, a curriculum designed to prepare medical residents for fellowship interviews has not yet been described. We aim to describe and assess the utility of an interview curriculum that we implemented at Northwell, with the goal of better preparing our medical residents for fellowship interviews and improving their chances at securing a position through the Fellowship Match.

Methods

Setting and participants

All third-year medical residents applying for subspecialty fellowship (N = 18) in the Internal Medicine Residency Program at the Zucker School of Medicine at Hofstra/Northwell (Northwell) were asked to participate in the interview curriculum in July/August 2017. One medical resident was unable to participate due to prior commitments, leaving 17 medical residents to participate in the curriculum. In addition, 1 medical resident who participated in the curriculum decided not to pursue fellowship and thus did not complete an actual fellowship interview. In total, 16 medical residents completed the interviewing curriculum and went on to complete an actual fellowship interview at Northwell.

Curriculum development

The interview curriculum was constructed through multiple sources. All of the fellowship directors at Northwell were queried to find out the most common questions that are asked on interviews. In addition, we spoke with our former third-year medical residents who had already gone through interviews to find out what they had been asked. We also used our own curriculum that we had developed for interviewing medical students for residency, which incorporates behavioral questioning. We drew heavily on our own years of experience, interviewing hundreds of candidates a year for residency. Finally, we also incorporated the guidelines discussed in Smith’s article.[8]

Program description

The curriculum began with a 1-hour didactic component in July 2017, encompassing various aspects of interview preparedness with the medical residents. The didactic component was presented by the IM program director and one of the chief medical residents and included steps that go into preparing for a fellowship interview, demonstrating appropriate interview etiquette, and anticipating the types of questions that may be asked on interview day. Following the didactic component, medical residents were given a summary sheet to review in the weeks to come (Supplemental Appendix 1). The following month, August 2017, 17 medical residents participated in an Objective Structured Teaching Exercise (OSTE). The OSTE followed the format of an actual fellowship interview. Medical residents were instructed to dress professionally and bring their curriculum vitae (CV). The OSTE included the medical resident as the learner, an IM faculty member as the interviewer, and another IM faculty member as the observer. There was 1 faculty interviewer and 1 faculty observer per resident, for a total of 34 faculty members. Of note, the faculty interviewer was not a member of the same subspecialty to which the medical resident was applying. This was done to reduce bias when the medical resident actually interviewed for their specialty of choice at Northwell. The interview lasted no longer than 20 minutes and followed a set script that the interviewers were trained to follow. The faculty observers were also given this script ahead of time. The interview was observed through a 2-way mirror by the faculty observer. Once the interview was completed, the interviewer, observer, and medical resident were given 10 minutes to fill out their surveys, after which they debriefed as a group. An additional 20 minutes was allotted for the debrief session, where feedback was provided to the medical residents from both the interviewer and the observer. A few months following the OSTE, the medical residents would then have their actual fellowship interview at Northwell in the Fall of 2017, with the fellowship director of the specialty to which they were applying.

Hypotheses

We hypothesized that self-perception of preparedness would improve over the continuum from pre-curriculum to post-mock interview and then post-actual interview. We also hypothesized that faculty would rate an improvement in the medical residents’ interview skills from the mock interview to their actual interview.

Program evaluation

Resident perceived preparedness in addition to resident perceived performance and faculty-rated performance was assessed using surveys developed for this study (Supplemental Appendices 2-8). Surveys #1 and #2 focused exclusively on how prepared medical residents felt to interview. Surveys #3, #5, and #6 evaluated medical residents’ performance on the mock interview. Finally, surveys #4 and #7 evaluated medical residents’ performance on the actual interview (Supplemental Appendix 9).

Statistical analysis

The Wilcoxon signed rank test was used to assess whether there was a difference between medical resident pre and post assessments (from Survey #1 to #2 and from Survey #3 to #4). In addition, the Wilcoxon signed rank test was used to evaluate whether there was a difference between faculty pre and post assessments (from Survey #5 to #7 and from Survey #6 to #7). A P value of <.05 was considered statistically significant. The data analysis for this article was generated using SAS software, Version 9.4 of the SAS system for Windows. Copyright © 2002-2012 SAS Institute Inc., Cary, NC. This study was deemed exempt by the Institutional Review Board of Northwell Health.

Results

Of the 18 potential participants who were initially supposed to take the interview course, 17 participants took the interview course (94%) and were included in the perceived preparedness analysis reflected in Table 1. Of the 17 participants who took the interview course, only 16 participants went on an actual fellowship interview (94%) and were included in the resident perceived performance analysis reflected in Table 2. Survey #1 evaluated medical resident preparedness prior to the interview curriculum and Survey #2 evaluated medical resident preparedness after the curriculum, but prior to their actual interview. From Survey #1 to Survey #2, the medical residents showed a statistically significant increase in all perceived preparedness parameters measured (Table 1). These measures included increases in overall interview preparedness (P = .001), knowledge of interview formats (P < .001), knowledge of behavioral questions (P < .001), knowledge of how to handle questions with an unknown answer (P = .04), and knowledge of how to answer personal questions (P = .02).

Table 1.

Medical resident preparedness.[a]

Survey items	Pre-test (N = 18)	Post-test (N = 17)	P value[b]
	n (%)	n (%)
Interview preparedness			.001
Very prepared	0 (0)	2 (11.8)
Mostly prepared	3 (16.7)	12 (70.5)
Neutral	8 (44.4)	2 (11.8)
Unprepared	7 (38.9)	1 (5.9)
Knowledge of interview formats			<.001
Yes	5 (27.8)	17 (100)
Knowledge of behavioral questions			<.001
Yes	4 (22.2)	17 (100)
Knowledge of how to handle questions with unknown answers			.04
Yes	6 (33.3)	13 (76.5)
Knowledge of how to answer personal questions			.02
Yes	7 (38.9)	15 (88.2)

Medical Resident Pre-Curriculum Preparedness Survey versus Medical Resident Post-Curriculum Preparedness Survey.

P value for the Wilcoxon signed rank test.

Table 2.

Medical resident perceived performance and faculty rated performance.

Survey Items	Medical resident perceived performance[a]			Faculty rated performance A[b]			Faculty rated performance B[c]
	Post-OSTE (N = 17)	Post-actual interview (N = 16)	P value[d]	Post-OSTE (N = 16)[e]	Post-actual interview (N = 15)[f]	P value[d]	Post-OSTE (N = 17)	Post-actual interview (N = 15)[f]	P value[d]
	n (%)	n (%)		n (%)	n (%)		n (%)	n (%)
Overall interview skills			.008			.99			.66
Excellent	0 (0)	1 (6.3)		4 (25.0)	3 (20.0)		4 (23.5)	3 (20.0)
Very good	5 (29.4)	10 (62.5)		10 (62.5)	10 (66.7)		8 (47.1)	10 (66.7)
Satisfactory	11 (64.7)	5 (31.2)		2 (12.5)	2 (13.3)		4 (23.5)	2 (13.3)
Needs improvement	1 (5.9)	0 (0)		0 (0)	0 (0)		1 (5.9)	0 (0)
Dress			.008			.13			.06
Well-groomed	8 (47.1)	16 (100)		11 (68.8)	15 (100.0)		12 (70.6)	15 (100.0)
Neat but could improve	6 (35.2)	0 (0)		5 (31.2)	0 (0)		5 (29.4)	0 (0)
Neutral	3 (17.7)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Disheveled	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Posture			.01			.22			.63
Good posture and poised	4 (25.0)	10 (62.5)		11 (68.8)	14 (93.3)		14 (82.4)	14 (93.3)
Poised but could work on posture	7 (43.8)	4 (25.0)		5 (31.2)	1 (6.7)		3 (17.6)	1 (6.7)
Neutral	3 (18.7)	2 (12.5)		0 (0)	0 (0)		0 (0)	0 (0)
Poor posture	2 (12.5)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Level of Interest			.008			.45			.11
Enthusiastic	6 (35.3)	13 (81.2)		11 (68.8)	12 (80.0)		9 (52.9)	12 (80.0)
Interested	11 (64.7)	3 (18.8)		4 (25.0)	3 (20.0)		7 (41.2)	3 (20.0)
Neutral	0 (0)	0 (0)		1 (6.2)	0 (0)		1 (5.9)	0 (0)
Disinterested	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Level of engagement			.002			.99			.99
Good eye contact and engaged	7 (41.2)	15 (93.8)		13 (81.2)	13 (86.7)		14 (82.4)	13 (86.7)
Engaged but could have better eye contact	9 (52.9)	1 (6.2)		3 (18.8)	2 (13.3)		3 (17.6)	2 (13.3)
Neutral	1 (5.9)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Avoided eye contact/poor engagement	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Responses to questions overall			.005			.4			.17
Excellent	1 (5.9)	3 (18.8)		7 (43.8)	9 (60.0)		6 (35.3)	9 (60.0)
Very good	3 (17.6)	11 (68.7)		8 (50.0)	6 (40.0)		8 (47.1)	6 (40.0)
Satisfactory	13 (76.5)	2 (12.5)		1 (6.2)	0 (0)		3 (17.6)	0 (0)
Needs improvement	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Fluency of speech			.01			.99			.79
Excellent	2 (11.8)	4 (25.0)		8 (50.0)	8 (53.3)		7 (41.2)	8 (53.3)
Very good	3 (17.6)	9 (56.2)		7 (43.8)	6 (40.0)		7 (41.2)	6 (40.0)
Satisfactory	11 (64.7)	3 (18.8)		1 (6.2)	1 (6.7)		3 (17.6)	1 (6.7)
Needs improvement	1 (5.9)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Ability to think on the fly			.001			.12			.09
Excellent	0 (0)	4 (25.0)		4 (25.0)	6 (40.0)		4 (23.5)	6 (40.0)
Very good	3 (18.7)	9 (56.3)		8 (50.0)	8 (53.3)		5 (29.4)	8 (53.3)
Satisfactory	12 (75.0)	3 (18.8)		4 (25.0)	1 (6.7)		7 (41.2)	1 (6.7)
Needs improvement	1 (6.3)	0 (0)		0 (0)	0 (0)		1 (5.9)	0 (0)
Ability to speak regarding research			.002			.38			.37
Excellent	1 (5.9)	11 (68.8)		8 (50.0)	4 (26.7)		6 (35.3)	4 (26.7)
Very good	9 (52.9)	4 (25.0)		7 (43.8)	9 (60.0)		4 (23.5)	9 (60.0)
Satisfactory	6 (35.3)	1 (6.2)		1 (6.2)	2 (13.3)		5 (29.4)	2 (13.3)
Needs improvement	1 (5.9)	0 (0)		0 (0)	0 (0)		2 (11.8)	0 (0)
Ability to speak regarding experiences on CV			<.001			.29			.29
Excellent	3 (17.6)	12 (75.0)		2 (12.5)	5 (35.7)		3 (17.6)	5 (35.7)
Very good	7 (41.2)	4 (25.0)		13 (81.3)	9 (64.3)		10 (58.8)	9 (64.3)
Satisfactory	7 (41.2)	0 (0)		1 (6.2)	0 (0)		3 (17.7)	0 (0)
Needs improvement	0 (0)	0 (0)		0 (0)	0 (0)		1 (5.9)	0 (0)
Ability to speak regarding motivation for field			<.001			.23			.09
Excellent	1 (5.8)	10 (62.5)		6 (37.5)	6 (40.0)		4 (23.5)	6 (40.0)
Very good	8 (47.1)	6 (37.5)		7 (43.8)	9 (60.0)		8 (47.1)	9 (60.0)
Satisfactory	5 (29.4)	0 (0)		2 (12.5)	0 (0)		4 (23.5)	0 (0)
Needs improvement	3 (17.7)	0 (0)		1 (6.2)	0 (0)		1 (5.9)	0 (0)
Ability to ask appropriate questions			.01			.15			.04
Excellent	2 (11.8)	6 (37.5)		5 (31.2)	7 (46.7)		5 (29.4)	7 (46.7)
Very good	4 (23.5)	6 (37.5)		7 (43.8)	8 (53.3)		7 (41.2)	8 (53.3)
Satisfactory	8 (47.1)	4 (25.0)		2 (12.5)	0 (0)		2 (11.8)	0 (0)
Needs improvement	3 (17.6)	0 (0)		2 (12.5)	0 (0)		3 (17.6)	0 (0)

Abbreviation: OSTE, Objective Structured Teaching Exercise.

Medical Resident Post-OSTE Performance Survey versus Medical Resident Post-Actual Interview Performance Survey.

Faculty Interviewer Post-OSTE Performance Survey versus Fellowship Director Post-Actual Interview Performance Survey.

Faculty Observer Post-OSTE Performance Survey versus Fellowship Director Post-Actual Interview Performance Survey.

P value for the Wilcoxon signed rank test.

16 responses of 17 distributed surveys (response rate = 94.1%).

15 responses of 16 distributed Surveys (response rate = 93.8%).

Medical resident perceived performance on the mock interview (Survey #3) was then compared with how they perceived their own performance on their actual interview (Survey #4). The medical residents demonstrated a statistically significant improvement in all performance measures, including overall interview skills (P ⩽ .01) (Table 2). On Survey #3, only 29.4% of medical residents rated their overall interview skills as being “excellent” or “very good.” This percentage increased to 69.8% on Survey #4 (P = .008). Furthermore, improvement was seen in ability to think on the fly, with only 18.7% of medical residents rating themselves as being “excellent” or “very good” during their OSTE and 81.3% of medical residents rating themselves as being “excellent” or “very good” on their actual interview (P = .001).

Faculty interviewers’ ratings of the medical residents after the mock interview (Survey #5) were then compared with fellowship directors’ ratings of the medical residents after their actual interview (Survey #7). No statistically significant differences were found in any of the parameters measured (Table 2). In terms of overall interview skills, faculty interviewers rated 87.5% (n = 14) of the medical residents as “excellent” or “very good,” whereas fellowship directors rated 86.7% (n = 13) of the medical residents as “excellent” or “very good.” The measure closest to achieving statistical significance was ability to think on the fly, with faculty interviewers rating 75% (n = 12) of the medical residents as “excellent” or “very good” compared with fellowship directors rating 93.3% (n = 14) of the medical residents as “excellent” or “very good” (P = .12).

Finally, faculty observers’ ratings of the medical residents during the OSTE (Survey #6) were compared with fellowship directors’ ratings of the medical residents during their actual interview (Survey #7). There were no statistically significant differences between how the faculty observers rated the medical residents’ performance compared with the fellowship directors on all measures except the ability to ask appropriate questions (Table 2). For the ability to ask appropriate questions, faculty observers rated 70.6% (n = 12) of medical residents as “excellent” or “very good, whereas fellowship directors rated 100% (n = 15) of medical residents as “excellent” or “very good” (P = .04). With regard to overall interview skills, faculty observers rated 70.6% (n = 12) of the medical residents as “excellent” or “very good,” whereas fellowship directors rated 86.7% (n = 13) of the medical residents as “excellent” or “very good,” but this improvement was not statistically significant (P = .66).

Discussion

Adding to prior work which has shown benefits of mock interviews for medical students,[3,4] our study suggests that the development and institution of a formal fellowship interview preparation course can improve medical resident preparedness for interviews and perceived performance during interviews. Medical resident overall preparedness increased significantly, from 16.7% prior to the course, compared with 82.4% after the course. The positive effect was also reflected in improved knowledge of interview formats and how to answer various types of questions. Medical resident perceived overall interview skills also increased significantly from 29.4% of medical residents saying their skills were excellent or very good in their mock interview to 68.8% in their actual interview.

The medical residents’ perception of improvement observed is likely multifactorial. For many medical residents, the fellowship interview process remains mysterious and prior to this study, there was no formal preparation course at our institution. A prior study looking at mock interviews for students applying to pediatric specialties found that the greatest benefit likely came from exposure to the process.[4] Increased exposure allows applicants to prepare in a more focused and less stressful manner. Our didactic session addressed a knowledge deficiency regarding pre-interview preparation, proper interview etiquette, different types of interviews, potential interview questions, and post-interview correspondence. Undergoing a mock interview experience may improve comfort and confidence levels and thus reduce stress during the real interview process. Finally, evaluation by faculty members can serve to identify weaknesses in medical residents that can then be specifically addressed on their own.

It should be noted that the faculty interviewers and observers in the OSTE rated the medical residents much higher on most performance categories than the medical residents rated themselves. Therefore, despite most medical residents receiving high ratings from the fellowship directors during their actual Northwell interview, not enough of an improvement was seen to demonstrate statistical significance in most performance categories.

We must acknowledge several limitations. This is a single center study of 1 residency program run over a single year, limiting generalizability. The sample size was small. There was no assessment by non-fellowship director faculty observers on the actual interview, whereas the OSTE was scored by both a mock interviewer and an observer. All evaluations were completed by faculty members from within our health system, which may contribute to bias. We are unable to isolate the effects of the didactic portion of the course versus the effects of the OSTE. In addition, the environment we created for the OSTE may have been different than that of the actual interview. For our surveys, some of the questions with only “Yes” and “No” answers limited diversity of the answers. Finally, in our OSTE, medical residents were not paired with an interviewer from the subspecialty the medical resident was applying to. This decision was made purposefully to avoid possible bias during the actual fellowship interview.

The next steps would be to have the same faculty member who observed a medical resident’s mock interview observe the actual fellowship interview either via 2-way-mirror as in the mock interview or via video recording. This would provide a more accurate comparison of pre and post-intervention performance. In addition, the number of evaluators in both mock and actual interviews could be increased and evaluators external to our health system could be included to provide different perspectives and feedback to our medical residents. Larger studies spanning multiple programs over several years would increase generalizability and statistical power to observe statistically significant differences. In future studies, we also plan to look at fellowship match rates as an attempt to objectively assess the effects of our interview curriculum.

Conclusion

Recent studies have shown improvement in residency match rates for medical students[3] and pharmacy students[5] after completing mock interviews. However, to the best of our knowledge, the effects of mock interviews on fellowship match rates have not been reported. The results of our study and overall positive anecdotal feedback from the medical residents, faculty, and fellowship directors suggest value in continuing and further improving our formal fellowship interview preparation course.

Click here for additional data file.

Supplemental material, MDE_Fellowship_Interview_Article__Manuscript_Journal_of_Medical_Education_Appendixes_MRS_xyz16799fe202174 for A Novel Curriculum to Prepare Internal Medicine Residents for Fellowship Interviews by Matthew Sinclair, John Raimo, Kelvin Wong, Myriam Kline and Karen Friedman in Journal of Medical Education and Curricular Development