Internal medicine fellowship directors' perspectives on the quality and utility of letters conforming to residency program director letter of recommendation guidelines.

Background: In May 2017, the Alliance for Academic Internal Medicine (AAIM) published guidelines intending to standardize and improve internal medicine residency program director (PD) letters of recommendation (LORs) for fellowship applicants.
Objectives: This study aimed to examine fellowship PDs impressions of the new guidelines, letter writers' adherence to the guidelines, and the impact of LORs that conformed to guidelines compared to non-standardized letters.
Methods: The authors anonymously surveyed fellowship PDs from January to March 2018 to gather input about LORs submitted to their programs during the 2017 fellowship application cycle.
Results: A total of 78% of survey respondents were satisfied with letters that followed the AAIM guidelines, whereas 48% of respondents were satisfied with letters that did not. Fellowship PDs felt that letters that followed the AAIM guidelines were more helpful than letters that did not, especially for differentiating between applicants from the same institution and for understanding residents' performance across the six core competency domains. Fellowship PDs provided several suggestions for residency PDs to make the LORs even more helpful.
Conclusion: Fellowship PD respondents indicated that LORs that followed the new AAIM guidelines were more helpful than letters that did not.

Introduction

Letters of recommendation (LORs) are consistently cited as highly important for determining whom to invite for fellowship interviews and for making ranking decisions [1-5] despite major limitations in the discriminatory function and interpretability of the traditional narrative letter of recommendation (NLOR) [5,6]. The standardized LOR (SLOR) is intended to reduce some of the limitations of the NLOR, including leniency bias, lack of a shared lexicon, low inter-rater reliability, and other known biases [6-9]. Moreover, SLORs have been shown to be more efficient to complete for the writer and easier to interpret, even for inexperienced readers [10-13].

The Resident to Fellow Interface Committee (RFIC) of The Alliance for Academic Internal Medicine (AAIM) published guidelines in May 2017 that sought to structure the format and content of internal medicine (IM) residency program directors’ (PDs’) LOR to create a SLOR [5]. These guidelines recommend a brief description of the applicant’s residency, the resident’s performance in each of the six core competencies, a summary of scholarly work, unique personal characteristics, any performance or professionalism concerns, a statement of the resident’s overall suitability for training in their chosen subspecialty, and a length of two pages.

This study sought to assess fellowship PDs’ opinions about the IM residency program letters they received during the 2017 fellowship application season, which followed the publication of the AAIM SLOR guidelines.

Materials and methods

Members of the RFIC created a survey via SurveyMonkey (SurveyMonkey, San Mateo, CA, USA) with the aim of assessing fellowship PDs perspectives and to obtain feedback on whether modification of the guidelines is warranted. We disseminated the survey in multiple ways. First, the link was posted on the IM residency PD discussion forum with instructions for residency PDs to forward the survey link to the IM fellowship PDs at their institution; the survey prompt was sent via the discussion forum five times. Second, AAIM sent the survey request directly to fellowship PDs twice via email, once in December 2017 (780 recipients) and again in January 2018 (759 recipients). Finally, a request for survey participation was placed in the AAIM Connection, a weekly e-newsletter for all AAIM members, in January 2018.

Survey responses were collected from December 2017 to April 2018. A prescreening question to verify role as fellowship PD status determined qualification to complete survey. Comparing conforming to nonconforming letters, respondents were asked to assess helpfulness using a Likert scale (‘extremely less helpful,’ ‘less helpful,’ ‘about the same,’ ‘more helpful,’ ‘extremely more helpful’); these results were collapsed to ‘more helpful’ (which included ‘extremely more helpful,’ ‘more helpful’), ‘less helpful’ (included ‘extremely less helpful,’ ‘less helpful’), and ‘about the same.’ Respondents were also asked to assess helpfulness of LOR guideline components, with response options consisting of ‘very helpful,’ ‘mostly helpful,’ ‘slightly helpful,’ and ‘not at all helpful.’ Satisfaction was rated on a Likert scale (‘very satisfied,’ ‘satisfied,’ ‘about the same,’ ‘dissatisfied,’ ‘very dissatisfied’), with ‘satisfied’ used to describe answers of either ‘satisfied’ or ‘very satisfied.’ The survey included three open-ended questions, with one soliciting suggestions to improve upon the AAIM guidelines, another asking for perceived benefits, and the third asking for overall comments or feedback. One author, ABO, performed the qualitative analyses. Narrative responses were reviewed with classification groupings created by theme in parallel, then responses were re-read and classified into the different themes as appropriate.

Results

Of the approximately 770 fellowship PDs listed in the AAIM membership database, 200 respondents indicated that they held the role of fellowship PD and were qualified to complete the survey (26%). The most represented specialties among the respondents were cardiovascular medicine (20%), gastroenterology (18%), rheumatology (15%), and nephrology (15%). A total of 83% of respondents described their program as being based at a ‘university hospital.’

A total of 56% of respondents were aware that AAIM had developed guidelines for IM residency PD LORs. A majority of respondents reported that most IM PDs followed most but not all of the guidelines. Fellowship PDs reported that letters that followed the AAIM guidelines were more helpful than letters that did not in the following areas: meaningfully comparing applicants, differentiating applicants from the same institution, understanding residents’ performance across the six core competency domains, and consistency in terms used to describe resident (Table 1).

Table 1.

Fellowship PD responses to: ‘Compared to letters that did not follow the AAIM guidelines, how helpful were letters that followed the guidelines in the following areas?’.

	More Helpful	About the Same	Less Helpful
Differentiate applicants from the same institutions	60%	36%	4%
Performance in six core competencies	59%	32%	10%
Consistency in terms used to describe resident	57%	36%	7%
Meaningfully compare applicants	52%	43%	5%
Assessing overall qualification for your program	50%	44%	6%
Selecting resident to interview	45%	50%	5%
Differentiate applicants from different institutions	44%	47%	9%
Ranking applicants	36%	54%	9%
Assessing fit for my program	37%	53%	10%

Overall, 78% of respondents reported being satisfied with letters that followed the AAIM guidelines (17% of these reported being ‘very satisfied’), whereas 48% of respondents reported being satisfied with letters that did not conform to the AAIM guidelines (no respondents selected ‘very satisfied’). More than three-fourths of fellowship PD respondents rated the following elements of the guidelines as being mostly or very helpful in making recruitment decisions: applicant unique characteristics (e.g. level of engagement, degree of initiative) (88%), serious performance-related issues (performance-based extension in training, curtailment of clinical privileges, and formal probation, if it occurred) (83%), and applicants’ scholarly contributions (78%) (Table 2).

Table 2.

Fellowship PD responses to: ‘Please rate the overall helpfulness of the following elements of the AAIM guidelines to your recruitment-related decisions.’.

	Mostly or Very Helpful	Slightly Helpful	Not at All Helpful
Applicant unique characteristics (e.g. level of engagement, degree of initiative)	88%	9%	3%
Serious performance-related issues (performance-based extension in training, curtailment of clinical privileges, and formal probation, if it occurred)	83%	12%	5%
Applicant’s scholarly contributions	78%	20%	3%
Skills sought to master beyond resident requirement	73%	24%	3%
Suitability for subspecialty	64%	29%	7%
Applicant’s achievement in all six core competencies	53%	41%	6%
Paragraph describing residency program	46%	46%	8%

One quarter of the 60 respondents who provided suggestions to improve upon the AAIM guidelines mentioned the desire for a standardized, summative ranking scale applied to all applicants – this was by far the most frequent suggestion. Some additional respondents noted that, even with the new guidelines, discriminating between different applicants is difficult (e.g. ‘all applicants sound good’). Several respondents did not find that addressing each core competency in the letter was helpful, and that doing so can obscure applicants’ unique strengths and weaknesses, as well as make the letters all sound similar. Several writers felt that the letters are too long, with some mentioning that addressing all six competencies and also providing a program description produces unnecessary length that sounds similar across different residents. Other respondents wanted greater emphasis on the applicants’ personal characteristics, including their work ethic, collegiality, empathy, motivation for inquiry, and response to difficult situations. A few comments expressed the desire to have serious professionalism issues highlighted more directly and for letter writers to address the question ‘would you want this applicant to take care of a family member of yours?’ A few respondents asked for greater publicity around the guidelines to help more residency PDs use them. Some representative suggestions for improvement in the AAIM guidelines are included in Table 3.

Table 3.

Representative comments from fellowship PDs in response to ‘What suggestions do you have to improve upon the AAIM guidelines?’.

‘We received 549 applications for three positions in our cardiology fellowship program. Letters have to have a bottom line statement. For example, ‘In comparison to all trainees I have trained in the past X years, this resident is among the top X%’ and ‘In comparison to trainees in this resident’s graduating class, Dr. X is in the top X%.’

‘It would be helpful to have a real grading scale but none of us believe it would be “real” – everyone thinks their trainees are in the top 10–25%, unless they’re in the top5%.’

‘Including the six competencies specifically does not add significantly to the overall assessments of strengths and weaknesses.’

‘Quantitative focus on competencies is helpful in distinguishing between top applicants and the others, but it’s not at all helpful in distinguishing strong applicants’

‘Uniformity in letters makes it more difficult to discern unique characteristics of the applicants, and to discern subtleties in the descriptions provided. There is more of a tendency for letter writers to cut and paste large blocks of verbiage that obfuscates the ability of the reader to determine the types of characteristics of the applicants that are important.’

‘I am looking for residents who have developed motivation for inquiry, shown empathetic behavior, gone up and beyond the assigned work and worked will in teams. Please incorporate this feedback in the letters very discretely.’

Fifty-six respondents provided interpretable responses to ‘What benefits did you find with the AAIM guidelines?’ Four themes emerged, each representing between 15% and 20% of comments: helpfulness of the program description; completeness of information; comparability of applicants, particularly those from the same program; that there were no or minimal benefits. In response to ‘Please provide any overall comments or feedback,’ there were 40 interpretable responses. Of these, 15–20% of responses were classified into each of these most common themes: ‘thank you/great work/helpful,’ uncertainty about the helpfulness of the guidelines, and ongoing indistinguishability of applicants.

Discussion

In this survey, IM fellowship PDs’ impressions of program letters that used the new AAIM Guidelines for a Standardized Fellowship LOR were positive: fellowship PDs were much more likely to be satisfied with letters that followed the guidelines than letters that did not. Similarly, letters that followed the guidelines were considered to be more helpful in a number of domains that allowed for discrimination between applicants from the same institution, judging the applicant’s performance in the six ACGME competencies, and providing meaningful comparisons between applicants. Very few respondents (≤ 10%) indicated that conforming letters were less helpful than nonconforming letters in any of the domains discussed in Table 1, or that any element of the new format (listed in Table 2) was ‘not at all helpful.’

We found that fellowship PDs’ impressions of the new guideline ranged from gratitude and praise for the guideline, including the completeness and description of residency programs, to skepticism and criticism of these same components by other PDs. We received suggestions for improvement in the structure of PD LORs. Throughout the years-long deliberation to create guidelines, it became apparent that no single letter type would satisfy all fellowship PDs. Several comments raised concerns that letters following the new guidelines were too long and too similar to each other. The guidelines do include a target length of only two pages, yet fully addressing all of the components included in the guidelines may make this length difficult to achieve.

The concept of providing fellowship programs with a class rank for each applicant is not new. In fact, it figured prominently in discussions during the development of the guidelines and, at times, caused discussions to stall. Many residency PDs oppose ranking on practical and theoretical grounds. Prior efforts to enforce a ranking component of a standardized LOR have led to misleading inflation; for example, in a sample of standardized faculty LORs for emergency medicine residency, > 95% of applicants were ranked in the top third relative to their peers [13]. A formal ranking of residents may also result in a more competitive, less collegial residency environment that could be deleterious to learning and resident wellness [14-16]. Finally, as grading schemata are inherently subjective, the residency PD priorities – including weighting of particular applicant characteristics – may not match the priorities of the fellowship PD, rendering the grading system unhelpful in judging fit for a particular fellowship.

This study has several limitations. The survey dissemination resulted in a ‘convenience sample’ that may not represent the full population of fellowship PDs; it is possible that PDs supportive of the new guidelines might have preferentially responded, but it is also possible that critics of the new SLOR format may have been more likely to respond. The sample size was smaller than ideal, which precluded analyses comparing different subspecialties. Our respondents were providing impressions immediately following the first application season after the guidelines were released. Many fellowship PDs were not aware that the guidelines were in place, which limited the usefulness of their input.

On the whole, our results support the continued use of the new guidelines, with fellowship PDs rating letters that followed the guidelines more favorably. Ongoing education is needed to raise awareness and adoption of the guidelines by residency PDs for subsequent recruitment cycles. With continued promotion of the guidelines, adherence to the guidelines is likely to improve during the first few years of implementation, as has been seen previously described with similar AAIM LOR efforts for residency [17].