Perspectives on the quality and utility of letters conforming to the AAIM guidelines.

Background Standardized letters of recommendation (SLOR) have become common features of the medical school to residency transition. Research has shown many advantages over the narrative letter of recommendation including improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors. Currently, at least four specialties require fellowship SLORs. Internal medicine adopted its SLOR in 2017. Previous research showed fellowship program directors' satisfaction with the 2017 guidelines. Little is known about residency program directors' acceptance and adherence to the guidelines. Objectives The study sought to assess the adoption rate of each component, barriers to adoption, time commitment, and alignment with intended goals of the guidelines. Methods Anonymous survey links were posted to an internal medicine discussion forum prior to the guidelines in spring 2017 and twice following the guidelines in fall 2018 and winter 2019. Two-sample tests of proportions were used to compare respondent characteristics with known survey population data. Pre- and post-survey comparisons were assessed for statistical significance with Pearson chi-squared statistic. Results The response rate varied from 30% to 35% for each survey period. Medical knowledge, patient care, interpersonal and communication skills, professionalism, and scholarly activity were reported frequently (>96%) at baseline. Inclusion of residency program characteristics, systems-based practice, practice-based learning and improvement, and skills sought to master increased over the study period. Conclusions The new guidelines improved uniform reporting of all core competency data. Overall, the gains were modest, as many pre-survey respondents reported high rates of including components within the guidelines.

Introduction

Several specialties have reported a benefit in shifting to standardized letters of recommendation (SLOR) or letters of evaluation (SLOE) for residency recruitment. Advantages over the traditional narrative letter of recommendation (NLOR) include improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors [1-8]. Currently, at least four specialties require SLORs [9]. It is unknown whether SLORs can similarly benefit fellowships given the limited adoption of fellowship SLORs. Distinct from SLOR in its emphasis on evaluation over recommendation, the SLOE is required in emergency medicine residency applications and asks evaluators for a qualitative ranking. Internal medicine (IM) adopted a fellowship SLOR in May 2017 to encourage competency-based assessment while maintaining an advocacy function [10]. The Alliance for Academic Internal Medicine (AAIM) recommended inclusion of performance summaries for each of the six Accreditation Council for Graduate Medical Education (ACGME) competencies, mastered skills beyond program requirements, any performance-related extensions, and scholarly activities.

Our objective was to assess awareness, adherence, barriers, satisfaction, and benefits of the IM SLOR among IM residency program directors’ (PDs). We also sought to assess changes in these measures over time. Although a survey of IM fellowship directors indicated high favorability and acceptability, we sought to address a gap in the literature on IM PDs’ attitudes related to the AAIM fellowship guidelines [11].

Methods

Survey development

An AAIM expert committee of community-based and university residency and fellowship PDs developed the survey instrument. To optimize content validity, the committee drew heavily from a verbatim translation of the guidelines’ stated goals and SLOR section headers. To enhance a common understanding of the standards, including terms such as ‘skills sought to master beyond curriculum’, an explanation of terms appeared in the original AAIM white paper and was regularly broadcasted to members through available communication channels.

Members piloted the survey and recommended revisions to improve uniform interpretability and to mitigate survey fatigue. Five-item Likert response options were replaced with dichotomous responses. The length of the survey was shortened. For post-surveys, the authors adopted a categorization of program types that AAIM surveys commonly used to assess representativeness.

Setting and participants

The committee conducted three convenience samples using an anonymous Survey Monkey link (SurveyMonkey, San Mateo, CA, USA). The authors solicited participation by posting on the Association of Program Directors in Internal Medicine (APDIM) discussion forum. Respectively, 84% and 91% of ACGME internal medicine residency programs residency program directors were members of AAIM between 2017–2018 and 2018–2019 and would have received daily digests of forum content [12]. Data about total IM program populations were obtained from the ACGME website.

Released prior to issuance of the AAIM guidelines, the first survey opened from February to March 2017 and included six requests to participate. The committee conducted its first of two post-surveys (PS1) following the release of the guidelines, posting four reminders from July to August 2017. The authors conducted a second follow-up (PS2) survey from October 2018 to February 2019 and posted six reminders on the APDIM discussion forum. Frequencies are reported. PDs were not queried whether they participated in the survey previously.

Analysis

We calculated the survey population by using the number of residency programs that were AAIM members for each respective academic year. To assess the representativeness of survey data, we compared respondents to survey population using two-sample test of proportions. Data analysis was performed in Stata 14.2 (StataCorp LLC, College Station, TX; 2015) and included pre- and post-survey comparisons for statistical significance with Pearson chi-squared statistic. We obtained program types from the American Medical Association Fellowship and Electronic Interactive Database Access System online (FREIDA), excluding non-AAIM members [13]. Differences were considered statistically significant at the P ≤ .05 level. The Alliance for Academic Internal Medicine approved this study.

Results

The overall response rate was 35% for the pre-survey, 34% for PS1, and 30% for PS2. There was no difference (Table 1) between survey population and respondents based on program type except with respect to the proportion of community-based, university affiliated (CBUA) programs for pre-survey and PS1 and PS2. The percentage of CBUA programs was lower in both post-survey periods than in the survey population.

Table 1.

Comparing respondents to survey population based on program type

	TotalEligible Population (Pre-Survey)a	TotalEligible Population (Both Post-Surveys)b	Pre (2108)	First Post	P-valuec	Second Post	P-Valuec
Qualifying Question			130 (100%)	135 (100%)		116 (88%)
Program Type
Community-based	42 (11%)	55 (14%)	–	22 (17%)	0.4	22 (19%)	0.19
Community-based, University Affiliated	190 (51%)	202 (52%)	–	56 (44%)	0.02*	47 (42%)	0.04*
Other	10 (3%)	8 (2%)	–	1 (<1%)	0.31	0 (0%)	0.12
University-based	131 (35%)	127 (32%)	–	49 (38%)	0.53	44 (39%)	0.23

aN = 373.

bN = 392.

cTwo-sample test of proportions used to compare survey respondents and survey population.

*P < 0.05.

An increased percentage of survey respondents indicated using any template to write LORs (Table 2). Among template users, the second survey period showed that use (‘every time’, ‘mostly’ or ‘occasionally’) of AAIM guidelines increased to 78% (93). Based on the presurvey results, programs reported high use of several components of the guidelines prior to its release: medical knowledge (MK) (110, 96%), patient care (PC) (113, 100%), interpersonal and communication skills (ICS) (114, 99%), professionalism (P) (111, 96%), and scholarly activity (SA) (111, 96%) (Table 2). Following release, areas that experienced shifts with statistical significance included a paragraph discussing applicant’s residency program (38% to 87%), skills sought to master beyond curriculum (36% to 58%, p = 0.003), systems-based practice (SBP) (57% to 77%, p = 0.006), and practice-based learning and improvement (PBLI) (60% to 77%, p = 0.009).

Table 2.

Use of AAIM Guidelines and Their Individual Components

	Pre	Post (2018)	Post (2019)
Respondents	116	119	104	P-value*
Use of Guidelines
Used Guidelines (every time, mostly, occasionally)		93 (78%)	93 (87%)
Did not use guidelines (almost never, never)		26 (22%)	14 (13%)
Components of Guidelines Included:
Applicant Learning Milieu
Applicant charateristics	0 (0%)	112 (93%)	99 (95%)
Residency program characteristics	44 (38%)	104 (87%)	92 (88%)	0.001**
Competencies (ACGME)
Interpersonal and Communication Skills (IPC)	114 (99%)	118 (98%)	101 (98%)	0.79
Medical Knowledge (MK)	110 (96%)	115 (96%)	101 (98%)	0.57
Patient Care and Procedures (PC)	113 (100%)	117 (98%)	100 (98%)	0.26
Practice-Based Learning and Improvement (PBLI)	67 (60%)	90 (76%)	79 (77%)	0.009**
Professionalism (P)	111 (96%)	118 (98%)	100 (97%)	0.68
Systems-Based Practice (SBP)	63 (57%)	81 (69%)	78 (77%)	0.006**
Performance Management
Curtailment of clinical privileges	83 (75%)	80 (72%)	71 (73%)	0.85
Formal probation	85 (77%)	83 (75%)	76 (78%)	0.88
Training extension	75 (67%)	75 (68%)	76 (78%)	0.15
Special Notes
Scholarly activity	111 (96%)	116 (97%)	97 (93%)	0.47
Skills sought to master beyond minimum requirements	40 (36%)	69 (58%)	53 (52%)	0.003**
Suitability for fellowship	94 (84%)	98 (83%)	90 (87%)	0.76
Standardized Test Scores
In-Training Exam (ITE)	7 (6%)	5 (4%)	3 (3%)	0.5
USMLE Steps	5 (5%)	4 (3%)	2 (2%)	0.59

In addition, program directors reported increasing comfort with reporting pertinent applicant characteristics. The percentage of responses that indicated ‘major’ or ‘moderate’ positive effect increased for each effect between PS1 and PS2 (Table 3). There were statistically significant differences for ‘major’ or ‘moderate’ positive effect in ability to structure letter (45% to 60%, p = 0.045) and ability to highlight distinctive aspects of my program or hospital (50% to 56%, p < 0.001).

Table 3.

Effects of Guidelines on Letter of Recommendation Writers

	First Postsurvey (2018)			Second Postsurvey (2019)
	n=107			n = 95
	Major or Moderate Negative Effect n(%)	Neutraln(%)	Major or Moderate Positive Effect n(%)	Major or Moderate Negative Effect n(%)	Neutral n(%)	Major or Moderate Positive Effect n(%)	P-value
Ability to comment on each of the six ACGME competencies	7 (7)	63 (59)	37 (35)	6 (6)	48 (51)	41 (43)	0.449
Ability to distinguish applicants	18 (17)	59 (55)	20 (28)	10 (10)	50 (52)	36 (38)	0.224
Ability to evaluate candidates for fit for a particular fellowship	10 (9)	82 (76)	16 (15)	5 (5)	70 (74)	20 (21)	0.327
Ability to highlight distinctive spects of my program or hospital	4 (4)	50 (47)	53 (50)	1 (1)	41 (43)	53 (56)	<0.001
Ability to structure letter	10 (9)	49 (46)	48 (45)	3 (3)	35 (37)	57 (60)	0.045
Number of direct observations performed on resident	4 (4)	97 (92)	5 (5)	0 (0)	89 (94)	3 (3)	0.144

Discussion

Study findings showed a major overall shift in uniform performance reporting as recommended in the AAIM guidelines, including applicants’ ACGME core competency attainment. The majority of programs were already commenting on many areas of competency attainment such as medical knowledge, patient care, and interpersonal and communication skills. Thus, the guidelines had a modest effect on these areas. However, the template reinforced the uniform inclusion of all six competencies, as well as other areas relevant to fellowship recruitment. Reporting of SBP, PBLI, performance-based extensions, skills sought to master, and a residency program paragraph were increased. This standardized performance reporting provides a foundation for a cohesive educational handover between residency and fellowship. Although SBP and PBLI remained the least cited competencies across all surveys, lower reporting of these competencies is reported in other specialties and may reflect broader challenges in universal construct interpretation and meaningful methods of evaluation [6].

As discussed in the original white paper, the goals for publication of guidelines were to improve the ability to distinguish candidates, tailor and structure letters while highlighting relevant features of the residency training program. Overall, reporting of ‘moderate’ or ‘major’ positive effects increased over time, though at least half of respondents were neutral on the effects for four of these areas. Although we did not directly assess, the time spent on each letter may have shaped perceptions about its benefits in year one. Subsequent to the AAIM guidelines, several studies on the SLOE raised concerns about gender and racial equity [14-16]. Future research on the AAIM guidelines should seek to assess any effects of bias on applicants.

The limitations of our survey included the use of an anonymous survey link, self-selection bias, and inability to assess other factors to compare respondents to non-respondents. The difference in the proportion of CBUA programs may have also affected results. The authors sought to mitigate concerns about content validity during survey development by using exact phrases from the guidelines. However, the pilot did not collect validity evidence. This is also considered a limitation of this study.

Conclusion

The AAIM LOR guidelines resulted in modest improvements in standardized components of the fellowship LOR for many areas. Increasing mention of competency development for SBP and PBLI, as well as performance-based extension shifts the LOR toward an increased balance between advocacy and evaluation letter. The increasing positive perceptions of the guidelines by IM PDs in the post-survey period shows general satisfaction with these guidelines. The results implied an improvement in the educational transition between residency and fellowship.