Gender-based differences in letters of recommendation in applications for general surgery residency programs in Canada.

BACKGROUND: In Canada, residency programs do not have many objective measures for ranking candidates. Instead, ranking relies on subjective measures such as letters of reference, which can be affected by the genders of the writer and the applicant. Our study assesses letters of recommendation for a general surgery program in Canada to categorize differences in reference letters based on the genders of applicant and letter writer.
METHODS: We assessed 215 reference letters from 51 general surgery candidates for systematic differences in the descriptors used for male and female applicants and differences based on male and female authorship.
RESULTS: Female applicants were more often described as mature, pleasant and flexible. Male applicants were more often described as having initiative, completing research, earning awards and performing extracurricular activities. Female writers were more likely to highlight an applicant's interest, initiative, response to feedback, knowledge of their limits, flexibility, communication, achievement in research and awards, confidence and ability to be a good assistant. Significantly more female applicants had female letter writers, compared with male applicants.
CONCLUSION: These differences may affect the acceptance of applicants based on their gender and the genders of people who recommend them. Future research is required to explore how these differences in how applicants are described may affect residency selection committees' perceptions and rankings of applicants.

Various aspects of residency program applications have been shown to be affected by the genders of both applicants and writers of recommendation letters. There is growing concern in the surgical community that linguistic bias within these letters may be affecting how applicants are perceived and how likely they are to be accepted by a program.1–6

Several American studies in surgical fields have shown systematic differences in reference letters written for applicants of different genders.1–6 Men were often described in terms of leadership, scholarship and achievement, or potential as a future leader.1,2 Physical descriptions, “doubt raisers” (e.g., faint praise), descriptions of work ethic, and reference to feelings and family life were most often applied to female applicants.1,2

In Canada, residency programs do not have percentage-based course grades or board examinations as objective measures of ranking candidates. Instead, ranking relies heavily on subjective measures (i.e., interviews, electives and letters of reference) as the main determinants of offering residency positions.7 Thus, gender bias may have an even greater impact on applicants in the Canadian context.

The differing semantics used according to gender may alter how applicants are assessed at key application moments. In our study, we sought to assess letters of recommendation for a general surgery residency training program in Canada to categorize characterizations of the applicant based on the genders of the applicant and the letter writer. We hypothesized that there are key differences in descriptors used for male and female applicants and by male and female writers, and that these differences will conform to previously described biases. The interaction between the gender of the applicant and of the writer may identify bias in how letters may be perceived by residency selection committees.

Methods

We invited a cohort of general surgery residency candidates who attended an interview in the 2018 residency selection cycle at Western University in London, Ontario, Canada, to participate. A research associate approached potential participants to request their consent for examination of their reference letters after the match cycle was complete and all general surgery residency spots across Canada were filled.

Once the residency match was complete, we accessed the reference letters via the Canadian Resident Matching Service (CaRMS) secure website. We converted each letter from PDF to text format in Microsoft Word (Microsoft, version 16.28). We then labelled the letters as male or female based on the gender of the applicant (qualified by pronoun use in the letter) and deidentified all personal information. We removed headings and salutations, and determined the letter’s word count.

Qualitative analysis

We employed a grounded theory, qualitative research methodology in which the 4 authors reviewed all the letters. We developed codes of applicant descriptors and used an iterative process to generate themes of descriptors individually. We then combined and distilled all identified themes into a final comprehensive list by consensus.

We then divided the list of applicants among the study authors, and each individual applicant letter was coded by 2 people independently. The authors independently read the letters and documented the occurrence of the pre-identified themes in each letter. They also recorded gender of the applicant; gender of the letter’s writer; frequency of uses of first name, last name, and the title “Dr.”; as well as any phrases that were felt to be unusual or not captured by the prespecified categories. For each section, a third author then went through the collected data and adjudicated any discrepancies.

Statistical analysis

We performed data analyses using SPSS (IBM, version 26). We used χ2 analysis to identify any significant differences in the frequency that each theme appeared in reference letters based on applicant gender.

In addition, we used RStudio (version 1.1.456) to perform text-mining of the letters, split by gender of the applicant, to identify phrase frequencies for letters of male and female applicants.

Ethics approval

The study was approved by the Research Ethics Board at Western University (REB No. 111631).

Results

For the application cycle of 2017–2018, 60 applicants were selected to be interviewed for our academic general surgery residency program. Of these, 51 applicants provided consent to participate in the study. Each applicant had between 3 and 5 reference letters, for a total of 215 letters for examination. Overall, 126 (58.6%) letters were from applicants who identified as female and 89 (41.4%) were from applicants who identified as male. Of the 215 letter writers, most were men (n = 161, 74.8%) and a minority (n = 54, 25.1%) were women.

Significantly more female applicants had female letter writers (n = 41, 32.5%) than male applicants (n = 13, 14.6%, p = 0.002) (Figure 1).

Fig. 1

Proportion of letters of recommendation written by male and female writers based on the gender of the applicant.

There were no differences in length of letters provided for female and male applicants (361 v. 364 words, p = 0.88). Female writers wrote significantly longer letters than male writers (411 v. 345 words, p = 0.011).

Qualitative results

We identified the following themes: knowledge, intelligence, interest, natural ability or intuition, preparedness, “functioned as a resident,” clinical skills, note-taking, efficiency, initiative, response to feedback, dedication to specialty, hard-working, self-sacrificing, sense of responsibility, maturity, technical skills, teamwork, integration, professionalism, judgment, positive disposition, “knew their limitations,” young man or woman, appearance, shyness, humble, enthusiasm, polite or respectful, flexibility, attitude, communication skills, research or awards, desire to have in writer’s own training program, kindness, confidence, endorsement or comparison to other students, good future resident, good future surgeon, extracurricular activities, personal information about applicant, good assistant, patient-centred care, leadership and teachability. In addition, we further qualified technical ability as average or no concerns, above average, and superb or outstanding.

Differences based on gender of applicant

Female applicants were more likely than male applicants to be described as mature (χ2 = 5.922, p = 0.02), pleasant (χ2 = 6.247, p = 0.01) and flexible (χ2 = 5.869, p = 0.02). In contrast, male applicants were more likely to be described as having initiative (χ2 = 4.819, p = 0.02), earning research and awards (χ2 = 4.851, p = 0.03) and performing extracurricular activities (χ2 = 7.949, p = 0.005). We did not observe any differences in the frequency of other identified descriptors (Table 1), nor did we observe significant differences by applicant gender in how technical skill was described (χ2 = 0.206, p = 0.9).

Table 1

Frequency of descriptors in letters of reference

Descriptor	No. (%) of letters containing descriptor		p value
	Female applicantsn = 126	Male applicantsn = 89
Knowledge	104 (82.5)	74 (83.1)	0.7
Intelligence	29 (23.0)	24 (27.0)	0.5
Interest	75 (59.5)	56 (62.9)	0.6
Natural ability or intuitiveness	17 (13.5)	9 (10.1)	0.5
Preparedness	33 (26.2)	30 (33.7)	0.2
Functioned like a resident	53 (42.1)	39 (43.8)	0.8
Clinical skills	78 (61.9)	52 (58.4)	0.6
Clinical notes	19 (15.1)	18 (20.2)	0.3
Efficient	29 (23.0)	16 (18.0)	0.4
Initiative	36 (28.6)	14 (15.7)	0.03
Responsive to feedback	14 (11.1)	7 (7.9)	0.4
Dedicated to specialty	58 (46.0)	44 (49.4)	0.6
Hard working or dedicated	88 (69.8)	61 (68.5)	0.8
Self-sacrificing	33 (26.2)	32 (36.0)	0.1
Sense of responsibility	31 (24.6)	25 (28.1)	0.6
Mature	38 (30.2)	14 (15.7)	0.02
Technical skills	89 (70.6)	68 (76.4)	0.3
Teamwork	77 (61.1)	64 (71.9)	0.1
Integration (“fit in”)	28 (22.2)	19 (21.3)	0.9
Professional	35 (27.8)	25 (28.1)	1.0
Judgment	49 (38.9)	42 (47.2)	0.2
Pleasant or positive disposition	55 (43.7)	24 (27.0)	0.01
Knew their limitations	9 (7.1)	5 (5.6)	0.6
Young man or woman	10 (7.9)	10 (11.2)	0.4
Appearance or smile	5 (4.0)	2 (2.2)	0.5
Quiet, shy, understated or reserved	13 (10.3)	10 (11.2)	0.8
Humble	4 (3.2)	4 (4.5)	0.6
Enthusiasm	30 (23.8)	21 (23.6)	0.7
Polite or respectful	16 (12.7)	17 (19.1)	0.2
Flexible	8 (6.3)	0 (0.0)	0.02
Attitude	28 (22.2)	15 (16.9)	0.3
Communication	54 (42.9)	37 (41.6)	0.9
Research and awards	53 (42.1)	51 (57.3)	0.03
Desire to have in training program	56 (44.4)	38 (42.7)	0.8
Kind or compassionate	25 (19.8)	12 (13.5)	0.2
Confident	14 (11.1)	4 (4.5)	0.08
Comparison to others	98 (77.8)	72 (80.9)	0.6
Good future resident	55 (43.7)	30 (33.7)	0.1
Good future surgeon	36 (28.6)	28 (31.5)	0.6
Extracurricular or “well rounded”	12 (9.5)	21 (23.6)	0.005
Personal facts	11 (8.7)	14 (15.7)	0.1
Good assistant	22 (17.5)	16 (18.0)	0.9
Patient-centred care	17 (13.5)	8 (9.0)	0.3
Leadership	21 (16.7)	14 (15.7)	0.8
Teachable	16 (12.7)	8 (9.0)	0.4

Differences based on gender of letter writer

Female writers were more likely to use descriptors that highlighted an applicant’s interest (χ2 = 6.812, p = 0.009, initiative (χ2 = 7.674, p = 0.006), response to feedback (χ2 = 3.895, p = 0.048), flexibility (χ2 = 6.174, p = 0.013), communication skills (χ2 = 3.824, p = 0.05) and confidence (χ2 = 13.533, p < 0.001). They commented on how applicants’ knew their limits (χ2 = 4.930, p = 0.03), completed research and earned awards (χ2 = 4.686, p = 0.03) and made good assistants (χ2 = 9.448, p = 0.002). Male writers did not use any descriptors more frequently than female writers. There were no differences in the frequency of other descriptors.

Text-mining analysis

Text-mining software provided the 10 most frequently used phrases in letters of reference (Table 2). Both male and female applicants were often described as having good technical skills, being like a junior resident and being above average. They were also both commended on their knowledge base, work ethic, communication skills and their ability to provide patient care. Given the different number of male and female letters, it is difficult to compare frequency data by gender.

Table 2

Phrases identified in reference letters to general surgery based on gender of applicant*

Rank	Female applicant			Male applicant
	Phrase	Frequency	Proportion of letter’s words, %	Phrase	Frequency	Proportion of letter’s words, %
1	Technical skills	44	1.61	Knowledge base	31	1.61
2	Junior resident	38	1.39	Technical skills	31	1.61
3	Knowledge base	38	1.39	Work ethic	31	1.61
4	Work ethic	38	1.39	Communication skills	21	1.09
5	Hard working	34	1.24	Junior resident	20	1.04
6	Above average	31	1.14	Cognitive skills	16	0.83
7	Communication skills	29	1.06	Patient care	15	0.78
8	Pleasure to work	23	0.84	Research projects	15	0.78
9	Patient care	21	0.77	Above average	13	0.68
10	Support of application	20	0.73	Problem solving	13	0.68

The frequency is the number of times this phrase occurred in a combined document of all female or male applicant letters. The proportion of the letter’s words shows the percentage of the total words in the combined document of all female or male applicant letters that are the phrase words.

Female applicants were more commonly described as being hard-working, a pleasure to work with and having support for their application. Male applicants were more often described in terms of their cognitive skills, research projects and problem solving.

Discussion

In our study of letters of references for applicants to a general surgery residency program in Canada, we observed differences by gender of both applicants and letter writers.

A Canadian perspective on gender biases in letters of recommendation to surgical program admissions committees is a critical addition to our understanding of how implicit gender bias may affect applicants’ opportunities to match to a surgical training program. In the United States, surgical applicants have numeric grades from medical school, as well as scores from the United States Medical Licensing Examination. These provide objective data and cut-offs for ranking of applicants, in addition to subjective data.8 In the Canadian system, the applicants receive pass or fail scores from their medical schools and have no standardized test scores at the time of application to residency programs. Eneh and colleagues7 assessed the importance of different components of CaRMS applications to the staff who adjudicate applicant ranking. After consideration of interview scores and elective participation, letters of reference were ranked as the most crucial factor to reviewers.7 Our findings suggest there may be subtle biases that could affect adjudication of applications to general surgery programs in Canada.

In most areas, the letters of reference for applicants to our general surgery residency program were similar across both genders. However, we did identify some differences. Female applicants were more likely to be described as mature, pleasant, flexible, and hard-working, whereas male applicants were more likely to be described as having initiative, good cognitive skills, being problem solvers, earning research and awards and performing extracurricular activities. These findings are consistent with established research in medicine showing that men are more often described by their research, skills and abilities, whereas women are described as teachers and nurturers.9,10

The relative dearth of information in this field has been recognized and multiple surgical centres have begun to investigate the reference letters that are relied upon so heavily. The fields of ophthalmology, general surgery, orthopedics, transplant and urology are among those in which letters of reference have been evaluated.1–3 In these studies, men were described by their ability, achievements, awards, leadership and scholarship, as well as their power and ability to be a leader.1–3 In contrast, female applicants were more frequently described for their work ethic and by “grindstone” adjectives that emphasize effort over ability (e.g., hard working). They were also referred to as delightful or enthusiastic, more “feeling” (e.g., caring or compassionate) and their letters had more references to their family.1,2

Interestingly, these findings are not universal. Many of these studies were unable to find significant differences between letters based on gender.4,5 Several potential explanations for such findings have been suggested. A lack of differences may mean that letters accurately represent applicants or may suggest that some surgical fields are beginning to pay more attention to the descriptors they use in their letters and the potential for bias.5 Another very likely explanation is that letter writers may reuse a template for multiple applicants, regardless of gender, and make very few modifications.4

Some of our findings differ from those of previous research. A review of letters provided to an otolaryngology program in 2008 found that female applicants more frequently had their appearance described, and male applicants were more likely to have their gender term applied (e.g., “young man”).11 Neither of the more recent studies, including ours, found this difference, which may show that overt biases, such as use of physical descriptors, are no longer socially acceptable but that more subtle biases persist.1

In our study, letters for female and male applicants were of similar length. This differs from the findings of previous studies, which have consistently shown that male applicants receive longer letters.1,8 Importantly, it has been previously shown that the letters rated most highly by general surgery program committees were twice the length as lower-rated letters.8 The discrepancy in our study may be an artifact of writers using the letter structure recommended by CaRMS, which tends to be longer than the free-form letters used by writers in other studies, and which includes specified topic headings. These data points may have skewed our interpretation of average letter length.

Interest in recruiting the applicant to the writer’s own program in a letter of reference is known to be one of the “most telling positive factors on a student’s behalf.”8 This was seen with similar frequency in letters for both male and female applicants in our study, as well as in other recent publications.1

We found that gender of the letter writer was correlated with use of different qualifiers. Female writers were more likely to describe an applicant’s level of interest, initiative and response to feedback; suggest the applicant knew their limits, was flexible or was a communicator; and comment on their research and awards, confidence and ability to be a good operative assistant. In the field of otolaryngology, female letter writers were more likely to mention that the applicant was a team player and compassionate, and less likely to write a letter of minimal assurance (i.e., a letter that shows a limited understanding of the applicant’s qualifications).11 The lack of difference by gender in the use of “team player” in our study may be owing to the understanding that general surgery is a very team-oriented specialty and, as such, teamwork was almost universally mentioned in applicant letters.

An unexpected finding of our study is the small number of female writers compared with male writers, as well as the tendency for female applicants to have more letters of reference from female writers than male applicants. Only 25% of writers were women, but women represented 32.5% of letter writers for female applicants. In general, the lack of female academic general surgeons and female surgeons in leadership positions may explain why there were fewer female writers. A recent review of academic general surgeons in Canada showed 27.4% were women.12 This may confound our analysis, as any differences in female applicants’ letters may also be attributable to the predominance of female writers for this subgroup. Additional exploration is required to understand what factors motivate female applicants to approach female staff for letters of recommendation. Possible reasons for female applicants seeking out letters of reference from female writers include the enhanced approachability of a supervisor of the same gender or mentorship opportunities, both formal and informal.

Future research is required to explore how different descriptors used in residency letters might affect the selection committee members’ perceptions of applicants’ abilities and suitability for general surgery residency, and how this might translate into the ranking of applicants. Ways to mitigate these differences must also be investigated. Standardized letters of reference,13 as well as gender-bias calculators (https://slowe.github.io/genderbias/), have been proposed.14 In addition, further bias training for staff and residents writing these letters may be a way to diminish the biases that enter these letters, likely without conscious intent. Staff write letters for fellowship applications and job recommendations, as well as for residency applications, and overarching education would mitigate what is likely a broader issue. We must acknowledge that there are other lenses through which applicants may be perceived, including race, ethnicity and sexual orientation. These deserve the same level of research engagement as gender to fully explore and confront application biases if programs continue to use subjective means of assessing applicants such as reference letters.

Limitations

Limitations to our study include the time-restricted scope as we focused on a single application year of a single institution; however, students applied from multiple institutions across Canada. In addition, the applicants approached for this study were preselected for interview at our institution based on their application and as such, it may be biased toward a high quality. Unfortunately, this step could not be circumvented for privacy reasons. Lastly, our aim was to objectively assess descriptors, which entailed placing each adjective into a predefined category. This approach enabled statistical analysis of the data, but meant we could not capture a wealth of phrases that stood out and made an impression on the group. These phrases may affect how a selection committee viscerally feels about the candidate’s application (e.g., “they are … tenacious and walk the walk,” “she has poise … she is a shining star”). In addition, the manner in which we grouped phrases may have affected the statistical analysis. For example, referencing the applicant’s home medical school was equivalent to discussing their family, home life or country of origin, as all of these were noted as “personal details.” This may have prevented more subtle differences from being captured.

Conclusion

We observed differences in how applicants to general surgery residency programs were described according to the genders of both applicants and letter writers. These differences may affect the acceptance of applicants. Future research is required to explore how these differences may affect the perceptions of residency selection committees and their rankings of applicants.