Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW: In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential. RECENT
FINDINGS: References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.

Introduction

The annual urology match organized by the American Urological Association (AUA) in conjunction with the Society of Academic Urologists (SAU) determines where aspiring urologists will undergo their residency training. The urology residency match has long been recognized as one of the most competitive application processes within the field of medicine—a reputation that is not unearned, as the AUA reports 20% of applicants who submitted a rank list in the most recent application cycle did not successfully match [1]. In recent years, both the resident selection process and the field of urology have undergone important changes, as evidenced by the expansion in the body of literature about the residency match over the past 5 years. Searches for “urology residency” AND “application” in PubMed, Embase, and Google scholar from January 2010 to March 2020 reveal a nearly fourfold increase in publications pertaining to urology resident selection in the past 5 years (Fig. 1). Recent developments include demographic changes, with studies demonstrating that urology has had the highest rate of growth in female residents compared with other surgical fields and that rate of women matching into urology is at an all-time high [1, 2]. Concurrently, greater awareness of burnout and career dissatisfaction among practicing urologists and urology residents has resulted in an emerging profession-wide emphasis on wellness and work-life-balance, particularly during training [3, 4], which may affect how residents evaluate the strengths and fit of various training programs [5, 6]. In addition to shaping applications’ expectations for residency programs, these considerations may also underlie the recent scrutiny of the growing financial and emotional burden that the residency selection process places on applicants [5-7].

Fig. 1

Number of publications on urology residency match over time. Searches for “urology residency” AND “application” were performed on PubMed, Embase, and Google Scholar from January 2010 to present and titles and abstracts for search results were reviewed. Articles relevant to the urology residency application were divided into two 5-year periods: January 2010 to March 2015 and April 2015 to March 2020

Further complicating this picture is the recent announcement by the National Board of Medical Examiners (NBME) that starting as early as January of 2022, the organization will forego the three-digit numerical scoring system for the United States Medical Licensing Examination (USMLE) Step 1 exam in favor of a simple pass/fail designation [8]. Step 1, which the NBME describes as an assessment of medical students’ mastery of basic pre-clinical biomedical sciences, has long been viewed by many residency programs as an objective way to assess applicants [9]. In the absence of numerical Step 1 scores, programs may be compelled to identify alternative means of assessing the relative strengths of applicants. The uncertainty surrounding these changes presents an opportunity to redefine and refocus the goals of residency selection. Herein, we performed a critical review of how urology residency applicants have been evaluated and selected in recent years, with a focus on identifying unmet needs and opportunities for improvement.

Transcript and Clinical Grades

Although transcripts have traditionally been used as an objective measure of assessing applicants’ performance throughout medical school, there are several important factors that may complicate their utility in assessing prospective residents. While many institutions have transitioned to a pass/fail pre-clinical curriculum, clerkship grades, particularly in the core surgery clerkship, remain a potentially useful indicator of clinical performance and abilities [10]. However, the variability of clerkship grades across institutions has long been an issue, casting doubt on their reliability as an evaluative tool [11]. This is particularly true for the surgery clerkship, where grade inflation is common and there is no clear consensus about what constitutes an “honors”-level performance [12]. Furthermore, there have been a number of recent studies demonstrating that clinical evaluations, which often are an important component of clerkship grades, may be subject to implicit biases and may disproportionately disadvantage traditionally underrepresented groups in medicine [13-15]. Given concerns about the objectivity of clinical grades, some institutions have begun advocating for pass/fail clerkship evaluations [16].

Despite their known limitations, clinical grades remain an important component of assessing urology applicants. In a study that sampled applicants in the 2015 urology match, the number of “honors” grades was one of the few positive predictors of a successful match [17]. This was echoed in a survey of 131 urology residency program directors, who were asked what criteria were most important in assessing prospective residents [18••]. Among the 10 most important criteria, 3 included clerkship grades in urology, surgery, and non-surgical fields. In addition, over 60% and 40% of program directors surveyed said that grades in urology and surgery, respectively, were “extremely important” factors in selecting trainees. Furthermore, program directors cited membership in the Alpha Omega Alpha (AOA) honor society, an indirect indication of class rank, as the fourth-most important factor considered [18••]. There is also evidence that an applicant’s overall trend in clerkship grades can predict performance in residency. In a recent study at Mayo Clinic, the authors assigned 53 urology residents who had trained at their program an overall 10-point performance score and attempted to identify predictors of excellent performance from the cohort’s residency applications [19••]. The study demonstrated that among other characteristics, obtaining “honors” in all medical school clerkships and membership in AOA were both associated with excellent performance during urology residency. The same authors also conducted a separate, but similar study which found that “honors” in all clerkships was the strongest predictor of surgical skills, clinical communication skills, and common sense as residents [20].

One of the tools that might help better contextualize grades and transcripts is the Medical Student Performance Evaluation (MSPE) or Dean’s letter, a document that is meant to summarize a student’s academic performance throughout medical school. However, the MSPE is often limited in its utility, as most letters do not compare the performance of students against their classmates and institutions may differ in how they choose to describe applicants [21, 22]. One survey of urology program directors found that the majority considered the MSPE to be of little to no importance in assessing applicants [22].

Medical School Reputation

The movement towards pass/fail grading in both the pre-clinical and clinical settings has made the comparison of clinical aptitude between applicants more difficult. In this context, it might be expected that residency programs would begin using the relative reputations of applicants’ medical schools as an indirect means of contextualizing students’ performance. In fact, some studies have demonstrated an association between attendance at a lower-ranked medical school and a lower probability of successfully matching [17]. A separate study also found that attendance of a higher-ranked medical school was associated with better performance on the urology in-service examination among residents [23]. However, several more recent studies have also suggested that medical school prestige is a poor predictor of success in both the urology match and in residency. The Mayo Clinic study of residents’ performance found that among other attributes, coming from a medical school that was not ranked in the top 50 of U.S. News and World Report Rankings was independently associated with resident success [19••]. Similarly, a study of 850 applicants who applied to urology residency at Columbia University over a two-year period and successfully matched into urology found that nearly two-thirds of matched applicants came from a medical school ranked outside of the top 30 [24]. In the previously described survey of urology program directors, medical school prestige was only a moderately important factor in candidate evaluation and ranked below letters of recommendation, USMLE exam scores, clinical rotation grades, AOA status, class rank, and research publications [18••]. However, it is notable that a quarter of the program directors surveyed stated that medical school prestige was an extremely important factor in decisions about applicants.

The strongest association between medical school reputation and success in the urology match was identified in a 2011 study of 5-year AUA match data which found that some medical schools disproportionately match students into urology [25]. The study reported a number of attributes for these schools, including a statistically significant correlation between U.S. News and World Report medical school rankings and number of students successfully matching into urology. The authors noted that while medical school ranking could be a factor in successfully applying into urology, there were a number of other potentially unrelated explanations for why certain schools produced more urology residents, including better mentorship, strong leadership from the departmental chair and program director, and a larger, more prominent urology department.

Research Experiences

Research output is also a criterion that is used to distinguish urology applicants, although it is often given less weight than other components of the application. Nationally, surveyed program directors place research publication as the sixth most important factor in assessing applicants. This component is below urology letters of recommendation, USMLE scores, and academic performance [18••]. However, the value of research productivity was reflected in a survey of applicants to Columbia University which found that successfully publishing manuscripts listed as “submitted” on the residency application was one of the strongest predictors of match success [26]. Publications may be a more important factor for programs with a strong academic culture that expect residents to engage in scholarly activity and pursue a career in academic medicine. One study found that an applicant’s number of published abstracts was associated with matching into highly ranked programs, which are often ranked based on academic output metrics such as citations, publications, and research grants among program graduates [17]. For programs who value academic productivity among graduates, medical school research output may be a predictor of future academic activity, as having at least one publication prior to residency has been reported to be significantly associated with an eventual academic career [27]. However, when assessing research output of urology residents, the same study found that the only independent predictor of high-volume research output during residency was whether training programs had a protected research year (i.e., 6- vs. 5-year program).

There are concerns regarding emphasis on research output among residency applicants. It has been reported that up to 5% of applicants may misrepresent their research productivity, particularly among applicants with a greater number of unpublished manuscripts [26]. These findings may reflect an increasing pressure among applicants to inflate their apparent research productivity and also highlight the inherent difficulties in verifying applicants’ reported research output. Additionally, emphasis on research experiences may further favor students who attend medical schools affiliated with a higher-ranked urology program [25]. These considerations limit the utilization of research experience as a point of comparison between applicants.

Letters of Recommendation

Letters of recommendation, particularly those written by practicing urologists, may in theory offer an assessment of a candidate’s urology-specific experiences, skillsets, personal qualities, and areas of weakness. For many program directors, this is one of the most important data points when evaluating potential urology residents. In the survey of program directors, urology references were the single most important factor in assessing candidates, ranked above USMLE Step scores and clinical grades [18••]. The value that program directors assign urology references stands in contrast to the relatively low importance of Dean’s letters and non-urology letters of recommendation, with program directors rating non-surgical references as the second least important factor in match decisions, ahead of only athletic prowess [18••].

However, letters of recommendation can be a flawed metric for assessing candidates. There is considerable variability in the quality and formatting of recommendations. For instance, while a select number of institutions use templates for their letters of recommendation, most letters are written without standardization in format and word choice [18••]. Some letter writers may also use choice keywords or subtext to distinguish applicants when writing references for multiple applicants. Letter writers may also be chosen by applicants for their reputation and esteem, and not on the basis of meaningful clinical or professional relationships. The heterogeneity of reference formatting, as well as their lack of clarity in assessing applicants’ strengths and weaknesses, often makes it difficult to interpret and compare letters between candidates. Additionally, letters of recommendation may be influenced by implicit biases. A linguistic content analysis of 460 letters of recommendation submitted in the urology residency match found that letters for male applicants contained more references to personal drive, hard work, and power than letters written for female applicants [28•]. Given these findings and the importance placed on urology references, the overreliance on non-standardized recommendations may be exacerbating existing gaps in diversity among urology applicants and residents.

Interviews and Away Rotations

Both interviews and visiting sub-internship rotations allow programs to assess applicants in person, which can provide meaningful, firsthand information on how applicants engage and interact with current residents, faculty, support staff, and patients. Away sub-internships are becoming increasingly common. A recent survey of 173 applicants who interviewed at 18 institutions demonstrated that 95% of urology applicants completed at least one away rotation and nearly 70% performed more than one [29]. Additionally, completing visiting sub-internships increases the likelihood of successfully matching, as 87% of program directors report that they gave special consideration to applicants who completed an away rotation at their institution [17, 18••]. There are several concerns about the increasing reliance on away rotations, which include exacerbating the financial burden of applying into urology as well as assigning undue significance to a students’ performance during a single month at the expense of achievements over the course of multiple years of medical school [29, 30]. In addition, there are concerns about the increasing unavailability of visiting sub-internship positions given the steadily rising volume of applications received by each program or due to schedule disruptions such as the current COVID-19 pandemic. The decision of who receives limited sub-internship positions may further complicate the challenges in selecting urology residents.

In-person interviews are undoubtedly a valuable tool for identifying residents who might be successful within an institution’s culture. The Mayo Clinic study on resident performance found that having no negative comments on an interview day was a predictor of excellence as a resident [19••]. However, interviews also present challenges for both the institution and the applicant. Because of the rising number of applications that programs receive, urology training programs often find it difficult to decide how to distribute interviews and may do so on the basis of flawed metrics such as applicants’ perceived interest in the program [22]. Additionally, the interview process is financially burdensome on medical students, with several studies citing that applicants pay an average cost of $5000–7000 per interview cycle and $500 per program interview [29, 30].

Furthermore, illegal and inappropriate questions remain an issue during interviews, despite policies and protections put in place by the National Residency Match Program (NRMP) and the United States Equal Employment Opportunity Commission [30-34]. Recent studies report that 44–85% of urology applicants are asked illegal questions and that women are more likely to receive illegal questions during their interview day [33, 34]. One study reported that female applicants were almost twice as likely to be asked illegal questions than male applicants [34]. These questions most often involve pregnancy status, parental status, and future plans for children [30, 34]. These types of questions have no role in the selection of candidates. In order to eliminate illegal questions and ensure equal opportunity for all interviewees, there must be greater accountability among residency programs with strict adherence to existing codes of conduct regarding interview practices.

Volunteer and Work Experiences

Although volunteerism and work experience can be personally enriching and facilitate the development of unique skillsets and perspectives, they are often difficult to quantify and compare between applicants. There have been few studies investigating how volunteer and work experiences are assessed in the urology match process. In the survey of program directors, community service ranked above only college prestige, non-surgical references, and athletic prowess in importance [18••]. Other surgical fields have found that the quality of volunteer experiences is only loosely associated with overall application strength [35]. With the elimination of Step 1 as a screening tool, there may be a greater push to assign numerical scores to other components of the application, which may include volunteer work [36]. Developing a standardized scale to evaluate volunteer work might allow for a more balanced and holistic approach to assessing applicants.

USMLE Board Scores

Step 1 and to a lesser extent Step 2 clinical knowledge (CK) have long been used as screening tools to assign interviews and narrow down a large pool of applications. Because these exams were initially conceived as formative assessments of basic biomedical and clinical knowledge mastery and not as screening tools, there has been an emerging scrutiny of the role of these examinations in the residency application process [36-38]. The concerns about the perceived over-emphasis of Step 1 scores center on its potential role in restricting racial and socioeconomic diversity, as well as its detrimental effect on the pre-clinical learning environment [37]. This conversation has culminated in the recent decision announced on February 12, 2020, to eliminate reporting of numerical Step 1 scores as early as January 1, 2022 [8].

Recent literature has demonstrated that Step 1 has played an important role in the selection of urology residents. Higher Step 1 scores are associated with successfully matching into urology residency, with over 80% of programs using a Step 1 cutoff to assign interviews [17, 22]. Program directors consider it to be the second most important factor in assessing candidates, below only urology letters of recommendation [18••]. Additionally, a study of program directors found that having a Step 1 score of less than 220 was the single most detrimental factor for a urology application, even worse than having a previous unsuccessful attempt at matching [18••]. Despite its many limitations when evaluating potential residents and predicting future success, Step 1 scores are not without value and have been associated with performance on specialty board exams in urology as well as other surgical fields [20, 23, 39, 40].

The recently announced changes in Step 1 scoring likely portend a greater role for Step 2 CK as a method of screening applicants. In many ways, Step 2 CK is a more sensible choice of screening examination, as it is tailored towards clinical management and diagnosis in contrast to Step 1’s focus on basic science. Furthermore, the Mayo Clinic resident performance study reported that Step 2 CK scores—and not Step 1 scores—may be an independent predictor of excellent performance among residents. Regardless, the simple substitution of Step 1 with Step 2 CK would likely be a marginal improvement at best and at worst a missed opportunity to genuinely reform and improve the process of selecting urology residents. Many of the same concerns about fairness, diversity, and the impact on student learning will likely persist with a Step 2 CK-driven screening process. The elimination of Step 1 as a factor in urology admissions presents a unique opportunity to shape the match process and the moment demands leadership, creativity, and clarity (Table 1).

Table 1

Areas to improve the residency application process

Goals	Possible solutions
1. Promoting diversity in urology residency	Increase early exposure to urology during medical school.
	Better utilize social media to connect with medical students and potential applicants.
	Eliminate illegal interview and match practices to ensure that all applicants have an equal opportunity to match.
	Identify standardized metrics to quantify life, work, and volunteer experiences to better select applicants with unique perspectives and skillsets.
2. Reducing the burden on applicants	Limit the number of applications that students are allowed to submit.
	Promote greater transparency among residency programs about their relative strengths, institutional cultures, and desired applicants. Make this information readily available online.
	Focused interviews assessing candidates based on personal and professional fit.
3. Replace Step 1 with an equitable, transparent, and useful screening measure	Create a standardized, urology-specific letter of recommendation that is clear and honest about applicants’ strengths and weaknesses.
	Promote a culture that discourages simply lavishing applicants with praise in letters of recommendation and instead encourages thoughtful assessments of candidates’ readiness for residency and personal goals in urology.
	Identify validated, standardized approaches to quantify and compare all aspects of the application.

Opportunities for Improvement

Recruiting diverse classes of residents is an essential component to ensuring that patients are treated by a community of urologists with a full breadth of experiences, perspectives, and opinions. Although recruitment of female applicants has increased at a favorable rate, the overall representation of women and underrepresented racial minorities among urology residents compares poorly to other surgical fields [2, 41]. In addition to stimulating interest in and exposure to urology among these groups of students, perhaps through greater outreach using social media [42], it is imperative to remove structural impediments to successful matching [43, 44]. This will involve tightening the interview process by ensuring that policies are enforced and inappropriate behavior reported and eliminated [44]. It may also involve designing evaluative measures that better quantify and capture the diversity of life experiences and allow program directors to better assess how these perspectives and skillsets might contribute to residency programs.

Increasing diversity also involves ensuring socioeconomic diversity of trainees. There is ample evidence in recent years that the expense associated with applying to urology residency is burdensome and steadily increasing. Although match rates have generally increased, the average number of applications submitted per applicant has gradually increased each year [1]. Additionally, interview expenses have increased and are compounded by the cost of away rotations, which are becoming an increasingly important component of assessing candidates. Greater awareness of the financial burden of matching into urology has led many to call for a limit to the number of applications that medical students are allowed to submit, a proposition favored by both programs and applicants [22, 29, 45, 46]. In addition to limiting the number of applications that can be submitted, there is a need for greater transparency among residency programs. Recently, there has been a greater emphasis on understanding the changing goals and needs of trainees [4–6, 47]. Currently, there is considerable variability in how much information regarding particular programs is available to applicants [48]. In order to ensure that applicants are focusing their application efforts appropriately, it is imperative that programs provide easily accessible, detailed information about their culture, expectations for trainees, research focuses, case logs, and other important resident applicant considerations [5].

Finally, the elimination of Step 1 creates a vacuum in screening and assessing applicants that would only partially be filled by a greater reliance Step 2 CK. These changes present an opportunity for the field of urology to better dictate how medical specialties should select residents. Given the declining early exposure to urology in medical school, this may be an opportunity to simultaneously author a formal urology-specific curriculum for students and ensure that students are familiar with the field when making career decisions [49, 50]. This could be concurrently deployed alongside a formalized, objective method of assessing applicants’ performance with a focus on aptitudes and skills relevant to urology. One of the emerging proposals is a standardized, templated urology-specific letter of recommendation coupled with a greater emphasis on honesty and transparency. There is precedent in other fields such as otolaryngology and emergency medicine, which have adopted similar standardized letters of recommendation [51, 52]. In order to maximize the probability for such a standardized letter to succeed, there needs to be a cultural shift towards offering fair, honest assessments of candidates’ relative strengths and weakness, which would be an improvement over the existing reference arms race in which letter writers offer increasingly effusive, but often empty, praise. Program directors may also use this opportunity to design other methods of screening applicants, perhaps by quantifying other components of the residency application and allowing for a more holistic, fair comparison of applicants’ life experiences. Whatever efforts are undertaken, there needs to be an emphasis on ensuring that these metrics are equitable, useful, and transparent.

Conclusions

Although USMLE Step 1 scores, clinical grades, away rotations, and letters of recommendation from urologists have been the most important factors in evaluating residents in recent years, the process of urology resident selection is in a state of transition. Changes in applicant demographics, awareness of the burden on applicants, and recently announced changes to Step 1 scoring offer an opportunity to assess the current state of the urology match and devise solutions to improve equity and transparency. There is an opportunity for leadership from urology programs to improve exposure to urology, limit the financial and emotional burden of application, reduce inappropriate interview practices, and devise fair and useful metrics to assess prospective residents. Finally, this period of transition in residency selection provides a unique opportunity to engage in research on medical education and resident evaluation. Ongoing efforts to improve the process of resident selection may yield important contributions to this active field of research.