Impact of Doximity Residency Navigator on Graduate Medical Education Recruitment.

OBJECTIVE: To assess residency applicants' use and perceptions of Doximity Residency Navigator (DRN) and to analyze the impact of Doximity reputation rankings on application, interview acceptance, and match list ranking decisions. PARTICIPANTS AND METHODS: We developed and distributed a survey seeking feedback from residency applicants to describe their use of DRN during the 2017 residency recruitment and match process. The dates of the study were March 1, 2017, through May 8, 2017.
RESULTS: We received responses from 2152 of 12,617 applicants (17%) across 24 graduate medical education programs. Sixty-two percent of respondents (n=1339) used DRN during the residency application, interview, and match list process. Doximity reputation rankings were noted to be valuable or very valuable to 78% of respondents (958 of 1233). Overall, 79% of respondents (977 of 1241) reported that Doximity reputation rankings influenced their application, interview acceptance, or match list ranking decisions. When asked about the accuracy of Doximity reputation rankings, 56% of respondents (699 of 1240) believed that rankings were slightly accurate or not accurate. The most commonly used resources to research potential residency programs were residency program websites, American Medical Association resources, and DRN.
CONCLUSION: Most survey respondents used DRN during the application, interview, and match ranking process. Doximity reputation rankings were found to be the most valuable resource in DRN, although more than 50% of responders had doubts about the accuracy of reputation rankings.

The process of selecting residency programs to submit applications to and interview with can be a daunting task for those seeking graduate medical education (GME) positions. Applicants have a variety of resources available to research potential residency training programs. The National Residency Matching Program (NRMP) surveyed applicants to residency programs in 2015 and found residency program reputation, geographic location, interview day experience, and perceived goodness of fit to be the most important factors that applicants considered when applying to and ranking residency programs. Accurate assessment of a residency program's reputation has long been a difficult task due to varying degrees of interpretation and bias.

Doximity is the largest social networking application for health care professionals and medical students. The Doximity Residency Navigator (DRN; Doximity, Inc) was developed to “help medical students make informed residency decisions and to increase transparency in the residency match process.” The DRN provides several tools to help applicants research prospective training programs, including reputation rankings for residency programs across multiple specialties, resident and alumni satisfaction surveys, and objective data (eg, training program size, board certification rate, sex balance, alumni publication data).

Previous studies have called into question the validity of Doximity reputation rankings of residency programs due to the lack of objective and outcome-based data used to formulate these rankings.3, 4 Nonetheless, previous studies using surveys have shown that Doximity reputation rankings influence applicants' behaviors when applying to and ranking residency programs.5, 6 To date, there are no large cohort studies evaluating the impact of DRN on medical students' residency selection across a wide distribution of GME programs. Therefore, the purpose of this study was to assess residency applicants' use and perceptions of DRN and to analyze the effect of Doximity reputation rankings on the application, interview acceptance, and match list rankings of applicants at a single sponsoring institution.

Participants and Methods

This study was approved by the Mayo Clinic Institutional Review Board and was conducted from March 1, 2017, through May 8, 2017. During the 2016-2017 NRMP Main Residency Match application period, the authors identified 12,617 applicants to 24 Mayo Clinic School of Graduate Medical Education (MCSGME) residency training programs. A survey was developed seeking feedback from applicants about their use of DRN during the residency application, interview selection, and match process. The survey was developed by 2 of us (B.B.S. and T.R.L.), with demographic questions modeled after a similar study of anesthesiology residency applicants. The survey was reviewed and edited independently by all study authors. For additional content validity, the survey was reviewed by the Mayo Clinic Center for Clinical and Translations Science support staff, including 2 analysts and a statistician. The survey was piloted for content validity by administration to 15 current postgraduate year 1 residents with experience using Doximity across medical and surgical specialties at MCSGME. The edited survey was reviewed again by all the authors and then finalized (Supplemental Appendix, available online at http://www.mcpiqojournal.org).

The survey included questions about demographic characteristics, type of medical school attended, specialty(s) applied to, number of residency program applications submitted, NRMP Main Residency Match results, applicant use and perception of DRN and Doximity reputation rankings, and other resources used to research residency programs. Applicants who did not use DRN were asked why they chose not to use the tool and then were asked what resources they did use to research residency programs during the application process. Additional space was provided for applicants to add comments where survey choices were not comprehensive (eg, specialty[s] applied, reasons not to use Doximity, other resources used to research residency programs).

The survey was distributed to all applicants to MCSGME programs participating in the NRMP Main Residency Match in April 2017. Weekly reminders were sent to nonresponders for 3 consecutive weeks. We used the Research Electronic Data Capture (REDCap) tool at Mayo Clinic for survey distribution. A 4-point Likert scale was used to score the DRN features that applicants found most valuable (very valuable, valuable, slightly valuable, not valuable) and to assess applicant views of the accuracy of Doximity reputation rankings (very accurate, accurate, slightly accurate, not accurate).

Data analysis consisted of descriptive statistics using REDCap. Comments were reviewed individually by a study author (B.B.S.), grouped according to common themes, and reported.

Results

During the study period, 12,617 applicants to MCSGME residency training programs participating in the NRMP Main Residency Match were identified. A total of 2152 applicants (17%) completed the survey and were included in the data analysis. Minor differences are present in the denominators of the data because not all survey respondents answered each question. Demographic data for residency applicants are outlined in Table 1.

Table 1

Residency Applicant Demographic Informationa

Variable	Values
Sex (No. [%]) (n=2147)
Male	1258 (59)
Female	889 (41)
Age (y)
Mean ± SD	28.5±4.1
Median	27
Range	19-58
Medical school (No. [%]) (n=2152)
US allopathic	1261 (59)
International	774 (36)
US osteopathic	117 (5)
Specialty applied to (No. [%]) (n=2148)
Internal medicine	622 (29)
Surgery	213 (10)
Family medicine	202 (9)
Anesthesiology	192 (9)
Pediatrics	172 (8)
Neurology	126 (6)
Emergency medicine	115 (5)
Orthopedic surgery	113 (5)
Radiology	108 (5)
Psychiatry	100 (5)
Pathology	91 (4)
Obstetrics and gynecology	75 (4)
Dermatology	72 (3)
Otolaryngology	65 (3)
Physical medicine and rehabilitation	57 (3)
Neurologic surgery	46 (2)
Radiation oncology	41 (2)
Plastic surgery	39 (2)
Ophthalmology	22 (1)
Child neurology	19 (<1)
Otherb	19 (<1)
Medicine/pediatrics	15 (<1)
Thoracic surgery	15 (<1)
Vascular surgery	10 (<1)
Urology	9 (<1)
Preventive medicine	2 (<1)
Medical genetics	1 (<1)
Nuclear medicine	1 (<1)
No. of residency programs applied to (No. [%]) (n=2142)
≥51	1258 (59)
21-30	262 (12)
31-40	233 (11)
41-50	202 (9)
11-20	141 (7)
1-10	46 (2)
Successful 2016-2017 NRMP match (No. [%]) (n=2027)
Yes	1722 (85)
No	305 (15)

NRMP = National Resident Matching Program.

Other specialties included pediatrics/psychiatry, interventional radiology, and transitional year.

The use of DRN by residency applicants is outlined in Table 2. Of the 2152 applicants who completed the survey, 1339 (62%) actively used DRN during the application, residency interview, and match list process. Of the 1335 applicants who used DRN and answered the question, 1157 (87%) used the tool before sending out applications, 906 (68%) used it during the interview process, and 686 (51%) used it while creating their rank order lists. Of the 1186 applicants who matched into a GME training program, only 314 (26%) believed that the use of DRN helped them match successfully.

Table 2

Doximity Residency Navigator (DRN) Use by Applicants

Variable	Applicants (No. [%])
DRN use during residency application/interview/match (n=2150)
Yes	1339 (62)
No	811 (38)
When did you access DRN? (n=1335)
Before sending out applications	1157 (87)
During the interview process	906 (68)
While making rank order list	686 (51)
Do you believe DRN helped you match successfully? (n=1186)
Yes	314 (26)
No	872 (74)
Did DRN expand your geographic options of residency programs? (n=1185)
Yes	577 (49)
No	608 (51)

Resources available in DRN identified by applicants as very valuable, valuable, slightly valuable, and not valuable to their research of residency programs are listed in the Figure. The most valuable resource in DRN was the reputation ranking, which was noted to be valuable or very valuable to 958 of 1233 applicants (78%). Data analyzing the influence of Doximity reputation rankings on an applicant's application, interview acceptance, and match list rankings are outlined in Table 3. Overall, 977 of 1241 applicants (79%) reported that Doximity reputation rankings influenced their application, interview acceptance, or match list ranking decisions. Despite this, 699 of 1240 applicants (56%) believed that the rankings were not accurate or only slightly accurate.

Figure

Features of Doximity Residency Navigator least and most valuable to applicants. aLarge public hospital, Veterans Affairs hospital, pediatric hospital, or American Osteopathic Association dual-accredited training program. bRural or urban setting.

Table 3

Impact of Doximity Reputation Rankings on Applicant Residency Decisions

Variable	Applicants (No. [%])
Did Doximity reputation rankings influence application to residencies? (n=1241)
Yes	742 (60)
No	499 (40)
Did Doximity reputation rankings influence acceptance or rejection of interviews? (n=1241)
Yes	499 (40)
No	742 (60)
Did Doximity reputation rankings influence match rankings? (n=1239)
Yes	605 (49)
No	634 (51)
How accurate do you believe that the Doximity reputation rankings are? (n=1240)
Very accurate	46 (4)
Accurate	495 (40)
Slightly accurate	615 (50)
Not accurate	84 (7)

Of the 811 applicants (38%) who reported that they did not use DRN during the residency application process, 621 (77%) were unaware of DRN and 130 (16%) did not find the information useful to their residency application. Applicants who chose not to use DRN commented that the content in DRN and the reputation rankings seemed “unreliable, biased, subjective,” with questionable validity.

Other resources used by applicants to research residency programs are listed in Table 4. The most frequently used resources were residency program websites (1676 of 2014, 83%), American Medical Association resources (1389 of 2014, 69%), DRN (1339 of 2150, 62%), Association of American Medical Colleges (AAMC) resources (1012 of 2014, 50%), and NRMP resources (1010 of 2014, 50%).

Table 4

Resources Used by Applicants to Research Residency Programsa

Resource	Applicants (No. [%])
Residency program websites (n=2014)	1676 (83)
AMA resourcesb (n=2014)	1389 (69)
Doximity Residency Navigator (n=2150)	1339 (62)
AAMC (n=2014)	1012 (50)
NRMP (n=2014)	1010 (50)
Student Doctor Network (n=2014)	849 (42)
Other online or print resources (n=2014)	236 (12%)
Social media (n=2014)	201 (10)
Medscape (n=2014)	85 (4)

AAMC = Association of American Medical Colleges; AMA = American Medical Association; FREIDA = Fellowship and Residency Electronic Interactive Database; NRMP = National Resident Matching Program.

FREIDA Online, the AMA Residency & Fellowship Database.

Discussion

Most applicants (1339 of 2150; 62%) applying to GME programs at our institution who completed the survey used DRN during the application, interview, and match ranking process. Doximity reputation rankings were found to be the most valuable resource in DRN by the applicants. Of 1241 applicants who reviewed Doximity reputation rankings, 977 (79%) reported that the rankings influenced their application, interview acceptance, or match list rankings of residency programs. Despite the significant impact of Doximity reputation rankings, 699 of 1240 applicants (56%) believed that the reputation rankings were not accurate or were only slightly accurate. Furthermore, only 314 of 1186 applicants (27%) believed that DRN helped them match successfully during the NRMP 2017 Main Residency Match.

Identifying tools that accurately assess residency programs before applying to and interviewing is important for applicants to GME programs. The NRMP 2017 Main Residency Match was highly competitive, with 43,157 applicants vying for 31,757 residency positions. In response to the competitive nature of the match, applicants are applying to and interviewing at an increasing number of residency programs to improve their chances of matching.10, 11 Application and interview expenses, as well as the time required to interview, at an increasing number of programs can be prohibitive to prospective residents. For example, a 2015 survey by the AAMC reported that during the application and interview process, mean ± SD expenses were $3422±$2853 per applicant (range, $80-$25,000). With limited financial resources and restricted time for interview travel, it is becoming increasingly important for applicants to have valid tools to research residency programs.

As stated by Doximity, Inc, DRN is a tool to help applicants “make informed residency decisions and to increase transparency in the residency match process.” The use of DRN by applicants has become more prevalent since its release in 2014. The present study shows that of the 811 applicants (38%) who did not use DRN, 621 (77%) were unaware of its existence. This demonstrates the possibility that increased awareness may result in a dramatic increase in utilization by residency program applicants. Doximity compiles objective data from a variety of public sources, conducts annual satisfaction surveys, and partners with residency programs to ensure individual residency program data in DRN is accurate and current. Doximity reputation rankings are derived from surveys sent to board-certified physicians and modeled after the annual physician survey from which U.S. News & World Report's Best Hospitals rankings are calculated. Previous studies have called into question the validity of the Doximity reputation rankings due to the lack of objective and outcome-based data used to formulate these rankings.3, 4, 5, 6 Consistent with previous studies,5, 6 the present study found that applicants question the accuracy of Doximity reputation rankings. Despite this apprehension, the present study indicated that Doximity reputation rankings influence the application, interview choice, and match list rankings of applicants, suggesting that applicants will use whatever information is available to evaluate residency programs.

Ongoing dialogue among applicants, medical schools, accreditation bodies, match organizations, program directors, and other parties, such as Doximity, Inc, to provide comprehensive information about residency programs to those seeking GME training is essential to improving the current application, interview, and match process. This is critical because the number of applicants seeking GME positions will increase due to medical school expansion. Individual residency programs should monitor their respective program websites and DRN, in addition to American Medical Association, AAMC, and NRMP resources, for accurate content because these resources are frequently used by applicants to research programs. Going forward, it is in the best interests of GME programs and prospective applicants for Doximity and sponsoring institution leadership to work together to ensure that valid metrics are collected and accurately reported in the DRN.

The low response rate in the present survey (2152 of 12,617 applicants [17%]) is a limitation of this study. Despite the low response rate, this is the largest multispecialty survey examining the use of DRN available in the literature. In addition, because the survey was distributed after the NRMP 2017 Main Residency Match, recall bias may have influenced the results as applicants may have been unable to determine the influence of DRN and Doximity reputation rankings on their application, interview, and match decisions. A large, multicenter, multispecialty study (possibly as a component of the NRMP applicant survey) is needed to further analyze the influence of DRN on applicants' decisions during the NRMP season.

Conclusion

Most survey responders used DRN during the application, interview, and match ranking process. Doximity reputation rankings were found to be the most valuable resource in DRN, and most applicants reported that the rankings influenced their application, interview acceptance, or match list rankings of residency programs. Despite this, more than 50% of respondents had doubts about the accuracy of Doximity reputation rankings. Given the availability and increasing utilization of social networking applications such as DRN, program directors and administrators at institutions that sponsor GME will need to develop strategies to ensure accuracy of content and how to best use these platforms to attract the best applicants.