Factors used in Applicant Ranking of Orthopedic Foot and Ankle Fellowships and the Availability of Online Information.

BACKGROUND: The Internet is often the first resource used by applicants to evaluate fellowship programs. However, information on these websites can be often incomplete, inaccessible, and/or inaccurate. The primary objective of this study was to examine key factors that orthopedic foot and ankle fellowship applicants use to rank programs. The secondary objective was to assess both the accessibility and availability of the information on orthopedic foot and ankle fellowship program websites.
METHODS: A Qualtrics survey was distributed via e-mail to those who matched into an orthopedic foot and ankle fellowship position from years 2008-2020. A comprehensive list of orthopedic foot and ankle fellowship programs was created. Program websites were evaluated for accessibility as well as the quality of recruitment and educational content.
RESULTS: There were a total of 114 survey responses out of 644 invites (17.7%). The most important factors for establishing a rank list were operative experience, current faculty, and program reputation. Eighty-five percent (41/48) of orthopedic foot and ankle fellowship websites were directly accessible using Google. On average, accessible orthopedic foot and ankle fellowship websites contained only 57% (11.5/20) of the content deemed desirable.
CONCLUSION: Orthopedic foot and ankle websites are widely accessible and have higher recruitment and educational quality content scores compared with previously published data. The most important factors for establishing a rank list are consistent with previous literature. Those who ranked operative experience as one of the most important factors when establishing a rank list did not complete more operative cases than those who did not. LEVEL OF EVIDENCE: Level IV.

Introduction

An increasing number of orthopedic surgery residents plan to pursue orthopedic subspecialty fellowship training. When ranking orthopedic fellowship programs, prior studies suggest applicants deem experience, autonomy, staff members, and prestige of the program to be the most important criteria. The Internet is often the first resource used by applicants to evaluate fellowship programs. However, the quality and accessibility of fellowship program websites vary. The information on these websites can be incomplete, inaccessible, and/or inaccurate. Orthopedic Foot and Ankle fellowships have participated in the San Francisco Residency and Fellowship Match (SF Match) since 2008. A list of available fellowships is available through the American Orthopaedic Foot & Ankle Society (AOFAS) and Fellowship Residency Electronic Interactive Database (FREIDA)—an artifact of the programs that are ACGME accredited. The primary objective of this study was to examine the most important factors that orthopedic foot and ankle fellowship applicants use to rank fellowship programs. The secondary objective was to assess both the accessibility and availability of the information applicants desire on orthopedic foot and ankle fellowship program websites.

Materials and Methods

A Qualtrics survey was distributed via e-mail to those who applied to orthopedic foot and ankle fellowships through the SF Match from 2008-2020 and who matched into an orthopedic foot and ankle fellowship position. All survey responses were weighed equally, regardless of year of fellowship completion. The survey inquired about factors (Table 1) individuals felt were most important while applying for foot and ankle fellowship, requested an estimate of the number and type of cases they were involved in during fellowship, as well as the calendar year fellowship was completed. It was estimated that approximately 120 survey responses were required for analysis based on multiplication of the approximate number of fellows per year (50), the number of years the SF Match has been in place for orthopedic foot and ankle fellowships (12), and a 20% response rate. Both survey content and statistical analysis are similar to previous studies. Survey respondents were asked to rank the following factors as most important, somewhat important, or not important at the time they applied for a foot and ankle fellowship.

Table 1.

Factors Influencing Fellowship Rank.

Program reputation	Fellow-run clinic
Number of fellows	Research opportunities/requirements
Current faculty	Job placement
Salary	Geography/location
Call schedule	Ease and compatibility of working with faculty
Perceived happiness of current fellows	Teaching students/residents
Operative experience	Work-life balance
Impression of the program from the interview	Moonlighting opportunities
Business/practice management training	Other (ie, free text entry)

Operative experience was further explored by asking survey respondents to estimate the number and type of procedures performed during fellowship as well as rank the following factors as they pertain to operative experience: fellow autonomy, exposure to complex and unusual cases, total number of cases, and wide breadth/comprehensive operative experience. Statistical analysis of those factors deemed important to establish a rank list were performed using both means and 95% confidence intervals. A Kruskal-Wallis test was used to compare groups. Spearman correlation was used to evaluate correlation between ranks of different factors. A value of P <.05 was assessed as significant.

The availability of information on program websites was assessed using methods reflected in published literature. A comprehensive list of orthopedic foot and ankle fellowship programs was created using both the AOFAS database and FREIDA. Both the AOFAS database and FREIDA were examined for links to orthopedic foot and ankle fellowship program websites. Additionally, a Google search was performed using “program name + orthopedic foot and ankle fellowship” to assess accessibility and access fellowship program websites. The first 10 Google search results were examined for direct links to fellowship program websites. Websites were assessed for quality using methods previously described regarding fellowship information available on websites. Website content was catalogued via the presence or absence of program description, application requirements, selection criteria, AOFAS link, SF Match program link, location description, program contact information, fellow listing, faculty listing, and salary. Educational content was also catalogued via the presence or absence of call responsibilities, didactic instruction, journal club, research requirements, evaluation criteria, rotation schedule, operative experience, office experience, meetings attended, and courses attended. If the information presented on a fellowship website was absent, incomplete, or unclear regarding any of the aforementioned categories, it was designated to be absent. One point was awarded for the presence of each of the aforementioned content areas with a maximum score of 20. This was designated the total content quality score. The total content quality score was further divided into a recruitment content quality score and an educational content quality score, both with a maximum score of 10. Content scores were compared by fellowship geographic region, affiliation with a top 20 orthopedic hospital and/or medical school (per the 2020-2021 US News & World Report), ACGME accreditation status, and number of fellows. Student t-tests were employed to compare 1-way affiliation with a top 20 orthopedic hospital, affiliation with a top 20 medical school, ACGME accreditation status, and number of fellows. To compare the quality of fellowship programs by geographic location, 1-way analyses of variance were utilized. A value of P <.05 was assessed as significant.

Results

A total of 48 orthopedic foot and ankle fellowships were listed on the 2021-2022 fellowship programs offering positions published by the AOFAS. The AOFAS document offered direct, functioning links to 42% (20/48) of the fellowship programs listed; however, 58% (28/48) of programs had nonfunctioning, indirect, or absent links to fellowship websites. A total of 7 programs were listed on the FREIDA database, all of which were listed on the AOFAS program offerings document. The FREIDA database offered no direct, functioning links to any orthopedic foot and ankle fellowship. Using a Google search, 85% (41/48) of orthopedic foot and ankle fellowship websites were accessible with a direct, functioning link compared with the 15% (7/48) of programs with nonfunctioning, indirect, or absent links to fellowship websites. A total of 7 programs were excluded from quality content analysis because of the absence of program websites or nonfunctioning program websites. Therefore, a total of 41 orthopedic foot and ankle fellowship websites accessible through Google and/or the AOFAS were included in quality content analysis. Table 2 summarizes the accessibility of orthopedic foot and ankle fellowship websites.

Table 2.

Links to Program Websites Available Through Google, AOFAS, or FREIDA.

	Google,n (%)	AOFAS Database,n (%)	FREIDA,n (%)
Number of programs with direct, functioning links	41 (85)	20 (42)	0 (0)
Number of programs with nonfunctioning, indirect, or absent links	7 (15)	28 (58)	48 (100)

Abbreviations: AOFAS, American Orthopaedic Foot & Ankle Society; FREIDA, Fellowship Residency Electronic Interactive Database.

Analysis of accessible orthopedic foot and ankle fellowship website content quality revealed a mean total content quality score of 11.5 ± 2.8 (maximum score of 20). The maximum observed total content quality score was 16, and the minimum score was 5. Orthopedic foot and ankle fellowship websites achieved a mean score of 5.9 ± 1.7 (maximum score 10) when analyzing the quality of recruitment information. The maximum score for quality of recruitment information was 9, and the minimum score was 3. Analysis of the quality of fellow education information revealed a mean score of 5.6 ± 1.8, with a maximum score of 9 and a minimum score of 0 (maximum score 10).

Investigation of the quality of recruitment information suggests that the most commonly included piece of recruitment information is program description, present on 98% (40/41) of program websites. The least commonly reported component of recruitment information was fellow selection criteria, present on 0.05% (2/41) of program websites. Analysis of the quality of fellow education information reveals that the most commonly included piece of educational information is operative experience, present on 90% (37/41) of program websites. The least commonly included component of fellow education information was fellow evaluation criteria, present on 0% (0/41) of program websites. Figure 1 illustrates the availability of desired information on orthopedic foot and ankle fellowship websites. Comparison of website quality by ACGME accreditation status, affiliation with a top 20 orthopedic hospital, affiliation with a top 20 medical school, and the number of fellows per fellowship did not reach significance.

Figure 1.

The availability of information on orthopedic foot and ankle fellowship websites.

Of those orthopedic foot and ankle fellowship websites with functioning websites, 32% (13/41) of programs were located in the South, 27% (11/41) were located in the Northeast, 24% (10/41) were located in the Midwest, and 15% (6/41) were located in the West. One program was located outside of the United States.

There were a total of 114 survey responses out of 644 invites (17.7%). A total of 7 survey respondents completed fellowship in 2009, 8 in 2010, 2 in 2011, 6 in 2012, 5 in 2013, 4 in 2014, 8 in 2015, 11 in 2016, 6 in 2017, 12 in 2018, 12 in 2019, and 19 in 2020. When asked to rank 18 total factors in order of importance concerning fellowship selection, the top 5 most important factors were “operative experience,” “current faculty,” “program reputation,” “ease and compatibility of working with current faculty,” and “perceived happiness of current fellows.” The 5 least important factors were “moonlighting opportunities,” “salary,” “fellow-run clinic,” “teaching residents and students,” and “job placement.” These rankings changed over time; compared with those that applied after 2016, fellows that applied before 2016 were more likely to deem “program reputation” as important (P = .05), whereas a greater number of individuals deemed “ease and compatibility of working with faculty” as important after 2016 as compared to those applying before (P = .008). We analyzed the differences of top 5 factors between a higher number of total cases done and a lower number of total cases done, where a higher number of total cases was defined as those who completed ≥435 cases. Analysis demonstrates that those who valued “current faculty” had more operative experience than those who did not (P = .034). Those who ranked “operative experience” as one of their top 5 most important factors completed a higher number of cases (448 cases) than those who did not (403 cases); however, this did not reach significance (P = .238). Those who ranked “geographic location” as one of their top 5 most important factors completed a higher number of cases than those who did not (P = .04). Those who ranked “work life balance” as one of their top 5 most important factors did not complete a significantly different number of cases from those who did (P = .596). There was no statistically significant difference in the total number of cases completed by fellowship year (P = .69). The average number of cases completed during fellowship was 460 cases. A distribution of the average number of cases completed by fellowship year is shown in Figure 2.

Figure 2.

Average number of cases completed by fellowship year.

Discussion

More than 90% of orthopedic surgery residents plan to complete a fellowship. The Internet is one method used by orthopedic fellowship applicants to evaluate prospective fellowship programs. The Internet is an underutilized tool for fellow recruitment. Several studies attempted to identify factors that impact how orthopedic surgery residents establish a rank list. Niesen et al reported operative experience and staff members at a fellowship to be the most valued factors in selecting a fellowship program whereas salary and research resources were ranked least important in establishing a final rank. Factors used to establish a rank list by orthopedic foot and ankle fellowship applicants have not been examined.

The quality of information on orthopedic fellowship program websites has been previously examined. In an analysis of orthopedic foot and ankle fellowship program websites, there was substantial variability in both the quality and accessibility of the program websites. Similarly, websites for orthopedic sports medicine, pediatric orthopedic surgery, and orthopedic trauma fellowships were evaluated for content and accessibility and were ineffective in conveying fellowship program information or lacked ease of accessibility.

Currently, there is no resource for residents to use when evaluating a fellowship program in terms of operative experience. In previous studies, this has been an important factor to applicants in establishing a rank list. However, the mean number of cases and types of cases performed during fellowship is not known. This information is also unlikely to be published or made available elsewhere. Programs do not publish the cases performed by fellows as such information is nuanced and often misunderstood by lay people. Typically, such information is orally transmitted during an interview. The number and types of cases may be valuable information for future applicants as they make their assessment of various fellowship programs.

Assessment of both the accessibility and the content quality on orthopedic foot and ankle fellowship websites by Hinds et al compared with our data reveals several trends and improvements concerning the information published on websites. Whereas previously 15% (7/47) of programs accessed using the AOFAS fellowship database and 62% (29/47) of programs accessed using Google had direct links to fellowship websites, our data suggest that a higher percentage of programs have direct links to their orthopedic foot and ankle fellowship website using both the AOFAS fellowship database (42%, 20/48) and Google (85%, 41/48) (Figure 3). Additionally, accessible orthopedic foot and ankle fellowship websites contained a greater percentage of examined content—a mean of 57% (11.5/20) compared with 44% (proportion not reported). Analysis of the quality of recruitment information reveals a mean score of 5.9 ± 1.7 compared with the previously reported mean score of 4.4 ± 1.7 out of 10. Consistent with previous literature, program description and program contact information remained the most frequently reported components of recruitment information, whereas selection criteria remained the least frequently reported component. Similarly, analysis of the quality of fellow educational information reveals a mean score of 5.6 ± 1.8 compared with the previously reported mean score of 4.3 ± 2.3 out of 10. Figure 4 compares previously reported content quality scores to our data. Operative experience and research requirements remained the most frequently reported components of educational information as well as didactic instruction. Whereas courses attended by fellows was previously reported to be the least commonly published component of educational information, our data suggest evaluation criteria to be the least commonly published component. The increase in orthopedic foot and ankle fellowship website accessibility as well as the increase in quality of total content, recruitment information, and education-related information is perhaps due to an increasing reliance on Internet-based communications amid the coronavirus pandemic and an increasingly digital world. Ultimately, an increase in the accessibility and quality of orthopedic foot and ankle fellowship websites renders applicants more informed and therefore allows for increasingly informed decisions pertaining to fellowship applications.

Figure 3.

The accessibility of orthopedic foot and ankle fellowship websites: a comparison.

Figure 4.

Mean content quality scores: a comparison.

Comparison of survey data with current literature reveals that the priorities of fellowship applicants has not changed in establishing a rank list. Niesen et al reported the most important factors to be operative experience/autonomy, staff members, program prestige, number of cases, and satisfaction of past fellows. Similarly, we found the most important factors to be operative experience, current faculty, program reputation, ease and compatibility of working with current faculty, and perceived happiness of current fellows. Those factors residents deemed least important when establishing a rank list varied compared with previously published literature. Whereas the least important factors reported by Niesen et al were research resources, salary, call/rounding requirements, facilities, and location, we found the least important factors to be moonlighting opportunities, salary, fellow-run clinic, teaching residents and students, and job placement. The factors fellowship applicants ranked as somewhat important were not subjected to statistical analysis.

Total number of cases completed varied based on several factors. Those who valued current faculty completed a higher number of total cases compared to those who did not. This is perhaps due to increased rapport and therefore greater autonomy granted to the fellow. Interestingly, those who ranked operative experience as one of their top 5 most important factors did not complete significantly more cases than those who did not. Similarly, those who valued work-life balance did not complete significantly fewer cases than those who did not. The operative volume between programs is not equal and therefore cannot explain this finding. Though prioritization of operative experience does not correlate with increased case volume, operative volume does not appear to be uniform across programs. Notably, survey respondents reported completing as few as 155 cases and as many as 1000 cases during their fellowship with an average of 460 cases.

The orthopedic foot and ankle fellowship experience appears to be relatively stable in terms of operative volume. There was no significant difference in the total number of cases completed by fellowship year. Fewer applicants after 2016 felt “program reputation” was an important factor and a greater number deemed “ease and compatibility of working with faculty as important.” This was distinctly contrary to prior survey studies of orthopedic residents. Interestingly, Li et al surveyed orthopedic residents and found that fellowship program reputation was among the top 2 most important factors when selecting a fellowship program. This difference may be attributable to subspecialty fellowship selection or perhaps a realization that time working directly with specific faculty is likely to be much greater in fellowship than in residency.

Limitations of the study include a low survey response rate—a 17.7% response rate is unlikely to be representative and therefore introduces selection bias. Response rates vary throughout the literature. Niesen et al surveyed orthopedic surgery residents about factors influencing fellowship selection; however, the response rate could not be calculated. Chun et al explored factors used in ranking pediatric otolaryngology fellowships and surveyed both applicants and fellowship directors with response rates of 68% and 48%, respectively. Recall bias may also influence survey responses, especially since some survey recipients completed fellowship as early as 2009. In particular, the large distribution of cases completed during fellowship may be attributable to recall bias. Although the largest number of survey responses was from those who most recently completed fellowship in 2020 and on average survey respondents were 4 years removed from fellowship, 28% of respondents were at least 6 years removed from fellowship. Opinions from survey data do not always reflect behaviors. Additionally, we used criteria for evaluating website content that is based on similar studies, which may not represent a comprehensive list of the information desired by fellowship applicants. Fellowship programs may also not publish certain information on their websites such as subjective factors valued by applicants, number of cases completed by fellows, selection criteria, and evaluation mechanisms. Furthermore, it is possible that fellowship program websites may update or change their website content prior to the publication of this study because fellowship positions are offered on a yearly basis. However, heretofore, there have been no attempts to quantify the number and type of cases performed by surgeons-in-training during their fellowship year in foot and ankle.

Examination of those factors used to rank orthopedic foot and ankle fellowship programs as well as the availability of information desired by fellowship applicants may reveal both strengths and weaknesses in fellowship program website content. This data can then be utilized to tailor orthopedic fellowship program website information, thereby improving both the quality and the accessibility of information to fellowship applicants.

Conclusion

In the age of information, there is an increasing reliance on program websites for information. As more surgeons pursue fellowship training, dispersal of useful information through program websites will be increasingly important for applicants and as a recruitment tool. Fellowships can use this article to best represent the strengths of their programs and applicants can use this information to critically assess program websites.