Analysis of Website Accessibility and Content for All 92 Accredited Hand Surgery Fellowship Programs in the United States: An Update.

Purpose: The internet is an important information source for hand surgery fellowship applicants. A previous analysis of hand fellowship websites in 2014 demonstrated they were often inaccessible and incomplete. Given the increased importance of virtual information, we performed an updated assessment of the accessibility and content of hand fellowship program websites.
Methods: Websites of 92 accredited hand surgery fellowship programs were evaluated for the following: (1) accessibility; and (2) the presence of 13 fellow recruitment and 13 fellow education criteria, as defined in prior studies. We used Mann-Whitney U and Kruskal-Wallis tests to assess whether the geographic region, number of fellows, or affiliation with a top orthopedic hospital or medical school were associated with website content.
Results: Functional website links that redirect to the appropriate fellowship program website are provided for 47 (51.1%) of 92 programs on the American Society for Surgery of the Hand fellowship directory. All missing websites were accessible via independent Google searches. Fellowship program websites contained an average of 13.9 ± 4.4 total criteria (range, 3-23). Of the 15 criteria examined in both 2014 and 2021, there were significant (P < .05) increases in the prevalences of 4: current fellow(s), salary, social media links, and operative experience. Conclusions: Despite a slight increase in accessibility since 2014, nearly half of hand surgery fellowship program websites remain inaccessible from the American Society for Surgery of the Hand directory. Program websites averaged approximately half of the criteria analyzed, with many websites failing to provide information deemed important by applicants. Clinical relevance: Our study provides an impetus for improving the accessibility and content of hand surgery fellowship websites. A website that incorporates criteria examined in this study can serve as an effective recruitment tool by providing consistent baseline information that may help applicants decide which programs align with personal values and future career goals.

Orthopedic surgery, plastic surgery, and general surgery residents interested in pursuing a hand surgery fellowship often use the internet to learn about hand fellowship programs as they begin the application process. Hand fellowship applicants seeking more information about the hand fellowship match process on the National Resident Matching Program website are redirected to the American Society for Surgery of the Hand (ASSH) website for information regarding individual fellowship programs. The ASSH maintains a comprehensive directory of all accredited hand surgery fellowships and provides basic information about each accredited hand fellowship program, including the program director name, number of annual available positions, faculty names, and website link. Although background information about each program is provided on the ASSH directory, much of the fellowship information useful to prospective applicants is found on individual program websites.

Many previous studies have demonstrated that applicants across diverse specialties and subspecialties cite program websites as an important factor when deciding where to apply, where to interview, and how to rank programs during the match process.3, 4, 5, 6, 7, 8, 9 Almost all (98%) anesthesiology residency applicants reported that they used residency program websites, with 56% reporting that the website content impacted their decision regarding where to apply. Additionally, 40% of emergency medicine residency applicants cited poor website quality as 1 reason for not applying to a program. The coronavirus disease 2019 (COVID-19) pandemic further elevated the importance of training program websites as a primary information source for prospective applicants. With many programs pausing in-person interviews, applicants were unable to visit programs in person and were left with fewer opportunities to experience a program’s culture. Yet, despite the importance of these websites, recent analyses indicate that program websites are often inaccessible and incomplete.10, 11, 12, 13, 14

Trehan et al evaluated the content and accessibility of hand surgery fellowship websites in 2014 (study published in 2015), and reported that websites were often inaccessible and failed to convey important information desired by fellowship applicants. However, in light of the COVID-19 pandemic and an increased emphasis on the use of program websites and social media by residency and fellowship programs as a means of communicating with applicants, programs may attempt to improve their online presence.15, 16, 17 The purposes of this study were to: (1) perform an updated assessment of the accessibility and content of hand surgery fellowship program websites and compare the results to the Trehan et al study to discern temporal trends in website accessibility and quality, particularly post-COVID-19; and (2) assess whether recruitment and education content on fellowship program websites varies based on program characteristics, such as the geographic region, number of fellows, affiliation with a top orthopedic hospital, and affiliation with a top medical school. We hypothesized that websites would be more accessible and achieve higher information potential than when last evaluated in 2014, as demonstrated by a greater percentage of program websites with a direct link populated following a Google search and by a greater percentage of program websites containing the recruitment and education criteria evaluated in this study.

Materials and Methods

Program eligibility

We obtained a list of all active hand surgery fellowship programs from the ASSH Fellowship Directory. All accredited hand surgery fellowship programs were included in our study based on programs listed in the Fellowship Directory on June 1, 2021. All websites were evaluated by 2 independent authors (SAC & MX) from June 1, 2021, to July 1, 2021.

ASSH database

Information recorded from the ASSH directory for each active program included program contact information, such as an email address and telephone number; the application deadline; interview date(s); the number of available positions; and whether the program accepted applicants from orthopedic surgery, general surgery, or plastic surgery residencies. Additionally, the website hyperlink listed for each program was evaluated for functionality. Website hyperlinks were either declared nonfunctional, indirect (a functional hyperlink but linked to another website requiring >1 click to reach the appropriate hand surgery fellowship website), or direct.

Google accessibility

A Google (http://www.google.com) search was conducted to determine the accessibility of program websites from outside the ASSH directory. We performed 2 searches for each program, and evaluated the first page of results (10 listings) for direct links to program websites. Two separate search terms were included in an attempt to cover the possible terms used by applicants: (1) “program name + hand fellowship”; and (2) “program name + hand surgery fellowship.”

Primary analysis: prevalence of recruitment and education content on fellowship program websites

All accessible program websites were evaluated for information regarding fellow recruitment and education criteria. The criteria being assessed were marked as present if the website provided information on the topic, regardless of the quantity or quality of information presented. This method is consistent with website content analyses used in previous studies and was done in order to maintain objectivity.,,,18, 19, 20, 21 To develop a list of website criteria, a qualitative analysis of the criteria listed in several prior fellowship website analyses (including hand fellowship websites) was completed by study authors (SAC & MX).,,,18, 19, 20, 21 These criteria were consistently divided into 2 categories: recruitment and education. All authors subsequently voted on the criteria most applicable to current applicants, and 26 total criteria were ultimately selected for inclusion in our study: 13 criteria related to the fellow recruitment process and 13 criteria related to fellow education (Table 1). Fifteen of the 26 total criteria assessed were derived from the previous Trehan et al study examining hand surgery fellowship website content. Examples of these criteria include application requirements, an application due date, and current fellow(s) from the fellow recruitment category and program description, didactics, and call responsibilities from the fellow education category. These criteria were selected to provide a direct comparison between fellowship websites in 2014 and 2021. Chi-square tests were used to compare the prevalences of the 15 recruitment and education criteria that were assessed in both 2014 and 2021. The remaining 11 criteria included in our study were based on other previously published fellowship website analyses and the current needs of fellowship applicants based on recent surveys administered to graduate medical education applicants.,,, These 11 additional criteria include the following: an application link, interview dates, a location description, video content, the selection criteria, association links, affiliated-hospital information, a faculty listing, academic meeting attendance, a journal club, and the evaluation criteria. All 26 of the criteria used to evaluate current fellowship program websites can be observed in Table 1.

Table 1

Hand Surgery Fellowship Website Recruitment and Education Criteria

Recruitment	Education
Application due date∗	Academic conferences
Application links	Affiliated-hospital information
Application requirements∗	Didactics∗
Association link	Evaluation criteria
Current fellow(s)∗	Faculty listing
Geographic placement of graduates∗	International opportunities∗
Interview dates	Journal club
Location description	Microsurgical training course or laboratory∗
Past fellows∗	On-call requirements or daily schedule∗
Salary∗	Operative experience∗
Selection criteria	Program description∗
Social media links∗	Research requirement∗
Video content	Rotation schedule∗

Also evaluated in the Trehan et al 2015 study.

Secondary analysis: website recruitment and education content based on program characteristics

Fellowship programs were stratified by the geographic region, number of fellows, affiliation with a top orthopedic hospital, and affiliation with a top medical school. For geographic location, programs were divided into 4 groups—Midwest, Northeast, South, and West—based on regions defined by the United States Census Bureau. For the number of fellows, programs were divided into 2 groups based on the information provided on the ASSH fellowship directory: (1) 1 fellow; or (2) 2+ fellows. For affiliation with a top orthopedic hospital, programs were divided into 2 groups: (1) affiliated with a top-20 US News and World Report orthopedic hospital; or (2) not affiliated with a top-20 US News and World Report orthopedic hospital, based on 2022 rankings. For affiliation with a top medical school, programs were divided into 2 groups: (1) affiliated with a top-20 US News and World Report medical school; or (2) not affiliated with a top-20 US News and World Report medical school, based on 2022 rankings. A Shapiro-Wilk test revealed that website content scores were not normally distributed. As such, we compared the recruitment and education criteria on fellowship program websites using the Mann-Whitney U test or Kruskal-Wallis test, as appropriate. If a statistically significant difference was observed based on the geographic region, a post hoc Dunn’s test was used to determine which program regions differed from each other. P values of <.05 were deemed significant.

Results

As of June 1, 2021, there were 92 hand surgery fellowship programs, with 192 available positions, accredited by the Accreditation Council for Graduate Medical Education. The 92 programs varied in the types of applicants they permitted to apply to their program, based on the applicant’s completed residency program. Forty-two fellowship programs accepted applicants who completed plastic surgery, orthopedic surgery, or general surgery residencies. Forty fellowship programs accepted applicants from orthopedic surgery or plastic surgery residencies. The remaining 10 fellowship programs accepted only orthopedic surgery residents (n = 8), only plastic surgery residents (n = 1), and a combination of orthopedic surgery or general surgery residents (n = 1). The ASSH directory provided application deadlines for all 92 programs listed, with December 15 (n = 61 programs), January 31 (n = 24 programs), and November 15 (n = 22 programs) the only 3 application due dates listed.

Website accessibility: Google and ASSH database

Overall, all 92 hand surgery fellowship programs listed on the ASSH database had functional websites that were accessible via an independent Google search when using either of the search parameters included in this study: “program name + hand fellowship” or “program name + hand surgery fellowship.” Hyperlinks provided on the ASSH directory were less reliable than a Google search with regards to redirecting the user to the appropriate fellowship website. Website links were listed for 90 (97.8%) of 92 hand surgery fellowship programs on the ASSH directory. Of the links provided, 76 (84.4%) of 90 were functional. Of the functional links, only 47 (61.8%) of 76 provided a direct link to the proper fellowship program website (Fig. 1).

Figure 1

Accessibility of hand surgery fellowship program websites from the ASSH online directory.

Website content: program overview, application and recruitment, and education criteria

There was high interobserver reliability between the 2 independent website reviewers (SAC & MX), with 89.5% agreement and a κ coefficient of 0.810.

Hand surgery fellowship program websites contained a mean ± standard deviation of 13.9 ± 4.4 (range, 3–23 criteria) out of the 26 total fellow recruitment and education criteria (Fig. 2). Table 2 displays the prevalences of all 26 criteria analyzed in this study on current hand surgery fellowship program websites.

Figure 2

Hand surgery fellowship program websites, sorted by total website content score (recruitment criteria + education criteria).

Table 2

Frequency of Recruitment and Education Criteria on Hand Surgery Fellowship Websites

Category	Mean (SD)	Low	High	Criteria	n (%)
Recruitment	5.6 (2.5)	1	11	1. Social media links2. Application requirements3. Application due date4. Application link5. Current fellow(s)6. Salary7. Interview dates8. Past fellows9. Location description10. Video content11. Geographic placement of graduates12. Selection criteria13. Association links	73 (79.3)70 (76.1)64 (69.6)55 (59.8)46 (50.0)42 (45.7)39 (42.4)32 (34.8)31 (33.7)29 (31.5)24 (26.1)7 (7.6)2 (2.1)
Education	8.2 (2.8)	2	13	1. Program description2. Affiliated-hospital information3. Operative experience4. Faculty listing5. Academic meeting attendance6. Journal club7. Didactics8. Research requirement9. Rotation schedule10. Microsurgical training course or laboratory11. On-call requirements or daily schedule12. Evaluation criteria13. International opportunities	92 (100)83 (90.2)83 (90.2)73 (79.3)75 (81.5)68 (73.9)67 (72.8)62 (67.4)61 (66.3)47 (51.1)34 (37.0)9 (9.8)7 (7.6)

Of the 13 fellow recruitment criteria analyzed, fellowship program websites contained a mean ± standard deviation of 5.6 ± 2.5 criteria (range, 1–11 criteria). Common recruitment criteria featured on hand surgery websites included social media links (n = 73; 79.3%), application requirements (n = 70; 76.1%), and application due dates (n = 64; 69.6%). Figure 3 compares the prevalences of the 7 recruitment criteria examined in both the 2015 Trehan et al study and the present study. The prevalences of 3 recruitment criteria increased significantly when comparing 2014 and 2021 websites: current fellow(s) (28% vs 50%, respectively; P = .01), salary (28% vs 46%, respectively; P = .02), and social media links (0% vs 79%, respectively; P < .01).

Figure 3

The prevalences of fellow recruitment criteria examined in both the 2015 Trehan et al study and the present study.

Of the 13 fellow education criteria analyzed, fellowship program websites contained a mean ± standard deviation of 8.2 ± 2.8 criteria (range, 2–13 criteria). Common education criteria featured on program websites included a program description (n = 92; 100%), affiliated-hospital information (n = 83; 90.2%), and operative experience (n = 83; 90.2%). Figure 4 compares the prevalences of the 8 fellow education criteria examined in both the 2015 Trehan et al study and the present study. Operative experience was the only education criteria that was included on a significantly greater proportion of fellowship websites in 2021 than in 2014 (90% vs 11%, respectively; P < .01).

Figure 4

The prevalences of fellow education criteria examined in both the 2015 Trehan et al study and the present study.

Larger fellowship programs with ≥2 current fellows included more fellow recruitment (P = .03) and fellow education (P < .01) criteria on program websites than smaller programs with 1 current fellow. There were no differences in website recruitment or education content when stratifying programs by region, affiliation with a top orthopedic hospital, or affiliation with a top medical school (Table 3).

Table 3

Comparison of Hand Surgery Fellowship Website Content by Program Characteristics

Characteristic	N	Recruitment Criteria, Mean (SD)	P	EducationCriteria, Mean (SD)	P
Region			.573		.830
Northeast	31	5.3 (2.5)		8.0 (2.9)
South	25	5.4 (2.4)		8.3 (2.8)
Midwest	19	6.2 (2.0)		8.4 (3.1)
West	17	5.6 (2.0)		8.6 (2.5)
Fellows			.031∗		.001∗
1	36	4.9 (2.2)		7.0 (2.8)
>1	56	6.0 (2.5)		9.1 (2.5)
Affiliated with top-20 orthopedic hospital
Y	16	5.5 (1.9)	.976	8.5 (2.6)	.795
N	76	5.6 (2.6)		8.2 (2.9)
Affiliated with top-20 medical school			.332		.897
Y	23	6.1 (2.5)		8.2 (3.2)
N	69	5.4 (2.5)		8.3 (2.7)

Indicates statistical significance.

Discussion

An accessible and informative fellowship website can serve as an effective recruitment tool. However, the ASSH fellowship directory, which provides access information for program websites, often has nonfunctional hyperlinks. With regards to website quality, websites contained an average of 13.9 out of a possible 26 fellow recruitment and education criteria (range, 3–23 criteria), suggesting highly variable websites with the potential to better convey information to prospective applicants. Only 4 of the 15 criteria analyzed in both 2014 and 2021 showed a significant (P < .05) increase in prevalence on program websites over the past 7 years, and many criteria deemed important by fellowship applicants remain infrequent on hand fellowship program websites. Improving the accessibility and quality of the hand fellowship program websites may enable applicants to make more informed application decisions based on their future career goals.

Several studies have demonstrated the importance of the program website as an information source for applicants.3, 4, 5,,, However, a website can only serve as an effective information source if it is accessible. Our results indicate that only 47 (52.2%) of 90 website links provided on the ASSH fellowship directory directly link to the appropriate fellowship program website. It would be beneficial for individual fellowship programs to ensure the link provided on the ASSH directory is routinely updated so that applicants can easily access accurate, program-specific websites.

With regards to the recruitment criteria evaluated, our results indicate significantly (P < .05) increased prevalences on fellowship program websites for 3 of the 7 criteria analyzed in both 2014 and 2021: current fellow(s), salary, and social media links. Since 2014, the proportion of hand fellowship program websites providing social media links increased from 0% to 79%. This drastic increase in social media presence reflects an overall trend regarding the evolving importance of social media as both an information source and a potential recruitment tool for fellowship programs. The COVID-19 pandemic resulted in a shift to virtual interviews. As a result, residency and fellowship programs used social media to communicate with applicants via the creation of Twitter, Instagram, and Facebook accounts dedicated to recruitment., Even if there is a return to in-person interviews, the use of social media as a recruitment tool is likely to remain important. A recent study of radiology residency applicants found social media played a “vital role” during the application season. Hand surgery fellowship programs can use an effective social media presence, such as the incorporation of social media accounts (such as Twitter, Instagram, and Facebook) on program websites, to provide an inside look at the program culture, which is important to applicants but difficult to discern online.

The incorporation of video content on fellowship program websites may also reveal the program culture. Despite being “highly valued” by the large majority of graduate medical education applicants, video content was present on only 31.5% of hand fellowship websites. Video content that provides an overview of the fellowship program or enables current fellows to describe their experiences would benefit current applicants.

Slight increases in the proportions of fellowship program websites containing information regarding current fellow(s) (+22% compared to 2014), past fellows (+6%), and geographic placement of graduates (+7%) suggest that some programs have responded to an increased demand for alumni information by current applicants., In the 2019 survey of residency applicants conducted by the National Resident Matching Program, “desired geographic location” was the most important factor in deciding which programs to apply to and rank highly. The creation of an “alumni database” that describes the employment information of previous fellows, which has been successfully implemented by a few fellowship programs, would allow for more targeted applications from current applicants based on personal geographic preferences, which have been shown to drive the match process in the past.31, 32, 33

With regards to the education content analyzed in this study, there were increases in the proportions of websites containing information about the fellow’s operative experience (+79% compared to 2014), research requirement (+11%), and on-call requirements or daily schedule (+8%); however, the percentage of programs with detailed information about the on-call requirements or daily work schedule (37%) remains low. Although on-call requirements or daily work schedule information was often absent, some fellowship websites shared detailed weekly schedules or provided a “day in the life” video prospective of the current fellow(s).

Improving the accessibility and content of hand surgery fellowship websites could result in more informed application decisions. Additionally, most of our study criteria will remain constant over time, with only a few criteria (such as current fellows and past fellows) requiring annual updates. Therefore, after an initial investment of resources to improve the fellowship program website, there would be minimal future action required to maintain website quality.

There are several limitations to our study. First, the list of 26 criteria analyzed in this study is not validated; however, the criteria used in this study have all been used in prior analyses of residency and fellowship website content and were determined by the study team to be relevant to current hand fellowship applicants.,,, Additionally, we used a binary scoring system when evaluating websites for the presence or absence of the 26 criteria used. As such, the quality of each criterion evaluated was not assessed, with the exception of noting a few examples of informative websites that could serve as models for other fellowship programs. However, the binary scoring system we implemented was used in all previous fellowship website content analyses that our study criteria were based upon.,,18, 19, 20,,, Furthermore, while we assessed the presence of various criteria, we did not confirm the accuracy of the information posted on program websites. For example, a recent change to a program’s application deadline may not necessarily be included on their website. Finally, website content may be updated by fellowship programs, and updates since the data collection process was completed would not be reflected in this cross-sectional study. While we did not assess each program website for the date of last update, this information could provide insight as to a program’s commitment to a strong online presence, and may be useful to gather in future studies.

In summary, there is potential for improvement with regards to both the accessibility and content of hand surgery fellowship program websites to meet the needs of current applicants. This study provides an impetus for fellowship programs to update their websites so that they can achieve full information potential, which can help applicants make informed decisions about which programs align with their career goals.