Website evaluation for shoulder and elbow fellowships in the United States: an evaluation of accessibility and content.

HYPOTHESIS AND/OR
BACKGROUND: When examining the access and content related to shoulder and elbow fellowship websites, only 64% of programs had individual websites in a query performed 5 years earlier. The purpose of this study was to re-evaluate content about individual programs listed on the American Shoulder and Elbow Surgeons (ASES) website and on individual program websites and compare the results to prior data.
METHODS: The ASES website was accessed to determine both the number of ASES-recognized shoulder and elbow fellowships and the number of direct links to fellowship program websites. A Google search was also performed to determine the ease of access to fellowship program websites. Each website was then evaluated for content in regard to their recruitment and educational program.
RESULTS: The ASES website includes contact information and a brief description for 29 programs with 40 reported positions. When trying to identify links to program websites, there were functioning links to 6 programs (21%) and absent/nonfunctioning links for the remaining 23 (79%). Through a Google search, there were functioning links to 22 (76%) and absent/nonfunctioning links for 7 (24%) programs. All 29 program websites had faculty listing and program contact info whereas 28 (97%) had a description of their program. In terms of educational content, 17 (59%) included description of operative cases and 18 (62%) had descriptions of rotations/curriculum. DISCUSSION AND/OR
CONCLUSION: Individual shoulder and elbow fellowship program websites provide varied content and accessibility. In the intervening 5 years, there has been minimal improvement in the accessibility of individual fellowship websites from the ASES website.

The Orthopedic Shoulder and Elbow Fellowship programs are recognized by the American Shoulder and Elbow Surgeons (ASES) and the match is run by the San Francisco Match (SFM). There are 29 orthopedic shoulder and elbow fellowship programs recognized by the ASES based in the United States and Canada that are listed on the ASES website (https://www.ases-assn.org/shoulder-and-elbow-fellowships/). When deciding which fellowship programs to apply to, the fellowship's website is an essential resource where applicants can ideally learn more about the program's specific clinical and educational curriculum and priorities.

With the Internet becoming ingrained in our undergraduate and graduate education, more and more residency and fellowship applicants turn to the Internet for information in regard to individual programs. A study of hand surgery fellowship applications found that applicants valued websites more than attendings' opinions when selecting fellowship programs. In addition, 70% of pathology residents rated the fellowship's website as the most important source from which to obtain information.

The accessibility of orthopedic fellowship websites has been studied for orthopedic surgery subspecialties including sports medicine,, adult reconstruction, foot and ankle, hand,, trauma, spine, pediatrics, musculoskeletal oncology, and shoulder and elbow. When looking at the accessibility and content related to shoulder and elbow fellowship websites specifically, the results were poor, with only 64% of programs having individual websites. However, that query was performed 5 years ago, and programs may have updated the information present on their websites since that initial search.

The purpose of this study was to (1) evaluate the content about individual programs listed on the ASES website, (2) evaluate the content listed on each fellowship program's individual website, and (3) examine the change in accessibility and content in the prior 5 years. We hypothesize that there will be an increase in the number of fellowship programs with individual websites and a general increase in the content provided on each website.

Materials and methods

The ASES website maintains a list of all active shoulder and elbow fellowships. The fellowship programs included were active as of September 2019 according to the ASES website, which provides information regarding each program. On the ASES website, it was identified which of the fellowship listings had a link to the individual program's website via a direct link, via links requiring multiple steps, or an absent/nonfunctioning link. The search engine Google was then used to search for “program name + shoulder elbow fellowship” and access to the program website with a direct link on the first page, via links requiring multiple steps, or with an absent/malfunctioning link was noted. Google was used as it is the most commonly used search engine. As per prior studies,, the first 10 search results were reviewed to identify links to fellowship websites.

The investigations into website characteristics focused on 2 domains: recruitment content and educational content. The presence of various subsets of these categories was recorded and reported in a descriptive manner. If information about these domains was obtained through either the ASES website or through the program website, it was recorded as being accessible through the Internet.

Recruitment factors included the following 9 categories: program director/coordinator contact information, program description, salary/benefits, listing of current faculty members, description of fellowship location, selection criteria, application requirements, and current employment of past fellows. For application requirements, if detailed application requirements were noted, we judged this as being present. If they advised to apply through the SFM, we judged this as not being present.

Educational factors included the following 10 categories: information regarding operative experience/case logs, rotation schedule/curriculum, call schedule/responsibilities, description of didactic instruction, meetings/courses, description of clinic schedule/responsibilities, criteria for evaluation of fellow competency, research, examples of research published at the institution, journal clubs. These are categories previously described, but some are more open to interpretation as to what exactly entails “presence” on a website. The details on how they have been previously recorded is lacking. For operative experience, this was deemed to be present if there are examples of case logs or numeric estimations of operative experience. Rotation schedule/curriculum was counted if either a sample schedule or description of the program curriculum was present. Research was counted if there was mention of research requirements or specific opportunities present at the institution. Importantly, we are identifying presence of these domains via online accessibility and not quality or depth of information on these categories.

Results

A review of the ASES database identified 29 shoulder and elbow fellowship programs with 40 reported positions. There were direct links to 4 programs (14%), links requiring multiple clicks for 2 (7%), and absent/nonfunctioning links for the remaining 23 (79%). Through a Google search, there were direct links to 19 (66%), links requiring multiple steps for 3 (10%), and absent/nonfunctioning links for 7 (24%) programs.

The program website and the ASES database were examined for fellow recruitment and educational experience content quality. For recruitment (Fig. 1), the most consistent factors listed were program contact information (29, 100%), faculty listing (29, 100%), and program description (28, 97%). Others included salary (22, 76%), location description (5, 17%), selection criteria (3, 10%), and past fellows (7, 24%). In addition, 13 (45%) had application requirements, though the majority advised to apply through the standardized application through SFM.

Figure 1

Percentage of programs listing specific recruitment content on the ASES website or individual program website.

The most consistent educational component mentioned was regarding a research requirement/expectation (26, 90%), though only 2 programs (7%) provided examples of research previously performed by fellows. Information about meetings/courses was present in 4 (14%), description of office/clinic duties or schedules in 14 (48%), a numeric description of anticipated operative experience/case log examples in 17 (59%), information about rotations/curriculum in 18 (62%), evaluation criteria and journal club information in 4 (14%), didactics in 15 (52%), and call responsibilities in 11 (38%) (Fig. 2).

Figure 2

Percentage of programs listing specific program educational content on the ASES website or individual program website.

Discussion

The content presented on an orthopedic shoulder and elbow fellowship website is often the first and only source of information a fellowship applicant obtains in regard to that specific fellowship. Therefore, it is important that these websites be easily accessible and provide adequate content for applicants to make an informed decision on whether or not to apply to that fellowship. Our hypothesis was found to be incorrect, for although there were improvements in some areas, we found no change or a decrease in many others.

The orthopedic shoulder and elbow fellowship match is administered by the SFM and sponsored by the ASES. There is no direct list of all shoulder and elbow fellowships present on the SFM website, but the ASES website does maintain a complete list of all ASES-recognized shoulder and elbow fellowship programs. For applicants, there is information about programs on the ASES websites as well as on individual websites, though only 6/29 programs on the ASES website had an accessible link to the individual program websites. Through a Google Internet search, only 22/29 programs had an individual fellowship website.

When combining the ASES program-specific website and each fellowship's individual website, less than 60% of all programs had information regarding operative experience, descriptions of rotations/curriculum, on-call responsibilities, application requirements, selection criteria, and information regarding past fellows. Although nearly 100% of programs have content related to a program description and faculty listing, only 25% of programs had information regarding current or past fellows and their current employment.

Our study was performed as an updated assessment of accessibility and content in shoulder and elbow fellowship websites and is comparable to a study by Young et al where the authors accessed the ASES website in 2014, 5 years before our study. In the intervening 5 years, there has been no improvement in the accessibility of individual shoulder and elbow fellowship websites from the ASES website, with 5/28 programs having direct functional links to individual websites in 2014, compared to 4/29 with direct links and 2/29 with a link requiring multiple steps in 2019. However, there has been an increase of 10% in the number of fellowship programs with individual websites, increasing to 22 from 18. In terms of educational criteria (Table I), there was a decrease in the number of programs discussing research opportunities and case descriptions, an increase in the number of programs discussing rotations/curriculum, and the same number discussing on-call expectations. Over the 5-year interval, there has been essentially no change in information in regard to current/past fellows and the current employment of past fellows (Table II). In terms of specific content on the ASES website (Table III), there was an increase in the number of programs listing contact info for the program (program coordinator or program director) from 17 up to 29, and nearly 100% of websites had a description of their respective program.

Table I

Comparison of education criteria

	ASES website		Individual website
	Young et al14 (n = 28)	Current study (n = 29)	Young et al14 (n = 18)	Current study (n = 28)
Research requirements	23 (82)	21 (72)	16 (89)	19 (68)
Current and previous research	0 (0)	0 (0)	2 (11)	2 (7)
Rotation schedules	8 (29)	12 (41)	7 (39)	15 (54)
On-call expectations	6 (21)	6 (21)	4 (22)	9 (32)
Journal clubs	9 (32)	5 (17)	8 (44)	6 (21)
Case descriptions	25 (89)	12 (41)	16 (89)	11 (39)
Meetings and conferences sponsored	14 (50)	1 (3)	12 (67)	4 (14)

ASES, American Shoulder and Elbow Surgeons.

Data for Young et al were collected on July 2014. Data for the current study were collected on September 2019. Total number of programs for each criteria are listed with the percentage in parentheses.

Table II

Comparison of fellow and faculty information

	ASES website		Individual website
	Young et al14 (n = 28)	Current study (n = 29)	Young et al14 (n = 18)	Current study (n = 28)
Previous fellows	0 (0)	0 (0)	5 (28)	7 (25)
Job choice of previous fellows	0 (0)	0 (0)	4 (22)	7 (25)
Attending faculty	27 (96)	28 (97)	14 (78)	21 (75)

ASES, American Shoulder and Elbow Surgeons.

Data for Young et al were collected on July 2014. Data for the current study were collected on September 2019. Total number of programs for each criteria are listed with the percentage in parentheses.

Table III

Comparison of recruitment information

	ASES website		Individual website
	Young et al14 (n = 28)	Current study (n = 29)	Young et al14 (n = 18)	Current study (n = 28)
Description of application process	12 (43)	3 (10)	14 (78)	13 (46)
Coordinator/director contact info	17 (61)	29 (100)	12 (67)	20 (71)
Fellow salary	16 (57)	18 (62)	5 (28)	11 (39)
Program description	28 (100)	28 (97)	18 (100)	21 (75)

ASES, American Shoulder and Elbow Surgeons.

Data for Young et al were collected on July 2014. Data for the current study were collected on September 2019. Total number of programs for each criteria are listed with the percentage in parentheses.

When compared to other specialties, shoulder and elbow fellowships are lacking in accessibility. A study looking at orthopedic sports medicine fellowships found that 47% of programs listed on the American Orthopaedic Society for Sports Medicine had a functional link to the program's personal website, compared with 25% with shoulder and elbow programs through ASES. Similarly, a study looking at orthopedic foot and ankle fellowships found that 49% of programs had a direct link to fellowship websites from the American Orthopaedic Foot & Ankle Society database, whereas 37% of hand surgery fellowships had a functional link from the American Society for Surgery of the Hand directory.

Although a specific fellowship-associated professional society is a logical starting point for potential applicants, Google search is a search engine commonly used to find program websites. Shoulder and elbow fellowships had similar percentages of programs with either direct links or links requiring multiple steps to access the program (76%) when compared with orthopedic foot and ankle fellowships (72%), pediatrics (74%), spine (71%), and adult reconstruction (77%) but still lagged behind the percentages for orthopedic sports medicine (93%),, hand (96%), musculoskeletal oncology (82%), and trauma (94%).

As the Internet becomes more ubiquitous in our lives, an easily accessible website is arguably of high importance, and certainly of high convenience, for prospective applicants. Among hand surgery fellowship applicants, a study by Meals et al found that applicants valued websites more than attendings' opinions when selecting fellowship programs. In a study of pathology residents applying for fellowship, 70% of pathology residents rated the fellowship's website as the most important source from which to obtain information. Similarly, almost 80% of medical students applying to US internal medicine residency programs found residency program websites helpful in deciding where to apply, and almost 70% found them helpful in determining where to interview.

We propose the following changes that could help improve the ease of access to information regarding each individual fellowship program. First, ensure all fellowship programs have a working website on the ASES domain. Second, each ASES website should contain a standardized list of information. Potential criteria include program director and program coordinator contact information, program location, specific applicant requirements, selection criteria, current fellows, past fellows, current faculty, operative experience/fellow case logs, rotations/curriculum, on-call responsibilities, office/clinic responsibilities, and examples of recent research projects/publications. This helps provide each applicant a standardized list of criteria from which to compare each program.

There were several limitations to this study. One, we attempted to re-create the criteria used to assess website content in the Young et al study, but there may be differences secondary to differing interpretations of each criteria by the differing authors.

Second, we only assessed the presence of a certain criteria on each website and did not attempt to evaluate the quality of the content. Although some programs may have had more detailed notes on application requirements, others simply referred the applicant to ASES for their application. Some programs were more descriptive than others regarding the clinical schedule and expectations. A program may make mention of their operative experience and at the same time not provide in-depth case logs. Some had more detail about how the curriculum is implemented in the program vs. noting the topics that are expected to be covered throughout the year, and the same went for didactics where some noted the schedule and others had further detail about the structure and topics to be covered. Research requirements/expectations were incredibly variable, with most programs noting a research requirement and others giving more explicit details about the type of projects to be expected, though only a minority (7%) actually had examples of previous research. Although this study did not look into the details about the research aspect of fellowship, it is notable that despite the nearly ubiquitous research requirement there is very little guidance on the actual machinations of accomplishing this and presents as an area of improvement on the individual and ASES websites.

Third, when using the Google search engine, we only analyzed the top 10 matches for individual fellowship websites, consistent with prior studies., It is possible that websites could be found further down in the list of results.

Finally, this search was performed in September 2019, and the ASES website and individual fellowship websites may have updated their content since the time of our search.

Conclusion

Improving the accessibility and quantity of information on orthopedic shoulder and elbow fellowship websites may benefit both programs and applicants. Currently, websites for shoulder and elbow fellowship programs recognized by the ASES provide a varied and relatively poor accessibility and content. Although the ASES lists all programs with a brief description, having a direct link to each individual program could be beneficial. An effort should be made by individual programs to improve their websites, potentially with standardized information. Having easier access to information could allow applicants to make more informed decisions regarding their fellowship applications.

Disclaimer

The authors, their immediate families, and any research foundations with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.