The Content and Accessibility of Orthopaedic Residency Program Websites.

BACKGROUND: Applicants to orthopaedic surgery residency programs face a competitive match. Internet resources such as program websites allow prospective applicants to gauge interest in particular programs. This study evaluated the content and accessibility of orthopaedic surgery residency program websites.
METHODS: Existing orthopaedic surgery residency programs for the 2020 application cycle were identified on the Electronic Residency Application Service (ERAS) website. Individual program websites were accessed through links directly from the ERAS website, and a Google search for each program was performed to corroborate accessibility. Programs websites were then reviewed and evaluated on the presence of 20 criteria selected for their potential to influence resident recruitment (10) and education (10), respectively. The results were compared with the lone 2001 study and with orthopaedic fellowship website analyses.
RESULTS: One hundred eighty-nine orthopaedic surgery residency programs were accredited at the time of the study. Only 6 programs (3.2%) did not have a website identifiable through ERAS or Google searches, leaving a final sample size of 183 websites. Approximately 73.3% of all recruitment content and 44.9% of education content were present on the websites available. There was a significant increase in all available recruitment and education content (p < 0.05) when compared with the lone 2001 study. Orthopaedic residency program websites provide comparable recruitment content at a higher rate (71.1%) than orthopaedic fellowship websites (59.6%) but fall slightly below average in presentation of education content (44.9% vs 45.9%).
CONCLUSION: This is the first study in nearly 20 years to assess the content and accessibility of orthopaedic residency program websites. There is noticeable variability in the presentation of website content, but approximately 73.3% of recruitment content and 44.9% of the educational content were easily accessible through internet search. Orthopaedic surgery residency programs and their applicants may benefit from standardization of program websites and an increase in recruitment and education content.

Introduction

Orthopaedic surgery continues to be one of the most competitive specialties for graduating medical students with an annual match rate of approximately 75% to 80%. The match rate has remained stable despite yearly increases in both the number of applicants and number of applications[1]. The match is designed to provide applicants and programs with an unbiased selection based on their relative rank lists, and significant effort goes into the formulation of these rank lists before match day and even before the interview season.

During the application process and residency search, prospective applicants have nominal resources to evaluate programs, often relying on mentors or current residents with outside knowledge of programs. For prospective applicants and residency programs alike, the primary objective of the interview process is to evaluate candidacy while serving as an opportunity for formal introduction and information gathering. Before the interview, applicants will often turn to online resources in order to learn more about the programs to which they are applying. The Electronic Residency Application Service (ERAS) is the universal application processing service that prospective orthopaedic residents will review, and the ERAS website is a valuable resource with links to orthopaedic residency programs and information about the application requirements of each program accredited by the Accreditation Council for Graduate Medical Education. Another resource is the Fellowship and Residency Electronic Interactive Database (FREIDA) available through the American Medical Association for a fee, but sometimes available without cost through select institutions. Many applicants, however, will supplement their research into residency programs with an online search engine such as Google.

For the 2020 residency application cycle, 1,265 United States and Canadian Medical Graduates (USGs) applied to orthopaedic surgery residency programs with each applicant applying to a mean of 81.5 programs[2]. Each program then reviewed an average of 548.5 USG applications and an additional 39.0 applications from International Medical Graduates (IMGs)[2]. Eight hundred forty-nine orthopaedic surgery residency positions were offered, and of these positions, all but 5 were filled by 714 MD seniors and graduates, 118 DO seniors and graduates, and 12 IMGs[3].

Although studies of the content and accessibility of nonorthopaedic residency websites have been performed[4-7], to the authors’ knowledge, no study has specifically evaluated the content and accessibility of orthopaedic surgery residency websites since 2001[8]. Orthopaedic surgery fellowship websites, however, have been analyzed[9-23], and this study uses a similar protocol[9,10,14,23] to assess orthopaedic residency website content and accessibility.

The primary objective was to measure the content and accessibility of recruitment and educational domains and compare the results with previous data. A secondary objective was to compare applicable domains with orthopaedic fellowship websites. It was hypothesized that the website content and accessibility of orthopaedic surgery residency programs would be greater than the lone previously published article on the subject and comparable with that observed in orthopaedic fellowship programs.

Methods

Existing allopathic and osteopathic orthopaedic surgery residency programs in the United States were identified on the ERAS website. Absent, direct, and indirect links or links requiring more than two clicks were codified. All data were recorded between the dates of March 25, 2020, and April 2, 2020, and data analysis was performed on Microsoft Excel, version 16.35 (Appendix 1, http://links.lww.com/JBJSOA/A202).

Using the criteria adapted from previous studies on fellowship program websites, orthopaedic residency program websites were evaluated based on two domains: recruitment and educational content[9,10,14,23]. Each domain contained 10 criteria, selected for their potential to influence recruitment and education, respectively. Recruitment domains included (1) application requirements, (2) current fellows, (3) current residents, (4) faculty listing, (5) location description, (6) past residents and their rmployment, (7) program contact information, (8) program description, (9) salary and benefits, and (10) selection criteria. Educational domains included (1) call schedules, (2) didactics schedule, (3) evaluation criteria and competencies, (4) examples of research, (5) journal club, (6) meetings and courses, (7) office and clinic descriptions, (8) operative experience and/or case log, (9) research requirements or opportunities, and (10) rotations and curriculum. Data were coded as “0” for absent or “1” for present based on the presence of these criteria in each program’s website. Quantitative or qualitative descriptions of these criteria were sufficient to warrant a score of “1,” with the exception of operative experience. Unless a case log or the number of cases was present, operative experience would receive a score of “0” as qualitative descriptions of operative experience are insufficient to receive a score of “1.”

On the interview trail, the presence of fellows is a talking point that is commonly addressed. Although the merits and detriments of fellow presence on a resident’s educational experience may be argued, it is an important dynamic which may affect prospective applicants and was thus coded as a variable under the recruitment domain. The presence of fellows was categorically assessed, and the websites were given a “0” if no information on fellows was provided or a “1” if there was information available. Programs without fellowships were required to mention the absence of fellows to receive a “1” in this category.

No funding was sought or obtained to support this investigation. The departmental Institutional Review Board declared this study exempt from oversight because the data obtained for this study were freely accessible and publicly available.

Results

One hundred eighty-nine orthopaedic surgery residency programs were accredited and available for application through ERAS. Only six programs (3.2%) did not have a website identifiable through ERAS or Google searches, leaving a final sample size of 183 websites (Table I). Two programs were not participating in the match for 2020, and at the time of data collection, 11 programs had updated their listings to reflect that they were no longer accepting applications for this cycle, inactivating the hyperlink to their program’s website.

Approximately 73.3% of all recruitment data were present on the 183 accessible websites. The most commonly identified recruitment data on orthopaedic surgery residency websites was program contact information, program description, and application requirements, with 98.9%, 96.2%, and 93.4%, respectively. Salary and benefit information was present in 74.3% of websites. Lists of faculty were present in 90.2%, residents in 92.9%, and fellows in 39.3% of websites. Descriptions of selection criteria were scarcely observed (8.7%), but past residents and their employment were more readily available (63.4%) (Fig. 1).

Fig. 1

Availability of recruitment content.

Only 44.9% of all education data were available. Schedules for call, didactics, and journal club were present in 43 (23.5%), 121 (66.1%), and 103 (56.3%) websites, respectively. One hundred twenty-four programs identified meetings or courses that residents attended (67.8%). The office and clinical schedule as well as a quantitative representation of operative experience or case log was present in 42.1% and 11.5% of program websites, respectively. One hundred fifty-nine programs mentioned rotations/curriculum (86.9%), but only 13 reported the evaluation criteria or competencies (7.1%). One hundred twenty-six programs stated research requirements (68.9%), but only 35 provided examples of research (19.1%) (Fig. 2).

Fig. 2

Availability of education content.

Only five recruitment domains and six educational domains were consistent between the original[8] and current study. Of these domains, there was a statistically significant improvement in all criteria listed (p < 0.05) (Figure 3 and 4). Our data were compared with existing data for foot and ankle, hand, joint reconstruction, pediatric, shoulder and elbow, sports, and trauma orthopaedic fellowship websites (Tables II and III).

Fig. 3

2001 to 2020 comparison: recruitment content.

Fig. 4

2001 to 2020 comparison: education content.

Under recruitment content, orthopaedic residency websites listed salary/benefits more than all specialties other than foot and ankle fellowship websites. Residency websites listed past residents/employment more than fellowship websites but provided similar reports of application requirements, program contact information, and program descriptions. Orthopaedic residency programs presented data of current fellows in 39.3% of websites, the highest percentage observed in the studies considered, and more than one standard deviation above the mean.

Under educational domains, 65% to 68% of residency, joints and shoulder and elbow fellowship websites presented meetings/course information, which was more than one standard deviation above the mean. Residency websites trailed only foot and ankle fellowship websites in presentation of rotations/curriculum content with 86.9% of data reported compared with 89.6%. Residency websites consistently fell within one standard deviation of the mean for presentation of content related to office/clinic, evaluation criteria/competencies, research, examples of research, journal club, didactics, and call. Orthopaedic surgery residency programs, however, were more than one standard deviation below the mean regarding data of operative experience/case logs (11.5%).

Discussion

This is the first study in nearly 20 years to evaluate the content and accessibility of orthopaedic surgery residency program websites. In 2001, Rozental et al. used FREIDA, Yahoo (www.yahoo.com), AltaVista (www.altavista.com), and Excite (www.excite.com) to guide their search. The authors came across 154 accredited programs at the time and only 113 websites (73.4%)[8]. Since then, an additional 35 programs have earned accreditation, and all but six have accessible websites (96.8%). As hypothesized, the accessibility and availability of recruitment and educational content regarding individual orthopaedic surgery residency programs has significantly increased.

The use of internet has grown exponentially since 2001; therefore, this study must also be considered in the context of its era and can be treated as a standalone snapshot of the 2020 content of orthopaedic surgery residency websites. Approximately 73.3% of the recruitment domain and 44.9% of the educational domain content were easily accessible through internet search. This study provides a starting point for further refinement of content on individual websites, although only the presence of these domains and not the quality of the presentation was evaluated.

There is significant variability in the content presentation of orthopaedic residency and orthopaedic fellowship websites. Compared with orthopaedic fellowship websites, orthopaedic residency programs provide the most recruitment content but fall below average in presentation of education content. One contributor to poor educational content reporting is that residency programs report operative experience/case log at the lowest rate of any of the studies evaluated. However, this finding may be due to author interpretation because this study did not accept qualitative descriptions of operative experience as sufficient to warrant the criteria as present.

With the rapid proliferation and growing importance of internet resources, it is not surprising that orthopaedic surgery residency program websites have improved in the last 20 years. Perhaps what is more surprising is that not all programs use this opportunity to capitalize on brand marketing to attract applicants who may be interested in the program. As applicants may struggle to make themselves stand out in a crowded pool of qualified students and graduates in search of the same job, so too may programs have difficulty differentiating themselves from one another.

As the number of applications that programs review increases, the proportion of applications receiving in-depth review decreases[24]. To the authors’ knowledge, there is no research to suggest that better information from a website will limit the number of applications or applicants to orthopaedic surgery residency programs. However, uniform transparency through standardization of orthopaedic surgery residency program websites has the potential to benefit both prospective applicants and programs alike. Applicant access to more program content provides for better-informed decision-making. Furthermore, programs have the opportunity to use online resources to attract applicants and enhance their online presence.

Our study demonstrates that there is still opportunity for improvement in the online presentation of both education and recruitment content by residency program websites. To our knowledge, there are currently no transparency or standardization mandates for orthopaedic residency program websites. In the interest of improved mobile access and information-sharing, the authors recommend that influential orthopaedic residency education bodies, such as the American Orthopaedic Association’s Council of Orthopaedic Residency Directors, consider promotion of orthopaedic residency program website standardization.

There are several limitations to this study. All data were collected in late March to early April 2020, at which time approximately 11 orthopaedic surgery residency programs had already updated their ERAS listings to reflect that they were no longer accepting applications. Two were not accepting applications for the 2020 application cycle. Although data were collected during a short interval, we cannot confirm that websites were not altered by individual programs during the time of data collection. In addition, we used the criteria adapted from previous studies[9,10,14,23] to perform this analysis, and there may be variation in author interpretation of the results. We acknowledge that “recruitment” and “education” are arbitrary categories and are not entirely as binary as implied. Rather, they are a categorical necessity to compare our work with that of others. Furthermore, the quality of the content presented by orthopaedic surgery residency programs was not directly evaluated, but the presence of content was. Therefore, although this study may allow direct comparison of program websites solely on the presence of content, the methodology does not support comparisons of website or program quality. A further limitation of this study is that no assessment of data accuracy or interrater reliability is provided because data were collected by a single member of our team during a short interval.

Conclusion

This is the first study since 2001 to assess the content and accessibility of orthopaedic surgery websites and demonstrate improvement in the content and accessibility from previous data. Compared with orthopaedic fellowship websites, orthopaedic residency programs provide the most recruitment content but fall below average in presentation of education content, particularly regarding reporting of operative experience. There continues to be opportunity for improvement in orthopaedic residency website content and accessibility from which both applicant and program may benefit.

TABLE I

Online Residency Program Accessibility

Link Type	Electronic Residency Application Service	%	Google	%
Absent/nonfunctioning link	21	11	6	3
Website with direct link	145	77	181	96
Website with indirect link	23	12	2	1

TABLE II

Residency and Fellowship Website Comparison: Recruitment Content*

Recruitment Criteria	Residency	F & A†	Hand‡	Joints§	Peds#	S & E**	Sports††	Trauma‡‡	Average	SD
Application requirements	93.4%	47.9%	97.0%	82.8%	74.0%	77.8%	70.0%	41.0%	73.0%	19.9%
Current fellows	39.3%	14.6%	32.0%	25.9%	23.0%	27.8%	36.0%	35.0%	29.2%	8.1%
Faculty listing	90.2%	100%	64.0%	65.5%	n/a	77.8%	93.0%	29.0%	74.2%	24.2%
Location description	75.4%	8.3%	n/a	n/a	n/a	n/a	n/a	n/a	41.9%	47.4%
Past residents/employment	63.4%	16.7%	n/a	13.8%	n/a	22.2%	n/a	31.0%	29.4%	20.1%
Program contact info	98.9%	100%	85.0%	56.9%	97.0%	38.9%	n/a	n/a	79.5%	25.7%
Program description	96.2%	97.9%	100%	98.3%	100%	100%	100%	100%	99.0%	1.4%
Salary/benefits	74.3%	100%	28.0%	34.5%	31.0%	27.8%	30.0%	25.0%	43.8%	27.7%
Selection criteria	8.7%	n/a	n/a	n/a	n/a	n/a	n/a	16.0%	12.4%	5.1%
Average	71.1%	60.7%	67.7%	53.9%	65.0%	53.2%	65.8%	39.6%	59.6%	10.3%

n/a = not available; Current Residents were removed from consideration in this figure.

F & A = Foot and Ankle: Khwaja et al., 2020;

Hinds et al., 2016;

Young et al., 2018;

Pediatrics: Davidson et al., 2014;

S & E = Shoulder and Elbow: Young et al., 2016;

Yayac et al., 2017;

Shaath et al., 2018.

TABLE III

Residency and Fellowship Website Comparison: Education Content*

Education Criteria	Residency	F & A†	Hand‡	Joints§	Peds#	S & E**	Sports††	Trauma‡‡	Average	SD
Call	23.5%	18.8%	32.0%	13.8%	34.0%	22.2%	19.0%	20.0%	22.9%	6.9%
Didactics	66.1%	45.8%	68.0%	n/a	69.0%	n/a	69.0%	53.0%	61.8%	9.9%
Evaluation criteria/competencies	7.1%	10.4%	n/a	n/a	n/a	n/a	n/a	20.0%	12.5%	6.7%
Examples of research	19.1%	2.1%	n/a	13.8%	n/a	11.1%	20.0%	n/a	13.2%	7.2%
Journal club	56.3%	41.7%	54.0%	31.0%	54.0%	44.4%	56.0%	47.0%	48.0%	8.8%
Meetings/courses	67.8%	18.8%	21.0%	67.2%	17.0%	66.7%	42.0%	39.0%	42.4%	22.4%
Office/clinic	42.1%	14.6%	38.0%	25.9%	57.0%	n/a	30.0%	33.0%	34.4%	13.4%
Operative experience/case log	11.5%	16.7%	49.0%	96.6%	60.0%	88.9%	64.0%	76.0%	57.8%	31.1%
Research	68.9%	91.7%	75.0%	89.7%	83.0%	100%	61.0%	49.0%	77.3%	17.1%
Rotations/curriculum	86.9%	89.6%	28.0%	31.0%	63.0%	38.9%	28.0%	65.0%	53.8%	25.8%
Average	44.9%	35.0%	45.6%	46.1%	54.6%	53.2%	43.2%	44.7%	45.9%	6.1%

n/a = not available.

F & A = Foot and Ankle: Khwaja et al., 2020;

Hinds et al., 2016;

Young et al., 2018;

Pediatrics: Davidson et al., 2014;

S & E = Shoulder and Elbow: Young et al., 2016;

Yayac et al., 2017;

Shaath et al., 2018.