Availability and Quality of Online Information on Sub-Internships in U.S. Orthopaedic Residency Programs.

BACKGROUND: The purpose of the present study was to assess the availability and quality of online information regarding sub-internships in orthopaedics among U.S. orthopaedic residency programs.
METHODS: Each U.S. orthopaedic surgery residency program web site was assessed for the following 4 criteria: any mention of a sub-internship offered by that program, contact information regarding the sub-internship, a list of learning objectives to be met by the rotating student during the sub-internship, and presence of a web page dedicated solely to the orthopaedic sub-internship. Each web site was given a sub-internship score (SI score) from 0 to 4 based on how many of the above criteria were met.
RESULTS: From the 151 analyzed U.S. orthopaedic surgery residency program web sites, 69 (46%) did not have any mention of a sub-internship and thus received a score of 0, 4 (3%) received a score of 1, 18 (12%) received a score of 2, 20 (13%) received a score of 3, and 40 (26%) received a score of 4. The average SI score was 1.05 for the community-based orthopaedic residency programs, compared with 1.98 for the university-based orthopaedic programs (p = 0.003). Subgroup analysis based on SI scores (0 vs. 1 to 4) revealed that the higher-score group (1 to 4) had a higher percentage of university-based programs than the lower-score (0) group (80% vs. 62%; p = 0.003) and was associated with a greater number of residents per program than the lower-score group (mean, 26.4 vs. 21.0; p = 0.04). There was a weak association between the SI score and the number of residents in a given program (R2 = 0.074, p = 0.0004).
CONCLUSIONS: The availability and quality of online information regarding sub-internships offered at orthopaedic residency programs in the U.S. are variable. Nearly half of the programs did not have any available online information on their web sites regarding orthopaedic surgery sub-internships. Larger and university-based orthopaedic programs have more robust information regarding sub-internships than smaller and community-based programs. CLINICAL RELEVANCE: There needs to be greater awareness and more uniformly accessible online information regarding orthopaedic surgery sub-internships for senior medical students seeking elective orthopaedic rotations prior to applying for residency training.

Orthopaedic surgery is widely considered one of the most competitive specialties for a fourth-year medical student to match into a residency position. In the 2018 National Residency Matching Program (NRMP), 171 Accreditation Council for Graduate Medical Education (ACGME)-approved orthopaedic surgery programs listed 742 PGY-1 (postgraduate year-1) positions[1]. There were 849 U.S. senior student applicants, and 4 unfilled positions, with an overall match rate of 81% for U.S. senior students[1]. It is well known that securing a position at an orthopaedic surgery residency program is a competitive process. Therefore, applicants put great effort into making their application as strong as possible in order to be accepted into a desirable residency position. However, as reported by Baldwin et al., the orthopaedic sub-internship or “away rotation” has recently become a critical factor in successful orthopaedic surgery residency applications[2]. Although that study was conducted in 2009, the study design assumed that only 75% of students would be enrolled in a sub-internship. In 2018, we can assume that nearly every applicant to orthopaedic surgery enrolls in a sub-internship and that, if the 2018 student demographics were applied to the 2009 study, the importance of the sub-internship would have been even more profound.

The sub-internship (also referred to as sub-I, acting internship, AI, away rotation) is typically a 4-week rotation that fourth-year medical students enroll in at an institution that is often different from their home institution. One of the primary goals of the sub-internship is to allow both the visiting student as well as the residents and faculty of the host program to assess the rotating applicant’s “fit” in the program on the basis of the student’s interpersonal skills, clinical aptitude, and ability to integrate into the program culture[3]. Thus, besides acquiring clinical skills and knowledge and assessing the appropriateness of the training from the student’s perspective, the sub-internship can be considered a 4-week-long audition during which the host institution can evaluate the desirability of having the applicant as a future resident in its training program. This acquaintance may allow residency program directors to separate and stratify applicants with greater precision when ranking applicants for the NRMP “Match.”[3] In addition, while not guaranteed, the sub-internship often gives the student a greater chance at receiving an interview at that institution for a residency position.

While previous reports have examined the content and quality of online information pertaining to visiting students for residency program web sites for specialties such as pediatrics[4], dermatology[5], general surgery[6], neurosurgery[7], and otolaryngology[8], there is little discussion in the current literature regarding the availability and quality of online information for orthopaedic sub-internships. The authors of those reports from other medical specialties came to the similar conclusion that the general quality and availability of information on residency program web sites was inadequate and lacking in quality when pertaining to medical student applicants. The purpose of the present study was to assess the availability and content of online information available for sub-internships in orthopaedics among U.S. allopathic orthopaedic residency program web sites.

Materials and Methods

We browsed the web sites of 151 orthopaedic residency programs on the Electronic Residency Application Service (ERAS) (https://services.aamc.org/eras/erasstats/par/display.cfm?NAV_ROW=PAR&SPEC_CD=260) using the Google search engine on December 28, 2017. A second search was performed on February 23, 2018 to check for any updated information that may have been missed during the initial search. The search terms always followed the same format: “X orthopaedic surgery residency,” where “X” would be the name of the program as listed in ERAS. Each orthopaedic surgery residency program web site was assessed for the following 4 criteria: (1) any description or mention of a sub-internship offered by that program, (2) contact information regarding the sub-internship, (3) a list of learning objectives to be met by the rotating student during the sub-internship, and (4) a web page dedicated solely to the orthopaedic sub-internship offered by that program. Given the lack of any previously published quantifiable measure for grading online content of sub-internships, we allotted each orthopaedic program web site a non-weighted sub-internship score (SI score) of 0 to 4 depending on how many of the above criteria were met for the individual web site. Program web sites that did not have any information regarding sub-internships were given a score of 0, program web sites that met any 1 of the 4 criteria were given a score of 1, and so on. Programs were further stratified according to size, including the listed number of residents and full-time faculty, and according to the program’s geographic location (Northeast, South, Midwest, Southwest, and West). Programs also were categorized according to whether they were university-based or community-based. A university-based residency program was defined as one in which the main teaching hospital for postgraduate orthopaedic training was also the primary hospital affiliated with a medical school. A community-based program, on the other hand, was one in which the main teaching hospital was not the primary teaching hospital for a medical school.

All data were recorded with use of a specifically designed data-collection sheet by a single individual (R.R.). The collected information included the prevalence of available online information and the corresponding SI score. The data were further analyzed for possible relationships between the SI score and various demographic criteria such as the total number of residents, the number of full-time teaching faculty members in the department of orthopaedics, geographic location, and university-based versus community-based status of the program.

Statistical analysis was performed with use of SAS software (version 9.4; SAS Institute). The unpaired t test was used for continuous variables, and the chi-square test was used for categorical variables. SI scores were compared for various subgroups, including by university-based versus community-based programs and by the number of residents and the number of full-time teaching faculty. One-way analysis of variance (ANOVA) was used to compare SI scores of programs among the 5 geographic regions. Pearson correlation was used to assess the relationship between the number of residents or faculty and the program’s SI score. Univariate and multiple logistic regression analyses were utilized. The level of significance was set at p < 0.05.

Results

A total of 151 orthopaedic surgery residency program web sites were assessed with use of the search terms described above. Of these 151program web pages, 69 (46%) had an SI score of 0, 4 (3%) had a score of 1, 18 (12%) had a score of 2, 20 (13%) had a score of 3, and 40 (26%) had a score of 4 (Table I).

Of the 151 programs, 42 were community-based programs and 109 were university-based programs. The average SI score was 1.05 for the community-based programs, compared with 1.98 for the university-based programs (p = 0.003) (Fig. 1). There was a weak association between the SI score and the number of residents in a given program (R2 = 0.074, p = 0.0004) (Fig. 2). There was no significant relationship between the SI score and the number of full-time faculty (R2 = 0.0045, p = 0.1969) (Fig. 3). Subgroup analysis based on SI scores (0 vs. 1 to 4) revealed that the higher-SI-score group had a greater percentage of university-based programs than the lower-score group (80% vs. 62%; p = 0.003). While the higher-score group was associated with a greater number of residents in the program (mean, 26.4 vs. 21.0; p = 0.04), the number of faculty (p = 0.57) and the geographic region of the program (p = 0.13) did not seem to affect the SI score (Table II). Based on multivariate analysis, both variables—the type of program (university-based vs. community-based) and the total number of residents—were independently associated with a higher SI score (p < 0.0001).

Fig. 1

Comparison of the average SI scores for university-based programs and community-based programs.

Fig. 2

A weak correlation was found between program size in terms of number of residents and the SI score (R2 = 0.074, p = 0.0004).

Fig. 3

No correlation was found between program size in terms of the number of full-time faculty and the SI score (p = 0.1969).

Discussion

Our study indicated that nearly half of the U.S. orthopaedic residency programs offered no information concerning a sub-internship. Previous investigators have reported on the importance of the sub-internship’s potential to generate a successful match into an orthopaedic surgery residency position[3]. Despite the established competitiveness of matching into orthopaedic surgery and the perceived importance of the sub-internship, the evident lack of online information concerning the orthopaedic sub-internship is concerning and warrants further attention.

A notable finding in our study was the difference in the average SI score between university-based and community-based orthopaedic surgery residency programs. This difference could be due to various reasons, such as the possibility that university-based programs keep their web sites more up to date than community-based programs do. It also could be due to the possibility that university-based programs may tend to offer sub-internships more frequently than community-based programs do and therefore could have higher corresponding SI scores. Furthermore, university-based programs may attract a different group of students than community-based programs do. The potential causes for this difference between the 2 types of programs and the possibility of self-selection require further study.

Another interesting finding was the linear relationship between the SI score and the number of residents in the program (R2 = 0.074, p = 0.0004) (Fig. 2). Additionally, there was a slightly higher total number of residents in the program web sites scoring 1 to 4 compared with those scoring 0 (mean, 26.4 vs. 21.0; p = 0.04). The size of a program may have been associated with how often the individual program updated its online information regarding the sub-internship. Again, this association may also be due to the possibility that larger programs tended to accommodate visiting students more readily than smaller programs. We found no significant difference between programs with a score of 0 and programs with a score of 1 to 4 with respect to geographic location or the number of full-time faculty.

Rozental et al.[9] utilized a similar approach to ours in assessing orthopaedic surgery department online web sites and concluded that orthopaedic surgery web sites “underutilize the Internet as a source of clinical and educational services.” Our study similarly suggests that a large number of U.S. orthopaedic surgery residency programs are not fully utilizing the Internet for promoting and offering detailed information regarding sub-internship rotations to prospective resident applicants. It is plausible that programs may be able to attract more competitive applicants by having as much online information as possible concerning the sub-internship. With more information at their disposal, applicants can apply to the sub-internship with a better understanding and expectation of the program and their clinical rotation. The orthopaedic sub-internship serves as a 2-way street: program directors may use this additional exposure to assess applicants’ “fit” with the program, and applicants likewise can use this experience to assess the quality of training and culture of the program[10]. Thus, it is likely that programs with more detailed information about their sub-internship would give a more favorable impression to applicants than programs with little to no information about the sub-internship.

Currently, many programs utilize the Visiting Student Application Service (VSAS) in addition to the program’s web site in order to promote their sub-internship[11]. VSAS is a centralized application that organizes all of the orthopaedic sub-internships into 1 location and allows applicants to choose which program(s) into which they wish to enroll for a sub-internship rotation. Most students apply for elective rotations during the spring of their third year of medical school, and many do not have full access to VSAS until January or February of their third year. While VSAS provides information regarding sub-internships for prospective applicants, this information may not be complete as most programs that utilize VSAS have a link to the program web site should the applicant wish to seek additional information. Therefore, programs would still benefit from improving the information about their sub-internships on the program web site as this portal is likely the primary location that applicants browse when seeking more information about the residency program.

While our study was able to generate some interesting findings that will be relevant for both the residency applicant and the training program, it had some limitations. First, although 171 ACGME orthopaedic residency programs participated in the 2018 Match, the data-collection sheet for our study was created in 2016, when only 151 programs participated in the Match. Therefore, the additional 20 programs that participated in the 2018 Match were not included in our analysis. Additionally, there was no method to determine whether the programs that received an SI score of 0 actually offered a sub-internship but lacked online information or simply did not offer a sub-internship at all. The possibility of contacting the appropriate residency program coordinator for these programs was discussed and ultimately abandoned because of the possibility of introducing selection bias. There is also a possibility that a program may have had more in-depth information regarding a sub-internship but the information could not be found because the web page was too difficult to navigate. Additionally, while we did try to quantify the online information regarding sub-internship by using a SI score, this score was not weighted and has not been previously used or validated. Furthermore, residency program web sites are not the only way in which applicants can garner information about sub-internships. Other avenues such as counselling from the Dean of Student Affairs, mentorship from senior students, and web site forums such as Orthogate and Student Doctor Network are common methods that medical students utilize to obtain information on sub-internships that may have been overlooked in our study. Information technology and administrative support for the residency programs are other factors that can influence the availability of residency program web sites that were overlooked in our study. Future studies can assess how these factors impact the quality and availability of orthopaedic residency program web sites. Investigators also can survey matched applicants to gain further insight as to why programs are not keeping their web sites up to date and whether applicants are utilizing these web sites.

In summary, the availability and quality of online information regarding sub-internships at U.S. orthopaedic residency programs are variable. Nearly half of the programs did not have any available online information on their web sites regarding the orthopaedic surgery sub-internship. Larger and university-based orthopaedic programs seemed to have slightly more robust information regarding sub-internships than smaller and community-based programs. There needs to be greater awareness and more uniformly accessible online information regarding orthopaedic surgery sub-internship for senior medical students seeking elective orthopaedic rotations prior to applying for residency training. We believe that both student applicants and programs would benefit from an improvement in the quality of information online regarding sub-internships and that this matter needs to be further studied and discussed among educators as well as in the peer-reviewed literature.

TABLE I

Sub-Internship (SI) Scores Among Programs

SI Score	No. of Programs
0	69 (46%)
1	4 (3%)
2	18 (12%)
3	20 (13%)
4	40 (26%)

TABLE II

Sub-Internship (SI) Score Outcomes According to Various Factors

	No Information (0) (N = 69)	Some Information (1-4) (N = 82)	P Value
Type of program*			0.003
University-based	43 (62%)	66 (80%)
Community-based	26 (38%)	16 (20%)
Size of program
Mean no. of residents	21.0	26.4	0.04
Mean no. of faculty	28.6	32.4	0.57
Geographic region*			0.13
Northeast	25 (36%)	20 (24%)
South	19 (28%)	17 (21%)
Midwest	11 (16%)	25 (30%)
Southwest	8 (12%)	8 (10%)
West	6 (9%)	12 (15%)

The values are given as the number of programs, with the percentage in parentheses.