A Nationwide Survey of Educational Resource Utilization and Perception Among Internal Medicine Residents.

BACKGROUND: New virtual resources ("novel resources") have been incorporated into medical education. No recent large studies about their use and perception among internal medicine (IM) residents exist.
OBJECTIVE: Characterize the use and perceived helpfulness of educational resources.
DESIGN: Nationwide survey from December 2019 to March 2020. PARTICIPANTS: IM residents in the USA. MAIN MEASURES: Residents were surveyed on their use and their perceived helpfulness of resources for both attaining general medical knowledge and for point-of-care (POC) learning. Traditional resources included board review resources, clinical experience, digital clinical resources (e.g., UpToDate), journal articles, pocket references, professional guidelines, textbooks, and residency curricula. Novel resources included Twitter, video streaming platforms (e.g., YouTube), online blogs, podcasts, and Wikipedia. KEY
RESULTS: We had 662 respondents from 55 residency programs across 26 states. On average, residents used 9 total resources (7 traditional and 2 novel). Digital clinical resources and clinical experience were used by all residents and found helpful by the highest percentage of residents (96% and 94%, respectively). Journal articles were next (used by 90%), followed by board review resources and residency curricula (both used by 85%). Their perceived helpfulness varied, from 90% for board review resources, to 66% for journal articles and 64% for residency curricula, the lowest perceived helpfulness of any traditional resource. Podcasts and video streaming platforms were used as frequently as textbooks (58-59%), but were rated as helpful more frequently (75% and 82% vs 66%, respectively).
CONCLUSIONS: Digital clinical resources, video streaming platforms, and podcasts were perceived as helpful, underscoring the importance of ensuring their integration into medical education to complement clinical experience and other traditional resources which remain highly valued by residents. IMPORTANCE: Our findings can inform residency programs as they transition to virtual curricula in the wake of the COVID-19 pandemic.

INTRODUCTION

Medical education has evolved over the past two decades with the growth of the internet and digital media. In 2004, two-thirds of residents used the internet for health-care-related purposes.[1] In 2006, UpToDate and online peer-reviewed journals were used more frequently than textbooks among residents.[2] Since then, virtual platforms (e.g., YouTube, founded in 2005, or Twitter, founded in 2006) have developed into common sources of medical education. They have been shown to positively impact the educational experience of learners[3] and affect the clinical practice of residents.[4]

Platforms such as YouTube, Twitter, and podcasts have been integrated into graduate medical education to promote engagement, but not all residents engage in online learning opportunities.[5] Incorporating Twitter into board review improved positive outcomes on in-training exam scores among surgery residents,[6] and residency-specific Twitter pages have been incorporated into internal medicine (IM) training programs to enhance resident education.[7] A survey in a single department of surgery in 2016 found that 90% of respondents used videos for surgical preparation, and YouTube was the most highly used source.[8] However, a 2019 study found that YouTube videos demonstrating surgical techniques were found to vary in quality.[9] Podcasts are utilized by pediatric, emergency medicine, and anesthesia residents.[10-12] A survey of pediatric residents who listened to a podcast focused on urinary tract infections in 2016 found that 98% of the residents found it educational and 93% of the listeners found the experience enjoyable.[10]

In the last decade, a single study examined the overall use of novel technologies (e.g., YouTube and Twitter) in IM resident education. This 2010 study of 189 IM residents in five residency programs found that UpToDate was their preferred resource for knowledge acquisition, ahead of journal articles, textbooks, core residency curriculum, and attending chalk talks.[13] Patient care and bedside rounds, rated as effective by 73.5% of residents, was found less effective than all of these resources except textbooks and residency core curriculum.[13] Given the changes in medical education in the past decade, we sought to characterize the current use and perceived helpfulness of novel technologies (e.g., YouTube and Twitter) and traditional educational resources (e.g., UpToDate and textbooks) among IM residents to inform educators.

METHODS

Study Population

We surveyed US IM residents immediately prior to the outbreak of the COVID-19 pandemic in the USA from December 23, 2019, to March 8, 2020. Cleveland Clinic’s Institutional Review Board approved this study as exempt.

Data Collection

Participants were recruited using purposive convenience sampling. We emailed residency programs in the USA through publicly available email addresses and personal contacts. Programs were selected to ensure geographic diversity. Program leaders were asked to email the online survey to their IM residents. Participants were asked to complete an anonymous survey using the SurveyMonkey platform. After completing the survey, participants were offered a chance to enter a $150 Amazon gift card raffle through a separate question not linked to their survey responses.

Survey Development

Questions were informed by existing literature[5,13] and expert opinion. We identified categories and examples of most available learning resources including board review resources and question banks (such as MKSAP, MedStudy, or UWorld), clinical experience (including accessing point-of-care information, presenting patients, bedside teaching, rounding, and informal chalk talks), structured residency educational curriculum, peer-reviewed journal articles, digital clinical resources (such as UpToDate, Epocrates, Medscape, or Dynamed), pocket references, and textbooks (digital or paper). Online blogs, podcasts, Twitter (this was restricted to content other than articles/linked obtained from it), Wikipedia, and streaming platforms (listed as YouTube or other video sites) were included as prior research demonstrated their recent integration into the field of medical education.[5]

The survey included a section assessing resources used “to attain general medical knowledge,” and a section for point-of-care (POC) learning (e.g., when they had to make a decision on the spot). Residents were asked about recent use of the resources (3 months for general medical knowledge, 1 month for POC), and if so, to rate each resource that they used on a 5-point Likert scale ranging from “very unhelpful” to “very helpful.” They were also asked to identify their “go-to” resource for each section. For the POC questions, we excluded clinical experience, podcasts, video streaming platforms, Wikipedia, and residency curricula because of their perceived impractical use for point-of-care decision-making.

The survey asked participant age, gender, ethnicity, postgraduate year (PGY), state of residence, description of the program (with “Community” defined as non-university and non-medical school affiliated, “Academic” defined as university or medical school affiliated, “Military,” or “Other”), and average total daily time spent commuting to their clinical site.

Before deployment, we field tested the survey for content and face validity with 7 residents who were excluded from the study population. We refined the survey based on their feedback. The full survey is available as Appendix A.

Categorization of Resources

We categorized Twitter, video streaming platforms (e.g., YouTube), online blogs, podcasts, and Wikipedia as “novel resources” since they are newer, non-peer reviewed, and less characterized in the literature. All other resources were categorized as traditional.

Analysis

To describe our respondents’ characteristics, we grouped states into geographic regions as defined by the United States Census Bureau.[14] We created a binary indicator or commute time (≤ 30 min vs more and grouped) and residency program type (academic vs other). We compared our results to the Accreditation Council for Graduate Medical Education’s (ACGME) annual demographic information of US residents[15] to determine whether the age, sex, and race of our respondents was representative. Of note, age in the ACGME data is only reported for PGY-1 residents.

We described the use and helpfulness of each resource for general medical knowledge and POC learning. Resources were considered helpful if indicated “very helpful” or “helpful” by respondents. We grouped educational resources used for point of care and general medical knowledge to understand their overall frequency of use and the degree to which they were perceived helpful by recent users. To understand whether resource use varied across geographical region, we compared the use of educational resources by region using chi-square tests. We conducted a secondary analysis comparing the resource use of residents who reported not using or finding helpful structured residency curriculum to those who did use it and find it helpful because IM residency programs spend a large amount of time developing and individualizing this content for their learners. This would allow us to identify which resources are used by residents who do not use or do not find helpful structured residency curriculum and help inform change. A multivariable logistic regression model was used to identify associations between demographics, commute time, program, and region characteristics on the use and perceived helpfulness of residency curricula. The analysis was conducted using Stata 14.0 and Microsoft Excel.

RESULTS

We received responses from 662 residents in 55 programs (from 115 programs that were contacted) across 26 states for a program-response rate of 48%. Program names were not reported by 9 respondents. As shown in Table 1, the average age was 29 (range 21–44) and 45% of respondents were female. Level of training included 38% who were PGY-1, 30% PGY-2, 28% PGY-3, and 4% PGY-4. Academic programs were represented by 81% of residents. Commute time of greater than 30 min was reported by 16% of respondents and did not differ across regions (P = 0.91). The majority of respondents were from the Midwest (39% across 16 programs) followed by the Northeast and South (27% of respondents each from 13 and 18 programs, respectively) and the fewest respondents from the West (7% from 8 programs). When compared to IM resident demographics as reported by the ACGME, we had more respondents from the Midwest (39% vs 23%) and less from the West (6% vs 15%).

Table 1

Survey Respondent Demographics Compared with ACGME Data

Demographic characteristics	Survey respondents (N = 662)	2018–2019 ACGME data (N = 28,420)	P value
Mean age	29.3	29.6	0.002
Sex			0.93
Male	360 (54)	15,497 (55)
Female	296 (45)	11,315 (40)
Ethnicity			0.99
White	(44)	7948 (28)
Asian or Pacific Islander	(30)	6072 (21)
Hispanic	(8)	1528 (5)
Black	(5)	1196 (4)
Other or unknown	(12)	11,648 (41)
Geography*			< 0.01
Northeast programs	182 (27)	159 (29)
Midwest programs	257 (39)	126 (23)
West programs	45 (6)	82 (15)
South programs	178 (27)	170 (31)
Post-graduate year
PGY-1	252 (38)	-	-
PGY-2	199 (30)	-	-
PGY-3	187 (28)	-	-
PGY-4	24 (4)	-	-
Program type
Community programs	127 (19)	-	-
Academic programs	533 (81)	-	-
Commute time < 30 min	247 (37)	-	-

*We received responses from 13 programs in the Northeast, 16 programs in the Midwest, 18 programs in the South, and 8 programs in the West

The regional differences in resource use are shown in Table 2. Most were used at similar rates throughout the different regions, while others varied. For example, professional organization guidelines were used by 87% of residents in the Northeast compared to 75% in the West.

Table 2

Comparison of Percentage of Residents Who Used Educational Resources by Region

	Northeast	Midwest	South	West	P value*
Traditional resources
Board review resources	84%	80%	91%	93%	< 0.01
Clinical experience	99%	100%	100%	100%	0.15
Digital clinical resource	100%	100%	99%	100%	0.44
Journal article	86%	92%	90%	87%	0.15
Textbooks (digital or paper)	62%	55%	61%	53%	0.41
Pocket reference	68%	72%	73%	73%	< 0.01
Professional organization guidelines	87%	85%	79%	75%	0.02
Structured residency curriculum	72%	61%	61%	60%	< 0.01
Novel resources
Blogs	26%	24%	19%	13%	0.18
Podcasts	55%	56%	64%	64%	0.23
Twitter	21%	29%	22%	7%	0.01
Wikipedia	51%	56%	45%	47%	0.11
YouTube	65%	53%	59%	51%	0.05

*Chi-square

Table 3 describes the use of resources for general medical knowledge and POC decision-making. Clinical experience and digital clinical resources were the most frequently used resources for attaining general medical knowledge (100% and 98% respectively) and were also rated helpful by the most residents for general medical knowledge (94% and 96%). Residency curriculum was used by 85% of residents but was rated helpful by 64%. At the POC, digital clinical resources were used and rated helpful by almost all residents. Professional organization guidelines were the next most commonly used resource for point of care learning; 68% of residents used guidelines and 87% of users found them helpful. Digital clinical resources were the most common go-to resources for both general medical knowledge (47%) and point-of-care use (87%).

Table 3

Survey Responses for General Medical Knowledge and Point-of-Care Use

	General medical knowledge		Point-of-care use
	Frequency N = 662 (%)	Rated helpful* (%)	Frequency N = 647 (%)	Rated helpful* (%)
Traditional resources
Board review resources	553 (84)	498 (90)	264 (41)	187 (71)
Clinical experience	660 (100)	621 (94)	-	-
Digital clinical resources	651 (98)	627 (96)	640 (99)	627 (98)
Journal articles	569 (86)	377 (66)	398 (62)	313 (79)
Pocket references	369 (56)	263 (71)	337 (52)	287 (85)
Professional guidelines	515 (78)	428 (83)	438 (68)	380 (87)
Textbooks (digital or paper)	372 (56)	257 (69)	164 (25)	109 (66)
Residency educational curriculum	561 (85)	359 (64)	-	-
Novel resources
Online blogs	149 (23)	85 (57)	89 (14)	54 (61)
Podcasts	388 (59)	290 (74)	-	-
Twitter	155 (23)	91 (59)	50 (8)	27 (54)
Wikipedia	339 (51)	222 (65)	-	-
YouTube	383 (58)	329 (86)	-	-

*Only residents who used a resource rated its helpfulness

On average, residents used 9 total resources (7 traditional and 2 novel). Grouped results are shown in Figure 1. After digital clinical resources and clinical experience, journal articles were the next most common (used by 90% of the residents) and were found helpful by 66% of users. Board review resources and residency curriculum were both used by 85% of the residents and found helpful by 90% and 64% respectively. Textbooks and streaming platforms were both used by 58% of the residents. Video streaming platforms were rated as helpful by more residents than textbooks (82% vs 66%, respectively). Podcasts were used by 59% of residents and were rated as helpful by 75% of them.

Figure 1

Resource use and perceived helpfulness by internal medicine residents for acquisition of either general medical knowledge or point-of-care learning. This graph demonstrates the combined percentage of IM residents who used each resource for either point of care decision-making or general medical knowledge and the percentage of residents who found each resource helpful among users.

Residency curriculum was either found unhelpful or not used by 46% of residents. When compared to the overall group, fewer residents in this group used online blogs, Wikipedia, podcasts, video streaming platforms, pocket references, professional guidelines, and peer-reviewed journal articles, while more used Twitter and textbooks. These differences were minor, and no resource deviated in use by more than 5% between the two groups. Residents in this subgroup found Twitter, textbooks, and podcasts more helpful than the overall group by a margin of 4%, 2%, and 2%, respectively. The biggest differences in perceived helpfulness were observed in peer-reviewed journal articles (found helpful by 59% in the subgroup compared to 66% overall) and pocket references (found helpful by 65% in the subgroup compared to 71% overall). Residents who had a commute time greater than 30 min had significantly higher odds of not finding their residency curriculum helpful (adjusted odds ratio 1.57; 1.02, 2.40). In the adjusted regression, there was no significant association between program type, demographics, and region on odds of not using a residency curriculum or finding it helpful.

DISCUSSION

In our study of over 650 residents across 26 states prior to the COVID-19 pandemic, we found that all residents used digital clinical resources and clinical experience for general medical knowledge or point-of-care learning, and virtually all residents found these resources helpful. The overwhelming prevalence and high favorability of digital clinical resources underscore the importance of ensuring their integration into resident curricula, including when they are converted to virtual platforms. Despite a large amount of effort that goes into developing a residency curriculum, it was only found helpful for general medical knowledge by 54% of respondents (users and non-users). We also found that a large number of residents are using novel resources. Video streaming platforms and podcasts were used as frequently as textbooks, and they were rated as more helpful. Users of video streaming platforms rated their helpfulness similarly to users of professional guidelines, but 25% more residents reported use of professional guidelines. These findings suggest that video streaming platforms may be underutilized relative to their perceived helpfulness compared to some traditional resources.

Our results are similar to previous studies that identified the widespread use and value of digital clinical resources, although residents in our study rated clinical experience as more helpful.[13,16] A survey of 299 IM residents in 2012 found that 85% used UpToDate for point-of-care decision-making at least daily.[16] The 2010 survey found that 89% of residents cited UpToDate as their first choice for answering clinical questions but did not distinguish between general medical knowledge and point of care learning.[13] We found that all current IM residents use some type of Digital Clinical Resource, such as UpToDate or DynaMed. Our results show that their use extends into general medical knowledge with 99% of respondents using them for point-of-care and 98% for general medical knowledge. The same survey determined that 74% of residents found “patient care/bedside rounds” effective. This corresponds to our survey’s clinical experience category, which was rated helpful by 94% of residents. The 20% difference may be attributed to additional categories included in the 2010 survey like “attending chalk talks” and “nursing staff/non-physician providers.”

We found much higher use of video streaming platforms and similar use of podcasts compared to a prior study. A 2014 survey found that YouTube videos were used by a quarter of emergency medicine residents.[17] In contrast, we found that 58% of residents used video streaming platforms. The 2014 survey also found that podcasts were more beneficial than textbooks (endorsed as beneficial by 70% vs 54%) despite similar rates of use.[17] We found that similar rates of podcast and textbook use (59% and 58% respectively) and the perceived helpfulness of podcasts were 75% compared to 69% for textbooks. While these differences may reflect specialty-specific preferences, we believe that an increase in the use of video streaming platforms and podcasts for medical education since 2014 is much more likely given the dramatic increase in educational material online in the last 5 years.

Despite being a commonly used resource, residency curricula were rated helpful by only 64% of residents who used them, which made them the least helpful of all traditional resources. This perception is similar to the previously mentioned 2010 study that found 60% of IM residents viewed residency core curriculum as effective.[13] Developing a residency curriculum is a time-consuming and resource-intensive task, and it is one of the few resources a program can tailor to the perceived needs of its residents. We found that residents who had a longer commute time (greater than 30 min) had higher odds of not using or liking their curriculum. The reason for this finding is unclear. It is possible that a long commute can conflict with commitments of a traditional residency curriculum and the desire to attain work-life balance. For instance, a prior study found that “family responsibilities” was a common inhibiting factor for independent reading among IM residents.[2] Negative implications of long commute time on wellbeing for emergency medicine residents have been speculated[18] but have not been formally studied, especially in IM residents. While our survey did not collect data on work hours, characterizing whether increased hours correlate with decreased perceived value or use of structured curricula is a potential area for future research.

Our findings suggest that strengthening and expanding resources identified as helpful including board review resources and video streaming platforms may enhance the learning experience of residents. Residents may benefit from maximizing the incorporation of digital clinical resources into the traditional curriculum. Novel resources like video streaming platforms and podcasts, which are available remotely, were also rated as more helpful than residency curricula. Understanding this preference is especially pertinent in light of the recent COVID-19 pandemic which has forced hospitals and educational institutions to transition to remote web-based educational structures to comply with social distancing measures.[19,20] While a general shift to virtual medical education has been described[21], very few residency programs have published how they are adjusting their curriculum to comply with the need for social distancing. The one paper we found on IM residents reported using an interactive case-based videoconference modeled after their traditional in-person didactics.[22] It is difficult to predict how the evolving nature of the pandemic will impact clinical experience, the second most highly rated resource. Changes to the clinical experience of the residents will likely vary largely from program to program, and are to a large degree outside of the control of residency program educators. In contrast, changes to the residency curriculum are relatively easy to control and implement. For example, instead of streaming traditional lecture formats for remote education, program leadership could instead spend time curating existing YouTube videos or podcasts that met the perceived educational gaps unique to each residency program.

One of the limitations of this study is that we used convenience sampling and participants who responded may have been different than those who did not. However, we have no reason to suspect participants who did not respond would have used educational resources differently than responders. While age, race, and sex distributions were similar across geographical region, we found participants from the western region of the USA were under-represented compared to total residents as reported by the ACGME (6% vs 15%), and participants from the Midwest were over-represented (39% vs 23%). Our findings of the use and relatively low perceived helpfulness of the residency curricula are limited by the fact that the term may have been perceived in a variety of ways, including lectures, handouts, and online modules. Lastly, we are unable to determine the reason some resources were not utilized. It could be because users found it unhelpful or for other reasons such as unfamiliarity or unavailability of the resource. While it was outside the scope of this project to understand why residents preferred particular resources, future qualitative research can identify the driving factors behind these preferences.

In this study, the largest survey of IM resident resource utilization to date, we found that clinical experience and digital clinical resources were the most widely used resources by IM residents for attaining general medical knowledge and point-of-care learning. In addition, these resources were perceived as the most helpful. Although video streaming platforms and podcasts were used less frequently than almost all traditional resources, their perceived helpfulness surpassed that of textbooks, pocket references, and residency curricula. In our increasing virtual landscape, residency programs could benefit from further incorporating digital clinical resources, video streaming platforms, and podcasts into their educational curriculum. These resources can be used to complement clinical experience and other traditional resources that remain highly valued by IM residents.

(DOCX 19 kb)