Evaluating an Oncology Video Curriculum Designed to Promote Asynchronous Subspecialty Learning for Internal Medicine Residents.

Internal medicine (IM) residents frequently see patients in subspecialty clinics. However, there are few published core subspecialty curricula targeted to residents' learning and practical needs, and little guidance exists regarding delivery of core subspecialty content to residents rotating across multiple clinical sites. Our study objective was to evaluate a novel oncology video curriculum for IM residents as a model for asynchronous subspecialty resident learning. Using the cognitive theory of multimedia learning, we developed a five-part oncology video curriculum targeted specifically to the needs of IM residents. All second- and third-year residents rotating in oncology clinics from October 2018 to March 2019 at a single training program were invited to participate. We evaluated curricular demand, efficacy, and acceptability, using completion rates, knowledge tests, and a survey. Twenty-eight of 31 (90.3%) residents utilized the curriculum. Resident knowledge improved after utilizing the modules, by 36.9% from pre- to posttests (95% CI [31.3-42.5]; P<0.001) and 13.7% from pre- to delayed posttests (95% CI [7.5-20.0]; P<0.001). Twenty-four of 31 (77.4%) answered the survey. Most residents agreed or strongly agreed that the curriculum contributed to their knowledge (95.2%) and added educational value beyond the clinical rotation (93.1%). Our curriculum evaluation supports the asynchronous delivery of oncology education targeted to the learning needs of IM residents using a novel core video curriculum. These curricular methods provide a model for delivering subspecialty education to IM residents with complex and busy schedules.

Introduction

Independent physician practice in internal medicine (IM) requires a wide range of knowledge spanning multiple subspecialties to provide high-quality patient care after residency. However, clinical and other competing responsibilities frequently take precedence over dedicated learning for busy residents, who must often learn “on the job.” Despite the need for oncology and other subspecialty education for IM residents, both for the American Board of Internal Medicine (ABIM) boards and for future practice, there is a striking lack of published core subspecialty learning materials targeted to IM residents. IM residents need to finish residency feeling knowledgeable and confident caring for patients with cancer, currently the second leading cause of death in the United States [1]. The literature lacks guidance on how to teach core oncology topics to residents, particularly those in programs with busy schedules that do not allow for synchronous didactics [2]. Innovative approaches are needed to address this gap, since curricula are crowded, residents lack abundant spare time to devote to learning, and residents are rarely all in the same place at the same time—now more than ever due to the SARS-CoV-2 pandemic.

Asynchronous learning in residency programs for other specialties has been shown to have similar or better efficacy than traditional lectures [3], and may be a useful addition to IM residency curricula for busy residents with complex ambulatory schedules [4]. Therefore, to address this educational gap, we developed and evaluated an asynchronous oncology curriculum targeted to the needs of IM residents.

Based on a targeted needs assessment described in the Methods, we chose videos as our educational medium. There are few published video-based subspecialty curricula for educating IM physicians, and even fewer targeting residents [5-7]. There is one published online oncology curriculum for residents, but the content was presented primarily in a text-based format [8]. Other existing online oncology videos either target the public, researchers, medical students, or higher level learners (oncology fellows or attendings) [9-14].

Conceptual Framework

We utilized the cognitive theory of multimedia learning (CTML) to produce our videos [15]. CTML is based on cognitive load theory (CLT), which describes how to optimize learning to facilitate processing of key information in the learner’s working memory for encoding in long-term memory [16-18]. Cognitive load (CL) is the mental effort that a given task requires from an individual’s working memory. CTML proposes a number of solutions to avoid cognitive overload in multimedia learning in order to focus learners’ sensory channels on key aspects of multimedia content, and thus reduce CL to save space in the limited working memory.

We focused on three key CTML principles: signaling, segmenting, and weeding [19]. Signaling utilizes on-screen text or symbols to highlight important information. Segmenting is the chunking of information to allow learner engagement with smaller pieces. Weeding is the elimination of interesting but extraneous information from the video. This learning theory, along with our own experiences as generalist and subspecialist trainees and educators, informed our conceptual framework.

In this article, we (1) describe the development and evaluation of our oncology video curriculum, and (2) describe the asynchronous integration of our curriculum into a residency rotation across multiple clinical sites.

Methods

Setting and Participants

The University of California, San Francisco (UCSF), is an urban academic medical center with an IM residency program of approximately 180 residents. Second- and third-year residents spend approximately 50% of their time on ambulatory rotations. One month of ambulatory oncology is required during either the second or third year, during which residents work in oncology clinics approximately two to three half-days per week, with the remainder of the time mostly devoted to primary care clinics, didactics, or research. Our study included all UCSF second- and third-year residents on a required ambulatory oncology rotation between October 2018 and March 2019.

Problem Identification and Needs Assessment

Following Kern’s curriculum development model [20], we identified the problem as inadequate oncology education for IM residents and reviewed the ABIM Blueprint as a general needs assessment to determine which oncology topics to include in the curriculum [2, 21]. We performed a targeted needs assessment by reviewing UCSF and national Accreditation Council for Graduate Medical Education (ACGME) survey data and ascertaining resident needs. ACGME annual survey results from 2009 to 2018 showed that UCSF IM residents experienced higher dissatisfaction with oncology didactic and clinical experiences compared to the national average for oncology and to other IM subspecialties at UCSF.

Prior to designing our curriculum, we performed a survey- and focus group-based targeted needs assessment of UCSF IM residents to identify areas for improvement. Residents requested online resources describing a basic approach to common cancers and more direct learning from oncologists. However, UCSF residents and oncologists work at different clinical sites, and residents are spread across multiple clinical sites at any given time. Given these challenges, we felt that asynchronous online videos featuring oncologists were a logical solution.

Goals and Objectives

The goal of our oncology video curriculum was to provide exposure to fundamental concepts in oncology relevant to general IM practice. We aligned the learning objectives of our videos with the residency ambulatory oncology curriculum objectives, which were (1) describe the initial workup for common malignancies. (2) Explain how treatment differs between early- and advanced-stage malignancies.

Educational Strategies

We created five videos covering four cancers commonly encountered in general IM practice based on our review of the ABIM blueprint: early-stage breast, metastatic breast, colorectal, pancreas, and prostate cancer. Representative screenshots can be found in Supplementary Figure 1. Our attempts to schedule filming with faculty who specialized in lung cancer were unsuccessful. We asked participating faculty to choose video content based on their answer to the question, “What should residents training to become strong general internists know within your field of practice?” We coached the faculty on theory-driven methods for optimal presentation of the content (including how to introduce their content, chunk content effectively, and summarize key points) but not on the content itself.

As part of our instructional design process, we utilized CTML strategies to optimize resident CL. We incorporated on-screen signaling to highlight key points—figures from relevant studies and inset text slides to summarize key information. We segmented the videos as outlined in an introductory roadmap with clear transition slides, and we edited (weeded) the videos extensively to minimize extraneous content. After editing, the videos were each approximately ten minutes or less to match resident needs and maximize participation [22]. We encouraged the featured faculty to use a conversational, enthusiastic presentation style to enhance learner engagement. We uploaded the videos to a private YouTube account and integrated video links into modules using Qualtrics. Each module included a pretest and an identical posttest containing answers and key teaching points.

Implementation

After a two-month pilot with 11 residents who provided positive feedback in response to an open-ended email and did not recommend any changes to the modules, we performed a six-month study with all second- and third-year UCSF IM residents on a mandatory ambulatory oncology rotation or an ambulatory oncology elective. We emailed module links and study information to rotating residents at the beginning of each month and sent weekly reminders until the end of the month. We encouraged but did not require module completion. We worked with oncology clinic faculty to provide ten minutes of optional protected time for review of the modules during each oncology clinic and encouraged residents and faculty to discuss any questions while in clinic together, but residents could also complete the modules outside of clinic. Participation was voluntary without incentive.

Evaluation

We evaluated three specific areas: demand, examining to what extent a curriculum is likely to be used; efficacy, examining whether the curriculum shows promise in helping residents learn the material; and acceptability, examining to what extent the curriculum is satisfying to residents [23].

To examine demand, we measured module completion rates. To examine efficacy, we tested resident knowledge using multiple-choice questions. The pre- and posttests, based on video content, consisted of 4-5 multiple-choice questions with a single best answer and were integrated into the modules. In addition to the immediate posttest, we emailed an identical posttest at least two months later to residents who had completed one or more modules to assess delayed recall. We sent only the test questions relevant to the modules each resident had completed. To examine acceptability, we surveyed residents at the end of each month to understand their experience with the video modules. We created the surveys using Qualtrics, and used 5-point Likert agreement items evaluating learner sentiments about the video modules, as well as multiple-choice and free-response items asking about the logistics of module use, including affordances and barriers. A free-response item asked for general feedback on the modules. The survey only displayed questions to residents about modules they had completed. Residents who completed at least two modules were asked to answer survey questions about the curriculum as a whole, in addition to questions about each individual completed module.

Data Analysis

We deidentified test and survey responses prior to analysis. We calculated descriptive statistics for all test and survey items. We compared mean pretest, posttest, and delayed posttest scores using multilevel linear mixed-effects modeling on repeated measures, using chi-square tests to determine statistical significance of contrasts of marginal linear predictions. We analyzed quantitative data with Stata SE, version 15.0 (StataCorp).

Two investigators (N.I. and L.S.) independently performed an inductive content analysis of all free-text survey responses to identify salient themes and compared results through iterative discussion [24].

Institutional Review Board Approval

This study was deemed exempt by the UCSF Institutional Review Board for Human Subjects Research.

Results

Demand

From October 1, 2018, to March 31, 2019, 31 residents rotated through ambulatory oncology clinics. Twenty-eight (90.3%) residents participated in at least one module—with one completing one module, three completing two modules, five completing three modules, three completing four modules, and 16 completing all five modules. Twenty-four (77.4%) residents responded to the survey, three (9.7%) of whom did not use any video modules and answered questions only about barriers to implementation. There were 130 total video views accounting for 1,009 minutes of watch-time.

All survey respondents who completed at least one module (n = 21) did so because they felt the modules would enhance their learning. Three (14.3%) also believed it was a course requirement. Two (9.5%) felt the videos were too short, one (4.8%) too long, and 18 (85.7%) just the right length. Two (9.5%) watched in clinic and 19 (90.5%) watched outside of clinic. Of the two who watched in clinic, one watched during protected time provided by the preceptor, and one was not given protected time and watched before clinic or between patient encounters. Eighteen (85.7%) residents stated that they were not offered protected time in clinic despite all preceptors previously agreeing to offer protected time. Of these residents, 13 (72.2%) said having protected time would increase the likelihood of watching the videos. Three (14.3%) residents discussed the content with their oncology preceptor, two of whom felt that these discussions improved the educational value of the videos. Of those that did not discuss the content with their preceptors, 15 (83.3%) said they would have liked to.

Efficacy

Test scores improved by 36.9% from pre- to posttests (95% CI [31.3-42.5]; P<0.001) and by 13.7% from pre- to delayed posttests (95% CI [7.5-20.0]; P<0.001; Supplementary Table 1). There were statistically significant increases in scores from pre- to posttests for all modules and significant increases in scores from pre- to delayed posttests for the colon and pancreatic cancer modules. The other three modules also showed an increase in delayed post-test scores, but the differences were not statistically significant.

Acceptability

Of the 21 survey respondents who completed two or more modules, 20 (95.2%) agreed or strongly agreed that the curriculum as a whole contributed to their oncology knowledge. Eleven (52.4%) felt more confident caring for oncology patients after using two or more modules. Of the 87 responses regarding whether individual modules added educational value beyond the oncology clinical experience, 81 (93.1%) were marked agree or strongly agree. Mean survey ratings and standard deviations for survey items regarding the curriculum as a whole as well as each individual module are in Table 1.

Table 1

Quantitative survey results for curriculum acceptability

Item (number of responses)	Meana	SD
Overall curriculum (n = 21)
I was satisfied with the videos as an educational experience.	4.29	0.90
The videos contributed to my knowledge of oncology.	4.67	0.73
I would recommend these videos to other internal medicine residents.	4.29	1.06
After watching the videos, I feel more confident caring for oncology patients.	3.57	0.98
After watching the videos, I feel more confident approaching oncology questions on the ABIM Board Exam.	3.76	0.94
The videos will change my practice caring for oncology patients.	3.52	0.98
The videos contributed positively to my oncology clinic experience.	4.29	0.72
The information presented in the videos is important for internal medicine residents to learn.	4.57	0.68
Colon cancer module (n = 19)
This video contributed to my knowledge of oncology.	4.11	1.20
The information presented in the video is important for internal medicine residents to learn.	3.98	1.13
I would recommend this video to other internal medicine residents.	3.63	1.38
I was satisfied with this video as an educational platform to deliver medical content.	3.74	1.37
I found this video to provide additional educational value to my oncology rotation experience.	4.00	1.25
Early breast cancer module (n = 19)
This video contributed to my knowledge of oncology.	4.68	0.48
The information presented in the video is important for internal medicine residents to learn.	4.68	0.58
I would recommend this video to other internal medicine residents.	4.47	0.84
I was satisfied with this video as an educational platform to deliver medical content.	4.32	0.89
I found this video to provide additional educational value to my oncology rotation experience.	4.47	0.77
Metastatic breast cancer module (n = 16)
This video contributed to my knowledge of oncology.	4.63	0.50
The information presented in the video is important for internal medicine residents to learn.	4.75	0.45
I would recommend this video to other internal medicine residents.	4.50	0.82
I was satisfied with this video as an educational platform to deliver medical content.	4.44	0.81
I found this video to provide additional educational value to my oncology rotation experience.	4.69	0.48
Pancreatic cancer module (n = 16)
This video contributed to my knowledge of oncology.	4.69	0.48
The information presented in the video is important for internal medicine residents to learn.	4.63	0.50
I would recommend this video to other internal medicine residents.	4.56	0.51
I was satisfied with this video as an educational platform to deliver medical content.	4.63	0.50
I found this video to provide additional educational value to my oncology rotation experience.	4.69	0.48
Prostate cancer module (n = 17)
This video contributed to my knowledge of oncology.	4.82	0.39
The information presented in the video is important for internal medicine residents to learn.	4.71	0.47
I would recommend this video to other internal medicine residents.	4.53	0.87
I was satisfied with this video as an educational platform to deliver medical content.	4.53	0.87
I found this video to provide additional educational value to my oncology rotation experience.	4.82	0.39

aLikert scale, 1–5; 1, strongly disagree; 5, strongly agree

Content Analysis

Resident comments addressed three key themes: module structure and content, clinical relevance, and practical considerations. Themes, subthemes, and representative quotes are summarized in Table 2. Additionally, residents identified malignant hematology, chemotherapy mechanisms and side effects, lung cancer, and hepatocellular carcinoma as additional areas they would like to see included in future iterations of the video curriculum.

Table 2

Themes and representative quotes from thematic analysis of resident comments on oncology video curriculum

Theme	Subtheme	Representative quotes
Module structure and content	Organization of the modules	• “I loved how the pre-test primed me to watch out for relevant learning objectives.” • I would like more “case-based questions.”
	Delivery of educational content	• Messages from educators were “clear”, “digestible”, and “succinct.” • Valued consistent “use of take-home points.”
	Quality of educational content	• “The prostate cancer and pancreatic cancer videos were standouts and really excellent.” • “Talking too fast” interfered with their ability to learn.
	Duration	• “The videos are the perfect length.”
Clinical relevance	General internal medicine focus	• “I thought the videos provided excellent approaches to the various malignancies, especially for the purposes of a generalist.” • Some components were too specific: “I will not be prescribing breast cancer therapy.”
	Applicability to future practice	• “[The videos] struck the perfect balance of highlighting critical information that general internists could apply in clinical practice without being overwhelming.”
	Core oncology topics	• “I really liked the pearls on how certain common cancers might present.” • “I appreciated the bread and butter summary of current management of common cancers.” • “[These videos] were more about updating someone who already knew about the disease, rather than starting from scratch. We need videos that start at the beginning.”
Practical considerations	Direct application to residency curriculum	• “The perfect foundation to have going into clinic to see solid tumor patients. Like a ‘Cliff notes’ version so that I am informed enough to know where to start when seeing patients in oncology clinic.” • “Overall I think the videos are an excellent addition to the oncology block and really help fill in gaps in core knowledge where clinic may fall short (since clinic is so dependent on patient census).” • “I didn’t have a single patient in oncology clinics with any of the cancers addressed in the videos.”
	Time prioritization	• “No free time,” as the reason for not participating. • “Unclear whether protected time would be offered.” • “I can pick and choose what to watch (though I watched them all and found them helpful).”
	Additional resources	• “I would like to be able to have a 1-page PDF on each cancer that I could save to an Evernote.”

Discussion

We developed and evaluated a novel oncology video curriculum targeted to IM resident needs and relevant to general IM practice. Our evaluation demonstrated resident demand for the curriculum, efficacy in promoting resident learning, and curriculum acceptability and value among learners.

Our results are noteworthy because the optimal means by which to deliver asynchronous oncology curricula to residents has not been determined. These results are particularly relevant and important in light of the SARS-CoV-2 pandemic, when much of medical education has moved to asynchronous learning, clinical oncology opportunities for residents may be more limited in some training programs, and residents may be taken from oncology clinical rotations to care for patients with COVID-19. While this study does not compare our method of delivery to other possible ways to deliver content asynchronously, we have demonstrated a rigorous, theory-driven, reproducible method for developing a core video-based curriculum for asynchronous delivery of oncology content to residents, and propose that other residency programs can use this curriculum development and delivery process to address program-specific subspecialty educational needs (Fig. 1).

Fig. 1

Recommended workflow to create asynchronous internal medicine subspecialty curricula for residents using video modules

Our data taught us several lessons about developing video-based subspecialty curricula. We learned that residents may utilize asynchronous subspecialty resources without incentivization even when they are optional. Case-based materials and clear take-home points are desirable. Residents preferred materials that covered the basics rather than overly detailed or “cutting edge” information; subspecialized faculty may require guidance to help them stick to the basics. Creating protected time for utilization of asynchronous curricula and providing ample opportunity for direct application to clinical care are crucial.

We found that targeting our video module design to resident learner level and resident practical needs was feasible as curriculum developers and well-regarded by users. The alignment of our curricular goals and objectives with those of the existing residency curriculum, as well as the integration of the modules into the relevant clinical rotation, allowed for maximizing the direct relevance of the modules for the residents. Other institutions and subspecialties may use our development and implementation process for asynchronous subspecialty education for busy residents rotating across multiple clinical sites, during the SARS-CoV-2 pandemic, or for future residency curricular planning as medical education increasingly utilizes remote learning at all levels of training [25].

This study has a few limitations. To maximize resident access to the curriculum, we omitted a control group, so it is difficult to isolate the effect of the video curriculum compared to the rotation. However, the difference between pretest and immediate posttest scores is likely fully explained by the video modules, and residents strongly agreed that this curriculum added additional educational value to their clinical experience. Though the curriculum currently includes only a subset of cancers that a practicing internist will see, future iterations may include all common cancers. Lastly, online videos, particularly in fast-moving fields like oncology, may need to be updated over time.

In conclusion, this highly rated oncology curriculum is a first step toward developing a standardized, asynchronous approach to the delivery of oncology and other subspecialty curricula for IM residents. Curricula such as ours can help ensure that all IM residents have a high-quality subspecialty learning experience before entering independent practice, where broad knowledge across subspecialties is critical. While this curriculum may act a model for all IM programs to adopt and adapt, recommendations regarding competencies or standards in oncology and other subspecialty education should also be considered by residency accrediting bodies and IM professional organizations.

(PDF 9190 kb)