A Neurology Clerkship Curriculum Using Video-Based Lectures and Just-in-Time Teaching (JiTT).

Introduction: Just-in-time teaching is an educational strategy that involves tailoring in-session learning activities based on student performance in presession assessments. We implemented this strategy in a third-year neurology clerkship.
Methods: Linked to core neurology clerkship lectures, eight brief video-based lectures and knowledge assessments were developed. Students watched videos and completed multiple-choice questions, and results were provided to faculty, who were given the opportunity to adjust the in-person lecture accordingly. Feedback was obtained by surveys of students and faculty lecturers and from student focus groups and faculty. Student performance on the end-of-clerkship examination was analyzed.
Results: Between October 2016 and April 2017, 135 students participated in the curriculum, and 56 students (41.5%) responded to the surveys. Most students agreed or strongly agreed that the new curriculum enhanced their learning and promoted their sense of responsibility in learning the content. Faculty agreed that this pedagogy helped prepare students for class. Most students watched the entire video-based lecture, although there was a trend toward decreased audience retention with longer lectures. There were no significant changes in performance on the end-of-clerkship examination after implementation of just-in-time teaching. In focus groups, students emphasized the importance of tying just-in-time teaching activities to the lecture and providing video-based lectures well in advance of the lectures. Discussion: Just-in-time teaching using video-based lectures is an acceptable and feasible method to augment learning during a neurology clinical clerkship. We believe this method could be used in other neurology clerkships with similar success.

Educational Objectives

By the end of this curriculum, students will be able to:

Develop a basic approach to the evaluation and treatment of common and serious neurological diseases.

Demonstrate enhanced knowledge of neurological diseases in the following categories: neuromuscular disorders, cerebrovascular disease, headache, movement disorders, epilepsy, neuro-immunology, neuro-oncology, and behavioral neurology.

Report greater comfort and confidence in the clinical evaluation and management of patients with neurological disease.

Introduction

Medical knowledge is growing at an exponential rate.[1] This plethora of medical content translates to an increase in the amount of information a medical student is expected to acquire in a finite amount of time. Given these circumstances, it is necessary for medical schools to embrace a pedagogy that delivers medical information in an effective and efficient manner. Prober and Heath wrote that making lectures more comprehensive and memorable, engaging, and self-paced can improve medical education in the setting of these time constraints.[2]

Traditional lectures use a one-way transfer of knowledge mechanism that typically results in learning that is passive and superficial.[3] Pedagogical methods that promote active learning may be superior, and some early data suggested that these types of methods improve learning outcomes and student attitudes when compared to traditional lectures.[4,5] Advances in technology have facilitated an explosion in innovative approaches to active learning.

One innovative method of teaching that promotes active learning is just-in-time teaching (JiTT). JiTT uses web-based technologies to allow students to answer conceptual questions in order for the instructor to adjust the lecture just in time.[6] JiTT creates an environment where there is a feedback loop between the classroom and web-based materials. Because instructors know how students have performed on a pretest, they can identify and address any misconceptions students may have. Thus, both students and instructors are better prepared, allowing for more efficient use of classroom time. JiTT may promote good study habits and interactivity, lower attrition rates, and increased student satisfaction and cognitive gains.[7]

Prober and Khan presented a proposed reimagined model of medical education based on Khan's described education model for K–12 students.[8] This reimagined model encompasses a framework of core evergreen knowledge that can be created through a series of short (under 10-minute) instructional videos viewable at a student's own pace. This flipped classroom experience can allow for better use of classroom time. Another component of this model involves the creation of dynamic and interactive sessions, as well as encouraging students to partake in deeper learning activities.[8]

We have developed a curriculum that combines both JiTT and a flipped classroom for medical students on their neurology clinical clerkship. Students acquire core evergreen knowledge through short (5- to 12-minute) video-based lectures (VBLs) recorded by faculty and subsequently answer United States Medical Licensing Examination–style questions in preparation for class. Student performance on these questions is provided to the instructor, as part of the JiTT component of the curriculum. Students can watch these VBLs as many times as needed in order to obtain mastery of the material and directly send questions to the instructors on what they find most difficult.

Our curriculum builds on the success of other VBL teaching modules, with the added element of pre-lecture knowledge questions that can be used to tailor the classroom experience. The curriculum is unique in providing a broad overview of clinical neurology aimed at medical students in the clinical clerkship.[9-12]

Methods

This curriculum was developed for medical students at the Yale School of Medicine, who participated in the intervention as part of a 4-week neurology rotation in the 12-week Medical Approach to the Patient clinical clerkship. Every student in these clerkships was given the opportunity to participate in the JiTT curriculum and was provided with an introduction to the project. Participation in the curriculum was voluntary, and no identifiable individual student information was collected. The period of implementation extended between September 9, 2016, and June 16, 2017, for a total of nine 4-week clerkship blocks. From September to December 2016, the students were in their third year of medical school. Because of a change in our medical school's curriculum, from January to June 2017, there was a mixture of students in their second and third years of medical school.

In addition to a number of other structured educational experiences, during their neurology rotation students all participated in a total of 10 didactic lectures, which included eight lectures in different neurological subspecialties and two lectures in neuroradiology. The main objective of these lectures was to build core medical knowledge, specifically as it related to diagnosis and treatment of neurological disease. The JiTT curriculum was based around the eight existing neurology subspecialty lectures. Each of the eight components of the JiTT curriculum was self-contained, so facilitators could choose to implement all or only a selected portion of the curriculum, based on their existing clerkship didactic curriculum.

Educational Intervention

Eight VBLs and sets of multiple-choice questions (MCQs) were developed in conjunction with faculty lecturers. An iterative approach was used, so that for each 4-week block between September 2016 and April 2017, one to three lectures were developed and introduced, and faculty and student feedback could be applied to subsequent lectures. Lecturers were asked to identify three to five of the most important themes and concepts addressed in their existing didactic lecture. Using these themes and concepts, four to eight MCQs were developed by two of the authors. These questions were based on clinical vignettes and were written in accordance with methods outlined in the National Board of Medical Examiners (NBME) question-writing manual.[13] The questions were then reviewed and edited by the faculty lecturer before being finalized. The final MCQs, with answer explanations, are included in Appendix A.

VBLs were developed by two of the authors and edited for content by each of the faculty lecturers. The template was created using PowerPoint, and the VBLs were recorded by the faculty lecturer and edited using Camtasia screen-capture and video-editing software. The videos were designed with a uniform style and were brief (5–12 minutes), focusing on a small number of teaching points, and easily watched by a busy medical student in a single sitting. Several published recommendations regarding VBLs were incorporated into the development of the videos, as was the prior experience of the authors.[9-12,14,15] Details about the lectures are outlined in Table 1, and the lectures themselves are included in Appendices B–I.

Table 1.

Description of Video-Based Lectures Used in the Neurology Clerkship

Video	Title	Duration	Date Introduced	Topics Covered
1	Neuromuscular Disorders	6:33	October 2016	Peripheral nervous system overview, amyotrophic lateral sclerosis, Guillain-Barre syndrome, myasthenia gravis, muscle disease, monitoring respiratory status.
2	Cerebrovascular Disease	8:45	November 2016	Overview of ischemic and hemorrhagic stroke types, transient ischemic attack, acute management of ischemic stroke including thrombolysis, etiology of acute ischemic stroke.
3	Headache	10:32	November 2016	Migraine physiology, epidemiology of headache, migraine clinical features and treatment, cluster headache clinical features and treatment, trigeminal neuralgia clinical features and treatment.
4	Movement Disorders	11:10	January 2017	Definition and terminology, an approach to classifying movements, comparison of common causes of tremor, dystonia, Parkinson disease and Parkinson-plus syndromes, other common hyperkinetic movement disorders.
5	Epilepsy	8:45	February 2017	Definitions and terminology, differential diagnosis of seizures, seizure classification, diagnostic tests in seizure evaluation, risk of recurrence and when to treat, considerations in choosing antiepileptic drugs.
6	Neuro-immunology	12:18	April 2017	Epidemiology of multiple sclerosis, clinical presentation of multiple sclerosis, multiple sclerosis diagnostic criteria, multiple sclerosis treatment, neuromyelitis optica, other common neuro-immunologic disorders.
7	Neuro-oncology	9:51	April 2017	Primary brain tumors, metastatic brain tumors, brain tumor treatment, paraneoplastic syndromes.
8	Behavioral Neurology	5:12	April 2017	Neuroanatomy of cognition, Alzheimer disease, frontotemporal dementia, Lewy body dementia, vascular dementia.

Students were instructed to view the VBLs and complete the MCQs prior to each lecture, although participation was voluntary. Instructions were sent to students by email at least 5 days prior to the in-classroom lecture, with a deadline for completion of at least 24 hours prior to the in-classroom lecture in order to allow time to send student results to faculty lecturers. The VBLs were viewed on a video-sharing platform, with a link at the end of each video to an online survey platform that contained the MCQs and answers. The online survey software allowed students to get immediate feedback on their performance on the MCQs. Aggregate student performance was sent to faculty lecturers, along with instructions to modify their classroom lecture accordingly. At the end of the online survey, there was a free-text box giving students the opportunity to identify which elements of the VBLs they found most confusing. These comments were also provided to faculty lecturers.

Technical Considerations

VBLs should be uploaded to a web-based platform for easy access by students. Options include video-hosting platforms or learning management systems (LMSs) with file-sharing capabilities. MCQs should be uploaded to a web-based survey platform or LMS that is capable of compiling results for each student cohort.

Faculty Development

Participating faculty were provided with an in-person explanation of the JiTT format and were advised that they would receive data about student performance on the pre-lecture MCQs before the scheduled lecture time. Faculty were encouraged to use a case-based, interactive format for in-person sessions and to adapt the session based on student performance on the MCQs, spending more time on topics with poor MCQ performance and less time on topics with good MCQ performance.

Preparing Students for the Curriculum

At the start of the clerkship (e.g., during orientation activities), the curriculum was explained to students, including the format and learning objectives. Students were told to expect further instructions about the curriculum by email throughout the clerkship.

Assigning VBLs and MCQs to Students

An email was sent to all students approximately 1 week before each scheduled in-person session, with instructions on how to access the VBL and set of MCQ questions. Students were given a deadline on when to complete the MCQs, usually at least 24 hours before the scheduled in-class lecture time.

Providing MCQ Results to Faculty

MCQ results were compiled and sent by email from administrative staff to the faculty responsible for the in-person session. Ideally, this occurred 24 hours before the scheduled session in order to give faculty members time to adapt in-person activities.

Program Evaluation

Feedback on the curriculum should be obtained through end-of-clerkship surveys, evaluation forms, and in-person group feedback sessions. Faculty feedback should also be sought, either in person or by survey. Student scores on end-of-clerkship examinations should be evaluated to determine potential effects on learning.

Outcome Measures

At the end of each clerkship block, students and lecturers were asked to respond to a survey describing their overall satisfaction with the JiTT curriculum (Appendices J & K). These surveys were adapted from surveys used from a previous study of JiTT in a surgical residency program.[16] One of the authors also met with each group of students at the end of the clerkship blocks to seek qualitative feedback, which was used to guide development of subsequent curricular elements.

Viewership analytics were used to evaluate student engagement with the VBLs. We calculated audience retention at various time points during the video, including the 50%, 90%, and 100% time points, as the fraction of viewers at these time points compared to the number of students who opened the video in the first place. We were also able to identify what proportion of students in the clerkship viewed the videos by calculating the number of original views of each video divided by the total number of students enrolled in each clerkship block.

At the end of the clerkship block, all students were required to complete a knowledge exam, which had been developed by the clerkship director. Two of the authors categorized each question in the 30-item knowledge exam by subspecialty. We then compared student performance on items that had been associated with JiTT compared to performance on items without JiTT.

Statistical Analysis

Pre- and post-JiTT student performance on knowledge exam items was compared using the unpaired t test. A Pearson regression analysis was used to examine the relationship between video length and audience retention.

Results

A total of 56 out of 135 students (41%) responded to the postcurriculum survey (Table 2). Over 95% of students felt that the VBLs helped them prepare for the in-person lecture, and over 80% felt that the pre-lecture MCQs helped reinforce lessons they had learned in the video. Students felt strongly that the videos helped them to learn key concepts, and the format made them feel more responsible for preparation for the lecture. Students also enjoyed the format overall and thought that more lectures should be in this format. They felt less strongly that the format allowed them to better identify areas of difficulty and did not feel as strongly that it was helpful to identify these areas.

Table 2.

Student Responses to the Postcurriculum Survey (N = 56)

Question	No. Respondents (%)
	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
Perceptions of prework:
Completing the presession videos helped me with learning the material.	0 (0)	1 (1.8)	2 (3.6)	18 (32.1)	35 (62.4)
Answering the presession multiple-choice questions helped me reflect on/reinforce what I saw on video.	0 (0)	1 (1.8)	8 (14.2)	17 (30.3)	30 (53.5)
Asking about the most difficult/confusing topics in the question section after watching the videos allowed me to clarify aspects of the topic I wanted to learn more about.	0 (0)	2 (3.6)	21 (37.4)	21 (37.4)	12 (21.4)
Lecture format:
The lecture format used in the classroom helped with my learning.	0 (0)	2 (3.6)	8 (14.2)	29 (51.7)	17 (30.3)
Perceptions of learning:
The JiTT/FC format helped me to better learn the key concepts of the given topic.	0 (0)	1 (1.8)	5 (8.9)	23 (41.0)	27 (48.2)
The JiTT/FC format helped me to see what I did not understand about the key points of the given topic.	0 (0)	3 (5.4)	16 (28.5)	19 (33.9)	18 (32.1)
The format (presession videos, clinical vignettes) made me feel more responsible for preparing for the core curriculum sessions.a	0 (0)	3 (5.4)	8 (14.2)	22 (39.2)	22 (39.2)
Overall evaluation of the curriculum:
I believe more lectures should be given in the JiTT/FC format.	1 (1.8)	1 (1.8)	7 (12.5)	21 (37.4)	26 (46.4)
I enjoyed learning using the JiTT/FC format.	0 (0)	2 (3.6)	5 (8.9)	17 (30.3)	32 (57.1)

Abbreviations: FC, flipped classroom; JiTT, just-in-time teaching.

aOne respondent did not give an answer for this question.

Faculty Satisfaction

Seven of the eight faculty members (87%) responded to the faculty survey; responses are shown in Table 3. All faculty agreed or strongly agreed that JiTT helped prepare students for class, and all enjoyed using the format. Faculty did not as strongly agree that the presession MCQ results helped them better understand areas in need of more attention, but most faculty members believed that more lectures should be given in the JiTT format.

Table 3.

Faculty Responses to the Postcurriculum Survey (N = 7)

Question	No. Respondents (%)
	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
JiTT/FC helped prepare students for class.	0 (0)	0 (0)	0 (0)	3 (43)	4 (57)
Presession multiple-choice questions results helped me better understand areas that need more attention.	0 (0)	0 (0)	1 (14)	6 (86)	0 (0)
I enjoy teaching using the JiTT/FC format.	0 (0)	0 (0)	0 (0)	3 (43)	4 (57)
I believe more lectures should be given in the JiTT/FC format.	0 (0)	0 (0)	1 (14)	3 (43)	3 (43)

Abbreviations: FC, flipped classroom; JiTT, just-in-time teaching.

Viewership Analytics

Viewership analytic data were captured from October 2016 to March 2017, after which the video-hosting platform was changed. Data were captured for a total of 15 cycles, including five different VBLs; timing of viewing and audience retention are shown in the Figure. The majority of viewers watched the video the day before the lecture, but a significant minority watched 2 or more days before. Some students watched the day of the lecture, and a few returned to the videos after the in-person lecture was completed. On average, 85% of students were still watching halfway through the videos, and 80% were still watching at the 90% point. There was a rapid drop-off between the 90% time point and the end of the lecture, suggesting that many students closed the video before it ended. There was a trend toward decreased audience retention for longer videos, but this was not statistically significant.

Figure.

Viewership analytics for five of the eight video-based lectures from October 2016 to March 2017. A: Timing of unique views of each video related to the in-person lecture. B: Audience retention throughout the duration of each video (mean audience retention and 95% confidence interval are shown in black). D/O indicates Disorders; Neuromusc, Neuromuscular.

Knowledge Examination Results

Results of the end-of-service exam are shown in Table 4. There was no significant difference in mean pre- and postimplementation subscores in any subspecialty category, nor was there any difference in the overall score pre- and postimplementation of the JiTT format.

Table 4.

Mean Subscores on the Postclerkship Knowledge Examination

Category	% (SD)
	Pre-JiTT/VBL	Post-JiTT/VBL
Dementia	99.7 (0.6)	100 (0.0)
Epilepsy	65.2 (37.9)	76.6 (24.2)
Headache	85.4 (20.5)	85.1 (11.5)
Movement disorders	71.9 (26.4)	71.7 (27.2)
Neuro-immunology	93.0 (10.4)	92.2 (8.0)
Neuromuscular	93.6 (8.5)	88.3 (9.5)
Neuro-oncology	81.1 (27.0)	80.9 (21.1)
Vascular neurology	69.1 (27.4)	69.6 (23.9)

Abbreviations: JiTT, just-in-time teaching; VBL, video-based lecture.

Student and Faculty Qualitative Feedback

Overall, the qualitative comments from students and faculty in interviews and surveys were positive, with particular strengths including the challenging nature of the pre-lecture MCQ questions, appropriate length of videos, and focus of the videos on providing a framework rather than too many details. Several students felt that many instructors did not adequately adjust their teaching style for the in-person lecture. As one student put it, “They simply gave the same lecture they were going to regardless.” A few students felt that a postlecture quiz might provide additional solidification of the concepts.

Discussion

We have found that JiTT using VBLs was an acceptable and feasible method of delivering medical knowledge in neurology to medical students on their neurology clinical clerkship. In our experience, the majority of students believed that this pedagogy enhanced their learning and was enjoyable to learn with; they also felt more responsible for learning the material when using these resources. Most faculty members felt that this resource allowed them to have a better understanding of which content areas needed to be further clarified in class and prepared students for the classroom lecture.

VBLs can be an effective method for delivering information to students. They allow students to engage with material on their own time line and pace[15] and can ensure that students are better engaged when they participate in class sessions. Well-designed VBLs may have distinct advantages over other preparation methods, such as assigned readings, in that they can be completed in a relatively short period of time, the experience of the student is standardized, and there may be cognitive advantages to the simultaneous display of words and pictures.[15,17] Adding the MCQs may further augment the experience as studies of test-enhanced learning have shown that retention can be improved by leveraging the retrieval effect using frequent testing.[18] Our students strongly felt that answering the presession MCQs helped reinforce the lessons of the videos and made them feel more responsible for preparing for the classroom lectures.

There was no statistically significant difference in the end-of-clerkship knowledge examination scores between pre-JiTT and post-JiTT groups. A possible explanation for this finding is the format and content of the end-of-clerkship examination. While we used the lecturers' assigned objectives to create United States Medical Licensing Examination–style MCQs and followed the NBME question-writing manual's recommendations,[13] the end-of-clerkship knowledge examination MCQs did not necessarily follow this particular format or map directly to the objectives. For example, the knowledge exam had negative A-type (one-best-answer) questions such as “All of the following are true EXCEPT,” a format discouraged by the NBME.[13] To correct this, we have created and will subsequently implement a new knowledge exam that has a format consistent with the presession MCQs. We also intend to randomize students to get only one aspect of the curriculum (VBLs only, MCQs only, both, or neither) to explore further which element of the curriculum may be most effective. We recognize that changing the knowledge exam introduces the risk of teaching to the test. However, because the old end-of-clerkship exam focused on some material that was not strongly rooted in the existing learning objectives, we have made a conscious decision to take this risk in order to introduce better alignment into the curriculum. In addition, we did not include any mechanism to study long-term retention in our protocol, and this could be considered in future studies of our curriculum.

Some studies have shown that longer VBLs are less engaging than shorter lectures, and we saw a trend toward lower viewership in our longer videos.[14] Overall, however, our rates of audience retention were high, possibly because of the efforts we made to keep the VBLs short, engaging, and focused on a broad overview of the topic, rather than excessively detailed.[15] We acknowledge that audience retention is not a perfect measurement of student engagement. It is possible that students were playing the VBLs but not actually watching them. However, the fact that there was a rapid drop-off in audience retention at the 90% point in the VBL might provide evidence of true engagement. Students would be unlikely to close a VBL before it was complete if they were not actually watching, while students who were actively engaged in a VBL may have decided to close it during a summary or conclusion section because no new information was being shown.

A limitation to our evaluation of this resource is that it was conducted in only a single institution. There may be different levels of engagement, acceptability, or even student performance at other institutions with different cultures or approaches to didactic learning. Moreover, each institution may have different objectives in the neurology clinical clerkship and may not require the use of all eight of our VBLs. In addition, the student response rate for the survey was only 41%, and thus, it is possible that true enthusiasm for the curriculum was lower than what we report. Students who enjoyed the curriculum less may have been less likely to respond to the survey.

Some students felt the in-person lecture was not always completely adjusted to the JiTT format. This created two major problems. First, there was repetition of concepts that students already had reviewed and understood. Second, there was not enough extra emphasis on the concepts students had identified as particularly challenging. In implementing this resource, we would strongly recommend meeting with each faculty lecturer to explain the process, emphasizing that the classroom lecture may need to be modified based on student performance on the MCQs.

With the rapidly growing wealth of medical knowledge that future doctors are expected learn in a finite amount of time, it is important to make use of innovative advances in teaching and technology. Creating each VBL and set of MCQs required a significant up-front time investment.[19] We hope that by sharing this resource, our efforts can benefit a broader group of learners. Our focus was on producing a curriculum containing standard and evergreen neurological topics that would be common to the neurology clerkship learning objectives of most medical schools. This curriculum allows students to learn the basic framework of key concepts in clinical neurology at their own pace and to potentially revisit these concepts as many times as needed until mastery is achieved. Faculty can become aware of which concepts students are struggling with, letting instructors efficiently address shared misconceptions amongst students. This will allow for an “education that wrings more value out of the unyielding asset of time.”[2]

A. JiTT Multiple-Choice Questions.docx

B. Neuromuscular.mp4

C. Cerebrovascular Disease.mp4

D. Headache.mp4

E. Epilepsy.mp4

F. Movement Disorders.mp4

G. Neuro-Immunology.mp4

H. Neuro-Oncology.mp4

I. Behavioral Neurology.mp4

J. Postcurriculum Student Survey.docx

K. Postcurriculum Faculty Survey.docx

All appendices are peer reviewed as integral parts of the Original Publication.