Taking the Learner on a Journey - An analysis of an Integrated Virtual CME Program in Epilepsy during the COVID-19 Pandemic.

The COVID-19 pandemic has significantly changed the way we treat patients and educate healthcare professionals (HCPs). In summer 2020, the International League against Epilepsy (ILAE) implemented a virtual CME program with three integrated program elements addressing challenges in patient treatment as well as challenges caused by the forced transition to a virtual environment. Despite the highly competitive environment with exponential increase of webinars offered to HCPs, the program achieved high participation and satisfaction rates. Over 60% of participants indicated a change in their clinical practice after the interventions. With our outcomes evaluation, we aimed to better understand how well such an integrated program resonates with the learner and if it can make a difference in a highly competitive environment by supporting educators to become more adaptive and responsive to learner needs. Our pilot project was shown to be well accepted, achieving high satisfaction and perceived impact by the learner. In the light of an upcoming "digital fatigue" and a wish to return to face-to-face, we reiterate the value of the digital approach and recommend continuing along this successful path as we believe that taking a learner on a digital educational journey has been successful in a highly competitive and challenging environment.

Introduction

The COVID-19 pandemic has marked a significant disruption in the lives of people, requiring social distancing and limiting personal contacts. Unsurprisingly, it has become a stress test on healthcare systems in general as well as how medical education is provided to the learner, thus creating unprecedented challenges to medical education [1,2]. Medical congresses have been cancelled and replaced by virtual congresses and webinars leading to an increase in webinars ranging from 300% to 3250%, according to different sources [3,4]. Medical schools and institutions also faced the necessity for a significant increase in online learning. While this was initially well perceived by physicians and healthcare personal in general, physicians felt overwhelmed by the increased amount of offers and frequency of invitations [5,6]. A study from India examining the barriers in using E-learning methods for medical education suggests that Internet and utility infrastructure are major problems in accessing E-learning [7]. Apart from hardware difficulties, Tabatabai [8] argues that maintaining standards in delivering educational content and accessing students’ learning can be extremely challenging when moving from face-to-face to E-learning [8].

The International League Against Epilepsy (ILAE) launched its educational platform “ILAE Academy“ in the middle of the COVID-19 pandemic in July 2021 to provide curriculum-based education to physicians worldwide. In parallel, the ILAE assessed educational program opportunities complementary to their core curriculum that fulfilled physician needs. While for a long time e-learning has been perceived as a replacement of face-to-face activities, more recent research has shown that it can enhance the learning process, making it more effective and accessible, especially in low- to medium-resource countries [9]. As part of this understanding, it was decided to capitalize on the benefits of digitization in health care and education and to develop an educational program that went beyond a one-time intervention, offering a virtual learning journey, based on the best practice in instructional design [10,11]. It was complementary to the ILAE curriculum and had the objective of improving knowledge, understanding and application of current and upcoming scientific research in epilepsy related to COVID 19 and relevant for HCPs active in epilepsy. Through a highly interactive approach, including case-based learning, integrated tests and different digital formats, the project aimed to counteract the “digital fatigue” that has been identified in the literature [5]. Our review revealed that current literature about education under COVID-19 is largely focused on the evaluation of single or distinct series of single interventions. It provides limited insights into theoretical underpinnings of professional development and medical education and its subsequent application of knowledge in the context of COVID-19 [12-14]. Therefore, our project consequently aimed to better understand how well such an integrated program resonates with the learner and if it can lead to differentiation in a highly competitive environment by adding value to providers and physicians.

By sharing our experience about taking a learner on a virtual learning journey, we strive to address the lack of understanding in the current literature and support educators to become more adaptive and responsive to learner needs under this global crisis.

Design & methodology

Program design

The development of the comprehensive educational program was based on the predisposing-enabling-reinforcing instructional framework as developed by Green and Kreuter (see Figure 1 detailing the conceptual assumptions) [15,16].

Figure 1.

Program design based on best practice from instructional learning design

The program consisted of three elements:

A case study as a predisposing activity, in order to create or reinforce “teachable moment” [16].

A webinar as an enabling learning activity, aiming to “develop competence related to teachable moment” [16]: The Virtual Epilepsy Symposium.

3 microlearning modules as reinforcing activity to “assist in recall of competence” [16].

The program was implemented between August 2020 and December 2020 and targeted at Health Care Professionals (HCPs) treating patients with epilepsy. The educational content was defined with the overarching objective to improve knowledge, understanding and application of current & upcoming scientific research in epilepsy related to COVID 19 and relevant for HCPs active in epilepsy. A needs assessment was initially performed based on literature review and complemented by insights from the predisposing activity.

1. Predisposing activity:

HCPs who registered for the Virtual Epilepsy Symposium, starting August 2020, received a link to a case vignette about a patient with COVID-19, discussing how to diagnose the disease and how to manage associated epilepsy. Upon completion of the case vignette, they were invited to participate in a short survey. The survey complemented the literature review and was intended to identify challenges that HCPs were facing during the COVID-19 pandemic. It consisted of open questions such as “what challenges have you faced in your diagnosis and treatment of patients during the COVID-19 pandemic” and “in what way has the COVID-19 pandemic impacted your patients”.

The objective of the case vignette was to

-Prepare participants for the upcoming virtual symposium in order to increase homogeneity in the participant group with regards to level of knowledge.

-Assess participant needs for the upcoming virtual symposium in order to complement the needs assessment based on literature review with the specific needs in the target group and consequently adjust the content of the core activity according to their needs.

2. Virtual Epilepsy Symposium

The Virtual Epilepsy Symposium was implemented as a live symposium on 8 September 2020 4–6pm CEST and repeated as a recording on September 10th 6–8am CEST. Communication of the symposium was based on various channels (ILAE website, email invitation & reminders, regional ILAE chapters).

The agenda development was based on the initial needs assessment. It revealed challenges in patient care reinforced and/or caused by COVID-19 and by the forced transition to telemedicine as well as neurological and psychological consequences of COVID-19 for patients with epilepsy as important knowledge and skills gaps. These were subsequently addressed in the symposium.

The objectives for the symposium were to:

Provide relevant and actual content, complementary to the ILAE curriculum to existing users of the ILAE Academy and thus increase satisfaction.

Enhance understanding of the complex challenges faced by epileptologists caused by the COVID 19 pandemic and the consequences derived from it for the treatment of patients with epilepsy.

Assist epileptologists and general neurologists to deal with these challenges.

3. Microlearnings

The 3 microlearnings, short educational programs of 6–8 minutes duration, were developed following the virtual symposium and consisted of short sequences from the live symposium and complementary slides. As for the virtual epilepsy symposium the chosen content was based on the specific needs of the participants in the symposium and based on the feedback received in the post-symposium survey by ILAE and the survey data provided by Advancing Knowledge in Healthcare Inc. (AKH Inc.)) who provided accreditation in a joint providership with ILAE to the program. Selection criteria included interest of the learner as confirmed by the number of questions related to the topic and by answers to “what was your most important learning” and “what was the most interesting part for you”.

The objective of the microlearnings was to reinforce the learning of the virtual symposium and to provide an educational continuum to the learner. Rollout started in December, 3 months after the symposium.

The program received accreditation by ACCME through a joined providership with AKH Inc.

After completion of the program the case vignette and the microlearnings were moved to the ILAE academy and are still accessible from there.

Outcomes evaluation

Our outcomes evaluation of the 3 program components follows the expanded outcomes framework by Moore et al. [16] using the following data sources:

Google Analytics (Moore level 1, participation analytics).

Survey monkey questionnaires at the end of each educational intervention (Moore level 2/3 (satisfaction & (subjective) knowledge gain (self-assessment by the learner)).

Survey Data provided by AKH Inc.

Subjective Moore Level 3a was limited to the post-meeting survey of the Virtual Epilepsy Symposium.

The organisational evaluation was limited to a formative assessment of the organisational opportunities & challenges faced over the course of the project and the potential of the project to differentiate from a single intervention approach.

Results

A total of 2253 HCPs registered for the program with 468 HCPs participants in the predisposing activity and 954 unique participants in the virtual symposium (see Table 1 for details). The majority of the participants in the predisposing activity as well as at the symposium came from the Asian Pacific Region while Europe and the US were clearly underrepresented.

Table 1.

Registration and attendance at the symposia

	Live Symposium (Sept. 8th)	Recorded Symposium with live Q&A (Sept 10th)	Sept 8th & 10th combined
Registration
Total Registrations	1,638	909	2,547
Total Unique Registrations	-	-	2,253
Registered for both	-	-	294
Attendance
Total Attendance (Unique per airing)	676	318	994
Total Individual Attendance	-	-	954
Attended both (Individuals who attended 8 & 10 Sep)	-	-	40
Duration (minutes)	119	123	-
Average View Time (minutes)	73	73	-

Source: ILAE Google Analytics

Around 25% of those who registered for the virtual symposium accessed the patient case vignette prior to the symposium between August 13th and 10 September 2020. The microlearnings as reinforcing activity after the virtual symposium were accessed by around 30% of the participants since 10 December 2020, with the majority of accesses to the site in January 2021 (Table 2).

Table 2.

Participation case vignette and microlearnings

	Case Vignette	Microlearning 1	Microlearning 2	Microlearning 3
Participation
Unique User Engagement	468	145	94	44
Average View Time (min)	7,16	NA	NA	NA

Source: ILAE Google Analytics

Satisfaction with the program (Figure 2) was confirmed in all surveys with over 90% of participants responding that the symposium met or exceeded their expectations.

Figure 2.

Satisfaction with the program

With regard to the learning impact (Figures 3 and 4) over 60% of participants of learners indicated that the content met their educational needs, being highly relevant for their clinical practice. Over 90% of participants indicated that following the symposium they were better prepared to assess and modify their practice due to the changed environment as well as being better able to discuss the neuropathology of COVID-19. Also approximately 60% of participants indicated a change in their clinical practice following the educational intervention.

Learning impact – ability

Learning impact – behavioural change

Critical assessment of results

The critical questions in order to assess the success of the project were

How did we perform in the highly competitive environment that significantly changed during the planning period?

Did we achieve our learning objectives?

Performance

While no quantitative objectives had been defined, a priori a registration rate of over 2000 and a participation rate of around 1000 can be considered as very successful when benchmarked against comparative projects, especially in the Asia Pacific region where on average only 30% of registrants actually attend the virtual event [17,18]. For Europe and the US, registration and participation rates were below expectations, potentially reflecting the ease of access to continuing education and higher availability of courses. Still other reasons such as impact of COVID-19 pandemic on HCP workload with countries being in different pandemic phases and impact on the health care system may also have impacted participation rates.

Learning objectives

For all learning objectives the learning impact has been confirmed at a subjective Moore 3 level. For details please refer to Table 3.

Table 3.

Findings per learning objective

Objectives	Findings	Comments
Provide relevant and actual content, complementary to the ILAE curriculum to existing users of the ILAE Academy and thus increase satisfaction	95% of respondents indicated content was objective, scientifically sound & current 65% of respondents indicated content was extremely relevant, 30% somewhat relevant 90% of respondents indicated the symposium fulfilled or exceeded their expectations	Due to moving the project from the ILAE academy to ILAE the objective could only be assessed with regards to participants and not in relation to the ILAE academy
Enhance understanding of the complex challenges faced by epileptologists caused by the COVID 19 pandemic and the consequences derived from it for the treatment of patients with epilepsy	>90% of respondents feel better able to assess and modify their practice due to changed needs of their patients with Anti-seizure Medication (ASM) and complex epilepsies within the pandemic > 90% of respondents feel better able to recognise patient risks and know how to deal with anxiety	The achievement of the objective was further confirmed in individual statements in the post-survey: “A practical outlook to facing the current difficulties“
Assist epileptologists and GNs to deal with these challenges	50% of respondents indicated that they will change their current practice 60% indicated that they will seek further information	The achievement of the objective was further confirmed in individual statements in the post-survey: “Sharing of experience of lecturers about dealing with epilepsy patients in this time of pandemic and problems that our patients have because of increased stress, inability to access their physician, Anti-epileptic Drugs (AEDs), etc.”

Discussion

The COVID-19 pandemic had a disruptive effect on medical education forcing educators to explore new approaches under unique and challenging circumstances. This has been a project in continuing medical education where a virtual learning continuum had been set up consisting of different interventions (case-based learning, live interactive online symposium and microlearnings) with needs assessments integrated in the project design to adjust the following intervention based on needs identified in the previous program.

When we developed our program we faced two major challenges related to the COVID 19 pandemic:

First of all our educational content was significantly impacted by the course of the pandemic. While in the beginning of content development, interest and motivation to learn more about COVID-19 was very high, this changed when we moved closer to implementation and a sense of “COVID-19 fatigue” appeared [19]. We addressed this in our program by moving from teaching knowledge about COVID-19 to discussing learnings from the COVID-19 pandemic.

Second we had to counteract a certain “digital fatigue” that had appeared over the course of the pandemic as shown in a study implemented in November 2020 assessing physicians attitudes during the 6 months period between March and September 2020 in which over 70% of physicians felt overwhelmed by the frequency of webinars and online webinars [5]. Furthermore, according to Bassett et al. “increasing the number of options can lead to a choice paralysis and decrease satisfaction; the so-called “paradox of choice” [20].

However, despite this exponential increase of webinars and virtual symposia at the time of implementation, our study participants expressed high satisfaction rates with registration and participation rates also being over average as compared to literature, exceeding our expectations.

Adult learning principles suggest that engaging with the learner over a longer period of time with several interventions leads to better outcomes than one time interventions as repetition of content through multiple interventions is supposed to increase recall and knowledge gain [16]. In our program an immediate online confirmation of registration provided direct access to the first educational intervention, the case study, reducing the barrier for the learner to participate in the first educational intervention which may have lead to the high participation rate in the case study and been a critical success factor in our overall program. The impact of the microlearnings as reinforcing activity cannot be assessed as the participation rate was relatively low, possibly due to the time gap between the event and the launch of the microlearning launch of 3 months.

Our results also show that transition to a digital environment facilitates the integration of the different components and offers direct access for us as educators to the learner as well as for the learner to the educational intervention.

However we have also seen that participation rates vary significantly with high participation rate in the Asia Pacific region and low participation in Europe and the US. We do not have a sound explication for this, various factors may have played a role such as number of webinars available, impact of COVID 19 on care, and communication of the program.

Furthermore we have seen that the live symposium achieved higher participation rates than the recorded symposium that was streamed 2 days later, combining the recorded version with a live Q&A. This is different to the “anywhere-anytime” feature that has been considered beneficial, according to literature [5].

Limitations

Our program took place in summer/fall 2020 in a very difficult environment where the shift from face to face to virtual just started. The high number of concurrent virtual programs was unexpected. At the same time HCPs worked under unprecedented circumstances, with social distancing and uncertainty about the future. Today, one year on, digital conferences have become the “new normal” which may make it difficult to transfer results into today’s environment.

Second the evaluation of the program was based on integrated short post-intervention surveys integrated into the actual educational intervention. This allowed us to receive immediate feedback on educational needs and thus adjustment of the content to the learners needs. Our program did not foresee a structured outcomes evaluation to assess the motivations and barriers to participate in the program and to enable behavioural change. This allows us only to suspect and discuss the potential reasons why we achieved the observed outcomes. Differences on a country level with regards to registration & participation rates cannot be answered.

Finally the evaluation of the learning impact of the program was limited to survey-based subjective feedback of the learner. Knowledge test questions integrated into the learning content of the case study and the microlearnings aimed to strengthen individual knowledge and capability gain of the learner were not evaluated by us.

Recommendations

While we have seen that an integrated virtual CME program achieves high learner satisfaction, confirming that it was able to fulfil educational needs of the learner, we are not yet able to safely state that it is superior to single interventions. The integrated surveys and analysis only allowed assessment of the impact of an individual program element. In order to gain better insights into the impact of the whole learning experience and the motivations and barriers of learners to engage in the learning journey a structured outcomes evaluation based on behavioural models after completion of all three program elements is required. The Theoretical Domains Framework (TDF) is developed by synthesising 33 psychological theories aimed at behaviour change, thus helping implementation scientists and interventionists take on theoretical approaches [21]. It can be used by researchers and practitioners to investigate implementation problems, quality improvement and medical education programs and enables interventionists to better understand the psychological mechanisms of behavioural change.

For future analysis we further suggest to include feedback from the different stakeholder groups involved, such as faculty and providers, in order to get a more comprehensive and objective understanding of current reality. This will also help to better understand differences in registration and participation as seen in our program.

Conclusion

The forced transition by COVID-19 to a digital educational environment has paved the way for digitisation in health care in an unprecedented way and has given us the opportunity to pilot new online approaches and programs. Taking a learner on a digital educational journey was successful in a highly competitive and challenging environment. Our pilot project was well accepted, achieving high satisfaction and perceived impact by the learner. In the light of possible “digital fatigue” and a wish to return to face-to-face learning, we reiterate the value of the digital approach and recommend continuing on this successful path. Our educational project may serve as a first role model for future programs, in combination with intensified research on the motivations, opportunities and barriers driving participation, satisfaction and learning outcomes.