Webinar-based continuing medical education in otorhinolaryngology during the COVID-19 pandemic in Germany: a longitudinal study.

OBJECTIVE: Reliance on webinars for continuing medical education (CME) has increased since the onset of the COVID-19 pandemic. Here, we aimed to evaluate the teaching methods used in these webinars.
DESIGN: Retrospective, longitudinal study.
SETTING: 20 CME-approved webinars, conducted March-December 2020 in Germany, and lasting 2.25 hours each. PARTICIPANTS: Otorhinolaryngologists pursuing CME credits.
INTERVENTIONS: Postwebinar participant assessments of the speaker, effects on practical work, desired scientific content, technical quality, interactions, attention and future training behaviour.
RESULTS: On average, 780 participants joined each webinar. The mean survey response rate was 35% (n=282). When asked how well the speaker had mastered the content, 38% responded 'very well', 44% responded 'well', 14% indicated 'satisfactory' and 4% chose 'sufficient'. The frequency of webinars was considered appropriate by 92%, too high by 4% and too low by 4% of participants. The measured attention of the participants was 90%. After the COVID-19 pandemic lockdown, 68% of participants preferred online teaching. The average costs per participant were €3.50 (about $4.25 or £3.15) per webinar.
CONCLUSIONS: Although the pandemic context likely influenced the results, we conclude that periodic ear, nose and throat webinar training during COVID-19 in 2020 was an effective alternative delivery method. We found high attention rates, high levels of participant satisfaction and low costs. Evaluations and re-evaluations will be necessary to adapt teaching concepts successfully and ensure high levels of teaching and learning efficiency. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Interaction with participants during webinars might play a crucial role in ensuring attendee attention for a successful webinar.

Limited continuing medical education and professional development opportunities, especially at the beginning of the COVID-19 pandemic, might have affected participant responses in this study.

More online continuing education courses should be assessed in future studies.

Introduction

Digitised learning is a product of digital development and global access to the worldwide web. Systematic reviews and meta-analyses have revealed that digitised learning methods, such as online learning, provide advantages for enhancing knowledge and skills. There is a long tradition of effectiveness assessment of online learning for physicians. For example, Wutoh et al1 concluded that internet-based continuing medical education (CME) programmes are just as effective as traditional CME formats in imparting knowledge. Webinars are among the tools available for digitised and online learning, offering access to a large number of participants, enabling interactions and providing a virtual congress like atmosphere. Acceptance of webinar sessions seems to be high in medical education. Several recent reports have emphasised reduced costs, flexibility, convenience and time savings, and many studies have investigated learning efficiency associated with webinar education.2–6

Continuing professional development (CPD) and CME are concepts of lifelong training for medical professionals to maintain and upgrade their knowledge. CME is focused only on one profession and on knowledge/education, whereas CPD is a more inclusive term that encompasses systematic maintenance, improvements and broadening of knowledge and skills. Casebeer et al7 showed that internet-based CME activities are effective for supporting physicians in making evidence-based clinical choices. With the onset of the COVID-19 pandemic, knowledge maintenance and upgrade have required alternative approaches, and webinars have represented one solution. Since 2014, the German Professional Association of ENT (ear, nose and throat) Physicians (Deutscher Berufsverband der HNO-Ärzte) has organised regular webinars that are conducted a few times each year. Although these webinars target mainly attending ENT physicians, the participants also can include medical students, interns, fellows and residents. Based on this empirical experience, we have developed a teaching concept for webinar training.

From March to December 2020, during the COVID-19 pandemic, we conducted 20 ENT webinars. In the present study, we aimed to describe webinar organisation in detail and our experiences with organising a webinar series during the pandemic, to investigate the results of participant evaluations and to present our teaching concepts.

Materials and methods

Webinar criteria and financing

Webinars were conducted for members of the German Professional Association of ENT Physicians. Presenters had to declare all conflicts of interest. Sponsorship by any commercial source was not allowed. Advertising, either for products or institutions, was prohibited. Company or product names had to be blacked out.

Webinar software and teaching content

For webinar training, we used the GoToWebinar software tool from GoToMeeting, which could accommodate up to 1000 participants. From March to December 2020, we conducted 20 ENT webinars, each lasting 135 min. The webinar software included options for asking short questions and sending immediate responses (online voting) and a chat tool for posing questions and making comments to involve listeners during the webinar.

Except for the initial teaching topics, webinar topics were chosen according to proposals and evaluations from participants. The webinar teaching topics are summarised in table 1.

Table 1

Content of 20 webinars (W1–W20) and the number of participants for each

Webinar number	Webinar teaching content	Participants
W1	Hearing and hearing aids	695
W2	Vertigo and dizziness	832
W3	Menière’s disease	847
W4*	Surgery involving the nasal septum	734
W5	Cochlear implants	785
W6*	Sonography in head and neck	841
W7	Vestibular schwannoma	683
W8	Biologicals in chronic sinusitis	738
W9	Hearing implants	749
W10*	Sleep medicine	740
W11	Case reports	564
W12	Diagnostics/therapy in impaired hearing	903
W13*	Middle ear surgery	674
W14	Antibiotics	821
W15*	Diagnostics/therapy for thyroid gland	697
W16	Legal aspects of ENT practice	637
W17*	Infectiology and COVID-19	922
W18	Medical assessment	898
W19	Salivary glands	871
W20*	Tinnitus	978

*Participants were asked to complete an evaluation after the webinar.

ENT, ear, nose and throat.

Webinar coordination team

The initial teaching concept included a speaker (and a cospeaker in some cases), two moderators, one director and two secretaries. Both moderators were ENT physicians. One was the head of a department of otolaryngology, head and neck surgery at a German university, and the other was an ENT specialist working as a consultant for outpatient clinics.

Webinar organisation process

Each speaker, a scientific expert in a selected area, was invited some weeks in advance to speak at a webinar session. The learning objectives and teaching elements of the webinar were discussed with the moderators before it was presented. CME accreditation was received by the local medical board. The moderators and the speaker(s) prepared the required CME questions.

After the presentation was prepared, a test webinar, lasting about 60 min, was conducted (including the speaker(s), moderators and director) to allow for software familiarisation and presentation training. Briefly, the session involved a detailed discussion of the process, including the methodology, interactive elements, technical questions, behaviour in response to technical problems and strategies for troubleshooting. Each speaker prepared short CME questions in an order that matched the teaching content and practised a presentation before the moderator team.

Administrative assistants were responsible for all external communications for the event. Every webinar was announced by email. Webinar invitations were sent twice to all members of the German Professional Association of ENT Physicians during the week before the webinar. A reminder was sent on the day of the webinar. Interesting questions or cases from the participants were sent in advance by email to the moderators. Each webinar was accessible via computer, tablet and smartphone and was planned to last for 2 hours 15 min.

The webinar started with the moderators giving an introduction about the webinar team and comments on the organisational process. Then, the director provided a technical introduction for the participants. Interactive elements (eg, CME questions, chat and online discussion with the speaker) were explained. The participants had the opportunity to ask content-related questions during the webinar via a chat connection. The moderators selected the questions for the speaker to address. The director was responsible for the presentation of CME questions and for responding to technical questions. When technical problems occurred (eg, sound interference, sound delays, picture or video quality issues), they were announced by the director. The moderators and director were in contact via mobile phone.

Interactive teaching elements

The speaker’s presentation included all of the media elements for the main lecture (eg, videos and animations). After presenting a section, the speaker initiated an interactive discussion that lasted at least 10 min, and each webinar involved three or four such interactive discussions. The moderators presented the speaker with interesting questions, either received from the participants before the webinar or extracted from questions posted to the chat during the webinar. Furthermore, as noted, participants had three or four opportunities to join the speaker online in discussions after each content section. In addition, we presented several multiple-choice CME questions during the discussion to involve learners and ask them to test their learning and discuss it afterwards with the speaker. Responses were collected immediately, followed by a discussion of the participants’ answers. Participants also had the opportunity to ask the speaker questions via email after the webinar ended. All webinars were recorded. A video of the presentation was posted on the homepage of the German Professional Association of ENT Physicians immediately after the webinar.

Quality control

Quality control was performed with an online evaluation (table 2) after each webinar. We used elements of the standardised Student Evaluation of Educational Quality8 and added supplementary questions that characterised online training (eg, interactions and technical transmission quality). We used a survey software tool (SurveyMonkey) for evaluations. Immediately after each webinar, we provided participants with a PowerPoint presentation via a QR code and a web link. The evaluation was open for 4 days. Participants received the evaluation link in an email, which was sent twice during the 4-day postwebinar interval. The survey was anonymous, and all questions had to be answered. We issued evaluations for 7 of the 20 webinars. The evaluations included 12 questions (table 2), which were identical for each webinar selected, but the questions were offered in random order. The calculated time to complete the evaluation was about 3 min. Results were analysed after each webinar.

Table 2

Evaluation form provided by SurveyMonkey online software, via a web link or QR code (smartphone)

Question number	Evaluation questions	Response options
Q1	How well was the teaching material mastered by the speaker?	1 (deficient) to 6 (very well)
Q2	How would you rate this speaker overall?	1 (low) to 6 (high)
Q3	How inclined are you to recommend this speaker to a colleague?	0%, 50%, 100%
Q4	How satisfied were you with the content/topics chosen for the ENT webinars up to now?	1–5 stars
Q5	Did the webinar affect any changes/updates in your daily practical-clinical activity?	0 (no influence) to 100% (predominant influence)
Q6	How well was the webinar organised, in terms of presentation, questions, video materials, images, interactions, chat, etc?	1 (extremely well) to 5 (not at all)
Q7	How good was the overall technical transmission quality?	1 (deficient) to 6 (very well)
Q8	Please indicate the percentage of online training compared with in-person training or congresses after the COVID-19 pandemic.	Slider 0%–100%
Q9	How likely is it that you will attend future ENT webinars using this format and this time window?	Most definitely, presumably, unlikely
Q10	The number of credits earned for this webinar was …	Appropriate, too little, too much
Q11	Does your medical association require written proof of CME credits or can you provide evidence digitally?	Written, digital, not sure
Q12	How do you rate the possibility of being able to ask questions?	1 (low) – 6 (high)
Q13	What topics would you like to see presented in future webinars? What suggestions do you have for improvements?	(Free-text response)
Q14	The frequency of webinars during the coronavirus pandemic this year was:	Appropriate, too low, too high

ENT, ear, nose and throat.

All questions included space for adding suggestions for improvement or to justify the answer. Furthermore, participants had the opportunity to share their opinions and make suggestions after the webinar.

Attention

Webinar attention was analysed with the GoToMeeting statistics tool, mainly by calculating the number of participants that completed the CME questions.

Costs

The cost calculation included the fee for the webinar account and for certification, and the costs of office supplies, website maintenance, effort and shipping the certificates of attendance.

Statistical analysis

Statistical analyses were performed with SPSS for Windows V.20.0.0 and Excel (Microsoft 365). The number of participants was obtained from the GoToMeeting statistics tool. Averages and percentages of evaluation responses were calculated with the SurveyMonkey statistics tool for all seven evaluations.

Patient and public involvement

No patients or members of the public were involved in the study.

Results

On average, 780 participants (median 767, SD=104, 564–978) attended each of the 20 webinars. Webinar 20 had the most participants (n=978; topic: tinnitus, table 1), and webinar 11 had the fewest (n=564; topic: case reports). The mean survey response rate was 35% (282 answers, median 293, SD=55) per evaluated webinar.

From the seven evaluated webinars (n=5586 participants), we received complete evaluations from 1974 participants. The evaluation results for Q1 (how well the speaker mastered the material) are shown in figure 1 and those for Q7 (how good the technical transmission quality was) are shown in figure 2. The mean ratings for all speakers also are shown. Speaker expertise (Q2) was rated ‘very well’ by 38%, ‘well’ by 44%, ‘satisfactory’ by 14% and ‘sufficient’ by 4% of participants. Less than 1% of participants rated speaker expertise as‘“poor’ or ‘deficient’.

Figure 1

Summary of the evaluation responses for seven webinars. Results are shown for question 1 (Q1): how well was the teaching material mastered by the speaker? The mean number of respondents was 282 (35% of all participants) per webinar. The y-axis shows the mean percentage of respondents that gave the indicated answer. The x-axis gives the descriptions of options 1–6. The horizontal line indicates the median, and the cross indicates the mean. Less than 1% of participants felt that the speaker’s mastery of the topic was sufficient, poor or deficient.

Figure 2

Summary of the evaluation responses for seven webinars. Results are shown for question 7 (Q7): how good was the overall technical transmission quality? The mean number of respondents was 282 (35% of all participants) per webinar. The y-axis shows the mean percentage of respondents that gave the indicated answer. The x-axis gives descriptions of options 1–6. The horizontal line indicates the median, and the cross indicates the mean. all tables and figures are originally created by the authors for this manuscript only.

For Q3 (‘how inclined are you to recommend this speaker to a colleague?’), 92% of participants gave a ‘100% recommendation’, 7% gave a ‘50% recommendation’ and 1% gave ‘no recommendation’. The mean weighted-average rate of satisfaction with webinar content (Q4) was 4.6 (indicated with a slider ranging from 1 to 5 stars). For how the material influenced their daily practical clinical activities (Q5), 26% reported that the webinars had a ‘predominant’ (100%) influence, 26% reported a ‘probable’ (50%) influence, 24% reported a ‘possible’ (25%) influence, 7% reported a limited influence (<10% probability) and less than 1% reported ‘no’.

The organisation of the presentation, including the questions, video material, images and interactions (Q6), was rated as average. After the COVID-19 pandemic, online training was compared with in-person training (personal presence, Q8). We observed that 68% of participants preferred online teaching and 22% preferred in-person training. On average, when asked how likely they were to take part in one of the next webinars (Q9), 99% of participants responded ‘most definitely’, less than 1% responded possibly and 0% responded ‘unlikely’. The number of CME credits for this webinar (Q10) was assessed, on average, as ‘appropriate’ by 91%, as ‘too little’ by 3%, and as ‘too much’ by 6% of participants. On average, CME credits could be transmitted digitally (Q11) for 29% of participants, whereas 29% required ‘written proof’ and 4% were not informed about the procedure. In the open question (Q13), constructive notes and comments about the webinars included the length (too long), starting time (too late or too early), discussion (too short, too extensive), privacy, tone quality and more interactive capabilities (video).

The frequency of webinars (Q14) was assessed as appropriate by 91% of participants, too frequent by 3% and too infrequent by 6%. On average, 90% of participants were paying attention.

According to the results and analysis of these evaluations, we adjusted factors associated with teaching methods, attention maintenance and webinar content for later sessions. For attention, after each content section, we shifted to using interesting chat questions to present to the speaker. Furthermore, we invited participants 3–4 times after a content section to online discussions with the speaker. In addition, CME questions were presented, along with the percentage of responses and an explanation by the speaker of the correct answers.

The organisational effort required for each webinar involved about 12 hours for the speaker (including preparation, test webinar, webinar and postwebinar email questions) and about 60 hours for the organisational team. Finally, the average cost for one webinar was €3.50 (about $4.25 or £3.15) per participant.

Discussion

The present study clearly shows that these webinars represent a popular CME option during the ongoing COVID-19 pandemic, especially regarding speaker quality, learning effects, organisation and practical content. We found high levels of acceptance, high satisfaction rates and steadily high attention. Our evaluations represent a cross-section of ENT physicians in Germany, but the relative consistency of our results supports the notion that this learning format offers clear advantages.

Successful educational events in real time, such as webinars, require careful organisational and teaching preparation. In our series, we relied on our experience with webinars conducted before the COVID-19 pandemic, and we refined the organisational structure and teaching concepts based on empirical observations and evidence gained from the evaluations and participant feedback.

During the pandemic in 2020, our webinars were attended by a large number of participants presumably because of the lack of alternatives, uninterrupted CME requirements and the desire for professional exchange. In our webinar series evaluations, ENT physicians expressed a need for more online learning as part of medical education in the future. About one-third of our participants indicated that they would prefer mainly online learning after the pandemic. Thus, the pandemic circumstances have led to a rethinking of the role of internet-based CME/CPD. Regardless, ongoing quality control through continuous evaluation is necessary to adapt to the changing needs of participants, select appropriate course topics and update teaching principles to improve webinar quality.

Currently, a standardised instrument is needed that is specifically designed for evaluating webinars and that incorporates some unique features of online events, in particular interactions with the audience, knowledge control and technical transmission quality. Because webinar pedagogy (‘distance teaching’) differs in its concepts from in-person teaching, the maintenance of attention (organisation) by delivering an engaging webinar (eg, speaker quality) seems to be a crucial factor in the overall assessment. Based on praise from the participants about the interactive features, we infer that continual interaction between participants and with the speaker was the most important teaching element in maintaining high attention. Webinar audience interactions can be realised in discrete time periods (10–15 min) after a content section. Interactions comprise a continuous selection of interesting questions from the chat with subsequent presentation to the speaker, 3–4 times scattered online discussion throughout the webinar between the participants and the speaker and a CME question with an online presentation of results and explanation of correct solutions by the speaker, respectively. This variety of interactive elements seems to be suitable for maintaining attention. Thus, interaction with participants seems indispensable for ensuring their attention and a successful webinar.

The limitations of the study include potential bias introduced by the same people completing several surveys if they attended several webinars. The survey was anonymous, so we do not know how many participants completed surveys for more than one webinar and whose views are therefore reflected multiple times in the results. The phrasing of Q8 might have carried some bias because ‘online training’ is a broader category than ‘webinars’. Furthermore, especially in the beginning of the COVID-19 pandemic, teaching opportunities were reduced because of partial or complete lockdowns, which could have affected responses to the evaluations.

Webinar organisation was time consuming for the speakers and the organisational team but cost little for the individual participant. Holding webinars every 2 weeks could reduce the fee per webinar, but it would not reduce the organisational effort. Although we assumed that pandemic-imposed event cancellations and social distancing drove the high participation rates and attention in these webinars, this delivery method remains effective and feasible. However, successful webinars require careful organisation to maintain audience attention. Webinars have an important role as a teaching and learning tool,9 and further research is necessary to evaluate how clinically oriented webinars can be delivered with high quality and benefit for practice improvement in a strong network of peers.