National, Virtual Otolaryngology Training Day in the United Kingdom During the COVID-19 Pandemic: Results of a Pilot Survey.

The COVID-19 pandemic has forced an unprecedented worldwide change in our daily lives, particularly in healthcare. In response to the lockdown measure it has brought on, some deaneries cancelled planned teaching days, while others had delivered remote smaller scale teaching sessions electronically. Due to significant regional variation, the National Association of Program Directors in ENT (NAPDENT) set up a national educational program, matched to the Intercollegiate Surgical Curriculum Program. This is delivered virtually and for free. This pilot survey evaluates participants' learning experience of the first NAPDENT National Otology Training Day, which was held on 20 April 2020 via Zoom. This is a pilot survey evaluating the first online, national educational program delivered by the NAPDENT in the United Kingdom. In the times of the COVID-19 pandemic, our program has rapidly learnt from this initial experience and continues to build on lessons learnt for future editions. It is feasible to augment the national ENT curriculum with a collaborative, online teaching program. Regional buy-in is required to make this work. This modality allows for access to subject experts across the country. Synchronous delivery allows for participant interaction, while an online repository allows for asynchronous viewing when work or personal commitments do not allow live attendance. The key to participant engagement is adequate structure, appropriate breaks and opportunities for interaction. Even once the COVID-19 pandemic has passed, this teaching modality will remain as an opportunity to educate ENT surgeons of the future.

INTRODUCTION

The COVID-19 pandemic has forced an unprecedented worldwide change in our daily lives, particularly in healthcare. The UK introduced lockdown measures on March 23, 2020. A joint policy statement from the Royal Surgical Colleges issued on March 13, 2020 recommended nonessential activities to be postponed, including educational courses to either be postponed or delivered via video call or webinar.

Health education bodies across England, Wales, and Scotland oversee training and educational delivery to postgraduate healthcare professionals in the United Kingdom. Local delivery is coordinated by regions or “deaneries,” with each medical and surgical specialty led by a Training Program Director (TPD).

Within ENT higher surgical training an educational program is most frequently delivered as a monthly face-to-face “teaching day” including mainly didactic materials, with occasional workshop and simulation held at a central location within the local deanery. Trainees within the deanery use study leave on these specific days to travel to the specified location. In the pre-COVID 19 era, these regional teaching days were always face-to-face. There is approximately 450 ENT Specialty Trainees Year 3 to 8 (ST3-ST8) in the United Kingdom, who have to maintain 70% teaching day attendance within an academic year. National-level teaching was delivered at annual subspecialty conferences and biennial British Academic Conference of Otolaryngology held at various locations across the country. Further opportunities were also provided by the annual Live International Otolaryngology Network global broadcast of surgical demonstrations with a focus in Otology and Neurotology.

In response to the COVID-19 pandemic, deaneries had varied approaches to education delivery. Some cancelled planned teaching days, while others had delivered remote smaller scale teaching sessions electronically. Due to significant regional variation, the National Association of Program Directors in ENT (NAPDENT) set up a national educational program, matched to the Intercollegiate Surgical Curriculum Program. This is delivered virtually and for free. This pilot survey evaluates participants’ learning experience of the first NAPDENT National Otology Training Day, which was held on April 20, 2020 via Zoom (Zoom Video Communications Inc., 2020)

MATERIALS AND METHODS

Study Design

The NAPDENT National Otology Training Day was advertised 5 days prior to all ENT TPDs in the United Kingdom and on the Association of Otolaryngologists in Training's (AOT) website and mailing list of about 1500 recipients, which included trainees and consultants as well as staff grades, associate specialists and specialty doctors (SAS doctors). Participants were required to register in advance to receive Zoom meeting login details. The questionnaire was sent to all registered participants after the session. Participants had 2 weeks to complete the survey.

Questionnaire Design

The pilot questionnaire was divided into 4 sections—demographics, presession, session content, and postsession. There were 60 questions in total, which consisted of single-answer multiple choice, 4-point interval scale responses and free-text. The “session content” questions were multiplied by 5 to reflect each of the 5 sessions delivered. All nonfree text questions were compulsory. A Google Form online link was sent to all the registered email addresses of participants. The information was collected anonymously with no incentives offered to participants.

The 4-point interval scale was used. This was preferred to the traditional 5-point Likert scale because the primary objective was to capture direction (positive or negative) of responses, rather than their intensity level of agreement of disagreement. The 4-point scale also eliminates the neutral response option, which reduces ambivalence and forces participants to present their true feelings regarding the subject.

The questions were initially designed based on the Kirkpatrick model of learning evaluation, which includes 4 outcome levels: satisfaction, learning value, behavioral change, and organizational change or patient outcome. These were reflected mainly in the latter half of the questionnaire within sections “session content” and “postsession.”

The former half of the survey focused on attendees’ demographics, devices used and level of preparation involved prior to viewing the session. All questions were peer reviewed by 4 ENT consultants.

RESULTS

Demographics

Two hundred and thirty people registered for the session. There was fluctuation in attendance during the 5-hour session with a minimum of 170 and a maximum 250 people logged on at any one time. One hundred and twenty participants (52%) responded to the questionnaire. Individuals from England, Scotland, Northern Ireland, and the Republic of Ireland attended. There were no participants from Wales. Forty percent of participants were from North West and Mersey deanery (see Fig. 1 ). Most participants (62%; n = 75) were specialist trainees in ENT, but there was a varied audience including foundation trainees, core trainees, clinical fellows, specialty, and associate specialist doctors and consultants (see Table 1 ). Communication for TPDs was the most influential source of advertisement (54%, n = 65).

FIGURE 1

Map distribution of participants with their corresponding deaneries presession

TABLE 1

Participants’ Demographics

Participants Details
		Responses % (n)
Grade	Foundation year 1/2	0.8 (1)
	Core/specialty trainee year 1/2	10.8 (13)
	Specialty trainee year 3	10 (12)
	Specialty trainee year 4	13.3 (16)
	Specialty trainee year 5	12.5 (15)
	Specialty trainee year 6	8.3 (10)
	Specialty trainee year 7	12.5 (15)
	Specialty trainee year 8	5.8 (7)
	Consultant	5 (6)
	Masters of surgery/education/clinical fellow	13.3 (16)
	Staff grade/associate specialist/specialty doctor	5.8 (7)
	Others	1.7 (2)
Where did you hear about the session?	Association of Otolaryngologists in Training (AOT)	16.7 (20)
	Training program director	54.2 (65)
	Word of mouth	29.2 (35)

Ninety-seven percent (n = 116) of respondents found the registration process easy. Sixty-eight percent (n = 82) watched the online session from home, while the rest watched from work (Fig. 2 ). There were different types of devices used: 64% (n = 77) laptops, 20% (n = 24) mobile phones, and 16% (n = 19) desktop computers.

FIGURE 2

Where did you watch the session from?

Seventy-nine percent (n = 95) did not research the speakers or topics prior beforehand. Eighty-nine percent (n = 107) planned to stay for the whole session, with 80% of those managed to watch until the end. Thirty-six respondents provided free text comments on reasons behind not watching the session in entirety—58% (n = 21) stated work commitments, 36% (n = 13) already had personal commitments scheduled due to short notice of the session and technical issues were stated by the remaining 6% (n = 2).

Session Content

There were 5 sessions in total. The majority of participants perceived the duration of these to be “perfect” (71%). However, more than 30% participants perceived session 2 and 5 to be “just over” or “too long” (see Fig. 3 ). Interestingly, it was session 1 that had the longest actual duration. Further 4-point interval scale questions relating to delivery, content, and relevance of the topics presented in each session were also asked in the questionnaire (see Fig. 4 ). Sessions 2 and 5 received an overall more negative scale response compared to the other sessions, which suggested that actual time duration is a secondary factor. Figure 4 outlined the responses to each session's content and overall rating.

FIGURE 3

Duration of each session and the perceived length reported by participants

FIGURE 4

Session content evaluation and overall ratings

Questions from participants during the session were asked using the chat function with a nominated moderator who presented them to the speakers. An average of 18 questions were asked during each session. These were either answered verbally by the presenting speaker or generated further discussion within the forum involving both consultants and trainees. Figure 9 demonstrated that 83% (n = 99) found it a more comfortable means to ask questions compared to a face-to-face lecture setting. A majority of respondents also felt that it was easy to use (93%, n = 112), their questions were answered adequately (63%, n = 75) and it was moderated well (91%, n = 109).

FIGURE 9

The chat function evaluation

Postsession

A vast majority of participants found it easier to attend and concentrate in this format compared to face-to-face sessions (see Figs. 5 and 6 ). Ninety-one percent of respondents suggested the ideal frequency of scheduled breaks in a typical 4-hour session should be every 45 to 90 mins (see Fig. 7 ). Overall, participants regarded the virtual training day to have an overwhelmingly positive learning value (see Fig. 8 ). Figure 9 demonstrated positive evaluation of the chat function as a tool for interaction with the speakers and other participants.

FIGURE 5

Did you find it easier to attend this virtual session than face-to-face session?

FIGURE 6

Concentration level

FIGURE 7

Suggested frequency of scheduled breaks in a 4-hour session

FIGURE 8

Learning value of the session (percentage and (n) corresponds to combined total of “somewhat agree” and “strongly agree” responses)

DISCUSSION

The COVID-19 pandemic has brought about profound medical education adaptations that have shifted the classroom into the virtual world. This newly-created, national venture was the result of a national collaboration of ENT training program directors. It aimed to deliver aspects of the national Intercollegiate Surgical Curriculum Program curriculum utilizing online learning.

The specialty-specific (knowledge) content of the program was rated highly. The national reach of the training day allowed for collaboration and access to experts from around the country. During the sessions, participants were able to ask questions via a chat function. A nominated moderator fielded and relayed questions to the speaker. These interactive modalities were viewed as easy to use, adequate and non-threatening ways to engage with the live content, as shown in Figure 9. The session was developed as a mixture of instructional and case-based learning. Synchronous content delivery with experts allowed for collaborative learning, where learners received information and were able to communicate with each other simultaneously.

Lessons learnt regarding the use of the new online teaching modality have been profound. In keeping with previously described components of e-learning content - development, delivery, management and standardization—each of these posed challenges and opportunities. Development of the specialty-specific content had to be adapted to the new virtual environment, and flexible enough to cater to live interaction. The session was designed for junior registrars (ST3-5), but remote attendance enabled other training and non-training grade doctors to attend. Investment into an institutional subscription to Zoom Pro eliminated the meeting duration limit and increased the potential attendees to up to 500 people. The teaching session was recorded and made available on YouTube with links available on the AOT website for free, so learners could attend asynchronously if required, or revisit material in the future. The videos were edited accordingly prior to being uploaded by removing any patient data to maintain confidentiality. They were also bookmarked based on the topics presented to allow more targeted viewing. This investment into information technology infrastructure is vital to store the material in an accessible online repository, supporting short- and long-term educational goals. Standardization of a new learning material is vital to promote its compatibility and usability across different computer systems. Therefore, the use of free existing established software, such as Zoom, was preferred as evidenced by the participants’ positive responses to ease of online registration, variability of devices used and ease of access from both home and the workplace.

Another aspect that has become apparent in this endeavor is the crucial role of advertising for sessions of online learning. Advertisement from TPDs was shown to be the commonest source of communication about the session, indicating that regional leadership is vital for the success of this national scheme. This may be useful as well to address the perceived absence or low number of trainees from certain deaneries (see Figure 1). Interestingly, trainees from Republic of Ireland (4.2%, n = 5) joined the session because of the advertisement from AOT and “word of mouth” as their TPD was not included in the initial communication to all UK ENT TPDs. Other methods of communication, such as advertisement on social media, will also need to be considered for future sessions.

The latest systematic review on e-learning in medical education by Regmi et al identified several factors, categorized into enablers and barriers, influencing the enhancement of lifelong learning. They found that e-learning was more effective than traditional learning when it could deliver stimulating integrated content, facilitate learning acquisition and integration into practice, provide flexibility to address short and long-term educational needs, as well as encourage learning autonomy by improving engagement and retention. Factors in e-learning that may pose a barrier include poor motivation and expectation, resource intensive, unsuitability for some disciplines and a relatively high IT skills requirement for both facilitators and participants. Interactive online education provides a novel intersect of behaviorist, cognitivist, constructivist and connectivist approaches to learning . An underlying appreciation of different learning theories can help educators tailor online teaching to appropriate audiences. The most significant element of this multimodal learning approach is its flexibility to evolve in pace with learning needs and technological advances.

A majority of respondents did not research the topics or speakers prior to the session. There may be several factors affecting this, including the short advertisement notice of only 5 days, perceived familiarity of topics, unfamiliarity of the session format (e.g., didactic or interactive) or the training level at which topics were presented and personal or work commitments, amongst other factors. In relation to this, some respondents commented on the “free text” section about the need for presession materials, in the form of MCQs, summary of topics and panel discussion cases. An active trend in current medical education is the concept of a flipped classroom, which is a type of blended learning where the instructional lecture-type information is viewed at home and the teacher has a more facilitator role in case-based discussions in the classroom. Taking all of these factors on board, future sessions will aim to provide relevant presession materials that will support a wider variety of learning styles. This will also set participants’ expectations in order to boost their learning motivation, which is one of the key determinants of e-learning success identified by Regmi et al.

Remote attendance of online sessions provided greater flexibility on where participants can attend from, which was reflected by the overwhelming positive response on this as demonstrated in Figure 5. However, Figure 6 showed that 44% of respondents selected “somewhat agree”, rather than “strongly agree” and 23% expressed disagreement that this format allowed for better concentration level. This may be explained by Figure 2, which showed 68% viewed it from home, where access to a quiet space or lack of unforeseen distractions might not be as readily available as in a purpose-built lecture theatre, for example.

Data on perceived duration of individual presentations, shown in Figure 3, and responses on ideal frequency of scheduled breaks, shown in Figure 7, will inform us on the design of future sessions in order to maximize concentration and learning value obtained. The data suggested that in a typical 4- to 5-hour online teaching session, each session should last for 40 to 60 minutes, with a scheduled break after 1 or 2 sessions.

Figure 4 demonstrated that a higher negative scale rating in slide design, delivery style and content relevance impacted poorly on the overall rating of a session as shown in Session 2. Addressing these aspects during session preparation may help speakers in the future to improve audience engagement and retention of materials delivered.

The learning value of the virtual training day was evaluated using questions based on the Kirkpatrick model of learning evaluation. Feedback from participants indicates that behavioral change may have been affected by the session, resulting in Kirkpatrick Level 3 training (Fig. 8).

The program was rated highly overall not only due to its high quality specialty-specific content, but also because it was delivered through the new technology-enhanced learning modality, which allows for greater collaboration, provides repository of shared knowledge and increases accessibility while maintaining participants’ engagement. Table 2 summarized our key recommendations.

TABLE 2

Key Recommendations for Organizing Virtual Teaching Session

Key Recommendations
1. It is possible to deliver high-quality, specialist education on a national level using existing online technology
2. Live, interactive delivery allows for a collaborative learning experience
3. Institutional investment and faculty engagement with online platforms are key to program success
4. Clear learning objectives advertised enhances engagement
5. Targeted advertising for online learning is critical to reach the appropriate audience
6. Remote attendance allows flexibility for attendees
7. Attendees value sessions lasting 40-60 minutes

Study Limitations

The survey questions were peer reviewed independently by 4 ENT consultants to ensure its effectiveness in measuring the intended outcomes of the study. However, given the wider context of operating within an emerging pandemic, we were not able to further pilot the questions to a smaller focused group of participants. After reviewing the results, we acknowledged that there was a high number of questions and some questions measured the same outcome. The former aspect might have introduced an element of fatigue and indifference towards the later stages of the questionnaire. Therefore, we would recommend future iterations to be more streamlined.

We were also unable to quantify how many people registered but did not watch the session as it was proven very difficult to keep a record on the number of participants throughout the session because people were logging in and out. Moreover, there may be more than one registered participant using one device and those that did not watch the session or only parts of it would be less likely to fill in the questionnaire.

Given the relatively short 5 days advertisement period, the range of 170 to 250 attendees logged on at a given time could be considered as a success. Extrapolating from the 62% figure of the questionnaire responders who were specialty trainees, our session likely have reached more than a third of all specialty trainees across the United Kingdom. We acknowledged that this number could still be improved by continuing to outline clear learning objectives for each session and allowing longer advertisement period to allow trainees to make the necessary arrangements to attend virtually.

Another aspect of online delivery that was not measured in this survey was faculty experience, which will be included in our future sessions. It is equally important for them to have an overall positive outlook.

Last, as we planned for future training days covering other ENT topics, we would also like to recommend a formal method to track learner's attendance in their portfolio, such as in the form of awarding Continuing Professional Development points.

CONCLUSIONS

This pilot survey evaluates the first online, national educational program delivered by the NAPDENT in the United Kingdom. In the times of the COVID-19 pandemic, our program has rapidly learnt from this initial experience and continues to build on lessons learnt for future editions. It is feasible to augment the national ENT curriculum with a collaborative, online teaching program. Regional buy-in is required to make this work. This modality allows for access to subject experts across the country. Synchronous delivery allows for participant interaction, while an online repository allows for asynchronous viewing when work or personal commitments do not allow live attendance. The key to participant engagement is adequate structure, appropriate breaks and opportunities for interaction. Even once the COVID-19 pandemic has passed, this teaching modality will remain as an opportunity to educate ENT surgeons of the future.