Rapid Adaptation of Cancer Education in Response to the COVID-19 Pandemic: Evaluation of a Live Virtual Statistics and Research Skills Workshop for Oncology Trainees.

Due to COVID-19, an annual interactive statistics and research methodology workshop for radiation oncology trainees was adapted at short notice into a live virtual format. This study aimed to evaluate trainee opinions around the educational value of the workshop, logistical aspects and impact on interactivity. A post-course on-line survey was completed by 26/42 trainee attendees (response rate 62%). For five pre-specified learning outcomes (LO), 58 to 69% of trainees agreed that the LO was completely or largely met (Likert scores 6 and 7 on a scale 1 = not met at all; 7 = completely met). All trainees felt that logistical aspects of the workshop including organisation, accessibility to the platform and sound/image quality were good or excellent. With regard to opportunities for interaction and suitability for small-group 'break-out' sessions, the majority felt that interaction could be adequately maintained whilst just under a quarter felt the delivery method was not fit for the purpose. Networking/social engagement with peers and teachers was the factor most impaired using the live virtual delivery format. Over three-quarters of trainees replied they would favour the current event or other educational sessions being offered (at least as an option) in a virtual format in the future. Cost and convenience were given as the major non-COVID-19-related benefits of virtual on-line learning. These preliminary findings provide valuable feedback to help adapt or develop further on-line educational and training initiatives that will be necessary in the COVID-19 pandemic period and beyond.

Introduction

The far-reaching consequences of the COVID-19 pandemic are playing out currently across all spheres of life in all parts of the world [1, 2]. The enormity of the community health and economic impact of the novel coronavirus infection is clearly of paramount significance. However, there will be countless ramifications that permanently alter elements of our work and personal lives, some of these for the better. In this way, the current dire situation offers opportunities for innovation over the longer term.

For cancer patients undergoing radiation therapy and systemic treatments and for staff looking after them, there are a special set of complex considerations in maintaining good quality clinical care during the pandemic [3, 4]. Other core departmental functions such as quality assurance, research activity, and professional training and education need to continue in some form despite additional challenges. Such a crisis draws into sharp focus where efficiencies and alternative practices need to be investigated as a priority both in the realm of clinical care and across other domains.

The SMART workshop is a full-day interactive education session designed to enhance trainee knowledge and skills in statistical methods, critical appraisal of medical literature and research methodology [5, 6]. It is a mandatory component of training in radiation oncology in Australia and New Zealand under the auspices of the Royal Australian and New Zealand College of Radiologists and is run each year in association with the Trans-Tasman Radiation Oncology Group (TROG) annual scientific meeting. Five days before the 2020 edition of the SMART workshop, a decision was made to convert it to a live virtual format. This brief article describes trainee-reported outcomes on educational value and logistical and social aspects of the rapidly implemented real-time virtual delivery format.

Material and Methods

Forty-three trainees within the Australian and New Zealand radiation oncology (RO) training program had registered to attend the SMART workshop in March 2020. Workshop learning outcomes (LO) are that (after the workshop) trainees are able to: (i) describe features of different trial study designs, (ii) describe study outcome measures/endpoints, (iii) justify selection of suitable endpoints, (iv) apply new knowledge for the purposes of critically appraising the literature and (v) consider ways in which they might participate in research.

Several weeks prior to the decision to adapt the workshop format, trainees had been sent the usual ‘homework’ pack including select research methodology questions to consider in relation to three published RO trials, provided as learning prompts. Five days before the workshop, trainees were informed by e-mail of the change to a live virtual format. Forty-eight hours before the workshop, trainees and faculty were sent an e-mail with login instructions to access the video-conferencing platform.

The SMART workshop program is composed largely of short didactic lectures given by the biostatisticians to align with learning outcomes, followed by highly interactive small-group sessions led by the radiation oncologist facilitators [6]. This year, several key faculty including the convenor, two biostatisticians, one of seven facilitators and a support officer met in a single location in Sydney respecting COVID-19 social distancing requirements at the time. The other six facilitators, two biostatisticians and 42 (of 43 registered) trainees logged into the video-conferencing platform externally from across six Australia states and both islands of New Zealand. A technical support officer and the conference event manager also connected from remote locations to assist with logistics and troubleshooting.

The entire original content of the SMART Workshop was delivered via the live virtual format. However, the ‘small’-group sizes were increased compared to the usual in-person workshop due to unfamiliarity with the as yet untested technical aspects of running the virtual breakout sessions. During the small-group sessions, trainees were divided and directed into four distinct digital ‘rooms’ in which around ten residents were tutored by one or two radiation oncologists and a biostatistician. Interactivity was enabled through all participants knowing, usually seeing, who was on-line and each having access to a microphone. Trainees could also use the text chat window to ask questions or comment. Just like the usual workshop model, after each breakout group session, the whole cohort came back to one virtual conference room to present their group’s responses and for a debrief.

Two international guest speaker talks were retained in the program. As usual, they were invited to provide inspiration to trainees to engage in research. Speakers, one presenting virtually in real-time and one recorded, were located in New Zealand and the UK, respectively due to travel restrictions. Only the lunchtime and post-workshop networking events were cancelled.

Three days following the workshop, all 42 participating trainees were invited to provide feedback via a SurveyMonkey® questionnaire (see Supplementary material). Due to the short time-line, formal institutional ethics approval was not sought. Trainees were assured that responses were anonymous and neither individual nor collated responses would be linked to training progression. Demographic questions related to level of training and prior SMART workshop attendance. Trainees were asked their views on the educational value of the workshop components (didactic and interactive) and to rate their experience against the pre-prescribed LOs. Opinions around the logistical, technical and networking/social aspects of the virtual delivery format were sought as well thoughts about digital delivery methods for future educational activities.

Responses were collected using five- and seven-item Likert scales with freehand text answers encouraged, focusing on areas for improvement. Quantitative data were analysed using percentages and means.

Results

Demographics

Twenty-seven of 42 RO trainees completed the first section of the survey after two reminder e-mails, giving a response rate of 64.3%. Forty-four per cent (12/27) were in their first 2 years of training, 37% (10/27) in their third year and the remainder in the last 2 years of a minimum 5-year training program. One trainee skipped all questions beyond demographics leaving full input from 26/42 trainees (response rate 62%).

Educational Value

Figure 1 shows trainees’ views on the degree to which the virtual live workshop met the five pre-articulated LOs (shown in ‘Materials and Methods’) on a Likert scale of 1 (= not met at all) through to 7 (= completely met). For the five LOs, between 58 and 69% of trainees agreed that the LO was completely or largely met (Likert scores 6 and 7). None of 26 trainees gave scores of 1 or 2 for any LO.

Fig. 1

Degree to which learning outcomes were met

With respect to the perceived suitability of the level of teaching, taking all workshop components into consideration, 38% (10/26) of trainees felt that the level was not optimal for them. Judging from the open-ended comments that followed, trainees were divided as to whether the level was too sophisticated or too basic. Correlation of these findings with year of training and/or previous attendance was not possible due to the anonymity of responses.

Views on Virtual Delivery Method and Suitability for Purpose

All trainees responding to the survey felt that logistical aspects of the workshop including organisation, accessibility of the video-conferencing platform and sound and image quality were good or excellent. With regard to the ‘opportunity for participant interaction’ and ‘suitability for small-group break-out sessions’, 23% (6/26) and 19% (5/26) of trainees, reported the delivery method as not being fit for purpose. Figure 2 shows trainees’ ratings around logistical aspects of the virtual live workshop and views on suitability of the format for interactive group work in particular.

Fig. 2

Ratings for digital delivery method according to logistical factors and suitability for purpose

Over three-quarters of responding trainees indicated they favoured the SMART workshop and/or other (un-named) educational sessions being available via an online platform as part of their future RO training.

Impact on Social Engagement and Networking

The survey included the question: ‘In your view, how much did being able to see (i.e. on video) other participants and teachers enhance the ability to interact?’ to which 77% (20/26) trainees answered in the affirmative; 62% felt this factor was ‘very-’ or ‘extremely important’. Figure 3 shows responses relating to the impact of the virtual delivery format on their ability to network with others at the workshop (teachers, peers and invited speakers) as well as other RO researchers attending the subsequent TROG scientific meeting, also delivered online for the first time [7].

Fig. 3

Impact on ability to network with others

Between 12 and 23% of trainees felt there was ‘no impact’ or ‘a little impact’ on their ability to network with other groups linked to the workshop (listed above). The vast majority (up to 88% for interaction with peers) indicated that the virtual format impacted ‘quite a lot’ or ‘very much’ in this domain.

Comments and Suggestions for Improvement

A rigorous qualitative thematic analysis was beyond the scope of this study. However, open-ended comments solicited around workshop LOs and suggested improvements, revealed some common opinions. One strong message was trainees’ recognition of the short turn-around time and appreciation of not missing out on core learning. Only one trainee felt the SMART workshop should have been cancelled. Most favoured aspects were the small-group interactive sessions, having access to biostatisticians’ expertise and the value of basing learning on publications of real RO research. Main areas suggested for ‘improvement’ were having access to a virtual workshop as a standard training option and ensuring input from all participants in small-group sessions. Reduced cost and travel time were the most common reasons trainees favoured having the option of ongoing virtual meetings, especially those training in non-metropolitan centres. Equally, many trainees stressed the value of in-person interactions. A typical example quote was:

‘There are benefits of meeting virtually, and this was so important in these COVID times, however ultimately nothing can replace the engagement and community that come with face to face meetings.’

Discussion

An annual statistics and research methods workshop for Australian and New Zealand RO trainees was rapidly converted into a live virtual experience. The response rate to an evaluation survey sent to participants was remarkably high (62%) considering the pressure trainees were under soon after the outbreak of the coronavirus epidemic in these countries. With escalating changes within workplaces, including the threat of hospital redeployment, it would be totally understandable if trainees had chosen to ignore an optional survey.

On-line training methods for education within the medical setting have received increasing attention in recent years [8, 9], including within the field of RO [10-13]. Similarly, virtual reality tools have huge scope for RO professional [14, 15] and patient [16] education. There is far less evidence, if any, around the feasibility and value of adapting face-to-face interactive education into a mirrored live virtual format. This report provides some considerations for the delivery of existing curriculum elements via real-time virtual methods going forward, both during the pandemic and beyond.

Despite the rapid turnaround time, the workshop logistics were viewed favourably by the vast majority. It is noteworthy that no trainee dropped out after log-in. Similarly, despite the lack of familiarity with interactive meeting software platforms in early March, navigation through the on-line workshop ‘rooms’ for the various groups was pleasingly smooth, albeit assisted by professional technical support.

Limitations of the current study are that views of the responding trainees may not necessarily reflect those of the whole group. The overall number of respondents was quite small despite a reasonable response rate for a voluntary survey. The survey format may have favoured trainees more au fait with electronic learning methods leading to bias. Generalisability across other geographical regions cannot be assumed. Our countries represent high-income training environments in which trainees are typically well supported to attend educational sessions and are well resourced with regard to modern computer technologies. Trainees were not asked specifically about challenges of participation from home such as having privacy or competing childcare/home schooling responsibilities. The findings from this study will clearly not be applicable to all types of learning necessary for oncology training, particularly where the clinical and workplace contexts are central to learning. However, for ‘non-medical expert’ or ‘intrinsic’ skills [17], such as professionalism, quality improvement, leadership/management and scholar/research skills, as well as for more didactic clinical teaching, live virtual and other digital formats may offer a useful substitute for face-to-face and on-the-job teaching.

Of special interest was the impact of the virtual format on learner/teacher and learner/learner interactivity, a key feature of the workshop design founded on theoretical constructs for effective learning [18]. Most trainees felt that interactivity was still achieved though hindered to some extent. Previous research supports that student interactivity can be successfully achieved for real-time virtual teaching methods [19]. If future virtual versions of this workshop or other activities are convened, focus on maximising interactivity would be important. Strategies might be to request that cameras were always turned on during small-group sessions and having access to participant photos and names. Bandwidth issues requiring that video be turned off to improve the speed of connection might be an impediment in some situations. User proficiency across numerous on-line platforms has quickly developed over the months since the workshop both for learners and teachers. It is probable that if the workshop had been scheduled, a few weeks or months later, interactivity would have been more effortless for all involved.

Not surprisingly, networking and social engagement aspects of the workshop were hardest hit by the change of format. Interaction with peers and mentorship by senior researchers have been reported as strong factors in trainee engagement in research within our countries [4]. One trainee commented that s/he and peers had used mobile phone group messaging to network ‘behind the scenes’ as a substitute for in-person chatting, especially during lunchtime. It is hard to imagine, however, that an adequate replacement for constructive mingling and building of research-oriented networks in the one location could be achieved in an on-line space.

Findings of this study do suggest that live virtual formats for education might offer a useful supplement or alternative mode of delivery for some curriculum content and for some learners. The expansion of ‘blended learning’ i.e. mixed digital and in-person methods, in health care education, including for RO offer promise [8, 20, 21]. On-line learning, whether real-time or not, can have major cost and convenience savings as well as being more climate-friendly. For trainees living in regional areas, the benefits of increasing live virtual education offerings, still allowing interaction with peers, are self-evident. For parents (likely more women) balancing child-care responsibilities, these formats could be particularly attractive.

Conclusion

In oncology, as for other specialities, we will continue adapting much of our traditional teaching to online formats. This process, accelerated by the pandemic, was already in train to improve efficiency and accessibility for learners. The findings of the current study support the feasibility of live virtual formats for education delivered to moderately large groups of trainees. Feedback from learners participating in a cancer-specific statistics and research methods workshop gives a taste of the acceptability and educational value of the virtual approach in delivering some components of training. Furthermore, the results of this evaluation are instructive in directing faculty, supervisors and training bodies to areas where strategies for improvement, such as optimising interactivity and ‘networking’ between participants, might be focused going forward.

(PDF 59 kb).