Evaluation of Blended Online Learning in Three Spinal Surgery Educational Courses.

COVID-19 pandemic created a need to improvise and redefine blended learning to be executed fully online. Background information on the effectiveness of fully online blended learning activities, especially for surgical disciplines is limited. This study describes a fully online blended learning course format on spinal surgery and aims to provide data regarding it effectiveness. Fully online blended courses on three topics of spinal surgery designed as six-week asynchronous and followed by 3-day live parts. Learning gaps (LGs) were identified with a survey at the beginning of asynchronous part, at its end, and at the end of the live part. The effectiveness of the asynchronous and live parts was assessed by LGs and a quiz, login statistics of learners and faculty and a post-course survey. Participants' LGs decreased in all courses, statistically significant in two. Faculty and learner login rates significantly correlated with each other. Faculty and learner satisfaction was very high. A fully online blended learning course can be delivered effectively on spine surgery with a high participant and faculty satisfaction rate. The asynchronous part contributes to learning significantly.

Introduction

Online learning is the form of learning that uses the internet as a delivery platform [1]. Blended learning traditionally refers to a combination of online and face-to-face (F2F) learning, merging the advantages of online and F2F activities by providing prolonged exposure to learning for participants, flexibility with their time, a comprehensive range of educational resources, and enabling peer learning [1-3]. Despite these advantages, E-learning has been slow in uptake, especially in medical education as many concerns and problems have been reported [4]. A systematic review on factors leading to failure identified dependency on internet availability and speed, perception of impersonal learning, potentially high initial cost, and the requirement of programming expertise as potential problems [5]. Whereas, E-learning covers a wider range than blended learning [6]. Recent reports demonstrated the effectiveness of blended learning as an educational instrument for medical education, including postgraduate and subspeciality training in medicine [3,7,8].

The Covid-19 pandemic necessitated an abrupt switch to online learning globally due to significant mobility limitations and has been instrumental in expediting a change in our educational paradigm by creating the state of need and urgency to trigger change [9,10]. It also created the need to improvise and redefine blended learning to be executed fully online. A recent study suggests repercussions extending beyond the pandemic period and that blended learning may be here to stay based on its proven effectiveness [11].

This paper describes and critically analyses three courses in spinal surgery that had to be planned and executed as a combination of online asynchronous learning followed by online synchronous (live) learning. This online live part of learning could not include hands-on skills training that was initially intended for the F2F live part. Instead, it had live and interactive case discussions and surgical technique videos.

Based on this unique experience, the objectives of the current study were to:

Describe a fully online blended learning format as a method in postgraduate spinal surgery education

Establish proof for a fully online blended learning as a feasible and effective format

Identify the contributions of the two parts of blended learning (asynchronous and synchronous) to improvements in learner-assessed learning gaps

Analyse the factors affecting these improvements.

Material and Methods

Course Design

The original design for the blended courses was asynchronous online learning over six weeks followed by a 2-day live F2F event. Due to the travel and meeting restrictions caused by the Covid-19 pandemic, this design had to be converted to a blended format with both the asynchronous and the synchronous (live) parts held online. All theoretical knowledge was supplied to the learners during the asynchronous part, and the live event was dedicated to higher learning activities, such as case discussions and demonstrations of surgical techniques and discussions based on them. Of the six courses that had been planned, three courses had to be cancelled because of the inability to perform an accurate conversion to the fully online format (Cervical Degenerative) or because of a lack of registrations (Thoracolumbar Trauma and Surgeon Leadership). Consequently, three courses on Adult Spinal Deformity (AD), Endoscopy (Endo), and Minimally Invasive Spinal Surgery in Degenerative and Traumatic Disorders (MISS) were planned and executed.

All courses were designed using backward planning by the course chairpersons (CPs) to identify the patient problems to be addressed, performance gaps, and expected learning objectives (LOs). CPs then prepared the scientific programme (Table 1) for both the asynchronous and live online parts based on these LOs and assigned faculty (six for each course, two of whom were responsible for the asynchronous part). The LOs were used as a basis for pre-course self-assessment questions that were used to calculate the learning gaps (LG) for each LO, defined as the difference between the desired level and present level of competency as assessed by the participant on a Likert scale of 1 to 5 [12]. A set of 9 multiple-choice assessment questions (MCQs) was integrated into one course (Endoscopy).

Table 1.

Programme structure for fully online blended learning

Unit	Tools, content, and format	LOs covered
Asynchronous learning
Self-assessment and MCQs	LMS, assessment, and questions	All (one LG and one question per LO)
Week 1	LMS, content, discussion forum, messaging, live case discussion*	Subset of LOs
Week 2, etc	LMS, content, discussion forum, messaging, live case discussion*	Subset of LOs
Mid-point MCQs and assessment	LMS, assessment, and questions	All (one LG and one question per LO)
Synchronous (live) learning
Session 1	Videoconferencing app, case discussions, small group discussions, demos, etc	Subset of LOs
Session 2, etc	Videoconferencing app, case discussions, small group discussions, demos, etc	Subset of LOs
Post-event questions and assessment	LMS, assessment, and questions	All (one LG and one question per LO)

MCQs – multiple-choice questions; LG – learning gap; LO – learning objectives; LMS – learning management system. *Weekly live case discussions were included in the asynchronous part only in the AD course.

Faculty Development

At the time planning started, our organisation had a large pool of trained faculty for F2F events who needed training updates to optimise their performance in an online learning environment. A dedicated faculty development programme was designed and delivered by the AO Education Institute and the course directors to address this need. This programme consisted of interactive webinars with the CPs and faculty to outline expectations in terms of time and effort and addressed the core educational concepts and principles.

Training for asynchronous learning began two weeks before the asynchronous part started and consisted of an interactive live session during which the educational context and the technical aspects were discussed, followed by two weeks of asynchronous learning. Training for the live online event took place the week before the live online event started and focused on the methods for managing online discussions and a refresher on the technical aspects.

In addition to faculty, the staff supporting the courses participated in these training sessions to ensure a pool of trained and knowledgeable faculty and the staff was prepared.

Online Learning Tools

The learning management system (LMS) used for the asynchronous part was Totara, a Moodle-based platform. It housed the educational content in all formats (articles, chapters, blogs, videos, etc) and provided interactivity in the form of Discussion Forums, Quizzes, Surveys, and Messaging, as well as login and posting data.

NewRow was used for the live part as the online “virtual classroom” (video conferencing and presentation application) and was linked to the LMS to provide access to the live sessions directly from the course home page. NewRow provided standard functionality such as video conferencing, presentation display, sharing screens, and breakout rooms for small group discussions. It also provided a content playlist that avoided time and continuity challenges that are experienced when individual presenters share their screen each time.

Data and Outcome Measures

Data on participation rates and self-perceived (subjective) ability, and question scores (objective) were collected through the LMS. Participants were also invited to evaluate the content relevance and faculty performance and to answer open text questions like; “What went well?” and “What could be done differently next time?”.

Analysed outcome measures are:

LGs at three different time points (entry, mid-, and end of course),

Question scores at three different time points for Endo only (entry, mid-, and end of course)

Login statistics for both the learners and faculty

Results of post-course evaluations (faculty and participants)

Analysis

Learning gaps and question scores were analysed for any changes by Wilcoxon’s Signed Rank Test for repeated measures. Login statistics for faculty and participants were analysed for potential correlations (collinearity) using Pearson’s r test. All analyses were performed using open-source statistical analysis software (JASP, version 0.14.1, www.jasp-stats.org, Amsterdam, The Netherlands).

The open text responses and comments were categorised as negative, neutral, and positive and counted.

Results

Three courses, on Endoscopy (Endo), Adult Deformity (AD), and Minimally Invasive Spinal Surgery in Degenerative and Traumatic conditions of the Spine (MISS), were delivered with 26, 14, and 19 registered participants respectively (56.5% course fill rate on average). All registered learners actively participated in the Endo and MISS courses and 12 of the 14 in the AD course. The learning gaps at the three-time points of pre-, mid-, and post-course were calculated and are shown in Table 2.

Table 2.

Average learning gaps at three time points and statistical comparisons

Course (n participants, LOs)	Pre-course LG (mean±SD)	Mid-course LG (mean±SD)	Post-course LG (mean±SD)	P value*
Endoscopy [26.9]	2.39 ± 0.23	1.47 ± 0.21	0.8 ± 0.14	Pre vs mid: 0.004 Mid vs post: 0.004 Pre vs post: 0.004
MISS [19.6]	1.95 ± 0.17	1.21 ± 0.1	0.85 ± 0.12	Pre vs. mid: 0.031 Mid vs post: 0.031 Pre vs post: 0.031
AD [12.4]	1.18 ± 0.2	0.79 ± 0.19	0.38 ± 0.16	Pre vs. mid: 0.250 Mid vs post: 0.125 Pre vs post: 0.125

LG: Learning Gap, *Wilcoxon’s signed rank test

Participants’ gaps pertaining to all LOs decreased in all three of the courses. These decreases were significantly different between all-time points for the Endo course, and between the pre- and mid- and pre-and post- time points for the MISS course, and not significant for the AD course. In the Endo course the MCQ scores (number of correctly answered questions out of nine) increased through all time points (Table 3), but only those differences between the pre- and mid-, and pre- and post- time points reached statistical significance.

Table 3.

MCQ scores (number of correctly answers out of 9 questions) at three time points for the endoscopy course

Quiz mark	Pre-course # (n = 27)	Mid-course # (n = 26)	Post-course # (n = 17)	P value*
# correct answers for 9 questions	5.81 ± 1.39	7.15 ± 1.52	7.41 ± 0.94	Pre vs mid: 0.002 Mid vs post: 0.484 Pre vs post: 0.002

*Wilcoxon’s signed rank test

Table 4 summarises the overall gain in self-assessed ability (decrease in LG) and increase in the question scores for the asynchronous and the synchronous learning periods.

Table 4.

Improvements in learning gap point (on a Likert scale 1 to 5) and MCQ scores (0 to 9) in all three courses by period (* = p < 0.05 for the quiz results only, the pooled LGs were not analysed for any statistical differences)

Course		Asynchronous(Pre – Mid)	Synchronous(Mid – End)
Endoscopy	LG decrease	0.93	0.68
	MCQ score increase	1.34*	0.26
Adult deformity	LG decrease	0.39	0.41
MISS	LG decrease	0.76	0.38

The login and posting activities of faculty (including the CPs and EAs) and participants aggregated for all three courses are shown as a line graph (Figure 1).

Figure 1.

Login and posting activity of all three courses aggregated during both the asynchronous and synchronous parts of the course. The x-axis shows time points in weeks beginning at the date of the first asynchronous activity. The week 12/6/2020 signifies the synchronous learning activities; all weeks before that represent the asynchronous learning part and all time points after signify post-course discussions.

This figure suggests collinearity between participant activity and faculty activity which was further studied by correlation analysis that yielded r values of 0.852, 0.844, and 0.962 for Endo, AD, and MISS courses, respectively (Table 5).

Table 5.

Correlations between faculty and learner online activity for all three courses

Course/activity	Faculty activity (total # of login and posts)	Learner activity (total # of login and posts)	Pearson’s r	P-value*
Endoscopy	3,383	15,386	0.852	<0.001
Adult deformity	861	2,059	0.844	<0.001
MISS	1,046	4,738	0.962	<0.001

* Pearson’s correlation test

And finally, the results of the post-course feedback questions (different for faculty and learners) are summarised in Table 6.

Table 6.

Summary of faculty and participant post-course feedback. Faculty evaluation includes their perception of the workload as appropriate or not and the overall rating of the courses. Learner evaluation comprises the analysis of three open ended questions. The responses were classified as positive, neutral, or negative. An example of positive sentiments may be: “the team was excellent”, or “had a great time, learned and experienced so much”, whereas an example of negative sentiments may be: “sometimes we lost time in connections between the faculties”, or, “in the 1

Faculty evaluation	Excellent	Good	Neutral	Poor
Workload during courses		15		1
Overall rating of the courses	9	5	1	1
Learner evaluation	Excellent	Good	Neutral	Poor
Overall rating of the courses	16	13	4	0
Learner evaluation (sentiments)	Positive	Neutral	Negative
What went well today?	54	63	0
What would you like us to do differently?	19	83	15
Is there anything else you would like to share?	36	80	1

All three courses were associated with high faculty and learner satisfaction. It is noteworthy that the learners’ negative sentiments were related to technical difficulties such as unstable or slow connections or camera or microphone problems, or similar challenges the faculty had experienced, causing delays or disruptions in the smooth running of the programme. It is also worth mentioning that most “neutral” answers were those suggesting no changes in the format would be necessary, so even these may be considered as being reasonably positive sentiments.

Table 7 summarises the overall evaluation results of spine courses conducted by our organisation in a F2F format in 2019 and 2020. The 2019 courses Microdecompression and percutaneous fixation, Endoscopy (two levels of introduction and advanced), and Complex cervical problems were purely F2F activities. The results are very comparable between F2F and blended online courses in 2020 regarding overall evaluation parameters.

Table 7.

Summary of data from post-event evaluation of AO Spine Davos courses 2019 (face-to-face) and 2020 (online)

Evaluation question (most on a Likert scale where 1 = lowest and 5 = highest)	2019 face-to-face courses(3 full days duration)				2020 blended online courses(6 weeks asynchronous + 3 days of 3 hours synchronous activity)
	Microdecompression and percutaneous	Complex cervical problems	Endoscopy (introduction)	Endoscopy (advanced)	Adult deformity	Minimally invasive spine surgery	Endoscopy
Responders (% of participants)	n = 9 (43%)	n = 15 (44%)	n = 13 (54%)	n = 12 (60%)	n = 9 (64%)	n = 14 (70%)	n = 20 (74%)
What was the overallimpact of this educationalevent? [High impact]	100%	93%	100%	100%	100%	93%	95%
To what degree were the stated objectives met? [Average for all objectives]	3.86	4.21	4.09	4.43	4.37	3.97	4.37
How useful was the content to your daily practice?	3.89	4.00	4.00	4.08	4.11	3.57	4.05
How effective were all faculty in the role they played?	3.67	4.13	4.08	4.25	4.44	3.93	4.20
Would you recommend this event to your colleagues?	100%	93%	100%	100%	100%	100%	90%
Please rate the venue/location	3.44	3.80	3.77	4.00	3.78	3.14	3.60

Discussion

In this study, we describe a fully online blended learning format as a method in postgraduate spinal surgery education and provide data that suggests this format can lead to significant improvements in learners’ subjective (LGs) competency levels and objective (MCQs) knowledge. Individual contributions of the two parts of blended learning (asynchronous and synchronous) to improvements in learning gaps and factors pertaining to these improvements were also demonstrated.

Educational Efficacy

Although accurate measurement of educational efficacy appears challenging, this study used objective and subjective parameters for this purpose. Based on the Kirkpatrick model [13] of four levels of learning (Reaction, Learning, Behaviour, Results), our survey targets the first level to gather impressions from participants and faculty, whereas the assessment questions target the second level. Using two separate tools, this level was assessed both subjectively and objectively. Our findings suggest that, in all three courses, learners reported improved levels of ability following the course (by decreased LGs) (statistically significant in two courses). Also, the end-of-course learning gaps (averaged for all objectives) on all courses below 1.0 indicated that the participant’s self-reported gaps in knowledge have substantially been reduced.

Data suggesting that this format resulted in learning attests to its efficacy as a concept. That narrowing of the learning gaps in the AD course had not reached statistical significance may be because the participants started it with a higher level of self-perceived ability (and knowledge) resulting in relatively small pre-course LGs (1.2 on average vs. 2.38 for Endo and 1.95 for MISS). Subsequently, their improvements could not be demonstrated to be as significant as in the other courses. Another possibility is that the relatively small number of participants (lower statistical power) in the AD course did not allow for the detection of any significant difference. Although not reaching statistical significance, the AD course ended up with the lowest average post-course LG of 0.4, supporting the idea that these participants also increased their knowledge. The quiz results suggest improved knowledge by participants after the activities, although the number of questions was limited and it is difficult to be certain that the questions included represent and cover all the spectrum of the LOs they have originated from. Because of this, we shall limit ourselves to stating that there was a significant improvement in the number of correct answers to quiz questions at the two time points analysed, which may further support the idea that the participants increased their knowledge.

Secondly, our findings suggest that substantial reductions in LGs took place at the asynchronous component of the courses (Tables 2 and 4). The learning gap improvements in the asynchronous part were bigger than the live online part for the Endo and MISS courses while roughly the same for the AD course. In addition, the improvement in the MCQ scores was higher in the asynchronous part then in the live online part of the Endo course.

This finding is in line with the reported advantages of distance learning and may be associated with the frequency and effort spent at the online platform [14,15]. As shown in Table 5 and Figure 1, participants remained engaged with the faculty and each other throughout the courses and even for several weeks following the course. Participants’ exposure to the course materials and the faculty started immediately once the asynchronous part started, remained at a high level throughout and peaked during the live event. We can speculate that this exposure afforded time enough not only for reading and watching the content but, more importantly, for reflection and discussing their learning with the other participants and faculty. This is highlighted by a comment posted by one participant:

“The weekly learning materials together with a blend of highly experienced, friendly faculty, and participants at various levels of the spine endoscopy learning curve created a unique learning discussion platform for six weeks before the live event. The simplification of each procedure into well-defined phases combined with numerous practical tips from faculty was a special highlight,

This finding is in line with the findings of Lindeman and colleagues [16] in a study where they compared the effectiveness of a blended learning curriculum with the traditional (lectures plus clinical experiences) curriculum and found that the blended online curriculum’s teaching ratings were significantly higher [16]. Likewise, Bock and colleagues demonstrated a similar finding in the context of Oral and Maxillofacial Surgery Clerkships [17]. As both of these studies are based on experiences in undergraduate settings, they are not directly comparable to our study. In this regard, our data suggests that this approach seems to be of similar effectiveness in both undergraduate and graduate surgical training. In addition, although any direct hands-on skills training was not included as a part of our courses, some learners have stated that they have learned new technical skills as well, as evidenced by the feedback from an endoscopy course as:

I operate my first L5-S1 TELD with craniocaudal approach.with GA without muscle relaxant … wonderful approach.I follow dr.xxxx’s instructions, and I get perfect result postoperative … Thanks

It is possible that such a level of learning may only be achieved in relatively more experienced learners. Nevertheless, our experience reported hereby suggests that the online part of a blended curriculum was at least as effective as the traditional methodology (see Table 7 for a comparison of fully F2F and fully online courses of 2019 and 2020 respectively).

Factors Associated with Improvements in Learning

Table 4 shows the total number of logins and posts for all three courses as 5,290 for faculty members and 22,183 for participants. It is also important to note that faculty and participant involvement remained highly correlated. We may thereby assume that one of the most important factors that improve participation is faculty being present at all times, communicating, answering questions, and providing feedback. This has also been emphasised by other authors [18-20] in different contexts, none surgical. It seems reasonable that faculty presence is important in our context as well simply because surgical education has traditionally been more reliant on learning directly from other people.

In this regard, learning communities and participants learning from each other in addition to faculty may be important [21]. In this regard, the time of exposure afforded to participants may be an important factor in group learning. This has previously been shown to increase learning efficiency in language education [22], but we still lack definitive evidence on it in our context. On the other hand, prolonged exposure periods may be deleterious as well and be discussed regarding their effect on participant retention. It is suggested that online courses have higher rates of dropout compared to traditional courses, ranging between 40% to 80% [23]. That we had very negligible rates of non-completion (only 2 out of 14 in the AD course) may have been related to a favourable time to content relation, avoiding drop-outs due to cognitive overload [24].

Faculty and Learner Feedback on Fully Online Blended Courses

As shown in Table 6, both the faculty and learners were satisfied with the courses overall, with very few negative responses or sentiments. Concerning faculty, courses were deemed as poor by only one, which may be related to a perception of overload. This format may be very demanding for the faculty and because of this, they need to be trained and prepared for this aspect of a blended course [24]. Our faculty training programme addressed this issue specifically and may be considered as having achieved its purpose based on the low rate of faculty dissatisfaction or perception of being overwhelmed.

Regarding participants, the responses were overwhelmingly favourable. Negative responses were mainly related to the technical aspects and shortcomings or problems with the LMS or the videoconferencing system. Also, an overwhelming portion was due to participants’ internet connection speeds and qualities (and, to a lesser extent, of faculty). This factor has been recently reported in a review by Almaiah and colleagues as one of the most critical factors influencing e-learning system usage [25]. Although it is not directly related to the learning design of the course(s), the possibility that many participants may not have high-speed internet access available needs to be considered, especially in the live parts of educational events. This may also be one reason for the participants being very enthusiastic about the asynchronous learning, as these do not require high-speed connections.

Pitfalls

Although it was initially planned as a group of six courses, three of these had to be forsaken. Lack of registrations and cancellations resulted in underutilisation of available course capacity. One factor that had affected all online educational activities globally is the so-called “webinar fatigue”, an abundance of free online educational offerings that eventually overwhelmed potential learners. Besides, pricing strategies for online educational activities had become arbitrary. It is very plausible that the under-registration problem is closely associated with the fact that our courses were relatively highly priced (900 Swiss franks per course). Another factor may be the reluctance of the faculty to believe that training in a surgical discipline can be executed in a fully online format, since this concept has not yet been thoroughly tested in surgical specialities when it was offered and proven to be a valid surgical education context.

Limitations

The main limitation of this study is the relatively small number of courses and participants that could be evaluated. On the other hand, the discrepancy between small goup learning, with higher educational effectivity and less statistical power, and learning in large groups, with less educational effectivity, but appropriate statistical power, is unlikely to be ever resolved.

A second shortcoming is the lack of control group(s) that had used blended online and F2F formats. Senkoylu and co-workers have reported on the learning efficacy of blended online and F2F learning in spine surgery [8] and our experience reported here echoes their findings, that study may serve only as a historical reference because of the difference in setting and context. The ideal control group for the present study would have been a blended online and F2F course running in parallel with one (or more) of the courses included in this study but could not be executed due to the limitations to F2F learning imposed by the pandemic. Thus, his absence renders any comparison between these formats virtually impossible at this point. Our future work will include the development of such parallel courses which would serve as real prospective controlled studies.

Practice Points

A fully online blended learning course can be designed and delivered effectively in spine surgery education

Significant factors for dissatisfaction are problems with technology related to software, staff and faculty unfamiliarity with the software, or the speed and quality of the internet connections of participants (or faculty)

The success of online activities is highly dependent on faculty time and effort and faculty training must be an integral part of this context