Effectiveness of an Online Course on Fracture-Related Infections.

This study aims to evaluate the effectiveness of an online course to enable orthopaedic surgeons to acquire the core competencies necessary to prevent and treat fracture-related infections (FRI). This study included orthopaedic surgeons and residents from Latin American countries who attended an online course focused on FRI. The online course included: didactic lectures, small-group clinical case discussions, and panel case discussions. The course was delivered using Zoom® platform and designed to address four core competencies: prevention, definition and diagnosis, antimicrobial therapy, and surgical treatment. An online questionnaire was created distributing 16 questions through six clinical scenarios. Participants were invited to answer the questionnaire before and after the course. Sixty of the 78 course participants answered the pre-course, and 42 the post-course assessment. Relative to before the course, the mean post-course assessment score rose significantly for prevention of FRI (4.1 before and 4.5 after; p = 0.014), definition and diagnosis (2.4 before and 3.4 after; p = 0.001), and surgical treatment (2.2 before and 2.8 after; p = 0.011). The final score encompassing all four core competencies also rose significantly (2.7 before and 3.3 after; p = 0.001). The online course on FRI was feasible and effective, significantly increasing course users' knowledge of overall competency in managing FRI.

Introduction

Medical education is essential for patient care and online learning, such as videoconferencing and eLearning platforms, has been used extensively to deliver lectures, tutorials, and courses remotely via handheld devices and laptops[1]. The recent coronavirus outbreak (COVID-19) has created profound changes in healthcare systems and considerable challenges for medical education programmes [1-3]. Consequently, a change in medical education has been necessary to deliver education in a pandemic setting.However, this rapid movement towards online educational platforms has resulted from an emergent response to a health crisis and little is known about the effectiveness of these tools.

To be effective, medical education programmes must have measurable outcomes and be based on competencies. O’Malley et al. defined competency as “the unique combination of knowledge, skills, and attitudes that enables one to perform in practice; this describes what a physician must be able to do to diagnose and treat patients with a specific clinical problem or issue”[4]. Competency-based medical education is based on clearly-defined learning objectives, an explicit curriculum, and an assessment framework to ensure that learners have acquired the necessary knowledge and skills they need to transition into independent practice. [5,6]

Postoperative bone infection is a severe complication that can arise during the treatment of fractures. Several scientific organisations have collaborated to generate a definition for fracture-related infection (FRI) that addresses the unique characteristics of this clinical entity[7]. Furthermore, in a series of publications, this consensus group reviewed best practices for the diagnosis and treatment of FRI [7,8]. Most of these guidelines were published in English over the last three years and much of this information has not yet been incorporated into classic textbooks. As such, many young surgeons and residents in non-English speaking countries remain unaware of these new concepts. Thus, an online “FRI course” appeared to us to be a perfect model to evaluate whether an online course format can be effective at updating course participants and introduce new competencies.

The objective of this study was to evaluate the effectiveness of an online course to enable orthopaedic surgeons to acquire the key competencies necessary for the prevention and treatment of fracture-related infections.

Materials and Methods

This prospective observational study included orthopaedic surgeons and orthopaedic surgery residents from several Latin American countries who attended an online course dedicated to managing FRI. The course was conducted using an online format. The research ethics committee approved the study, and all participants consented to participate.

The Study Sample

The study population was composed of orthopaedic surgeons and orthopaedic surgery residents who attended the online FRI course and answered the study’s online assessment questionnaire. Information regarding demographic data (age and gender) and each study participant’s country of residence was gathered. The participants’ clinical experience was evaluated based on the number of years they had been practicing as orthopaedic surgeons and whether they had already completed an orthopaedic residency training programme. All faculty instructors had access to the questions before the course and were invited to respond to the post-course assessment.

Course Structure

For the study, a 12-hour course was planned, divided into four 3-hour modules, one module per week for four consecutive weeks. The course involved eLearning activities divided into three major teaching methods: lectures, small-group clinical case discussions, and panel case discussions involving all the participants and faculty instructors. The course was delivered using the Zoom® platform. For small-group discussions, ten discussion groups – each having a maximum of eight participants and one or two facilitators – were created.

Definition of Competencies

The course programme was designed to address four core competencies related to FRI management: prevention, definition and diagnosis, antimicrobial therapy, and surgical treatment. To cover each of these competencies, the following activities were planned: 15-minute lectures; small group discussions involving real clinical cases; and panel discussions. Two of the authors were responsible for deciding the educational objectives for each lecture and which competencies the participants should acquire by the end of the activities.

Course Evaluation

To assess the effectiveness of this online learning tool, an online questionnaire was created based on 20 questions spanning six clinical scenarios. These questions were intended to assess the four previously-mentioned core competencies, among which ten different component competencies were tested, all related to the management of patients with FRI. These ten component competencies were, as part of the core competency prevention: (a) use of prophylactic antibiotics, n = 3 questions; for defining and diagnosing FRI (b) diagnosing an FRI and (c) classifications systems, n = 5 questions; for antimicrobial therapy (d) local and (e) systemic therapy, n = 3 questions; and for surgical treatment (f) early and (g) late infection management, (h) debridement, (i) intra-operative tissue sampling, and (j) implant removal versus retention, n = 9 questions (Supplementary Material 1). Two types of question were used: true/false questions and multiple-choice questions. All course participants were invited by email to answer the questionnaire two weeks before and three weeks after the course (first invitation to post-course assessment at two weeks and second at three weeks after the course). Participants were not allowed to answer the questionnaire more than once or change answers. The order of the clinical scenarios and questions varied between the pre- and post-test, but there were no changes in question content. The correct answers to the questions were not discussed with the participants during the course. The faculty instructors had access to the questions two weeks before the course, and to the result of the pre-course evaluation on the first day of the course. They were taught to address the competencies in the lectures and discussions, but to avoid making any direct references to the pre-course evaluation.

Statistical Analysis

All continuous variables were summarised as a mean, standard deviation, and range. For normally-distributed variables, unpaired and paired t-tests were used to compare inter- and intra-group means. Pearson chi-square analysis was used for categorical variables. Pre- and post-course responses were analysed both together and separately, depending on the course learning competency. Considering that many issues regarding FRI remain controversial, some multiple-choice questions had more than one correct answer. A given response option was considered correct or incorrect when greater than 66% agreement was established among the faculty instructors who participated in this process, and was further validated using the best-quality evidence available in the literature. Among the instructors, because more than 66% consensus was not achieved for four of the 20 questions in the original survey, these four questions were not included in the calculation of core or overall competencies and are reported separately. The four omitted questions asked about the best surgical option for early intramedullary infection; number of tissue samples to collect during surgery; where to take samples from; and the best way to manage infected non-union after plate fixation (Supplementary Material 2).

Each given competency ultimately was graded from 0 to 5 (with zero indicating no competence and 5 complete competence) by dividing the number of correct responses by the total number of questions testing that competency, then multiplying that fraction by 5. For example, if a participant had correct answers for three out of five questions covering a given competency (i.e. Competency in Surgical Treatment), then the final competency score would be: (5/5) multiplied by 3 = 3. Mean scores for competency then were compared between competencies (unpaired t tests) and within competencies before versus after the course (paired t tests). Also, the absolute frequency of each answer was measured for each question in the pre- and post-course evaluation, and the proportion of variation evaluated. Data were analysed using SPSS® version 25, with the threshold for statistical significance set at α = 0.05.

Results

A total of 78 participants signed up and concluded the course. Of these, 60 answered the pre-course assessment and 42 the post-course assessment. Thirty-four participants answered the pre-and post-course evaluations. Thus, an independent analysis was carried out comparing all 60 with these 34 regarding pre-course answers, and no statistical difference was observed (Supplementary Material 3). No significant differences were observed in the demographic characteristics (age and gender) or country of residence between course participants who completed the pre- and post-course evaluations, nor in the years of surgical practice. However, the percentage of participants who had finished residency training was lower in the post-course assessment (p = 0.002) (Table 1). Nine of the 16 faculty instructors answered the assessment used to support decisions regarding correct answer for the questions. The pre- and post-course results are summarised and correct responses for each question listed in Table 2 through 6.

Table 1.

Demographic characteristics and clinical experience of course participants who completed the pre- versus post-course assessment

	Selection	Pre-course	Post-course	p
Number of participants	Number	60	42
Age, mean ± SD (range)	Years	35.9 ± 6.2 (27–60)	36.4 ± 6.2 (30–60)	0.71
Gender, number (%)	Male	50 (83.35)	32 (76.2%)	0.37
Country, number (%)	Peru	14 (23.3%)	7 (16.7%)	0.63
	Chile	11 (18.3%)	12 (28.6%)
	Colombia	9 (15%)	6 (14,3)
	Others	26 (43.4%)	17 (40.5%)
Completed residency training, number (%)	No	48 (80%)	27 (64.3%)	0.002
Years in practice, mean ± SD (range)	Years	5.2 ± 5.4 (0–24)	6,3 ± 5.6 (0–24)	0.44

SD = standard deviation

Table 2.

Comparing levels of the core competency – Infection Prevention – before and after the on-line course

		% instructor	% of correct responses		Pre to post	Statistical
Competencies	Correct response	agreement	before	after	change	significance
Selection of prophylactic Abx	1st generation cephalosporin	100%	98%	98%	0%	NS
Duration of prophylactic Abx	Single dose pre-op	89%	58%	81%	23%	p = 0.016
Timing of prophylactic Abx	30–60 minutes pre incision	89%	90%	95%	5%	NS
	Overall proficiency score:		4.11	4.56	0.45	p = 0.014

Abx = antibiotic

For the core competency – Infection Prevention (Table 2) – a 23% increase in correct responses was noted between the responses given before versus after the online course for the administration of a prophylactic antibiotic, a difference that was statistically significant. Minimal to no change was observed for either the choice or timing of that antibiotic; however, 98 and 90% of the responses were correct at baseline (before the course), leaving almost no potential for improvement. The overall core competency score increased from 4.11 to 4.56 (p = 0.014).

For the core competency – defining and diagnosing FRI (Table 3) – a significant increase in correct answers was observed for four of the five questions (with p values ranging from 0.01 to < 0.001), and there was a marked increase in the overall core competency score, from 2.43 to 3.43 (p < 0.001). On the other hand, whereas 93% of the course participants who completed the baseline questionnaire correctly agreed with the statement that wound dehiscence with bone exposure is a confirmatory criterion for infection, this percentage was slightly lower after the course, at 85%.

Table 3.

Comparing levels of the core competency – Definition and Diagnosis – before and after the on-line course

		% instructor	% of correct responses		Pre to post	Statistical
Competencies	Correct response	agreement	before	after	change	significance
Wound dehiscence with bone exposure is confirmatory of infection …	TRUE	89%	93%	85%	−8%	p = 0.160
Per the FRI definition, a patient with purulent drainage is considered …	Infection confirmed	100%	79%	92%	13%	p = 0.083
Further imaging studies needed for late infection after plate fixation.	None	67%	14%	42%	28%	p = 0.003
Preferred system for classifying an early intramedullary infection =	Early (Willenegger & Roth)	89%	33%	79%	46%	p < 0.001
Preferred system for classifying a late infection after plate fixation =	Late (Willenegger & Roth)	78%	29%	72%	43%	p < 0.001
	Overall proficiency score:		2.43	3.43	1.00	p = 0.001

For the core competency antimicrobial therapy for FRI (Table 4), the percentage who correctly selected rifampicin to treat a Staphlococcus aureus infection increased by 35% after the course (p = 0.001). However, the percentage of correct answers fell for the remaining two questions, and there was virtually no increase at all in the overall core competency score after the course (2.94 to 2.97).

Table 4.

Comparing levels of the core competency – Antimicrobial Therapy – before and after the on-line course

		% instructor	% of correct responses		Pre to post	Statistical
Competencies	Correct response	agreement	before	after	change	significance
Venous antibiotic therapy should be for at least 4 weeks.	FALSE	78%	55%	48%	−7%	p = 0.462
When possible, rifampin should be used to treat S. aureus infection	TRUE	89%	40%	75%	35%	p < 0.001
Best ABx to pair with PMMA for MSSA infection	Vancomycin	67%	82%	65%	−17%	p = 0.59
	Overall proficiency score:		2.94	2.97	0.03	p = 0.910

ABx = antibiotic

Finally, for the core competency pertaining to the surgical treatment of FRI (Table 5), the percentage of correct responses increased for all five items, with increases ranging from 12 to 25%, one of these changes statistically significant. The overall core competency score also increased significantly, from 2.25 to 2.85 (p = 0.011).

Table 5.

Comparing levels of the core competency – Surgical Treatment – before and after the on-line course

		% instructor	% of correct responses		Pre to post	Statistical
Competencies	Correct response	agreement	before	after	change	significance
Best option for early intramedullary infection.	SD+SC+STR+Abx	67%	57%	69%	12%	p = 0.209
Delayed intramedullary infection – implant removal or retention	IR+MCD+IMF	100%	48%	60%	12%	p = 0.252
Delayed intramedullary infection – what not to do.	Always retain stable implant	89%	63%	75%	12%	p = 0.220
Approach to bone debridement	Radical (oncologic margins)	67%	30%	55%	25%	p = 0.013
How to store and ship samples	Sterile container, no saline	67%	29%	46%	17%	p = 0.090
	Overall proficiency score:		2.25	2.85	0.60	p = 0,011

SD+SC+STR+Abx = surgical debridement, sample collection, soft tissue reconstruction, and antibiotics

IR+MCD+IMF = implant removal, medullary canal debridement, and intramedullary fixation

Across the 16 questions asked, the numbers of questions where the percentage of correct answers increased, remained unchanged, and decreased were 12, one, and three, respectively, with six of the 12 increases statistically significant. The average score when all four core competencies were combined significantly increased between the pre- and post-course assessments, from 2.78 to 3.38 (p = 0.001). Comparing the four core competencies, however, there were significant differences both in the level of competency attained and the extent of improvement, ranging from virtually no improvement in the course participants’ competency with antimicrobial therapy to a marked 20% absolute and 41% relative increase in their competency defining and diagnosing FRI (Table 6). An analysis involving only the 34 participants who answered the pre and post-course evaluations showed similar results concerning the average score of the four core competencies and the final score with all core competencies combined (Supplementary Material 4).

Table 6.

Analysis comparing the pre- and post-assessment grades for each competence and the final score

Competence	Pre-course grade(Mean – SD)	Post-course grade(Mean – SD)	P*
Prevention	4.11 (1.08)	4.56 (0.74)	0.014
Definition and diagnosis	2.43 (1.09)	3.43 (1.5)	0.001
Antimicrobial therapy	2.94 (1.41)	2.97 (1.35)	0.910
Surgical treatment	2.25 (1.14)	2.85 (1.18)	0.011
Final score	2.78 (0.63)	3.38 (0.89)	0.001

* T test for independent samples with different variances

Discussion

The global COVID-19 pandemic has altered everyday interactions, due to the necessity of social distancing to reduce the spread of the virus[9]. As medical education providers, like speciality societies, cancelled their meetings and conventions, a migration to technology-based virtual learning experiences was observed. As a result, many institutions have implemented eLearning platforms and virtual events to adhere to social distancing rules and continue with their mission of providing medical education [9,10].Electronic learning (eLearning) is generally defined as using the internet and other media distribution methods to enhance learners’ knowledge and performance[11]. The widespread use of smartphones, tablets, and multimedia platforms affords new ways to deliver educational content. Electronic resources, such as WhatsApp and Twitter, are part of our lives, and most are open to using these technology-driven tools as education facilitators[10]. The use of video-conferencing applications like Webex®, Google meet®, and Zoom® increased recently, and these applications have many features that make them perfect tools for meetings, webinars, and case discussion sessions [9,12].

Convenience is probably the most beneficial aspect of online learning[9]. Other advantages are the ability to revisit recorded sessions, connect individuals in different geographic regions, and reduce costs[12]. There is tremendous versatility in this new eLearning technological toolkit: allowing online systems to mimic live classroom interactions, live-group discussions, and feedback pools. Contrary to the traditional classroom, webinars can be offered either live or on-demand. The online format also can be expanded, giving participants more exposure to experts in their field than they may might had with traditional learning[10].

Although virtual education may be convenient, it is unlikely to ever replace hands-on and face-to-face clinical experiences entirely[10]. There are dynamic aspects that take place during in-person interactions that cannot be replicated virtually. In surgical specialities, virtual education encompasses virtual case scenarios, online tutorials, videos, and images. Furthermore, developers may incorporate various updated educational strategies like spaced-repetition learning and “blended-learning’, which combine online and face-to-face instruction. Besides this, the ability to network and develop meaningful relationships is more likely to occur during in-person encounters[9]. These limitations notwithstanding, the scarce evidence available suggests that online learning is no less effective teaching clinical skills than traditional means[13].

Most eLearning tools are limited to teaching the cognitive processes necessary for learners to develop the cognitive elements they need to perform psychomotor tasks[14]. However, advancement needs a degree of automaticity and thinking achieved only through simulation and clinical practice[14]. As such, eLearning could serve as an adjunct to improve a curriculum’s effectiveness, especially when the curriculum has a dominant cognitive component[14]. The majority of studies evaluating the learning and acquisition of skills through written/oral examinations have used performance-based assessments in a non-clinical simulated environment, and an objective assessment using pre- and post-testing for email or web-based programs is a well-known method of evaluation [14,15]. Promising results have been demonstrated employing a standardised needs assessment protocol[16]. An evaluation system focused on the relevance of the educational activities is essential and the use of pre- and post-event multiple-choice questions to estimate the extent of knowledge gained is a well-known strategy[17]. In the present study, the assessment questionnaire we utilised to evaluate learning gaps was administered before the online course. Questions were designed to address the competencies that participants should learn by the end of the course. The post-course assessment revealed marked increases in almost all of the competencies, providing evidence that the online format effectively addressed knowledge gaps.

Some competencies, like those related to antibiotic prophylaxis, are well defined in the literature, and their recommendations are based on good-quality evidence [18,19]. Our questions were used to reinforce these concepts and ensure that most participants could assimilate these competencies. The high percentages of correct answers obtained in the post-test for antibiotic prophylaxis indicate that, after the course, participants were able to apply tasks related to this subject properly.

Regarding diagnosis and classification, after the course most of the participants were able to diagnose FRI based on clinical confirmatory criteria (wound breakdown or purulent drainage). A significant shift towards using the Willenegger and Roth classification system [20] was observed in the post-course survey, though there is no evidence to favour one specific classification system over another. Still, the FRI consensus group recommends using a time-based classification system that emphasises the importance of biofilm maturation for FRI management [21-24].

Antimicrobial therapy competencies approached some critical concepts in FRI, such as using rifampicin to treat S. aureus infection and the duration of administered intravenous antibiotic therapy. The standard recommendation is at least two weeks of intravenous therapy before transitioning to oral therapy. However, in a recent multicenter randomised controlled trial, earlier transition was not found to diminish treatment efficacy. This concept of a shorter course of intravenous therapy has been advocated in recent guidelines and after-course results have demonstrated that course participants can assimilate this concept [8,25].

The surgical treatment of FRI has two main alternatives: (a) debridement, antimicrobial therapy, and implant retention or (b) debridement, antimicrobial therapy, and implant removal. Implant stability and bone union must be considered when deciding between implant removal versus retention [21,23]. Our post-course survey results demonstrated an increase in the number of correct answers in competencies related to surgical treatment, with course participants exhibiting considerable improvement in their ability to correctly recommend surgical treatment to the detriment of clinical management. An increase in the correct recommendations for implant retention or removal and a staged surgical management decision was observed.

Although post-course results revealed substantial improvement, the final average competency score of 3.1 out of 5 is far from a desirable level; that said, the baseline competency level was low (2.1). Among the nine faculty instructors who took the course assessment, the final score was roughly 3.7. As among the course participants, among the instructors, the learning competence scores were high for antibiotic prophylaxis and very low for antibiotic therapy and surgical treatment. These findings may reflect the lack of consensus in some FRI topics and heterogeneity in the participants’ clinical practices.

Study limitations include the small sample size and that participants were limited to one geographic area (Latin America). Additionally, a small number of participants did not complete the pre and post-course survey. The post-course assessment also was completed three weeks after the course, so our results represent just a short-term view of the participants’ understanding, and cannot confirm any true intention for them to change their practice. The few true or false questions included in the assessment can also be considered a limitation, increasing the chance of correct answers. Our results should be interpreted with considerable caution, considering the peculiarities of the virtual format and curriculum used. Of note, this course format has been used extensively in face-to-face AO courses and was just adapted to be delivered online. The same is true for the pre and post-course evaluation format, which have also been used extensively to identify learning gaps and back-ward planning [16,17]. In part, this FRI course’s online transition benefited from the absence of psychomotor activities, like practical exercises on synthetic bone or time spent in a cadaver lab. Future studies comparing the virtual format against face-to-face courses could provide additional information about the benefits of each model in curriculum improvement.

Conclusions

Transitioning an FRI course from an in-person to online format was both feasible and effective. After course evaluations demonstrated a significant increase in the course participants’ knowledge levels in most of the competencies evaluated.

Click here for additional data file.