Difference in postcourse knowledge and confidence between Web-based and on-site training courses on resuscitative endovascular balloon occlusion of the aorta.

AIM: Effective courses are essential for highly invasive procedures such as resuscitative endovascular balloon occlusion of the aorta. However, the coronavirus disease pandemic has forced the postponement of on-site educational courses due to transmission concerns. Few studies have examined the effectiveness of Web-based education in highly invasive procedures. To address this knowledge gap, this study aimed to investigate whether knowledge acquisition and confidence after the Web-based course are different from those acquired after the on-site course, using pre- and postcourse test scores.
METHODS: The increase in scores before and after the course was compared between the on-site and Web-based courses. The questions reflected knowledge about seven different topics in the course modules. In addition, participants were asked about their self-rated confidence about three topics before and after the course.
RESULTS: Thirty learners completed the on-site course, and 21 learners completed the Web-based course. Forty-seven learners completed both the precourse and postcourse tests. In both courses, the difference between the precourse and postcourse test scores showed a statistically significant increase in knowledge (on-site course: increased score, 1.8; 95% confidence interval, 0.8 to 2.8; Web-based course: increased score, 1.6, 95% confidence interval, 0.5 to 2.5). However, the difference was not statistically significant in the self-rated confidence scores about "sheath and catheter removal" among learners of the Web-based course.
CONCLUSION: Knowledge increased significantly in both the on-site and Web-based courses. However, the Web-based course might not be sufficient to give learners confidence in the procedures.

BACKGROUND

The field of emergency medicine involves many highly invasive procedures, such as resuscitative endovascular balloon occlusion of the aorta (REBOA). The REBOA has been recognized as effective in providing temporary hemostasis in patients exposed to fatal blood loss. , , However, correct knowledge and skills are essential because of the multitude of complications and the need to become proficient in the placement technique. ,

Thus, effective courses are essential for proficiency. , , , However, the coronavirus disease (COVID‐19) pandemic has forced the postponement or cancelation of various educational courses due to transmission concerns.

Web‐based education is effective not only for knowledge but also for procedures. , , However, these studies have shown effective education for minimally invasive procedures. Few studies have examined the effectiveness of Web‐based education in highly invasive procedures, such as REBOA.

To address this knowledge gap, this study aimed to investigate whether knowledge acquisition and confidence after the Web‐based course, which cannot provide a hands‐on station, are different from those acquired after the on‐site course, using pre‐ and postcourse test scores.

METHODS

Setting and context

The REBOA working group of the Japanese Society of Diagnostic and Interventional Radiology in Emergency, Critical Care, and Trauma (DIRECT) conducted the REBOA course (DIRECT‐REBOA course) in 2019. However, in 2020, we decided to conduct the course Web‐based because of COVID‐19. The DIRECT‐REBOA course was created and piloted in Japan to address the critical need for a standardized handling of REBOA in an emergency and critical care setting. The course content was based on the standard handling of REBOA mainly for emergency and critical care residents and attendants. The on‐site course material was modified after discussions with the course instructors. The instructors included emergency physicians, intensivists, interventional radiologists, and acute care surgeons to ensure the quality of the course.

The DIRECT‐REBOA course comprises several components, including didactic lectures, skill stations using models, and simulation. The overall course objective is “to provide knowledge and technical skills for adequate and safe handling of REBOA.” The content of didactic lectures includes seven topics, including insertion, sheath and catheter removal, and complication. At the skill stations, learners actually apply the skills taught in the lectures to mannequins. As the Web‐based course does not allow the hands‐on practice of actual procedures, only basic procedure schemas or videos were presented. The instructors discussed the patients’ overall pre‐, during, and postprocedural care during this course. Table 1 shows a detailed timetable of the on‐site and Web‐based course.

Table 1

Detailed timetable of the Web‐based training course on resuscitative endovascular balloon occlusion of the aorta (REBOA)

Length (min)	Content
5	Opening remarks
10	Course description, pretest
35	Lecture 1: Basic knowledge of REBOA
35	Lecture 2: REBOA in non‐trauma
10	Break
35	Lecture 3: Arterial access
35	Lecture 4: REBOA techniques
10	Break
35	Lecture 5: Complications of REBOA
35	Lecture 6: Resuscitative thoracotomy and REBOA
20	Posttest, Q&A session
5	Closing remarks

Participants, data collection, and statistical analysis

The learners’ precourse test was distributed just before starting the first lecture, and the postcourse test was distributed immediately after completion of the course. The test questions were created by course instructors based on the objectives of each lecture. A series of questions were vetted to ensure that they reflected the course content, wording, and appropriateness. The test consisted of 28 true/false questionnaires. Both the pre‐ and postcourse test questions were written in Japanese and were the same. The questions reflected seven different topics in the course modules, including insertion, complication, indication, and management. Students were required to answer all questions, and each question received one point for each correct answer. Therefore, the maximum score of this test was 28 points. In addition to assessing knowledge, participants were asked about their self‐rated confidence regarding insertion, sheath and catheter removal, and complication before and after the course. Self‐rated confidence was measured using 10‐point Likert scales (1 = least confident, 10 = most confident). Table 2 shows the actual test used before and after the course.

Table 2

Questionnaire taken before and after courses on resuscitative endovascular balloon occlusion of the aorta (REBOA)

Question 1. Are the following statements about the position of the sheath true or false?

Puncture the superficial femoral artery.

Puncture just above the inguinal ligament.

It is crucial to confirm the level of bifurcation by echocardiography due to the high femoral artery bifurcation.

A real‐time ultrasound‐guided puncture does not improve safety.

Question 2. Are the following statements about the REBOA true or false?

REBOA has a high hemostatic effect.

Some data suggest that REBOA could be harmful.

REBOA is a safe device with no complications.

REBOA has established itself as an effective resuscitation tool with solid evidence.

Question 3. Are the following statements about REBOA insertion true or false?

ZONE 2 is always the preferred position.

The umbilicus marks the inferior border of ZONE 1.

The pubic symphysis marks the inferior border of ZONE 3.

Both the catheter and the sheath should be grasped before balloon dilation.

Question 4. Are the following statements about the management of REBOA true or false?

The target proximal arterial pressure is 60 mmHg.

Compression hemostasis is not possible during the sheath and catheter removal of a 7Fr sheath.

Do not remove only the REBOA catheter, leaving the sheath in place.

Partial REBOA is a method to reduce organ ischemia.

Question 5. Are the following statements about the complications of REBOA true or false?

If the physician feels resistance when inserting a guidewire, try pushing it in.

Organ damage by reperfusion is more likely to occur with intermittent dilation than continuous dilation.

To reduce lower extremity ischemia, insert a large‐bored sheath.

If the vessel is tortuous, the Landmark method cannot work effectively.

Question 6. Are the following statements about REBOA and aortic clamping true or false?

The decision on which to perform REBOA or aortic clamping is easy to make.

When the pulse is not palpable, it is better to choose REBOA.

It is difficult to switch from aortic clamping to REBOA.

Aortic clamping is highly invasive compared to REBOA.

Question 7. Are the following statements about the difference between blunt trauma and nontrauma true or false?

REBOA is contraindicated in AAA rupture.

In postpartum hemorrhage, REBOA should be placed in ZONE 1.

REBOA is not effective in upper gastrointestinal bleeding.

REBOA is effective for high bleeding risk cesarean section.

AAA, abdominal aortic aneurysm.

The institutional review board approved the protocol with waiver of informed consent because the pre‐ and postcourse tests were anonymized immediately after being paired. A Wilcoxon matched‐pairs signed‐rank test was carried out on the pre‐ and postcourse test scores. Stata/SE 16.0 (Stata Corp) was used for data analyses. The threshold for significance was set at p < 0.05.

RESULT

Thirty learners completed the on‐site course in December 2019, and 21 learners completed the Web‐based course in October 2020. Forty‐seven learners completed both the precourse and the postcourse tests. The complete details of the sample characteristics are shown in Table 3. In summary, the median postgraduate years of the learner was 6 years (interquartile range [IQR], 4–9 years), and 43% (19/44) of the learners were board‐certified emergency physicians.

Table 3

Basal characteristics of learners who participated in training courses on resuscitative endovascular balloon occlusion of the aorta (REBOA)

Variable	Overall (N = 47)	On‐site course (n = 26)	Web‐based course (n = 21)	p‐value
PGY, median (IQR)	6 (4‐9)	7 (5‐10)	6 (4‐9)	0.34
EM board‐certified, n (%)	19 (40.4)	13 (50.0)	6 (28.6)	1.00
No previous experience of REBOA, n (%)	10 (21.2)	6 (23.1)	4 (19.0)	0.45

EM, emergency medicine; PGY, postgraduate year.

The median precourse test score was 25.4 (IQR, 24.4–26.4), compared with 27.2 (IQR, 26.8–27.6) for the postcourse test in the on‐site course. In the Web‐based course, the mean precourse test score was 25.7 (IQR, 24.7–26.7), compared with 27.3 (IQR, 26.7–27.8) for the postcourse test. In both courses, the difference between the precourse and postcourse test scores showed a statistically significant increase in knowledge among participants (Table 4). Thirty‐two learners showed an improved score, while nine learners maintained the same score.

Table 4

Pre‐ and postcourse scores in on‐site and Web‐based courses on resuscitative endovascular balloon occlusion of the aorta

Variable	Precourse	Postcourse	Difference	p‐value
On‐site (N = 26)
Test score	25.4 (24.4–26.4)	27.2 (26.8‐27.6)	1.8 (0.8–2.8)	<0.010
Self‐rated confidence of insertion	4.5 (3.4–5.7)	6.8 (6.1–7.6)	2.3 (1.4–3.2)	<0.010
Self‐rated confidence of sheath and catheter removal	4.7 (3.6–5.7)	6.8 (6.0–7.6)	2.2 (1.4–2.9)	<0.010
Self‐rated confidence of complication management	3.6 (2.7–4.5)	5.9 (5.2–6.6)	2.3 (1.5–3.1)	<0.010
Web‐based (N = 21)
Test score	25.7 (24.7–26.7)	27.3 (26.7–27.8)	1.6 (0.5–2.6)	<0.010
Self‐rated confidence of insertion	3.6 (2.5–4.7)	5.1 (4.2–6.0)	1.5 (0.0–2.9)	0.045
Self‐rated confidence of sheath and catheter removal	4.1 (3.0–5.3)	5.3 (4.5–6.3)	1.2 (−0.3–2.7)	0.150
Self‐rated confidence of complication management	3.1 (2.2–4.1)	5.0 (4.2–5.8)	1.9 (0.5–3.2)	0.010

The learners’ self‐rated confidence scores are presented in Table 4. The increases in the self‐rated confidence scores were statistically significant among learners in the on‐site course. However, the difference was not statistically significant in the self‐rated confidence scores about “sheath and catheter removal” among learners in the Web‐based course.

DISCUSSION

There was a significant increase in scores between the pre‐ and postcourse tests for both the on‐site and Web‐based courses. In addition, the self‐rated confidence levels in all the procedures significantly increased during the on‐site course. In contrast, in the Web‐based course, not all fields showed significantly increased self‐rated confidence in the procedures.

The significant increase in test scores could be an indication that didactic lectures are an efficient method of transferring knowledge. The lecture content was the same for the Web‐based and on‐site courses; thus, the results suggest no difference in the educational effectiveness of on‐site and Web‐based courses in terms of knowledge transfer. Web‐based resources have been shown to be practical educational resources. This result is consistent with previous studies. , ,

However, self‐rated confidence about sheath and catheter removal was not significantly increased in the Web‐based course. Self‐rated confidence in a procedure is a concept that encompasses not only what one “knows” but also what one “can do.” Thus, the Web‐based course, which consisted of didactic lectures that mentioned technical procedures, was insufficient for learners to gain self‐rated confidence. A significant increase was observed in the on‐site course, which included hands‐on sessions, suggesting that a combination of lectures and hands‐on sessions plays a significant role in gaining self‐rated confidence.

Emergency physicians are required to perform various procedures during shifts. “Just knowing” is insufficient for them. In this COVID‐19 pandemic era, when it is difficult to undertake on‐site training, Web‐based training has become popular. However, educators need to be aware that there are still challenges in learning procedural skills through a Web‐based course only. For students to gain confidence in the highly invasive procedure, taking the course while handling the actual devices or instructors peer‐reviewing videos of students performing the actual procedures after the course could be possible solutions.

This study had several limitations. First, because knowledge and confidence were measured before and immediately after the course, it is unclear whether knowledge can be retained in the long term. It could be useful to reexamine knowledge and confidence after a year of the course with postcourse clinical experience. Although the differences in learning effects due to the learning methods were evident in this study, it is necessary to prove the scientifical and clinical usefulness of the course itself by measuring the long‐term retention of knowledge so that this result can be informative to educators. Second, although the goal of education for a clinician is to enable them to “do” in a clinical setting, it is unclear whether students actually perform REBOA placement. Third, true/false tests could be insufficient to measure knowledge. The purpose of this study was to compare the scores before and after the course. Thus, the measurement method did not need to be rigorous. Finally, the number of students was limited.

CONCLUSION

Knowledge increased significantly in both the on‐site and Web‐based courses. However, the Web‐based course might not be sufficient to give learners confidence in the procedures.

DISCLOSURE

Approval of the research protocol: The research was approved by the Ethics Committee of Tokyobay Urayasu Ichikawa Medical Center (reference no. 684).

Informed consent: The need for informed consent was waived because the pre‐ and postcourse tests were anonymized immediately after being paired.

Registry and registration number of the study: N/A.

Animal studies: N/A.

Conflict of interest: None.