Development of a Standardized Assessment of Simulation-based Extracorporeal Membrane Oxygenation Educational Courses.

Background: In 2020, the Extracorporeal Life Support Organization education task force identified seven extracorporeal membrane oxygenation (ECMO) educational domains that would benefit from international collaborative efforts. These included research efforts to delineate the impact and outcomes of ECMO courses. Objective: Development of a standardized online assessment tool to evaluate the effectiveness of didactic and simulation-based ECMO courses on participants' confidence, knowledge, and simulation-based skills; participant satisfaction; and course educational benefits.
Methods: We performed a prospective multicenter observational study of five different U.S. academic institution-based adult ECMO courses that met Extracorporeal Life Support Organization endorsement requirements for course structure, educational content, and objectives. Standardized online forms were developed and administered before and after courses, assessing demographics, self-assessment regarding ECMO management, and knowledge examination (15 simple-recall multiple-choice questions). Psychomotor skill assessment was performed during the course (time to complete prespecified critical actions during simulation scenarios). Self-assessment evaluated cognitive, behavioral, and technical aspects of ECMO; course satisfaction; and educational benefits.
Results: Out of 211 participants, 107 completed both pre- and postcourse self-assessment forms (97 completed both pre- and postcourse knowledge forms). Fifty-three percent of respondents were physician intensivists, with most (51%) practicing at academic hospitals and with less than 1 year of ECMO experience (50%). After the course, participants reported significant increases in confidence across all domains (cognitive, technical, and behavioral, P < 0.0001, 95% confidence interval [CI], 1.2-1.5; P < 0.0001, 95% CI, 2.2-2.6; and P = 0.002, 95% CI, 1.7-2.1, respectively) with an increase in knowledge scores (P < 0.001; 95% CI, 1.4-2.5). These findings were most significant in participants with less ECMO experience. There were also significant reductions in times to critical actions in three of the four scored simulation scenarios. The results demonstrated participants' satisfaction with most course aspects, with more than 95% expressing that courses met their educational goals.
Conclusion: We developed and tested a structured ECMO course assessment tool, demonstrating participants' self-reported benefit as well as improvement in psychomotor skill acquisition, course satisfaction, and educational benefits. Course evaluation is feasible and potentially provides important information to improve ECMO courses. Future steps could include national implementation, addition of questions targeting clinical decision making to further assess knowledge gain, and multilanguage translation for implementation in international courses.

Extracorporeal membrane oxygenation (ECMO) has become a mainstay therapy for life-threatening cardiopulmonary failure. Its use has increased significantly in the past two decades by almost 10-fold in adults and 4-fold in children (1, 2). The 2009 influenza pandemic and the current coronavirus disease (COVID-19) pandemic have seen a significant increase in ECMO use, primarily in adults (3–8), with a growing number of centers reporting to the Extracorporeal Life Support Organization (ELSO), from 184 centers in 2010 to 492 in 2020 (2). Since its founding in 1989, and with five international chapters covering North America, Asia-Pacific, Europe, Latin America, and Southwest Asia and Africa, ELSO has been recognized as the international body providing guidance on education, guidelines, original research, and publications on ECMO support (9, 10).

With the increase in ECMO use, there has been a parallel surge in ECMO education programs worldwide, reflecting recognition of the high cost, relatively low volume, intense resource use, and high-risk nature of ECMO support, mandating that clinicians be properly trained (11, 12). Despite this growth, however, there is a lack of consensus on the curriculum and structure of these education programs (13). As such, ELSO created a dedicated task force, the Extracorporeal Life Support Organization education task force (ELSOed), charged with defining international educational needs and identifying opportunities for standardization and international collaboration. An ELSOed position statement in 2020 outlined educational priorities (10) to include standardizing an ECMO curriculum for delivery at ECMO courses, to standardize a process for ECMO course evaluation, and to outline mechanisms to evaluate the educational benefits of ECMO courses worldwide. Although ECMO courses are provided internationally through several venues, ELSO-provided and ELSO-endorsed courses are the only internationally available courses known to follow a similar structure for both didactic and simulation components (14, 15).

The Kirkpatrick evaluation model is a well-defined method for objectively measuring the effectiveness of training programs (16). This model outlines the four-level evaluation framework: reaction, learning, behavior change, and organizational performance. Reaction evaluates the trainees’ perceptions about the program and the usefulness of the material to their work; learning gauges the participants’ developed expertise and knowledge and is commonly assessed before (“pretest”) and after (“posttest”) training. Behavior change and organizational performance assess the differences in participants’ behavior and impact on their work after completing the program, thereby requiring long-term follow-up.

The goal of the present study was to develop a structured online course assessment tool based on the Kirkpatrick evaluation model to assess the educational benefits of ECMO courses using self-assessment, knowledge assessment, and psychomotor skill evaluation forms in a simulation setting.

Methods

Study Design

Using online forms, we performed a prospective assessment of U.S.-based adult ECMO courses that are ELSO run or ELSO endorsed. The courses were offered at academic institutions. ELSO-run courses are provided directly through ELSO, whereas ELSO-endorsed courses are provided through centers whose ECMO simulation-based courses have undergone review by ELSO to ensure they follow the same structure. Although there are other non-ELSO courses available worldwide, these are not guaranteed to follow a standardized format. In addition, ELSO is internationally recognized to be the largest body to oversee clinical and educational ECMO-related guidelines worldwide. The study was reviewed by the institutional review board at Washington University in St. Louis and was granted a waiver of consent.

ELSO and ELSO-endorsed ECMO courses have a standard structure for 25 hours over 3.5 days with 14.5 hours of simulation. Participants are provided with a copy of the ELSO Red Book and the ECMO Specialist Training Manual on arrival. Didactics consist of structured lectures covering ECMO basics, circuit components, physiology, cannulation, complications, and literature review, as well as interactive case studies provided by physicians, nurses, and perfusionists with ECMO expertise. Simulation makes up more than 50% of the course duration and is immersive, with two or three instructors per every six participants. Scenarios are created with high fidelity, incorporating an ECMO circuit connected to a mannequin via a simulated dual-site cannulation (17). In addition to realistic circuit variables, this setup incorporates patient vital signs, ventilator settings, laboratory test results, and radiologic images. Simulation scenarios are followed by structured debriefing sessions according to the three-phase debriefing technique (18).

Study Subjects

All course participants were eligible for this study and received individual links to pre- and postcourse assessment forms hosted on the Google Forms platform. Completed form data were centrally collected, deidentified, and analyzed.

Assessment Forms

Pre- and postcourse forms were developed by representatives from the ELSOed course work group (A.S.S., E.C., R.K., and B.Z.) following the first two levels of the Kirkpatrick evaluation model: reaction and learning. In July 2018, a literature review was performed to identify previous ECMO simulation course surveys. This review identified nine assessment tools focused primarily on specific aspects of ECMO courses or limited members of the ECMO team, with none including all representatives of the interprofessional ECMO team, and covering the impact of the courses on the participants’ confidence levels, perceived educational benefits, and impact on ECMO-related skills (17, 19–30). During the 7-month period from July 2018 to February 2019, through virtual meetings, the pre- and postcourse surveys were developed and modified following commonly recognized steps of medical education survey development (31–33). Leading, double-barreled, vague, or negatively worded questions were intentionally avoided during survey development. The survey questions were focused on ECMO-relevant clinical aspects using clear and familiar terms and avoiding incomplete or overlapping answers (32). Feedback from pretesting of the surveys in March 2019 was concerning for their length, so a modified version was developed and subsequently used for the study course assessments (see Supplementary Digital Content 1 section in the data supplement).

Course administrators were blinded to the form results. Precourse forms included 1) participant demographics; 2) participant self-assessment regarding ECMO management; and 3) knowledge assessment. Postcourse forms included 1) participant self-assessment regarding ECMO management; 2) knowledge assessment; 3) skills assessment during the course; and 4) overall course evaluation, including lectures, faculty, facilities, course logistics, perceived educational benefits, and perceived bias. The self-assessment forms covered cognitive, technical, and behavioral aspects of ECMO management. Assessment categories were defined to cover the basic knowledge and management skills identified as learning objectives for the courses. All questions were scored on a 5-point Likert scale (from “least confident” to “most confident” for self-assessment and from “poor” to “excellent” for course assessment questions). Knowledge assessment forms were composed of 15 multiple-choice simple-recall questions on ECMO physiology, management, and circuit setup. There were no repeat questions between pre- and postcourse knowledge assessments. The questions were developed by pooling seven question sets from currently provided international ECMO courses. These were then ranked by question clarity, ease, and relevance to the ELSOed-identified learning objectives, first by three international ECMO educational expert members of ELSOed, then by the survey development group (A.S.S., E.C., R.K., and B.Z.). The top 90 questions were selected for the final question database to avoid question repetition.

Four ECMO emergency simulation scenarios were repeated during the courses, initially as a standalone simulation and then, on the final day of the course, as part of a megasimulation in which several scenarios were presented back to back as part of a clinical scenario. For each of these scenarios, actions critical to successful resolution of the scenario were identified for timing. As the multidisciplinary teams participated in these scenarios, times to each of these critical actions were recorded for the group as a whole. Time to action during simulation was recorded by a core group of four senior educators (including T.M. and B.Z. and two nurse ECMO specialists with more than 5 years of ECMO simulation experience) within the simulation group to limit interoperator variability. Groups were maintained throughout the course to reduce variability arising from group dynamics. The scenarios included breach of circuit, recirculation on venovenous (VV) ECMO, ventricular tachycardia (VT) on VV ECMO with conversion to venoarterial (VA) ECMO, and air entrainment into the ECMO circuit.

Statistical Analysis

Statistical analysis was conducted in R (https://www.r-project.org), and figures were produced using the ggplot2 package (https://ggplot2.tidyverse.org). Categorical variables are presented as count (percent), and quantitative variables are presented as the median and interquartile range (IQR), unless otherwise stated. The Wilcoxon signed rank sum test was used to compare pre- and postcourse results. Pre- and postcourse intergroup analyses were performed using Kruskal-Wallis one-way analysis of variance tests. These subgroup analyses were solely exploratory. A P value <0.05 was considered significant. Cohen’s d was used to assess effect size based on differences between means, with d of 0.2 representing small, 0.5 representing medium, and 0.8 representing large effect sizes.

Results

Between March 2019 and February 2020, 211 participants in five ELSO and ELSO-endorsed courses participated in the study. Of these, 107 (51%) completed both pre- and postcourse assessment forms, with 97 participants (91%) completing the pre- and postcourse knowledge assessment forms (Figure 1). The median duration between precourse form completion and course start day was 2.61 days (IQR, 1.05–5.7), with 1.05 days (0.2–4.95) between the course end day and postcourse form completion. Physician intensivists made up more than half of participants (53%), followed by nurses (11%), with the majority working in academic hospitals (51%). The majority (62%) of respondents had more than 5 years of clinical experience, although half (50%) had less than 1 year of ECMO experience. Table 1 displays the participant demographics and ECMO practices provided at the participants’ home institutions.

Figure 1.

Participant enrollment diagram. CONSORT (Consolidated Standards of Reporting Trials) diagram of the participant enrollment process from five ELSO-endorsed courses from March 2019 through February 2020 (four general ECMO and one VV course) with 211 participants, 156 completed precourse demographics, 100 completed precourse knowledge tests, and 138 precourse self-assessments. For the postcourse assessments, 102 completed the knowledge tests, and 107 completed the self-assessments. In total, 107 participants completed both pre- and postcourse self-assessments, and 97 completed pre- and postcourse knowledge tests. ECMO = extracorporeal membrane oxygenation; ELSO = extracorporeal life support organization; VV = venovenous.

Table 1.

Participant demographics

Parameter	n (%)
Sex
Male	62 (58)
Female	45 (42)
Age group
21–30 yr	5 (4.7)
31–40 yr	57 (53)
41–50 yr	29 (27)
51–60 yr	9 (8.4)
>60 yr	7 (6.5)
Specialty
Physician – intensivist	57 (53)
Physician – anesthesiologist	6 (5.6)
Physician – pulmonologist	6 (5.6)
Physician – cardiac surgeon	2 (1.9)
Physician – cardiologist	2 (1.9)
Physician – trainee	2 (1.9)
Physician – transplant surgeon	3 (2.8)
Nurse	12 (11)
Nurse practitioner	10 (9.3)
Respiratory therapist	5 (4.7)
Other	2 (1.9)
Hospital setting
Community hospital	46 (43)
Government hospital	4 (3.7)
Academic hospital	55 (51)
Other	2 (1.9)
Age group managed
Pediatric	5 (4.7)
Neonatal	2 (1.9)
Adult	96 (90)
Pediatric and adult	4 (3.7)
Years in practice
<5 yr	41 (38)
>5 yr	66 (62)
ECMO experience
<1 yr	54 (50)
>1 yr	53 (50)
ECMO duration at local hospital
<6 mo	2 (1.9)
1–2 yr	18 (17)
2–5 yr	41 (38)
>5 yr	32 (30)
Do not have ECMO and are not planning to	1 (0.9)
Do not have ECMO but are planning to	13 (12)
ECMO modes at local hospital
None	8 (7.5)
VV	8 (7.5)
VA	1 (0.9)
Both VV and VA	90 (84)
ECMO capability at local hospital
No ability to cannulate or refer for ECMO	4 (3.7)
Can cannulate patients onto ECMO but then refer to outside institutions	9 (8.4)
Can cannulate and care for ECMO patients	94 (88)

Definition of abbreviations: ECMO = extracorporeal membrane oxygenation; VA = venoarterial; VV = venovenous.

Self-Assessment

Participants reported improved self-confidence with management of patients receiving ECMO. This improvement was consistent across the three domains: cognitive (precourse, mean ± standard deviation [SD], 3.16 ± 0.88; postcourse, 4.5 ± 0.43; P < 0.001; 95% confidence interval [CI], 1.2–1.5), technical (precourse, mean ± SD, 1.7 ± 0.85; postcourse, 4 ± 0.76; P < 0.001; 95% CI, 2.2–2.6), and behavioral (precourse, mean ± SD, 2.4 ± 1.1; postcourse, 4.2 ± 0.78; P < 0.001; 95% CI, 1.7–2.1) (Figure 2). In subgroup analysis, significant improvement in self-assessment scores persisted across specialties and experience levels (Supplementary Digital Content 2 section).

Figure 2.

Pre- and postcourse self-assessment results. Box-and-whisker plots of the pre- and postcourse self-assessment results. There was a significant increase in the reported postcourse self-assessments across all three tested categories: cognitive, technical, and behavioral aspects of ECMO management (Wilcoxon signed rank sum test). CI = 95% confidence interval; Post = postcourse; Pre = precourse.

Knowledge Assessment

There was a significant improvement in knowledge assessment scores (10 [9-12] vs. 12 [11-13] for pre- vs. postcourse; P < 0.001; 95% CI, 1.4–2.5). There were no significant differences in either the pre- or postcourse test scores among participants’ specialties, age groups, or clinical experience levels. In subgroup analysis by specialty and ECMO and clinical experience, the significant improvement only persisted for participants with less than 1 year of ECMO experience (P = 0.001; 95% CI, 1.5–2.9). In Cohen’s d calculation, there was a moderate effect size (⩾0.5) in all subgroup analyses, with larger effect sizes (⩾0.7) among nonintensivists, nonphysicians, and participants with less than 1 year of ECMO experience (Figure 3).

Figure 3.

Pre- and postcourse knowledge assessment tests. (A) Box-and-whisker plots of the pre- and postcourse knowledge assessment tests. There was no significant difference in either pre- or postcourse test results between the participant subgroups (Kruskal-Wallis one-way analysis of variance). There was a significant increase in the postcourse scores for all participants, in addition to the subgroups of intensivists, nonintensivists, physicians, and nonphysicians (Wilcoxon signed rank sum test), with a moderate to large effect size by Cohen’s d calculation for all groups. (B) Box-and-whisker plots of the pre- and postcourse knowledge assessment tests, by years of clinical experience. There was no significant difference in either pre- or postcourse test results in subgroups by clinical experience. There was a significant increase in the postcourse test results for both groups of participants with clinical experience less or more than 5 years (Wilcoxon signed rank sum test). There was a higher effect size in participants with less than 5 years of clinical experience than in those with more than 5 years of clinical experience (Cohen’s d test). (C) Box-and-whisker plot of the pre- and post-knowledge assessment test scores by groups according to ECMO experience. There was no significant difference in the pre- or postcourse test results, but there was a significant increase in the postcourse results for both groups, regardless of previous ECMO experience (Wilcoxon signed rank sum test). We found a larger effect size in the participants with less than 1 year of ECMO experience versus those with more than 1 year of ECMO experience (Cohen’s d). CI = 95% confidence interval; Post = postcourse; Pre = precourse.

Psychomotor Skill Assessment

Times to critical actions were recorded during breach of circuit, recirculation on VV ECMO, VT on VV ECMO with conversion to VA ECMO, and air entrainment into the ECMO circuit scenarios. For the breach-of-circuit scenario, there were significant decreases in all times to critical actions: recognition of pathology, clamping the ECMO circuit, and resumption of ECMO support (pre- vs. postcourse mean ± SD, 88 ± 152 vs. 39 ± 56 s; P = 0.05; 310 ± 228 vs. 171 ± 215 s; P = 0.02; 393 ± 238 vs. 246 ± 214 s; P = 0.008, respectively). There were similar results in the recirculation scenario tasks of recognition of pathology, decreasing blood flow, and cannula imaging (pre- vs. postcourse mean ± SD, 264 ± 167 vs. 107 ± 91 s; P < 0.001; 440 ± 181 vs. 202 ± 146 s; P = 0.04; 314 ± 192 vs. 160 ± 109 s; P < 0.001, respectively) and in the VT on VV ECMO tasks of initiating cardiopulmonary resuscitation, call for VA ECMO, and convert to VA ECMO (pre- vs. postcourse mean ± SD, 82 ± 151 vs. 24 ± 49 s; P = 0.002; 278 ± 150 vs. 101 ± 62 s; P < 0.001; 563 ± 59 vs. 467 ± 134 s; P = 0.05, respectively). For the air entrainment scenario, there was a significant decrease in time to task completion for the recognition of pathology (pre- vs. postcourse mean ± SD, 38 ± 75 vs. 9 ± 9 s; P = 0.02), but there were no significant differences in clamping the ECMO circuit, deairing the circuit, or resumption of ECMO support (pre- vs. postcourse mean ± SD, 74 ± 90 vs. 56 ± 64 s; P = 0.4; 98 ± 121 vs. 132 ± 189 s; P = 0.6; 470 ± 178 vs. 341 ± 163 s; P = 0.07, respectively) (Supplementary Digital Content 3 section). Table 2 displays the pre- and postcourse times to critical actions in addition to the postcourse changes for all the recorded tasks.

Table 2.

Time to critical actions in simulation scenarios, in seconds

Critical Action	Precourse Time (s), Median [IQR]	Postcourse Time (s), Median [IQR]	Delta Time (s), Median [IQR]	P Value*	95% CI
Breach of circuit
Recognize pathology	42 [9, 79]	14 [3, 52]	0 [-59, 5]	0.05	−89, 0.49
Clamp ECMO circuit	260 [110, 493]	111 [42, 206]	−15 [−302, 12]	0.02	−335, −20
Resume ECMO	287 [199, 600]	172 [109, 310]	−80 [−328, 35]	0.008	−328, −38.5
Recirculation on VV ECMO
Recognize pathology	202 [139, 360]	92.5 [45, 156]	−80 [−289, 0]	<0.001	−227, −65
Decrease ECMO flow	496 [274, 600]	169 [118, 254]	0 [−220, 121]	0.04	−369, −21
Cannula imaging	290 [154, 491]	146 [82, 194]	−114 [−299, 16]	<0.001	−265.5, −67.5
VT on VV ECMO
Initiate CPR	30 [15, 68]	10 [7, 16]	−16 [−56, 0]	0.04	−55.5, −10
Call for VA ECMO	240 [178, 382]	85 [54, 133]	−140 [−229, 0]	<0.001	−240, −135
Convert to VA ECMO	600 [527, 600]	467 [367, 600]	0 [0, 422]	0.05	−289, 19
Air entrainment
Recognize pathology	12 [5, 30]	5 [3, 12]	−2 [−17, 2]	0.02	−24.5, −0.5
Clamp ECMO circuit	30 [18, 103]	27.5 [15, 86]	0 [−48, 23]	0.4	−59.5, 21.5
Vent ECMO circuit	49 [30, 152]	39 [26, 176]	0 [−30, 34]	0.6	−35, 108
Resume ECMO	541 [256, 600]	298 [204, 502]	0 [−158, 168]	0.07	−345, 100

Definition of abbreviations: CPR = cardiopulmonary resuscitation; ECMO = extracorporeal membrane oxygenation; IQR = interquartile range; VA = venoarterial; VT = ventricular tachycardia; VV = venovenous.

Paired t test.

Course Evaluation

On a 5-point Likert scale, there was a high degree of satisfaction with several aspects of these courses, including lectures, faculty preparedness and knowledge, and overall course impression (median and IQR, 5 [5-5], 5 [5-5], and 5 [5-5], respectively). There was variability in the assessment of simulations and course logistics (median and IQR, 4.8 [3-5] and 5 [4.8–5], respectively). Of the respondents, 96% expressed that the training fulfilled their educational goals, and 97% reported that the courses provided information that could be implemented in clinical practice. In addition, more than 95% of participants stated that the delivered information was balanced and free of bias and that the teaching faculty presented statements of potential conflicts of interest (Supplementary Digital Content 4 section).

Discussion

This study presents the first implementation of a structured assessment tool for simulation-based ECMO courses evaluating changes in participants’ self-reported confidence in managing patients receiving ECMO, assessment of their knowledge gain, change in psychomotor skills, course evaluation, and evaluation of perceived educational benefits. We present the preliminary results from five U.S.-based courses. Our findings show 1) improved participant self-confidence in managing patients receiving ECMO across all specialties and experience levels; 2) improved knowledge in participants with less than 1 year of ECMO experience; and 3) significant reductions in times to complete critical actions during simulated ECMO emergency scenarios. Although course assessment scores were overall very positive, there was more variability in the assessment of the simulation and, to a lesser extent, the courses’ logistics. The majority of the respondents expressed that training in the courses fulfilled their educational goals and that the information learned during the courses would be implemented in their clinical practice.

To objectively assess the educational benefits of ECMO courses, we followed the commonly used Kirkpatrick model of training course evaluation by prospectively evaluating course participants both before and after the courses. We selected levels 1 and 2 of the Kirkpatrick model because they are the short-term levels. Our study population was heterogeneous, with a variety of disciplines and specialties represented as well as varied clinical and ECMO experience levels. This is of pragmatic value, representing the typical multidisciplinary team involved in managing patients receiving ECMO.

Our study showed the ability of the assessment tool to identify significant improvement in participant self-confidence with ECMO management. These findings are consistent with previously reported improvements in self-confidence in participants of 1-day simulation-based ECMO courses for intensive care physicians, nurses, thoracic surgery residents, and ECMO specialists (22, 23, 28). Others have reported improvement in confidence levels for a group of ECMO-novice practitioners after simulation training (25). In addition to including a significantly larger cohort at five different courses nationwide, our results complement these studies by demonstrating improvement across specialties and clinical or ECMO experience levels. In addition, the tool presented in this study assesses improvements across the three categories: cognitive, technical, and behavioral. Such a structured assessment is of special importance in ECMO education, where the multidisciplinary team’s confidence in both technical and nontechnical aspects is essential to effectively provide this life-sustaining therapy (34).

We found the most significant improvement in knowledge assessment scores in the participants with less ECMO experience in subgroup analysis. These findings likely represent the small number of participants tested in this pilot study and highlight the need for future larger studies to evaluate the ability of the knowledge assessments to elucidate participants’ knowledge gain from the ECMO courses. The significant improvement in participants with the least ECMO experience is one of the goals of the standardized curricular approach stated by ELSOed in the 2020 position statement. These findings are consistent with other studies showing significant knowledge gain in ECMO-naive critical care fellows after simulation-based training (30). Similar knowledge gain after 1-day and 2-day high-fidelity ECMO simulation courses has also been shown with other course evaluations (27, 28). Despite the significant knowledge gain in our participants, there was a noticeably narrow range for both pre- and postcourse scores. We supplemented the knowledge evaluation with effect size analysis that demonstrated a more significant effect in participants with less ECMO clinical decision-making experience, including nonphysicians, nonintensivists, and those with less clinical and ECMO experience.

For the simulation assessments, there was a significant decrease in the time to critical action completion in all tasks for three of the four evaluated scenarios. These findings are consistent with previous reports (30). Zakhary and colleagues found that simulation-based training improved time to completion of critical actions for novice ECMO practitioners. More important, the authors found that these results persisted in long-term follow-up (30). Similarly, it has also been shown that there was a decrease in the number of technical errors after simulation, even though reduction in reaction time did not reach statistical significance (35), whereas Burkhart and colleagues reported a significant improvement in time to critical action after simulation training (23).

Previous studies of ECMO courses have been limited, with the majority focusing on the feasibility and benefit of high-fidelity simulation (20, 21, 23, 24, 26, 30). A 2020 publication highlighted an abbreviated ECMO course for nonsurgical clinicians (28) and reported an improvement in participants’ knowledge and confidence after the 1-day ECMO course. Their study was limited to participants with no previous ECMO experience and focused on the initiation of ECMO with no training in the subsequent management of patients or in weaning of ECMO support. The nature of the knowledge tests and the confidence assessments was not clearly delineated in the study. The present study developed and used a tool for assessing the impact of ECMO courses with participants of multidisciplinary representation and of varying experience levels in addition to providing a structured assessment of postcourse knowledge, confidence, and skill gain.

Strengths and Limitations

This study has several strengths. First, it provides a structured evaluation for simulation-based ECMO courses. Second, the study reflects the applicability of the assessment model across a large multidisciplinary cohort that represents the different specialties involved in caring for patients receiving ECMO. Finally, we outline a structured approach to assess the short-term educational benefits of simulation-based ECMO courses that can be applied on both national and international levels.

The present study should be interpreted within the context of certain limitations. First, only 50% of participants completed all the pre- and postcourse forms. This is likely due to the fact that completion was voluntary and is important to take into account when gauging the generalizability of the results. Our results, in addition to the constructive deidentified feedback to course organizers collected on these forms, provide an incentive for course organizers to include assessment form completion as a mandatory part of future courses. Second, the study was limited to U.S.-based courses. We are currently translating the assessment forms to allow pilot implementation at international courses. Third, as mentioned above, the knowledge questions were limited to rapid-recall questions, which likely limits our ability to detect significant differences in knowledge acquisition and critical application; the incorporation of multistep logic questions may allow a more in-depth evaluation of participants’ knowledge gain. Furthermore, evaluation on Kirkpatrick levels 1 and 2 has limited ability to extrapolate findings to real-life performance and patient outcome improvements. In addition, it is important to acknowledge the limitation of self-confidence as a surrogate for competence, as extensively studied in medical education. This highlights the need for long-term follow-up of participants to evaluate the impact of the ECMO course on their daily work practices and allowing assessment of levels 3 and 4 of the Kirkpatrick model. Last, this study shows the results of piloting the assessment tool in ECMO courses following the ELSO-recommended structure, either ELSO run or ELSO endorsed. However, broad implementation of the present tool across all ECMO simulation courses would require assessing the tool’s performance in ECMO courses not following the ELSO-recommended structure.

Conclusions

This study demonstrates the successful development of a comprehensive ECMO course assessment tool. This tool demonstrated improvement in participants’ self-confidence with management of patients receiving ECMO in addition to knowledge acquisition and time to critical action in simulated scenarios after standardized simulation-based ECMO courses. Future directions include national and international implementation and evaluation of long-term effects of ECMO courses on participants’ work practice.