A learning curve of a novel multimodal endotracheal intubation assistant device for novices in a simulated airway: a prospective manikin trial with cumulative sum method.

Background: Awake fiberoptic intubation is conventionally performed in anticipated difficult airways. However, obstruction by secretions and sputum makes it challenging for novices. A prototype of a novel multimodal endotracheal intubation assistant device (MEIAD) was developed for an indication of airway according to end-tidal carbon dioxide (ETCO2) and image. At the tip, 4 sampling tubes collected ETCO2 concentration. The airway direction is located according to an advanced algorithm based on 4 directions' concentrations. It assists awake intubation, especially with unclear view field. The objective was to analyze the learning curve of MEIAD for novices on a manikin by cumulative sum method (CUSUM) and evaluate the utility.
Methods: A total of 16 novice residents with less than 2-year clinical experience were enrolled. After instruction, each individual exercised 40 insertions with MEIAD on a difficult airway simulation. Insertion success (defined as a visualization of the carina within 120 seconds), insertion time (the time from when the guiding scope entered the nasal cavity to the carina was visible), and self-confidence score (subjective score with a numerical rating scale from 0 to 10) were recorded. The acceptable and unacceptable failure rates of CUSUM were set as 15% and 30%, respectively. The exercises were divided into 2 phases (phase 1: 1-20, phase 2: 21-40) for further evaluation. All continuous data were expressed by median (IQR, interquartile ranges) and analyzed using Mann-Whitney test. All categorical variables were expressed as percentages and compared by the χ2 test.
Results: Among the 16 residents, 15 were able to cross the lower decision boundary in an average of 21.27±9.51 attempts using the novel device. The insertion time [24.0 (17.0-42.0) vs. 17.5 (14.0-28.0) seconds, P<0.001] and success rate (88.4% vs. 97.5%, P<0.001) were improved with increased experience. The confidence score was significantly improved from 2.5 (1.3-4.0) to 7.0 (7.0-8.0). Conclusions: MEIAD showed a satisfactory learning curve and efficacy on the manikin for novices. However, as a small exploratory manikin trial, the results cannot be replicated in clinical practice. MEIAD is expected to be further improved and potential to be an alternative device for difficult airways. 2022 Translational Pediatrics. All rights reserved.

Introduction

Awake fiberoptic intubation is regularly performed in patients with anticipated difficult airways (1). However, many challenges are still associated with endotracheal intubation under the guidance of fiberoptic bronchoscopy. The visual field is limited and easily obstructed by secretions and blood in the airways (2). At the same time, learning to perform awake fiberoptic intubation is difficult and time-consuming for novice anesthesiologists (3). It is considered that more than 25 operations are needed for an anesthesiologist to be proficient with fiberoptic bronchoscopy (2). End-tidal carbon dioxide (ETCO2) is considered to facilitate endotracheal intubation with spontaneous breath (4,5). Previous studies have shown that ETCO2 monitoring assisted orientation and improved the safety of awake blind nasotracheal intubation (6,7) and reduced intubation time for inexperienced anesthesiologists (8). At present, there are still no intubation devices that integrate synchronous ETCO2 monitoring with visualization technology for clinical practice.

The multimodal endotracheal intubation assistant device (MEIAD, patent number: ZL202030298413.9) is a novel flexible intubation scope independently developed by the anesthesiology department at Shanghai Ninth People’s Hospital of Shanghai Jiao Tong University. The MEIAD prototype () consisted of an operating handle, a guiding scope, and a sensor system. At the tip of the steerable, flexible scope, it was equipped with a camera and 4 gas sampling tubes (), which collected ETCO2 concentration from 4 directions based on the principle of infrared light absorption and used an advanced algorithm to locate the airway direction according to differences in ETCO2 concentration. The direction of the airway based on ETCO2 concentration was displayed on the screen simultaneously with the endoscopic image to assist operators in identifying airways during awake intubation, especially when the field of view is unclear.

Figure 1

MEIAD diagram. (A) MEIAD prototype. (B) The tip of the guiding scope. MEIAD, multimodal endotracheal intubation assistant device.

The acquisition of the skill is also an important part of evaluating the utility of a device. The cumulative sum control chart (CUSUM) is a statistical analysis technique that can be used to evaluate learning curves (9). In medicine, CUSUM analysis can be used to monitor clinical performance and quality and to assess the time for operators to achieve expected levels (10). CUSUM is used in the learning analysis of many anesthesia practices, including intraspinal anesthesia, endotracheal intubation, and nerve blocks under the guidance of ultrasound (11-14). CUSUM is an efficient method to judge the number of attempts to achieve proficiency.

The objective of this study was to analyze the learning curve of MEIAD for inexperienced anesthesiology residents on a manikin by CUSUM analysis and evaluate the utility of the novel device. We present the following article in accordance with the STROBE reporting checklist (available at https://tp.amegroups.com/article/view/10.21037/tp-22-405/rc).

Methods

Participants

We conducted the prospective, observational, manikin trial from October 2020 to November 2020. The Clinical Research Ethics Committee of our institution deemed this study to be exempt from ethical review as it involved only a manikin and simulation training, which was considered a regular part of medical education. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

A total of 16 anesthesiology residents with 1–2 years clinical experience in our institution were enrolled. All participants had experience in nasal/orotracheal intubation with direct/video laryngoscopy. We excluded residents who had experience in fiberoptic bronchoscopy from our study. None of the residents had prior experience with MEIAD. All participants were voluntary and provided informed consent. The data collection was anonymous. The residents were named sequentially from A to P.

To simulate a difficult airway, we used a cervical collar to limit the mouth opening and neck extension of the manikin (BZ-51, Shanghai Baijiao, China). In addition, we connected the simulated lungs of the manikin to a CO2 accumulator, which generated CO2 when pressed to simulate exhaled CO2 during awake intubation ().

Figure 2

The difficult airway simulation. A cervical collar limited the mouth opening and neck extension of the manikin. The simulated lungs were connected to a CO2 accumulator. CO2, carbon dioxide.

Protocol and measurements

All participants received instruction and demonstration by an experienced anesthesiologist, including basic knowledge of airway anatomy and endotracheal intubation, an explanation of MEIAD, manipulation of the device, and demonstration on the manikin. Each resident then completed 40 exercises of nasotracheal insertion using MEIAD on the difficult airway manikin.

The manikin was positioned neutrally and the residents stood at the head of the manikin. After lubricating the guiding scope, the residents inserted the scope into the nasal cavity and slowly introduced the scope forward to find the glottis. An assistant pressed the CO2 accumulator at a frequency of 10 times per minute to simulate the expiration of CO2 in the airway. The residents adjusted the scope direction according to the indication of the airway analyzed by ETCO2 and the visualized image to see the glottis and then advanced the tip of the scope through the glottis until the carina was visible (). The correct MEIAD positions were confirmed by the instructor viewing the image of the carina under the scope. In this study, the placement of an endotracheal tube was not required. We considered the insertion of the scope into the airway as the endpoint. In addition, the confidence in difficult airway management was scored subjectively by residents with a 11-point numerical rating scale (NRS) scale (ranging from 0 to 10, 0= very unconfident, 10= very confident) before and after 40 consecutive attempts.

Figure 3

The MEIAD screen showing the scope image and the indication of airway based on ETCO2. (A) The indication of airways analyzed by ETCO2 in different directions. (B) A successful insertion with the image of the carina. MEIAD, multimodal endotracheal intubation assistant device; ETCO2, end-tidal carbon dioxide.

The main data collected included success or failure and insertion time of each operation and the individual confidence score. Successful insertion of MEIAD was defined as correct placement of the device into the airway within 120 seconds. If the insertion time exceeded 120 seconds or if the scope entered into the esophagus, the operation was recorded as a failure. The insertion time was defined as the time from when the guiding scope entered the nasal cavity to when the tracheal carina was visible. The primary outcome was the learning curve of MEIAD established by CUSUM analysis. The secondary outcomes were an evaluation of the utility of MEIAD and the degree of improvement in confidence after simulation training.

Statistical analyses

To construct the learning curve according to CUSUM analysis, 4 basic variables were required: acceptable failure rate (p0), unacceptable failure rate (p1), standard type I error (α), and standard type II error (β) (15). Detailed calculations in CUSUM analysis based on the 4 variables are described in . The calculation of the CUSUM value started from zero. If a successful operation was performed, a value of S was subtracted from the previous CUSUM score. Conversely, a value of 1-S was added to the previous CUSUM score. Therefore, a negative trend in CUSUM values indicated a good trend for more successful attempts. When the curve was below the acceptable failure rate boundary h0, there was no significant difference between the actual and acceptable failure rate, and competence was considered obtained. If the CUSUM score was over the unacceptable failure rate boundary h1, the operator did not master the practice. When the curve score lines were between h1 and h0, it was not clear enough to judge the competence of operation.

Table 1

Calculations of the CUSUM analysis

Variables	Equations	Numeric values
α (type I error)	False positive	0.1
β (type II error)	False negative	0.1
p0	Acceptable failure	0.15
p1	Unacceptable failure	0.3
a	ln((1 − β)/α)	2.19722457
b	ln((1 − α)/β)	2.19722457
P	ln(p1/p0)	0.693147181
Q	ln((1 − p0)/(1 − p1))	0.194156014
S (CUSUM)	Q/(P + Q)	0.218815863
h0 (Lower decision boundary)	−b/(P + Q)	−2.476295126
h1 (Upper decision boundary)	a/(P + Q)	2.476295126
Number of cases for p0	[(h0(1 − α) − αh1)/(s − p0)]	36
Number of cases for p1	[(h1(1 − β) − βh0)/(p1 − s)]	31

CUSUM, cumulative sum method.

As an exploratory study, 16 residents were included based on previous studies of learning curves and simulation training (16,17). The value of α and β were set conventionally as 0.1. In this study, we set p0 as 15% and p1 as 30% (p1 is usually twice the value of p0) (17). Based on the calculation, the number of operations required to achieve the acceptable failure rate p0 and unacceptable failure rate p1 was 36 and 31, respectively. Therefore, all residents attempted 40 insertions in the study.

The CUSUM chart of the learning curve was constructed by Microsoft Excel (Microsoft Office 2016) and statistical analyses were performed by SPSS 22.0 (IBM, Armonk, NY, USA). Continuous data are expressed by median (IQR, interquartile ranges), and categorical variables are expressed as percentages. We grouped the 40 insertions for every resident into 2 groups of 20 attempts each (phase 1: 1–20, phase 2: 21–40) to evaluate the learning effect of different periods. We used the Mann-Whitney test to compare insertion time and confidence score between the 2 groups because the data did not conform to a normal distribution. The success rate between periods was compared by the χ2 test. All tests were two-sided with a significance level of 0.05.

Results

All of the 16 novice residents (male/female: 8/8), aged 24.44±1.55, completed 40 trials of MEIAD (640 insertions in total), and all attempts were recorded completely. The CUSUM curves of the 16 novice residents using MEIAD are shown in . According to the CUSUM analysis, 15 of 16 novice residents crossed the lower decision boundary h0 after 21.27±9.51 attempts. The 15 residents were considered proficient in endotracheal intubation with MEIAD on the difficult airway simulation to achieve an acceptable failure rate of 15%. Resident C had a CUSUM value between h0 and h1 after 40 attempts of MEIAD and was considered to not have obtained the skill at the acceptable level within 40 attempts.

Figure 4

CUSUM of 16 residents’ insertions. CUSUM, cumulative sum method.

The 40 insertions were stratified into 2 groups of 20 insertions each. The insertion success and insertion time of each group are shown in . The insertion time of the 2 groups was 24.0 (17.0–42.0) seconds vs. 17.5 (14.0–28.0) seconds, respectively (P<0.001). In terms of success rate, the difference between the 2 groups was also statistically significant (88.4% vs. 97.5%, respectively, P<0.001). As the number of attempts to master the use of MEIAD was approximately 22, it can be assumed that the last 20 attempts were the plateau of the learning curve. During the plateau period, the insertion time was 17.5 (14.0–28.0) seconds and the success rate was 97.5%.

Table 2

Operation results of 2 phases in training MEIAD

Outcomes	Phase 1	Phase 2	P value
Insertion time(s)	24.0 (17.0–42.0)	17.5 (14.0–28.0)	<0.001
Number of successes	283 (88.4)	312 (97.5)	<0.001

Data are shown as median (IQR) or number (%). MEIAD, multimodal endotracheal intubation assistant device; IQR, interquartile ranges.

In addition, after training on the simulation, the NRS score of the residents’ confidence in difficult airway management was significantly improved from 2.5 (1.3–4.0) to 7.0 (7.0–8.0).

Discussion

Our study showed that inexperienced residents achieved an accepted success rate of 15% using MEIAD in a difficult airway simulation after about 22 attempts. During the plateau period, the insertion time was 17.5 (14.0–28.0) seconds and the success rate was 97.5%.

With an incidence of 0.5–12.8%, the difficult airway is a major challenge for anesthesiologists during perioperative airway management (18). For anticipated difficult airway, awake endotracheal intubation under the guidance of fiberoptic bronchoscope is recommended to guarantee patients’ security. Some relevant studies have shown that the success rate of awake fiberoptic bronchoscope intubation was 78–100% (19-22). Although fiberoptic bronchoscopy is advised for difficult airways, it takes considerable training to obtain the skill, particularly for novice residents (23,24). In addition, fiberoptic bronchoscope is easily affected by sputum and secretions from the oropharynx or nasopharynx, making the glottis difficult to identify. The novel device added indication of airway based on the algorithm of ETCO2 to the endoscope in order to play an auxiliary role in difficult airway awake intubation with unclear vision or abnormal anatomy. The first-generation prototype of the instrument was completed and feasibility of endotracheal intubation was preliminarily verified on pigs with spontaneous breath in our previous study (25). In this study, we focused on the learning curve and efficacy of the novel device.

As a novel device, it is similar to a fiberscope except for the multimodal indication of airway direction. For acquisition of the awake intubation skill with fiberscope, Smith et al. observed that after performing 18 fiberoptic nasotracheal intubations in patients with normal airways, trainees could complete 70–80% intubations within 1 minute, and it took about 45 manipulation attempts to reach the expected “expert level” (26). Dalal et al. analyzed the learning curve of 16 residents in a study using fiberscope in a normal airway simulation for nasotracheal intubations, and it took 27–58 attempts to be proficient in nasotracheal fiberoptic intubation (27). Moreover, once proficiency was attained, the average operation time of fiberoptic bronchoscopy was about 50–120 seconds in previous studies (28,29). The differences in learning curves, operation time, and success rate among the studies may be related to the airway simulations and varying definitions of success. In our study, the insertion time was defined from the entry of the nasal cavity to the vision of the carina. We didn’t perform subsequent endotracheal intubation and the insertion time was defined as the time of scope examination because the main purpose of this study was to explore the learning curve and utility of this novel intubation device. The process of advancing endotracheal tubes may have an impact on the utility and effectiveness of the device itself (30). In addition, the identification and positioning of airway is considered a crucial part of endotracheal intubation. Successful identification of the airway and delivery of the guiding scope into the airway have positive implications for the completion of endotracheal intubation operation. Therefore, in our study, the main endpoint was the correct insertion of the novel device instead of completion of the entire endotracheal intubation process. With the accumulation of experience, the differences in insertion time and success rate between the first 20 attempts and the last 20 attempts were statistically significant, and it was considered that the learning plateau period was reached at the last 20 attempts. Therefore, with acceptable training on the simulation, the insertion time of MEIAD on the manikin was 17.5 (14.0–28.0) seconds, and the success rate was 96.6%. As a novel and self-developed intubation assistant device, MEIAD showed a satisfactory learning curve and efficacy on the manikin.

A learning curve is a way to monitor performance development over time. There are many ways to construct a learning curve, such as with a graph, table, or statistical technique (31). CUSUM analysis is a useful method for spotting subtle, slow, prolonged degradation in a process under control and thus an effective method to evaluate a novel skill. However, CUSUM analysis requires a clear description of the success and failure of the operation, and the assessment condition cannot be changed randomly during the research process. In terms of endotracheal intubation, the success and failure of the operation are not uniformly defined. In early relevant studies involving CUSUM analysis of endotracheal intubation, the p0, p1, α, and β values were also set differently. The requirement for strict quality control in medical training is important, but there is still debate over whether or not these restrictions should be gentler for novice residents. Since this study was an exploratory study of a novel device prototype and the operators were novice anesthesiologists (grades 1–2), we set the acceptable failure rate at 15% and defined success as the correct insertion into the airway within 120 seconds. These variables were identical to prior CUSUM-based studies regarding learning curves of tracheal intubation (16,17). Under this condition, the residents became proficient at using MEIAD on the difficult airway simulator after 22 manipulations. If a stringent acceptable failure rate is set or success is defined as a shorter operation time, the number of people who acquire skills within the specified attempts may also decrease accordingly. For example, setting the acceptable failure rate as 10%, only 11 residents could cross h0 within 40 manipulation attempts. If a successful operation was defined as less than 90 seconds, 2 residents could not cross the lower decision boundary h0 in 40 operation attempts. Therefore, the results of CUSUM analysis of the learning curve need to be comprehensively considered in combination with the assigned p0, p1, α, and β values.

Simulation-based education is an important approach for teaching and training in difficult airway management (32,33). For some unconventional operations with a slow learning curve, simulation-based training can be especially effective for reducing harm to patients caused by novices in the early stage of clinical operations. Consistent with the results of our study, simulated practice is an effective approach for improving the self-confidence of beginners. Since MEIAD is still in the development and verification stage, a simulated airway is suitable for initial exploration.

There were several limitations to this study. Our study involved the use of a manikin, and due to its relatively simple structure, the results obtained on the simulator cannot be fully replicated in clinical practice under the same conditions (34). It is certain that the clinical process will be more challenging and the learning curve will decline more gradually. Further, according to the literature, common physical methods for constructing a difficult airway are cervical collar and simulated tongue edema (35). The advantage of MEIAD is to assist with an obscured or unclear view, and the simulated tongue edema method is more suitable for constructing an unclear view. Due to the limitation of the manikin, the tongue was unable to be inflated to simulate tongue edema. At the same time, it also cannot simulate the clinical scenario of sputum and secretions blocking the view field (36), so the advantage of multimodal tracheal intubation aids was not fully realized. The initial prototype MEIAD used in this study has not yet been put into production and clinical application. As a proof-of-concept study, we confirmed the utility and efficacy of MEIAD on the airway simulator, and further confirmation in a different clinical setting or involving operators with different experience is needed. Considering the diversity and complexity of actual clinical scenarios, the device will be modified and improved before clinical application.

In conclusion, novice residents could be basically proficient in MEIAD after 22 practice sessions on the simulated airway. At the stable stage of the training, the insertion time was 17.5 (14.0–28.0) seconds, and the success rate was 97.5%. It is expected that MEIAD will be further improved and promoted and has the potential to be an alternative airway device.

The article’s supplementary files as