Changing the view: impact of simulation-based mastery learning in pediatric tracheal intubation with videolaryngoscopy.

OBJECTIVE: To determine the effect of a training program using simulation-based mastery learning on the performance of residents in pediatric intubations with videolaryngoscopy.
METHOD: Retrospective cohort study carried out in a tertiary pediatric hospital between July 2016 and June 2018 evaluating a database that included the performance of residents before and after training, as well as the outcome of tracheal intubations. A total of 59 pediatric residents were evaluated in the pre-training with a skills' checklist in the scenario with an intubation simulator; subsequently, they were trained individually using a simulator and deliberate practice in the department itself. After training, the residents were expected to have a minimum passing grade (90/100) in a simulated scenario. The success of the first attempted intubation, use of videolaryngoscopy, and complications in patients older than 1year of age during the study period were also recorded in clinical practice.
RESULTS: Before training, the mean grade was 77.5/100 (SD 15.2), with only 23.7% (14/59) of residents reaching the minimum passing grade of 90/100. After training, 100% of the residents reached the grade, with an average of 94.9/100 (SD 3.2), p<0.01, with only 5.1% (3/59) needing more practice time than that initially allocated. The success rate in the first attempt at intubation in the emergency department with videolaryngoscopy was 77.8% (21/27). The rate of adverse events associated with intubations was 26% (7/27), representing a serious event.
CONCLUSIONS: Simulation-based mastery learning increased residents' skills related to intubation and allowed safe tracheal intubations with video laryngoscopy.

Introduction

Obtaining an advanced airway through tracheal intubation (TI) is an essential skill for handling many emergencies. However, it is less frequently necessary and technically more difficult in pediatrics when compared to the adult population. TI in controlled situations, such as anesthesia, occurs in the first attempt at rates above 98%; however, in pediatric intensive care units (ICU), success in the first attempt occurs only in around 60% of cases.

In the pediatric emergency department, TI is a risky procedure due, among other factors, to the severity of patients itself, the absence of pre-procedure fasting, and the relative inexperience of professionals performing the intubation. Studies in different services such as Australia, Japan, South Korea, and the United States reported success rates in the first attempt of 52%–78%.4, 5, 6, 7 A Brazilian study, carried out in a tertiary hospital in São Paulo, found that only 39% (37/94) of patients were intubated in the first attempt.

Tracheal intubation-associated events (TIAEs), such as cardiorespiratory arrest, esophageal intubation, vomiting with aspiration, hypotension, laryngospasm, pneumothorax, selective intubation, epistaxis, tooth or lip trauma, arrhythmia, pain, or agitation that delay the procedure9, 10 are associated with worse outcomes during hospitalization of children in pediatric ICUs.

Traditionally, intubation occurs under direct visualization of the vocal cords with the aid of a laryngoscope, with the use of videolaryngoscopy being a more recent alternative. Videolaryngoscopy devices allow airway visualization through small high-resolution cameras and portable monitors, in an attempt to improve the vision and success rate of the laryngoscopy, with its use being associated with a lower occurrence of TIAEs. Despite the possible benefits of its use, there is still resistance to the adoption of the video laryngoscope, due to the concern with costs and also with its use for the training of new professionals. To implement its universal use, it is necessary to train professionals to work with it.

The mastery learning training strategy consists of the following steps: (i) baseline assessment; (ii) clear training objectives, sequenced in units of increasing difficulty; (iii) participation in sustained educational activities (in this case the deliberate practice on an intubation manikin); (iv) a minimum passing grade in each step;(v) formative assessment with specific feedback until activity completion with mastery is achieved; (vi) summative assessment with advancement to the next unit, until skill acquisition at the mastery level; and (vii) continuous practice until mastery acquisition. Just-in-time and just-in-place trainings are those that take place during the working hours and at the workplace of the professional.

There is only one study using mastery learning for intubation training with videolaryngoscopy, showing the superior performance of eight residents for intubations in simulated cases of adult patients using this strategy when compared to the performance of eight residents submitted to traditional training, which was maintained after six months of training; however, the performance in real intubations in the clinical environment was not evaluated.

Considering the relative difficulty and severity of TI in pediatric emergency, the potential benefit of using videolaryngoscopy, and the need for training resident pediatricians in airway management skills, a TI training program with videolaryngoscopy was established using a strategy of simulation-based mastery learning.

The hypothesis was that with training, residents perform excellently in a simulated case, and that this can be reflected in greater success rates in intubations in real situations.

This study aimed to determine the effect of this training program on the performance of residents in pediatric intubations with videolaryngoscopy, both in simulated cases and in intubations in the emergency department. To the best of the authors’ knowledge, there are no reports in the literature of this strategy being used for training tracheal intubation in pediatrics.

Materials and methods

Design

This was a retrospective cohort study that evaluated a database, which included resident physicians’ performance before and after training, as well as the outcome of tracheal intubations. This was a specific analysis related to the training of residents in a project that evaluated intubation safety in the emergency department.

Sample

A total of 80 s pediatric residents during two years of follow-up were eligible for the study. Inclusion criteria were being at the pediatric emergency stage and having undergone training. Exclusion criteria were refusal to participate in the study or not having attempted at least one intubation in patients older than 1 year during the pediatric emergency stage. All intubation records for 24 months between July 2016 and June 2018 were evaluated in patients older than 1 year of age and younger than 18 years.

Study site

This study was carried out in the emergency department of an exclusive pediatric tertiary university hospital, which does not treat polytrauma patients and is a referral for emergency care in patients with underlying diseases.

Training

According to the service schedule, a second-year pediatric resident is assigned at each shift to perform emergency procedures, if these occur. These residents attended a valid pediatric advanced life support (PALS) course, have practical classes in a skill laboratory with intubation training during the training period, and participate in on-site simulations in the emergency room and skill laboratory simulations, with scenarios that include the need for advanced airway management. For the training of TI with videolaryngoscopy, a strategy was created using the principle of mastery learning, including elements of the preparation, of the TI procedure itself with direct laryngoscopy and videolaryngoscopy, and the checking of the intubation.

An intubation checklist (Table 1) and four model cases for training were prepared by one of the study authors (TC) based on elements of the intubation and publication record describing just-in-place intubation training, i.e., at the professional’s workplace.3, 9 A group of specialists consisting of seven emergency pediatricians reviewed the checklist to ensure face and content validity.

Table 1

Training checklist.

Task	Performedcorrectly	Not performed/performed incorrectly	Comments
Preparation
Procedure indication
RSI indication
RSI medications
ECG monitoring
Sat monitoring
BP monitoring
Requests capnograph
Requests venous access
Hand hygiene
IPE gloves
IPE mask and goggles
Requests BVM
Tests BVM
Requests cushion/positions patient
Requests mask of correct size for the patient
Requests material for aspiration
16. Requests material for fixation
Requests stethoscope
Requests adequate-size cannula
Tests cuff/requests syringe
Requests guidewire
Requests VL
Chooses correct blade
Tests VL
Requests correct size LM
Minimum of three professionals (intubator, medication, and intubation aid)
Procedure
Pre-oxygenates
Waits for the effect of RSI drugs
Laryngoscope in the left hand
Visualizes vocal folds
Inserts tube in the trachea
Inserts the tube at the correct depth (3x cannula)
Insufflates cuff
Ventilates with BVM
Checklist
Checks expansibility
Checks capnometry
Checks saturation
Auscultates
Fixes tube
Requests chest X-ray
Total (mastery grade = 36 orhigher)
*if any of the items is not applicable, for instance RSI in CRA, consider it correct and make a note it in comments.
Resident:	Date:	Evaluator:

RSI, rapid sequence intubation; ECG, electrocardiogram; Sat, oxygen saturation; BP, blood pressure; IPE, individual protection equipment; BVM, bag-valve-mask; VL, video laryngoscope; LM, laryngeal mask; CRA, cardiorespiratory arrest.

Each item was given an equal value and a score of 1 (done) or 0 (done incorrectly or not done) was determined. To determine the passing score, the Angoff method was used, where specialists are introduced to the assessment instrument, individually judge item by item, and classify whether an average participant would do it correctly or not. After a discussion, knowing the classification given by colleagues, a new classification is made, and the average value of this new evaluation generates the minimum passing grade. The minimum passing grade of 34.7 items on the checklist was reached. To facilitate grade interpretation, 2.5 points are attributed to each correct item in the checklist, determining scores of 0–100 points. The minimum passing grade that would be 87.5/100 was rounded off to 90/100 (or 36 checklist items) for practical reasons. Before starting the training, all residents were evaluated using this checklist by an emergency physician participating in the study (VC). For reliability control, the study's main investigator (TC) performed the initial assessments of eight participants (10% of eligible residents) independently.

At the beginning of their training, each resident was trained individually by an assistant physician (VC) in the emergency department, using videolaryngoscopy, materials needed for intubation, and an intubation manikin (Pediatric Intubation Trainer Parts; Laerdal – Norway). The videolaryngoscope used has a standard Macintosh blade format, allowing both direct laryngoscopy and videolaryngoscopy (McGrath Mac Enhanced Direct Laryngoscope; Medtronic – United States).This device did not have a No. 1 blade at the time of the project and, therefore, patients younger than 1 year were excluded from this study. During the training, a theoretical explanation about the intubation procedure was provided, as well as a presentation of the available device, step-by-step demonstration of the procedure, deliberate practice of both intubation by videolaryngoscopy and by direct laryngoscopy, and feedback. After the training, the residents were tested again, unless they had already achieved a minimum passing grade in the pre-test. If mastery was not proven, the resident had new opportunities for practice and was tested again. The complete activity lasted between 30 min and two hours, depending on the resident's performance. Fig. 1 shows the type of visualization that the resident had during training.

Fig. 1

Resident training using simulator and videolaryngoscopy.

Each resident was only allowed to intubate patients once they had undergone training and achieved mastery within the proposed training objectives, i.e., at least 90% of the items correctly answered. Actual intubations performed in the emergency department were always supervised by an attending physician, who was able to view the screen and suggest corrections, if necessary.

Outcomes

Primary: Performance in case of intubation with videolaryngoscopy in an intubation simulator, pre- and post-training, assessed through the percentage of correct items in the checklist.

Secondary: Rate of successful intubations with videolaryngoscopy at the first attempt in real emergency situations, of intubation-related events, and assessment of the degree of glottic visualization, assessed through analysis of tracheal intubation records.

Power of the study

This study had a maximum convenience sample of 80 residents, 40 per year of study. Based on the preliminary test of the checklist with pediatric emergency residents during the preparation of the instrument, the authors estimated a baseline performance in the population of residents of 75/100 at the pre-test and a 15-point increase in the post-test, for a mean of 90/100 in the checklist, with a standard deviation of 30. It was calculated that if 42 residents (approximately 50% of the sample) participated in the study, considering an alpha of 0.05 and a paired t-test between two means, the study would have a 90% power to detect a statistically significant difference of 15 points in performance during evaluation (a 20% improvement in performance).

Statistical analysis

The data were treated according to the outcomes of the variables. For continuous variables, the evaluation was carried out using mean values (standard deviations) if the distribution was normal or medians (range or interquartile range) if the distribution was not normal, according to the Kolmogorov–Smirnov or Shapiro–Wilk test. The comparison of means or medians was performed using Student’s t-test, or the Mann–Whitney or Kruskal–Wallis test, according to the necessity. For categorical variables, the assessment was in terms of percentage. The comparison of categorical variables was performed using the chi-squared test or Fisher’s exact test.

The data were analyzed using the software SPSS (IBM SPSS Statistics for Windows, Version 22.0. NY, United States). A p-value <0.05 was considered as statistically significant.

Ethics

An informed consent was requested to utilize the data. The project was approved by the hospital's ethics and research committee, under the code CAAE:56983516.7.0000.0068.

Results

Participants

All 80 eligible residents participated in the training. Fifty-nine second-year residents underwent simulation-based training and had attempted at least one intubation in a patient older than 1 year during the pediatric emergency residency. A total of 21 residents were excluded for not having attempted at least one intubation in children older than 1 year during the study period.

Before the training with simulation, the average grade was 77.5/100 (SD 15.2), with only 23.7% (14/59) of residents reaching the minimum passing grade of 90/100. After the training, 100% of residents reached the passing grade, with an average of 94.9/100 (SD 3.2), p < 0.001, t-test, which demonstrates a 23% increase in performance after the training. The inter-rater reliability (VC-TC) was 0.71 by Cohen's kappa. Only 5.1% (3/59) of the residents needed more practice time than initially allocated. These had extended practice time and were tested again until they reached the mastery. Fig. 2 summarizes the results related to the training.

Fig. 2

Mean grades and standard deviations in the evaluation checklist for simulated intubation cases, before and after training.

TI procedures

A total of 50 patients older than 1 year were submitted to TI with videolaryngoscopy during the study period, and in 27 the procedure was performed by second-year residents. The median age of the patients was 3.5 years (1–18). Most patients, 89% (24/27), had an underlying disease.

The success rate in the first attempt at TI with videolaryngoscopy was 77.8% (21/27). The rate of intubation-associated events was 26% (7/27). There was one case of an event considered severe, in which the patient had vomiting with aspiration. The degree of glottic visualization, measured by the Cormack-Lehane classification, in TI with videolaryngoscopy, showed a median grade I classification of (I–III).

Discussion

In this study, conducted in a single emergency department of a tertiary hospital, the training of residents using simulation-based mastery learning was able to guarantee excellent performance in a simulated case. Success rates in tracheal intubations in patients older than 1 year were also compatible or superior to those in the international literature.

The obtained data reinforce the results of other studies with skill training in critical procedures, such as the passage of a central venous catheter, thoracentesis, chest drainage, lumbar puncture, and ultrasonography using purposeful practice based on simulation with mastery learning.16, 26 This training model with mastery learning before real procedures is in contrast to the traditional procedure training model for residents, the “see one, do one, teach one” concept, which poses a potentially greater risk for the patient and is invariably insufficient when it is intended to train low-frequency, but highly important procedures, as are many of the procedures in the pediatric emergency room.

The goal of training professionals is to trigger changes in practice. One strength of this study was that it assessed clinical performance after training, the so-called Kirkpatrick level four. The observed result, of 77.8% successful intubation in the first attempt, is close to and sometimes higher than the level of success described in several pediatric emergency departments around the world, which varies between 52% and 78%,4, 5, 6, 7 and the reference of 15 North American ICUs, with a 62.2% success rate.

Although it is possible to claim that part of this result should be attributed to the device itself, the fact is that prior to the simulation-based training, the videolaryngoscope, although available, was rarely used; moreover, the success of first-time intubations in this service was much less, at around 40%. It is noteworthy that 21 of the 80 trained residents did not attempt at least one intubation in children older than 1 year in the pediatric emergency stage during the study period. Even though it is possible that these residents intubated patients at other stages of the residency, where this procedure is more common, such as in intensive care or neonatology, the relative rarity of intubations reinforces the need to keep residents systematically trained.

As for the adverse events related to tracheal intubation during the study period, the rate found, of 26%, is still high when compared to the rate of 20% described in a registry of 15 pediatric ICUs in the United States, which shows that there is still room for improvement regarding the safety of intubations in this service.

The degree of measured glottic visualization using the Cormack-Lehane classification system was ideal with the use of a videolaryngoscope, which may explain the high success rate. However, since visualization occurs indirectly, video laryngoscopy usually makes it easy to visualize, but creates difficulties for the passage of the tracheal cannula, unlike direct laryngoscopy, where the greatest difficulty is usually to adequately visualize the airway.

Some limitations of this study comprise the fact that it was carried out in a single center, with distinctive characteristics of the treated population, with greater potential severity of patients, which may partly explain the success rates prior to training and the rate of complications. As this study measured the outcomes of a group of residents in which all were trained, it is not possible to guarantee that the good performance in real cases occurred only because of the training and not due to other factors, such as the Hawthorne effect or because the residents had more general training time over the study period. Moreover, the medical residency organization is also differentiated and, therefore, the results found herein should be interpreted with caution, avoiding generalizations.

Simulation-based mastery learning increased intubation-related skills of residents and allowed greater safety in tracheal intubations. This training model requires that residents demonstrate their skills in a simulated environment before performing this invasive and risky procedure on patients. The results reinforce the need for robust training programs on critical procedures in pediatric emergencies. Future studies are required to evaluate the effect of this training model compared to other teaching methods, for all age ranges and in clinical outcomes such as mortality, long-term intubation complications, and hospital length-of-stay.

Conflicts of interest

The authors declare no conflicts of interest.