Establishing quality indicators for pre-hospital advanced airway management: a modified nominal group technique consensus process.

BACKGROUND: Pre-hospital advanced airway management is a complex intervention composed of numerous steps, interactions, and variables that can be delivered to a high standard in the pre-hospital setting. Standard research methods have struggled to evaluate this complex intervention because of considerable heterogeneity in patients, providers, and techniques. In this study, we aimed to develop a set of quality indicators to evaluate pre-hospital advanced airway management.
METHODS: We used a modified nominal group technique consensus process comprising three email rounds and a consensus meeting among a group of 16 international experts. The final set of quality indicators was assessed for usability according to the National Quality Forum Measure Evaluation Criteria.
RESULTS: Seventy-seven possible quality indicators were identified through a narrative literature review with a further 49 proposed by panel experts. A final set of 17 final quality indicators composed of three structure-, nine process-, and five outcome-related indicators, was identified through the consensus process. The quality indicators cover all steps of pre-hospital advanced airway management from preoxygenation and use of rapid sequence induction to the ventilatory state of the patient at hospital delivery, prior intubation experience of provider, success rates and complications.
CONCLUSIONS: We identified a set of quality indicators for pre-hospital advanced airway management that represent a practical tool to measure, report, analyse, and monitor quality and performance of this complex intervention.

Pre-hospital advanced airway management is a critical feature of many mass casualty events.

However, quality indicators to evaluate pre-hospital advanced airway management are lacking.

An international group of pre-hospital medicine experts conducted a consensus process to identify quality indicators for pre-hospital advanced airway management.

Seventeen quality indicators were identified, covering all steps of pre-hospital advanced airway management.

This practical tool can help measure, report, analyse, and monitor quality and performance of this complex intervention.

Continuous improvements in pre-hospital critical care has allowed advanced diagnostic, therapeutic, and supportive procedures such as pre-hospital advanced airway management (PAAM) to be delivered without delaying time to definitive care., However, the therapeutic benefit remains unclear and there is evidence that it may even be harmful. Studies often suffer from limited external validity because of the heterogeneity of the data collected. To address this concern, templates have been developed to standardise documentation and reporting of PAAM. Further, studies often struggle to reliably evaluate this complex intervention, the considerable heterogeneity in providers, and the techniques used that might influence outcomes and the quality of care.

The inherent heterogeneity of multiple steps, interactions, and variables in complex interventions suggest that traditional methods such as systematic reviews are of limited value.6, 7, 8, 9 Instead, a quality improvement approach using quality indicators (QIs) may be more suitable, as the measurement of complex interventions through datasets is accessible, practical, and needs less risk adjustment. Measuring the quality of PAAM will allow systems to monitor processes and provider quality to target quality improvement and the professional development of the providers, and define the level of quality required to have a positive impact on patient outcome.

The aim of this study was to use expert consensus development methodology to develop a set of QIs to evaluate PAAM, viewing it as a process with potential for improvement.

Methods

Study design

The study was conducted between October 2016 and June 2018 and included a narrative literature review, followed by a modified nominal group technique (mNGT) consensus process, comprising three email rounds and a consensus meeting among an international group of experts., The experts were selected based on scientific merits within the field of emergency airway management, and especially in the pre-hospital setting. They were all senior physicians in pre-hospital critical care, recruited among medical societies (e.g. the European Pre-hospital Research Alliance [EUPHOREA] and the European Airway Management Society's [EAMS] council) and the professional networks of the project group. Geographically, the experts were from Europe, North America, and Australia. The expert group was unaware of its composition until the consensus meeting, and anonymity was guaranteed for each email round. Since the study did not include any sensitive data it was exempted from a formal ethical review by the Regional Committee for Medical and Health Sciences Research Ethics of Western Norway (Reference number 2017/260).

Definitions

PAAM was defined in accordance with the Utstein-style template as ‘any airway management beyond manual opening of the airway and use of simple airway adjuncts, such as an oropharyngeal airway’. PAAM includes both the introduction of a supraglottic airway device or a tracheal tube (either through the natural orifice or through front of neck access) and the consecutive controlled or assisted ventilation. The latter also includes bag-valve-mask (BVM) ventilation, noninvasive mechanical ventilation, or other ventilatory support in case of failed insertion of an airway device.

Process

Figure 1 describes every step of the identification and selection of the QI for PAAM, and the experts' tasks at each round of the mNGT.

Fig 1

Flowchart detailing identification and selection process of the quality indicators for pre-hospital advanced airway management. QI, quality indicator. ∗Nine QIs tied for 10th place. †Two QIs tied for 3rd and 5th place. ‡One QI was divided into two QIs for measurement feasibility.

Literature search

Potential QIs for PAAM were first identified from the literature by the research group. Guidelines, recommendations, and studies addressing the relationship of PAAM and patients' outcome (Supplementary File 1) were analysed to identify best practices for PAAM. Also, potential or validated QIs for advanced airway management in an emergency setting were searched for, as to our knowledge, QIs specific to pre-hospital setting have not been published yet. Then, the consensus process aimed to identify the most relevant QI for PAAM, starting with a wide set of potential QIs and gradually honing in on a subset. Considering that the quality measurement of PAAM represents only one area of quality monitoring in pre-hospital critical care, while ensuring a sufficient description of the procedure, the project group aimed for a final set of around 20 QIs.

Expert panel questionnaire

Questionnaires sent to the experts were designed on an excel spreadsheet (Microsoft Corporation, Redmond, WA, USA) and contained a list of QIs structured according to the three categories described by Donabedian: structure, process, and outcome (definitions in Supplementary File 2). The questionnaire in the first round contained the potential QIs issued from the literature search. On each step of the consensus process, the questionnaires were sent by email to each expert individually, who submitted their responses to a data manager. The project group was blinded to the data submitted by the experts to the data manager, who had no role in the study design, analysis of the data, or interpretation of the study results.

Assessment of consensus

In the first round, the experts were asked to rate the importance of each QI for measuring quality of PAAM using a Likert scale, ranging from 1 (‘totally disagree’) to 5 (‘totally agree’). A mean Likert rate of 4 or higher was used to determine the proportion of QIs in each of the structure, process, and outcome category for the final list of QIs. These proportions were necessary to give the adequate tasks to the experts and ensure that all three categories would be represented in the final set of QIs. In the second and third round, the process of selection of the QIs followed a detailed scheme involving six calculation methods, attributing varying importance to nomination and ranking (Supplementary File 3).

During the consensus meeting, each QI was evaluated for its legitimacy as a QI. Further, the name, definition, potential categories, and values for each QI were revised and agreed upon by consensus among the experts. A supervisor of the project group moderated the discussions between the experts, while the main investigator documented the discussions and answered questions from the experts, but was not involved in the discussions.

After the consensus meeting, the experts confirmed their agreement on the list of 20 QIs from the consensus meeting. Finally, the set of QIs was assessed for usability according to the National Quality Forum's (NQF) Measure Evaluation Criteria by the project group and the necessary revisions were adapted.

Results

Study participants

Twenty-one experts were invited to contribute to the consensus process, 16 of whom accepted the invitation. All 16 experts participated and answered to each of the three email rounds. From these 16, nine attended the consensus meeting and 15 agreed on minor corrections required for the final list of 17 QIs. During the three email rounds, the 16 experts answered 99.7% (n=3876) of the 3888 items of the questionnaires (243 for each expert).

Consensus process

QIs explored at each step of the project are presented in Fig. 1. In round one of the consensus process, 15 structure, 22 process, and 15 outcome QIs obtained a mean Likert rate of 4 or higher. With 35 additional QIs proposed by the expert panel, a total of 74 QIs were identified, comprising 19 structure (26%), 38 process (51%), and 17 outcome (23%) QIs. This distribution was applied to the predefined limit of 20 QIs of the final set (respectively, 5/10/5 QIs).

Final quality indicators

A total of 17 QIs met the NQFs Measure Evaluation Criteria (Table 1). A more precise presentation and definitions required for a proper practical use of the indicators are presented in their specification sheets (Supplementary File 2). One indicator was split into two for measurement feasibility. Four QIs that did not fulfil the NQF Measure Evaluation Criteria were excluded, but led to experts' recommendations presented in Supplementary File 4.

Table 1

Quality indicators for pre-hospital advanced airway management, ranked‡ by importance according to expert panel consensus.

QI Nr	Quality indicator name	Short definition
Structure-related QIs
QI 1	Overall intubation clinical practice	Overall number of successful intubations performed by the provider in the hospital and pre-hospital setting before the recorded attempt∗,‡
QI 2	Pre-hospital intubation periodic exposure	Number of successful intubations performed by the provider in the pre-hospital setting during the 12 months before the recorded attempt∗,‡
QI 3	Intubation periodic exposure	Number of successful intubations performed by the provider in the hospital and pre-hospital setting during the 12 months before the recorded attempt∗,‡
Process-related quality indicators
QI 4	Intubation attempts∗	Total number of intubation attempts for the given patient
QI 5	Capnography for tube position confirmation	Rate of (quantitative) continuous waveform end-tidal CO2 monitoring and documentation, for tracheal tube placement confirmation, immediately after advanced/definitive airway insertion
QI 6	Preoxygenation method	Rate of patients where preoxygenation was performed with a BVM or an automated ventilator, with PEEP
QI 7	Preoxygenation duration	Duration of the preoxygenation phase, using a BVM or an automated ventilator with PEEP
QI 8	RSI for intubation	Rate of rapid sequence induction including an anaesthetic drug (induction) and an NMBA (paralysis), for intubation of patients with vital signs
QI 9	Laryngoscopy duration†	Duration of the ‘no oxygenation time’ during laryngoscopy∗
QI 10	Intubation Indication threshold (attitude)	Rate of intubation of trauma patients with GCS<9 compared with all trauma patients with GCS<9
QI 11	EtCO2 monitoring during transport	Rate of intubated patients with continuous EtCO2 (capnometry) monitoring during transport to hospital, compared with all intubated patients
QI 12	Automated ventilation during transport	Rate of patients ventilated with an automated ventilator during transport to hospital (after insertion of advanced airway device), compared with all patients with an inserted advanced airway device and ventilated during transport to hospital
Outcome-related quality indicators
QI 13	First attempt success	Rate of successful tracheal intubation at first attempt, compared with all patients who at least got one intubation attempt∗,‡,†
QI 14	Overall intubation success	Rate of successful tracheal intubation, compared with all patients who at least got one intubation attempt∗,‡,†
QI 15	Desaturation during laryngoscopy†	Rate of patient with SpO2 decrease below 90% or ≥10% from baseline during intubation/laryngoscopy†
QI 16	Complications§	Rate of complications observed during the intervention and clearly associated with the pre-hospital airway management, compared with all patients who underwent at least one intubation attempt†.
QI 17	Normoventilation at hospital delivery	Rate of patients with an inserted advanced airway device in place who are normoventilated at handover in hospital: EtCO2=4–6 kPa (30–45 mm Hg), PaCO2=4.67–6.67 kPa (35–50 mm Hg), compared with all ventilated patients with an inserted advanced airway device in place (patient still ventilated by the pre-hospital ventilator or BVM).¶For TBI patients: EtCO2=4–4.67 kPa (30–35 mm Hg), PaCO2=4.67–5.33 kPa (35–40 mm Hg), according to the Brain Trauma foundation.

BVM, bag-valve-mask; GCS, Glasgow Coma Score; NMBA, neuromuscular blocking agent; QI, quality indicator; RSI, rapid sequence induction; SAD, supraglottic airway device; TBI, trauma brain injury.

Intubation attempt: an attempt is each time the laryngoscope blade passed the front teeth. Correction of the tube’s depth is not defined as a new attempt.

Laryngoscopy duration: defined as the time between the moment the preoxygenation mask is removed from the face of the patient and the moment the tube position is confirmed in the trachea (preferably with capnography).

Complications contain the items of the updated Utstein-style airway template. Immediately recognised/corrected oesophageal intubation; not immediately recognised/corrected oesophageal intubation; tracheal tube misplaced in left or right main stem bronchus; incorrect positioning or difficult ventilation with SAD; dental trauma; aspiration or vomiting during airway management (and not present before); cardiac arrest during airway management; complications during surgical or percutaneous airway management (e.g. bleeding or pneumothorax); new hypoxia during airway management; new bradycardia during airway management; new hypotension during airway management. The three latter ones are defined as follows: hypoxia: adults and children: SpO2<90%; hypotension: infants <1 yr: SBP<70 mm Hg, children 1–10 yr: SBP<70+ (2×age), children >10 yr: SBP<90 mm Hg, adults: SBP<90 mm Hg or decrease >10% from baseline value; bradycardia: newborn to 3 yr: <100 beats min−1, 3–9 yr: <80 beats min−1, 10–16 years: <60 beats min−1, adults: <50 beats min−1.

Intubation success: a success is defined by a tube confirmed in the trachea (preferably by at least two different techniques, one of them ideally being quantitative EtCO2 measure immediately after insertion).

Services with blood gas analysis possibility should use PaCO2.

Discussion

Based on review of the scientific literature and an mNGT consensus process, we identified 17 QIs for PAAM that cover structure, process, and outcome categories.

All the QIs in the structure category address the skill level of providers, emphasising provider skills more than equipment or infrastructure. Although anaesthesiologists can more easily achieve advanced airway experience in their routine clinical practice in a given time period and tend to perform better in PAAM compared with non-anaesthesiologists, several studies have reported overall intubation success rates >99% for PAAM providers regardless of base speciality or professional background.2 14 Therefore, to objectively assess the skill level of the providers, the number of intubations performed (QI 1) is considered key and should be monitored from the early stages of training, along with intubation success rate (QI 13, QI 14) and the frequency of intubation over a given period of time (e.g. 1 yr) (QI 2, QI 3). These indicators should be monitored individually for each provider, as the learning curve is very variable among the providers.15, 16, 17

Although most experimental studies describing learning curves have aimed for a performance goal of 90% overall individual intubation success, it is not yet clear what success rate should be achieved to confirm that a provider is competent to perform pre-hospital intubations safely, and what regular clinical exposure is required to prevent skill fade. Higher levels of regular practice have been associated with a lower incidence of difficult airway situations and more experienced providers tend to have a lower threshold for intubation (QI 10). Services with higher rates of attempted intubations had higher survival at hospital discharge for trauma patients with a Glasgow Coma Score (GCS) <9., Although PAAM standards might be adapted to the available resources and conditions, they should at least meet those of in-hospital emergency airway management, as suboptimal performance of intubation plays an important role in outcomes.

A primary goal of PAAM, opening and securing a threatened airway, is achieved when a cuffed tracheal tube is placed into the trachea. Ideally, two different techniques should be used to confirm correct placement of the tube, one of them being quantitative continuous waveform capnography immediately after insertion. The rate of use of this measurement and its documentation (QI 5) have been identified as the second most important QI in the process category, and its importance is supported by two studies., The use of rapid sequence induction (RSI) for pre-hospital intubation (QI 8) is associated with a higher overall and first attempt success rate (QI 14, QI 13), as intubation without drugs has been found to be associated with increased complications and mortality.,, RSI was defined as presented by the National Institute for Health and Care Excellence (UK). The choice of the hypnotic agent, neuromuscular blocking agent, and use of an additional opioid should be left open to the expertise of the provider and adapted to the clinical circumstances, as there is no evidence for an ideal agent for pre-hospital RSI.

Although some patient-related/injury-related variables cannot be modified, the skill level of the provider and the use of appropriate RSI techniques can be the focus of quality improvement. Both have an impact on the number of intubation attempts (QI 4), which has been identified as the most important QI in the process category and is highly correlated with intubation success (QI 14)., Also, repeated laryngoscopic manoeuvres are related to increased complication rates, morbidity, and mortality., Advanced airway management aims to ensure optimal oxygenation and ventilation, which is at least as important as securing the airway of the patient. They are highly related to complications and their therapeutic margin is narrow, as deviation from normoxia and normocapnia can be deleterious., Optimal oxygenation is different at different stages of PAAM. Once an airway device has been inserted, normoxia is the goal for most of the patients. However, normoxia during the pre-hospital phase was not retained as a QI, mainly because most emergency medical services (EMS) can only rely on pulse oximetry and are not able to perform blood gas analysis in the field. Instead, the rate of desaturation during induction/intubation (QI 15) was suggested as a way to monitor oxygenation, as it is a significant complication that is likely to occur during PAAM.

During preoxygenation, hyperoxia is required in order to prevent desaturation. In the hospital setting, the end-tidal fraction of oxygen (EtO2) is measured routinely in order to monitor preoxygenation and a certain value is usually targeted before starting induction. In the pre-hospital setting, EtO2 monitoring is not yet routinely available and preoxygenation must be done empirically based on experimental studies. As vital capacity breathing is not applicable in most pre-hospital patients, gently assisted inspiratory support and PEEP (in the absence of contraindications) using a BVM or a ventilator may accelerate the procedure and has been identified as the best method (QI 6), achieving a higher EtO2 in a shorter time than other methods. However, in critically injured patients, preoxygenation method and time might be adjusted to the circumstances. Finally, preventing or at least prolonging time to desaturation during intubation by performing optimal preoxygenation might improve the likelihood of first attempt success (QI 13), as desaturation is a common reason for aborted attempts.

Ventilation can be monitored by measuring the arterial or end-tidal CO2 pressure (Paco2 or EtCO2) and can be adjusted accordingly. Except for certain specific situations, patients should be normoventilated and thus handed over to the next level of care with an EtCO2 (respectively Paco2) within the normoventilation range (QI 17). Although measurement of Paco2 is preferable, most EMS are not performing blood gas analysis routinely in the field yet. To improve the quality of the ventilation, EtCO2 should be monitored continuously from insertion of the definitive airway device until hospital arrival (QI 11), as over- and under-ventilation plays an important role in the outcome of intubated patients, especially in traumatic brain injury.,, Patients should be ventilated with an automated ventilator (QI 12), as targeted pre-hospital ventilation using EtCO2 monitoring and automated ventilation has been associated with a decrease in severe iatrogenic hyperventilation and decreased mortality.37, 38, 39 Finally, both increase safety by allowing early recognition of tube dislocation, disconnection, or misplacement.

Monitoring complications is critical when measuring the quality of this complex intervention, as the rate and type of complications related to PAAM (QI 16) are tightly coupled to quality and influenced by the skill level of the provider, and almost all QIs of the process category. As expected, some of the complications listed in QI 16 were already identified as QIs in previous studies., Reliable collection of the data required to calculate the QI might be challenging for some EMS, especially for the QI relying on self-reporting and, moreover, in the pre-hospital environment. However, modern technologies can help reduce this limitation and increase reliability and accuracy of documentation. For example, electronic medical charts with automated vital signs recording will allow a higher precision in the collection of QI 15, QI 16, and QI 17. The use of video recording (videolaryngoscopy, bodycam, or both) might help with the time measurements required for QI 7 and QI 9.

Finally, the growing use of electronic medical charts should allow the systematic collection of the data to calculate the QI to be feasible within the available resources, as several variables are probably already systematically collected or can be easily added to the medical chart. However, feasibility both from a provider point of view and from a technical point of view will be a necessary next step in the implementation of systematic quality monitoring.

This study has several limitations. First, experts were recruited based on scientific merits within the field of PAAM. As described in other quality improvement programs, we could also have recruited expert clinical providers and other relevant stakeholders. However, most of our experts are clinically active in an EMS, which adds a clinical perspective to evaluating the importance and feasibility of collecting QIs. We intentionally aimed to enlarge the expert panel beyond the EUPHOREA members, and the country represented in this group, by inviting several council members of the EAMS, and experts from the professional networks of the project group. The latter might represent a selection bias and its effect could have been reduced by randomly selecting an acceptable number of experts from a larger list of potential experts. Further, the result of a consensus process is dependent of the group composition and could even be different if the mNGT was repeated with the same experts. We believe that by blinding the experts to each other in the first three rounds, some of the potential bias was reduced. Also, the QIs identified in this study were based on an initial group of suggested QIs based on scientific evidence. Second, we only performed a narrative literature review before the consensus process. Although a systematic review may have introduced more scientific strength to the end results, neither is a mandatory step in a consensus process, which is often started from scratch. The final list of QIs only contains four QIs that were not identified in the literature search, suggesting that this step was effective and useful. Third, we may have guided the number of indicators per category. Murphy et al. left indicator number open until the end and reached different proportions. Nevertheless, we calculated the proportions according to the Likert ratings given by the experts during round one and the spread among the categories might be different when addressing different topics. Finally, poor response rate from participants is often a serious limitation in this type of process. Although we would have preferred a higher participation rate at the consensus meeting, the overall response rate of nearly 100% during the three email rounds underlines the remarkable work of a committed group of experts.

In summary, by combining a review of scientific evidence with an mNGT process with international experts, we identified 17 QIs for PAAM. The QIs represent a practical tool to measure, report, analyse, and monitor quality and performance of this complex intervention. Adopting a continuous quality improvement approach will enable EMS systems not only to monitor their own performance, but also to compare their process and quality measurements with other EMS services, identify areas to focus on with quality improvement interventions, and measure their improvement.

Authors' contributions

Conceptualisation: AK, PNC, SJMS

Study design: AK, GAS, SJMS, AJK

Ethics approval coordination: AK, GAS, SJMS

Narrative literature review: AK

Data analysis: AK, AJK, GAS, JR, SJMS

Writing of the manuscript: AK, AJK, PNC, SJMS

Review of the manuscript: GAS, JR, JKH, DL

All authors have read and agreed to the published version of the manuscript.