Consensus-driven model to establish paediatric emergency care measures for low-volume emergency departments.

Narrative

Background

Engagement in quality improvement (QI) work is essential to ensuring emergency departments (EDs) are meeting the needs of all patient populations. In the USA, children represent approximately 27% (35 million) of all ED visits.1 Over 50% (18 million) of these children are seen in EDs that see fewer than 10 000 paediatric patients per year, with only 7% of children seen in extremely high-volume EDs (>50 000 paediatric patients per year).1 Among the 5000 EDs in the USA, 39% see fewer than 5 children per day, 30% see 5–14 children per day and 17% see 15–25 children per day.2 In the absence of repetitive performance and measurement, as seen in high-volume centres (greater than 10 000 paediatric visits annually), it may be difficult to assess the quality of care processes.3 In the USA, less than 50% of EDs report the inclusion of paediatric-specific elements in a department-wide QI plan.2 The aim of this manuscript is to identify a series of paediatric emergency care quality measures that would be valuable for low-volume EDs (<10 000 paediatric patients per year).

The National Paediatric Readiness Project applies a systems approach (care coordination, QI, policies and procedures, staff competencies, patient safety and equipment and supplies) to ensuring high-quality emergency care for children among diverse EDs.2 Paediatric readiness, as determined by the National Paediatric Readiness Assessment using a weighted 100-point scale, is associated with decreased paediatric mortality among critically ill and injured children.2 4–10 Facilities that incorporate paediatric-specific QI initiatives demonstrate a 26-point increase in their paediatric readiness score.11 Engagement in paediatric readiness efforts is high, yet integration of paediatric QI efforts in EDs is lagging.2

While over 400 paediatric emergency care performance measures have been proposed and prioritised, widespread uptake has been limited.2 12–14 Infrequent paediatric patient encounters make it difficult to assess the cause and effect of care processes. Furthermore, process and outcome measures within the paediatric emergency care landscape often require complex data extraction which may be especially difficult to implement in lower-resourced facilities.12–14 To date, no comprehensive assessment of the quality of paediatric emergency care delivery exists in low-volume EDs (<10 000 visits/year). The National Pediatric Readiness Quality Initiative (NPRQI) focuses on the undifferentiated patient experience that encompasses standard phases of care in the ED: assessment, interventions, diagnostics and disposition. We describe the development of a core set of NPRQI quality measures targeting common paediatric patient reports for adoption in low-volume EDs.

Methods

NPRQI conducted a five-phase modified Delphi process from November 2019 through January 2021.15 The study consisted of two content arms. Arm 1 included the cross-cutting clinical assessment and four clinical reports. Arm 2, which focused on behavioural health, was added 6 months into the study due to the evolving behavioural health crisis among children. The Delphi process included an initial meeting (Arm 1—in-person, synchronous; Arm 2—virtual, synchronous), a confidential online survey (Arm 1, Arm 2) and subsequent virtual meetings (Arm 1, Arm 2, synchronous). Both arms followed the same series of steps and were only separated by time.

Consensus panel

The panel consisted of 41 members who were either identified by their respective national professional society as a content expert or were selected based on the following criteria: expertise in paediatric emergency care applied research, emergency medical services for children, QI, QI data registries, specific areas of clinical practice, clinical practice setting, healthcare system networks, regulatory agencies and federal partners (table 1, online supplemental appendix A).

Table 1

Characteristics of consensus panel

Characteristic	Participants, % (N) N=41
Pediatric Emergency Care Applied Research (EA, EK*, CM, RS, SD*, TC*)	14.6 (6)
Emergency Medical Services for Children (CM, CN, EL, HH, MGH)	12.2 (5)
Quality Experts from National Professional Societies	26.8 (11)
American Academy of Family Physicians (DF)
American Academy of Pediatrics (RP, SJ)
American College of Emergency Physicians (IB, JA, KG)
American College of Surgeons Committee on Trauma (AJ)
Emergency Nurses Association (RK, SS)
National Association of State Emergency Medical Services Officials (AV)
Pediatric Trauma Society (LG)
Quality Improvement Data Registries (BM)	2.4 (1)
Health System Networks	4.9 (2)
US Acute Care Solutions (SI)
Hospital Corporation of America (HCA) Healthcare (AY)
Regulatory body	2.4 (1)
The Joint Commission (TE)
Federal partners	4.9 (2)
Health Resources and Services Administration (LL)
National Highway and Traffic Safety Administration, Office of Emergency Medical Services (EC)
Physician specialty	65.9 (27)
Paediatric emergency medicine (CM, HH, LA, MG, RP, RS, SI, SJ)
Emergency medicine (BM, CN, IB, JA, JL†, KG, KS†)
Trauma (AJ)
Family medicine (DF)
Behavioural health* (BZ, EK, JH, KD, NU, SD, SP, SR, TC, VF)
Nursing background	19.5 (8)
Emergency medicine (AR†, AY, BW, CR, CT, DG, RK, SS)
Trauma (CT, LG, SS)
Practice in low-volume ED setting (AR†, AY, CT, DG, JL†, KG, KS†)	17.1 (7)

Panellist affiliations are listed in online supplemental appendix A.

*Arm 2 panellist, members of the Emergency Medicine Quality Improvement Collaborative for Kids (EMQUICK). Co-chaired by Drs Susan Duffy and Tom Chun, EMQUICK works collaboratively to assist in the development and implementation of rigorous, evidence-based quality improvement measures for paediatric mental healthcare in EDs.

†Measurement feasibility assessment in low-volume EDs.

EDs, emergency departments.

Phase 1—determination of measures for consideration

The study team (KR, EE, KB, LG) and a subset of research panellists (ARM 1: AJ, CM, HH, LA, RS, BM; ARM 2: TC, SD) collated existing paediatric emergency care quality measures,10–12 specialty-specific professional guidelines, consensus statements and evidence-based reviews to identify measures for the ED setting. A total of five clinical reports (‘domains’) were identified based on the prevalence of clinical reports among ED visits, availability of evidence-based guidelines, alignment with national priorities and presence of validated screening tools.16 17 The five clinical domains selected for inclusion were: blunt head trauma, respiratory reports, seizures, behavioural health and vomiting. Cross-cutting measures for clinical assessment and interfacility transfer were included to capture foundational processes of care delivery, independent of the clinical report. Additional clinical domains were excluded due to scope. All proposed measures were characterised by clinical domain, the six domains of quality, phase of ED care (assessment, interventions, diagnostics, disposition) and measure type (process or outcome).18–20 Structural measures were excluded as they are the focus of the National Paediatric Readiness Assessment.5 Structural measures for behavioural health, proposed by the Emergency Medicine Quality Improvement Collaborative for Kids behavioural health consortium, are included in online supplemental appendix B for future consideration and consensus building; however, they were deemed outside of scope for NPRQI. This initial set of process and outcome measures were presented to the larger consensus panel for review and approval. Measures with a vote of greater than 50% were included in an online survey instrument (Research Electronic Data Capture; Vanderbilt University).

Phase 2—evaluation of measures

The consensus panel was charged with rating each measure based on the National Quality Forum (NQF) Measure Evaluation Criteria: feasible for data collection in a low-volume, low-resourced ED setting, usable to an ED care team, important for patient-centred outcomes and scientifically acceptable.21 The goal was to identify fewer than six measures per clinical domain (assessment, interfacility transfer, clinical reports and behavioural health). Arm 1 added three additional stakeholders (AR, JL, KS) from low-volume facilities to assess feasibility, usability and importance of the measures in the ED setting. Each member completed a confidential, online survey rating each measure using the NQF Criteria on a 3-point Likert scale.21 Aggregate responses were compiled and the mean score was calculated for each measure (online supplemental appendix C). Measures with a mean score of greater than 2 or greater than 1.8 for behavioural health, were included in Phase 3.

Phase 3—panel feedback: usability and importance

On completion of the survey, the panel was reconvened to share aggregate results: group mean and SDs for each measure (Arm 1 in-person; Arm 2 virtual). Panellists participated in open discussions regarding usability and importance. An importance statement was developed for each of the measures as further validation of usability and importance. Panellists had an opportunity to champion the inclusion of a measure or provide justification for exclusion. Those measures that did not achieve consensus (typically less than 80%) were excluded.

Phase 4—measure feasibility assessment

The feasibility of measures was determined based on viability of data extraction, accessibility of variables for each measure and complexity to implement. For both arms, the NPRQI leadership team (KR, EE, KB, LG) conducted a feasibility assessment based on whether discrete variables existed in emergency medical records for proposed measures and for Arm 2, whether the information would be accessible due to behavioural health privacy concerns and if discrete variables existed. Feedback was obtained from a subset of research panellists (AJ, BM, CM, HH, LA, RS, SD, TC) with respect to feasibility. Measures were presented to the consensus panel (virtual) and refined for clarity of interpretation and feasibility.

Phase 5—final review

The final list of measures was reviewed in totality with the panel (virtual) to review the final list of measures and importance statements, and ensure consensus. Each measure was further categorised by its Donabedian classification (process or outcome measure) and according to the phase of care in the ED (assessment, diagnostics, interventions and disposition).19

Results

The breadth of measures considered reflected NPRQI’s approach to identify cross-cutting care processes and common ED reports among undifferentiated paediatric patients. The seven clinical domains included two cross-cutting domains (recognition of the sick or injured child (8 measures) and effective transfer (4 measures)) and five clinical domains (blunt head trauma (10 measures), seizures (11 measures), respiratory reports (12 measures), vomiting (8 measures) and behavioural health (17 measures). Based on these clinical domains, 70 total measures were proposed for inclusion in NPRQI.

A total of 65 measures (53 for Arm 1 and 12 for Arm 2) were included in Phase 1 of the modified Delphi process (online supplemental appendix B). All measures were categorised according to the four phases of ED care: assessment, interventions, diagnostics and disposition. Behavioural health measures were categorised as follows: assessment (7 measures, 58.3%), interventions (1 measure, 8.3%), diagnostics (1 measure, 8.3%) and disposition (3 measures, 25%). The remaining clinical domain measures were categorised as follows: assessment (14 measures, 26.4%), interventions (19 measures, 35.8%), diagnostics (11 measures, 20.8%) and disposition (9 measures, 17%). In Arm 1, six measures scored 2.0 or less and were excluded from subsequent phases. In Arm 2, six measures scored 1.8 or less and were eliminated from subsequent phases. Phase 3 and 4 panel discussions resulted in the exclusion of 18 and 5 additional measures, respectively, from Arm 1 and the exclusion of 0 and 2 additional measures, respectively, from Arm 2. Measure exclusion during phases 3 and 4 was due to: lack of a validated screening tool (eg, human trafficking screening), lack of clarity in interpretation (eg, intubation following head trauma), outside locus of control for ED implementation (ED length of stay for mental health patients), redundancy with cross-cutting care process measures and/or complexity of data collection (table 2). Measures excluded based on complexity or feasibility to implement included: percentage of patients with abnormal vital signs included in provider notification process, percentage of families who received transfer packets, percentage of CTs that used appropriate weight-based dosing, percentage of patients with return precautions, medication dosing errors and behavioural health follow-up after discharge.

Table 2

Modified Delphi process for measures selection

Clinical domain	Proposed measures	Measures in each phase
		Phase 1: review and approve	Phase 2: National Quality Forum evaluation21	Phase 3: measure usability and importance	Phase 4: measure feasibility	Phase 5: final review
Cross-cutting care processes
Assessment	8	8	8	6	5	5
Transfer	4	4	4	3	3	3
Clinical report
Blunt head trauma	10	10	9	5	4	4
Seizures	11	11	9	5	3	3
Respiratory reports	12	12	10	6	6	6
Vomiting	8	8	7	4	3	3
Behavioural health	17	12	6	6	4	4
Total	70	65	53	35	28	28

A final review by a subset of research panellists (AJ, BM, CM, HH, LA, RS, SD, TC) confirmed the 28 final quality measures for inclusion in NPRQI: cross-cutting care processes (8 measures, 28.6%), blunt head trauma (4 measures), seizures (3 measures), respiratory reports (6 measures), vomiting (3 measures) and behavioural health (4 measures) (table 3). All were classified as process measures. By phase of care in the ED the measures are characterised as follows: assessment (8 measures, 28.6%), interventions (13 measures, 46.4%), diagnostics (3 measures, 10.7%) and disposition (4 measures, 14.3%).

Table 3

National Pediatric Readiness Quality Initiative quality measures

Intervention bundle	Donabedian classification	Phase of care	Quality measures
Recognition of a sick or injured child	Process	Assessment	Percentage of paediatric patients with weight documented in kilograms only.
			Percentage of paediatric patients with pain assessed.
			Percentage of paediatric patients with vital signs re-assessed.
		Intervention	Median time from collection of first set of vital signs to first intervention (eg, oxygen, medication).
		Disposition	ED length of stay (ED arrival to discharge*).
Timely and effective transfer to appropriate resources	Process	Disposition	Percentage of transferred paediatric patients who met the site-specific criteria for transfers.
			Time from arrival to transport.
			Percentage of transferred paediatric patients that were discharged from the receiving centre <24 hours of arrival.
Adherence to evidence-based guidelines† for management of blunt head trauma	Process	Assessment	Percentage of paediatric patients with a full set‡ of vital signs obtained.
			Percentage of paediatric patients with a Glasgow Coma Scale reassessment.
		Diagnostics	Percentage of patients with a head CT that met one or more PECARN§ criteria.
		Intervention	Percentage of paediatric patients that received hypotonic saline.
Adherence to evidence-based guidelines for seizures	Process	Assessment	Percentage of paediatric patients with a neurologic reassessment.
		Intervention	Percentage of paediatric patients that received at least one additional class of antiepileptics (for patients requiring ≥2 doses of benzodiazepines).
		Diagnostics	Percentage of paediatric patients who underwent invasive diagnostic assessments: blood glucose, blood work, urinalysis, lumbar puncture and head CT.
Adherence to evidence-based guidelines for respiratory reports	Process	Intervention	Percentage of paediatric patients with asthma or croup that received a steroid.
			Median time to steroids in patients diagnosed with asthma or croup.
			Percentage of paediatric patients ≥2 years with a diagnosis of asthma that received beta agonist.
			Median time to beta agonist administration in patients ≥2 years with a diagnosis of asthma (ED arrival to beta agonist administration).
			Percentage of patients that received an antibiotic.
		Diagnostics	Percentage of patients that underwent a chest X-ray.
Assess the timeliness and variability of interventions for vomiting	Process	Intervention	Percentage of paediatric patients that received an antiemetic.
			Time to first antiemetic (ED arrival to antiemetic administration).
			Percentage of patients that received oral rehydration.
Acute suicidality encounters	Process	Assessment	Percentage of patients who had a structured suicide screen.
			Percentage of patients with a positive suicide screen who had a structured suicide assessment.
		Intervention	Percentage of patients with a positive suicide screen who had a consultation with a licenced mental health professional.
			Percentage of patients with a positive suicide screen that received a discharge safety plan.

*For purposes of standardisation, discharge is defined to be the moment of physical departure from the ED.

†Evidence-based guidelines.

‡Includes temperature, heart rate, respiratory rate, blood pressure, pulse oximetry, mental status and pain assessment.

§Paediatric Emergency Care Applied Research Network.

ED, emergency department; PECARN, Paediatric Emergency Care Applied Research Network.

Discussion

Twenty-eight measures that capture five common clinical presentations and two cross-cutting processes of care were developed by the NPRQI with the participation of a diverse panel of experts and national organisations. These measures encompass the four phases of care in the ED: assessment, diagnostics, intervention and disposition. This approach allows for a prospective, feasible and patient-centred focus to paediatric-specific QI efforts. Low-volume EDs may never have sufficient patient encounters to focus on final diagnoses for timely QI efforts. By targeting critical cross-cutting processes and common clinical presentations, EDs can assess quality of care among a larger undifferentiated paediatric population.18 The patient experience model, as highlighted within the Institute for Healthcare Improvement’s Triple Aim, is based on the care provided in response to a clinical presentation, not a specific diagnosis.22 The measures within each clinical domain were derived from evidence-based research that links early and appropriate assessment, targeted diagnostics, timely interventions and disposition to improved outcomes.

The NPRQI represents a new framework for the inclusion of quality measures for all categories of paediatric care divided across four phases of care: assessment, diagnostics, interventions and disposition. Assessment serves as the foundation for paediatric patient safety. For example, early recognition of altered mental status is essential to timely management of head trauma.23–25 Early recognition of tachycardia and hypotension are core components of sepsis recognition.26 In turn, early recognition of the critically ill or injured child leads to timely administration of evidence-based interventions linked to improved outcomes. For example, early administration of steroids are associated with decreased hospitalisation among children with moderate and severe asthma and, among children with seizures, treatment delays can result in prolonged seizure activity.27 28 In addition to improved outcomes, timeliness of interventions improves patient experience.29 Diagnostic testing should be undertaken judiciously and in conjunction with family-centred care. Unnecessary exposure to radiation in the paediatric patient is both costly and associated with potential harm.30 31 Invasive procedures and unnecessary testing, too, can adversely impact patient experience and cost.32 Adherence to standardised, site-specific transfer criteria and processes promotes timely access to necessary resources and may minimise unnecessary cost to patients and families.33 A first step to adhering to evidence-based guidelines in specific populations is optimising care processes across all phases of ED care.

The past decade has fostered significant growth in the development of evidence-based guidelines as research in paediatric emergency care has shifted from single-centre data to multicentre studies enabling researchers to address low frequency, high-risk conditions in a more systematic manner. Multicentre research has also allowed for creation of evidence-based guidelines and validated decision rules for common paediatric reports. This approach is exemplified by the Emergency Medical Services for Children-funded Paediatric Emergency Care Applied Research Network (PECARN), which has published over 150 articles on a diverse range of paediatric emergency medicine topics.17 While much of the research from multicentre trials focuses on high risk, low-frequency events such as sepsis, some of the most prominent guidelines generated from PECARN relate to common clinical conditions encountered at virtually every ED in the USA: management of closed head injury in children, bronchiolitis management, pain management, suicidality and paediatric patient safety considerations.17 31 34–36

The NPRQI measures individually may overlap with the development of paediatric measures by others.12–14 What differs is the context in which they are implemented which focuses on clinical presentation rather than a specific diagnosis, and that they are imbedded in a set of measures that capture all ED phases of care for managing the patient, not the diagnosis. This approach can be applied across low-volume EDs where specific diagnoses are infrequent, but the phases of ED care are universal. The large proportion of NPRQI measures (28.6%) that fall within cross-cutting care processes further supports engagement by very-low volume EDs (fewer than five paediatric patients per day). These measures set the foundation for measuring the quality of ED care provided for most children in the USA. In the absence of pay-for-performance incentives, the adoption of paediatric-specific quality measures in low-volume EDs (where most children seek care) will depend on the ease of data collection, relevance to a large proportion of the paediatric population and linkage to patient-centred outcomes. These measures strive to be feasible at the local level and actionable for the ED care team.

Families rely on EDs of close proximity to meet the needs of critically ill and injured children, most of which see fewer than 15 children per day.1 37 The potential impact of quality measures on care delivery is dependent on relevance to the population served, and uptake by those who can implement change. Engagement of frontline practitioners in QI efforts is essential to ensuring the success of improvement strategies. The measures proposed herein are a first step to recognising variability in care within a single ED as the focus is optimisation of processes of care that are central to the delivery of high-quality care for all paediatric patients. The potential impact of the 28 NPRQI quality measures is that any ED can immediately improve processes of care that are linked to improved health outcomes in children.

The NPRQI measures provide a foundation for any ED care team to measure adoption of evidence-based guidelines for paediatric care using a patient-centred, provider-driven approach to QI. Unlike many quality measures that rely on administrative data or a diagnosis-based retrospective review, the NPRQI measures were designed for any ED to assess performance and improve delivery of care to the undifferentiated paediatric patient. The NPRQI paediatric quality measures may be biased from using a consensus model. It was critical to engage a diverse group to participate in the modified Delphi process as well as create an environment for sharing differing perspectives. The subset of research panellists engaged in measures proposal and final review were chosen for their clinical, research and implementation expertise to ensure feasibility. The consensus panel was composed of multidisciplinary organisational representatives that intersect with low-volume EDs. With significant input from consensus panellists and national organisations, the NPRQI measures are relevant, feasible and linked to improved outcomes for paediatric patients with common clinical presentations.

Conclusion

The majority of paediatric emergency care is sought in diverse, low-volume ED settings with variable capacity and capabilities and where the majority of patients are adults. We worked to identify relevant, feasible, patient-centred and process-focused paediatric quality measures for adoption in any ED, especially low-volume settings. The quality measures proposed herein were developed as part of the NPRQI, the operational arm of the National Paediatric Readiness Project.4

The NPRQI is an effort to optimise emergency care delivery for children regardless of site-specific resources or infrastructure. The development of relevant and applicable paediatric quality measures is the first step in implementation of paediatric QI processes. The presence of QI processes that includes paediatric-specific measures is associated with improved paediatric readiness and increased survival from critical illness and injury. The NPRQI framework differs from previous efforts in the following ways: (1) diversity and national representation of the selected consensus panel, (2) focus on the undifferentiated patient, (3) adherence to the perspective of healthcare providers in rural and community settings and (4) prioritisation of easily accessible data points. This is the first set of paediatric emergency care quality measures that supports implementation in low volume, low-resourced EDs; provider-driven improvement strategies; and encompasses all four phases of care in the ED setting. The NPRQI measures will be used to create a national open access, electronic paediatric emergency care QI data platform and registry to facilitate engagement in QI efforts, support ED providers to assess the current state of paediatric emergency care delivery and benchmark performance across similar ED settings. Future efforts will focus on establishing performance benchmarks across variably resourced EDs, geographical regions and patient demographic groups to begin to identify and address disparities in the ability of our nation’s emergency care system to meet the needs of all children.