Retrospective chart review and survey to identify adverse safety events in the emergency medical services care of children with out-of-hospital cardiac arrest in the USA: a study protocol.

INTRODUCTION: Efforts to improve the quality of emergency medical services (EMS) care for adults with out-of-hospital cardiac arrest (OHCA) have led to improved survival over time. Similar improvements have not been observed for children with OHCA, who may be at increased risk for preventable adverse safety events during prehospital care. The purpose of this study is to identify patient and organisational factors that are associated with adverse safety events during the EMS care of paediatric OHCA. METHODS AND ANALYSIS: This is a large multisite EMS study in the USA consisting of chart reviews and agency surveys to measure, characterise and evaluate predictors of our primary outcome severe adverse safety events in paediatric OHCA. Using the previously validated Paediatric prehospital adverse Event Detection System tool, we will review EMS charts for 1500 children with OHCA from 2013 to 2019 to collect details of each case and identify severe adverse safety events (ASEs). Cases will be drawn from over 40 EMS agencies in at least five states in geographically diverse areas of the USA. EMS agencies providing charts will also be invited to complete an agency survey to capture organisational characteristics. We will describe the frequency and proportion of severe ASEs in paediatric OHCA across geographic regions and clinical domains, and identify patient and EMS organisational characteristics associated with severe ASEs using logistic regression. ETHICS AND DISSEMINATION: This study has been approved by the Oregon Health & Science University Institutional Review Board (IRB Approval# 00018748). Study results will be disseminated through scientific publications and presentations, and to EMS leaders and staff through local EMS medical directors, quality and training officers and community engagement activities. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This will be among the largest studies of adverse safety events in the emergency medical services (EMS) setting regarding the care of children with out-of-hospital cardiac arrest (OHCA), and is powered to identify characteristics of patients and EMS agencies that are associated with severe adverse safety events.

The results of this study will guide education and quality improvement efforts to improve the quality of EMS care for children with OHCA.

Data abstraction and assessment of the primary outcome, severe adverse safety events, is based on review by clinical experts who have undergone rigorous training and testing in the use of a validated and published Paediatric prehospital adverse safety Event Detection System tool with significant decision support.

Like similar studies, the chart review is limited by what is documented in the medical record, and the survey is limited by self-report.

Introduction

Up to 1.6 million children are transported by emergency medical services (EMS) in the USA each year.1 EMS provide life-saving care in the first few minutes of an emergency and are an essential component of the healthcare delivery system for critically ill and injured patients. High-quality EMS care is an important factor in the survival of children with out-of-hospital cardiac arrest (OHCA), one of the leading causes of paediatric death.2 3 Although survival from adult OHCA4–8 and paediatric in-hospital cardiac arrest9 have improved over the last two decades, paediatric OHCA survival has not improved.10 11

Severe adverse safety events (ASEs), defined as unintended deaths, injuries or complications caused by healthcare management,12 can occur at any time during delivery of care and are leading causes of preventable death in the USA.13 Our prior work demonstrated that critically ill and injured children experience high rates of ASEs in the prehospital setting and that children experiencing OHCA had the highest risk for a life-threatening ASE.14–17 While OHCA cases represented only 8% of paediatric transports using lights and sirens, they accounted for 34% of severe ASEs.14 The purpose of this study is to measure and characterise the prevalence of severe ASEs in the EMS care of children with OHCA and to identify patient and organisational factors associated with severe ASEs.

Methods and analysis

Objectives

The primary objective of this study is to quantify the occurrence, variation and predictors of severe ASEs in paediatric OHCA. We hypothesise that severe ASEs, which have the potential for severe or permanent harm, will be prevalent among children receiving prehospital care for OHCA. We also hypothesise that there are specific, predictable and modifiable organisational features of EMS agencies that are associated with severe ASEs.

Study design

This is a multicentre study of children in the USA who have received EMS care for OHCA from 2013 to 2019. There are two main components to the study: (1) a chart review of EMS patient care report(s) (charts) for all paediatric OHCA cases from at least five geographically diverse US regions, and (2) a detailed survey of organisational characteristics for all EMS agencies participating in the study. We will follow Strengthening the Reporting of Observational Studies in Epidemiology guidelines in manuscripts arising from this study.18 This study began with training clinical reviewers in November 2018, and will continue through April 2021.

Study population

Eligible patients are children <18 years of age who received treatment by EMS for OHCA, defined as cardiopulmonary resuscitation (CPR) performed by EMS or defibrillation at any time (including before EMS arrival by use of a defibrillator). We will also include children who became pulseless after EMS arrival (EMS witnessed OHCA) and before arrival at the receiving hospital. We will exclude cases where there is documentation of obvious signs of death such as rigour mortis or dependent lividity, or where the patient is pronounced dead without attempts at resuscitation by EMS; resuscitation is defined as delivery of chest compressions, cardioversion or defibrillation. We will also exclude cases where resuscitation efforts ceased within 10 min of initiation. We plan to separately evaluate cases where resuscitation was stopped within 10 min to identify potential areas for improvement.

OHCA cases meeting inclusion criteria from participating EMS agencies during a 7-year period from January 2013 to December 2019 will be included. We will recruit public and private EMS agencies from at least five US cities, counties or metropolitan areas, attempting to achieve broad geographic representation across multiple regions of the USA. Cities, counties and metropolitan areas included thus far are: DeKalb, Georgia; Pittsburgh, Pennsylvania; Milwaukee, Wisconsin; Portland, Oregon; San Bernardino and Riverside, California. Each of these sites is served by 3–20 EMS agencies. Together, these agencies serve over 2.5 million children, resulting in over 1700 expected EMS-treated paediatric OHCA cases over 7 years (table 1); recruitment is ongoing. Table 1 includes demographic information for each site.

Table 1

Cities and counties represented in our study of paediatric OHCA

US census region	West—Pacific		Midwest—East Central North	Northeast—Middle Atlantic	South—South Atlantic
Metropolitan area	Portland, Oregon (OR)	San Bernardino/Riverside, California (CA)	Milwaukee, Wilsconsin (WI)	Pittsburgh, Pennsylvania (PA)	Atlanta, Georgia (GA)	Total
County(s)	Washington County, OR; Clackamas County, OR; Multnomah County, OR; Clark County, Washington	San Bernardino County, CA; Riverside County, CA	Milwaukee County, WI	Allegheny County, PA	DeKalb County, GA; Decateur County, GA; Fulton County, GA
Sex (% female, average)	51	50	55	52	52
Race (% white only, not Hispanic or Latino, average)	84	31	64	80	45
Per capita annual income (US$) (average)	37 077	24 284	28 121	36 907	32 735
Paediatric population (N)	497 182	1 195 649	226 974	227 400	413 805	2 561 010
Expected number of eligible OHCA cases (2013–2019)*	348	837	159	159	290	1793

Demographic data are from the US Census Bureau, 2018.

*Eligible based on study inclusion criteria, using incidence of paediatric OHCA of 10 per 100 000 person-years. Depending on the yield of charts, we may randomly sample from areas with a higher number of charts to achieve as even of distribution as possible across geographic areas while still meeting our goal study sample of 1500 total OHCA cases.

OHCA, out-of-hospital cardiac arrest.

EMS response in the USA is varied and consists of a mix of teams with basic life support (BLS) and advanced life support (ALS) capabilities. In many jurisdictions, an OHCA call will generate a response from more than one team, which may include a first response team and a team with the capability to transport the patient; sometimes multiple agencies will dispatch teams simultaneously. In Portland, Oregon, for example, it is customary for the fire department to dispatch an ALS team to begin resuscitation, and a private ambulance company to concurrently dispatch a second ALS team to assist with resuscitation and transport of the patient where necessary. Thus, a single cardiac arrest case may involve one or more agencies, and could generate more than one chart.

Paediatric OHCA EMS chart review

Data source

Charts included in this study adhere to the National EMS Information System reporting guidelines and include structured data elements containing demographic and medical information, treatment details (including procedures and medications) and free-text narrative.19 For cases where a second chart was generated, we will evaluate both charts after they have been matched.

Our research team includes members from three US cities (Milwaukee, Pittsburgh and Portland) with strong relationships with local EMS agencies from prior studies related to OHCA. We have also partnered with American Medical Response, the largest private provider of EMS care in the USA, to identify at least two additional US cities or counties from which to obtain charts. Currently, these include DeKalb, Georgia as well as San Bernardino and Riverside, California. Together, we will obtain charts from over 40 participating agencies representing 5 of the 10 US Department of Health and Human Services regions, improving our ability to generalise our findings to other areas of the USA. Working with these partners or directly with individual EMS agencies where needed, we have begun executing data use agreements to allow transfer of charts to our Health Insurance Portability and Accountability Act (HIPAA)-compliant Box cloud storage system (Box, Redwood City, California, USA). We will identify patient care episodes meeting our inclusion criteria. For cases where a second chart was generated, charts will be matched. Charts will be de-identified and uploaded to the server by research staff at one of our study sites or by agency personnel.

Chart review instrument

We will use the Paediatric prehospital adverse safety Event Detection System (PEDS), the first validated and published chart review tool to detect ASEs in the prehospital care of children.20 21 The PEDS tool guides users in the identification of ASEs in six different domains of EMS care: (1) assessment and diagnosis, (2) clinical decision-making, (3) procedures, (4) airway and breathing, (5) medications and (6) fluid therapy. Users will classify severity and preventability of each ASE and identify potential contributing factors. Because the PEDS tool was designed for the entire range of paediatric transport cases (eg, trauma, drowning, etc, in addition to OHCA), we will streamline the tool specifically for use in OHCA. This study will focus on severe ASEs, which are defined as having the capacity to produce severe or permanent harm.

Establishment of taxonomy of common ASEs and gold standard

Based on our prior experience identifying ASEs in paediatric OHCA,15 our study team created a taxonomy of common ASEs (online supplementary file 1). This document was subsequently edited for ease of use during clinical reviewer training and represents a consensus of eight physicians with clinical expertise in this field as well as methodological and patient safety experts from our research team. The physicians who participated in clinical reviewer training were either paediatric emergency medicine or paediatric critical care physicians from sites around the country; many with previous experience with paediatric OHCA research. For each common error in paediatric OHCA care, the taxonomy includes a definition as well as assessments of the domain of care, severity and preventability.

The gold standard for this study is a consensus review performed by three core clinical investigators who are board-certified paediatric emergency medicine or paediatric critical care physicians and participated in the development and validation of the PEDS tool. Each of these investigators has been an active part of our core investigative team for at least 4 years, and has previously reviewed at least 30 EMS charts for validation and exhibited excellent inter-rater reliability. In order to ensure a high degree of consistency, consensus review initially consisted of all three core clinical investigators independently reviewing 10 charts, followed by discussion to resolve discrepancies and achieve full agreement. Subsequent consensus review will consist of independent review by at least two of the three core clinical investigators, followed by discussion to resolve disagreements. When necessary, the third core clinical reviewer will serve as the arbiter.

Data collection process

Data are collected and managed using Research Electronic Data Capture (REDCap), an electronic data capture tool hosted at OHSU.22 23 REDCap is a secure, web-based software platform designed to support data capture for research studies, providing 1) an interface for validated data capture; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads and 4) procedures for data integration and interoperability with external sources. Non-medical research staff have been trained by study investigators to read charts and to abstract data for the first section of the PEDS tool, which focuses on items that do not require clinical judgement, such as age and weight, timing of events including procedures, dosing and route of medication administration, etc. Clinical reviewers will confirm correctness of non-medical staff data entry for all charts, and will abstract the remaining data, including determining the presence of ASEs and their severity and preventability.

Reviewer training

All clinical reviewers will undergo reviewer training to standardise the chart review and abstraction processes. Reviewer training consists of an initial training session focused on the use of the PEDS tool, use of the REDCap data entry system and description of common pitfalls to avoid. Reviewers will then undergo two practice review rounds as part of training; in each round, they will review two to three training charts independently, then receive feedback and additional training based on their responses from the study team leadership.

After two rounds of practice review and feedback, each reviewer will review 10 charts. Responses will be compared with the gold standard consensus review of these charts performed by our core clinical investigators (described previously). For each patient care episode, reviewers may agree or disagree with the gold standard consensus review on the presence or absence of a severe ASE in each of the six domains of EMS care. Each clinical reviewer must meet a threshold for agreement with the gold standard before they will be able to perform independent review; this threshold is an overall agreement of at least 80% on the presence or absence of severe ASEs across the 10 charts, and at least 70% on the presence or absence of severe ASEs in each domain of care. Reviewers who do not meet both of these criteria for their first 10 charts will continue reviewing 10 new training charts at a time until they meet this agreement threshold.

Quality assurance

Once reviewers have achieved threshold agreement to begin independent chart review, they will be randomly assigned charts to review using the RAND function in Microsoft Excel. Initially, 20% of each reviewer’s charts will be randomly selected for consensus review by our core clinical investigators. Once a reviewer has completed 50 chart reviews with 80% agreement in the presence or absence of a severe ASE in each domain of EMS care, 10% of subsequent charts will be randomly assigned for consensus review throughout the duration of the study. If at any time during the quality assessment process agreement falls below 80%, additional reviewer feedback and training will be required. Reviewers also will be given the option to flag a chart for consensus review by our core clinical investigators.

EMS agency survey of organisational characteristics

We have developed a 16-item agency survey to capture organisational-level characteristics of EMS agencies that may be associated with severe ASEs in the care of children with OHCA (online supplementary file 2). Initial survey items were developed by our multidisciplinary research team consisting of paediatric emergency medicine or critical care physicians, patient safety experts, quantitative and qualitative research experts (including a statistician), EMS medical directors and research staff with expertise in data entry, analysis and project coordination. Survey items focus on agency size, OHCA response characteristics (eg, number of responding agencies, number of responding ALS and BLS units), annual volume of calls for patients aged <18 years, information resources available in the field, paediatric education, paediatric emergency care coordination and quality improvement and quality assurance processes used for paediatric care. We reviewed and edited the items for clarity, removed duplicate items and reordered items to improve the flow of the survey. The preliminary survey was then reviewed by two EMS medical directors not part of our study team, and items further edited based on input from these experts. The final survey contains 16 items, with a combination of closed and open response questions, including binary responses, multiple choice and free text boxes.

EMS agency operational leaders (eg, training officers) will be identified through EMS leadership contacts in cities where our research team is based and through American Medical Response for the remainder of study sites. Our study team members who are national EMS experts will aid in identification and recruitment of subjects as needed. We will contact EMS agency leaders by email and ask them to complete the survey using REDCap; after two initial attempts to contact agency leaders by email, we will contact them by telephone. If necessary, our study team will administer the survey by telephone and enter the data directly into REDCap on behalf of an agency. Where repeated attempts to contact an individual operational leader fail, we will contact additional leaders within each EMS agency following the same process.

Our study team will review each individual survey, and we will contact survey respondents to clarify any illogical or incomplete responses.

Variables

Table 2 describes key predictor and outcome variables that will be obtained from chart review and EMS agency survey. The primary outcome of the study is the presence or absence of a severe ASE, and will be obtained from chart review. A severe ASE is defined as an ASE with the capability to produce severe or permanent harm. The chart review will also capture variables describing patient demographics (eg, age, sex, weight, presence of chronic illness), the OHCA event (eg, initial cardiac rhythm, location, date, bystander CPR or defibrillation) and the EMS agency factors (eg, agency identifier, type of agency, treatment and response times, specific treatments provided). The EMS agency survey captures variables describing the agency’s size, annual paediatric call volume, response model for OHCA, information resources available in the field, level of coordination with other agencies if multiple agencies respond to OHCA events and pediatric-specific training and quality improvement activity.

Table 2

Key predictor and outcome variables

	Variable	Source	Variable type
Outcome variables	Presence of severe ASE in each domain of EMS care	Chart review	Binary
	Preventability of ASE	Chart review	Categorical
Patient-level explanatory variables	Patient age	Chart review	Categorical
	Patient sex	Chart review	Categorical
	Witnessed arrest	Chart review	Binary
	Bystander CPR	Chart review	Binary
	Presenting rhythm	Chart review	Categorical
	EMS response time	Chart review	Continuous
	EMS scene time	Chart review	Continuous
	Study site	Chart review	Categorical
Organisation-level explanatory variables	Number of ALS-trained providers on scene for OHCA response	Agency survey	Continuous
	Number of total providers on scene for OHCA response	Agency survey	Continuous
	Information resources available	Agency survey	Categorical
	Response model	Agency survey	Categorical
	Number of hours of annual paediatric training required	Agency survey	Continuous
	Level of training of paediatric care champion	Agency survey	Categorical
	Annual paediatric volume of ‘lights and sirens’ calls	Agency survey	Continuous

ALS, advanced life support; ASE, adverse safety event; CPR, cardiopulmonary resuscitation; EMS, emergency medical services; OHCA, out-of-hospital cardiac arrest.

Analyses

We will describe the frequency and proportion of severe ASEs in paediatric OHCA across regions and domains of EMS care (eg, airway and breathing procedures, clinical decision making, medications), and compare their distributions using χ2 tests. Logistic regression will be used to estimate odds of severe ASEs at the patient-level, based on patient and EMS agency organisation characteristics. We will account for agency-level clustering using correlated data regression models for binary data. We will evaluate key predictors selected for inclusion in the model based on a priori knowledge, testing associations between each predictor and ASEs, with adjustment for confounding in multivariable models. The number of cases with missing data for key predictors or primary outcome variables is anticipated to be small. However, we will evaluate the impact of missing data and robustness of our fitted models via sensitivity analysis.

Sample size considerations

Power and sample size computations were based on Generalized estimating equation (GEE) logistic regression models. We estimated the power to detect an OR of 2.0 for a binary organisation-level covariate under different assumptions at 0.05-significance level. An OR of 2.0 is a conservative estimate based on our prior chart review study and recent literature.10 24 Power calculations were performed with the following assumptions: the proportion of cases with severe ASEs is expected to be 0.67 (range 0.50–0.75),15 the proportion of agencies with a multi-agency OHCA response (as an expel of a binary predictor) will be 0.50 (range 0.20–0.80), intra-agency correlation is expected to be small with a range of 0.01–0.02 and there will be approximately 35 agencies with the average number of cases from each agency being the anticipated number of cases from that agency’s site divided by the number of agencies in that site. For a total sample size of n=1000 and the range of the proportion of cases with severe ASEs, binary predictor distributions and intra-agency correlations noted above, the power to detect an OR of 2.0 exceeds 90%. The sample size we anticipate collecting is 1500 OHCA cases.

Patient and public involvement

EMS stakeholders have been closely involved in our prior research as well as in the design of this study. As our understanding of ASEs advances, we plan to invite patients and families to help us develop our dissemination strategy and future research.

Ethics and dissemination

The Oregon Health and Science University Institutional Review Board has approved this study (IRB Approval# 00018748). All proposed research involves analysis of either existing electronic charts or surveys from EMS providers and agency staff. This study does not involve an intervention or attempt to alter usual care during its course, therefore the overall risk to patients and EMS agencies from participation in the study is minimal and relates solely to the potential for breach of confidentiality.

Transmission and handling of charts will be consistent with data use agreements. Each chart will be assigned a unique identification number for this study. Reviewers will only have access to charts they are assigned to review. Charts will not be downloaded to local computers for viewing by reviewers, but rather stored and viewed on our cloud-based Box storage system. Names and other identifying information from EMS agency surveys will not be shared. Chart review and survey data will be stored in a web-accessible password-protected Research Data Capture database housed on an OHSU secure server behind the institutional firewall. Access to data generated is restricted to study personnel. Study staff will be trained in standard institutional practices to maintain the confidentiality and security of data collected in this study. Data will be kept for 7 years past the close of the study. After the 7-year period, all study data will be destroyed. We will only publish aggregate results in manuscripts that result from this study.

To maximise the impact of this work and improve EMS care for children with OHCA, we will disseminate our findings in several ways. Research findings will be disseminated through publication in research journals and presentations at national scientific meetings. In addition, we will share our findings with EMS leaders and medical directors through participation in national meetings of these groups, and at state and local meetings where possible. We believe strongly that agencies participating in this research should learn as much as possible about practices associated with fewer ASEs, and we will share these findings with participating agencies.