A chart review tool to systematically assess the safety of prehospital care for children with out-of-hospital cardiac arrest.

Objective: Create an easy-to-use pediatric out-of-hospital cardiac arrest (OHCA)-specific chart review tool to reliably detect severe adverse safety events (ASEs) in the prehospital care of children with OHCA.
Methods: We revised our previously validated pediatric prehospital adverse event detection system (PEDS) tool, used to evaluate ASEs in the prehospital care of children during emergent calls, to create an OHCA-specific chart review tool. We developed decision support for reviewers, reviewer training, and a dedicated section for chart data abstraction. We randomly selected 28 charts for independent review by 2 expert reviewers who determined the presence or absence of a severe ASE for each care episode and identified the domain of care and preventability for each ASE. We calculated inter-rater agreement in the assessment of the presence or absence of a severe ASE using Gwet's first-order agreement coefficient (AC1).
Results: The PEDS-OHCA chart review tool has 6 sections, with a minimum of 70 and maximum of 667 total possible fields. We found inter-rater agreement of 0.83 (95% confidence interval, 0.63-0.99) between our 2 reviewers for the overall detection of a severe ASE and an average time to complete of 8 minutes (range, 2-25 minutes). Inter-rater agreement in the detection of a severe ASE in each individual domain ranged from 0.36 to 0.96. Conclusions: The PEDS-OHCA is the first chart review tool to systematically evaluate the safety and quality of EMS care for children with OHCA. This tool may help improve understanding of the quality of EMS care for children with OHCA, which is essential to improving outcomes.

INTRODUCTION

Background

Pediatric out‐of‐hospital cardiac arrest (OHCA) is a major public health problem, with over 23,000 children experiencing OHCA each year in the United States, of whom approximately 6%–12% survive. , To survive OHCA, children need high‐quality treatment throughout the continuum of care, including the essential early resuscitation provided by emergency medical services (EMS) personnel before arrival at a health care facility. Unfortunately, this critical component of care for children with OHCA is understudied and poorly understood, which may partly explain why outcomes for children with OHCA have not improved even as outcomes for adults with OHCA have. ,

Importance

Retrospective chart review is an important tool to understand care quality and has been used to systematically characterize adverse safety events (ASEs) and identify opportunities for improvement in emergency medicine , and inpatient care. , , We previously developed and validated the pediatric prehospital adverse safety event detection system (PEDS) to assess ASEs in the EMS care of children needing emergency care and identified that the incidence of a severe ASE occurring were increased when the call included cardiopulmonary resuscitation.

Goals of this investigation

The goal of this investigation was to create an OHCA‐specific chart review tool for the identification of severe ASEs in EMS care of children based on our successful experience developing the broader PEDS tool for EMS quality and safety.

METHODS

As a part of a multi‐part mixed methods study, we developed an OHCA‐specific version of our previously published PEDS tool and evaluated the reliability and usability of this new tool. We have previously described the protocol for this study, which adheres to STROBE guidelines and involves a detailed review of EMS electronic charts and a survey of EMS agencies. This study was approved by the OHSU Institutional Review Board (STUDY00018748).

Tool modification and decision support

Our team of investigators, which includes experts in pediatric emergency medicine and critical care, added 15 cardiac‐arrest specific fields, including Utstein variables, , , , , to our previously validated PEDS tool. The original PEDS tool contained 36 fields and covered the entire spectrum of clinical cases, whereas the current tool is for OHCA, exclusively. The added variables were selected based on the Cardiac Arrest Registry to Enhance Survival (CARES) guidelines for OHCA data collection. Added variables included whether the arrest was witnessed, whether a bystander performed cardiopulmonary resuscitation (CPR) or defibrillation before EMS arrival, the etiology of the cardiac arrest, and if return of spontaneous circulation was achieved. To increase efficiency for clinical reviewers, we moved many existing data elements that could be abstracted by a non‐clinical data abstractor (eg, patient demographics, EMS scene time, interventions performed, and the timing of these interventions) into a separate section. Reliability between abstractors was achieved through training sessions with clinical experts (C.O.E., M.H., and G.M.) and evaluated through multiple rounds of comparison of agreement and discussion. The details of this process are noted elsewhere. We created data fields to identify details of key interventions, including whether an intervention was indicated, delayed, or required multiple attempts to successfully complete. We made subsequent minor edits to the PEDS‐OHCA tool based on input from our team of 6 chart reviewers who are experts in pediatric emergency or critical care medicine.

To decrease subjectivity in the assessment of ASEs, we created a decision support matrix based on current PALS and NRP guidelines as well as the clinical expertise of our study team and chart reviewers who are pediatric emergency or critical care physicians or fellows (Table S1). This matrix is organized according to domains of care in the PEDS‐OHCA tool, and identifies the criteria, ASE subtype, severity, and preventability for common ASEs. We focused on ASEs identified in prior work examining pre‐hospital care for OHCA. ,

The Bottom Line

The authors developed and tested a chart review tool to identify severe adverse safety events during the out‐of‐hospital care of children with cardiac arrest. Averaging 8 minutes per chart and with substantial inter‐rater agreement, this tool provides a systematic approach to evaluate the safety and quality of emergency medical services care for pediatric cardiac arrest.

Using REDCap electronic data capture tools hosted at Oregon Health and Science University, , we enhanced decision support for clinical reviewers by using key elements of data abstracted by non‐clinical reviewers and displayed the information in the corresponding domain of the tool. In the procedures and airway domains, this included patient age and weight, type of vascular access, number of attempts and success in establishing access, electrical therapies applied, airway procedures performed, size of equipment used, and the time between arrival on scene to when key interventions were performed (Figure 1). For the medications and fluids domains, this included medications administered, dose, concentration, route, and also the time between arrival on scene and first dose. Based on interventions, dosing, and times from the charts, notices appear in REDCap when there appears to be a deviation from the guidelines in the ASE detection matrix (e.g., medication dose given is too high or low for the child's weight, or time between arrival on scene and starting bag‐valve‐mask ventilation is too long).

Example of the PEDS‐OHCA tool and decision support in REDCap

Tool testing and inter‐rater agreement

We randomly selected 28 charts from our population of over 1000 OHCA cases from January 2013 through December 2019. After preliminary review of 20 charts by each reviewer, we determined that review of 30 charts would provide an appropriate sample size to generate meaningful point estimates and confidence intervals. A total of 2 charts were unusable due to EMS reporting errors and missing data in the charts. We received EMS charts describing care episodes for children with OHCA from at least 5 United States cities, counties, or metropolitan areas and over 40 EMS public and private agencies. Randomization was performed in Microsoft Excel using the RAND() function. Each chart was independently reviewed by 2 core clinical investigators (M.H. and G.M.), who determined the presence or absence of a severe ASE for each care episode, and identified the domain of care and preventability for each ASE. Due to low prevalence of severe ASE, we calculated inter‐rater agreement using Gwet's first‐order agreement coefficient (AC1) to overcome the “kappa paradox” limitation of other approaches, such as Cohen's kappa or Fleiss kappa (ie, these measures do not behave well when the prevalence is near 0 or 1). , , The 95% confidence interval (CI) for the agreement coefficient was also obtained. We applied this approach to identify the level of agreement in the determination of a severe ASE for each care episode and within each domain of care. Each lower bound of 95% CI was compared to the following Landis and Koch benchmark limits to determine a level of agreement: 0–0.20, slight; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, substantial; and 0.81–1, almost perfect agreement. Using the lower bound of the 95% CI to determine these levels is a more conservative approach that allows us to interpret the results with stronger certainty.

RESULTS

The resulting PEDS‐OHCA chart review tool contains 6 sections, with a minimum of 70 possible fields. Branching logic elicits details of treatments (eg, endotracheal tube size, medication route, etc) with a maximum of 667 total possible fields (Table 1). To decrease the time burden on clinical reviewers, the first section can be completed by a data abstractor without a clinical background, such as a research assistant. In the remaining sections, clinical reviewers identify ASEs in each of the following domains: (1) assessment, impression, and/or diagnosis; (2) clinical decision‐making; (3) non‐airway procedures; (4) airway interventions; (5) medications; and (6) fluids. Each identified ASE is categorized by type (unintended injury or consequence, near miss, suboptimal action that can be improved, error, or management complication), potential to cause harm (no harm likely or near miss, mild temporary harm including additional treatment or adverse effect from unnecessary treatment, or permanent or severe permanent harm including death), and preventability (4‐point scale from not preventable to preventable).

TABLE 1

PEDS‐OHCA tool sections and field elements

Section name	Description	Minimum no. items	Maximum no. items	Free text, no. items	Multiple choice, no. items	Ordinal scale, no. items	Other
Case identification and background	Background, patient characteristics, timing of events, pre‐EMS and EMS interventions	40	340	107	106	99	28
Clinical background	Characteristics of the OHCA including Utstein variables, patient demographics, and outcomes	6	14	6	7	0	1
Assessment, impression/diagnosis	Details of safety events related to assessment, diagnosis, and decision‐making, as well as resuscitation protocols	2	60	3	45	2	10
Clinical decision‐making
Procedures	Details of safety events related to procedures and airway interventions	15	131	20	61	2	48
Airway
Medications	Details of safety events related to medications and intravenous fluids	3	99	6	31	2	60
Fluids
Overall	Primary factor associated with safety events and ranking of each domain's contribution to safety events	5	23	6	3	6	8
Total		70	667	148	253	111	155

Abbreviations: EMS, Emergency Medical Services; OCHA, Out of hospital cardiac arrest.

The average time to complete the clinical review of the charts was 8 minutes (range = 2–25 minutes). Reviewers identified a total of 63 severe ASEs, most of which were in the medications and procedures domains (Supplemental File 1). We found substantial inter‐rater agreement of 0.83 (95% CI = 0.63–0.99) between our 2 reviewers for the overall detection of a severe ASE in a care episode (Table 2). Estimated inter‐rater agreement in the detection of a severe ASE in each individual domain ranged from 0.36 to 0.96. The individual domain with almost perfect agreement was fluids (0.96; 95% CI = 0.88, 0.99). Next highest with substantial agreement was assessment, impression, and diagnosis (0.84; 95% CI = 0.65, 0.99). Agreement was moderate for non‐airway procedures (0.69; 95% CI = 0.41, 0.98) and medications (0.79; 95% CI = 0.55, 0.99), fair for clinical decision‐making (0.59; 95% CI = 0.28, 0.91), and slight for airway procedures (0.36; 95% CI = 0.01, 0.74).

TABLE 2

Inter‐rater agreement using Gwet's AC1

	Agreement (95% CI)	Level of agreement
Overall	0.83 (0.63, 0.99)	Substantial
Individual domains
Assessment, impression, and diagnosis	0.84 (0.65, 0.99)	Substantial
Clinical decision‐making	0.59 (0.28, 0.91)	Fair
Procedures (non‐airway)	0.69 (0.41, 0.98)	Moderate
Airway procedures	0.36 (0.01, 0.74)	Slight
Medications	0.79 (0.55, 0.99)	Moderate
Fluids	0.96 (0.88, 0.99)	Almost perfect

LIMITATIONS

This study has several recognized limitations. First, there is no gold standard in the determination of an ASE to which we can compare our tool beyond expert opinion. Although our clinical reviewers are experts in this area, we acknowledge there may be differences in opinion among experts when assessing whether an ASE occurred. We have focused on detecting severe ASEs to maximize reviewer agreement that the events we detect are clinically important, and we have created robust clinical decision support to further improve reliability. Although we have not yet reached ideal inter‐rater agreement, it is our hope that this tool and its accompanying decision support can be a catalyst in evaluating EMS quality of care during pediatric OHCA. Further evaluation of this tool is needed to establish validation.

Second, our finding of substantial inter‐rater reliability in the assessment of whether a severe ASE occurred during the care episode was based on review by experts in pediatric emergency medicine and EMS care for children, and we have not evaluated agreement among less experienced reviewers. In our ongoing study, we have developed a robust system for training clinical reviewers and assessing agreement to ensure that we have adequate reliability. However, external generalizability is unknown as this tool has not yet been tested outside of our research group. In addition, the relatively small sample size in this study resulted in wide confidence intervals. Our choice of sample size was based on informal review rather than formal power calculations; our outcome measure was rare enough to not meet an assumption of normality, making standard sample size tools not appropriate for our purposes. Future studies with larger sample sizes and diverse reviewers may resolve these limitations.

Last, all retrospective chart reviews are limited by the possibility of inaccurate or incomplete documentation in the chart, which can lead to underestimation of ASEs as compared to direct observation; conflicting information from multiple data sources or charts; changes to protocols over time; and the possibility of reviewers drawing different conclusions from the same chart. , Our team has attempted to mitigate these issues through creating a standardized approach to reconcile discrepancies and providing reviewers with the protocols relevant to the years of the study.

DISCUSSION

We have created and validated the first tool to systematically identify ASEs in the EMS care of children with OHCA. Our tool demonstrated substantial agreement in assessing the presence of severe ASEs, which have the potential to cause severe permanent harm or death. This is an essential first step in preventing morbidity and mortality from safety events during care for children with OHCA. In addition, this tool enables clinicians to systematically review the entire patient encounter, and determine the severity and preventability of ASEs in a relatively short period of time compared to traditional in‐hospital chart review. This work also acknowledges the difficulty in obtaining, and need for, high‐quality pre‐hospital documentation. Time pressures and other factors are at odds with comprehensive documentation, yet documents are often a key method of communication among clinical providers. Our research demonstrates the importance of research in this area.

Survival from pediatric OHCA is dismal. Less than 10% of children with OHCA survive to hospital discharge, with infants and young children having greater risk of mortality compared to teenagers. Although survival from OHCA in adults and in‐hospital cardiac arrest survival in children has improved since the early 2000s, , pediatric OHCA survival has not. , Previous studies have identified the need for longer CPR as a risk factor for mortality and worsened neurologic status, highlighting the critical importance of high‐quality resuscitation by EMS teams to improving outcomes. , Our prior work demonstrated that among children with life‐threatening emergencies, patients with OHCA were at highest risk for a severe ASE. , Additionally, almost half of ASEs in the EMS care of these patients were preventable, indicating that there are many opportunities to improve care during this critical period.

Our PEDS‐OHCA tool displayed substantial inter‐rater agreement in the identification of whether a severe ASE occurred during the care episode, a critical step to improving EMS care for children with OHCA. This substantial inter‐rater agreement compares favorably to agreement reported for other chart review tools. , , Many hospital studies of ASEs use trigger tools rather than a complete review of the chart. , Because the EMS chart is considerably shorter than most hospital records, clinical reviewers were able to review the entire chart in an average of 8 minutes while assessing the severity and preventability of ASEs. The ability to reliably evaluate a prehospital chart in a relatively short period of time makes this tool valuable in assessing the critical time before arrival to a health care facility for pediatric patients in cardiac arrest. In addition to its use as a research tool, the PEDS‐OHCA tool may be used by EMS agencies to identify opportunities for improvement as part of a robust, blame‐free quality improvement program. We have made the PEDS‐OHCA tool and its accompanying decision support publicly available to facilitate this use.

We also demonstrate a novel approach to analyzing agreement between reviewers when the prevalence is close to 0 or 1. In light of the well‐documented limitations to Cohen's kappa in the literature, , , , , , Gwet developed a chance‐corrected agreement coefficient that performs better than Cohen's kappa when the prevalence is close to 0 or close to 1. , , We also employed alternative approaches, such as Brennan and Prediger, Scott/Fleiss' pi, and Krippendorff's alpha, as a method of comparison. , , , , , , , However, we ultimately report Gwet's AC1 as the measure is more robust in the case that the prevalence is close to 0 or 1 and is less prone to chance‐agreement. , , ,

We encountered several obstacles during this testing process, including but not limited to: data quality, standard methods to replicate, novel approach to analysis, and small sample of charts given rarity of events. Additionally, the airway domain showed only slight agreement between reviewers. On further investigation, we believe that the lack of agreement may be due to differing reviewer interpretations of the documentation (e.g., assuming that CPR implies bag‐valve mask occurred). Other disagreement may stem from discrepancies in computer‐generated time stamps compared to narrative free‐text times. Although we have taken steps to mitigate these challenges, there is room for improvement to ensure a standardized approach to each chart.

Even with this standardized approach, we identified differences in how our reviewers assessed whether there was an ASE related to failure to ventilate within 2 minutes of arrival on scene, leading to weak agreement in the airway procedures domain. This may be because assessing this ASE relies on documentation from the beginning of the encounter when teams are trying to accomplish multiple goals and accurate documentation may be particularly challenging. As a result of this finding, we have further strengthened our guidance to reviewers.

Although we have not yet reached complete agreement and acknowledge the need for further evaluation of the data collection tools and methods for this study, we believe that the novel methods of this study and exposure of barriers faced provides a deeper look into the challenges associated with reviewing pre‐hospital pediatric OHCA.

Our results confirm the importance of reviewer training, decision support, and the need for a standardized approach to abstraction of data to ensure strong agreement. Previous chart reviews have also achieved good agreement through similar emphasis on abstractor training, abstractor monitoring, systematic data collection, and decision support. However, these methods are not widely applied to emergency medicine research. The ability to identify and characterize errors during pediatric OHCA will rely on the use of these methods for future studies.

We have created the PEDS‐OHCA, the first chart review tool to systematically evaluate the safety and quality of EMS care for children with OHCA. Our tool displayed substantial agreement between reviewers in the identification of severe ASEs, which have the potential to cause severe permanent harm or death. This tool is an important step forward in better understanding the quality of EMS care for children with OHCA, which is essential to improving outcomes.

AUTHOR CONTRIBUTIONS

The authors confirm contribution to the paper as follows: study conception and design: Carl O. Eriksson, Garth Meckler, Matthew Hansen, and Jeanne‐Marie Guise; data collection: Amanda Schoonover and Tabria Harrod; analysis and interpretation of results: Amanda Schoonover and Thuan Nguyen; draft manuscript preparation: Amanda Schoonover and Carl O. Eriksson. Jeanne‐Marie Guise will take primary responsibility. All authors reviewed the results and approved the final version of the manuscript.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

SUPPORTING INFORMATION

Click here for additional data file.

SUPPORTING INFORMATION

Click here for additional data file.