Development and validation of a novel image quality rating scale for echocardiography during cardiac arrest.

OBJECTIVE: Research into echocardiography (echo) during cardiac arrest has suffered from methodological flaws that limit aggregation of findings. We developed and validated a novel image rating scale for qualitative analysis of echo images obtained during resuscitation.
METHODS: A novel 5-point ordinal rating scale was developed and validated using recorded echo images from 145 consecutive cardiac arrest patients. Recorded echo images were reviewed in a blinded fashion by investigators experienced in cardiac arrest echo, and image quality was rated using this scale. Cardiac activity was subsequently classified as no activity, disorganized activity and organized activity. The primary outcome was inter-rater agreement using the image quality rating scale. Secondary outcome was the qualitative evaluation of the type of cardiac activity.
RESULTS: A total of 235 ultrasounds were analyzed by study investigators using the image quality rating scale. The overall image quality agreement between reviewers using the scale was good with a weighted kappa of 0.65. Agreement for image quality in subxyphoid images was greater than in parasternal images (0.65-0.52). Echo analysis of cardiac activity showed no activity (33%), disorganized activity (18%), and organized activity (49%). Agreement was great for presence or absence of "cardiac activity" and "organized cardiac activity" with a kappa of 0.84 and 0.78.
CONCLUSIONS: A novel image quality rating scale for echo during cardiac arrest demonstrates substantial agreement between reviewers. Agreement regarding the presence or absence, as well as the organization of cardiac activity was substantial.

Introduction

Focused trans-thoracic echocardiography (echo) during advanced cardiac life support (ACLS) has been proposed as a tool with potential to improve outcomes following cardiac arrest. The main applications of echo during cardiac arrest include identification of patients with poor prognosis related to cardiac activity and identification of potentially treatable pathologies.2, 3 Focused transthoracic echo during ACLS often involves a single view of the heart obtained during a 10 s (or less) pause in CPR, most commonly a subxyphoid, apical or parasternal view. In addition to providing diagnostic information regarding potentially treatable causes of the arrest, some observational evidence suggest that echo may help in the interpretation of electrocardiographic (ECG) rhythms by providing an image of the heart during CPR pauses.1, 4

Inconsistent definitions of cardiac activity and variable quality of echo images across studies represent a source of significant confounding, limiting the interpretation of the role of echo and the translation of this knowledge into resuscitation practice. The current study aimed to address some of these concerns by developing and evaluating a novel image rating scale designed to standardize the analysis of echo image quality. While other image rating scales have been described, no cardiac arrest-specific scale has been proposed and evaluated. Using a prospectively collected database of echo images, we assessed the agreement in the analysis of images between clinicians using this scale.

Methods

Study design and setting

This was a pilot study to describe and internally validate an image quality rating scale to assess the quality of echo images obtained during cardiac arrest resuscitations. A 5-point ordinal rating scale was developed using a modified Delphi technique. An initial rating scale was created by one of the investigators reviewing echo images generated from a large cardiac arrest study previously conducted. Iterative improvements of the scale were performed by a group of 2 experts in point of care echo and extensive experience in cardiac arrest echo. The two investigators conducted several independent, iterative reviews of the scale refining the content and language before coming together in three conference calls to address points of disagreement and ascertain consensus. After the 4th iteration, the two experts came to a final consensus and produced the final scale used in this study. The scale was validated through a secondary review of prospectively enrolled patients with recorded images from a cohort of consecutive non-traumatic cardiac arrest patients from a single emergency department (ED). Data were uploaded into a REDCap database (Research Electronic Data Capture, Vanderbilt University, Nashville, USA),6, 7 with the exception of echo images that were organized on a research computer. The hospital’s institutional review board approved this study as exempted of informed consent. The study was registered on Clinical Trials.Gov (NCT04248985).

Patient population

For the validation cohort, adult patients with atraumatic out-of-hospital cardiac arrest (OHCA) presenting to the ED with ongoing CPR were enrolled. The current study involves a pre-defined secondary endpoint that was part of a larger prospective study on ultrasound in cardiac arrest currently under review for publication. One hundred and sixty-eight patients presented during the 14-month study period, with 145 patients included in the study. Patients not included (n = 23) were excluded because of lack of recorded echo images. There were 51 clinicians involved in obtaining ultrasound images during this study.

Development of image quality rating scale

The image quality rating scale was developed using a modified Delphi technique through iterative improvements in the image-rating tool by experts in point of care echo. One of the investigators (RG) has over two decades of practice of emergency echo, having performed or interpreted over 10,000 studies and was the lead investigator in the largest study to date evaluating the role of echo in cardiac arrest resuscitation. The other investigator (FT) has extensive experience in emergency and resuscitation echo and is a testmur of the Critical Care Echocardiography board examination (CEEeXAM). An additional expert (TG) was used in the validation phase of this study, fellowship trained in ultrasound and currently the division director of the ultrasound division at UMASS Memorial Medical Center. A series of elements related to image quality were considered in order of ascending value for decision making during resuscitation; (1) Ability to identify the presence or absence of cardiac activity, (2) Ability to characterize the type of cardiac activity (e.g. organized vs disorganized), and (3) Ability to identify potentially treatable pathologies (e.g. cardiac tamponade or intra cardiac thrombosis). Of note, the inability to view areas of interest by either omission or by artifact results in a similar poor rating. The number and degree of image quality categories were chosen and ranked based on ability to detect the findings of importance, with a focus on minimal amount of detail to detect pathology or anatomy. The resulting image rating scale uses a 5-point Likert scale with image quality ratings from poor to excellent: 1 — unable to interpret, 2 — sufficient to detect only if the heart is beating, 3 — sufficient to determine the quality of cardiac activity (organized vs. disorganized), 4 – sufficient to visualize internal details of heart (inner myometrium, valves, pathological findings), 5 — sufficient for quantitative analysis (Fig. 1).

Fig. 1

Image rating scale.

Cardiac activity definition

The definition of cardiac activity for this study was based on previously published research.1, 9 Cardiac activity was defined as “any movement of the myocardium not including isolated valve movements or movement of blood in cardiac chambers”. Organized cardiac activity was defined as “movement of the myocardium with change in size of the ventricular cavity and synchronized movement of the ventricular wall” Disorganized cardiac activity was defined as “movement of the myocardium without change in size of the ventricular cavity, e.g. agonal or twitching activity”.

Echocardiography

Imaging was limited to transthoracic echo (TTE) performed during ACLS pauses for pulse checks or rhythm checks. The physician performing the resuscitation recorded the echo images in real-time during the resuscitation. Recorded images were reviewed independently by two physician experts in point of care echo blinded to each other and any clinical or therapeutic information. Image quality was assessed using the image quality rating scale. Images were subsequently categorized as having any cardiac activity or not, and further categorized as having organized cardiac activity or non-organized movements.

Outcome

The primary outcome was agreement of reviewers for image quality rating. Secondary outcome was agreement of reviewers for characterization of cardiac activity using established definitions.

Statistical analysis

Data were collected from the electronic medical record using a standardized data sheet and transcribed into an electronic database. Data elements included in the study database were chosen based on previously published recommendations. Data collection methods were discussed in our prior publication. Data included resuscitation-related data points, patient data points, and echo data points. The database was complete at the time of analysis with no missing data. Demographics and clinical data are presented as means with 95% confidence interval (95%CI). Agreement is provided using Weighted Kappa or Cohen’s Kappa. Interpretation of kappa analysis was consistent with prior publications. The analyses were performed using JMP Pro version 15 (SAS Institute Inc, Cary, NC).

Results

Two hundred thirty-seven echos were performed, 49% subxyphoid view and 51% parasternal view. Patient characteristics are detailed in Table 1. Agreement for image quality between the reviewers was substantial (weighted kappa 0.66 (95%CI 0.60–0.72). The average image quality was 2.9 out of 5 (95%CI 2.75–3.03). Agreement for each of the image quality levels differed, with increased agreement for the top and bottom of the rating scale, with highest agreement for the poorest image quality. There was no difference in agreement when analyzing by echo view (subxyphoid Kappa 0.66 (95%CI 0.58–0.73) vs parasternal 0.66 (95%CI 0.56–0.76)). Agreement between reviewers was moderate for patients with cardiac activity (kappa 0.55–95% CI 0.45–0.65) and substantial those without cardiac activity (kappa 0.61–95%CI 0.49–0.72). The data for each image quality level is detailed in Table 2. Image quality was negatively affected by cardiac activity, with decreased image quality in patients with no cardiac activity.

Table 1

Patient characteristics.

Clinical variable	(n = 145)
Gender, male – n (%)	97 (67.4)
Age - mean, (95% CI)	57.7 (56–60)
Initial rhythm
Asystole – n (%) – 95%CI	68 (46.9) – 39.0–55.0
PEA – n (%) – 95%CI	44 (30.1) – 23.4–38.3
Sinus – n (%) – 95%CI	3 (2.0) – 0.4–6.2
Vfib – n (%) – 95%CI	30 (21.0) – 14.9–28.0
Witnessed Arrest – n (%)	74 (51.7) – 43.0–59.0
Bystander CPR – n (%)	72 (50.4) – 41.6–57.7
ED presenting rhythm
Asystole – n (%) – 95%CI	64 (46.3) – 38.2–54.6
PEA – n (%) – 95%CI	57 (41.3) – 33.4–49.6
Sinus – n (%) – 95%CI	4 (2.9) – 1.1–7.2
Vfib – n (%) – 95%CI	13 (9.4) – (5.6–15.5
Mechanical compression device pre-hospital n (%) – 95%CI	108 (74.4) – 66.8–80.9
Endotracheal intubation pre-hospital n (%) – 95%CI	103 (72.0) – 64.1–78.7
Arrest length in min – AVG, (95% CI)	15.6 (13.6–17.7)

Table 2

Characteristics by image quality.

Clinical variable	Image rating 1 (n = 35)	Image rating 2 (n = 44)	Image rating 3 (n = 70)	Image rating 4 (n = 83)	Image rating 5 (n = 3)
Subxyphoid view, n (%)	26 (74%)	24 (55%)	32 (46%)	43 (52%)	1 (33%)
Cardiac activity, n (% no)	35 (100%)	32 (73%)	16 (23%)	18 (22%)	0 (0%)
Sex, n (% male)	24 (68%)	28 (64%)	44 (63%)	59 (71%)	3 (100%)
Mechanical compression device, n (% no)	7 (20%)	12 (27%)	26 (37%)	16 (19%)	0 (0%)

A total of 64.5% (133 of 206) of patients had some presence of cardiac activity. Agreement for cardiac activity (present or absent) was substantial (kappa 0.78 (95%CI 0.70–0.87) and agreement for type of activity (organized vs disorganized) was good (kappa 0.76 (95%CI 0.64–0.88). There was no difference in agreement (present vs absent) when analyzing by echo view (subxyphoid Kappa 0.77 (95%CI 0.65–0.89) vs parasternal 0.79 (95%CI 0.67–0.92)). There was also no difference in agreement (organized vs disorganized) when analyzing by echo view (subxyphoid Kappa 0.72 (95%CI 0.54–0.91) vs parasternal 0.78 (95%CI 0.64–0.94)).

Discussion

The majority of echo images obtained in this study were of high quality despite being obtained during short pauses in CPR. Echo during ACLS occurs within a 10 s window and prior research has not proven that high quality echo images are obtainable in 10 s. The American Society of Echocardiography recommends 45−60 min for an uncomplicated trans-thoracic echo. In one study single views of the heart during stress echocardiography took 67 s to acquire. Echo image acquisition times during ACLS average significantly less than 67 s, with a goal not to obtain perfect images, but to obtain images with sufficient information to guide ACLS. The majority of images obtained in this study had sufficient image quality to determine cardiac activity, degree of cardiac activity and cardiac pathology such as pericardial effusion or intra-cardiac clot.

Agreement was greatest for a level 1 image quality, defined as “images uninterpretable.” This is expected as it is likely the least subjective of the levels. Obtaining images in cardiac arrest is challenging and due to time constrains results in limited views of poor quality in some cases where images are completely missing. The fact that agreement for level 5 (best quality) was also greater than the other levels support the argument that clinicians agree on “perfect” and “terrible” but have less agreement for image quality in the middle.

We found substantial agreement on echo findings of cardiac activity using the most recent, and most widely used definitions in the literature.1, 9 Previous research into echo in cardiac arrest has focused on the presence of cardiac activity as a prognosticating event.1, 15 Some studies have explored degrees of cardiac activity or identification of alternative reasons for cardiac arrest.4, 9 The international liaison committee on resuscitation recently reviewed the research behind echo in cardiac arrest and noted that definitions of cardiac activity need to be uniform and that few studies have focused on agreement between reviewers. We suggest that the definition used in this manuscript as defined by Gaspari et al. be adopted as the standard definition to support research aggregation.

Other researchers have presented data on agreement and image quality during bedside echocardiography,17, 18, 19 but point-of-care echo in cardiac arrest has important differences from the broader use of point-of-care echo in non-resuscitative settings. Obtaining echocardiographic images during pauses in CPR requires image acquisition times that are a small fraction of the time spent on other patients. One echo image quality index focused on assessing the visibility of the endocardial boarder, but this method requires imaging through multiple cardiac cycles, which is not feasible during ACLS. It also assumes anechoic cardiac chambers to identify the endocardial boarder, a situation that is not ubiquitous in patients in cardiac arrest. Other echo image quality tools require multiple images, something not relevant to echo during cardiac arrest.17, 19 The cardiac arrest patient population presents significant challenges related to image quality, and our image rating scale focuses on image elements pertinent to cardiac arrest that may have been overlooked by previous researchers.

It has been noted that the research supporting the use of ultrasound in cardiac arrest suffers from methodologic rigor and does not currently support improved outcomes stemming from use of echo. Further research on the use of this imaging modality during ACLS is needed with sufficient rigor to address the best role in clinical care. Tools such as this image rating scale are needed to support both improved research methodology and clinical implementation in this patient population.

This study had several limitations. This is a small study from a single site. Images were reviewed by emergency physicians with significant experience in echo during cardiac arrest, and the findings may be different for individuals with less experience. Similarly, the quality of cardiac images obtained in this cohort may not be representative of the quality of ultrasound images in other emergency departments. Patients were excluded due to lack of recorded echo images, resulting in possible selection bias. In addition, other unmeasured variables may have impacted image quality or interpretation of cardiac activity. Finally, our study focused on TTE, but transesophageal echocardiography (TEE) is emerging as a well-suited modality during cardiac arrest resuscitation. Several studies using TEE in cardiac arrest have shown its reliably in producing high quality images, its potential to improve CPR quality,22, 23 and the ability to shorten chest compressions pauses intra-arrest. Future studies should evaluate image quality in TEE-guided resuscitations and potentially compare image agreement between the TTE and TEE.

Conclusion

This novel image quality rating scale demonstrates substantial agreement between clinicians. The ability to track and rate image quality is important for interpretation of clinical findings and aggregating data. This rating scale focuses on the image elements which are important and specific to echo during cardiac arrest resuscitation and therefore represents a valuable tool for clinicians and researchers in the field.

Author contribution

RG conceived and designed the study. FT contributed to methodology. RG and TG supervised the conduct of the study and data collection. AK provided database support and administration of imaging as well as contributions to manuscript preparation. RG and TG undertook recruitment of patients and managed the data, including quality control. RG provided statistical advice on study design and analyzed the data; RG drafted the manuscript, and all authors contributed substantially to its revision. RG takes responsibility for the paper as a whole.

Declaration of interest

None.

Ethics information

This study was reviewed and approved by the UMASS institutional Review Board.