Impact of the COVID pandemic on quality measures in a pediatric echocardiography lab.

Background: The COVID pandemic necessitated an altered approach to transthoracic echocardiography, especially in COVID cases. Whether this has affected echocardiography lab quality is unknown.
Objectives: We sought to determine whether echocardiography lab quality measures during the COVID pandemic were different than those prior to the pandemic and whether quality and comprehensiveness of echocardiograms performed during the pandemic was different between COVID and non-COVID patients.
Methods: The four quality measures (diagnostic errors, appropriateness of echocardiogram, American College of Cardiology Image Quality metric and Comprehensive Exam metric in structurally normal hearts) reported quarterly in our lab were compared between two quarters during COVID (2020) and pre-COVID (2019). Each component of these metrics was also assessed in randomly selected echocardiograms in COVID patients and compared to non-COVID echocardiograms.
Results: For non-COVID echocardiograms, the image quality metric did not change between 2019 and 2020 and the comprehensive exam metric improved. Diagnostic error rate did not change and appropriateness of echocardiogram indications improved. When COVID and non-COVID echocardiograms were compared, the image quality metric and comprehensiveness exam metric were lower for COVID cases (image quality mean 21.3/23 for non-COVID, 18.6/23 for COVID, p < 0.001 and comprehensive exam mean 29.5/30 for non-COVID, 27.7/39 for COVID, p < 0.001). In particular, systemic and pulmonary veins, pulmonary arteries and aortic arch were not adequately imaged in COVID patients. For studies in which a follow-up echocardiogram was available, no new pathology was found. Conclusions: At our center, though diagnostic error rate did not change during the pandemic and the proportion of echocardiograms ordered for appropriate indications increased, imaging quality in COVID patients was compromised, especially for systemic and pulmonary veins, pulmonary arteries and arch. Though no new pathology was noted on the small number of patients who had follow-up studies, we are paying careful attention to these structures to avoid diagnostic errors going forward.

Introduction

Early during the coronavirus diseases (COVID) pandemic, the American Society of Echocardiography made recommendations for performing transthoracic echocardiograms (TTE) in pediatric patients that had suspected or confirmed COVID infection [1]. The guidelines emphasized the need to minimize the duration of exposure to these patients, performing problem-focused studies, paying attention to the established Appropriate Use Criteria for initial and follow-up examinations when applicable, and maintaining quality of images during the scans [1], [2], [3]. These guidelines led us to adopt a more focused and abbreviated protocol for TTEs on patients with confirmed or suspected COVID. However, the impact of these changes on various components of quality of TTEs remains unknown. In our echocardiography lab, we aim at performing a comprehensive TTE examination during the initial evaluation and follow the standards defined by the American Society of Echocardiography [4], [5] and the American College of Cardiology Quality Network cardiovascular imaging metrics [6]. In addition, the 2014 Appropriate use Criteria for Initial Outpatient evaluation is integrated with our electronic TTE ordering system since 2017 [7].

The purpose of this study was to determine the impact of changes made during COVID pandemic on the TTE imaging quality, comprehensiveness, diagnostic error rate and appropriateness of TTE orders in patients with and without COVID infection.

Methods

The study was approved by the Institutional Review Board of Children's Healthcare of Atlanta. Data were retrospectively collected for four quality measures that were reported on a quarterly basis in our lab prior to the pandemic: 1) diagnostic error rate based on the number of major diagnostic errors on TTE for all patients that undergo cardiac surgery; 2) appropriateness of indication for initial outpatient TTE evaluation based on the 2014 Appropriate Use Criteria document [6], which has been integrated into the electronic health system at our center since May 2017 and incorporated into feedback for physicians on the appropriateness of TTE orders [7]; 3) initial TTE Image Quality Metric (IQM) and 4) Comprehensive Echocardiographic Exam (CEM) [6]. IQM and CEM have been defined by the Quality Network of the American College of Cardiology and were adopted by our center in Jan 2019. For IQM and CEM we evaluate 20 randomly selected TTEs performed on patients with structurally normal hearts during each quarter. The components of IQM scoring are comprised of image orientation, 2D imaging, color flow imaging and spectral Doppler display, with a maximum total score of 23. CEM scoring is based on imaging of vessels, valves, ventricles, veins, atria and situs, with a maximum total score of 30. TTEs performed by trainees were excluded.

All four quality measures were compared between two similar quarters (April to June, and July to September) in pre-COVID (2019) and COVID era (2020). Additionally, IQM and CEM of 40 randomly selected TTEs in patients with suspected or confirmed COVID were compared with those of non-COVID patients during the same timeframe. Records were reviewed to look for any missed echocardiographic findings if a follow-up TTE was performed on these patients. The number of images and study time was compared between COVID and non-COVID patients. The location in which the TTEs were performed in patients with suspected or conformed COVID was also recorded.

Statistical analyses

Statistical analyses were performed using SAS software version 9.4 (SAS Institute, Cary, North Carolina), and statistical significance was assessed at the 0.05 level. Normality of continuous variables was assessed using the histogram, normal probability plots and Anderson-Darling test for normality. Descriptive statistics are presented as counts and percentages for categorical variables and as median (25th, 75th percentile) for continuous data with skewed distributions or mean ± standard deviation for those with normal distributions. Non-normal continuous data were compared using Wilcoxon rank-sum tests, normal continuous data were compared using t-tests and comparisons between categorical variables were performed using chi-square tests, or the Fisher exact test when expected cell counts were <5.

Results

The quarterly volume of TTEs performed at the hospital during pre-COVID and COVID study period were similar (2687 ± 37 vs 2700 ± 425, p = 0.97). The average quarterly volume of initial outpatient TTEs performed in our clinics was also similar (1524 ± 130 vs 1170 ± 524, p = 0.45). A comparison of quality measures between the pre-COVID and COVID era is shown in Table 1 . There was no statistically significant difference in the mean IQM score between pre-COVID and COVID era in non-COVID patients with structurally normal hearts. However, the mean CEM score improved from pre-COVID to COVID era, specifically for imaging of coronary sinus (29/40 vs 38/40, p = 0.006), tricuspid regurgitation jet evaluation by Doppler in two views if available (34/40 vs 40/40, p = 0.011) and evaluation adequate for measurement of aortic valve, aortic root, and aortic sinotubular junction diameters measured in parasternal long-axis view (30/40 vs 37/40, p = 0.034). There was no significant change in major diagnostic error rate between the pre-COVID and COVID era (0.02 % vs 0.01 %, p = 0.18). Overall, there was an improvement in appropriateness of TTE orders in the COVID era, with the proportion of TTEs with indications rated Appropriate increasing from 90.3 % to 92.1 %, p = 0.028 (Table 1). There was no significant change in the TTEs ordered for indications rated Rarely Appropriate. Among the TTEs ordered for indications related Rarely Appropriate, there was an increase in the proportion of those related to syncope, (5/37 (13.5 %) vs 7/17 (41.2 %), p = 0.035) and palpitations and arrhythmias (8/37) 21.6 % vs 10/17 (58.8 %), p = 0.007) in COVID era compared to pre-COVID era, and a decline in those related to chest pain (10/37 (27 %) vs 0/17 (0 %), p = 0.02). Clinic notes of patients with TTE for indications rated Rarely Appropriate were reviewed for any history of COVID or MISC-related concerns, but none were noted.

Table 1

Comparison of quality measures between pre-COVID and COVID Era.

Metric	Pre-COVID EraApril-Sept 2019	COVID EraApril-Sept 2020	P-value
Image Quality MetricMean (SD), (Total score 23)	21.5 (2.0)	21.3 (1.5)	0.529
Comprehensive Exam MetricMean (SD), (Total score 30)	28.4 (1.7)	29.0 (0.9)	<0.001
Diagnostic ErrorsN (%)	15/665 (0.02 %)	9/694 (0.01 %)	0.180
Appropriateness RatingsN (%)
Appropriate	2753/3048 (90.3 %)	2153/2339 (92.05 %)	0.028
May be Appropriate	258/3048 (8.46 %)	169/2339 (7.23 %)	0.095
Rarely Appropriate	37/3048 (1.21 %)	17/2339 (0.73 %)	0.075

In the COVID era, the initial TTE IQM mean score was significantly less in COVID patients compared to non-COVID patients, 18.6 ± 2.5 vs 21.3 ± 1.5, p < 0.001, (Table 2 ). The specific components of IQM that were worse in COVID patients included image orientation for subcostal sagittal view, brightness level, balanced penetration resolution, presentation of region of interest, frame rate and gain level for color-Doppler, scale adjustment and gain level for spectral-Doppler. The mean CEM score was also significantly lower in COVID patients compared to non-COVID patients, 27.7 ± 3.7 vs 29.5 ± 0.8, p < 0.001, (Table 3 ). Imaging of the systemic and pulmonary veins, pulmonary arteries and aortic imaging (arch sidedness and branching, arch imaging by Doppler, and abdominal aorta by Doppler) scored significantly lower in COVID patients compared to non-COVID patients. Of the 40 studies in COVID patients, 5 were performed in the emergency department, 20 in the acute care floor and 15 in the critical care unit. The study location was not associated with IQM or CEM results. The number of images acquired per study was significantly less in COVID vs non-COVID patients (96 ± 28.7 vs 118 ± 18, p = 0.0002), though the scanning time was similar (29 ± 13 vs 33 ± 12, p = 0.11). A complete follow-up TTE was available in 20 out of 40 COVID patients and no pathology was noted on these TTEs.

Table 2

Comparison of initial transthoracic echocardiogram image quality metric between non-COVID and COVID patients in 2020.

Variable	Scale	Non-COVIDN = 40	COVIDN = 40	P-value
Mean (SD) score (Out of 23)		21.3 (1.5)	18.6 (2.5)	<0.001
Image orientation
Parasternal long axis	Yes	35 (87.5 %)	40 (100.0 %)	0.021
Parasternal short axis	Yes	39 (97.5 %)	40 (100.0 %)	0.314
Apical 4 chamber	Yes	36 (90.0 %)	38 (95.0 %)	0.396
Subcostal sagittal	Yes	29 (72.5 %)	17 (42.5 %)	0.007
Suprasternal notch	Yes	34 (85.0 %)	36 (90.0 %)	0.499
2D Imaging
Brightness level appropriate	Agree	28 (70.0 %)	13 (32.5 %)	0.003
	Somewhat agree	10 (25.0 %)	25 (62.5 %)
	Disagree	2 (5.0 %)	2 (5.0 %)
Balanced penetration: resolution	Agree	26 (65.0 %)	23 (57.5 %)	0.050
	Somewhat agree	10 (25.0 %)	17 (42.5 %)
	Disagree	4 (10.0 %)	0 (0.00 %)
Region of interest presented well	Agree	39 (97.5 %)	23 (57.5 %)	<0.001
	Somewhat agree	1 (2.5 %)	17 (42.5 %)
Color-flow imaging
Frame rate appropriate	Agree	38 (95.0 %)	11 (27.5 %)	<0.001
	Somewhat agree	2 (5.0 %)	29 (72.5 %)
Gain level appropriate	Agree	39 (97.5 %)	30 (75.0 %)	0.003
	Somewhat agree	1 (2.5 %)	10 (25.0 %)
Nyquist limit settings appropriate	Agree	37 (92.5 %)	31 (77.5 %)	0.060
	Somewhat agree	3 (7.5 %)	9 (22.5 %)
Spectral Doppler display
Choice of Doppler appropriate	Agree	40 (100.0 %)	40 (100.0 %)	–
Gain level appropriate	Agree	40 (100.0 %)	29 (72.5 %)	<0.001
	Somewhat agree	0 (0.00 %)	11 (27.5 %)
Scale adjusted to provide maximal signal size	Agree	38 (95.0 %)	18 (45.0 %)	<0.001
	Somewhat agree	0 (0.00 %)	22 (55.0 %)
	Disagree	2 (5.0 %)	0 (0.00 %)

Table 3

Comparison of comprehensive echocardiogram examination quality metric between non-COVID and COVID patients in 2020.

Variable	Non-COVIDN = 40	COVIDN = 40	P-value
Mean (SD) Score (Out of 30)	29.5 (0.8)	27.7 (3.7)	<0.001
Situs, Veins, Atria
Liver and stomach shown	40 (100.0 %)	37 (92.5 %)	0.077
Cardiac position	40 (100.0 %)	38 (95.0 %)	0.152
IVCa and aorta in relation to spine	40 (100.0 %)	37 (92.5 %)	0.077
IVC and SVCb in imaging and color	40 (100.0 %)	32 (80.0 %)	0.003
IVC connection to atrium	40 (100.0 %)	38 (95.0 %)	0.152
Two left and two right pulmonary veins	37 (92.5 %)	25 (62.5 %)	0.001
Coronary sinus visualized	38 (95.0 %)	38 (95.0 %)	1.000
Atrial septum imaging	40 (100.0 %)	37 (92.5 %)	0.077
Ventricles
Ventricular septum by color Doppler	40 (100.0 %)	38 (95.0 %)	0.152
Imaging for qualitative RVc function	40 (100.0 %)	38 (95.0 %)	0.152
Imaging of LVd function	40 (100.0 %)	39 (97.5 %)	0.314
LV end diastolic dimension or volume	40 (100.0 %)	39 (97.5 %)	0.314
LV end systolic dimension or volume	40 (100.0 %)	39 (97.5 %)	0.314
Evaluation for LV mass	40 (100.0 %)	39 (97.5 %)	0.314
LV Outflow by Doppler	40 (100.0 %)	39 (97.5 %)	0.314
RV Outflow by Doppler	40 (100.0 %)	38 (95.0 %)	0.152
Cardiac valves
Tricuspid valve imaging	40 (100.0 %)	40 (100.0 %)	–
Tricuspid regurgitation jet by Doppler	40 (100.0 %)	39 (97.5 %)	0.314
Mitral valve imaging	40 (100.0 %)	40 (100.0 %)	–
Mitral valve in short axis	38 (95.0 %)	36 (90.0 %)	0.396
Pulmonary valve imaging	40 (100.0 %)	39 (97.5 %)	0.314
Aortic valve imaging	40 (100.0 %)	37 (92.5 %)	0.077
Coronary arteries imaging	39 (97.5 %)	37 (92.5 %)	0.305
Vessels
Evaluation for aortic dimensions	37 (92.5 %)	36 (90.0 %)	0.692
Branch pulmonary artery imaging	39 (97.5 %)	33 (82.5 %)	0.025
Patent ductus arteriosus excluded	39 (97.5 %)	39 (97.5 %)	1.000
Ascending aorta imaging	31 (77.5 %)	35 (87.5 %)	0.239
Aortic arch sidedness and branching	40 (100.0 %)	34 (85.0 %)	0.011
Aortic arch imaging by Doppler	40 (100.0 %)	36 (90.0 %)	0.040
Abdominal aorta evaluated by Doppler	40 (100.0 %)	36 (90.0 %)	0.040

IVC = Inferior Vena Cava.

SVC = Superior Vena Cava.

RV = Right Ventricle.

LV = Left Ventricle.

Discussion

In this study from a large tertiary care pediatric center, the image quality and comprehensiveness of TTEs performed in COVID patients were significantly impacted by the changes adopted during the pandemic. However, the quality was maintained in non-COVID patients receiving initial complete evaluation at our center.

It is known that severe acute respiratory syndrome coronavirus-2 infection that causes COVID has rapid transmission and is associated with high morbidity and mortality. Given this concern, cardiovascular imaging societies have recommended considering the use of focused and abbreviated protocols and using laptop size echocardiography machines in these patients [1]. In addition, there have been other system-wide changes that were continuously evolving during the pandemic including closure of the clinics, lack of sedation for TTEs in uncooperative infants and toddlers, changes in staffing, telemedicine and remote reading that could have impacted quality of care in non-COVID patients. Therefore, this study not only evaluated patients with suspected or confirmed COVID, but also non-COVID patients who had initial complete evaluation with TTE. Studying the effect of each of these factors on various quality measures would have been quite challenging as these are hard to objectively capture in a retrospective manner. Fortunately, we did not find any significant changes in IQM in non-COVID patients during the pandemic and noted improvement in CEM. These two metrics were implemented at our center in Jan 2019 and since then, the sonographers have been receiving quarterly feedback on their studies. The improvement in CEM may be reflective of our educational efforts regarding these metrics.

It was not surprising that the CEM was affected in COVID patients given the use of abbreviated protocols. Initial evaluation in COVID patients had significantly deficient imaging of pulmonary veins, pulmonary arteries, arch situs, arch Doppler, and abdominal aorta Doppler. If a subsequent study was performed on these patients, attempt was made to image the structures that were not captured on the initial scan. IQM was lower in COVID patients due to suboptimal image orientation, brightness level, improper adjustment of Doppler gains and scale. At our center, we did not use laptop size machines to perform TTEs in COVID patients included in this study. One could speculate that the need to complete the study quickly to reduce exposure to the patient could have resulted in lack of effort for image optimization and hence the lower IQM. Though no pathology was noted in the COVID patients who had a subsequent study, it was a small sample size (20/40). A study in 171 adult patients with suspected or confirmed COVID, evaluated the scanning time and acquisition of minimum data set, and compared it to 50 controls from the pre-pandemic period [8]. They reported a significant reduction in median scanning time from 14 min (IQR 10–16 min) to 6 min (IQR 4–8 min) while following the abbreviated protocol. The number of images also reduced from a median of 55 (IQR 39–66) to 25 (17–36). Importantly, there was no difference in the proportion of studies that met the minimum data set (47/50 (94 %) in controls vs 158/171 (92.4 %) in COVID patients). In contrast to this study, in our study, even though the number of images decreased in COVID patients, the decrease in average scanning time did not reach statistical significance and this could be related to a relatively small sample size.

Quality of cardiovascular imaging can impact patient outcomes [9], [10]. Fortunately, we did not observe any change in the diagnostic error rate. One of the steps in providing quality care with TTE is appropriateness of the study indications. Though overall there was an improvement in appropriateness of TTE orders for initial outpatient evaluation, there was an increase in the proportion of TTEs ordered for the Rarely Appropriate indications related to syncope, palpitations and arrhythmias. We speculated that these findings may reflect heightened concern for the cardiac complications of COVID and lower threshold to obtain an echocardiogram given the ongoing COVID pandemic. However, upon review of clinic notes, no such association was found. Surprisingly, during the pandemic, no TTEs were ordered for chest pain indications rated R. Anecdotally, anxiety-related chest pain has been a common presentation in cardiology clinics through the pandemic and despite that we did not notice a rise in TTEs for indications rated R for chest pain. It is important to note that the overall number of indications rated R is quite small and the differences noted for specific indications may not be clinically meaningful.

This study is limited by its retrospective nature and restriction of the data to two quarters of the ongoing COVID pandemic. Being a single-center study, the results may not be generalizable. The factors that could have influenced the imaging metrics were difficult to capture objectively for statistical analysis. The representative sampling of TTEs assessed for IQM and CEM quality metrics was relatively small since we used the existing data from our quality assurance process for the echocardiography lab. Nevertheless, some important differences were noted in COVID patients compared to non-COVID patients that have been helpful in our educational efforts to improve quality of care during the pandemic.

Conclusion

During the COVID pandemic, the diagnostic error rate has not changed and the proportion of TTEs ordered for appropriate indications has increased. However, unlike non-COVID patients, the image quality as well as the exam comprehensiveness has decreased in those with suspected or confirmed COVID. In our center, this information has served as a guide to assure that adequate imaging of systemic and pulmonary veins, pulmonary arteries and the aorta is performed on the initial or subsequent studies.

CRediT authorship contribution statement

Anna-Claire Marrone: Conceptualization, methodology, investigation, formal analysis, writing (original draft) and writing (review and editing); Gemma Morrow: Investigation, data curation and writing (review and editing); Michael Kelleman: Methodology, formal analysis and writing (review and editing); Joan Lipinski: Methodology, data curation and writing (review and editing); William Border: Conceptualization, writing (review and editing) and supervision; Ritu Sachdeva: Conceptualization, methodology, formal analysis, writing (original draft), writing (review and editing) and supervision.

Declaration of competing interest

X The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.