A Teaching Intervention Increases the Performance of Handheld Ultrasound Devices for Assessment of Left Ventricular Ejection Fraction.

BACKGROUND: Few studies have demonstrated the utility of a teaching program for evaluation of left ventricular ejection fraction (LVEF) of echocardiographic images acquired with high-end machines. No study to date explored the value of similar programs when a handheld ultrasound device is used. The aim of this study was to determine whether a teaching intervention could improve the accuracy and the reliability of LVEF visual assessment of echocardiographic images acquired with HUD.
MATERIALS AND METHODS: Twenty echocardiograms acquired with a hand-held ultrasound device with a spectrum of LVEF were presented to 26 participants with varying experience in echocardiography (range 2-12 years) for single-point LVEF visual estimates. After this baseline assessment, participants underwent three training sessions which included analysis of the individual baseline results and review and interpretation of additional 60 cases from the same platform. After 2 months, 20 new echocardiograms were presented to the same 26 participants for visual LVEF assessment. For each participant, the visual LVEF for each case was compared with the reference LVEF (quantitative measurements by experts), and a difference of > ±5% was considered a misclassification.
RESULTS: The misclassification rate was 61% preintervention and decreased to 41% after intervention (P < 0.0001). The mean absolute differences in LVEF between visual estimates and reference before and after intervention for all readers were -7.9 ± 9.6 and -1.2 ± 7.8, respectively (P < 0.0001). Inter-rater repeatability analysis was performed using the intraclass correlation coefficient. The intraclass correlation coefficient for inter-rater reliability was fair preintervention (0.65, 95% confidence interval [CI] 0.59 0.71) and good after intervention (0.80, 95% CI 0.73 0.87), and there were no differences when categorized according to the level of experience.
CONCLUSIONS: A teaching intervention can improve the accuracy and the reliability in the visual LVEF assessment of images acquired with handheld ultrasound device. Copyright:

INTRODUCTION

Although not recommended by the American Society of Echocardiography/European Association of Cardiovascular Imaging (EACVI) as the primary method for the evaluation of left ventricular ejection fraction (LVEF),[1] visual assessment is still widely applied in particular when handheld ultrasound devices (HUDs) are used. In fact, HUD are mostly utilized by physicians at point of care as part of the physical examination for gross evaluation of the heart,[2] and in general, they do not have software for quantitative analysis. Thus, in this clinical scenario, the left ventricular systolic function is usually described in a qualitative manner (normal, mild, moderate, or severe dysfunction); still the interpretation should be as close as possible to the quantitative methods, with a high level of accuracy and reliability for being clinically acceptable.[3]

In our institution, HUD are used by cardiologists with different levels of expertise in echocardiography (fellows, specialists, and consultants) and in different clinical settings (including outpatient department, emergency department, heart failure unit, and critical care unit), thus suboptimal accuracy and interobserver variability (IOV) can be anticipated, and although there is no universal agreement for training, all agree on the need for didactic education and interpretation experience.[4]

With these concepts in mind, we sought to determine whether a formal teaching intervention could improve the accuracy and the reliability of visual LVEF assessment of echocardiographic images acquired with HDU.

MATERIALS AND METHODS

Pre-intervention

Twenty-six participants (17 cardiologists and 9 cardiology fellows) with different experience in echocardiography (mean years of experience 6.2 ± 3.2; range 2–12 years) reviewed 20 cases of two-dimensional echocardiograms selected from the ESC e-Learning platform of the EACVI.[5] These echocardiograms are part of the module on left ventricular assessment of the online course for HUD and have a LVEF measured by experts with quantitative methods as reference.

Each selected echocardiogram consisted of multiple views of the left ventricle, including parasternal, short-axis, and apical views, displayed in a quad screen format. The cases selected represented a spectrum of LVEF range and image quality. No information regarding clinical history or reason for referral was provided. All participants reviewed the cases simultaneously blinded to each other's interpretation and had an equal amount of time to review the case and provide a visual estimate of the LVEF as a single integer. Participants were assigned an answer sheet with a numerical identifier so that their answers remained anonymous but could be tracked.

Teaching intervention

The teaching intervention was conducted by two staff cardiologists (AS, SA) with long experience in echocardiography and consisted of three 1.5-h sessions with the participants divided in small groups. During the first teaching session, the individual preintervention results were displayed and discussed with the participants as a graph plotting the single-integer EF visual estimates of each participant. This allowed participants to understand if the visual LVEF may have been overestimated or underestimated. Then, the echocardiographic images were presented again, and discussion was centered on factors that influenced the participant's LVEF estimate. These included technical aspects of image acquisition, dropouts of endocardium, presence of wall motion abnormalities/aneurysm, impact of arrhythmias, and impact of abnormal septal motion on LVEF assessment. In addition, 60 new cases from the same platform were presented and discussed collegially.

Post-intervention

After 2 months, 20 new echocardiograms from the same platform were presented to the participants for visual assessment of LVEF using the same methodology described for the baseline assessment. These cases had similar range of LVEF and quality as those of the baseline assessment. Again, all participants were blinded to each other's assessment.

For each case, individual visual estimates of LVEF were compared with the reference values. Disagreement was defined as a > ±5% difference between visual estimate and the quantified LVEF. The whole methodology is summarized in Figure 1.

Figure 1

Quality improvement protocol for the assessment of left ventricular ejection fraction with handheld ultrasound device. IOV: Interobserver variability, LVEF: Left ventricular ejection fraction

Statistical analysis

Continuous data are expressed as mean ± standard deviation and categorical data as frequencies or percentages. McNemar Chi-square test was used to determine differences in misclassification rates before and after intervention. In addition, the mean LVEF for each case for all readers before and after the intervention was compared with the reference LVEF using both linear regression and Bland–Altman analysis.

Inter-rater repeatability analysis was performed using the intraclass correlation coefficient (ICC). ICC estimates and their 95% confidence intervals (CIs) were calculated based on mean rating, absolute agreement, and 2-way random effects model. Interpretation was as follows: <0.50, poor; between 0.50 and 0.75, fair; between 0.75 and 0.90, good; and above 0.90, excellent. All statistical analyses were performed using Statistical Package for Social Science (SPSS version 22.0).

RESULTS

Information regarding each case shown pre- and post-intervention is displayed in Table 1. In total, there were 520 responses (20 cases, 26 participants) both pre- and post-intervention.

Table 1

Characteristics of cases pre- and post-intervention

Preintervention			Postintervention
Case number	Readers’ mean±SD	Reference LVEF	Case number	Readers’ mean±SD	Reference LVEF
1	57±4.8	55	1	60±6.1	60
2	51±10.7	62	2	61±4.7	65
3	40±9.6	60	3	32±9.3	30
4	56±7.2	60	4	37±7.7	32
5	41±6.1	40	5	47±9.4	50
6	55±5.8	65	6	58±9.8	57
7	38±9.7	40	7	32±9.1	37
8	54±6.9	62	8	61±4.4	65
9	50±7.3	57	9	26±7.0	27
10	45±10.9	60	10	52±6.1	52
11	46±11.4	60	11	22±6.3	25
12	31±7.1	32	12	56±6.1	55
13	30±8.1	35	13	56±4.9	60
14	43±7.7	60	14	35±7.7	47
15	28±8.0	45	15	61±4.0	62
16	51±8.2	57	16	36±5.6	32
17	21±3.7	27	17	46±5.3	42
18	29±6.3	35	18	16±3.2	12
19	54±5.6	62	19	34±8.6	40
20	51±9.2	60	20	43±7.1	47

LVEF: Left ventricular ejection fraction, SD: Standard deviation

Accuracy

The classification of visual LVEF by readers pre- and post-intervention is shown in Table 2. The misclassification rate for the preintervention reads for the 520 responses was 61%, while after intervention, it decreased to 41% (P < 0.0001). Using a difference in LVEF of > ±10% (that is the cutoff generally accepted to be clinically relevant), the number of cases with misclassification decreased significantly from 188 (37%) to 69 (13%) (P < 0.001). Cases were further divided into two groups according to the degree of impairment (LVEF 30%–55% and LVEF <30% or >55%). For cases with LVEF 30%–55%, the misclassification rate was 51% pre- and 52% post-implementation (n. s.), while for cases with LVEF <30% or >55%, the misclassification rate was 64% pre- and 29% post-implementation (P < 0.001). The mean absolute differences in LVEF between visual estimation and reference before and after intervention for all readers were 7.9% ± 9.6% and 1.2% ± 7.8% (P < 0.0001) [Figure 2].

Table 2

Misclassification rates of individual visual estimates of left ventricular ejection fraction compared with reference ejection fraction for all cases before and after the intervention

Differences between gold standards and visual assessment of LVEF (%)	Preintervention, n (%)	Postintervention, n (%)
±0-5	205 (39)	306 (59)
±6-10	127 (24)	145 (28)
±11-15	76 (15)	39 (7)
≥±16	112 (22)	30 (6)

LVEF: Left ventricular ejection fraction

Figure 2

Linear regression (top) and Bland–Altman analysis (bottom) comparing the mean visual left ventricular ejection fraction for each case for all readers pre- and post-intervention in comparison with reference left ventricular ejection fraction

Inter-rater reliability

The ICC for inter-rater reliability was fair (0.65, 95% CI 0.59–0.71) preintervention and good (0.80, 95% CI 0.73–0.87) after intervention. When categorized by level of experience, there were no differences in the improvement of inter-rater reliability. Figure 3 shows an example of LVEF visual estimates from the 26 participants before and after intervention in two different cases with the same LVEF (32% by reference methods).

Figure 3

Visual left ventricular ejection fraction estimates before and after the intervention in two different cases with similar left ventricular ejection fraction by all readers. The reference left ventricular ejection fraction is represented by the red dots. The group mean is illustrated with a solid line, with ± standard deviation illustrated by the dotted lines. The standard deviation of the visual left ventricular ejection fraction estimates was reduced after the intervention

DISCUSSION

Assessment of LV function remains one of the most common clinical questions addressed by echocardiography, and several approaches can be used. In patients referred for a comprehensive transthoracic echocardiogram, experienced echocardiographers usually evaluate the LVEF offline and quantitative methods (volumetric three-dimensional or modified biplane Simpson's) are highly recommended.[1]

The advent of HUD has changed this paradigm, with two major implications. First, due to the minimalism and versatility of this equipment, physicians with different experience in echocardiography may use HUD at the point of care. As a consequence, a low grade of accuracy and a high IOV can be anticipated, which limit the clinical usefulness of these devices. Second, HUD is used as an extension of the physical examination for gross evaluation of the cardiac structures and function; thus, online qualitative assessment is usually applied including eyeball estimate of LVEF that is reported as normal, or mildly, moderately, or severely reduced, based on interval range. Although this approach is practical, a more stringent definition of the LVEF with single integer visual evaluation that closely represents the actual LVEF would be appropriate.

Furthermore, to have clinical value, the single integer value of LVEF should be similar among different observers. Thus, education and training on the use of HDU are highly recommended.[678] The results of the present study indicate the feasibility of a teaching program integrated with web-based expert interpretation of stored echocardiographic images, which could be ideally delivered also to remote communities.[9] In our experience, our cardiologists tended to underestimate the LVEF, and after the teaching program, we found a significant improvement in accuracy as well as in IOV.

Two previous studies have evaluated the utility of similar learning programs to improve accuracy and IOV of visual estimation of LVEF. Both studies utilized images acquired with standard echocardiographic equipment. With this exercise, Johri et al. demonstrated a reduction in misclassification rate (from 44% before intervention to 21% following intervention) and a 40% reduction in IOV (from ± 14% to ± 8.4%).[10]

With a similar self-directed program, Thavendiranathan et al. obtained a reduction in misclassification (overall from 56% to 47%), in particular for studies with severe LV dysfunction (from 62.7% to 39.5%), and an improvement in IOV (from ± 0.120 to ± 0.097) independent from the level of experience of readers.[11]

Our results confirm these findings and expand the validity of a training program also to the interpretation of images acquired with a HUD, which in general have a lower quality compared to high-end machines. The higher misclassification rate in the current study (61%) can be explained by the lower image quality of HUD compared to standard high-end machines. Yet, our teaching program resulted in a 33% relative reduction of misclassification (from 61% to 41%).

Furthermore, similar to the findings by Thavendiranathan et al.,[11] we found a significant improvement in misclassification with the intervention for the severe categories, with no improvement in the mild-to-moderate categories.

Limitations

Usually, the visual estimate is given as a range rather than a single value; thus, our assessment of the imprecision may be overestimated. However, using the same methodology pre- and post-intervention, we measured the reduction in IOV following the teaching intervention even if there is overestimation in both groups. We did not determine the need of repeated teaching intervention for LVEF assessment; thus, the long-term effect of the teaching program is still questionable.[12] We used LVEF values as determined by expert in a blinded fashion as the reference standard and the variability in the quantification is unknown.

CONCLUSIONS

A teaching intervention can improve the accuracy and the reliability in the visual assessment of LVEF of echocardiographic images acquired with a Handheld Ultrasound Device (HUD). The benefit of the intervention is independent from the level of expertise of the readers. The impact of this study is potentially important in improving reader confidence on the analysis of echocardiographic images obtained with a HUD.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.