Influence of observer experience on cardiac magnetic resonance strain measurements using feature tracking and conventional tagging.

AIM: CMR quantitative myocardial strain analysis is increasingly being utilized in clinical routine. CMR feature tracking (FT) is now considered an alternative to the reference standard for strain assessment -CMR tagging. The impact of observer experience on the validity of FT results has not yet been investigated. The aim of this study was therefore to evaluate the observer experience-dependency of CMR FT and to compare results with the reference standard.
METHODS: CSPAMM and SSFP-Cine sequences were acquired in 38 individuals (19 patients with HFpEF,19 controls) in identical midventricular short-axis locations. Global peak systolic circumferential strain (PSCS) together with LV ejection fraction (EF) and volumes were assessed by three observers (5,3 and 0 years of CMR-strain experience). Intermodality, intra- as well inter-observer variability were assessed.
RESULTS: Correlation between tagging and FT derived PSCS depended on observer experience (r = 0.69, r = 0.58 and r = 0.53). For the inexperienced observer tagging and FT derived PSCS differed significantly (p = 0.0061). Intra-observer reproducibility of tagging derived PSCS were similar for all observers (coefficient of variation (CV): 6%, 6.8% and 4.9%) while reproducibility of FT derived PSCS (CV: 7.4%, 9.4% and 15.8%) varied depending on observer experience. Inter-observer reproducibility of tagging derived PSCS for observer 1 and 2 as well as 1 and 3 for tagging (CV: 6.17%, 9.18%) was superior in comparison to FT (CV: 11.8%, 16.4%).
CONCLUSIONS: Reliability and accuracy of FT based strain analysis, more than tagging based strain analysis, is dependent on reader experience. CMR strain experience or dedicated training in strain evaluation is necessary for FT to deliver accurate strain data, comparable to that of CMR tagging.

Introduction

Cardiac magnetic resonance (CMR) is considered the gold-standard for cardiac functional analysis [1], as unlike other modalities, CMR allows for comprehensive and precise appraisal of the entire left- and right ventricle (LV&RV) [2]. Although qualitative assessment of LV wall motion in CMR cine-images (i.e. visual assessment) has been shown to be reader dependent [3], it is currently the standard clinical practice. In contrast, quantitative wall motion (i.e. strain assessment) assessment methods, such as CMR tagging, have been shown to deliver robust, reproducible results [4]. However, to date clinically strain assessment has not been widely adopted due the necessity of additional scan acquisition as well as off-line post-processing. CMR myocardial feature tracking (FT) enables rapid and therefore clinically feasible quantitative wall motion analysis using standard balanced steady state free precession (bSSFP) cine scans. Although FT offers several advantages, one of the main identified drawbacks is the increased inter-observer variability [5,6]. In this regard the impact of observer experience on validity and variability of FT derived strain has not yet been investigated. The aim of this study was therefore first, to evaluate the observer experience-dependency of CMR FT and second, to compare results with the current reference standard for quantitative wall motion analysis - CMR tagging.

Methods

Study population

Controls (Group A) and patients with heart failure but preserved ejection fraction (HFpEF) (Group B) were prospectively enrolled into the study. HFpEF was diagnosed using standard criteria [7] according to the current European Society of Cardiology guidelines [8]: 1. signs of heart failure (HF); 2. preserved systolic LV function (ejection fraction ≥50%); 3. evidence of echocardiographically diagnosed diastolic LV dysfunction (DD) and/or surrogate markers (e.g. hypertrophy, elevated plasma levels of BNP) of diastolic LV dysfunction. DD was evaluated and graded by means of echocardiography as previously described [9]. All subjects gave their written informed consent before being included in this study, which received approval by the local institutional review board.

MR Imaging

Examinations were performed using a 1.5 T clinical MR scanner (Ingenia, Philips Medical Systems, Best, the Netherlands). For functional analysis retrospectively gated SSFP sequences were acquired in the standard cardiac axes [10]. Ejection fraction was assessed in short axis bSSFP sequences with a minimum of 12 short axis slices and 30 phases reconstructed per slice. For the evaluation of FT strain additional prospectively gated bSSFP cine images with 25 cardiac frames per RR-cycle were acquired in the short axis orientation at the midventricular level.

To ensure the highest possible congruency of scanning parameters between tagged images and bSSFP cine images prospective ECG gating was employed. Further scan parameters were: FOV 370 mm, TE/TR of 1.4/3.0 ms, flip angle 50°, slice thickness 8 mm, and in plane resolution of 1.4 mm. Tagged images were acquired in identical positions using the same number of cardiac frames [25] and an identical trigger delay. For tagged images the following parameters were used: complementary spatial modulation of magnetization in a grid pattern with a grid-gap space of 8 mm; FOV 320 mm, typical TE/TR 6/33 ms, flip angle 25°.

All images were analysed by an experienced reader (reader 1: 5 years of CMR experience, 5 years of experience in strain analysis), a second reader with 2 years of CMR experience (reader 2: 1.5 years of experience in strain analysis), and one reader with 1 year of CMR experience and no experience in strain analysis (reader 3). The third reader received a 30 min tutorial in both FT and tagging derived strain analysis. FT and tagging derived global peak systolic circumferential strain (PSCS) were calculated. To investigate the intra- and inter-observer reproducibility FT and tagging analysis was performed twice by all readers with an interval of two weeks between the first and second analysis. All readers were blinded to their own intermodality results as well as to each other's inter-observer results.

TAG analysis

Dedicated harmonic phase-analysis software (Tag Track, GyroTools Ltd., Zurich, Switzerland) was used to calculate midmyocardial strain. As previously reported, short axis circumferential strain values were derived from mid-left-ventricular short axis slices [6,11]. Tracking is commenced after manually drawing a midmyocardial track-line in a diastolic phase with optimal myocardium-blood contrast. Automatic propagation of track-lines (endocardial, midmyocardial or epicardial) throughout the entire RR cycle is achieved by using the grid crossing points as points of orientation. In case of faulty propagation track lines were manually corrected.

FT analysis

Dedicated software (Diogenes; TomTec; Germany) was employed to perform FT strain analysis. Short axis circumferential strain was calculated from the same midventricular short-axis slice as used for tagging analysis. Based on an initial manually drawn endocardial contour in an end-diastolic image the LV endocardial borders are identified over the entire RR cycle. The Feature Tracking method has been previously described elsewhere in detail [6]. In brief, strain evaluation in bSSFP sequences is achieved by assigning each voxel of the endocardial/epicardial border a number of characteristics (e.g. brightness and dishomogeneities of the tissue) in a defined phase which are then tracked in the following phases. Strain information can then be deducted from the endocardial/epicardial motion. In case of faulty propagation the track line can be re-adapted to the endocardial border in a selected phase, the software then propagates a new track line based on the manually made corrections.

Statistical analyses

Statistical analyses were performed with MedCalc (Medcalc Software, Mariakerke, Belgium). Results are expressed as mean ± standard deviation (SD). Comparison between tagging and FT derived peak systolic circumferential strain (PSCS) were performed with the Wilcoxon signed rank test. P-values of <0.05 were considered statistically significant. Intra- as well as inter-observer comparisons along with reproducibility were assessed with Bland–Altman plots [12], the Wilcoxon signed rank test and the coefficient of variation (CV) [13]. Correlation between FT and tagging derived PSCS was evaluated with the Spearman correlation coefficient. Correlation coefficients were graded depending on their value (r ≤ 0.35- weak correlation; r = 0.36 to 0.69 - moderate correlation; r = 0.70–1.0- strong correlation).

Results

A total of 19 healthy controls (7 female) (Group A) (32 ± 11 years, mean ejection fraction 63 ± 3%) and 19 patients (12 female) with HFpEF (Group B) (67 ± 18 years, mean ejection fraction 60 ± 8%) were included in the study. The study protocol could be completed in all participants. Fig. 1 demonstrates strain curves of a healthy volunteer computed by each of the three readers employing both, FT and TAG analysis. Table 1 summarizes subgroup results for Tagging and FT.

Fig. 1

Example of tagging (upper images: 1A–2C) and Feature Tracking (lower images: 3A-4C) derived strain assessment in a healthy volunteer completed by each of the three observers (A: experienced reader; B: intermediately experienced reader; C: inexperienced reader). Contour lines are placed in a diastolic image in a cspamm and SSFP image (1A&3A). The respective software (tagging and FT) propagates the contour throughout the cardiac cycle (2A–C&4A–C), however corrections may be necessary. Tagging and Feature Tracking derived strain curves (1D & 3D) for observer 1 (blue graph), observer 2 (red graph) and observer 3(yellow graph). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 1

Subgroup comparison of tagging and feature tracking derived strain results.

	Tagging volunteers	Tagging HFpEF patients	Tagging overall	FT volunteers	FT HFpEF patients	FT overall
Observer 1	−21.99 ± 2.2%	−20.15 ± 4.2%	−21.04 ± 3.5%	−21.38 ± 3.3%	−20.41 ± 4.3%	−20.89 ± 3.8%
Observer 2	−21.7 ± 2.5%	−20.15 ± 4.3%	−20.91 ± 3.5%	−20.09 ± 3.1%	−19.19 ± 6.2%	−19.57 ± 4.9%
Observer 3	−22.44 ± 4.6%	−20.26 ± 5.2%	−21.32 ± 4.2%	−19.8 ± 3.9%	−18.87 ± 5.6%	−19.31 ± 4.8%
Observer 1 vs. Observer 2	p = 0.51/r = 0.78	p = 0.98/r = 0.93	p = 0.89/r = 0.84	p = 0.01/r = 0.71	p = 0.21/r = 0.63	p = 0.04/r = 0.71
Observer 1 vs. Observer 3	p = 0.29/r = 0.76	p = 0.9/r = 0.74	p = 0.59/r = 0.71	p = 0.01/r = 0.62	p = 0.13/r = 0.58	p = 0.02/r = 0.62

Tagging and FT derived peak circumferential strain results in healthy volunteers and patients with heart failure with preserved ejection fraction for observer 1 (experienced reader), observer 2(moderately experienced reader) and observer 3 (inexperienced reader) as well as interobserver comparison with Wilcoxon rank test and Spearman's correlation coefficient.

Bold data statistically significant p values < 0.05.

Reader 1

Using tagging analysis mean midventricular PSCS was −21.04 ± 3.5%, while FT derived mean midventricular PSCS was −20.89 ± 3.8%. Correlation was moderate (r = 0.65) and results did not significantly differ from each other (p = 0.74). Intra-observer variability of PSCS yielded identical mean differences yet an increased deviation for FT (0.4 ± 2.4 (95% CI: −1.1 to 0.4) (FT) vs. 0.4 ± 1.7 (95% CI: −0.95 to 0.18) (tagging)). The intra-observer coefficients of variation were 6% for tagging and 7.4% for FT derived PSCS. Results of subgroup intra-observer reproducibility are given in Table 2.

Table 2

Intra-observer reproducibility of tagging and feature tracking.

	Tagging volunteers	Tagging HFpEF patients	Tagging overall	FT volunteers	FT HFpEF patients	FT overall
Observer 1	4.8%	6.2%	6%	5.1%	8.1%	7.4%
Observer 2	5.2%	6.1%	6.8%	7.5%	12.4%	9.4%
Observer 3	7.2%	6.7%	4.9%	9.8%	19.4%	15.4%

Intra-reader reproducibility of tagging and FT derived peak systolic circumferential strain assessed by the coefficient of variation for multiple measurements.

Reader 2

Tagging derived mean midventricular PSCS was −20.91 ± 3.5%, while FT derived midventricular PSCS was −19.57 ± 4.9%. Correlation was moderate (r = 0.54), results did not differ significantly (p = 0.09). Intra-observer variability of PSCS yielded higher mean differences but similar deviation for FT (0.8 ± 1.8 (95% CI: −4.4 to −2.4) (FT) vs. 0.06 ± 2 (95% CI: −0.72 to 0.6) (tagging)). The intra-observer coefficients of variation were 6.8% for tagging and 9.4% for FT derived PSCS. Results of subgroup intra-observer reproducibility are given in Table 2.

Reader 3

Tagging derived mean midventricular PSCS was −21.32 ± 4.2%, while FT derived midventricular PSCS was −19.31 ± 4.8%. Correlation was only moderate (r = 0.48) and results differed significantly (p = 0.0061). Intra-observer variability of PSCS yielded slightly higher mean differences and an increased deviation for FT (1.17 ± 4.3 (95% CI: −2.6 to 0.2) (FT) vs. 0.32 ± 1.3 (95% CI: −0.74 to 0.09) (tagging)). The intra-observer coefficients of variation were 4.9% for tagging and 15.4% for FT derived PSCS. Results of subgroup intra-observer reproducibility are given in Table 2.

Inter-observer comparison

An excellent reproducibility as well as a strong correlation was found between reader 1 and 2 for tagging analysis (CV: −6.17%; r = 0.84). In comparison, FT inter-observer reproducibility and correlation for reader 1 and reader 2 were less strong (CV: 11.8%; r = 0.71) (Fig. 2). Inter-observer Bland-Altman analyses for reader 1 and 2 yielded a superior agreement for tagging (mean difference of −0.14 (95% CI: −0.74 to 0.46)) in comparison to FT (mean difference of 1.3 (95% −2.44 to −0.18)) (Fig. 2). Inter-observer comparison between reader 1 and reader 2 yielded significantly different results for FT derived strain (p = 0.02).

Fig. 2

Inter-observer correlation between observers 1 and 2 (tagging: A; Feature Tracking C) as well as 1 and 3 (tagging: B; Feature Tracking: D) for PSCS in healthy volunteers (blue dots) and in patients with HFPEF (red triangles). Bland-Altman Plots for interobserver agreement between observers 1 and 2/1 and 3 for tagging (E/F) and Feature Tracking (G/H) derived PSCS, the plots show higher inter-observer agreement for tagging derived PSCS. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

A good reproducibility as well as a strong correlation was found between reader 1 and 3 for tagging (CV: −9.18%; r = 0.71) (Fig. 2), in comparison FT inter-observer reproducibility and correlation for reader 1 and reader 3 were less strong (CV: 16.4%; r = 0.62) (Fig. 2). Inter-observer Bland-Altman analyses for reader 1 and 3 yielded a superior agreement for tagging (mean difference of 0.28 (95% −0.62 to 1.18)) in comparison to FT (mean difference of 1.7 (95% CI: −2.72 to −0.42)) (Fig. 3). Inter-observer comparison between reader 1 and reader 3 yielded significantly different results for FT derived strain (p = 0.008).

Fig. 3

Bland-Altman Plots for inter-observer agreement between observers 1 and 2/ 1 and 3 for tagging (A/B) and Feature Tracking (C/D) derived PSCS, the plots show higher inter-observer agreement for tagging derived PSCS.

Subgroup comparison

Tagging derived results for subgroup intra-observer reproducibility showed superior results for both groups with less variation between results (Table 2). Furthermore, significant differences were found between reader 1 and 2 as well as 1 and 3 for feature tracking derived results for the healthy control group in subgroup comparison (Table 1).

Discussion

The novel finding of the present study is that CMR FT based strain analysis is significantly more dependent on reader experience than CMR tagging. While a notable intra-/inter-observer and interstudy variability has been reported for CMR FT [5,14,15,16], to the best of our knowledge, this is the first study investigating the reader experience dependency of wall motion assessment using CMR feature tracking and comparing it to CMR tagging.

Although previous studies have demonstrated that qualitative CMR based analysis of wall motion, as performed in stress testing, is greatly dependent on reader experience [17] it remains the clinical CMR standard [1]. In a recent dobutamine stress test study it could be shown that FT derived quantitative wall motion parameters were less reader dependent than mere visual analysis [18], nevertheless inter-observer congruency was still an issue. In comparison CMR tagging based quantitative wall motion assessment is not only less reader dependent and more sensitive than the visual approach [19] but also very robust [20]. Previous studies comparing tagging and FT have mainly reported a rather good agreement and correlation between both modalities [6,14]. However, the current results indicate that a good correlation between the two modalities (CMR tagging vs. CMR FT) is only granted if the data is analysed by experienced readers. FT derived strain data of the inexperienced reader showed only moderate correlation with tagging. While FT derived strain results were lower than tagging derived strain results for all readers, it is noteworthy that the degree of strain underestimation depended on the experience of the reader.

It can be assumed that with growing experience placement and appraisal of track lines improves in both FT and tagging analysis. The intermediately experienced reader and the inexperienced reader either placed inferior FT track lines or had difficulties to adapt track lines in case of faulty propagation. In fact it has been established that suboptimal tracking in FT leads to an underestimation of strain [21]. While in theory both FT and tagging allow for track line adaptation, complete manual adaptation can only performed in tagging analysis. FT only allows for manual adaptation in one frame, followed by an automatic re-propagation of the track line, possibly nullifying the changes that were made before. As the FT software does not provide information on which clusters are tracked or on tracking quality, correction of track lines can be challenging and depends mostly on reader experience. Although analysis of tagged images is considered to be complicated and time consuming [6,22], results between readers are generally highly congruent, indicating that image analysis is not more difficult than FT, even for an inexperienced reader.

Tagging images required an acquisition time of ~14 s, while balanced SSFP images for FT analysis required an acquisition time of ~8 s, depending on the subject's heart rate and rhythm. Postprocessing duration for tagging-based strain assessment was ~5 min on average, while FT-based analysis of strain in balanced SSFP images could be accomplished in ~3.5 min, making FT-based strain the more rapid appraisal tool.

A high reader experience dependency and the related inaccuracy, potentially invalidate the advantages of quantitative strain assessment. In fact many cardiac diseases, which may be detected and characterized by strain analysis, are often characterized by only mild systolic or diastolic strain impairment [[23], [24], [25]]. A strain assessment tool used for clinical and/or scientific application ideally should deliver precise and robust results, otherwise mild systolic or diastolic dysfunction may either be overlooked or falsely diagnosed during primary assessment or follow-up.

Previous studies assessing the reproducibility of strain analysis found excellent results for tagging (CV 3.7% to 5.5%) [20] and good results for FT (CV 5%–13.3%) [26,27]. It can be assumed that inter-observer and interstudy reproducibility were tested by experienced readers. In the current study reproducibility of tagging results were similar for all readers, including the inexperienced reader, while reproducibility of FT significantly varied depending on reader experience. We believe that a scenario where inexperienced readers use strain analysis software is not uncommon and rather represents the actual clinical situation. Therefore it is of essential value that strain analysis results do not significantly vary depending on reader experience.

While reproducibility of tagging derived strain results was similarly strong for both patients and volunteers independent of reader experience, reproducibility of FT derived strain was inferior in patients in comparison to healthy volunteers, especially for the non-experienced reader. Although image quality was sufficient in all cases, typically image quality and respective endocardial border delineation is superior in healthy volunteers in comparison to cardiac patients [28]. Therefore, the increased variability of FT in part may be a result of the high contrast dependency of FT [26]. As previously concluded [5,15] the current FT algorithm should be improved to reduce variability which would also allow inexperienced readers to clinically and scientifically rely on this technique.

Limitations

Though the number of patients (n = 19) and controls (n = 19) included in the study was rather small, the total number of tests performed (n = 228) was large enough to demonstrate the increased observer dependency of FT.

In this study only circumferential midventricular strain assessment was analysed. Longitudinal strain, which is also considered a robust FT strain parameter, was not evaluated.

Prospective ECG-gating was employed for the acquisition of bSSFP sequences, which may have an impact on image quality. However, prospective ECG-gating was necessary in order to achieve the highest possible congruency between bSSFP and tagging sequences.

In conclusion, this study demonstrates that the reliability and accuracy of FT based CMR strain analysis, more than tagging based strain analysis, is dependent on reader experience. CMR strain experience or dedicated training is necessary for readers to produce accurate strain data with FT, comparable to that of CMR tagging.

Clinical relevance

In theory CMR FT allows for reliable estimation of myocardial strain from standard bSSFP images. While this offers the prospect of clinically feasible strain analysis and the deduction of additional, possibly clinically relevant information without the necessity of additional sequence acquisition, the current results indicate that FT required reader experience to deliver accurate strain results. These findings emphasize that dedicated training is necessary before non-experienced users perform FT strain analysis in a clinical or scientific setting.

Funding

No funding has been received for this study.

Conflict of interest

The authors report no relationships that could be construed as a conflict of interest.