Quality assurance of quantitative cardiac T1-mapping in multicenter clinical trials - A T1 phantom program from the hypertrophic cardiomyopathy registry (HCMR) study.

BACKGROUND: Quantitative cardiovascular magnetic resonance T1-mapping is increasingly used for myocardial tissue characterization. However, the lack of standardization limits direct comparability between centers and wider roll-out for clinical use or trials.
PURPOSE: To develop a quality assurance (QA) program assuring standardized T1 measurements for clinical use.
METHODS: MR phantoms manufactured in 2013 were distributed, including ShMOLLI T1-mapping and reference T1 and T2 protocols. We first studied the T1 and T2 dependency on temperature and phantom aging using phantom datasets from a single site over 4 years. Based on this, we developed a multiparametric QA model, which was then applied to 78 scans from 28 other multi-national sites.
RESULTS: T1 temperature sensitivity followed a second-order polynomial to baseline T1 values (R2 > 0.996). Some phantoms showed aging effects, where T1 drifted up to 49% over 40 months. The correlation model based on reference T1 and T2, developed on 1004 dedicated phantom scans, predicted ShMOLLI-T1 with high consistency (coefficient of variation 1.54%), and was robust to temperature variations and phantom aging. Using the 95% confidence interval of the correlation model residuals as the tolerance range, we analyzed 390 ShMOLLI T1-maps and confirmed accurate sequence deployment in 90%(70/78) of QA scans across 28 multiple centers, and categorized the rest with specific remedial actions.
CONCLUSIONS: The proposed phantom QA for T1-mapping can assure correct method implementation and protocol adherence, and is robust to temperature variation and phantom aging. This QA program circumvents the need of frequent phantom replacements, and can be readily deployed in multicenter trials.

Introduction

CMR parametric mapping permits the quantification and spatial visualization of changes in myocardial composition based on changes in T1, T2 relaxation times, and extracellular volume (ECV) [1]. Native T1-mapping has been shown to have narrow normal ranges within the same method, and sensitivity to a wide range of myocardial diseases [2,3]. Its advantages include relatively simple single breath-hold acquisitions, excellent reproducibility, and avoidance of gadolinium-based contrast agents.

However, T1 values depend on the protocol parameters, acquisition and reconstruction methods [4], currently requiring cumbersome establishment of within-center norms [1,5]. Native T1-mapping can differentiate disease from healthy tissue in single-site studies at a fixed field strength, and demonstrated reproducibility in multi-center studies under a uniform imaging set-up, but the lack of standardization limits direct comparability between centers and wider roll-out for clinical use. For clinical sites attempting to implement T1 mapping, it is often unclear how to install and validate the methods correctly before using them for clinical diagnosis, or for assessing novel therapeutics in multi-center trials. Despite SCMR consensus recommendations [1], there has been no working solution to-date for standardization – not even for a single T1-mapping technique on a single vendor platform. The use of local normal ranges and z-scores to adjust for differences have been proposed [1], but are suboptimal, as small sample size (e.g. 10–20) healthy volunteers can be prone to sampling errors [1,5]; outliers or biases could directly affect a center's ability to detect abnormal findings to diagnose disease. While these effects decrease with larger sample sizes, re-acquiring such data frequently just to monitor stability after each method or scanner upgrade becomes prohibitively cumbersome and expensive.

Validation and quality assurance (QA) of single-method deployment could be a first step to standardize T1-mapping techniques, increasing the confidence of individual centers in the set-up of CMR mapping for clinical use. The large, international multicenter Hypertrophic Cardiomyopathy Registry (HCMR) study [6] adopted a single T1-mapping method (Shortened Modified Look-Locker Inversion Recovery, ShMOLLI [7]) to maximize comparability of the datasets to power for outcomes. This study setup provided an opportunity to collect the required phantom data to develop a QA program for standardizing T1-mapping between centers, using an original multiparametric modelling approach. The derived QA model can reliably detect deviations from correct method implementation and protocol adherence, despite changes in phantom properties, including temperature variations [8,9] and aging effects, unlike most phantom solutions currently available [9,10].

Materials and methods

Study design

In 2013, a batch of 50 original dedicated QA phantom devices was designed for the prototype ShMOLLI T1-mapping method [7]. The manufacture details are provided in Supplemental Section S1. The external appearance and arrangement of the phantom compartments are shown in Fig. 1a, b, and the achieved T1 and T2 combinations in Table 1.

Fig. 1

HCMR QA phantom. (a) Phantom external appearance. (b) Phantom compartment arrangement. (c) An example of T1 map in QA post-processing. Black dashed lines indicate the automatically detected ROIs. The two phantoms on either side are not part of QA; their use is recommended to assure adequate coil loading.

Table 1

Phantom chemical composition and T1/T2 relaxation times measured at room temperature within one month of manufacture. ShMOLLI T1 are average values of a batch of 50 phantoms measured at 1.5 T and 3 T. The T2 are average values of five randomly selected phantoms at 1.5 T and another five at 3 T scanned in an additional single measurement.

Phantom compartment and formulation		ShMOLLI T1 [ms] (mean ± SD, N = 50)		Spin-echo T2 [ms] (mean ± SD, N = 5)
		1.5 T (21.3 ± 0.4 °C)	3 T (21.0 ± 0.5 °C)	1.5 T (21.0 °C)	3 T (21.0 °C)
A	0.5% Agar, 0.33% Carrageenan, 0.113 mM NiCl2	2529.5 ± 14.0	2461.0 ± 7.8	275.8 ± 2.1	265.9 ± 4.8
B	0.5% Agar, 0.626 mM NiCl2	1396.2 ± 5.9	1329.4 ± 2.2	266.2 ± 4.6	259.0 ± 2.2
C	Undoped 18 MΩ deionized H2O	3251.5 ± 12.5	3234.9 ± 27.8	2373.4 ± 184.3	2383.1 ± 153.9
D	1.9% Agar, 1.2 mM NiCl2	859.1 ± 2.9	804.71 ± 0.74	72.2 ± 0.7	71.4 ± 0.4
E	2% Agar, 0.77 mM NiCl2	1109.7 ± 12.5	1051.3 ± 1.9	69.6 ± 1.2	70.7 ± 0.8
F	2% Agar, 0.524 mM NiCl2	1397.8 ± 4.7	1328.0 ± 2.2	80.2 ± 1.4	78.4 ± 2.8
G	1.5% Agar, 0.1% Carrageenan, 4.5 mM NiCl2	323.1 ± 0.66	307.5 ± 0.6	72.4 ± 0.7	68.5 ± 1.1
H	3% Agar, 0.457 mM NiCl2	1428.8 ± 3.9	1368.1 ± 3.4	56.9 ± 0.4	56.5 ± 2.6
I	1.8% Agar, 2 mM NiCl2	610.2 ± 1.3	574.4 ± 0.5	74.0 ± 0.7	72.5 ± 1.5

The QA program aimed to assure the detection and compensation of potential differences in T1-mapping sequence properties between sites to track T1 measurement stability [6]. Centers operating Siemens MR scanners and performing measurements free of charge (N = 28) were provided a consistent protocol developed at Oxford core lab, adapted for various Siemens software platforms. The sites were provided with imaging manuals and HCMR QA protocols to perform repeated ShMOLLI-T1 acquisitions, an inversion recovery spin echo (IR-SE) acquisition and a multi-echo SE acquisition for reference T1 and T2 maps (protocol specified in Supplemental Section S2). The multi-center dataset collected from the HCMR sites served to monitor site-specific changes of the T1-mapping technique properties. Meanwhile phantoms were repeatedly scanned at Oxford core lab on a MAGNETOM Avanto (1.5 T) and a MAGNETOM Trio Tim (3 T) scanners (Siemens Healthcare, Erlangen, Germany). This high-volume dataset served to investigate temperature and age dependencies as well as to establish the QA protocol.

The phantom scans were uploaded by individual HCMR sites to Boston core lab and then sent to Oxford core lab for data analysis between July 2014 and December 2017, when the first phase of the HCMR study was completed [6]. 28 HCMR sites using Siemens scanners acquired and sent 78 scans (a complete list of HCMR sites providing the phantom scans is given in Supplemental Table S1). Meanwhile, locally at Oxford core lab, we acquired 441 phantom studies between October 2013 and June 2017. Each study scanned one to three phantoms, which provided a total of 1004 phantom scans for analysis.

All reference T1 and T2 maps were calculated offline by a C++ library [11]. We analyzed the T1 and T2 using the robust mean values from the circular (15 mm diameter) regions of interest within each of the phantom compartments (Fig. 1c), which were placed automatically by a machine learning algorithm [12]. Altogether, from each set of phantom scans, we obtained 9 sets of reference T1 (T1ref) and T2 measurements paired with 5 repeats of corresponding ShMOLLI-T1 (T1sh) for each compartment.

T1 and T2 dependencies on temperature and phantom aging

To investigate the T1 and T2 dependency on temperature, we scanned phantoms at temperatures ranging from 16 °C to 28 °C. To analyze phantom property drifts over time, four selected phantoms were scanned approximately every two weeks between February 2014 and June 2017, with an average of 148 studies acquired per phantom over a 40-month period. We evaluated time trends in T1ref and T2, corrected to room temperature (21 °C) using the established temperature dependencies.

T1 prediction model and T1-mapping quality assurance

We postulated that the measured cardiac ShMOLLI-T1 has a predictable dependency on T1ref and T2 in phantoms. Using the dataset acquired locally at Oxford core lab, we established a single empirical equation exploiting the inherent relationship, T1sh = Function (T1ref, T2), for all T1 and T2 combinations. The residual of fitting was calculated to establish the range of agreement for the QA program. The equation was then used to predict the expected ShMOLLI-T1 from T1ref and T2, and compare it with observed ShMOLLI-T1 in the same QA scan to check the agreement. Phantom scans with ShMOLLI-T1 of all compartments within the agreement range (the 95% confidence interval (CI) of the residual of fitting; see results section) would indicate an accurate ShMOLLI sequence deployment and, therefore, QA passed. If any ShMOLLI-T1 fell outside the agreement range, the pattern of discrepancy and the scanning parameters were then further inspected, to identify potential sources of error. These may include artefacts, reconstruction error or protocol error, and are given a conditional pass with specific recommendations for remedial actions. Failing to identify and rectify errors would suggest that the ShMOLLI sequence was deployed incorrectly, and, therefore, QA failed. The capacity of this method was validated on the multi-center HCMR phantom dataset for Siemens scanners, with proof-of-principle translation demonstrated on the Philips platform.

Results

T1 baseline values and temperature dependency

At the time of manufacture, all 9 compartments of the 50 phantoms made as a batch had consistent ShMOLLI-T1 values with coefficient of variation within 0.85%, all measured at room temperature of ~21 °C (Table 1). In contrast to in-vivo myocardial T1 values [7], ShMOLLI-T1 values of all phantom compartments were lower at 3 T than at 1.5 T by 0.5–6.8% (all p < 0.001). Phantom T2 were typically lower at 3 T by 2.1–3.7% (all p < 0.001), with the exception of compartment #E where T2 was 1.6% higher at 3 T than at 1.5 T (p < 0.001). In compartments #C and #H, the T2 difference did not reach statistical significance (Table 1).

In subsequent experiments, T1ref showed clear linear dependency on temperature at both 1.5 T and 3 T (Fig. 2a). The variation of predicted T1 changes with temperature was 2.6% ± 1.5% at 1.5 T and 2.6% ± 1.2% at 3 T for all compartments, relative to the baseline T1 in Table 1. The temperature sensitivity increased with baseline T1, following closely a second-order polynomial (R2 > 0.996; Fig. 2b).

Fig. 2

Temperature sensitivity of reference T1 in the Oxford core lab dataset at 1.5 T and 3 T. (a) T1 temperature dependency. Temperature sensitivity coefficients (ΔT1/Δt, ms/°C) are provided to the right of the graph, prefixed with the compartment ID. Regression lines are omitted for clarity. (b) T1 temperature sensitivity coefficients (Y-axis) follow a second-order polynomial to baseline T1 values (X-axis).

Impact of phantom aging on T1

A wide range of aging effects were observed on the reference T1 measurements, with the largest seen in compartment #B in three of the four phantoms scanned repeatedly at Oxford core lab (Fig. 3). These phantoms appear to have undergone a transition, whereby the baseline T1 values had increased by ~50%. This may be due to various factors during the manufacture, particularly the amount of air within the universal container tubes and the resilience of the caps. T2 characteristics are provided in supplemental Fig. S4 and S5.

Fig. 3

The varied appearance of age-related drifts in reference T1 (T1ref) in individual phantom compartments observed over a period of 40 months. All T1ref values are corrected to room temperature. Colors represent the four phantoms investigated.

T1 prediction model robust to temperature variation and phantom aging

Consistent with prior work, ShMOLLI-T1 generally followed a linear relationship with T1ref, with visibly larger underestimation in compartments with shorter T2 (Fig. 4a, red arrows). To establish a model independent of phantom age and temperature, we exploited the inherent relationships between ShMOLLI-T1 and T1ref and T2, based on the 1004 dedicated phantom datasets. We accounted for the linear dependency of ShMOLLI-T1 on T1ref, and an exponential dependency on T2 (coefficients calculated for 1.5 T and 3 T, respectively; full data samples in Supplemental Fig. S6). While this model significantly reduced the residual differences between ShMOLLI-T1 and reference T1 methods from 7.24% to 1.64% (p < 0.001), there was a remaining small trend visible in the fit residuals. This was rectified with a third-order polynomial correction. The final multivariate correlation model predicting the ShMOLLI-T1 () from reference measurements were thus established; see derived equations in Suplemental S3. The model predicted which agreed with real T1sh with high accuracy (R2 > 0.99, Fig. 4b), despite the temperature variation and phantom aging effects described above. Analysis of the residual errors of the ShMOLLI prediction model allowed the establishment of the 95% confidence interval CI = ±3.12%, and the 99.7% confidence interval CI = ±5.32%, robust to temperature and aging confounders. The 95% CI was used as the tolerance range for the QA (full data samples in Supplemental Fig. S7).

Fig. 4

ShMOLLI T1 (T1sh) against reference T1 (T1ref) in the Oxford core lab dataset before T2 correction (a) and after T2 correction (b) at 1.5 T and 3 T. (a) The visible deviations (red arrows) were driven predominantly by T2 effects. (b) The multivariate correlation model including T2 effects predicts T1sh with high accuracy (R2 > 0.99). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Clinical application: T1-mapping quality assurance

We received 94 phantom scans: 78 from 28 Siemens sites, 15 from 6 Philips sites, and 1 from a General Electric (GE) site (complete list of sites in Supplemental Table S1). The vendor and scan distribution reflected the resources available and local feasibility during this study, which precluded fair head-to-head inter-vendor comparisons. We were able to perform QA and present the findings in 78 scans (390 ShMOLLI-T1 maps) from the 28 Siemens sites (Table 2 and Fig. 5). We exemplified the need for further work for inter-vendor application of the QA model (Fig. 5f), subject to sufficient datasets acquired on those MR systems.

Table 2

QA results of 78 Siemens scans from 28 sites with proposed outcomes and actions.

QA Results	Description	No. scans (sites)	Action recommended
1. Passed	All T1 maps in the scan provided ShMOLLI-T1 (T1sh) values within the agreement range with expected T1^sh (Fig. 5a).	34 scans (15 sites)	QA passed.No further action required
	Disagreement between T1sh and T1^sh in one or more individual compartments; the rest were in the agreement range. The source of error can be identified as image artefacts not linked to cardiac T1 maps (Fig. 5b).	3 scans (3 sites)	QA passed.Consider technical investigation.
2. Warnings	Underestimated T1sh in individual acquisitions. At least one acquisition is within the agreement range. Incomplete recovery of longitudinal magnetization in individual acquisitions (Fig. 5c).	26 scans (16 sites)	QA passed, with warning of possible protocol adherence problems
3. Conditional	T1sh values outside the agreement range; Source of disagreement caused by T1 map fitting without ShMOLLI conditional fitting reconstruction, but accurate T1sh values were successfully restored offline (Fig. 5d).	7 scans (5 sites)	QA conditional on offline reconstruction. Require re-deployment of T1 sequence
4. Failed	T1sh values outside the agreement range; unable to identify source of error. Unable to restore accurate T1sh values offline (Fig. 5e).	6 scans (3 sites)	QA not passed. Technical investigation required
	QA could not be performed due to missing reference T1 or T2 sequences in the scan.	2 scans (2 sites)	Incomplete scan. Check protocols and repeat QA

Fig. 5

Phantom QA of T1-mapping for 6 example CMR centers. In each panel, the top graph shows observed ShMOLLI T1 (T1sh) and the bottom graph the residuals (y-axis) displayed against the expected (x-axis). (a) QA passed as all within 95% CI (green range). (b) QA passed with artefacts in individual compartment(s). (c) QA passed with warning. Underestimated T1sh in individual acquisitions. (d) QA conditional on T1sh offline reconstruction. Inline reconstruction failed (gray circles). (e-f) QA failed due to presence of patterns and variability of the observed residuals reaching outside the tolerance range. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

QA passed

QA was passed if the dataset from a session contained at least one accurate ShMOLLI-T1 acquisition for all 9 compartments. 34 scans from 15 sites gave ShMOLLI-T1 values within the prescribed 95% CI range of the expected ShMOLLI-T1, confirming accurate ShMOLLI deployment (Fig. 5a). Three scans from 3 sites showed departure of ShMOLLI-T1 values in individual phantom compartments, with the rest being within the agreement range. We identified the sources of these outliers as imaging artefacts in the reference T1-maps (Fig. 5b). These 3 scans were considered to have passed the QA, as the source of discrepancies was clearly outside the cardiac T1-mapping technology, and the ShMOLLI-T1 values of the rest of the compartments were within the agreement range.

Common error 1: Lower T1s due to inadequate waiting time

A further 26 scans from 16 sites provided at least one accurate ShMOLLI acquisition but with a characteristic pattern of greater underestimation of ShMOLLI-T1 values in longer T1 compartments in other acquisitions of the scan (Fig. 5c), suggesting incomplete recovery of magnetization. This was likely caused by inadequate waiting time after the previous image acquisition, frequency adjustment, or shimming adjustment. We reproduced this T1 underestimation at Oxford core lab with dedicated experiments, and studied the impact of incomplete magnetization recovery on the measured T1. The experimental details are provided in Supplemental Section S4. This error may be rectified by reminding the operators to wait at least 10 s in between T1-map acquisitions.

Common error 2: failed in-line reconstruction of T1-mapping

We detected incorrect in-line ShMOLLI-T1 reconstruction in 7 additional scans from 5 sites (Fig. 5d, gray circles). We were able to reconstruct and salvage T1-maps offline using raw T1-weighted images, and restored the accurate T1 values (Fig. 5d, blue points). This indicated correct T1-weighted image acquisition but inaccurate inline reconstruction on the MR system. Therefore, these sites passed QA conditionally on offline reconstruction, but required redeployment of the T1 sequence.

QA failed

Six scans did not provide T1 measurements falling wholly within the prescribed tolerance limit, and therefore did not pass QA (Fig. 5e). A further 2 scans from 2 sites contained no reference T1 or T2 acquisitions, and thus the QA was not performed. Similarly, incompatible measurement protocols prevented a convincing inter-vendor validation, with one example shown as a proof of principle (Fig. 5f).

Discussion

In this work, we have established a novel approach for quality assurance (QA) that is independent of the actual physical properties of the phantoms, bypassing the exhibited phantom sensitivities to temperature variations and aging. We have demonstrated how to use this QA to verify correct implementation of T1-mapping methods to within a prescribed tolerance range across multiple scanners and magnetic field settings; signature patterns of departure also identified common errors for actionable remediations.

Inter-method and inter-vendor applications

We have illustrated how to achieve this phantom QA solution using a single T1-mapping method, but the solution can be deployed to other mapping techniques. The HCMR Consortium had chosen a single T1-mapping method (ShMOLLI [7]) to maximize comparability of the datasets to power for outcomes. The ShMOLLI T1-mapping sequence possesses three characteristics which made this work possible within 5 years: (1) method stability and full accountability of its quantitative characteristics, such that it was feasible to compare it on a wide range of platforms within a single vendor setting; (2) heart-rate independence, alleviating the need to vary heart rate as part of the QA process; (3) ShMOLLI is a single universal technique suitable for measuring a wide range of T1 values (whether short or long), thus obviating the need for developing separate QA models, each for a specific MOLLI sub-variant for short or long T1s.

The QA solution may be applied to any other quantitative mapping methods across vendor platforms, subject to the exact application of the specified protocols. The provided scanning protocol (Supplemental Section S2) and presented QA formula can be readily tested on other vendor MR systems as published, subject to the availability of vendor research personnel and support of technical development sites. Even though our experience indicates that this is far from trivial, by setting clear and achievable QA goals, the proposed QA program offers for the first time a practical way towards a fully compatible inter-vendor T1 methodology.

Study limitations and future work

The proposed QA program was demonstrated on a single vendor platform. Although the general approach can be applied across vendors, the investigators' experience indicates that translation to other vendors and methods will require significant further technical investment, to address the large intervendor differences between documented similar methods [10]. The authors are open to collaborative requests to employ the proposed QA to improve the T1-mapping consistency in multi-vendor setting. In future, the described QA should be either implemented directly on the scanner or be provided as a prompt external service, so that end-users can assure method stability with the same frequency as the usual scanner QA. This QA is designed to test correct technical deployment of methods; operator compliance with the acquisition protocols and standardization of image processing [13] are required to minimize introduction of errors.

Conclusions

We presented the development of a practical MR phantom QA program for accurate and comparable T1 measurements for use in a large multi-center setting. The QA model is robust to phantom aging and ambient temperature variations, circumventing the need for manual temperature corrections and frequent phantom replacements. This provides an immediate and economical solution to verify correct T1-mapping method implementation, and identify common errors for remediation. The proposed QA program paves the way to widespread adoption of T1-mapping into routine clinical practice.

Funding

This work is funded by project grant PG/15/71/31731 (QZ, KW, SKP) and Oxford Biomedical Research Centre at The Oxford University Hospitals (IAP, SKP, VMF and SN).

Declaration of Competing Interest

SKP has patent authorship rights for U.S. patent US20120078084A1. Systems and methods for shortened Look Locker inversion recovery (Sh-MOLLI) cardiac gated mapping of T1. Granted March 15, 2016. IP is managed by Oxford University Innovations; the license exclusively transferred to Siemens Healthcare.

QZ, SKP, KW, IAP, VMF have authorship rights for pending patent PCT/GB2020/051189. A method for identity validation and quality assurance of quantitative magnetic resonance imaging protocols. Filed May 15, 2020. IP is owned and managed by Oxford University Innovations.