Inter-observer reproducibility of quantitative dynamic susceptibility contrast and diffusion MRI parameters in histogram analysis of gliomas.

Introduction

Grading of gliomas, the most common primary brain tumor (1,2), is important since the prognosis and adjuvant therapy after surgery differ according to tumor grade (3,4). Conventional imaging parameters, such as contrast enhancement on T1-weighted magnetic resonance imaging (MRI), do not provide a reliable indicator of tumor grade (5,6). Diffusion-weighted imaging (DWI) and perfusion-weighted imaging (PWI) provide a plethora of additional parameters to overcome the shortcomings of conventional MRI. Differentiation between low and high glioma tumor grades has been demonstrated with dynamic susceptibility-weighed (DSC)-MRI, yielding the relative cerebral blood volume (rCBV) (7–11). Additionally, the additional value has been documented of dynamic contrast-enhanced (DCE)-MRI derived parameters such as blood plasma volume (Vp) (7,8), volume transfer constant (Ktrans) (7,8,10,12), rate constant (kep) (9,10), and normalized tumor cerebral blood flow (nCBF) (10,12,13). In addition, quantitative DWI expressed in the apparent diffusion coefficient (ADC) shows significant differences between low- and high-grade gliomas (8,13,14).

In our opinion, a systematic review of the robustness of the various quantitative measures of perfusion and diffusion in brain tumors, and of the impact of the methodology of region of interest (ROI) selection, is lacking. The purpose of this study was therefore to compare inter-observer reproducibility of DSC-MRI and DWI-derived tumor characteristics in 30 gliomas. Three observers analyzed the MRIs according to the common practice of manual delineation of the tumor as visualized on the respective PWI and DWI series, and of the equally widespread use of small circular ROIs to simulate fast evaluation in clinical practice. Interclass correlation coefficients (ICC) were determined and differentiation between low- and high-grade glioma was assessed for each parameter.

Material and Methods

Study population

The protocol of this retrospective study was approved by the hospital’s institutional review board and patients’ written informed consent was waived. Between March 2006 and March 2014, patients with suspected brain lesions and scheduled for MRI were recruited. Inclusion criteria were: MRI scan protocol including both DSC and DWI series; and lesion diameter ≥2 cm on MRI. This resulted in the inclusion of 30 consecutive patients (16 boys/men; mean age = 40.6 years; age range = 5–78 years) with 30 lesions: 13 were low-grade and 17 were high-grade. Tissue samples were obtained in all patients using neuro-navigational guided core biopsy (n = 13) or surgical resection (n = 17).

MR protocols

All participants were examined on a 1.5-T MRI system (Magnetom Sonata, Siemens Medical Solutions, Erlangen, Germany) using the eight-channel head coil. Diffusion tensor imaging (b = 0 and 1000 s/mm2) was scanned with the following: single shot echo-planar imaging (EPI); TR = 5800 ms; TE = 92 ms; flip angle (FA) = 90°; slice thickness = 3 mm; slice gap = 3 mm; field of view (FOV) = 230 × 230 mm; matrix = 256 ×256; bandwidth = 1565 Hz/pixel; no averages. Diffusion gradients (40 mT/m) in 12 directions; 37 transverse slices; acquisition time = 2.2 min. ADC maps were calculated on the scanner. Finally, DSC perfusion: 70 transversal time series (EPI, epfid2d1); temporal resolution = 1.0 s; gadoterate meglumine (Gd-DOTA, Dotarem, Guerbet, 0.1 mmol/kg): TR = 1000 ms; TE = 40 ms; FA = 60°; slice thickness = 6 mm; slice gap = 7.8 mm; FOV = 230 × 230 mm; matrix = 256 × 256; and bandwidth = 1500 Hz/pixel.

Defining the lesions

All patients’ images were clinically interpreted by neuroradiologists with >10 years of experience. Three independent observers then analyzed the lesions: an expert neuro-radiologist with >10 years of experience (Observer 1); a radiology resident (Observer 2), and a medical physicist without any experience in brain imaging (Observer 3). All lesions were analyzed off-line (Matlab 2014a, The Mathworks, Natick, MA, USA). Three different methods were used to define the ROI. In method 1, a circle was drawn independently and with varying diameters to include the lowest pixel values on the ADC map and highest pixel values on the DSC time series at the time-point with the highest contrast enhancement (Fig. 1). These small ROIs had an average diameter of 9.1 mm (range = 3.7–17.8 mm). The lowest or highest region in the tumor was located in the tumor by visual evaluation and quantitative measurements of the pixel values. The diameter of the circle was adjusted according to the estimation of the observer. Additionally, freeform ROIs were drawn to encompass the whole tumor at the largest MRI cross-section only (method 2: 2D; Fig. 2; area = 184–5140 mm2) or summed MRI cross-sections (method 3: 3D; Fig. 3; area = 577–46,121 mm2).

Fig. 1.

Method 1, conventional small circle ROI. A 54-year-old man with a glioblastoma (high-grade). Circles of free diameter were drawn independently to include the lowest pixel values on the ADC map (left) and the highest pixel value on the PWI map (right).ROI: region of interest; ADC: apparent diffusion coefficient; PWI: perfusion-weighted imaging.

Fig. 2.

Method 2, 2D freeform ROI method. A 54-year-old man with a glioblastoma (high-grade). A freeform ROI encompassing the whole tumor was drawn at the largest tumor cross-section, independently on the ADC map (left) and on the PWI map (right).ROI: region of interest; ADC: apparent diffusion coefficient; PWI: perfusion-weighted imaging.

Fig. 3.

Method 3, 3D volume ROI method. A 54-year-old man with a glioblastoma (high-grade). ROIs were drawn independently on the ADC map (A) and on the PWI map (B); the tumor was visible on all slices, yielding summed voxels approximating the whole 3D volume of the lesion.ROI: region of interest; ADC: apparent diffusion coefficient; PWI: perfusion-weighted imaging.

DSC-MRI quantification

Voxelwise analysis of the DSC curve was performed to obtain the following parameters: maximum contrast enhancement (MCE); time-to-peak (TTP); wash-in rate (Win); wash-out rate (Wout); and the relative cerebral blood volume (rCBV).

Three time-points were defined: tonset = 10 s after start of the sequence; tmax = time at the peak of the signal; and tend = last time point of DSC curve. Win was calculated by the slope of a linear fit of the signal between tonset and tmax. Wout was calculated by the slope of a linear fit of the signal between tmax and tend. MCE was defined as (Smax–Sonset)/Sonset, where Smax is the signal intensity at maximum contrast enhancement and Sonset is the signal without contrast enhancement at tonset. TTP is the time between tonset and tmax. The rCBV was calculated by numerical integration of ΔR2* = -ln(S/Sonset)/TE (15), where S is the signal intensity. ΔR2* was not divided by the result for contralateral or centrum ROIs in order to obtain ICC values not complicated by additional ROI settings.

Comparisons and statistics

Statistical analyses were performed using SPSS (SPSS 23, Chicago, IL, USA). All data were tested for normality using Shapiro–Wilk tests. The ICCs between observers was calculated for all three methods using a two-way mixed model, single measures after applying a log transformation to normalize the PWI data. Low- and high-grade lesions were compared using independent t-tests for ADC data and Mann–Whitney tests for PWI data. The mean of all voxels in the circular ROI was taken to define the ADC, rCBV, MCE, TTP, Win, and Wout for each lesion for all methods. For the freehand 2D and 3D methods, next to the mean, four additional statistics were calculated by histogram analysis for ADC, rCBV, MCE, TTP, Win, and Wout: the 5% and 10% lowest values in the lesion (min5% and min10%) and the 5% and 10% highest values in the lesion (max5% and max10%) for each parameter. For both freehand methods, the best statistic was determined based on the highest ICC and best differentiation (highest significance) between low- and high-grade lesions. For the conventional small circle ROI, only the mean of the circle was used. A Ptwo-sided < 0.05 was considered a statistically significant difference for all tests.

Results

Histopathology of 30 lesions resulted in 13 low-grade and 17 high-grade gliomas: 12 astrocytoma (Grade II); one oligoastrocytoma (Grade II); 13 glioblastoma (Grade IV); three anaplastic astrocytoma (Grade III); and one anaplastic oligodendroglioma (Grade III).

Inter-observer reproducibility

The Shapiro–Wilk tests showed that within the chosen ROIs the pixel value distributions for ADC and five PWI parameters (rCBV, MCE, TTP, Win, Wout) were generally normal after the log transformation, so ICC were assessed for the mean values (Table 1).

Table 1.

Inter-observer reproducibility.

Method	Statistic	ADC	MCE	TTP	Win	Wout	rCBV
Circle	Mean	0.641	0.204	0.314	0.325	0.141	0.405
2D	Mean	0.854	0.890	0.974	0.876	0.892	0.923
	Min5%	0.532	0.839	0.923	0.552	0.620	0.861
	Min10%	0.554	0.844	0.926	0.677	0.860	0.705
	Max5%	0.639	0.905	0.860	0.888	0.922	0.814
	Max10%	0.749	0.903	0.874	0.890	0.926	0.842
3D	Mean	0.836	0.897	0.986	0.911	0.935	0.961
	Min5%	0.622	0.904	0.878	0.743	0.745	0.708
	Min10%	0.588	0.905	0.873	0.826	0.769	0.507
	Max5%	0.755	0.912	0.927	0.923	0.973	0.908
	Max10%	0.821	0.907	0.935	0.929	0.966	0.923

Three different methods were used to define the ROI (circle, 2D, and 3D). For each method, the six MRI parameters were calculated by using the mean or 5% or 10% lowest or respectively highest values in the ROI. The intraclass correlation coefficient (ICC) was determined for all combinations.

ROI: region of interest; ADC: apparent diffusion coefficient; MCE: maximum contrast enhancement; TTP: time-to-peak; Win: wash-in rate; Wout: wash-out rate; rCBV: relative cerebral blood volume.

Low-grade gliomas presented with sufficient signal change on both ADC and PWI maps to accurately draw the contours, as demonstrated in the example in Fig. 4. For all six parameters, inter-observer agreement of the mean values was better for 2D and 3D freeform ROIs (ICC = 0.836–0.986) than for circular ROIs (ICC = 0.141–0.641).

Fig. 4.

Example of low-grade tumor assessed using the 2D freeform ROI method. A 37-year-old woman with an astrocytoma. A freeform ROI encompassing the whole tumor was drawn at the largest tumor cross-section, independently on the ADC map (left) and on the PWI map (right).ROI: region of interest; ADC: apparent diffusion coefficient; PWI: perfusion-weighted imaging.

Differentiation of low- and high-grade gliomas

For PWI, both freeform 2D and 3D methods yielded significant differences between low- and high-grade gliomas when comparing the mean, max5%, or max10% of either TTP (P ≤ 0.015) or rCBV (P ≤ 0.001), as documented in Table 2. For Win (P ≤ 0.004) or Wout (P ≤ 0.007) the min5% and min10% resulted in significant differences for both methods. MCE as assessed by any ROI method yielded no significant differences between low and high grade (P ≥ 0.127).

Table 2.

Differentiation between low- and high-grade gliomas.

Method	Statistic	ADC	MCE	TTP	Win	Wout	rCBV
Circle	Mean	<0.001*	0.968	0.159	0.151	0.027*	0.002*
2D	Mean	0.189	0.987	0.004*	0.481	0.379	<0.001*
	Min5%	0.045*	0.127	0.022*	<0.001*	<0.001*	0.416
	Min10%	0.048*	0.169	0.156	<0.001*	<0.001*	0.772
	Max5%	0.595	0.842	<0.001*	0.715	0.927	<0.001*
	Max10%	0.726	0.855	<0.001*	0.753	0.927	<0.001*
3D	Mean	0.131	0.693	0.015*	0.497	0.085	<0.001*
	Min5%	0.005*	0.246	0.062	0.004*	0.005*	0.432
	Min10%	0.009*	0.381	0.364	0.004*	0.007*	0.599
	Max5%	0.265	0.723	<0.001*	0.855	0.413	<0.001*
	Max10%	0.238	0.766	<0.001*	0.874	0.373	<0.001*

Three different methods were used to define the ROI (circle, 2D and 3D). For each method the six MRI parameters were calculated by using the mean or 5% or 10% lowest or respectively highest values in the ROI. The significance of the differences between low- and high-grade was determined for all combinations.

*P < 0.05.

ROI: region of interest; ADC: apparent diffusion coefficient; MCE: maximum contrast enhancement; TTP: time-to-peak; Win: wash-in rate; Wout: wash-out rate; rCBV: relative cerebral blood volume.

For DWI, the ADC differentiated significantly (P ≤ 0.048) between low- and high-grade gliomas when choosing the minimum 5% or 10% of pixel values of the 2D and 3D freeform ROIs; however, reproducibility was low (ICC = 0.532–0.622). In addition, for the circular ROI method, the ADC was the only parameter yielding significant differences with reasonable inter-observer agreement (P < 0.001, ICC = 0.641).

Overall, the best ICC with significant grade differentiation could be obtained for the TTP parameter measured on a freeform 3D (ICC = 0.986, P = 0.015) and using the mean as measurement statistic (Table 3), closely followed by the result from a freeform 2D ROI (ICC = 0.974, P = 0.004).

Table 3.

Comparison of optimal ROI settings and statistic.

	Conventional circle			2D freeform			3D freeform
	Statistic	ICC	Low-/high-grade P	Statistic	ICC	Low-/high-grade P	Statistic	ICC	Low-/high-grade P
ADC (mm2/s)	Mean	0.641*	<0.001*	Min10%	0.554*	0.048*	Min5%	0.622*	0.005*
MCE	Mean	0.204*	0.968	Min5%	0.839*	0.127	Min5%	0.904*	0.246
TTP (s)	Mean	0.314*	0.159	Mean	0.974*	0.004*	Mean	0.986*	0.015*
Win (s–1)	Mean	0.325*	0.151	Min10%	0.677*	<0.001*	Min10%	0.826*	0.004*
Wout (s–1)	Mean	0.141	0.027*	Min10%	0.860*	<0.001*	Min10%	0.769*	0.007*
rCBV	Mean	0.405*	0.002*	Mean	0.923*	<0.001*	Mean	0.961*	<0.001*

For the 2D and 3D freeform method the best statistic is indicated based on the highest ICC and best differentiation between low- and high-grade lesions. For the conventional circle ROI only the mean of the circle was used.

*P < 0.05.

ROI: region of interest; ADC: apparent diffusion coefficient; MCE: maximum contrast enhancement; PWI: perfusion-weighted imaging; ROI: region of interest; TTP: time-to-peak; Win: wash-in rate; Wout: wash-out rate; rCBV: relative cerebral blood volume.

Discussion

This study showed that inter-observer agreement increased when gliomas were encompassed with a 2D freeform ROI compared to the conventional small circle ROI, while 3D freeform ROIs yielded no additional benefits.

Our finding of best reproducibility of whole tumor freeform ROI for the five DSC parameters, as well as ADC, compared with small circles of variable size positioned at highest PWI map and lowest ADC, is hard to compare with previous studies. Some only used small circular or elliptical ROIs of 20–40 mm2 (9–11,16) compared to others that only used freeform ROIs to delineate tumors (7,8,12–14), sometimes supplemented with “minimum ADC” (8) or lower and pixel value means for ADC (12). In just one study, both small presumably circular “hot spot” ROIs of 25–30 mm2 were set for each DSC parameter and compared with freeform ROI (7). Their conclusion that “hot spot ROI” analysis in DSC showed the best correlation with grading of gliomas appears to be at odds with our result. No assessment of reproducibility by ICC comparison was provided in that comparatively small study of nine low-grade gliomas and 17 high-grade ones (7).

In our case, reproducibility and differentiation was best for rCBV and TTP which apparently reflected the comparatively high ICC values obtained for these two DSC parameters in 2D and 3D freeform ROI. That the freehand mean and max5–10% offered the best differentiation for these parameters leads to the interpretation that the highest blood volume and enhancement delay encountered in glioma are most characteristic for tumor grade. In contrast, the rates of Win and Wout offered best differentiation at freeform ROI min5–10%. This fits with the knowledge that increased transfer rates are typical for malignancy (7–10,12).

Note that the most basic PWI parameter, MCE, though highly reproducible at especially 3D-freeform ROI max5% and max10% (ICC = 0.907–0.912), did not differentiate between low- and high-grade gliomas (P > 0.7). This is in line with a previous study showing that rCBV is a stronger predictor of glial tumor grade than the degree of enhancement (5).

The DWI parameter ADC was the only one providing significant grade differentiation along with reasonable reproducibility (ICC > 0.641) for the circular ROI, despite its lower ICC values compared with freeform 2D and 3D ROI. The worse result in terms of grade differentiation when just taking the mean of a freeform 2D or 3D ROI indicates that low ADC values are probably clustered in a limited part of the tumor and thus averaged out when taking the mean of the entire ROI. This is confirmed by the finding by Perez et al., that the ADC was worse in differentiating between glioma grades than a couple of DSC parameters when just taking the mean of the ROI (8). However, when taking the minimal ADC value of the ROI, they observed significant differentiation between glioma grades by the ADC, which is in accordance with our results. In addition, other compounds, such as fatty acids and calcifications, potentially present in oligodendroglioma, can affect the minimal ADC of a tumor which can be avoided by a manually selected area of low diffusion with a circle ROI (17,18).

Results obtained from multiple summed tumor slices (3D freeform ROI) were similar to those obtained with 2D freeform ROI, both in terms of reproducibility and in terms of tumor grade differentiation. It was expected that the circular ROIs would result in a lower reproducibility than the freeform methods because the circular diameters and locations were set differently by different observers. Another fair assumption is that the freeform 3D method, including the whole tumor, would result in better reproducibility than the freeform 2D method. Due to the heterogeneity of the tumor, especially for the high-grade glioma, one slice analysis, as applied in the freeform 2D method, might not reflect the features of the whole tumor. Our results of 3D analysis were, however, similar to the 2D analysis. The statistics such as min5% and max5% are already sensitive for the small number of pixels deviating from the mean in the 2D freeform slice. For tumor characterization, the 3D analysis therefore had no incremental value beyond the 2D freeform region. This might indicate that in the analysis of glioma, evaluation of a single 2D freeform region of the largest cross-section of the tumor might be sufficient, especially considering the time needed to draw the regions.

A limitation in our study was that only 30 patients were included. However, the results of this study documented that this was sufficient, considering that the reported observations generally meet high levels of significance (P < 0.001). Another issue was that ROIs were set independently for the DWI analysis and for the DSC analyses. This had the limitation that it made no sense to associate the ADC parameter values to those derived from DSC. Apart from that, it might even be best to set “hot spot” ROIs independently for each DSC parameter to account for tumor heterogeneity (7,9,10). Another limitation was that our analyses were performed off-line by Matlab. More consistent results could be obtained by more or less automated software packages such as Olea or NordicIce, although inter-software reproducibility has been found to be poor (16). The three observers in our study were of mixed levels of experience with neuroradiology, which turned out to be no real limitation as evidenced by the tabulated ICC values of up to 0.99. It has been shown that combining perfusion and DWI parameters can improve glioma grading accuracy (7,9,13,14), but exploration of that was beyond the scope of the present study aiming at the effects of ROI definition on the robustness of separate DSC parameters and ADC in relation to their potential to differentiate between low- and high-grade gliomas. Another topic, not explored in this study, with standard clinical DWI acquisitions of just 2 b values, is that with biexponential intravoxel incoherent motion (IVIM) analysis of DWI data differentiation between low- and high-grade glioma and other brain tissue types may improve (19).

In conclusion, DSC perfusion parameters rCBV and TTP yielded high inter-observer reproducibility and significant glioma grade differentiation for the means of 2D and 3D freeform ROIs. However, the more time-consuming alternative of 3D freeform ROI did not provide addition benefit. The small circular ROI yielded only significant differences with reasonable reproducibility for the DWI parameter ADC.