Automated versus manual post-processing of perfusion-CT data in patients with acute cerebral ischemia: influence on interobserver variability.

INTRODUCTION: The purpose of this study is to compare the variability of PCT results obtained by automatic selection of the arterial input function (AIF), venous output function (VOF) and symmetry axis versus manual selection.
METHODS: Imaging data from 30 PCT studies obtained as part of standard clinical stroke care at our institution in patients with suspected acute hemispheric ischemic stroke were retrospectively reviewed. Two observers performed the post-processing of 30 CTP datasets. Each observer processed the data twice, the first time employing manual selection of AIF, VOF and symmetry axis, and a second time using automated selection of these same parameters, with the user being allowed to adjust them whenever deemed appropriate. The volumes of infarct core and of total perfusion defect were recorded. The cerebral blood volume (CBV), cerebral blood flow (CBF), mean transit time (MTT) and blood-brain barrier permeability (BBBP) values in standardized regions of interest were recorded. Interobserver variability was quantified using the Bland and Altman's approach.
RESULTS: Automated post-processing yielded lower coefficients of variation for the volume of the infarct core and the volume of the total perfusion defect (15.7% and 5.8%, respectively) compared to manual post-processing (31.0% and 12.2%, respectively). Automated post-processing yielded lower coefficients of variation for PCT values (11.3% for CBV, 9.7% for CBF, and 9.5% for MTT) compared to manual post-processing (23.7% for CBV, 32.8% for CBF, and 16.7% for MTT).
CONCLUSION: Automated post-processing of PCT data improves interobserver agreement in measurements of CBV, CBF and MTT, as well as volume of infarct core and penumbra.

Introduction

In recent years, perfusion-CT (PCT) imaging has emerged as a useful tool to evaluate patients with suspected stroke [1, 2]. PCT has been advocated to detect and characterize the ischemic penumbra [3, 4], but also to assess blood–brain barrier permeability in an effort to predict the risk of hemorrhagic conversion after stroke [5-7]. As stated in the Acute Stroke Imaging Research Roadmap [8], one of the major obstacles to the widespread use of PCT technique to select acute stroke patients for reperfusion therapy is the lack of interobserver reliability, which led to a consensus recommendation for a more automated method of PCT data post-processing [8]

More specifically, the parameters to be selected during the post-processing of PCT data and, therefore, prone to influence the final PCT results are: the arterial input function (AIF), used to calculate the mean transit time (MTT) and the blood–brain barrier permeability (BBBP) maps; the venous outflow function (VOF), used to calculate the cerebral blood volume (CBV); the symmetry axis, used to calculate infarct and penumbra maps. These maps are generated by comparing MTT and CBV values in the ischemic and contralateral hemispheres and applying specific thresholds [9-11]. Several commercially available PCT software packages allow for automatic selection of the AIF, VOF and symmetry axis, and for automatic post-processing of PCT maps, with the possibility for the user to correct or adjust the aforementioned parameters in case the automatic selection is not adequate and/or not completely satisfactory.

The purpose of this study was to evaluate whether the reproducibility of quantitative PCT results can be improved by automated post-processing with automatic selection of the AIF, VOF and symmetry axis, corrected by the user if deemed necessary. In order to do so, we measured and compared the degree of interobserver reproducibility of PCT-derived quantitative measurements obtained using manual and automated post-processing methods in patients with suspected acute ischemic stroke. We selected one specific software implementation for our study, based on its ability to display maps of infarct core and penumbra based on CBV and MTT thresholds.

Materials and methods

Patients

Imaging data from 30 PCT studies obtained as part of standard clinical stroke care at our institution in patients with suspected acute hemispheric ischemic stroke were retrospectively reviewed with the approval of our institutional review board. At our institution, patients suspected of acute stroke and with no history of renal insufficiency or iodinated contrast allergy routinely undergo a stroke CT survey including noncontrast CT (NCT) of the brain, PCT at two cross-sectional positions, CT-angiogram (CTA) of the cervical and intracranial vessels, and post-contrast cerebral CT.

Perfusion-CT protocol

PCT studies were obtained on 16-slice (15 patients) and 64-slice (15 patients) CT scanners. Each PCT study involved successive gantry rotations performed in cine mode during intravenous administration of iodinated contrast material. Images were acquired and reconstructed at a temporal sampling rate of 1 image/s for the first 45 s. Additional images were acquired at 60, 90, 120, 150, and 180 s. Acquisition parameters were 80 kVp and 100 mAs. Two successive PCT series at two different levels were performed following the noncontrast CT and before the CTA. At each PCT level, two 10-mm-thick sections (16-section CT scanners) or eight 5-mm-thick sections (64-section CT scanners) were assessed. The first PCT series was performed at the level of the third ventricle and the basal ganglia, and the second PCT series was performed above the lateral ventricles. For each PCT series, a 40-mL bolus of iohexol (300 mg/mL of iodine, Omnipaque; Amersham Health, Princeton, NJ) was administered into an antecubital vein by using a power injector at an injection rate of 5 mL/s for all patients. CT scanning was initiated 7 s after the start of the injection of the contrast bolus.

PCT raw data post-processing

PCT data were analyzed using a research version of a PCT software package (Philips Medical Systems, Cleveland, OH) to calculate MTT, CBV, CBF, and BBBP maps. This software relies on the central volume principle, which is the most accurate for low injection rates of iodinated contrast material [12]. After motion correction and noise reduction by an anisotropic, edge-preserving spatial filter, the software applies curve fitting by least mean squares to obtain mathematical descriptions of the time-enhancement curves for each pixel, describing the wash-in and the wash-out of contrast in these pixels. A closed-form (noniterative) deconvolution is then applied to calculate the MTT map. The deconvolution operation requires a reference arterial input function and venous outflow function. For this purpose, the user draws regions of interest, typically within the anterior cerebral artery (ACA) and a vertical segment of the superior sagittal sinus (SSS). The software then identifies the appropriate reference voxels within these regions of interest. Alternatively, in the research version used in this study, the user may let the software automatically select the reference voxels for AIF and VOF, and adjust them if needed. Similarly, the user can manually define a symmetry axis, or let the software define one and adjust it if needed.

The CBV map is calculated from the area under the time-enhancement curves for each pixel. The central volume principle equation allows the calculation of cerebral blood flow (CBF = CBV/MTT). From the parametric maps the software generates summary maps of penumbra and infarct core. In our study, absolute CBV and relative MTT thresholds (comparing the voxels on the stroke side to their contralateral counterparts around the symmetry axis) were used as reported in literature (total perfusion defect: MTT > 145% of the contralateral side values; infarct core, CBV < 2.0 ml × 100 g−1) [9]. BBBP maps were calculated by applying the Patlak model to the delayed PCT acquisition [13]

Post-processing of the 30 CTP datasets was performed by two observers with more than 6 months of experience with PCT post-processing each. Both individuals were trained by a neuroradiologist who has 10 years of experience in PCT post-processing software application. Each observer processed the data twice, a first time employing manual selection of AIF, VOF, and symmetry axis, and a second time using automated selection of these same parameters, with the user being allowed to adjust them whenever deemed appropriate. The two processing sessions for each reviewer were performed more than 2 months apart, in order to reduce recall bias.

Data analysis

The volume of perfusion abnormality corresponding to infarct core and tissue at risk were recorded. The sum of the volumes of infarct core and tissue at risk was calculated and recorded under the label of “total perfusion defect”.

To obtain mean values of CBV, CBF, MTT, and BBBP, regions of interest (ROIs) were drawn in a single representative slice in the MCA territory of the ischemic and non-ischemic hemispheres at the level of the lateral ventricles in the slice just above the choroid plexus. This region of interest included white matter of the corona radiata and the superficial cortical gray matter (Fig. 1). These ROIs were systematically drawn in the exact same location for all post-processing rounds.

Fig. 1

Axial PCT map depicting the area of infarct core (red) and tissue at risk (green) in the right MCA territory. Also demonstrated are the regions of interest (ROIs) that were systematically drawn for all post-processing rounds in the MCA territory of the ischemic (yellow) and contralateral hemisphere (blue). Mean values of CBV, CBF and MTT measured in these ROIs were recorded

Statistical analysis

Interobserver agreement was quantified for each of the recorded PCT parameters using the approach proposed by Bland and Altman [14]. Two descriptive statistics of the interobserver agreement were calculated: the repeatability coefficient and the coefficient of variation. The repeatability coefficient is related to the absolute difference between the measurements performed by the two reviewers for the different patients and the different variables: in 95% of the cases, the absolute difference between the measurements performed by the two reviewers will be less than the repeatability coefficient. Lower repeatability values mean better agreement between two measurements. The coefficient of variation is calculated as a ratio of the repeatability coefficient over the mean value of the considered variables, and expressed in percentages. A lower coefficient of variation means better agreement between two different measurements is achieved.

Also, the measurements performed by the two reviewers for the different patients and the different variables were evaluated for linear regression. The slopes, intercepts and coefficients of correlation of the linear regressions were recorded.

Results

Our study population included 30 consecutive patients (14 males and 16 females; median age, 61; age range, 47–89) presenting between January and March 2008 with symptoms suspicious for acute hemispheric ischemic stroke. The average National Institute of Health Stroke Scale (NIHSS) score on admission was 13 (range, 3–21). In 26 patients, a diagnosis of ischemic stroke involving the MCA territory was made, while two patients had an ACA stroke and two were considered to have normal perfusion-CT studies.

Manual versus automated post-processing

The coefficients of variation for the volume of the infarct core and the volume of the total perfusion defect (Table 1) ranged between 5.8% and 31.0%. They were higher for the volume of the infarct core (15.7–31.0%) compared to the volume of the total perfusion defect (5.8–12.2%). Automated post-processing yielded improved interobserver agreement (15.7% for volume of the infarct core and 5.8% for the volume of the total perfusion defect) compared to manual post-processing (31.0% for volume of the infarct core and 12.2% for the volume of the total perfusion defect). Linear regression showed slopes and coefficients of correlation close to 1, both for the automated and the manual post-processing.

Table 1

Interobserver variability for the manual and automated post-processing methods for the volumes of the infarct core and of the total perfusion defect.

	Interobserver variability
	Manual post-processing	Automated post-processing	p value
Volume of infarct core (mL)
Mean ± SD	16.89 ± 21.01	17.42 ± 20.84	0.739
Repeatability coefficient	5.23	2.73
Coefficient of variation	31.0%	15.7%
Slope	1.069	0.987
Intercept	−1.133	0.631
Correlation	0.992	0.997
Volume of total perfusion defect (mL)
Mean ± SD	68.11 ± 49.60	68.36 ± 49.87	0.998
Repeatability coefficient	8.33	3.99
Coefficient of variation	12.2%	5.8%
Slope	1.015	0.995
Intercept	−2.239	0.698
Correlation	0.995	0.999

The mean volume of the regions of interest drawn in the MCA territory was 24.3 cc2 (range: 21.9–28.1 cc2). The coefficients of variation for the ROI-generated values of CBV, CBF and MTT in the ischemic hemisphere (Table 2) ranged between 9.5% and 32.8%. They were higher for CBV (11.3%–23.7%) and CBF (9.7%–32.8%) compared to MTT (9.5%–16.7%). Automated post-processing yielded improved interobserver agreement (11.3% for CBV, 9.7% for CBF and 9.5% for MTT) compared to manual post-processing (23.7% for CBV, 32.8% for CBF and 16.7% for MTT).

Table 2

Interobserver variability for the manual and automated post-processing methods for PCT values of cerebral blood volume (CBV), cerebral blood flow (CBF), mean transit time (MTT) and blood–brain barrier permeability (BBBP) in the ischemic brain parenchyma.

ROI on ischemic hemisphere	Interobserver variability
	Manual post-processing	Automated post-processing	p value
CBV (mL/100 g)
Mean ± SD	3.00 ± 0.95	2.89 ± 0.93	0.650
Repeatability coefficient	0.71	0.32
Coefficient of variation	23.7%	11.3%
Slope	0.724	0.849
Intercept	0.766	0.376
Correlation	0.921	0.930
CBF (mL/100 g/s)
Mean ± SD	18.94 ± 11.45	17.67 ± 9.84	0.808
Repeatability coefficient	6.21	1.71
Coefficient of variation	32.8%	9.7%
Slope	0.960	1.004
Intercept	1.030	−0.070
Correlation	0.948	0.994
MTT (s)
Mean ± SD	15.23 ± 21.02	15.67 ± 21.93	0.952
Repeatability coefficient	2.55	1.49
Coefficient of variation	16.7%	9.5%
Slope	0.966	1.035
Intercept	−0.300	0.006
Correlation	0.997	0.995
BBBP (mL/100 g/s)
Mean ± SD	1.10 ± 0.61	0.98 ± 0.61	0.514
Repeatability coefficient	0.40	0.33
Coefficient of variation	35.9%	34.2%
Slope	1.180	0.812
Intercept	−0.184	0.166
Correlation	0.945	0.953

The coefficients of variation for the ROI-generated values of CBV, CBF, and MTT were similar between the ischemic and non-ischemic hemispheres (Table 3).

Table 3

Interobserver variability for the manual and automated post-processing methods for PCT values of cerebral blood volume (CBV), cerebral blood flow (CBF), mean transit time (MTT) and blood–brain barrier permeability (BBBP) in the non-ischemic brain parenchyma.

ROI on contralateral, non-ischemic hemisphere	Interobserver variability
	Manual post-processing	Automated post-processing	p value
CBV (mL/100 g)
Mean ± SD	3.04 ± 0.67	2.94 ± 0.50	0.117
Repeatability coefficient	0.74	0.29
Coefficient of variation	24.3%	9.7%
Slope	0.734	0.921
Intercept	0.696	0.238
Correlation	0.805	0.942
CBF (mL/100 g/s)
Mean ± SD	34.40 ± 10.86	32.17 ± 8.31	0.082
Repeatability coefficient	12.19	4.69
Coefficient of variation	35.4%	14.6%
Slope	0.811	0.951
Intercept	6.863	1.632
Correlation	0.779	0.942
MTT (s)
Mean ± SD	5.59 ± 1.23	5.70 ± 1.16	0.307
Repeatability coefficient	1.67	0.45
Coefficient of variation	29.8%	8.0%
Slope	0.790	0.877
Intercept	0.985	0.680
Correlation	0.696	0.979
BBBP (mL/100 g/s)
Mean ± SD	1.10 ± 0.38	0.96 ± 0.38	0.424
Repeatability coefficient	0.40	0.25
Coefficient of variation	36.7%	25.6%
Slope	0.923	0.927
Intercept	0.092	0.083
Correlation	0.811	0.929

Interobserver agreement for BBBP measurements

The coefficients of variation for BBBP measurements ranged from 25.6% to 36.7%. Automated post-processing yielded improved interobserver agreement (34.2% in ischemic hemisphere and 25.6% in non-ischemic hemisphere) compared to manual post-processing (35.9% in ischemic hemisphere and 36.7% in non-ischemic hemisphere).

Corrections by the observers during automated post-processing of PCT raw data

During automated post-processing, both observers corrected the software selection of AIF in 3 cases each (10%). In these cases, which were the same for both observers, the automatically selected AIF voxels were on the anterior portion of the SSS, being manually repositioned to the ACA. Automatically selected VOF voxels were considered inadequate by both users in five cases (16.7%). These studies had head motion that was not completely corrected by the registration algorithm, with VOF voxels originally placed on the skull by the automatic algorithm. These were manually repositioned to the superior sagittal sinus. The symmetry axis was manually adjusted by each observer in one patient, whose head was severely tilted at the time of the scan.

Discussion

This study demonstrated that the reproducibility of PCT measurements, including CBV, CBF, MTT, and BBBP values, as well as of PCT abnormality volumes based on CBV and MTT thresholds, was improved when an automated post-processing algorithm minimizing user interaction was used. The average interobserver variability for CBV, CBF and MTT measurements using an automated post-processing approach was 10.4%, almost three times less than the interobserver variability of 27.1% observed for a manual post-processing approach. This improved interobserver reproducibility when using an automated approach was observed both in the ischemic parenchyma (10.1% with automated post-processing, 24.4% with manual post-processing) and in the non-ischemic parenchyma (10.8% with automated post-processing, 29.8% with manual post-processing). In agreement with previous studies [15, 16], our results showed that MTT was the most reproducible PCT parameter.

In this study, we reported not just the variability of measurements of CBV, CBF, MTT, and BBBP, but also the variability of measurements of infarct core and total perfusion defect volumes based on CBV and MTT thresholds. To our knowledge, the interobserver variability for measurements of infarct core and total perfusion defect volumes has not been reported before. The interobserver variability we obtained for the measurements of CBV, CBF and MTT using a manual processing approach is similar to what was reported in a prior published study comparing manual post-processing of PCT datasets by different CT technologists [16]. In another study, lower interobserver variability in case of manual PCT post-processing was achieved by the use of a uniform and standardized approach [17]. In the same study, paradoxically, greater interobserver variability was introduced with the use of an automated approach [17]. This reflects that not all automated PCT post-processing algorithms, as implemented in commercially available packages, are equivalent, and that each of them needs to be carefully evaluated, and possibly improved.

Since a completely automated approach to post-processing PCT data would yield a perfect degree of agreement (0% of interobserver variability), it is clear that the interobserver variability around 10.4% observed when using the automated approach was introduced by the user-defined manual adjustments. The manual adjustments performed by the observers in our study were mainly justified by an incorrect selection of the AIF and VOF by the automatic algorithm in studies with severe head motion artifact. Although these manual corrections were necessary to obtain accurate PCT results, they introduced interobserver variability for the “automated” PCT results. This again emphasizes the need for further improvement of automated PCT post-processing algorithms, with the ultimate goal being to completely eliminate user interaction.

Two main limitations should be noted in our study. The first one is that the results we obtained apply only to a specific software implementation. Therefore, the variability between automated and manual post-processing observed in our study should not be assumed to be similar to other automated PCT post-processing algorithms. The second limitation is the absence of a well-defined criterion of acceptability in the variability of PCT-derived measurements. However, prior studies with similar results [16, 17] have shown that an overall variability around 10% does not change the qualitative visual assessment of PCT maps, thus being very unlikely to alter clinical management decisions.

In conclusion, automated post-processing of PCT data improves interobserver variability in measurements of CBV, CBF, MTT and BBBP, as well as volume of infarct core and penumbra based on CBV and MTT thresholds. PCT post-processing automation is a way to ensure reliability of PCT technique when widely used in the community, which is one of the requirements for it to be used in the selection of stroke patients for acute reperfusion therapies.