Intra-observer and inter-observer agreements for the measurement of dual-input whole tumor computed tomography perfusion in patients with lung cancer: Influences of the size and inner-air density of tumors.

BACKGROUND: This study was conducted to assess intra-observer and inter-observer agreements for the measurement of dual-input whole tumor computed tomography perfusion (DCTP) in patients with lung cancer.
METHODS: A total of 88 patients who had undergone DCTP, which had proved a diagnosis of primary lung cancer, were divided into two groups: (i) nodules (diameter ≤3 cm) and masses (diameter >3 cm) by size, and (ii) tumors with and without air density. Pulmonary flow, bronchial flow, and pulmonary index were measured in each group. Intra-observer and inter-observer agreements for measurement were assessed using intraclass correlation coefficient, within-subject coefficient of variation, and Bland-Altman analysis.
RESULTS: In all lung cancers, the reproducibility coefficient for intra-observer agreement (range 26.1-38.3%) was superior to inter-observer agreement (range 38.1-81.2%). Further analysis revealed lower agreements for nodules compared to masses. Additionally, inner-air density reduced both agreements for lung cancer.
CONCLUSION: The intra-observer agreement for measuring lung cancer DCTP was satisfied, while the inter-observer agreement was limited. The effects of tumoral size and inner-air density to agreements, especially between two observers, should be emphasized. In future, an automatic computer-aided segment of perfusion value of the tumor should be developed.

Introduction

Computed tomography perfusion (CTP) is an important functional imaging technique of pulmonary lesions for the quantitative evaluation of blood perfusion and the associated metabolic activity of tumors.1, 2, 3 CTP can be used in single or double input mode for the diagnosis of lung nodules or masses.4, 5, 6, 7, 8 Recently, with the advances in CT equipment and mathematical methods for analysis, lung cancer CTP with a first‐pass maximum slope dual‐input artery mode was introduced to routine clinical practice.9 Compared to the single input artery mode, the dual‐input artery mode applied in lung CTP is based on the theory that lung cancers are supplied by both bronchial and pulmonary circulations.10, 11 Using the dual‐input artery mode, Yuan et al. first reported that the pulmonary index (PI) derived from dual‐input whole tumor computed tomography perfusion (DCTP) analysis is a valuable biomarker for identifying malignancy in solitary pulmonary nodules (SPNs).12 Compared to magnetic resonance imaging (MRI) and fluorine 18 fluorodeoxyglucose (FDG) positron emission tomography (PET)/CT, CTP is more specific and accurate in the diagnosis of SPNs.13, 14 Li et al. and Ohno et al. also demonstrated that DCTP analysis is useful in the prediction or early assessment of the treatment effect of chemoradiotherapy for non‐small cell lung cancer (NSCLC).15, 16

Previous studies have proposed that DCTP analysis is valuable in clinical use to estimate the hemodynamics of lung cancer.9, 12, 13, 14, 15, 16, 17 Some of these studies involved inter‐observer agreement analysis to measure advanced NSCLC or intra‐observer and inter‐observer agreement analysis to measure lung nodules.13, 14, 15 However, to our knowledge, tumor size‐based observer agreement for measuring DCTP in lung cancer has not been reported in a major academic journal. Furthermore, the inner‐air density of lung cancer (i.e. the bronchus sign and cavity), leading to an obvious difference of density in the tumoral area, might have an effect on observer agreement for the measurement of CTP and has also not been investigated in previous studies.18, 19, 20, 21, 22, 23

Therefore, the purpose of our study was to investigate the influence of tumor size and inner‐air density on the agreement of measurements of the first‐pass maximum slope DCTP analysis in lung cancer.

Methods

Patients and clinical manifestations

This prospective study was conducted as a part of an institutional review board‐approved clinical study of DCTP in patients with lung cancer. All patients were informed of the radiation exposure and potential harm and provided written informed consent before participating. From July 2012 to July 2014, 126 consecutive patients with suspected lung cancer were prospectively enrolled. The inclusion criteria were: (i) patients with suspected lung cancer; (ii) lesions sized ≥1.0 cm in diameter; (iii) solid or part‐solid nodules or mass; (iv) no history of any treatment, including chemotherapy, radiation therapy, or surgery for lung cancer; and (v) no history of hypersensitivity to iodinated contrast media. The exclusion criteria were: (i) contrast media injection failure, (ii) unfit Z axial coverage for dynamic CT scan, (iii) obvious image artifacts, (iv) follow up with proven metastatic tumors or benign lesions and (v) patients lost to follow up.

Thirty‐eight patients were excluded, leaving a total of 88 patients with proven bronchogenic lung cancer. The mean age was 63.4 ± 8.5 years (standard deviation [SD]; range 29–80; median 63 years). There were 55 men and 33 women. A definite histopathological diagnosis was obtained in 83 subjects by tumor resection (51 patients), CT‐guided biopsy (14 patients), and bronchofibroscopic biopsy (18 patients). The remaining patients were diagnosed by sputum examination (1 patient) or clinical treatment and follow‐up (4 patients). Forty‐eight patients had adenocarcinomas, 20 squamous cell carcinomas, seven small cell lung carcinomas, two sarcomatoid carcinomas, six NSCLC without definite pathological types, one poorly differentiated carcinoma, and four patients had malignant lung cancers clinically detected during follow‐up.

Dual‐input whole tumor computed tomography perfusion (DCTP)

Dual‐input whole tumor computed tomography perfusion was performed using a 320‐MDCT scanner (Aquilion ONE, Toshiba Medical Systems, Otawara, Japan). All patients were trained to hold their breath before examination. An 18‐gauge cannula was placed in the antecubital vein. A plain CT scan of the entire thorax (120 kV, detector configuration 160 × 0.5 mm, rotation time 0.75 seconds, pitch 0.869, slice thickness 5.0 mm, reconstruction interval 5.0 mm) was taken to localize the tumor. A dynamic volume scan was then taken using a flexible Z‐axial range of 6–16 cm (median 10 cm) to cover the entire tumor and the trunk of the pulmonary artery and left atrium. The DCTP dynamic volume scan mode has 18 intermittent volumes comprising three parts with different temporal intervals (TI = 1.5 or 2 seconds). The total scan time was 30.5 seconds (the time sequence display is shown in Fig 1; 1 mm thickness and 1 mm interval, 512 × 512 matrix, 320 mm field of view [FOV], reconstruction kernel FC 02, 100 mA, 0.5 seconds gantry rotation time). The adjusted parameters of the dynamic scan included the tuber voltage according to the patient's body mass index (BMI).2 The standard for chest CTP was used: BMI ≤ 20, 80 kV in 14 patients; 20 < BMI < 30, 100 kV for 74 patients; and BMI ≥ 30, 120 kV did not apply to any of the patients.2

Figure 1

The time sequence display of the dynamic volume scan using the Aquilion ONE system. After bolus injection, 18 intermittent volumes comprising three parts with different temporal intervals (TI) were acquired using breath‐hold behavior for a duration of approximately 31 seconds: a TI of two seconds for the first three scans from one to five seconds, a TI of 1.5 seconds for nine subsequent scans from 6.5 to 19 seconds, and a final TI of two seconds for the last six scans from 21 to 31 seconds.

The contrast medium (CM) dose (350 mg I/mL) was also adjusted according to patient weight: at least 35 mL for patients ≤70 kg; if the patient weighed >70 kg, then the dose would be equal to the weight multiplied by 0.5 mL/kg.2 The CM was injected using a dual‐head power injector (REF XD 2060 Touch, Ulrich Medical, Ulm, Germany) at an injection rate of 8 mL/seconds, followed by a 30 mL saline flush at the same injection rate.

The recorded dose‐length product and the estimated effective dose (measured with a conversion factor of 0.014 mSv/mGy × cm)18 for the dynamic DCTP scan were 395.49 ± 157.20 (SD) mGy × cm (median 376.7 mGy × cm) and 5.54 ± 2.20 (SD) mSv (median 5.27 mSv).

Quantitative DCTP analysis

Quantitative DCTP analysis was performed using perfusion software for the CT equipment (Body Perfusion, Toshiba Medical Systems, Otawara, Japan). Two radiologists with eight and three years experience in chest CTP imaging measured the perfusion parameters independently to evaluate inter‐observer agreement. The second radiologist calculated the measurement twice a month to evaluate intra‐observer agreement. Both radiologists were blinded to all clinical data and results.

The DCTP analysis was calculated using the maximum slope mode. The perfusion parameters contain the bronchial flow (BF), pulmonary flow (PF), and the perfusion index (PI, PI = PF/[PF + BF]). A global value representing the perfusion parameter for the entire tumor was calculated as the median of the perfusion value of all measured slices.19 The regions of interest (ROIs) were drawn freehand around the solid part of the tumor, excluding the surrounding air, calcification, and nearby pulmonary vessels. The radiologist also placed circular 10 mm diameter ROIs in the bilateral lung parenchyma, excluding the large vessels, to calculate the signal‐to‐noise ratios (SNRs) of the first dynamic CT image. Subsequently, the SNRs in both the tumor and lung parenchyma were calculated.20

Statistical analysis

The patient sample was divided into two groups depending on tumor size (nodules ≤3 cm and mass >3 cm) or inner density (with or without bronchus sign or cavity).21 Bland–Altman statistics were then used to assess the intra‐observer and inter‐observer agreements of DCTP measurements. The target percent of variation for reproducibility is <30%. The within‐subject coefficient of variation (WCV) and single‐measure intraclass correlation coefficient (ICC) were also calculated. All statistical analyses were performed using SPSS version 18.0 (SPSS Inc., Chicago, IL, USA) and MedCalc software version 12.2.1 (MedCalc Software, Ostend, Belgium). A two‐sided P value of <0.05 was considered statistically significant.

Results

Dynamic image quality

The SNR for lung parenchyma of all patients was 30.29 ± 9.07 (SD, range 11.94–50.92, median 28.26), and the SNR for the tumors was 1.82 ± 0.85 (SD, range 0.57–4.34, median 1.60).

Intra‐observer and inter‐observer agreements for lung nodules and masses

There were 37 lung masses and 51 nodules. The mean mass diameter was 4.6 ± 1.1 cm (SD, median 4.5 cm), while that of the nodules was 2.3 ± 0.5 cm (SD, median 2.4 cm). The mean values of DCTP parameters measured in patients with lung nodules and masses are listed in Table 1. Observer agreements for measuring DCTP parameters in all patients are listed in Table 2. Bland–Altman plots for inter‐observer agreements are shown in Figure 2. Observer agreements with nodules and masses, respectively, are listed in Table 3 and Bland–Altman plots are shown in Figure 3. The mean differences in intra‐observer agreement for measuring DCTP parameters of lung masses and nodules ranged from −1.30 to 2.5% and from −2.14 to 4.90%, and the WCVs ranged from 9.4 to 11.5% and 10.4 to 16.8%, respectively. Correspondingly, the mean difference in inter‐observer agreement (except for the measurement of BF value of lung nodules), ranged from −0.60 to −4.8% and −5.55 to −6.91%, and the WCVs from 12.1 to 14.4% and 14.1 to 19.8%, respectively.

Table 1

Dual‐input computed tomography perfusion parameters measured in patients with lung cancer

	Perfusion parameters
	Pulmonary flow (mL/min/100 mL)	Bronchial flow (mL/min/100 mL)	Pulmonary index (%)
Masses (n = 37)
Observer 1, first	45.07 ± 27.39	83.73 ± 31.69	35.34 ± 12.20
Observer 2	49.34 ± 35.29	84.69 ± 33.78	35.58 ± 12.00
Observer 1, second	46.65 ± 30.49	82.89 ± 33.77	36.11 ± 12.79
Nodules (n = 51)
Observer 1, first	60.80 ± 37.27	68.98 ± 38.36	48.58 ± 21.31
Observer 2	63.00 ± 33.96	67.22 ± 41.85	50.91 ± 21.58
Observer 1, second	61.33 ± 37.69	67.88 ± 39.95	49.39 ± 21.36
Tumors without inner‐air density (n = 68)
Observer 1, first	51.10 ± 32.55	78.80 ± 36.26	40.47 ± 17.50
Observer 2	54.83 ± 35.07	79.61 ± 38.46	41.55 ± 17.92
Observer 1, second	52.11 ± 34.11	78.56 ± 38.03	41.17 ± 17.87
Tumors with inner‐air density (n = 20)
Observer 1, first	61.70 ± 35.00	60.61 ± 34.94	52.16 ± 22.51
Observer 2	63.07 ± 33.60	56.81 ± 38.83	54.21 ± 22.30
Observer 1, second	66.90 ± 38.40	57.45 ± 35.65	54.53 ± 22.08

Table 2

Observer agreement for measuring all lung cancers (n = 88)

Perfusion parameters	Intra‐observer agreement					Inter‐observer agreement
	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)
Pulmonary flow	0.964	−2.90 ± 18.0	−38.2 to 32.4	35.3	12.8	0.942	−5.91 ± 24.6	−54.2 to 42.4	48.3	17.8
Bronchial flow	0.971	3.84 ± 19.5	−34.5 to 42.1	38.3	14.0	0.930	8.50 ± 41.4	−72.7 to 89.7	81.2	29.7
Pulmonary index	0.970	−2.78 ± 13.3	−28.9 to 23.4	26.1	9.6	0.945	−3.22 ± 19.4	−41.3 to 34.8	38.1	13.8

95% limits, 95% limits for mean difference; ICC, intraclass correlation coefficient for a single measure; MD, mean difference; RC, reproducibility coefficient; SD, standard deviation; WCV, within‐subject coefficient of variation.

Figure 2

Intra‐observer and inter‐observer agreements using Bland–Altman plots for all lung cancers (n = 88) measured by two different observers. The solid line shows the mean value, where the mean = (value 1 + value 2)/2. The dashed lines denote the means ± (1.96 × standard deviation). Ordinate value = ([value 1 − value 2]/mean) × 100. Pulmonary flow (PF) 1, 2, 3 = PF measured by observer 1 the first time, observer 1 the second time, and observer 2, respectively. The same process applies to bronchial flow (BF) 1, 2, 3 and perfusion index (PI) 1, 2, 3. Graphs show Bland–Altman plots for (a) PF, (b) BF, and (c) PI values measured by observer 1 and (d) PF, (e) BF, and (f) PI values measured between observers 1 and 2.

Table 3

Observer agreement for measuring lung cancer nodules (n = 51) and masses (n = 37)

Perfusion parameters	Intra‐observer agreement					Inter‐observer agreement
	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)
Pulmonary flow
Nodules	0.907	−2.14 ± 23.8	−48.9 to 44.6	46.7	16.8	0.937	−6.91 ± 27.3	−60.5 to 46.7	53.6	19.8
Masses	0.966	−1.3 ± 16.4	−33.6 to 30.9	32.2	11.5	0.939	−4.8 ± 20.1	−44.2 to 34.7	39.5	14.4
Bronchial flow
Nodules	0.976	4.90 ± 22.47	−39.1 to 48.9	44.0	16.1	0.942	16.5 ± 51.0	−83.6 to 116.6	100.1	37.6
Masses	0.952	2.5 ± 14.9	−26.8 to 31.7	29.3	10.6	0.891	−0.5 ± 17.3	−34.5 to 33.4	33.9	12.1
Pulmonary index
Nodules	0.967	−2.31 ± 14.7	−31.2 to 26.5	28.8	10.4	0.952	−5.55 ± 19.4	−43.6 to 32.5	38.0	14.1
Masses	0.910	−2.2 ± 13.2	−28.1 to 23.8	26.0	9.4	0.902	−0.6 ± 18.8	−37.4 to 36.2	36.8	13.1

95% limits, 95% limits for mean difference; ICC, intraclass correlation coefficient for a single measure; MD, mean difference; RC, reproducibility coefficient; WCV, within‐subject coefficient of variation.

Figure 3

Interobserver agreement using Bland–Altman plots for masses (n = 37) and nodules (n = 53) measured by two different observers. The solid line shows the mean value, where the mean = (value 1 + value 2)/2. The dashed lines denote the means ± (1.96 × standard deviation). Ordinate value = ([value 1 − value 2]/mean) × 100. Pulmonary flow (PF) 1 = PF measured by observer 1, PF 3 = PF measured by observer 2. The same process applies to bronchial flow (BF) 1, BF 3, and perfusion index (PI) 1, PI 3. Graphs show Bland–Altman plots for (a) PF, (b) BF, and (c) PI values for lung masses and (d) PF, (e) BF and (f) PI values for lung nodules.

Intra‐observer and inter‐observer agreements for lung cancers with or without inner‐air density

There were 20 lung cancers with inner‐air density, including cavities (7 tumors) and CT bronchus sign (13 tumors), and 68 lung tumors without inner‐air density. Figure 4 shows a colored map of the ROIs of the whole tumor, excluding the small cavity. The mean DCTP parameter values measured in patients with or without inner‐air density are listed in Table 1. The observer agreements for measuring DCTP parameters of both groups are listed in Table 4. Bland–Altman plots for agreements between the observers are shown in Figure 5. A higher intra‐observer agreement in lung cancers without inner‐air density was represented as low reproducibility coefficients (RCs) of approximately 25%, narrow 95% limits, and low WCVs of approximately 10%. A lower intra‐observer agreement in lung cancers with inner‐air density was nearly 1.5–2 times that (PI = 35.4/22.7), compared to the values of lesions without bronchus sign or cavities. Except for the PI RC and WCV values, inter‐observer agreement in the two groups was almost the same. However, the intra‐observer agreement for both groups was decreased compared to the inter‐observer agreement, with the exception of both agreements of PF and PI values approximated in lung cancers with air‐density.

Figure 4

Colored maps of dual‐input computed tomography perfusion in a 52‐year‐old man with peripheral adenocarcinoma in the right inferior lung. This nodule has a small cavity in the central portion. Air in the cavity may have resulted in miscalculation of the perfusion values. (a–c) show the maps color display of blood supply of the tumor for pulmonary flow (a), bronchial flow (b), and pulmonary flow (c). (d) shows computed tomography image without pseudo color processing.

Table 4

Observer agreement for measuring tumors with (n = 20) or without inner‐air density (n = 68)

Perfusion parameters	Intra‐observer agreement					Inter‐observer agreement
	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)	ICC	MD ± SD (%)	95% limits (%)	RC (%)	WCV (%)
Pulmonary flow
Tumors with air density	0.930	−7.91 ± 26.9	−60.6 to 44.8	52.7	19.4	0.952	−4.01 ± 24.6	−52.1 to 44.1	48.1	17.2
Tumors without air density	0.976	−1.43 ± 14.4	−29.7 to 26.8	28.2	10.2	0.938	−6.47 ± 24.8	−55.1 to 42.2	48.6	18.0
Bronchial flow
Tumors with air density	0.960	13.7 ± 31.0	−47.1 to 74.5	60.8	23.5	0.916	20.4 ± 49.9	−77.4 to 118.3	97.9	37.3
Tumors without air density	0.972	0.89 ± 13.5	−25.6 to 27.4	24.5	9.50	0.931	4.94 ± 38.2	−70.0 to 79.9	74.9	27.6
Pulmonary index
Tumors with air density	0.967	−6.35 ± 18.0	−41.7 to 29.0	35.4	13.2	0.961	−5.18 ± 19.2	−42.8 to 32.4	37.6	13.7
Tumors without air density	0.968	−1.73 ± 11.6	−24.4 to 20.9	22.7	8.20	0.933	−2.64 ± 19.6	−41.0 to 35.8	38.4	13.9

95% limits, 95% limits for mean difference; ICC, intraclass correlation coefficient for a single measure; MD, mean difference; RC, reproducibility coefficient; SD, standard deviation; WCV, within‐subject coefficient of variation.

Figure 5

Inter‐observer agreement using Bland–Altman plots for tumors with (n = 20) and without inner‐air density nodules (n = 68) measured by two different observers. The solid line shows the mean value, where the mean = (value 1 + value 2)/2. The dashed lines denote the means ± (1.96 × standard deviation). Ordinate value = ([value 1 − value 2]/mean) × 100. Pulmonary flow (PF) 1 = PF measured by observer 1, PF 3 = PF measured by observer 2. The same process applies to bronchial flow (BF) 1, BF 3, and perfusion index (PI) 1, PI 3. The graphs show Bland–Altman plots for (a) PF, (b) BF, and (c) PI values for lung cancers with inner‐air density and (d) PF, (e) BF, and (f) PI values for tumors without inner‐air density.

Discussion

Our results indicate that intra‐observer agreement in lung cancer on the whole was adequate, while inter‐observer agreement was more limited, especially for measurements of BF value in lung nodules and tumors with inner‐air density. Lung cancer size and tumors with inner‐air density might influence observer agreement of DCTP measurement.

This result was consistent with a previous finding of volume single‐input CTP in lung cancer using motion correction.22 The more difficult delineation of smaller tumors might primarily reflect the use of a freehand contour drawing method. Thus, the application of automatic computer‐aided delineation of the tumor and calculation of the volume perfusion value into future CT perfusion software might improve inter‐observer and intra‐observer agreements.

The agreements for measuring lung masses were better than those of the nodules. The lower intra‐observer agreement for smaller lesions was observed with wider 95% limits and higher WCVs. Although inter‐observer agreement was reduced in the nodule group, particularly the BF value, these results were acceptable for clinical use, and consistent with previous studies indicating the values for single or dual‐input CTP in lung cancer.22, 23 The volume scan mode, a measurement of the entire tumor and the introduction of motion correction techniques, might reflect relatively good agreement within observers.19, 22, 23

Moreover, we showed that lung cancers without inner‐air density (the cavity or the bronchus sign) had better agreement than those with inner‐air density, particularly regarding observer agreement. This finding indicated that the partial volume effect produced between the parts of soft tissue, and the air in the tumor played an important role in the reproducibility of lung cancer CTP, even when the function of motion correction was used to reduce misregistered dynamic images.

We used personalized protocols of dynamic scan and CM injection based on the BMI and weight of the patients to control X‐ray radiation and CM doses and to balance the image quality. To our knowledge, this study is the first published in a major academic journal. Following a previous study, we also designed a scan sequence comprising a TI of 1.5 seconds in the temporal window, covering the climax of pulmonary and systemic circulation, and a TI of two seconds before and after the window, to limit exposure time and maintain the accuracy of the perfusion calculation.24 The estimated radiation dose for the dynamic scan was 5.54 mSv. Considering the sensitivity to image noise, the maximum‐slope CTP images should have an acceptable SNR (a balance must be made between temporal resolution and the SNR).25 To some extent, the choice should be determined by the analysis method used. Thus, time–attenuation data from protocols that adopt a higher mAs value but a lower image frequency are appropriate for compartmental analysis.24, 25 The tumor SNR results illustrate this balance. For the broad use of CTP, detailed development, including dynamic protocol, hardware, and image algorithm are required to reduce the radiation dose and improve image quality.

This study has several limitations. First, the sample size was not predetermined in dimension with a power analysis. Second, other factors that might influence observer agreement analysis, such as the location, the duration of data acquisition, and image quality were not examined. Third, one of the two observers had limited experience in DCTP image analysis, and thus did not partake in intra‐observer agreement. This choice might decrease the significance of agreements.

In conclusion, the intra‐observer agreement for DCTP in lung cancer was satisfied, but the inter‐observer agreement was limited. Although DCTP was helpful for the diagnosis of lung cancer, a professional radiologist experienced in DCTP is required. Overall, observer agreement of large‐sized tumors with or without cavities or bronchus sign in solid parts was better than for small‐sized tumors with air density. In the future, automated segment analysis based on the density, morphology, and perfusion value of lung lesions should be developed to increase the reproducibility between observers.

Disclosure

No authors report any conflict of interest.