Higher agreement between readers with deep learning CAD software for reporting pulmonary nodules on CT.

Purpose: The aim was to evaluate the impact of CAD software on the pulmonary nodule management recommendations of radiologists in a cohort of patients with incidentally detected nodules on CT.
Methods: For this retrospective study, two radiologists independently assessed 50 chest CT cases for pulmonary nodules to determine the appropriate management recommendation, twice, unaided and aided by CAD with a 6-month washout period. Management recommendations were given in a 4-point grade based on the BTS guidelines. Both reading sessions were recorded to determine the reading times per case. A reduction in reading times per session was tested with a one-tailed paired t-test, and a linear weighted kappa was calculated to assess interobserver agreement.
Results: The mean age of the included patients was 65.0 ± 10.9. Twenty patients were male (40 %). For both readers 1 and 2, a significant reduction of reading time was observed of 33.4 % and 42.6 % (p < 0.001, p < 0.001). The linear weighted kappa between readers unaided was 0.61. Readers showed a better agreement with the aid of CAD, namely by a kappa of 0.84. The mean reading time per case was 226.4 ± 113.2 and 320.8 ± 164.2 s unaided and 150.8 ± 74.2 and 184.2 ± 125.3 s aided by CAD software for readers 1 and 2, respectively.
Conclusion: A dedicated CAD system for aiding in pulmonary nodule reporting may help improve the uniformity of management recommendations in clinical practice.

Introduction

The increasing demand for ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) has dramatically increased the workload of radiologists over the last decades. The number of cross-sectional studies needing reporting from radiologists increased by two-fold in the period 1999–2010 [1], and for CT specifically, the radiologist’s workload during on-call hours was reported to have quadrupled from 2006 to 2020 [2]. This pressure on the radiologist’s practice can increase missed cases and diagnostic errors [3], [4].

Some of this increased workload can be attributed to pulmonary nodules, a prevalent CT finding. One or more pulmonary nodules have been reported as an incidental finding in 14–31 % of patients undergoing chest CT imaging for any clinical indication [5], [6], [7] and in 51 % of lung cancer screening trial participants, pulmonary nodules were found at baseline [8]. Considering that more than 95 % of these findings are benign, it is crucial that pulmonary nodules are managed safely and cost-effectively to prevent unnecessary patient burden and healthcare utilization but still allow for the early detection of lung cancer or lung metastases.

Specific nodule characteristics help radiologists stratify the risk of malignancy. Characteristics such as size, composition, and location are implemented in malignancy risk prediction methods, like the Brock or PanCan risk prediction model [9], [10], to help determine the level of risk for developing lung cancer. Then there are guidelines that give recommendations regarding an appropriate follow-up such as the 2015 British Thoracic Society (BTS) guidelines and the 2015 Fleischner society guidelines [11], [12]. However, despite this, a low to moderate interobserver agreement is often reported between radiologists on pulmonary management recommendations [13], [14], [15], [16].

Computer-aided detection (CAD) systems have been developed to support radiologists in several tasks for reporting pulmonary nodules on chest CT, and some of these systems are commercially available. These CAD systems have shown high sensitivities on their own [17] and as a second or concurrent reader and have been shown to improve a radiologist’s sensitivity for reporting pulmonary nodules [18], [19], [20]. How CAD software affects pulmonary management recommendations remains to be determined. Therefore, this study aimed to evaluate the effect of CAD software on interobserver agreement of pulmonary nodule management recommendations.

Methods

Institutional review board approval was obtained for this single-center study and informed consent was waived due to its retrospective nature (reference number: 2018.0061). The study was performed in a large teaching hospital in the Netherlands. To prevent any diagnostic or treatment impact on patients as a result of the study, only scans older than 5 years before the start of the study were included. The image database of the institution was manually consulted for eligible studies between July 2013 and September 2013 by a resident radiologist. Fifty adult patients scanned with chest CT were selected for pulmonary nodule assessment. Eligibility was determined based on the initial radiology reports and the availability of prior scans in PACS. Predetermined stratification criteria ensured a patient cohort containing cases with and without nodules, as well as with or without prior imaging. The stratification criteria were as follows:(a) no pulmonary nodules, (b) pulmonary nodules without prior scans, (c) pulmonary nodules with prior scans which do not contain actionable nodules, or (d) pulmonary nodules with prior scans which include actionable nodules that require follow-up. Five, ten, five, and thirty patients were included in groups a to d, respectively for a cohort size of 50 patients. Patients with CT scans reporting more than 5 pulmonary nodules, a pulmonary mass (>30 mm in largest axial diameter), or interstitial lung disease were excluded from this study.

Image acquisition

The chest CT scans were performed on various multislice systems: Aquilion One (n = 56), Toshiba Medical Systems, Otawara, Japan, Sensation 16 (n = 25), Siemens Medical Solutions, Forchheim, Germany, and Gemini 16 (n = 4), Philips Medical Systems, Best, the Netherlands). Scans were performed at 100, 120, or 140 kVp at variable mAs. The image data were reconstructed with a lung filter kernel at a slice thickness setting of either 2.00 mm (n = 73), or 3.0 mm (n = 12). The convolution kernels used were FC08 (n = 2), FC18 (n = 13), FC55 (n = 15) and FC56 (n = 26) for Toshiba systems, B31f (n = 1) and B24f (n = 24) for Siemens systems, and A (n = 2), B (n = 1), L (n = 1) for Philips systems. Routine nonionic intravenous contrast was applied in 63/85 (74.1 %) main and prior scans (300mgI/ml Omnipaque, GE healthcare, IL, USA).

CT assessment

All scans of the study cohort were anonymized and migrated to a local test workstation which was identical to the workstation used in clinical practice. Two readers assessed all scans twice (two reading sessions) with a washout period of 6 months. The order in which the scans were to be reported was randomized at the start of each reading session. Reader 1 is a thoracic radiologist with 15 years of experience in reporting pulmonary nodules on chest CTs and reader 2 is a general radiologist with 13 years of experience in reporting pulmonary nodules on chest CTs. The workstation included AGFA enterprise imaging 8.1.2 (AGFA Healthcare N.V., Mortsel, Belgium) and Vitrea Enterprise Solution (Vital Images Inc, Minnetonka, Minnesota, United States] (“VITREA”), which includes a semi-automated volumetry tool but no volume doubling time calculator for this a web-based tool was available (http//:www.chest-xray.com/index.php/calculators/doublingtime). The first reading session was performed without a CAD system (unaided) and the second session was performed with the availability of the CAD outputs (aided) (Veye Chest v2.15.3, Aidence B.V., Amsterdam, NL). The CAD system automatically detects and segments pulmonary nodules and provides information such as nodule composition (solid, sub-solid), diameter, volume, and volumetric changes over time (growth percentage and volume doubling time). The CAD outputs are made available to the radiologists after processing within the reader’s workstation as two separate DICOM series of the original scan study. One series contains a single summary image of the nodule findings and the other contains the original axial chest series with an overlay highlighting the CAD’s nodule findings. Each reading session was recorded with screen recording software (Camtasia, TechSmith, Okemos, Michigan, United States).

The 50 main scans of each patient were assessed together with the 35 prior scans where applicable as one case. The readers were tasked to read the scans to determine the pulmonary nodule management recommendation and report relevant pulmonary nodules that contributed to their management decision and disregard any concurrent abnormalities. The readers reported the relevant nodules’ location, composition, volume, and if applicable nodule growth percentage and volume doubling time. If volumetry was not deemed reliable, the longest axial diameters were reported. An actionable nodule was defined as a non-calcified pulmonary nodule with a volume of between 65 mm3 and 14.000 mm3 or with the largest axial diameter between 5 mm and 30 mm that requires follow-up according to the reader. Finally, a nodule management recommendation grade based on the 2015 British Thoracic Society guidelines was determined for each case [12]. Figs. S1 and S2, included in the Supplementary materials present the flow diagrams used to come to the recommended patient management using on a 4-point grade (A-D). After both reading sessions had been completed, all cases with discrepant BTS grades between readers were re-evaluated during a consensus meeting and a consensus BTS grade was determined between the two readers.

Reading time assessment

Reading time was determined by at least two reviewers independently from the screen recordings. The start of the reading was defined as the moment where the main scan is opened in the viewer and the end was defined as the moment a new main scan is opened or the screen recording has ended. Discrepant reading times were re-evaluated by another reviewer to determine a final reading time.

Statistical analysis

To summarize patient demographics and radiological findings, continuous or discrete variables are presented as mean and standard deviation or median and range, where appropriate. Categorical variables are summarized in frequencies and percentages of the whole. To determine whether the mean reading time per scan was reduced by CAD a one-sided paired t-test was performed. A linear weighted kappa was used to assess the agreement of the BTS grade between readers and consensus. Confusion matrix analysis with exact binomial confidence limits of the BTS grades was performed to evaluate the diagnostic performance of readers versus the consensus reading. Statistical analyses were performed with R statistical software (R.4.1.1, R Foundation for Statistical Computing, Vienna, Austria) and Python programming language (version 3.9.7, Python Software Foundation, Delaware, USA).

Results

The mean age in years of the fifty included patients was 65.0 ± 10.9 (range 32–84) at the time of the main scan. 20 patients were male (40 %). A total of 64 and 63 nodules were reported by readers 1 and 2 unaided by CAD. Aided by CAD, readers 1 and 2 reported 41 and 44 nodules respectively. A summary of the radiological findings is provided in Table 1 and 2.

Table 1

Summary of radiological findings (n=50).

	Reader 1 unaided	Reader 1 aided	Reader 2 unaided	Reader 2 aided
Number of nodules reported	64	41	63	44
- Patients with nodules	41/50 (82.0 %)	41/50 (82.0 %)	44/50 (88.0 %)	40/50 (80.0 %)
Nodule locations
- Right
- UL	17/64 (26.6 %)	12/41 (29.3 %)	21/63 (33.3 %)	8/44 (18.2 %)
- ML	8/64 (12.5 %)	8/41 (19.5 %)	5/63 (7.9 %)	7/44 (15.9 %)
- LL	16/64 (25.0 %)	10/41 (24.4 %)	14/63 (22.2 %)	14/44 (31.8 %)
- Left
- −Right - UL - −ML −LL −Left −UL −LL	12/64 (18.7 %)	5/41 (12.2 %)	13 (20.6 %)	8/44 (18.2 %)
- LL	11/64 (17.2 %)	6/41 (14.6 %)	10 (15.9 %)	7/44 (15.9 %)
Nodule measurements
- Mean volume±sd (mm3)	567.2 ±626.8	736.3 ±835.0	613.9 ±791.3	632.0 ±720.0
- Count	29/64 (45.3 %)	40/41 (97.6 %)	35/63 (55.6 %)	42/44 (95.5 %)
- Mean diameter±sd (mm)	10.8 ±5.7	27.0 ±NA	10.0 ±3.5	17.8 ±8.6
- Count	35/64 (54.6 %)	1/41 (2.4 %)	28/63 (44.4 %)	2/44 (4.5 %)
Nodules composition
- Solid	58/64 (90.1 %)	36/41 (87.8 %)	57/63 (90.5 %)	38/44 (86.4 %)
- −Solid - Part-solid - −GGO	5/64 (7.8 %)	4/41 (9.8 %)	4/63 (6.3 %)	4/44 (9.1 %)
- GGO	1/64 (1.6 %)	1/41 (2.4 %)	2/63 (3.2 %)	2/44 (4.5 %)

Table 2

Agreement between readers and consensus on patient management recommendation as determined by the BTS grade (linear weighted kappa).

	Unaided	Aided
Between readers	0.61	0.84
Reader 1 and consensus	0.66	0.80
Reader 2 and consensus	0.57	0.87

For each patient, readers concluded each assessment by doing a recommendation for the patient management according to the BTS grade. A consensus session led to 27 (54 %) patients being assigned a BTS grade A, 5 (10 %) a grade B, 8 (16 %) a grade C, and 10 (20 %) a grade D. The linear weighted kappa between readers unaided was 0.61. Readers showed a better agreement with the aid of CAD, namely by a kappa of 0.84. The CAD-aided readings of each reader also showed a higher agreement with the consensus session than when readings were done unaided. The kappas between unaided sessions and consensus were 0.66 and 0.57 for readers 1 and 2, respectively. Between aided sessions and consensus, kappas of 0.80 and 0.87 were found for readers 1 and 2.

Anything other than a BTS grade A, requires a clinical follow-up of the reported pulmonary nodules. The sensitivity for finding a BTS grade A at consensus unaided was 0.83 (95 % CI: 0.61–0.95) for reader 1 and 0.76 (9 % CI: 0.55–0.91) for reader 2. A sensitivity of 0.85 (95 % CI: 0.66–0.96) and 0.92 (95 % CI: 0.73–0.99) was found for readers 1 and 2 aided, respectively. The specificities were 0.85 (95 % CI: 0.66–0.96) and 0.84 (95 % CI: 0.64–0.95) unaided and 1.00 (95 % CI: 0.85–1.00) and 0.96 (95 % CI: 0.80, 1.00) aided for reader 1 and 2.Fig. 1.

Fig. 1

Illustrative example of the CAD output as shown to the readers during the aided session of a growing part-solid nodule found in a 57 year old female patient.

The mean reading time per patient of Reader 1 was 226.4 ± 113.2 s unaided and 150.8 ± 74.2 s aided by CAD software. The mean reading time per patient of Reader 2 320.8 ± 164.2 s unaided and 184.2 ± 125.3 s aided by CAD software. Fig. 2 presents a boxplot of the reading times of each session and reader. For both readers 1 and 2, a significant reduction of reading time was observed of 33.4 % and 42.6 % respectively (p < 0.001, p < 0.001). Fig. 2 presents an example of a pulmonary nodule seen in a viewer unaided and aided by CAD. The reduced reading times with CAD could be attributed to the fact that the readers reported fewer actionable nodules during that session. A subgroup analysis of cases where an equal number of nodules was reported during both sessions, also showed reduced reading times, namely a reduction of 38.0 % and 30.3 % for readers 1 and 2.

Fig. 2

Boxplots of the reading times of each reader during unaided and aided sessions. Each box represents the median (bold horizontal dash) and the interquartile range. The tails and additional data points represent the full range of the reading times. The notch represents the 95 % confidence interval of the median.

Discussion

This study shows that a CAD system as a concurrent reader can reduce the interobserver variation of pulmonary nodule management recommendations while also reducing reporting times of pulmonary nodules on chest CT by 33–43 %.

Pulmonary nodules are the most common incidental finding on chest CT, yet interpreting these findings can be challenging. The interobserver variance between radiologists has been shown to be high for not only the number of nodules reported or for nodule classification, but also for follow-up recommendations [13], [15], [16], [21], [22], [23]. Gierada et al. compared the findings on 135 baseline screening CT scans over 16 radiologists and reported that only in 44 % of cases all radiologists agreed on whether the case was a positive or negative screening result and a kappa of 0.35 was found between radiologists for determining whether or not patient follow up was recommended [16]. Van Riel et al. estimated that 65.1 % of discrepant readings could potentially affect patient management in a retrospective study with 8 radiologists of 145 screening CT scans [13] and Penn et al. showed a moderate interobserver agreement on patient management based on the 2013 Fleischner Society recommendations (kappa of 0.56) [15]. The interobserver agreement was comparable in this study when the 2 readers were asked to give management recommendations during the unaided session. However, the kappa increased from 0.61 to 0.84 when aided by CAD. The sensitivity and specificity of detecting a BTS grade A by consensus were either the same or slightly higher during the CAD aided session. This suggests that by implementing a CAD system, no additional patients will be unnecessarily followed up or inappropriately omitted from further diagnostics.

Differences in nodule management recommendations between radiologists could have several causes. In a retrospective study with 6 readers evaluating the scans of 100 screening participants, there were 155 cases of disagreements in findings between readers of which 77 led to a different follow-up decision (yes or no follow-up). Of these 77 cases, 30 % of discrepancies could be attributed to measurement differences, 27 %, 27 %, and 16 % were attributable to the detection of nodules, choice of the target lesion, and nodule classification [23]. The CAD outputs provided a list of candidate nodules along with their measurements (diameter and volume), volume doubling times, and composition, therefore mitigating some of the largest sources of reader disagreements. Further research is warranted to determine if the changes in management recommendations due to the availability of CAD outputs lead to better adherence to pulmonary nodule guidelines. Increasing the uniformity of patient management recommendations will allow for more robust and effective triage algorithms in clinical practice and screening programs.

The reading times in this study were comparable to the reading times reported by Hsu et al. and Beyer et al. and both studies reported a significant reduction of readings with CAD aided readings [19], [24]. One study demonstrated a reduction of 15.8–29 % in reading times aided by CAD by six radiologists [24] and the other only 6.9 % on average over four radiologists [19]. This study showed higher reductions aided by CAD (33–43 %). There could be several reasons for this. One is that our cohort included 35 cases with prior scans to consider and only 5 cases without nodules. Beyer et al. and Hsu et al. included 50 % and 35 % of cases without nodules, respectively, and no cases with prior imaging. The current study included 20 % of patients without nodules described in the original report and 30 % of patients with prior imaging. Also, differences in the CAD systems used may have played a role.

A radiologist’s workload has substantially increased over the past decades due to higher demands of CT, among others. The prospect of population screening programs for lung cancer with low-dose CT [25], [26] will introduce even more pressure. A reduction in reading time with CAD could help radiologists keep up with demand. At our institution, approximately 11,200 new chest CTs are reported per year of which 55 % of cases have prior imaging. Although our research suggests an average reduction in reading time of about two minutes reporting pulmonary nodules, our cohort is not directly representative of the actual radiologist's workload and thus further research is warranted to determine the cost-effectiveness of CAD systems in the clinic.

Limitations of this study include those inherent to its retrospective setting and small cohort size. The manual selection of patients would have introduced selection bias. Another limitation is that both radiologists reported fewer actionable nodules when reading aided by CAD, most likely because the CAD system provided the radiologist with a list of nodules and therefore there was no need to personally keep track of all findings. This could have affected the reading time as less time was spent describing nodules. This study is also limited to the possible time saved reporting pulmonary nodules specifically and does not consider the time spent on interpreting and reporting on other radiological findings or the total reporting time. Therefore, its extrapolation to clinical practice is limited. Finally, no patient follow-up or histology was available for a golden standard. A consensus meeting to discuss BTS grades provided a surrogate golden standard.

Conclusion

A dedicated CAD system for pulmonary nodule reporting may improve the interobserver agreement on the management recommendations and which can contribute to the effectiveness of triage algorithms for detecting early-stage lung cancer patients.

CRediT authorship contribution statement

HL Hempel: Writing – original draft preparation, Data analysis, Data curation MP Engbersen: Writing- Original draft preparation, Visualization, Reviewing and Editing J Wakkie: Conceptualization, Reviewing and Editing BJ van Kelckhoven: Reviewing and Editing, Methodology, Investigation W de Monyé: Conceptualization, Methodology, Reviewing and Editing, Investigation, Supervision.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: MPE and JW declare being employed by Aidence BV, the other authors have nothing to declare.