Quantitative CT imaging features for COVID-19 evaluation: The ability to differentiate COVID-19 from non- COVID-19 (highly suspected) pneumonia patients during the epidemic period.

OBJECTIVES: COVID-19 and Non-Covid-19 (NC) Pneumonia encountered high CT imaging overlaps during pandemic. The study aims to evaluate the effectiveness of image-based quantitative CT features in discriminating COVID-19 from NC Pneumonia.
MATERIALS AND METHODS: 145 patients with highly suspected COVID-19 were retrospectively enrolled from four centers in Sichuan Province during January 23 to March 23, 2020. 88 cases were confirmed as COVID-19, and 57 patients were NC. The dataset was randomly divided by 3:2 into training and testing sets. The quantitative CT radiomics features were extracted and screened sequentially by correlation analysis, Mann-Whitney U test, the least absolute shrinkage and selection operator (LASSO) logistic regression (LR) and backward stepwise LR with minimum AIC methods. The selected features were used to construct the LR model for differentiating COVID-19 from NC. Meanwhile, the differentiation performance of traditional quantitative CT features such as lesion volume ratio, ground glass opacity (GGO) or consolidation volume ratio were also considered and compared with Radiomics-based method. The receiver operating characteristic curve (ROC) analysis were conducted to evaluate the predicting performance.
RESULTS: Compared with traditional CT quantitative features, radiomics features performed best with the highest Area Under Curve (AUC), sensitivity, specificity and accuracy in the training (0.994, 0.942, 1.0 and 0.965) and testing sets (0.977, 0.944, 0.870, 0.915) (Delong test, P < 0.001). Among CT volume-ratio based models using lesion or GGO component ratio, the model combining CT lesion score and component ratio performed better than others, with the AUC, sensitivity, specificity and accuracy of 0.84, 0.692, 0.853, 0.756 in the training set and 0.779, 0.667, 0.826, 0.729 in the testing set. The significant difference of the most selected wavelet transformed radiomics features between COVID-19 and NC might well reflect the CT signs.
CONCLUSIONS: The differentiation between COVID-19 and NC could be well improved by using radiomics features, compared with traditional CT quantitative values.

Introduction

Corona Virus Disease 2019 (COVID-19) has threaten the public health. Computed tomography (CT) is one of the most intuitive assessment tools for lung disease. During pandemic, CT has played an active role in finding lesions, evaluating the severity and prognosis of disease and differentiating between COVID-19 and other pneumonia. However, the current CT diagnosis mainly depends on the experience of clinicians according to the qualitative characteristics, which may lack consistency, repeatability and operability.

The qualitative CT manifestations might be not enough for differential diagnosis among different pneumonia with imaging overlaps. Most of Non-COVID-19 (NC) patients are diagnosed as community-acquired pneumonia (CAP) during pandemic [1]. CAP is the third most common cause of death worldwide. Common pathogens of CAP are Streptococcus pneumonia, Haemophilus influenza, influenza virus, RSV virus and so on. The outbreak of COVID-19 in China was at a time of high CAP incidence. At the same period in the past, influenza viruses often spread [2]. Influenza virus and new coronavirus have similar clinical manifestations, such as cough, fever and myalgia. Laboratory tests also present a decrease in white blood cells and lymphocytes. In addition, there still lack of uniform CT imaging standards to distinguish COVID-19 from NC. At present, the most common CT manifestations of COVID-19 pneumonia are: GGO, consolidation, mixed density and subpleural distribution. However, these signs often appear in patients with bacterial infections or other viral infections. Previous report have shown that the pathological features of COVID-19 greatly resemble those seen in SARS and Middle Eastern respiratory syndrome (MERS) coronavirus infection, which makes the identification of CAP and COVID-19 particularly difficult [3]. Although, RT-PCR testing is the current gold standard for the diagnosis of COVID-19. In the current emergency, the low sensitivity of RT-PCR implies that many COVID-19 patients may not be identified and may not receive appropriate treatment in time. Previous research supports the use of chest CT for screening for COVD-19 for patients with clinical and epidemiologic features compatible with COVID-19 infection particularly when RT-PCR testing is negative [4-6].

Under such circumstances, CT imaging is gradually required to transform from qualitative to quantitative, which is to discover the invisible difference under the veil of similar CT signs or quantify the manually interpreted CT signs. The digital medical images contain a large amount of invisible deep information. With the rapid development of computing power and algorithms, more and more useful quantitative imaging indicators could be derived and applied in clinical studies. Radiomics is a method that covers a large number of quantitative imaging features, from first-order statistical features to second or transformed higher-order features [7]. The radiomics has been widely utilized in tumors, which can indicate genotyping, tumor microenvironment and susceptibility [8]. Several texture-based studies in pneumonia have also been reported including: identification of lung tissue, differentiation between different types of pneumonia, discrimination of pneumonia and other lung diseases, and prognosis prediction of pneumonia [9-14]. However, there is still a lack of effective quantitative features or standard quantitative criteria for the differentiation between COVID-19 and NC but highly suspected patients.

Therefore, the current study aimed to evaluate the effectiveness of radiomics-based quantitative CT features in discriminating COVID-19 from highly suspected NC. Meanwhile, the differentiation performance of traditional quantitative CT features such as lesion volume ratio, GGO or consolidation volume ratio were also considered and compared with radiomics-based method, which was to comprehensively assess the value of quantitative CT imaging analysis in pneumonia diagnosis.

Methods

Patients

We have abided by ethical standards of journal and the study design was approved by institutional Ethics Committee of Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital. We retrospectively enrolled 1988 fever patients from four centers of fever clinic in Sichuan Province during the period of January 23 to March 23. 1781 patients were excluded which showed negative lung imaging and COVID-19 nucleic acid test and without epidemic exposure history. Data including severe CT imaging artifacts or contrast-enhanced CT images were also excluded. Among the rest population, 207 patients had positive pulmonary manifestations, of which 88 patients showed positive lung imaging with positive COVID-19 nucleic acid test. 119 of them were suspected COVID-19 pneumonia patients, who present initially negative COVID-19 nucleic acid test, but were identified as highly suspected who cannot be exclude from COVID-19 (Fig 1) according to CT manifestations and consultation by multidisciplinary team (MDT) (specialists from infection, radiology and respiratory department, and finally excluded by repeated nucleic acid test.) While, 62 patients were excluded from 119 suspected patients, because there was no clear microbiological evidence. Finally, 57 non-COVID-19 patients were selected with microbiological evidence of infection including influenza A or B virus infection, bacterial infection and mixed-type infection. The CT images, laboratory indicators and clinical characteristics were recorded in Table 1.

Fig 1

Screen flow chart.

Table 1

Demographics of patients infected with COVID-19 and non-COVID-19 patients.

Data are represented as n (%) or mean (SD).

	COVID-19		NC
	Training set(n = 52)	Testing set(n = 36)	Training set (n = 34)	Testing set (n = 23)
Age	44.5(14.4)	47.9(14.5)	42.1(16.3)	42.2(16.4)
Gender/male	25(48%)	23(64%)	18(53%)	17(74%)
Symptoms
Fever	12(23%)	7(19%)	26(76%)	17(74%)
Cough	13(25%)	13(36%)	23(68%)	14(61%)
Myalgia or fatigue	11(21%)	7(19%)	7(21%)	7(30%)
Dizzy	5(10%)	2(6%)	2(6%)	3(13%)
Diarrhea	1(2%)	2(6%)	1(3%)	0
Sore Throat	1(2%)	2(6%)	5(15%)	4(17%)
Headache	0	1(3%)	3(9%)	1(4%)
Chest pain	2(4%)	0	1(3%)	0
Hospital length of stay(day)	14.4(5.3)	13.8(4.6)	6.1(4.1)	5.3(3.3)

All patients underwent non-enhanced chest spiral CT scanning. We selected the initial CT images scanned when the patients were first admitted to the hospital. Images from all cases were collected and analyzed by two radiologists (H. P [a thoracic radiologist with 28 years’ experience] and SK. P [a radiology attending doctor with 7 years’ experience in interpreting chest CT images]). The results of the judgement were reached by consensus.

CT image acquisition

All patients underwent non-enhanced chest spiral CT scanning. We selected the initial CT images scanned when the patient admitted to the hospital. The CT scanners were as follows: UCT528 40-row 40-slice spiral CT (United Imaging, shanghai, china), Siemens 16-row CT (Emotion; Germany) and Siemens 64-row (SOMATOM Perspective); and the following scanning parameters were used: slice thickness 1.0mm or 2mm, tube voltage 130 kVp, tube current 30mAs or tube voltage 120 kVp, tube current was regulated by an automatic exposure control system(80 to 300 mAs). The patient was examined in the supine position. The scan range was from the entrance of the chest to the lower lung (costophrenic angle), continuous screening with single breath at the end of inspiration. Images were reconstructed with a slice thickness of 1.5 mm or 1 mm and an interval of 1.5 mm or 1 mm, respectively. All images were displayed with standard lung (width, 1200 HU; level, –600 HU) and mediastinal (width, 350 HU; level, 40 HU) window settings. All images were exported as digital imaging and communications in medicine format for quantitative image feature extraction.

Image analysis

The workflow of imaging analysis and data processing was illustrated in Fig 2. The CT image analysis was conducted by Lung Kit software (GE Healthcare, Version LK2.2) and the detailed procedure was described in the Supplementary Method in S1 File. To reduce the impact of different scanner and scanning parameters, all the images were firstly resampled into isotropic voxel size of 1 mm*1 mm*1 mm using trilinear interpolation. After interpolation into isotropic voxel, the image’s intensity values were rounded to the nearest integer HU value. The low-pass Gaussian filter with σ = 0.5 was then conducted to increase the reproducibility of the radiomics features to be extracted [15]. Afterwards, the five anatomic lung lobes were firstly automatically segmented, which was followed by automatic pneumonia lesion volume of interest (VOI). The margin of the VOI was checked and manually adjusted by an experienced thoracic radiologist (SK. P, a radiology attending doctor with 7 years’ experience in interpreting chest CT images) and the obviously swollen blood vessels involved in the lesion were excluded, if necessary. All the automatically segmented or manually adjusted VOIs were checked by a senior radiologist (H. P [a thoracic radiologist with 28 years’ experience]) to reach consensus. The distributed lesions were considered as a whole VOI in the following analysis steps.

Fig 2

Workflow of CT imaging analysis.

Following the CT value discretized with binWidth = 25HU, a total of 851 radiomics features were then extracted from segmented VOIs by using Python package Pyradiomics v2.2 (The formula for calculating each feature were described in the website: https://pyradiomics.readthedocs.io/en/latest/index.html) [16]. In brief, a total of 851 extracted radiomics features were categorized into three groups: first-order features, textural features, and transformed features. There were 32 first-order features consisting of 18 intensity statistical features and 14 morphological features. Among 75 textural features, there were 24 Gray Level Co-occurrence Matrix (GLCM), 16 Gray Level Run Length Matrix (GLRLM), 16 Gray Level Size Zone Matrix (GLSZM), 14 Gray Level Dependence Matrix (GLDM) and 5 Neighboring Gray Tone Difference Matrix (NGTDM) features. For the transformed images, first-order wavelet transform decomposed the VOI into 8 sub-ROIs by using either a high- or low-pass filter in three dimensional directions, including high–high–high, high–high–low, high–low–low, high–low–high, low–high–low, low–high–high, low–low–high, and low–low–low. The same set of texture features (18 intensity statistical features, 24 GLCM features, 16 GLRLM, 16 GLSZM, 14 GLDM, and 5 NGTDM) were calculated based on these wavelet-transformed images and 744 wavelet features were obtained finally. These common or wavelet-transformed radiomics features were mainly calculated as referring to the previous studies on pneumonia differentiation [7, 8].

For lesion volume ratio analysis, the lesion volume ratio in five lung lobes was respectively calculated automatically by LK2.2 after lesion VOI was delineated. The lesion ratio in each lung lobe was scored from 0 to 5 which was defined according to the volume ratio involved: 0, no lesion; 1, ≤5%; 2, 6%-25%; 3, 26%-49%; 4, 50%-75%; 5, >75% [17].

For lesion component analysis, the part solid component was defined as the lesion with CT value <-200 HU while the parts with CT value >-200 HU were defined as solid component. And the volume ratio of each kind of lesion component in the left and right lung was derived automatically.

CT Image processing results and manually interpreted CT signs were aligned by two radiologists (H. P [a thoracic radiologist with 28 years’ experience] and SK. P [a radiology attending doctor with 7 years’ experience in interpreting chest CT images]). The results of the judgement were reached by consensus.

Data preprocessing, model establishment and evaluation

The radiomics feature data was firstly preprocessed by replacing missing values with median values and z-score normalization. The dataset (n = 145; COVID-19 = 88, NC = 57) was randomly stratified into training (n = 86) and testing (n = 59) set using the ratio of 3:2.

Following the CT image analysis and data preprocessing, five logistic regression models were constructed for differentiating COVID-19 from NC. Model A was constructed by radiomics features; Model B was constructed by lesion volume ratio in five lung lobes; Model C was constructed by lesion ratio score in each lung lobe derived from lesion volume ratio; Model D was constructed by volume ratio of lesion part-solid and solid component in left and right lungs; Model E was combined model using lesion score and component ratio. For every model, the features were selected sequentially only in the training dataset because the distribution of test dataset was taken as unknown status.

For Model A, the radiomics features for model construction were selected by the procedure as follows. The redundant features were firstly reduced by correlation analysis at a cut-value of 0.6. Then Mann-Whitney U test was used to eliminate features without statistical difference in COVID-19 and NC groups. The significant level was 0.05. Next, the least absolute shrinkage and selection operator (LASSO) logistic regression method with 10-fold cross validation was used for further feature selection and regularization to improve the model accuracy and avoid overfitting. The remaining features with non-zero coefficients were kept and involved into multi-variate backward stepwise logistic regression with minimum AIC (Akaike Information Criterion) method to establish the model. Besides, to verify the reliability of selected radiomics features in the logistic regression model, 100-times bootstrapping and 100-fold leave-group-out cross-validation (LGOCV) were performed.

From Model B to Model D, the features of lesion ratios or scores were sequentially selected by Mann-Whitney U test and univariate logistic analysis. The features with statistically P<0.05 were used for respective logistic regression model construction and the model score was derived from Eq (1). And for combined Model E, the model score of Model C and Model E derived from logistic regression were involved directly to construct logistic regression model. in which β0 is the constant, βi is logistic regression coefficient and xi is the value of selected features in the model.

The five logistic regression models were finally constructed based on the training set. And models’ classification performances were evaluated by receiver operating characteristic (ROC) analysis. The area under the ROC curve (AUC), accuracy, sensitivity and specificity were derived. To internally validate the model performances in the testing set, the retained feature names, logistic regression coefficients and the ROC cut-off value (Youden index reached the maximum) obtained from the training set were applied in the testing set. Then the corresponding performance in the test data could be calculated. In addition, the calibration curves and decision curve analysis (DCA) curves were calculated to assess the models’ prediction performance and clinical benefit.

Statistical analysis

The continuous variables or ordinal variables were compared by t-test or Mann-Whitney U test. The distribution of different CT manifestations judged by radiologist in COVID-19 and NC groups were compared by chi-squared test or Fisher exact test when small sample sizes occurred. When three or more manifestation types were included and compared by chi-squared test, the adjusted standardized residuals was calculated to characterize cell significance and the absolute value of adjusted residual larger than 2.58 was considered to be significant [18]. The Cramer’s V coefficients [0.1 (small); 0.3 (medium); 0.5 (large)] were used to assess the association between the categorical variables [19]. For ROC analysis, the cut-off value in the training set at the maximum of Youden index of each model was calculated and the sensitivity, specificity, accuracy in the training and testing sets were derived at such cut-off value. The ROC curves between different models were compared by Delong test. The reported statistical significance levels were all two-sided, and the statistical significance was set at 0.05. All these statistical analyses were conducted with SPSS Software (Version 25, IBM, Chicago, IL), GraphPad Software (version 8, San Diego, CA) and R software (Version: 3.6.1, https://www.r-project.org). The following R packages were mainly utilized: “glmnet” for logistic regression including LASSO algorithm; “pROC” for ROC analyses; “rmda” for DCA analysis.

Results

Feature selection

1. Radiomics feature selection

Among 851 radiomics features, by using Spearman’s correlation coefficient cut-value of 0.6, 47 features with low correlation were remained. Then, 26 features which were significantly different between COVID-19 and NC groups (P < 0.05) were retained after Mann-Whitney U test. Afterwards, through univariate logistic regression analysis (18 features left) which was followed by 10-fold cross validation of LASSO logistic regression algorithm, 13 features with non-zero coefficients were retained and involved into multi-variate backward stepwise logistic regression. With minimum AIC (Akaike Information Criterion) method, 10 features were finally selected to establish the radiomics Model A, which included wavelet.LHL_firstorder_Skewness (wLHL_fS), wavelet.HHL_glcm_Idn (wHHL_glcm_Idn), wavelet.LLL_glszm_SizeZoneNonUniformityNormalized(wLLL_glszm_SZNUN), wavelet.LLH_glcm_InverseVariance (wLLH_IV), wavelet.HLH_gldm_DependenceNonUniformityNormalized (wHLH_gldm_DNUN), wavelet.HLH_glszm_SmallAreaEmphasis(wHLH_glszm_SAE), original_shape_Flatness (ori_Flatness), original_glszm_SizeZoneNonUniformity (ori_glszm_SZNU), original_firstorder_10Percentile(ori_f10P), wavelet.HHH_glcm_Imc1 (wHHH_glcm_Imc1). The LASSO feature selection procedure was shown in S1(A) and S1(B) Fig in S1 File.

The statistical differences of each radiomics feature between COVID-19 and NC groups in training and testing sets were illustrated in S2 Fig in S1 File and summarized in S1 Table in S1 File. The wavelet-HHL_glcm_Idn, wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-LLH_glcm_InverseVariance and, wavelet-HHH_glcm_Imc1features showed significant differences (P<0.05) between COVID-19 and NC in both of training and testing sets.

2. Lesion ratio feature selection

The lesion volume ratio, score and component features were also selected to construct Model B to Model D. The lesion volume ratio of left and right bottom lung lobes were selected as significant features in Model B. The lesion-volume-based score in five lung lobes were all included in the Model C. And for Model D using component volume ratio, the GGO volume ratios in left and right lungs were selected. The selected features in each model and their statistical differences between COVID-19 and NC in training and testing sets were summarized in S2 Table in S1 File. It showed that all these lesion ratio features were significantly different (P<0.05) in COVID-19 and NC groups. The bar plot of lesion volume ratio and component volume ratio in different lung lobes were illustrated in S3 and S4 Figs in S1 File.

Model establishment and evaluation

The logistic regression calculation formula constructing Model A to Model E were shown in Table 2. The classification score of each model derived by weighting logistic regression coefficients onto selected features were illustrated for COVID-19 and NC, as shown in S5 Fig in S1 File. The confusion matrix and the classification score of each model were illustrated in Fig 3.

Table 2

The coefficient of each feature in different logistic regression models.

Model A a
variables	Coef.
intercept	6.8715
wavelet-LHL_firstorder_Skewness	-3.1308
wavelet-HHL_glcm_Idn	9.1974
wavelet-LLL_glszm_SizeZoneNonUniformityNormalized	-3.6223
wavelet-LLH_glcm_InverseVariance	-4.8207
wavelet-HLH_gldm_DependenceNonUniformityNormalized	-3.7400
wavelet-HHH_glcm_MCC	0.6721
wavelet-HLH_glszm_SmallAreaEmphasis	4.7616
original_shape_Flatness	-4.8054
original_shape_MajorAxisLength	-0.3870
original_glszm_SizeZoneNonUniformity	4.9277
original_firstorder_10Percentile	-4.0563
wavelet-HHH_glcm_Imc1	-11.1807
wavelet-HHH_glszm_GrayLevelNonUniformityNormalized	1.7932
Model B
variables	Coef.
intercept	-0.5569
Volume ratio of bottom-right lung lobe	0.0700
Volume ratio of bottom-left lung lobe	0.0586
Model C
variables	Coef.
intercept	-2.5602
Middle-right lung lobe	0.3299
Upper-left lung lobe	-0.2769
Upper-right lung lobe	0.2282
Bottom-right lung lobe	1.1397
Bottom-left lung lobe	0.8675
Model D
variables	Coef.
intercept	-0.1688
Left_GGO	0.0693
Right_GGO	0.0644
Model E
variables	Coef.
intercept	0.2100
CT lesion score (Model C score)	1.3920
Component score (Model D score)	-1.0101

a: Model A: radiomics model; Model B: lesion volume ratio model; Model C: lesion score model; Model D: lesion component ratio model; Model E: combined model of lesion score and lesion component ratio.

Fig 3

The confusion matrix and the logistic model score of each model in differentiating non-Covid-19 (NC) and Covid-19 groups.

(A, B) Training and testing sets of Model A: radiomics model; (C, D) Training and testing sets of Model B: lesion volume ratio model; (E, F) Training and testing sets of Model C: lesion score model; (G, H) Training and testing sets of Model D: lesion component ratio model; (I, J) Training and testing sets of Model E: combination model using lesion score and component ratio.

By ROC analyses, the performances of Model A to Model E were summarized in Table 3 and the ROC curves and precision-recall curves were respectively illustrated in S6 and S7 Figs in S1 File. Among five differentiation models, the Model A using radiomics features performed best with the highest AUC, sensitivity, specificity and accuracy in the training and testing sets. The AUC, sensitivity, specificity and accuracy were 0.994 (0.984–1.0), 0.942 (0.8375–0.9862), 1.0 (0.8793–1.0) and 0.965 (0.8982–0.9923) in the training set, while the corresponding values in the testing set were 0.977(0.947–1), 0.944 (0.8091–0.9941), 0.870 (0.6703–0.9631), 0.915 (0.8125–0.9673), respectively.

Table 3

The performance of each classification model in the training and testing sets.

		Training set				Testing set
Model	Cut-value	AUC	Sensitivity	Specificity	Accuracy	AUC	Sensitivity	Specificity	Accuracy
		(95%CI)	(95%CI)	(95%CI)	(95%CI)	(95%CI)	(95%CI)	(95%CI)	(95%CI)
Model A: radiomics model	>0.66	0.994	0.942	1.0	0.965	0.977	0.944	0.870	0.915
		(0.984–1.0)	(0.8375–0.9862)	(0.8793–1.0)	(0.8982–0.9923)	(0.947–1)	(0.8091–0.9941)	(0.6703–0.9631)	(0.8125–0.9673)
of Model B: lesion volume ratio model	>-0.0532	0.739(0.635–0.844)	0.788	0.588	0.709	0.738	0.722	0.652	0.694
			(0.6580–0.8792)	(0.4220–0.7366)	(0.6056–0.7951)	(0.603–0.873)	(0.5586–0.8430)	(0.4478–0.8130)	(0.5680–0.7980)
Model C: lesion score model	>0.7429	0.82(0.733–0.908)	0.635	0.823	0.674	0.772	0.694	0.826	0.712
			(0.4984–0.7523)	(0.6611–0.9203)	(0.6056–0.7951)	(0.638–0.905)	(0.5303–0.8211)	(0.6226–0.9363)	(0.6211–0.8404)
Model D: lesion component ratio model	>0.3553	0.682(0.566–0.798)	0.538	0.882	0.674	0.681	0.639	0.826	0.712
			(0.4050–0.6666)	(0.7278–0.9593)	(0.5695–0.7644)	(0.535–0.828)	(0.4752–0.7758)	(0.6226–0.9363)	(0.5855–0.8123)
Model E: combination model	>0.9339	0.84(0.754–0.925)	0.692	0.853	0.756	0.779	0.667	0.826	0.729
			(0.5566–0.8015)	(0.6939–0.9403)	(0.6547–0.8350)	(0.652–0.906)	(0.5026–0.7986)	(0.6226–0.9363)	(0.6032–0.8265)

In addition, as shown in S7 Fig in S1 File, all the model performance indicated by precision-recall curves shared the same tendency as the ROC analysis. The radiomics model (Model A) performed better than other image-derived models both in the training and test group, which was followed by combined model (Model E). It indicated the robustness of the radiomics model. Moreover, 100-times bootstrapping and 100-fold leave-group-out cross-validation (LGOCV) were also used to verify the reliability of the selected radiomics features and the regression model (detail in Supplementary Method in S1 File). Except for wavelet. LHL_firstorder_Skewness, all other features appeared more than 50 times during multiple splits into training and testing set as shown in S3 and S4 Tables in S1 File. The estimation of model overoptimism by using 100-times bootstrapping and 100-fold LGOCV method was summarized in S5 Table in S1 File. The appearing frequency of each selected radiomics feature and optimism-corrected AUC (bootstrap: 0.9; LGOCV: 0.898) represent an acceptable reliability of the feature and the predictive capability of constructed radiomics model.

For CT volume-ratio based models, the Model E which combining lesion score and component ratio performed better than others, with the AUC, sensitivity, specificity and accuracy of 0.84, 0.692, 0.853, 0.756 in the training set and 0.779, 0.667, 0.826, 0.729 in the testing set. It could be observed that the scored lesion volume ratio could enhance the differentiation performance than lesion volume ratio in its absolute value. Meanwhile, by further combining the ratio score and component ratio features, the model performance could be further enhanced.

After the Delong test, the statistical differences in ROC of each model were summarized in S6 Table in S1 File. It can be seen that the imaging radiomics Model A is statistically different from all other models (P <0.05). While the best volume-ratio-based Model E (combined model) only showed statistically differences compared with Model B and Model D in the training sets. And there was no significant difference between Model C (lesion score) and combined Model E.

The analysis of the calibration curve showed that the radiomics Model A (C-index: 0.994 in the training group; 0.977 in the testing group) and combined Model E(C-index: 0.840 in the training group; 0.779 in the testing group) had good predictive performance in the training and testing group (S8 Fig in S1 File). The decision curve analysis also indicated that the use of radiomics model A to differentiate COVID-19 from NC might achieve more net benefit (>0.85) than using the full regimen treatment or no treatment for patients in the threshold range of 0.2–0.8, which possess a higher clinical application value (Fig 4).

Fig 4

The decision curves of different models.

The y-axis represents the net benefit. The y-axis represents the net benefit. Model A: red; Model B: blue; Model C: green; Model D: purple; Model E: orange. The solid grey line represents the hypothesis that all patients suffered from COVID-19 (ALL). The solid black line represents the hypothesis that no patient receives treatment (NONE). At any given threshold, the highest curve (radiomics Model A) is the optimal decision-making strategy to maximize net benefits compared to other models.

The statistics of CT manifestation and signs distributions in COVID-19 and NC

In order to find the association between the CT manifestations or signs and different pneumonia types, the distribution of CT manifestations and signs were compared using chi-squared or Fisher‘s exact test, including lesion distribution, GGO involvement pattern, lobe predomination, lesion contour, bronchial wall thickening, air bronchogram, tree in bud sign, interlobular septal thickening, Intralobular septal_thickening, pleura effusion, and lesion attenuation.

The counts and percentage of individual CT sign in each group were summarized in S7 Table in S1 File. In the training set, the lesion distribution, GGO involvement pattern, lesion contour, bronchial wall thickening, air bronchogram and tree in bud sign showed significant differences (P < 0.05) between COVID-19 and NC patients. While in the testing group, the significant differences existed in lesion distribution, GGO involvement pattern, lobe predomination, bronchial wall thickening, tree in bud sign and intralobular septal thickening. Considering the same significant CT sign which simultaneously existed in training and testing set, it could be found from the Cramer’s V value that lesion distribution (Cramers’ V in training and testing: 0.792, 0.596), bronchial wall thickening(Cramers’ V in training and testing: 0.591, 0.414), tree in bud sign (Cramers’ V in training and testing:0.487, 0.788) and GGO involvement pattern (Cramers’ V in training and testing: 0.371, 0.686) showed association with different pneumonia of COVID-19 and NC. It seems that the bronchial wall thickening and tree in bud sign happened in NC more frequently, compared with that of COVID-19 patients. In addition, from the aspect of adjusted residuals, the lesion central distribution more likely happened in NC patients while lesion peripheral distribution was more frequent in COVID-19 patients. Meanwhile, GGO involvement pattern of Patchy GGO appeared more frequently in COVID-19 patients while GGO involvement of Cluster-like GGO was more frequent in NC patients.

Discussion

For CT signs, both influenza and COVID-19 presented GGO or GGO with consolidation. And multifocal lesions, GGO, and consolidation are the common CT findings of viral pneumonia [20-22]. Nevertheless, there are some differences in the CT manifestations between COVID-19 and influenza virus pneumonia. COVID-19 presented a high frequency of clear lesion margin and shrinking contour compared with influenza pneumonia. In the aspect of clinicopathological findings, COVID-19 histological examination showed bilateral diffuse alveolar damage with cellular fibromyxoid, desquamation of pneumocytes and hyaline membrane, and pulmonary edema with hyaline membrane formation, suggestive of early-phase ARDS formation. The clinicopathological features of influenza pneumonia findings were bronchiolitis with hemorrhage and necrotic debris, diffuse alveolar damage and alveolar hemorrhage. In addition, organizing pneumonia with prominent fibrin exudation and pulmonary edema were also observed. Irregularly fragmented hyaline membranes were seen at the alveolar orifice. The similar pathological features of COVID-19 as those in SARS and MERS also complicate the discrimination between COVID-19 and NC.

The selected subjects of this study are the patients who were difficult to diagnose after the consultation of MDT in each hospital. For these highly suspected patients, our study found that using the AUC of the ROC curve as the evaluation criterion, the radiomics model using quantitative imaging features has a higher diagnostic efficacy (AUC = 0.994 (0.984–1.0)), while the other four models based on traditional quantitative CT features were slightly lower. Additionally, among all the models based on lesion or component volume ratio, the diagnostic efficiency of the single-factor model was not as good as the combined model. These results indicated that quantitative imaging analysis could obtain more comprehensive disease imaging information, which includes first-order, second-order and transformed higher-order imaging information. While this imaging information cannot be found by traditional manual interpretation.

Radiomics has played an active role in tumor researches. Recently, it was also adopted in studies discriminating pneumonia and other lung diseases. T. Zhang et al. [12] selected four independent prediction features (2 wavelet transform first-order energy features; 1 texture feature and 1 shape feature) for differentiation between focal tissue pneumonia (FOP) and peripheral lung adenocarcinoma (LA). A good model effect was obtained: AUC (0.956), accuracy (87.6%), sensitivity (85.3%) and specificity (89.7%). For the differentiation between pneumonia and tumors, lymphocyte recruitment or tumor cell proliferation may lead to a higher degree of image discrimination, so the use of lower-order radiomics features may produce better discrimination [12, 23]. However, the identification of different types of pneumonia may require more advanced quantitative radiomics features due to the higher degree of imaging overlap. Sang Ok Park et al. [10] defined the texture of normal tissues, ground glass density shadow tissue, mesh tissue, honeycomb tissue and emphysema or solid tissue, and applied the volume proportion of each texture to achieve a model with 82% accuracy distinguishing common interstitial pneumonia (UIP) and non-specific interstitial pneumonia (NSIP). In addition to the classification of the same type of pneumonia, CT features were also used to distinguish between different types of pneumonia. For example, Bei Wang et al. [11] used radiomics to distinguish between children’s pulmonary tuberculosis and community-acquired pneumonia, hoping to help overcome the difficulty of sampling children’s tuberculosis and imaging overlaps. Five radiomics features (GLCM or GLRLM texture features: 4; first-order statistical feature: 1) in the lung solid part and 6 features (shape feature: 4; GLRLM texture feature: 1; first-order statistical feature: 1) in lymph nodes were screened out, respectively. Compared with the diagnosis performance of senior radiologists (AUC: 0.791), the model combining radiomics and clinical features could well improve the diagnostic performance with AUC of 0.971. These studies indicated the active role of the imaging textures in pneumonia differentiation.

In our study, the wavelet-HHL_glcm_Idn, wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-LLH_glcm_InverseVariance and wavelet-HHH_glcm_Imc1 features showed significant differences (P<0.05) between COVID-19 and NC in both of training and testing sets. The selected features with statistical differences mainly focused in the texture features with wavelet transform. And these features also shared relatively higher absolute value for their logistic regression coefficients. For example, the wavelet-HHH_glcm_Imc1, wavelet-HHL_glcm_Idn and wavelet-LLH_glcm_InverseVariance have regression coefficients as -13.961, 9.42 and -6.056. In addition, the 100-times bootstrapping and LGOCV also indicated that these features showed their importance or higher appearing frequency (greater than 60/100, especially 93/100 and 82/100 for wavelet-HHH_glcm_Imc1 and wavelet-LLH_glcm_InverseVariance respectively) during the multiple split of dataset into training and testing set. However, compared with texture features, the first-order statistical features’ contribution was weaker. For example, for the first-order features of original_firstorder_10Percentile and wavelet.LHL_firstorder_Skewness, their appearing frequency or the weight in the logistic regression were less.

The first-order features can be used to describe some pathological features. For example, relative to normal lung parenchyma, alveolar damage and abnormal transparency caused by emphysema can be well reflected by the significantly reduced average attenuation. And the tissue of idiopathic pulmonary fibrosis can also be reflected by the decreased peak sharpness of intensity distribution curve [11]. Skewness, kurtosis, and entropy have also been used to differentiate lung malignancy [24]. However, first-order statistical features cannot describe the spatial relationship between voxel features. Therefore, in order to explore the heterogeneous distribution of different tissues spatially and their correlations, it is necessary to introduce second-order texture features or filtered conversion to highlight or emphasize texture features in certain frequency domains. We found that among the selected radiomics features, the gray level co-occurrence matrix feature extracted from wavelet-transformed image- wavelet-LLH_glcm_InverseVariance describes the local flatness or fluctuation of intensity, and its relatively lower value in COVID-19 indicated that the local inhomogeneity in the COVID-19 lesion was slightly higher than that of NC. However, from the overall texture of the lesion or the textures extracted from different wavelet-filtering channel, the lesion texture manifestation might be different. Wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-HHH_glcm_Imc1 and wavelet-HHL_glcm_Idn, respectively described intensity homogeneity among zone size volumes, mutual-information-based intensity dependence which is a measure of texture complexity and normalized inverse difference which measures intensity homogeneity. Compared with NC lesions, the relatively lower value of wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-HHH_glcm_Imc1 and higher value of wavelet-HHL_glcm_Idn, in COVID-19 patients indicated the more homogeneous texture property in the image of these wavelet transformation channels. Although it still need detailed biological or physiological explanations for these texture manifestations, they might be potential to reflect the CT signs of COVID-19 and NC from a quantitative perspective. For example, the more frequent central distribution, Cluster-like GGO or tree in bud sign in NC patients may be related to its higher texture unevenness. Therefore, the use of quantitative features might help distinguish the signs of different pneumonia in more detail, which can obtain the higher discrimination efficiency and quantify the manually interpreted CT signs. It could be suspected that the significance of the quantitative CT features might increase with the imaging overlap between different type of pneumonia, especially when it is not enough to distinguish by the CT manifestations.

Although the model using quantitative CT imaging features presented better performance, the misjudgments still existed which resulted in the imbalance of sensitivity and specificity. It also suggested that the representative quantitative imaging features should be considered from the perspective of individualization, and extracted from standardized big data. In addition, with assistance of artificial intelligence, specific clinical work still requires clinicians to make comprehensive judgments on patients based on their own experience and other auxiliary examinations.

In this study, automatic lesion extraction and manual modification were used to ensure the consistency and repeatability to a certain extent. On such basis, quantitative parameter extraction was carried out, including the volume ratio of the lesion in the lung lobe, the volume ratio of the density component and the quantitative radiomics features. We found that compared to common quantitative parameters, such as volume ratio and density component ratio, the model established by using radiomics can distinguish NC and COVID-19 more accurately, which also means that it is necessary for CT images interpretation transform from qualitative to quantitative. Although based on WHO guidelines, the diagnosis of COVID-19 is referred to comprehensive judgments such as epidemiology, clinical manifestations, nucleic acid testing, and imaging, and the imaging alone cannot be used to diagnose. Such CT radiomics model might be potential to provide additional information indicating the risk of COVID-19 especially when the nucleic acid testing was negative and the CT manifestation was overlapped with other pneumonia. In the current study, it took about 2 min 30 s per patient for delineation of pneumonia lesion VOI, 44s per patient to extract 851 radiomics features from each pneumonia VOI and less than 10s to output the risk probability being COVID-19 based on the model (the computer processor: Intel(R) Xeon (R) E-2276M CPU@2.8GHz). Therefore, if the model was validated as reliable as possible, the clinical practice could be imagined. Once the radiologist gets the patient’s CT images and input it into auto-processing platform, the risk probability for the patient could be outputted within 3 min 30s. If the imaging model is highly indicative of the risk of COVID-19, repeating the nucleic acid test multiple times is recommended. Moreover, different for image-based deep learning algorithm, the calculated and saved radiomics features could help further correlated with or explain the biological or pathological manifestations in a describable and mathematical manner. The limitations of this study are as follows. Firstly, the data of patients with COVID-19 pneumonia came from four medical centers in Sichuan Province. Some scan parameters are inconsistent, and the enrolled sample size for the current imaging study was not so large. And such small sample size was not supportive to further discriminate different infections with different microbiological evidence. Secondly, our patients were mainly mild in symptom, with very a few serious patients. Thirdly, our model is not applicable to COVID-19 patients with negative chest CT imaging. Meanwhile, the method of automatically delineating the lesions and manually modifying the boundary was used. In order to make the quantitative CT indicators more standard, the unified definition of lesion boundaries is required to make through big data and multi-center standards. In addition, although quantitative imaging features can improve the identification ability of the model, we still need to study the pathophysiological changes represented by these selected imaging features in detail.

Conclusion

The quantitative CT imaging features are helpful to improve the diagnosis efficiency of suspected COVID-19 patients during the epidemic. The differentiation between COVID-19 and NC could be well improved by using radiomics features, compared with traditional CT quantitative values such as lesion volume ratio or component density ratio.

Supporting information & supporting information of methods.

(PDF)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

12 Apr 2021

PONE-D-20-25966

Quantitative CT imaging features for COVID-19 evaluation: the ability to differentiate COVID-19 from Non- COVID-19 (Highly Suspected) pneumonia patients during the epidemic period

PLOS ONE

Dear Dr. Zeng:

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Specific comments for the inclusion of additional information are provided by the reviewers, who both suggested major revisions.

Please submit your revised manuscript by May 27 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gayle E. Woloschak, PhD

Academic Editor

PLOS ONE

Additional Editor Comments:

Both reviewers had suggested major revisions as noted from the comments. Specific comments involve the inclusion of additional information for the reader.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please amend your current ethics statement to address the following concerns:

a) Did participants provide their written or verbal informed consent to participate in this study?

b) If consent was verbal, please explain i) why written consent was not obtained, ii) how you documented participant consent, and iii) whether the ethics committees/IRB approved this consent procedure.

3. Thank you for stating the following in the Competing Interests section:

"The authors have declared that no competing interests exist."

We note that one or more of the authors are employed by a commercial company: GE healthcare (China).

3.1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

3.2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

4. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

5. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

6. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

7. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This manuscript describes how radiomics-based logistic regression model can differentiate COVID-19-associated pneumonia from non COVID-19 pneumonia when patients present positive CT findings. While the finding is promising, it is still questionable how reproducible the result is because the technical details do not seem to be sufficient, particularly the software package utilized. There was a very similar study recently published using the datasets acquired in a different Chinese province as well. Significance and scientific rigor will be strengthened if the paper describes how the software works, and discussion to potentially test the models described in this paper and the other paper against the data although it is not required because of potential challenges to acquire the data from other sides. Overall, there is a clear technical merit for the study presented in this manuscript which can be improved with further revision.

Specific comments are:

1. Line 162: The reference [11] does not seem to be connected with the sentence “Among the rest population, …”

2. Lines 187-188: Please correct the unit for x-ray tube voltage: 130kv to 130kVp, 120 kV to 120 kVp.

3. Line 197 (and thereafter): Please consider changing from “Imaging analysis” to “Image analysis”

4. Line 199: Is “Lung Kit” software available commercially from GE Healthcare? It is hard to find any information about it except for papers published by authors including an author from GE. If it is not available, the functionality and other important details for this software, for reproducibility, better be described.

5. Lines 198-210: The writing of radiomics processing is basically identical to similar papers, particularly one published in 2021 (Huang Y, et al. MBC Medical Imaging). Although it is understood that the same software and same processing method were used for different patient population, the only significance difference seems to be where the datasets were obtained (Yunnan vs. Sichuan). Significance of this paper is diminished by not including all available datasets, and repeating the same technique for the same kind of data – resulting in slightly different performances. It is imperative to make a good rationale over what was presented and found in the Hunag 2021 paper.

6. About the above point, it is a good discussion topic to be included to test the radiomics-based model in those already collected datasets in a different Chines province.

7. A discussion about how this radiomics-based model could be added for its utility in comparison to RT-PCR. For example, if a patient is presented with pneumonia, and positive CT findings, but initial PCR testing is negative, this model could be used to further assure the negative or positive COVID-19 diagnosis, which could potentially help select a more appropriate treatment course.

8. There are many grammatical errors throughout the manuscript. A thorough copyediting is strongly recommended for revision.

Reviewer #2: The objective of the study is to investigate the effectiveness of CT-derived image features (Radiomics) in discriminating COVID-19 from non-COVID Pneumonia (NC). The study includes 88 cases with COVID and 57 patients with NC. The radiomics features were selected sequentially based on correlation analysis, Mann-Whitney U test and the classification was performed by backward stepwise LASSO logistic regression models. The performance of radiomics feature subset were further compared with traditional CT lesion score and volumes. The classification model with radiomics features outperformed traditional CT measures with an Area Under the Curve (AUC) of 0.97 vs. 0.77 (combination model of lesion related features). Although the presented results clearly indicate the radiomics feature subset’s superior performance, there are few major concerns regarding feature selection process and model evaluation.

Major Comments:

1. The authors mention that the dataset is randomly divided into training and test set using the ratio of 3:2. Is the randomized test set consists of reasonable class distribution (stratified)?

2. It would help understand model robustness if authors can add precision-recall plots for each model.

3. It would add a great value to the manuscript if author can discuss the importance of selected features from the radiomics pool. Are the features selected were consistent across multiple splits of train and test?

4. It is not clear whether the feature selection was performed on training set only? Or was the process performed on all available instances prior to train/test division? This would affect the performance of selected features on the test set.

Minor Comments:

1. Line 114. Need to provide a citation

2. Line 121 Please mention full description of GGO as it is mentioned for the first time

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Sandeep Bodduluri

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

5 Jun 2021

Answer to Reviewer #1:

We sincerely thank the reviewer for the hard work and valuable suggestions.

Reviewer #1: This manuscript describes how radiomics-based logistic regression model can differentiate COVID-19-associated pneumonia from non COVID-19 pneumonia when patients present positive CT findings. While the finding is promising, it is still questionable how reproducible the result is because the technical details do not seem to be sufficient, particularly the software package utilized. There was a very similar study recently published using the datasets acquired in a different Chinese province as well. Significance and scientific rigor will be strengthened if the paper describes how the software works, and discussion to potentially test the models described in this paper and the other paper against the data although it is not required because of potential challenges to acquire the data from other sides. Overall, there is a clear technical merit for the study presented in this manuscript which can be improved with further revision.

Specific comments are:

1. Line 162: The reference [11] does not seem to be connected with the sentence “Among the rest population, …”

Response:

We thank the reviewer for the kind reminder. The reference [11] was misplaced here so we have removed the reference [11] in the revised version.

2. Lines 187-188: Please correct the unit for x-ray tube voltage: 130kv to 130kVp, 120 kV to 120 kVp.

Response:

We thank the reviewer for this kind reminder. We have corrected the unit of X-ray tube voltage from kV into kVp.

3. Line 197 (and thereafter): Please consider changing from “Imaging analysis” to “Image analysis”

Response:

Thank you very much for this kind suggestion. We have changed the description from “Imaging analysis” into “Image analysis”.

4. Line 199: Is “Lung Kit” software available commercially from GE Healthcare? It is hard to find any information about it except for papers published by authors including an author from GE. If it is not available, the functionality and other important details for this software, for reproducibility, better be described.

Response:

We thank the reviewer for this valuable question. “Lung Kit” software (GE Healthcare) is one commercialized software only for scientific research. The software contains analysis module for COVID-19 in which the four-step data processing flow includes:

(1) user-defined image preprocessing: such as resampling, intensity discretization, image filtering or denoising and so on. In the current research, all the images were firstly resampled into isotropic voxel size of 1 mm*1 mm*1 mm using trilinear interpolation. After interpolation into isotropic voxel, the image’s intensity values were rounded to the nearest integer HU value. The low-pass Gaussian filter with σ = 0.5 was then conducted to increase the reproducibility of the radiomics features.

(2) the automatic segmentation of lung lobes and pneumonia lesion: the five lung lobes were segmented based on deep learning algorithm Dense V-networks [Gibson, E., Giganti, F., Hu, Y., Bonmati, E., Bandula, S., & Gurusamy, K. (2018). Automatic Multi-organ Segmentation on Abdominal CT with Dense V-networks. 0062(c), 1–12. https://doi.org/10.1109/TMI.2018.2806309 ], including 2 lobes in left lung (upper, bottom) and 3 lobes in right lung (upper, middle, bottom). Based on the lung lobes segmentation, the pneumonia lesions were detected and volume of interest (VOI) was segmented as a whole. The machine-human collaboration was applied to guarantee correct segmentation as described in the main text and we also carried it here for reviewer’s convenience:

“The margin of the VOI was checked and manually adjusted by an experienced thoracic radiologist (SK. P, a radiology attending doctor with 7 years’ experience in interpreting chest CT images) and the obviously swollen blood vessels involved in the lesion were excluded, if necessary. All the automatically segmented or manually adjusted VOIs were checked by a senior radiologist (H. P [a thoracic radiologist with 28 years’ experience]) to reach consensus. The distributed lesions were considered as a whole VOI in the following analysis steps. “

(3) quantization of pneumonia lesion: the volume of each segmented lung lobe and pneumonia lesion were firstly calculated. The lesion volume ratio and lesion component analysis were conducted as described in the main text and we also carried it here for reviewer’s convenience.

“For lesion volume ratio analysis, the lesion volume ratio in five lung lobes was respectively calculated automatically by LK2.2 after lesion VOI was delineated. The lesion ratio in each lung lobe was scored from 0 to 5 which was defined according to the volume ratio involved: 0, no lesion; 1, ≤5%; 2, 6%-25%; 3, 26%-49%; 4, 50%-75%; 5, >75%[ F P, T Y, P S, S G, B L, L L, et al. Time Course of Lung Changes at Chest CT during Recovery from Coronavirus Disease 2019 (COVID-19). Radiology. 2020;295(3):715-21.].

For lesion component analysis, the part solid component was defined as the lesion with CT value <-200 HU while the parts with CT value >-200 HU were defined as solid component. And the volume ratio of each kind of lesion component in the left and right lung was derived automatically.”

(4) radiomics feature extraction based on the lesion VOIs:

The “Lung Kit” software carried open source of Python package Pyradiomics v2.2 (https://pyradiomics.readthedocs.io/en/latest/index.html) and provide user interface to set the parameters for feature extraction. In the current study, following the CT value discretized with binWidth = 25HU, a total of 851 radiomics features were extracted from segmented pneumonia VOIs [Larue RTHM, van Timmeren JE, de Jong EEC, Feliciani G, Leijenaar RTH, Schreurs WMJ, Sosef MN, Raat FHPJ, van der Zande FHR, Das M, van Elmpt W, Lambin P. Influence of gray level discretization on radiomic feature stability for different CT scanners, tube currents and slice thicknesses: a comprehensive phantom study. Acta Oncol. 2017 Nov;56(11):1544-1553. doi: 10.1080/0284186X.2017.1351624. PMID: 28885084 .].

In brief, a total of 851 extracted radiomics features were categorized into three groups: first-order features, textural features, and transformed features. There were 32 first-order features consisting of 18 intensity statistical features and 14 morphological features. Among 75 textural features, there were 24 Gray Level Co‐occurrence Matrix (GLCM), 16 Gray Level Run Length Matrix (GLRLM), 16 Gray Level Size Zone Matrix (GLSZM), 14 Gray Level Dependence Matrix (GLDM) and 5 Neighboring Gray Tone Difference Matrix (NGTDM) features. For the transformed images, first-order wavelet transform decomposed the ROI into 8 sub-VOIs by using either a high- or low-pass filter in three dimensional directions, including high–high–high, high–high–low, high–low–low, high–low–high, low–high–low, low–high–high, low–low–high, and low–low–low. The same set of texture features (18 intensity statistical features, 24 GLCM features, 16 GLRLM, 16 GLSZM, 14 GLDM, and 5 NGTDM) were calculated based on these wavelet-transformed images and 744 wavelet features were obtained finally.

And Lung Kit software has been applied in texture-based analysis of severity or prognosis and differentiation from influenza pneumonia in some published manuscript as well. [Li L, Wang L, Zeng F, Peng G, Ke Z, Liu H, Zha Y. Development and multicenter validation of a CT-based radiomics signature for predicting severe COVID-19 pneumonia. Eur Radiol. 2021 Mar 30:1–12. doi: 10.1007/s00330-021-07727-x. PMID: 33786655; PMCID: PMC8009273.][Huang Y, Zhang Z, Liu S, Li X, Yang Y, Ma J, Li Z, Zhou J, Jiang Y, He B. CT-based radiomics combined with signs: a valuable tool to help radiologist discriminate COVID-19 and influenza pneumonia. BMC Med Imaging. 2021 Feb 17;21(1):31. doi: 10.1186/s12880-021-00564-w. PMID: 33596844; PMCID: PMC7887546.][Ke Z, Li L, Wang L, Liu H, Lu X, Zeng F, Zha Y. Radiomics analysis enables fatal outcome prediction for hospitalized patients with coronavirus disease 2019 (COVID-19). Acta Radiol. 2021 Feb 18:284185121994695. doi: 10.1177/0284185121994695. Epub ahead of print. PMID: 33601893.][Xiao F, Sun R, Sun W, Xu D, Lan L, Li H, Liu H, Xu H. Radiomics analysis of chest CT to predict the overall survival for the severe patients of COVID-19 pneumonia. Phys Med Biol. 2021 Apr 12. doi: 10.1088/1361-6560/abf717. Epub ahead of print. PMID: 33845467.][Wei W, Hu XW, Cheng Q, Zhao YM, Ge YQ. Identification of common and severe COVID-19: the value of CT texture analysis and correlation with clinical characteristics. Eur Radiol. 2020 Dec;30(12):6788-6796. doi: 10.1007/s00330-020-07012-3. Epub 2020 Jul 1. PMID: 32613287; PMCID: PMC7327490.]

5. Lines 198-210: The writing of radiomics processing is basically identical to similar papers, particularly one published in 2021 (Huang Y, et al. MBC Medical Imaging). Although it is understood that the same software and same processing method were used for different patient population, the only significance difference seems to be where the datasets were obtained (Yunnan vs. Sichuan). Significance of this paper is diminished by not including all available datasets, and repeating the same technique for the same kind of data – resulting in slightly different performances. It is imperative to make a good rationale over what was presented and found in the Hunag 2021 paper.

Response:

Thank you very much for this valuable suggestion to help us strengthen the manuscript. Firstly, we feel sorry about making the reviewer confusing about the study object of these two studies. And we would like to describe the difference from the Huang’s study and significance of the current manuscript from two aspects.

(1) The patient population and study object of the current research

The etiology was different for the patient population in the current research. The pulmonary appearance of COVID-19 is very similar to that of influenza or some other mixed infection. Our control group patients are more representative of the majority of the population and are more universal. Among the patients in our control group, there are viral infections, bacterial infections, and mixed infections, which are more diverse. In addition, all the patients were collected during the COVID-19 epidemic. This kind of differentiation research might better reflect its value during the COVID-19 epidemic. Because in fact, when we encountered community-acquired pneumonia at the very beginning, it might be not possible to immediately diagnose whether the patient is infected with the virus. Besides, the same pathogenic infection may show different clinical manifestations and imaging characteristics in different regions. And the patients with independent pathogenic infection were relatively rare during the same epidemic to proceed the differentiation . So it is meaningful for us to compare COVID-19 patients and patients with other pathogenic infections during the COVID-19 epidemic. And the object of the current research was to comprehensively consider different kinds of quantitative CT imaging features and to evaluate their differentiation performance for the COVID-19 from other pathogenic infections during the COVID-19 epidemic.

(2) The image-based features involved in the current research

In the current study, one of the main idea is to evaluate the performance of quantitative CT-derived features in COVID-19 differentiation. The image-based features in the current study included radiomics features and quantized CT-manifestations. Therefore, we tried to construct five CT-value-derived models: Model A was by CT radiomics features; Model B by lesion volume ratio in five lung lobes; Model C by lesion ratio score in each lung lobe derived from leveled lesion volume ratio; Model D was constructed by volume ratio of lesion part-solid and solid component in left and right lungs; Model E was combined model using lesion score and component ratio. The construction and comparison for these different models were from the consideration that whether we could find out some simple quantitative CT-derived parameters instead of relatively complex calculation to fulfill the COVID-19 differentiation. Because the simpler the method, the easier for clinical practice. Therefore, the consideration of the image-derived features was different from the other study which involved radiomics features and CT signs or their combination.

We sincerely thank the reviewer again for this kind suggestion.

6. About the above point, it is a good discussion topic to be included to test the radiomics-based model in those already collected datasets in a different Chines province.

Response:

We deeply agree with the reviewer’s suggestion that an external validation of the model performance was necessary if the patient population or the enrollment criteria was the same. However, we felt very sorry that the COVID-19 data sharing between different province is limited by the policy and the data involved in the current research is the most available data that our hospital could collect around Sichuan province. But we would keep it in mind and try to externally validate the model if we have obtained the external data for other province or we would also consider to participate in multicenter-study if there is related project proposed in our country.

7. A discussion about how this radiomics-based model could be added for its utility in comparison to RT-PCR. For example, if a patient is presented with pneumonia, and positive CT findings, but initial PCR testing is negative, this model could be used to further assure the negative or positive COVID-19 diagnosis, which could potentially help select a more appropriate treatment course.

Response:

Thank you very much for this valuable suggestion on the clinical practice of the constructed model. We have added a discussion about the clinical practice potential for the radiomics model in the revised manuscript, and we also described it here for reviewer’s convenience.

Revised manuscript on LINE 508:

Although based on WHO guidelines, the diagnosis of COVID-19 is referred to comprehensive judgments such as epidemiology, clinical manifestations, nucleic acid testing, and imaging, and the imaging alone cannot be used to diagnose. Such CT radiomics model might be potential to provide additional information indicating the risk of COVID-19 especially when the nucleic acid testing was negative and the CT manifestation was overlapped with other pneumonia. In the current study, it took about 2 min 30 s per patient for delineation of pneumonia lesion VOI, 44s per patient to extract 851 radiomics features from each pneumonia VOI and less than 10s to output the risk probability being COVID-19 based on the model (the computer processor: Intel(R) Xeon (R) E-2276M CPU@2.8GHz). Therefore, if the model was validated as reliable as possible, the clinical practice could be imagined. Once the radiologist gets the patient’s CT images and input it into auto-processing platform, the risk probability for the patient could be outputted within 3 min 30s. If the imaging model is highly indicative of the risk of COVID-19, repeating the nucleic acid test multiple times is recommended. Moreover, different for image-based deep learning algorithm, the calculated and saved radiomics features could help further correlated with or explain the biological or pathological manifestations in a describeable and mathematical manner.

8. There are many grammatical errors throughout the manuscript. A thorough copyediting is strongly recommended for revision.

Response:

Thank you very much for kind reminder. We have carefully checked and revised the manuscript thoroughly.

Answer to Reviewer #2:

We sincerely thank the reviewer for the hard work and valuable suggestions.

Reviewer #2: The objective of the study is to investigate the effectiveness of CT-derived image features (Radiomics) in discriminating COVID-19 from non-COVID Pneumonia (NC). The study includes 88 cases with COVID and 57 patients with NC. The radiomics features were selected sequentially based on correlation analysis, Mann-Whitney U test and the classification was performed by backward stepwise LASSO logistic regression models. The performance of radiomics feature subset were further compared with traditional CT lesion score and volumes. The classification model with radiomics features outperformed traditional CT measures with an Area Under the Curve (AUC) of 0.97 vs. 0.77 (combination model of lesion related features). Although the presented results clearly indicate the radiomics feature subset’s superior performance, there are few major concerns regarding feature selection process and model evaluation.

Major Comments:

1. The authors mention that the dataset is randomly divided into training and test set using the ratio of 3:2. Is the randomized test set consists of reasonable class distribution (stratified)?

Response:

Thank you very much for this kind reminder.

Yes, the data was divided into a training and test set at a ratio of 3:2 using a stratified random sampling method. In the training set, there were 52 COVID-19 patients and 34 non-COVID-19 patients. In the test set, there were 36 COVID-19 patients and 23 non-COVID-19 patients.

The statistics difference of CT manifestations and signs features between training and testing set was summarized in Table C1. There was no significant bias for the CT manifestations and signs between training and test set.

Table C1 The statistics difference of CT manifestations and signs features between training and testing set.

We have also revised the manuscript (part “Data preprocessing, model establishment and evaluation”) on LINE 220 as follows:

“The dataset (n = 145; COVID-19 = 88, NC = 57) was randomly stratified into training (n = 86) and testing (n = 59) set using the ratio of 3:2.”

2. It would help understand model robustness if authors can add precision-recall plots for each model.

Response:

We thank the reviewer for this valuable suggestion to help us strengthen the manuscript.

Following the reviewer’s instruction, we have added the precision-recall plots for 5 models constructed in the manuscript, including (1) Model A: radiomics model; (2) Model B: lesion volume ratio model; (3) Model C: lesion score model; (4) Model D: lesion component ratio model; (5) Model E: combined model using lesion score and component ratio simultaneously. And the area under the PR curve for each model were summarized in the inset. As shown in Figure S7 , all the model performance shared the same tendency as the ROC analysis. The Radiomics model (Model A) performed better than other image-derived models both in the training and test group, which was followed by combined model (Model E). It indicated the robustness of the radiomics model.

Figure S7 The Precision-Recall (PR) curve of each model in the training (A) and testing (B) sets. The area under the PR curve for each model was labeled in the legend.

3. It would add a great value to the manuscript if author can discuss the importance of selected features from the radiomics pool. Are the features selected were consistent across multiple splits of train and test?

Response:

We thank the reviewer again for this valuable suggestion and help us check out some errors and strengthen the manuscript.

We carefully checked the radiomics features reliability measurement in the original manuscript and found out that the description for the method was wrong which should be corrected from “100-fold leave-group-out cross-validation (LGOCV)” into “100-times bootstrapping method”. In addition, we found out that some parts of discussion about the radiomics features should be corrected or supplemented.

Therefore, besides the 100-times bootstrapping used in the original manuscript, in the revised version, we also supplemented the method of 100-fold leave-group-out cross-validation (LGOCV) with the proportion of data in the sub–training sets (60%), to measure the reliability of the radiomics features involved in the final model and the model overoptimism. The reliability of the radiomics features was measured by the appearing frequency during multiple splits of training and test sets. While the model overoptimism was verified by the mean AUC in the multiple-split training and test sets. We have corrected, revised or supplemented the corresponded context in the method, result and discussion part for the revised manuscript. And we also summarized here for reviewer’s convenience.

(1) The method for the 100-times bootstrapping and LGOCV was described as follows.

(1-1) 100-times bootstrapping method

Step 1: Apparent performance. Taking the model’s AUC performance of predicting model developed in the original whole training dataset as the apparent performance.

Step2: BOOTSTRAP sample splitting. The whole original training dataset was repeatedly split into bootstrapped training and test sets. The bootstrapped training set with the same sample size as original training dataset was constructed by sampling with replacement from the original sample. The out-of-bag samples constructed the test set in each bootstrap.

Step3: BOOTSTRAP model establishment. Starting features were features used to establish radiomics model based on the original training dataset. And these features were further selected by backward stepwise logistic regression method with minimum AIC (the same as the modeling method in the manuscript) in each bootstrapped training set. The logistic regression model established from the bootstrapped training set was respectively tested in the “out-of-bag” test set. The AUC were obtained for bootstrapped training and test set in each bootstrap loop.

Step4: Model optimism among bootstrap. Calculate the model optimism as the difference between the AUC of bootstrapped training set and the test set.

Step5: The optimism-corrected performance. Repeating Step 2 to Step 4 for 100 times. The appearing frequency of each feature and the average optimism were calculated and recorded. Subtract the value from the apparent performance in step 1 and obtain an optimism-corrected performance.

(1-2) 100-fold leave-group-out cross-validation (LGOCV) method

Step 1: Apparent performance. Taking the model’s AUC performance of predicting model developed in the original whole training dataset as the apparent performance.

Step2: Multiple splitting of training and test sets. The whole dataset was repeatedly split into training and test sets. A proportion of data in the sub–training sets was set as 60% which was the same as the randomized stratification ratio of 3:2 in the original whole dataset. The rest 40% data were taken as test set in each loop. Such group splitting was repeated 100 times.

Step3: LGOCV model establishment. Starting features were features used to establish radiomics model based on the original training dataset. And these features were further selected by backward stepwise logistic regression method with minimum AIC (the same as the modeling method in the manuscript) in each LGOCV training set. The logistic regression model established from the LGOCV training set was respectively tested in the LGOCV test set. The AUC were obtained for LGOCV training and test set in each loop.

Step4: Model optimism among bootstrap. Calculate the model optimism as the difference between the AUC of LGOCV training set and test set.

Step5: The optimism-corrected performance. Repeating Step 2 to Step 4 for 100 times. The appearing frequency of each feature and the average optimism were calculated and recorded. Subtract the value from the apparent performance in step 1 and obtain an optimism-corrected performance.

(2) The results for the 100-times bootstrapping and LGOCV were summarized as follows.

The 100-times bootstrapping result showed that all of the 10 radiomics features except for wavelet.LHL_firstorder_Skewness (47 times in 100 folds) appeared more than 50 times during 100-times bootstrapping, which indicated the reliability of the selected features. As shown in Table S3.

Table S3 The appearing frequency of the selected radiomics features during 100-times bootstrapping.

The LGOCV result showed a similar appearing frequency of features with bootstrapping method, in which all of the 10 radiomics features except for wavelet.LHL_firstorder_Skewness (43 times in 100 folds) appeared more than 50 times during 100-times group splitting. As shown in Table S4.

Table S4 The appearing frequency of the selected radiomics features during 100-fold leave-group-out cross-validation (LGOCV).

The estimation of model overoptimism by using 100-times bootstrapping and 100-fold leave-group-out cross-validation (LGOCV) method was summarized in Table S5. The appearing frequency of each selected radiomics feature and optimism-corrected AUC (bootstrap: 0.9; LGOCV: 0.898) represent an acceptable reliability of the feature and the predictive capability of constructed radiomics model.

Table S5 Estimation of the area under the ROC curve and the model overoptimism for the radiomics model using 100-times bootstrapping and 100-fold leave-group-out cross-validation (LGOCV) method.

(3) The revised discussion about the selected radiomics features.

Revised manuscript on LINE 447 (the italic and bold font indicates the revised part):

In our study, the wavelet-HHL_glcm_Idn, wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-LLH_glcm_InverseVariance and wavelet-HHH_glcm_Imc1 features showed significant differences (P<0.05) between COVID-19 and NC in both of training and testing sets. The selected features with statistical differences mainly focused in the texture features with wavelet transform. And these features also shared relatively higher absolute value for their logistic regression coefficients. For example, the wavelet-HHH_glcm_Imc1, wavelet-HHL_glcm_Idn and wavelet-LLH_glcm_InverseVariance have regression coefficients as -13.961, 9.42 and -6.056 respectively. In addition, the 100-times bootstrapping and LGOCV also indicated that these features showed their importance or higher appearing frequency (greater than 60/100, especially 93/100 and 82/100 for wavelet-HHH_glcm_Imc1 and wavelet-LLH_glcm_InverseVariance respectively) during the multiple split of dataset into training and testing set. However, compared with texture features, the first-order statistical features’ contribution was weaker. For example, for the first-order features of original_firstorder_10Percentile and wavelet.LHL_firstorder_Skewness, their appearing frequency or the weight in the logistic regression were less.

The first-order features can be used to describe some pathological features. For example, relative to normal lung parenchyma, alveolar damage and abnormal transparency caused by emphysema can be well reflected by the significantly reduced average attenuation. And the tissue of idiopathic pulmonary fibrosis can also be reflected by the decreased peak sharpness of intensity distribution curve [Zhang T, Yuan M, Zhong Y, Zhang YD, Li H, Wu JF, et al. Differentiation of focal organising pneumonia and peripheral adenocarcinoma in solid lung lesions using thin-section CT-based radiomics. Clin Radiol. 2019;74(1):78.e23-78.e30.]. Skewness, kurtosis, and entropy have also been used to differentiate lung malignancy [Scrivener M, de Jong EEC, van Timmeren JE, Pieters T, Ghaye B, Geets X. Radiomics applied to lung cancer: a review. Translational Cancer Research. 2016;5(4):398-409.]. However, first-order statistical features cannot describe the spatial relationship between voxel features. Therefore, in order to explore the heterogeneous distribution of different tissues spatially and their correlations, it is necessary to introduce second-order texture features or filtered conversion to highlight or emphasize texture features in certain frequency domains. We found that among the selected radiomics features, the gray level co-occurrence matrix feature extracted from wavelet-transformed image- wavelet-LLH_glcm_InverseVariance describes the local flatness or fluctuation of intensity, and its relatively lower value in COVID-19 indicated that the local inhomogeneity in the COVID-19 lesion was slightly higher than that of NC. However, from the overall texture of the lesion or the textures extracted from different wavelet-filtering channel, the lesion texture manifestation might be different. Wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-HHH_glcm_Imc1 and wavelet-HHL_glcm_Idn, respectively described intensity homogeneity among zone size volumes, mutual-information-based intensity dependence which is a measure of texture complexity and normalized inverse difference which measures intensity homogeneity. Compared with NC lesions, the relatively lower value of wavelet-LLL_glszm_SizeZoneNonUniformityNormalized, wavelet-HHH_glcm_Imc1 and higher value of wavelet-HHL_glcm_Idn, in COVID-19 patients indicated the more homogeneous texture property in the image of these wavelet transformation channels. Although it still need detailed biological or physiological explanations for these texture manifestations, they might be potential to reflect the CT signs of COVID-19 and NC from a quantitative perspective. For example, the more frequent central distribution, Cluster-like GGO or tree in bud sign in NC patients may be related to its higher texture unevenness. Therefore, the use of quantitative features might help distinguish the signs of different pneumonia in more detail, which can obtain the higher discrimination efficiency and quantify the manually interpreted CT signs. It could be suspected that the significance of the quantitative CT features might increase with the imaging overlap between different type of pneumonia, especially when it is not enough to distinguish by the CT manifestations.

4. It is not clear whether the feature selection was performed on training set only? Or was the process performed on all available instances prior to train/test division? This would affect the performance of selected features on the test set.

Response:

Thank you very much for helping us make the method much clearer for the readers.

For five models constructed in the current study, each group of features were selected sequentially only in the training dataset because the distribution of test dataset was taken as unknown status. The finally selected feature names, logistic regression coefficient and ROC cut-off value determined based on the training set were copied to the test set for model’s internal validation. In addition, the radiomics features were conducted with median values replacement for missing values and Z-score normalization prior to the feature selection procedure.

We have revised the feature selection method in the manuscript (part “Data preprocessing, model establishment and evaluation”) and also copied it here for reviewer’s convenience.

Revised manuscript on LINE 219:

Method Part: Data preprocessing, model establishment and evaluation

“The radiomics feature data was firstly preprocessed by replacing missing values with median values and z-score normalization. The dataset (n = 145; COVID-19 = 88, NC = 57) was randomly stratified into training (n = 86) and testing (n = 59) set using the ratio of 3:2.

Following the CT image analysis and data preprocessing, five logistic regression models were constructed for differentiating COVID-19 from NC. Model A was constructed by radiomics features; Model B was constructed by lesion volume ratio in five lung lobes. Model C was constructed by lesion ratio score in each lung lobe derived from lesion volume ratio. Model D was constructed by volume ratio of lesion part-solid and solid component in left and right lungs. Model E was combined model using lesion score and component ratio. The logistic regression method was used for every model construction. For every model, the features were selected sequentially only in the training dataset because the distribution of test dataset was taken as unknown status.

For Model A, the radiomics features for model construction were selected by the procedure as follows. The redundant features were firstly reduced by correlation analysis at a cut-value of 0.6. Then Mann-Whitney U test was used to eliminate features without statistical difference in COVID-19 and NC groups. The significant level was 0.05. Next, the least absolute shrinkage and selection operator (LASSO) logistic regression method with 10-fold cross validation was used for further feature selection and regularization to improve the model accuracy and avoid overfitting. The remaining features with non-zero coefficients were kept and involved into multi-variate backward stepwise logistic regression with minimum AIC (Akaike Information Criterion) method to establish the model. Besides, to verify the reliability of selected radiomics features in the logistic regression model, 100-fold leave-group-out cross-validation (LGOCV) was performed.

From Model B to Model D, the features of lesion ratios or scores were sequentially selected by Mann-Whitney U test and univariate logistic analysis. The features with statistically P<0.05 were used for respective model construction. And for combined Model E, the selected significant features from lesion score and component ratio were involved simultaneously.

The five logistic regression models were finally constructed based on the training set. And models’ classification performances were evaluated by receiver operating characteristic (ROC) analysis. The area under the curve (AUC), accuracy, sensitivity and specificity were derived. To internally validate the models’ performances in the test set, the retained feature names, logistic regression coefficients and the ROC cut-off value (when Youden index reached the maximum) obtained from the training set were applied in the test set. Then the corresponding predictive performance in the test data could be derived. In addition, the calibration curves and decision curve analysis (DCA) curves were calculated to assess the models’ prediction performance and clinical benefit.”

Submitted filename: Response to Reviewers without track.docx

Click here for additional data file.

16 Jun 2021

PONE-D-20-25966R1

Quantitative CT imaging features for COVID-19 evaluation: the ability to differentiate COVID-19 from Non- COVID-19 (Highly Suspected) pneumonia patients during the epidemic period

PLOS ONE

Dear Dr. Zheng:

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please address the concern about the inclusion of more information about the lung kit in a revision.

Please submit your revised manuscript by Jul 30 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gayle E. Woloschak, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Thank you for your ethics statement, "Ethics approval and consent to participate: This retrospective study was approved by the institutional Ethics Committee of Sichuan Academy of Medical Sciences & Sichuan

Provincial People's Hospital and obtained the informed consent of the patients.Please amend your current ethics statement to address the following concerns:

In the ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records used in your retrospective study, including:

Please amend your current ethics statement to address the following concerns:

a) Did participants provide their written or verbal informed consent to participate in this study?

b) If consent was verbal, please explain i) why written consent was not obtained, ii) how you documented participant consent, and iii) whether the ethics committees/IRB approved this consent procedure.

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

The reviewers have asked for additional information to be added about the lung kit in order to enhance reproducibility of the work.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The response to my critiques for the previous submission looks adequate and appropriate. However, the details of “Lung Kit” software, in the response letter, better be in either the manuscript’s main text or at least in the supplementary material. It is still not clear how others could reproduce the results without knowing that it can be obtained for research.

Reviewer #2: The authors satisfactorily addressed all the comments raised in the initial review and made significant changes to the manuscript accordingly.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: SANDEEP BODDULURI

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Jul 2021

Dear editor,

We are very pleased to learn again from your letter about revision for our manuscript (PONE-D-20-25966R2). Thank you very much for providing us the further suggestion and the opportunity for us to resubmit our revised manuscript. We have revised the manuscript according to the comments from the reviewers. Our answers to the comments are described as follows, with revision tracks in the revised manuscript and supplementary material.

We hope that the revised manuscript could be considered acceptable for publication. We are looking forward to hearing from you.

1.) "Please amend your current ethics statement to address the following concerns:

This retrospective study was approved by the institutional Ethics Committee of Sichuan Academy of Medical Sciences & Sichuan Provincial People's Hospital and obtained the informed consent of the patients. （In the Cover Letter）

2.) manuscript format：We have modified it.

We sincerely thank the reviewer again for the kind patience and valuable suggestion during the peer-review process.

Submitted filename: Response to Reviewers_20210625.docx

Click here for additional data file.

3 Aug 2021

Quantitative CT imaging features for COVID-19 evaluation: the ability to differentiate COVID-19 from Non- COVID-19 (Highly Suspected) pneumonia patients during the epidemic period

PONE-D-20-25966R2

Dear Dr. Zheng,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Gayle E. Woloschak, PhD

Section Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All comments have been addressed appropriately. The manuscript is of good quality, and no further revision seems necessary.

Reviewer #2: The authors made significant changes to the manuscript and provided with additional statistical analysis to ensure the robustness of the prediction performance, as requested in the initial review.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Sandeep Bodduluri

5 Jan 2022

PONE-D-20-25966R2

Quantitative CT Imaging Features for COVID-19 Evaluation: the Ability to Differentiate COVID-19 from Non- COVID-19 (Highly Suspected) Pneumonia Patients during the Epidemic Period

Dear Dr. Zeng:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Gayle E. Woloschak

Section Editor

PLOS ONE