Diagnostic accuracy of CO-RADS in patients with suspected Coronavirus Disease-2019: A single center experience.

INTRODUCTION: COVID-19 Reporting and Data System (CO-RADS) is a tool for standardizing the reports of patients with suspected or confirmed Sars-CoV-2 infection. We performed a study of the performance of the CO-RADS in a triage scenario of patients in Brazil.
METHODS: Data from 426 Computed Tomography (CT) scans from March 2020 through December 2020 were assessed in an ambidirectional, both retrospective and prospective, for the assessment in one of the six categories of the CO-RADS. We assessed sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR) Youden's index, Positive and Negative Clinical Utility Index (UC + and UC- respectively) and diagnostic odds ratio (DOR). We also plotted Receiver Operating Characteristics (ROC) curve with Area Under the Curve (AUC) for CO-RADS of >4 (4 + 5).
RESULTS: For CO-RADS classification > 4 (4 + 5) considered positive, the AUC obtained was of 0.89 (95% CI of 0.02), sensitivity of 78% (95% CI of 0.3), specificity of 91% (95% CI of 0.3), PPV of 0.92 (95% CI of 0.02), NPV of 0.41 (95% CI of 0.03), PLR of 0.85 (95% CI of 0.2), and NLR of 0.23 (95% CI of 0.02).
CONCLUSION: CO-RADS demonstrated overall good diagnostic performance in stratifying patients with suspected Sars-CoV-2 infection, even those without confirmed laboratorial diagnosis, therefore being useful in a triage scenario with lack of resources.

Introduction

At the end of 2019, a viral infection with a new type of Coronavirus, later called SARS-CoV-2, was reported in Wuhan, capital of Hubei province, China. As of December 23, 2020, there have been 78,320,614 cases of the disease and 1,723,502 deaths worldwide. In Brazil, on the same date, there were 7,318,821 cases and 188,259 deaths. Among the countries most affected by the disease are the United States, Brazil, India and Russia.

In developing countries, including Brazil, there has been a diagnostic difficulty due to the lack of a rapid laboratory test that allows the patient to be adequately screened for isolation, due to the high risk of contagion and transmission of the disease. The RT-PCR exam considered as “gold standard” for the diagnosis can take up to a few hours to a few days for a result.

Computed Tomography (CT) of the chest has been important in the diagnostic aid of COVID-19 because it allows an immediate result, anticipating actions such as isolation and clinical management, in addition to often estimating the assessment of the severity and extent of pulmonary involvement, including in early stages of the disease.5., 6.

In March 2020, the Dutch Radiology Society proposed a system to standardize the result of chest tomography for the diagnosis of COVID-19, the CO-RADS (COVID-19 Reporting and Data System), to allow the tomographic reports to be uniform and replicable. This classification system proved to be useful in several scenarios of the initial screening of patients for admission to isolation or general units while waiting for the confirmatory test.8., 9., 10.

The objective of this study is to validate the diagnostic and screening performance of CO-RADS at a center located in a population with a high prevalence of COVID-19 and to show its usefulness in a scenario with a low availability of beds and confirmatory exams.

Methods

This ambidirectional, single-center study was approved by our institutional review board and the written informed consent was waived. The reporting was done in accordance with the recommendations of the Standards for Reporting Diagnostic Accuracy Initiative (STARD).

The retrospective branch of the study involved CT from March 24 to September 1, 2020, and the prospective branch from September 2 to December 31, 2020.

Eligibility criteria

The examinations of the included patients involved those with respiratory symptoms and/or risk factors of exposure to Sars-COV-2 and who were submitted to chest CT and to gold standard collection (RT-PCR) up to 14 days after the entry CT according to institutional guidelines and available of tests.

The clinical inclusion criteria for a patient to undergo a CT were: fever higher than 37.5 °C, cough, dyspnea, close relationship with a confirmed positive individual, travel or residential history in areas with high prevalence of disease contact with individuals with fever or respiratory symptoms from those areas within 14 days prior to the CT scan based on suspicion and risk stratification guidelines.

As the center where the study was conducted is open to the population and receives patients referred from other medical services, the study cohort was characterized as random.

Missing imaging, important clinical information in the records such as the result of the RT-PCR, symptom descriptions and outcome status (alive or dead) at the time of assessment implied the exclusion of the patient from the final study cohort.

CT technique and CO-RADS classification

All CT scans were performed using a single 64-slice CT scanner (Philips Brilliance®). All patients were scanned in supine position during single deep-inspiration breath-hold. The contrast medium was administered according to the radiologists' opinion on the overall findings (e.g. suspicion of pulmonary thromboembolism) and according to appropriateness criteria.

The CO-RADS Classification was based on the original article and involves seven categories:

CO-RADS 0: not interpretable scan technically insufficient for assigning a score.

CO-RADS 1: normal or non-infectious CT pattern (very low probability).

CO-RADS 2: CT typical for other infection but not COVID-19 (low probability).

CO-RADS 3: equivocal/unsure features compatible with COVID-19, but also other diseases

CO-RADS 4: Highly suspicious probability by CT patterns.

CO-RADS 5: Very high suspicious probability by CT patterns.

CO-RADS 6: proven RT-PCR positive for SARS-CoV-2 despite CT patterns.

Patients were considered highly suspicious for COVID-19 were those who had CT with categories 4 and 5, justifying isolation and the patients with CO-RADS 1 and 2 were considered as low probability and transferred to general beds, CO-RADS 3 patients were dependent on clinical and laboratory criteria for isolation. Fig. 1 summarizes the main imaging aspects for each CO-RADS classification.

Fig. 1

Examples of CT scans for each CO-RADS category analyzed. CO-RADS 2 (A) with micronodules in the lower left lobe with “tree in a bud” pattern suggestive of other infections. CO-RADS 3 (B) with pleural effusion and scattered ground glass areas that could be implied in congestive states, CO-RADS 4 (C), more confluent ground glass areas predominantly in the lower lobes, but with pleural effusion that did not allow the classification in the CO-RADS 5 category and CO-RADS 5 (D) with a typical ground glass centrilobular and peripheral distribution.

Data analysis

Three senior radiologists of the hospital, with more than ten years of experience and six other residents, were responsible for analyzing the images blinded to all clinical data and without knowledge of the RT-PCR of the patients. All exams had a double reading (senior radiologist 1 + senior radiologist 2 or senior radiologist+resident), one of them being done by one of the three senior radiologists and the discrepancy between readers was resolved with the opinion of a third senior radiologist.

Standard reference

RT-PCR testing performed on respiratory specimens obtained by nasopharyngeal and throat swabs served as a reference standard for the COVID-19 diagnosis. Clinical information and index test results were not available for the assessors of the reference standard.

Evaluation of performance tests

The index tests were one of the six CO-RADS categories. We assessed sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR) Youden's index, Positive and Negative Clinical Utility Index (UC + and UC− respectively). UC+ was obtained by multiplying the sensitivity with PPV and UC− by multiplying specificity with NPV. The diagnostic odds ratio (DOR) was assessed by the ratio of PLR and NLR.

We chose to evaluate UC+, UC−, Youden's and DOR for two reasons: these parameters give a better comprehension of the diagnostic performance in tests that the gold standard is not perfect or is under evaluation,14, 15., 16., 17. such as the RT-PCR and also the correlation with pre-test probability by these parameters gives a better idea of the usefulness of the test when it's most needed as shown in previous studies that correlates these parameters.14, 15., 16.

We also plotted Receiver Operating Characteristics (ROC) curve with the Area Under the Curve (AUC) for patients with CO-RADS categories >4 (CO-RADS 4 + 5) since these were the classifications that indicated isolation of a suspected patient in our center (thus considered “positive” until proven otherwise). We excluded CO-RADS 6 since those patients already had laboratorial confirmation and were automatically conducted to the specific sectors.

We also plotted the nomograms for a positive and a negative test (PPTP and NPTP respectively) for each CO-RADS classification.

Statistical analysis

Categorical variables were made by the chi-square test and the statistical significance was considered when p < 0.05. All the continuous values had their confidence intervals calculated with 95% reliability. Since this study involved a randomized cohort of patients, we did not perform any previous sample size estimation in our population.

The software used for the tabulation was Microsoft Excel® 2016 (Microsoft Corporation), the ROC curves were plotted on SPSS version 22 (IBM Corp., Armonk, NY, USA) and for the calculations of the values and plotting of nomograms we used the software r (r foundation).

Results

Flow and demographics of the patients selected

From 1024 consecutive patients initially screened from our database, we included a sample of 532 patients that met all the inclusion criteria and, after exclusion of CO-RADS 6 patients, we kept 426 patients, of those 273 (64%) obtained in the retrospective branch and 153 (36%) in the prospective branch, for the final analysis. The main reasons for exclusion as well as the patient flowchart are summarized in Fig. 2 .

Fig. 2

Patient flowchart for the selection in the final analysis.

The median age of the participants was 64.9 years (95% CI of 16.7). Male patients were 330 (54.5%), and Female patients were 276 (45.5%). Patient characteristics are summarized in Table 1 . We observed the highest percentage of deaths in CO-RADS 2 patients mainly because this category contemplated several diseases with tomographic features atypical for COVID-19 but still with a high mortality rate such as extensive lobar pneumonia, pulmonary edema, metastasis etc.

Table 1

Characteristics of the patients.

CO-RADS (n)	Age95% CI	Sex (female/male)	Deaths (%)
1 (71)	64.916.7	39/42	11 (15)
2 (16)	64.716.5	8/8	11 (69)
3 (95)	64.916.6	42/53	31 (32)
4 (56)	64.916.7	32/24	20 (36)
5 (188)	64.916.9	78/110	54 (29)
6 (106)	64.916.9	47/59	10 (9)

CI – confidence interval.

N = number of patients.

CO-RADS 1 classification was found in 71 (13.3%) patients, CO-RADS 2 in 16 (3.0%), CO-RADS 3 in 95 (10.5%), CO-RADS 4 in 56 (10.5%), CO-RADS 5 in 188 (35.3%) and CO-RADS 6 in 106 (19.9%) patients.

The number of RT-PCR-positive patients was 379 against 183 negative in the final cohort. The graph correlating the number of PCR positive patients according to their CO-RADS classification is found in Fig. 3 .

Fig. 3

Histogram showing the incidence of each CO-RADS category and PCR status.

Diagnostic performance of CO-RADS

For a CO-RADS classification of 2 considered positive, we obtained an accuracy of 0.63 (95% CI of 0.3), a sensitivity of 89% (95% CI of 0.3), a specificity of 28% (95% CI of 0.3), a PPV of 0.61 (95% CI of 0.02), an NPV of 0.43 (95% CI of 0.03), a PLR of 3.11 (95% CI of 0.2), and an NLR of 0.39 (95% CI of 0.02).

For a CO-RADS classification of 3 considered positive, we obtained an accuracy of 0.69 (95% CI of 0.3), a sensitivity of 67% (95% CI of 0.3), a specificity of 80% (95% CI of 0.3), a PPV of 0.92 (95% CI of 0.02), an NPV of 0.21 (95% CI of 0.03), a PLR of 0.83 (95% CI of 0.2), and an NLR of 0.42 (95% CI of 0.02).

For a CO-RADS classification of 4 considered positive, we obtained an accuracy of 0.82 (95% CI of 0.3), a sensitivity of 84% (95% CI of 0.3), a specificity of 79% 89 (95% CI of 0.3), a PPV of 0.84 (95% CI of 0.02), an NPV of 0.42 (95% CI of 0.03), a PLR of 1.06 (95% CI of 0.2), and an NLR of 0.19 (95% CI of 0.02).

For a CO-RADS classification of 5 considered positive, we obtained an accuracy of 0.84 (95% CI of 0.3), a sensitivity of 72% (95% CI of 0.3), a specificity of 91% (95% CI of 0.3), a PPV of 0.92 (95% CI of 0.02), an NPV of 0.41 (95% CI of 0.03), a PLR of 0.85 (95% CI of 0.2), and an NLR of 0.23 (95% CI of 0.02).

The other diagnostic performance indexes for each CO-RADS category are summarized in Table 2 . The ROC curve for a CO-RADS classification of >4 (4 + 5) is positive for COVID-19 is shown in Fig. 4 . The AUC obtained was of 0.89 (95% CI of 0.02).

Table 2

Diagnostic performance of CO-RADS 2, 3, 4, 5 and 4 + 5.

	TP	FP	TN	FN	S	E	PPV	NPV	PLR	NLR	DOR	Accuracy	Youden	UC+	UC−	p
CORADS 2	8	5	2	1	0.89	0.28	0.61	0.43	3.11	0.39	8	0.63	0.17	0.64	0.13	0.001
CORADS 3	50	4	16	25	0.67	0.80	0.92	0.21	0.83	0.42	2	0.69	0.47	0.61	0.17	0.001
CORADS 4	27	5	19	5	0.84	0.79	0.84	0.43	1.06	0.19	5,4	0.82	0.63	0.71	0.34	0.001
CORADS 5	78	5	75	30	0.72	0.91	0.94	0.42	0.77	0.29	2,6	0.81	0.66	0.68	0.39	0.001
CORADS 4 + 5	112	9	92	31	0.78	0.91	0.92	0.41	0.85	0.23	3.61	0.84	0.69	0.72	0.38	0.001

TP: true positive, FP: false positive, TN: true negative, FN: false negative, S: sensitivity, E: specificity, PPV: positive predictive value, NPV: negative predictive value, PLR: positive likelihood ratio, NLR: negative likelihood ratio, DOR: diagnostic odds ratio, UC+: Positive Clinical Utility Index, UC−: Negative Clinical Utility Index.

We obtained the following 95% confidence intervals: S and E: 0.3, PPV, NPV: 0.02, PLR of 0.2, NLR of 0.02, DOR of 1.0, Accuracy: 0.3, Youden: 0.2, UC+ and UC− 0.01.

Fig. 4

ROC curve evaluating the performance of CO-RADS >4 considered positive. The red dots are related to the several observations included. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

The PPTP obtained was 65% (95% CI of 3), 67% (95% CI of 3), 84% (95% CI of 5) and 94% (95% CI of 5) for CO-RADS 2, 3, 4 and 5 respectively. The NPTP obtained was 18% (95% CI of 10), 18% (95% CI of 10), 22% (95% CI of 10) and 28% (95% CI of 5) for CO-RADS 2, 3, 4 and 5 respectively. The nomograms for CO-RADS 2, 3, 4 and 5 considered positive are shown in Fig. 5 .

Fig. 5

Nomograms for the assessment of the positive (blue line) and negative (red lines) post-test probabilities for CO-RADS 2 (A), 3 (B), 4 (C) and 5 (D). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 2 summarizes all of the diagnostic performance values as well as the true positive, true negative, false positive and false negative results.

Discussion

This study analyzed the diagnostic performance of CO-RADS of a population in an ambidirectional manner, in other words, with retrospective and prospective branches associated in a blind and randomized manner. To the best of our knowledge, this was one of the validation studies for CO-RADS with the largest number of patients.8., 9., 18.

In general, patients with CO-RADS > 4 showed good agreement with the RT PCR indicated by the AUC of 0.89, similar to other studies as well as the sensitivity and specificity values, especially for CO-RADS 5 and 2 (16). Our values of specificity proved to be even superior to other multicenter studies, which may be indicative of a better adaptation of the use of CO-RADS throughout the pandemic.

However, the values of AUC, sensitivity and specificity are not very accurate because they do not take into account the prevalence of the disease in the population and are unable to relate to the pre-test probability of the findings and patients, which is why we conducted the other analyses.

To try to overcome these limitations, we used indicators recently described as of positive and negative clinical utility (>0.64 defined as of excellent). We found UC+ values that met this for CO-RADS 4 and 5. This means that, the tomographic findings related to CO-RADS 4 and 5 have value in considering a patient positive for COVID-19, even being rare or used in isolation, however, the absence of these values does not imply in the discard of the disease as indicated in the negative clinical utility value <0.5. This inference would prove to be hidden if we considered only average values such as the Youden's index for these CO-RADS, as indicated.13., 14

The effect of the prevalence of the disease in our population as well as the false negatives also impacted the negative and positive likelihood ratios of our study. However, when analyzing the DOR for each CO-RADS we observed that all of them presented a value above 1, which could infer that the classification of CO-RADS can also discern between patients with or without COVID-19 with statistical significance.

Another point to be considered refers to the values of PPTP indicated by the nomograms. Patients classified as CO-RADS>4 presented a PPTP of 84 and 94% indicating excellent probabilities of confirmation for COVID in these categories. In contrast, patients with CO-RADS 1 and 2 presented an NPTP of up to 20%, which indicates that trying to stratify patients according to CO-RADS is well founded.

Analyzing all the results retrospectively, it is clear that most of the negative results occurred in the initial period of the infection (7 days), generating a decrease in the specificity and negative clinical utility of CO-RADS. Even so, our specificity values are higher than other studies.

Patients who were classified as CO-RADS 3 and still had a low differentiation between the presence or absence of COVID-19 (lower DOR among the categories) due to overlap with other pathologies of similar tomographic aspect, such as pulmonary edema or influenza pneumonia, mainly in early stages.17., 18., 19., 20.

Throughout the course of the pandemic, several scores were proposed by the Radiological Society of North American and American Society of Thoracic Radiology with diagnostic performance indexes with an AUC ranging from 0.8 to 0.85. Our AUCs were significantly better especially through the refinement of specificity and considering only CO-RADS above 4 positive in the final analysis, which was proved significant in our analysis. Another explanation for CO-RADS to have a better diagnostic performance involves the fact that the RSNA Statement and several others have low sensitivity values since most viral pneumonias not related to Sars-Cov-2 have similar presentation especially in the early phases.

The main limitations of our study involved the retrospective portion mainly related to the lack of clinical data and the quality of the tests obtained at the beginning of the pandemic. We also had to deal with the CO-RADS learning curve, although fast, and even with the double Reading of exams, we could not rule out any incorrect classifications.

In Conclusion, CO-RADS proved to be useful for stratifying patients in emergency departments and screening suspect patients for COVID-19, allowing them to be separated from the non-COVID-19 patients, however low CO-RADS were not useful for ruling out infection in the early stages, and a high level of suspicion should be maintained in these patients if they have associated risk factors.

Ethics approval and consent to participate

This study was conducted after approval in the ethics committee of the Red Cross – Parana Chapter under the registration 007/2021.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Funding

The authors did not receive any funding from third parties or institutions for the conduction of this study.

CRediT authorship contribution statement

JGBG, GLOS, CLP, ASLB, ABB, IG, KS and VRF had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study Concept and Design: JGBH, CLP, ASLB, IG. Acquisition, analysis and interpretation of data: CLP, IG, GLOS, ABB, ASLB, KS. Drafting of the Manuscript: GLOS, JGBG, VRF. Critical revision of the manuscript for important intellectual content: JGBG, GLOS and VRF.

Declaration of competing interest

The authors declare that they have no competing interests.