True-Positive Rate of RSNA Typical Chest CT Findings for COVID-19 Pneumonia in an Increasingly Vaccinated Population.

Background RSNA COVID-19 chest CT consensus guidelines are widely used, but their true positive rate for COVID-19 pneumonia has not been assessed among vaccinated patients. Purpose To assess true positive rate of RSNA typical chest CT findings of COVID-19 among fully vaccinated subjects with PCR-confirmed COVID-19 infection compared with unvaccinated subjects. Materials and Methods Patients with COVID Typical chest CT findings and one positive or two negative PCR tests for COVID-19 within 7 days of their chest CT between January 2021 - January 2022 at a quaternary academic medical center were included. True positives were defined as chest CTs interpreted as COVID Typical and PCR-confirmed COVID-19 infection within 7 days. Logistic regression models were constructed to quantify the association between PCR results and vaccination status, vaccination status and COVID-19 variants, and vaccination status and months. Results 652 subjects (median age 59, [IQR, 48-72]); 371 [57%] men) with CT scans classified as COVID Typical were included. 483 (74%) were unvaccinated and 169 (26%) were fully vaccinated. The overall true positive rate of COVID Typical CTs was lower among vaccinated versus unvaccinated (70/169 [41%; 95% CI: 34, 49%] vs 352/483 [73%; 69, 77%]; OR (95% CI): 3.8 (2.6, 5.5); P < .001). Unvaccinated subjects were more likely to have true positive CTs compared with fully vaccinated subjects during the peaks of COVID-19 variants Alpha (OR, 16 [95% CI: 6.1, 42]; P < .001) and Delta (OR, 8.3 [95% CI: 4.2, 16]; P < .001), but no statistical differences were found during the peak of Omicron variant (OR, 1.7 [95% CI: 0.27, 11]; P = .56) Conclusion Fully vaccinated subjects with confirmed COVID-19 breakthrough infections had lower true positive rates of COVID Typical chest CT findings.

Summary

The true-positive rate of RSNA COVID-19 typical chest CT scans for PCR-confirmed COVID-19 infection was lower among vaccinated subjects compared with unvaccinated subjects except during periods of COVID-19 peaks.

■ In this retrospective study, true-positive rates of RSNA COVID-19 typical chest CT scans in 652 subjects were lower among fully vaccinated versus unvaccinated (41% vs 73%; P < .001).

■ Unvaccinated subjects were more likely than vaccinated subjects to have true-positive CT scans during peaks of Alpha (OR, 16; P < .001) and Delta (OR, 8.3; P < .001) variants.

Introduction

The COVID-19 pandemic continues to cause substantial morbidity and mortality worldwide despite the development of vaccines proven to be effective in reducing the risk of severe illness. This is in part due to heterogeneity of vaccination in the general population, waning vaccine immunity leading to breakthrough infections, and the rise of highly contagious variants of concern (VOC) such as Delta and Omicron.[1-5]

Nucleic acid amplification testing with the reverse transcription polymerase chain reaction (RT- PCR) has been the reference standard for detecting COVID-19 infections due to the high sensitivity and specificity. However, false negative rates have been reported from <5% to 40% depending on test type and specimen quality.[6,7] Radiology has played a central role in the pandemic, particularly chest CT, which has been used in diagnosis, triage, and outcomes assessment of patients presenting with COVID-19 pneumonia.[8] To aid in these efforts, the Radiological Society of North America (RSNA) released consensus guidelines to standardize reporting of chest CT findings related to COVID-19 pneumonia among patients with suspected or confirmed COVID-19 infection.[9] These guidelines classify chest CT findings of COVID- 19 pneumonia into four categories and have been widely adopted across radiology practices worldwide.[8]

The RSNA guidelines were developed before COVID-19 tests and vaccines were widely available. Since 2021, COVID-19 test results are often available before the radiologists interpret the CT. Our institution continues to use the RSNA COVID-19 classification system in patients with and without COVID-19 PCR results for triage and placement of potential COVID patients to prevent spread of the virus in the hospital setting. Although chest CTs remain a key component in the evaluation of patients to detect occult COVID infections given the false negative rate of PCR, the applicability of chest CT scan utilization and reporting guidelines among an increasingly vaccinated population has not been well characterized. Therefore, the purpose of this study was to assess whether vaccination influences the true positive rate of CT in diagnosis of COVID-19 pneumonia by evaluating the performance of RSNA COVID-19 Typical CT findings among patients with RT-PCR-confirmed COVID-19 infection.

Materials and Methods

Study setting

This HIPAA-compliant retrospective cohort study performed at a large, quaternary academic medical center was approved by the Institutional Review Board with a waiver of informed consent.

Study cohort

The institutional electronic database was queried for chest CT examinations performed between January 2021 and January 2022 that contained the classification of RSNA Typical Chest CT findings for COVID-19 pneumonia (COVID Typical) included in the radiology report. All COVID Typical results were identified through an institutional decision support reporting tool utilizing the RSNA COVID-19 consensus guidelines. This tool was used to standardize reporting of chest CT findings, assist in triage of patients with confirmed or suspected COVID-19 infection, and label each scan using one of four RSNA categories: Negative, Atypical, Indeterminate, and Typical.

Study inclusion criteria included: Patients with COVID Typical chest CT findings, as identified using the decision support reporting tool, and one positive or two negative PCR tests for COVID-19 within 7 days of their chest CT. For subjects who had two COVID Typical chest CTs from separate clinical encounters, only the earliest encounter was considered. Patients with only one negative test or no PCR test result for COVID-19 within 7 days of their CT and subjects with COVID-19 Typical on non-chest CT studies (i.e., abdominal CT) were excluded.

Data sources and independent variables

Demographic data, medical comorbidities, oxygen requirement (i.e., supplemental oxygen use, no supplemental oxygen use, and intubation), and clinical status (i.e., hospitalization, intensive care unit admission, emergency department visit, outpatient visit, and deceased status at time of analysis) were extracted from the electronic medical record (EMR; Epic Hyperspace, November 2021; Epic Systems Corporation). The following comorbidities were included based on CDC classification as risk factors for severe COVID-19: cancer history, chronic kidney disease, chronic obstructive pulmonary disease, diabetes, obesity, heart conditions (e.g., heart failure, arrhythmias, hypertension), and immunocompromised status.[10,11]

Vaccination status and presenting symptoms

Documented vaccination status was obtained through EMR review. Subjects were considered fully vaccinated if they were ≥14 days from a second mRNA-1273 (Moderna), BNT162b2 (Pfizer-BioNTech), or single Janssen (Johnson and Johnson) vaccine dose. Subjects were considered fully boosted if they were ≥14 days after their booster.

Reported symptoms up to a week before subjects’ presentations were obtained through EMR review. The most common presenting symptoms associated with COVID-19 included cough, shortness of breath, fatigue, chills, chest pain, gastrointestinal complaints, myalgias, cognitive change, and fever.[12]

COVID-19 status and true positives

Through EMR chart review, COVID-19 status was defined as having one positive (COVID positive infection) or two negative (COVID negative infection) RT-PCR tests for COVID-19 within 7 days of having a COVID Typical chest CT report. True positive CT scans were defined as COVID-19 PCR-positive scans with a COVID Typical chest CT report.

VOC Dominance

A VOC was considered dominant during a period when it represented >50% of all sequenced COVID-19 strains in Massachusetts. The VOCs (date ranges) include: Alpha (January 1, 2021 – July 2, 2021), Delta (July 3, 2021 – December 24, 2021), and Omicron (December 25, 2021 – Present).

Statistical analyses and power calculation

A random sample of eligible subjects was used for analysis. Given the previously reported institutional true positive rate of 85% among unvaccinated subjects, 400 unvaccinated and 200 vaccinated subjects were needed to detect at least a 10% decrease in the true positive rate between vaccination status groups when using a one-sided two-sample test of proportions, type-I error (α) of .05, and power (1-β) of .9.[8]

Descriptive summaries were computed for the entire sample. Continuous variables were summarized as the median and IQR (interquartile range) and categorical variables as frequencies (percent). Differences in the distributions of categorical/continuous variables by vaccination status were assessed using either the Chi-square/Fisher's exact test or the Wilcoxon test, respectively. Three logistic regression models were constructed to quantify the association between RT-PCR results and: 1) vaccination status, 2) vaccination status and the VOC peak at CT assessment, and 3) vaccination status and calendar month. We considered accounting for subject age, but after adjusting for VOC peak, the inclusion of age did not significantly improve the model fit and was omitted. A second definition of vaccination status and VOC peak was utilized to investigate the effect of timings of CT imaging and vaccination relative to VOC peak. Among vaccinated subjects, we recoded VOC peak to include the VOC at times of CT imaging and vaccination (e.g., Alpha Alpha denotes the subgroup that both received the vaccine and obtained a chest CT during the Alpha VOC). For all models, linear combinations of parameter estimates were computed to summarize true positive rates by vaccination status (percent, 95% confidence intervals [CI]) and time period (i.e., overall, VOC peak, calendar month) as well as vaccination status comparisons (odds ratios [OR], 95% CIs, p-values). All analyses were performed using R (version 4.2.0; R Foundation for Statistical Computing).

Results

Cohort characteristics

The initial database query resulted in 2069 results, of which 652 met inclusion criteria (Figure 1) with COVID Typical Chest CT findings (Figure 2). The median subject age was 59 years (IQR, 48–72); 57% (371/652) were men, 65% (422/652) were COVID-19 positive by RT-PCR, and 26% (169/652) were fully vaccinated against COVID-19 with a median time from vaccination to COVID Typical CT of 119 (IQR, 55–193 days) (Table 1). The most common presenting symptom (Table 1) was shortness of breath (72% [469/652]) and the most common comorbidity was hypertension (56% [364/652]). Most subjects were hospitalized (58% [377/652]) and used supplemental oxygen (60% [389/652]; Table 1).

Figure 1:

Flow chart of the study. PCR = polymerase chain reaction, RSNA = Radiological Society of North America.

Figure 2:

At the time of interpretation, all chest CT scans in this figure were classified as RSNA COVID- 19 Typical based on the RSNA consensus guidelines for classification of COVID-19 pneumonia. (A): Left: An axial chest CT pulmonary angiogram with contrast showing lung window of an unvaccinated 53- year-old male subject with a past medical history of hypertension and no smoking history at initial presentation during a period of COVID-19 Alpha variant predominance (February 2021). Right: An axial chest CT pulmonary angiogram with contrast showing lung window of a fully vaccinated 74-year-old female subject with a past medical history of hypertension, supraventricular tachycardia, and no smoking history at initial presentation during a period COVID-19 Alpha variant predominance (March 2021). (B): Left: An axial chest CT pulmonary angiogram with contrast showing lung window of an unvaccinated 43- year-old female subject with a past medical history of active smoking and obesity at initial presentation during a period of COVID-19 Delta variant predominance (August 2021). Right: An axial chest CT pulmonary angiogram of a fully vaccinated 80-year-old male subject with a past medical history of hypertension, type 2 diabetes, and nonsustained ventricular tachycardia at initial presentation during a period of COVID-19 Delta variant predominance (September 2021). (C): Left: An axial chest CT pulmonary angiogram with contrast showing lung window of an unvaccinated 58-year-old male subject with a past medical history of active smoking, type 2 diabetes, obesity, heart failure with reduced ejection fraction, and stage 5 chronic kidney disease at initial presentation during a period of COVID-19 Omicron variant predominance (January 2022). Right: An axial CT chest without contrast of a fully vaccinated 80- year-old male with a past medical history of heart transplant, stage 4 chronic kidney disease, hypertension, chronic obstructive pulmonary disease, and complete heart block status post permanent pacemaker at initial presentation during a period of COVID-19 Omicron variant predominance (January 2022).

Table 1:

Characteristics of Subjects with COVID-19 Typical CT Findings

Among fully vaccinated subjects, the majority were vaccinated with BNT162b2 (Pfizer-BioNTech) (62% [104/169]), while fewer were vaccinated with mRNA-1273 (Moderna) (25% [42/169]), and Janssen (Johnson & Johnson) (14% [23/169]). Compared with unvaccinated subjects, fully vaccinated subjects were significantly older (median age 69 years [IQR, 56–79] versus median age 56 years [IQR, 46–68]; P < .001), had higher rates of supplemental oxygen use (67% [113/169] versus 57% [276/483]; P = .03), and higher hospitalization rates (68% [115/169] versus 54% [262/483]; P = .002). Compared with unvaccinated subjects, fully vaccinated subjects had significantly higher rates of many comorbidities associated with COVID-19 disease severity[10,11] (Table E1). Similarly, true positive fully vaccinated subjects compared with true positive unvaccinated subjects retained higher rates of most of these comorbidities and additionally were more likely to be immunocompromised (39% [27/70] versus 11% [38/352]; P < .001; Table E2). Fully vaccinated subjects in the true positive group also had longer time from vaccination to COVID-19 Typical CT scan (median 175 days [IQR, 122–220]) compared with PCR-negative subjects (median 70 days [IQR, 40–150]; P < .001; Table 1). The most common final diagnoses of PCR-negative subjects from clinical encounters corresponding to their COVID-19 Typical CTs were pulmonary edema (23% [70/311]), interstitial lung disease (13% [40/311]), and aspiration (13% [39/311]) (Figure E1).

Table E1:

Fully Vaccinated vs Unvaccinated Subjects with RSNA COVID-19 Typical CT Scans

Table E2:

Fully Vaccinated vs Unvaccinated Subjects with True-Positive COVID-19 CT Scans

Figure E1:

Alternative diagnoses in COVID-19 PCR-negative subjects with RSNA COVID-19 Typical CT findings. Categories assigned based on microbiological laboratory evidence included “Community acquired pneumonia”, “Hospital acquired pneumonia”, and “Viral or atypical pneumonia”. Categories assigned based on clinical suspicion with or without laboratory evidence included “Aspiration pneumonia/pneumonitis”, “Unconfirmed/suspected pneumonia”, “Post-COVID lung findings”, “Pulmonary edema” and “Interstitial lung disease”. PCR = polymerase chain reaction, RSNA = Radiological Society of North America.

*The diagnoses in the “Other” category and associated frequencies included bronchiectasis x2, lymphangitic carcinomatosis x2, ANCA-associated vasculitis x2, diffuse alveolar hemorrhage x2, hepatic hydrothorax x2, cytokine release syndrome x1, pulmonary hemorrhage x1, loculated pleural effusion x1, pulmonary lymphoma x1, and differentiation syndrome x1.

COVID-19 true positive subjects

Overall, the COVID Typical true positive rate (Table 2) was higher among unvaccinated subjects [73% (352/483)] compared with fully vaccinated subjects [41% (70/169); OR: 3.8 (2.6, 5.5); P < .001]. True positive rates also varied with new VOC. Compared with fully vaccinated subjects, unvaccinated subjects had higher true positive rates during Alpha variant predominance [67% (62, 72) versus 11% (5,); OR: 16 (6.1, 42); P < .001] and Delta variant predominance [88% (81, 93) versus 47% (38, 57); OR: 8.3 (4.2, 16); P < .001], while differences in true positive rates were not detected between vaccination status groups during Omicron variant predominance [87% (60, 97) versus 79% (55, 92); OR: 1.7 (0.27, 11); P = 0.56].

Table 2:

True-Positive Rate by Vaccination Status and by Vaccination Status and Calendar Period/Month

In a monthly comparison of the true positive rate of COVID Typical chest CT (Table 2), unvaccinated subjects were significantly more likely to have true positive CT scans between April 2021 (OR, 20; 95% CI: 4.1, 94; P < .001) and July 2021 (OR, 11; 95% CI: 1.1, 106; P = .04). Between April 1, 2021 – July 1, 2021, the vaccination rate in Massachusetts rose from 20% to 60%, while COVID-19 cases decreased from 4750 to 61 cases between January 5 – July 1, 2021 (Figure 3 A-B). The months with no significant differences in true positive rates between fully vaccinated and unvaccinated subjects coincided with the rise of dominant VOC in August – October 2021 (Delta), and January 2022 (Omicron; Table 2).

Figure 3:

(A): Monthly absolute number of fully vaccinated and unvaccinated subjects with RSNA COVID-19 Typical CT scans between January 1, 2021, and January 31, 2022. Grey line represents the proportion of Massachusetts population fully vaccinated against COVID-19, with line generated from datapoints corresponding to the first day of each month.[29,30] The vaccination rates used for February 2021 and January 2021 corresponded to February 3, 2021, and January 5, 2021, because data closer the first days of those months were not available. (B): Proportion positive for COVID-19 of fully vaccinated and unvaccinated subjects with RSNA COVID-19 Typical CT scans between January 1, 2021, and January 31, 2022. Grey line represents incident PCR-confirmed COVID-19 cases in Massachusetts at the beginning of each month, with line generated using 7-day averages containing the first day of each month as proxies for incident cases.[30] PCR = polymerase chain reaction, RSNA = Radiological Society of North America.

A secondary analysis explored the relationship between the true positive rate based on predominant VOC and time from vaccination to true positive COVID Typical CT (Figure 3). The odds of having a true positive chest CT during Delta predominance in unvaccinated subjects was higher than in subjects who were either vaccinated during the period of Delta predominance (OR 22, 95% CI: 7.6, 66) or vaccinated during Alpha predominance (OR, 6.4; 95% CI: 3.2, 13). Fully vaccinated subjects who were vaccinated during the period of Alpha predominance were more likely to have true positive CT scans when imaged during a period of Delta predominance (OR, 9; 95% CI: 3.2, 25) or Omicron predominance (OR, 39; 95% CI: 3.8, 405; estimated from values in Table 3) when compared with those who were also imaged during Alpha predominance.

Table 3:

True-Positive Rate by Vaccination Status and Dominant COVID-19 Variant of Concern (VOC) Identified in Massachusetts and by Time from Vaccination to CT

Exploratory analyses of true positive rates stratifying by booster status in November 2021 – January 2022 was performed (Table E3). Unvaccinated subjects were more likely to have true positive CT scans compared with fully vaccinated subjects in November (OR, 6.8; 95% CI: 1.1, 43; P = .043) and December (OR, 13; 95% CI: 2.3, 69; P = .003), and even more likely to have true positive CT scans compared with fully boosted subjects during November (OR, 95; 95% CI: 7.7, 1180; P < .001) and December (fully boosted 0 of 6 [0%; 95% CI: 0%, 39%]).

Table E3:

True-Positive Rate by Booster Status and Calendar Month

Discussion

The RSNA consensus guidelines, adopted by many institutions, have provided a standardized framework for interpreting and reporting CT findings associated with COVID-19 pneumonia.[8,9] This study showed that among an increasingly vaccinated population, these guidelines continue to provide valuable information for triage and disposition of patients being evaluated for suspected or confirmed COVID-19 infections, as unvaccinated subjects had a higher overall true positive rate of COVID Typical Chest CTs compared with fully vaccinated subjects (352/483 [73%; 69, 77%] versus 70/169 [41%; 95% CI: 34, 49%]; OR (95% CI): 3.8 (2.6, 5.5), P < .001). Additionally, the overall true positive rate of 65% among all subjects with RSNA COVID Typical CT during the study period at our institution represents a substantial decrease from the true positive rate of 85% during the initial peak of the pandemic in 2020, likely due to the decrease in disease severity among the vaccinated.[8]

Breakthrough infections among vaccinated patients have been reported to have distinct imaging characteristics compared with those of unvaccinated patients, including higher rates of CTs negative for pneumonia or with mild severity scores.[17-19] In one study, the majority of subjects were immunosuppressed and all but one subject with immunosuppression (80%) had abnormalities on imaging.[18] Our study was concordant with previous literature, showing that fully vaccinated subjects were less likely to have true positive COVID Typical Chest CT and more likely to be immunocompromised compared with true positive unvaccinated subjects.[18,20] The increasing vaccination rates and evolving data may result in a decreasing proportion of patients presenting with true positive COVID Typical Chest CTs and may result in an update in RSNA COVID-19 reporting guidelines.

True positive rates, like positive predictive values, are influenced by disease prevalence in the population and are expected to fluctuate with factors that decrease or increase COVID-19 prevalence, such as vaccination rates or increased transmissibility of COVID variants.[21,22] For example, the Delta variant arose as the dominant-state VOC in mid-2021 and was more transmissible than Alpha.[4] This rise in the Delta VOC coincided with the increasing true positive rate among fully vaccinated subjects between July and September 2021.[13] Omicron became the dominant-state VOC on December 25, 2021, and its mutations caused increased transmissibility, leading to more breakthrough cases among vaccinated patients.[14-16] This may explain why there was no difference in the likelihood of true positive Typical Chest CT between fully vaccinated and unvaccinated subjects during January 2022.

The monthly true positive rate among fully vaccinated subjects was likely influenced by the time from vaccination, with true positive fully vaccinated subjects having longer time from vaccination to CT scan compared with fully vaccinated subjects with COVID Typical CTs and negative PCR. The subjects in this study were more likely to have true positive CTs with longer intervals between vaccination and CT, suggesting an attenuation in vaccine effectiveness with time. Recent studies have shown that the effect of time from receipt of vaccination and booster was noted to be particularly striking with Omicron, with vaccine efficacy against symptomatic disease after two doses of BNT162b2 reported to be 65% at 2–4 weeks post-vaccine, dropping to 9% after 25 or more weeks; similar waning immunity has been seen with other vaccine and VOC types.[23-25] This study also showed that unvaccinated subjects were even more likely to have a true positive scan compared with fully vaccinated subjects who received a booster, thus underscoring the role of vaccine boosters in COVID-19 severity.[26-28]

This study has several limitations. Although prior studies have shown concordance among radiologists interpreting chest CTs using the RSNA guidelines, this study is limited by the lack of a formal over-read of the included studies.[8] COVID-19 variant information was not available at the patient level, limiting our ability to assess the contribution of different VOC to the observed true positive rates. The greater availability and quicker processing of COVID PCR tests may have influenced interpretation, but this was not captured in the study. The decreased severity of breakthrough infections noted in recent literature may have influenced the true positive rate.[17-19]

In summary, the overall true positive rate of COVID Typical chest CTs was lower in fully vaccinated subjects compared with unvaccinated subjects. Our study and previous literature have shown that several mechanisms likely contribute to the lower true positive rate observed among fully vaccinated subjects, including proportion of population vaccinated. Thus, the utility of CT in detecting COVID-19 pneumonia among a vaccinated population should be considered in this context. While fully vaccinated subjects with confirmed COVID-19 infections had significantly lower true positive rates of COVID Typical chest CT findings in the study, Chest CT and the RSNA consensus guidelines maintain a key role in assisting radiologists and referring clinicians in the triage and evaluation of patients with suspected or confirmed COVID-19 infection among an increasingly vaccinated patient population.