Determinants of Chest X-Ray Sensitivity for COVID- 19: A Multi-Institutional Study in the United States.

PURPOSE: To evaluate the sensitivity, specificity, and severity of chest x-rays (CXR) and chest CTs over time in confirmed COVID-19+ and COVID-19- patients and to evaluate determinants of false negatives.
METHODS: In a retrospective multi-institutional study, 254 RT-PCR verified COVID-19+ patients with at least one CXR or chest CT were compared with 254 age- and gender-matched COVID-19- controls. CXR severity, sensitivity, and specificity were determined with respect to time after onset of symptoms; sensitivity and specificity for chest CTs without time stratification. Performance of serial CXRs against CTs was determined by comparing area under the receiver operating characteristic curves (AUC). A multivariable logistic regression analysis was performed to assess factors related to false negative CXR.
RESULTS: COVID-19+ CXR severity and sensitivity increased with time (from sensitivity of 55% at ≤2 days to 79% at >11 days; p<0.001 for trends of both severity and sensitivity) whereas CXR specificity decreased over time (from 83% to 70%, p=0.02). Serial CXR demonstrated increase in AUC (first CXR AUC=0.79, second CXR=0.87, p=0.02), and second CXR approached the accuracy of CT (AUC=0.92, p=0.11). COVID-19 sensitivity of first CXR, second CXR, and CT was 73%, 83%, and 88%, whereas specificity was 80%, 73%, and 77%, respectively. Normal and mild severity CXR findings were the largest factor behind false-negative CXRs (40% normal and 87% combined normal/mild). Young age and African-American ethnicity increased false negative rates.
CONCLUSION: CXR sensitivity in COVID-19 detection increases with time, and serial CXRs of COVID-19+ patients has accuracy approaching that of chest CT. 2020 by the Radiological Society of North America, Inc.

Summary

Serial chest radiography sensitivity for detection of novel 2019 coronavirus infection approaches chest CT, suggesting utility of chest radiography as an adjunctive diagnostic tool for COVID-19 patients.

● Sensitivity of CXR for COVID-19 increases from 55% at ≤2 days to 79% at >11 days after symptom onset.

● Accuracy of diagnosis of COVID-19 in serial CXR approaches that of chest CT.

● Normal or mild severity CXRs were the main determinants of false negative CXR interpretations.

Introduction

In December 2019, the first outbreak of the novel 2019 coronavirus disease (COVID-19) was reported in China in Wuhan City, Hubei Province(1). It rapidly achieved pandemic status, and the United States currently has the highest number of cases in the world. In the US, the medical community has been primarily focused on controlling viral spread through mitigation measures(2). Until favorable developments occur from drug discovery and vaccine development efforts, rapid and sensitive detection of COVID-19 infection is essential for triage of patients, both for timely isolation of COVID-19+ patients and for appropriate allocation of hospital resources and providers in hospital units with limited capacity.

The gold standard for diagnostic confirmation of COVID-19 infection in patients is a positive result from reverse transcriptase polymerase chain reaction (RT-PCR), but the time-to-result from labs (which may be as long as several days) has been a rate-limiting step in hospital care. As a result, imaging-based diagnostic tools are attractive as a companion option, with some reports suggesting high sensitivity of COVID-19 detection by computed tomography (CT) studies(3,4). However, questions have been raised about the reported high sensitivity of CT in those studies, which likely suffer from selection bias toward more severe cases(5). Some early reports have also described high specificity in diagnosis of COVID-19 compared to other entities, but these too may be biased by selection of more severe cases and, at least in one case, an artificial control group of other viral pneumonias(6,7). In addition, the ability to detect COVID-19 pneumonia on imaging likely varies over time but remains to be fully elucidated(8).

Here, we evaluate the imaging characteristics, sensitivity, and specificity of both chest X-ray (CXR) and chest CT in detecting COVID-19 infection in patients with lab-confirmed SARS-CoV-2 viral infection over time. We also evaluate the contributions of patient demographics and comorbidities that influenced false-negative imaging.

Methods

Patient selection

This retrospective study was conducted at two large urban academic medical centers, including three tertiary care hospitals and one community hospital. The respective Institutional Review Boards at the two academic medical centers approved the study, with waiver of informed consent. Inclusion criteria were patients who had both at least 1 nucleic acid amplification-based COVID-19 detection test and at least 1 CXR or CT that was performed within a week of testing. Patients younger than 18 years of age were excluded. A patient was regarded as COVID-19 positive if they had any positive test within the time interval. At the first institution, 182 patients positive for COVID-19 infection from March 1, 2020 through April 1, 2020 were randomly selected and then paired with age- and gender-matched control patients who tested COVID-19 negative. At the second institution, 72 positive for COVID-19 infection between March 1, 2020 through April 14, 2020 and age- and gender-matched control patients were then included in a similar process.

Image review

At each site, two board-certified, fellowship-trained thoracic radiologists reviewed each study separately. At the first institution, images were reviewed by two of the thoracic radiologists (MMH, ARH, FLJ, HH, and SCB who had between 5 and 29 years of experience). At the second institution, RAH with 4 years of experience and CW with 29 years of experience reviewed the studies. Reviewers were blinded to the diagnosis.

Chest X-Ray evaluation

Reviewers evaluated for the predominant pattern in chest radiographs, selecting from: interstitial opacities, interstitial and airspace opacities, atelectasis, diffuse airspace opacities, lobar consolidation, or peripheral opacities; if none of these patterns applied the reviewers would select ‘normal’. Reviewers also assessed for a craniocaudal gradient and the presence of pleural effusions. Reviewers were asked to assign a severity score for the chest x-ray findings: normal, mild, moderate, or severe. Reviewers also assessed the likelihood of COVID-19 based on the radiograph findings, with a score between 1 and 5 (1 being “Very unlikely,” 3 being “Intermediate likelihood,” and 5 being “Highly Likely”). Up to 3 serial chest radiographs were scored per patient.

Chest-CT evaluation

Reviewers were asked to look for the presence of nodules, ground glass opacities, consolidation, and septal thickening. The presence of pleural effusions and reverse halo sign were also assessed. Additionally, reviewers evaluated craniocaudal gradient, anterior/posterior gradient, and central/peripheral gradient; reviewers could score cases as having a gradient or diffuse involvement. The reviewers were asked to assign a chest CT severity score and a chest CT COVID-19 likelihood score.

Patient Demographics

Patient’s gender, age, ethnicity, height, and admission weight were collected. Patient history was reviewed for positive history for hypertension, diabetes mellitus, cardiac disease or anomaly, chronic lung disease, autoimmune disease, malignancy, stem cell transplant, solid organ transplant, or chronic kidney disease.

Data Entry and Statistics

Data was entered using the Castor EDC platform(9) and the REDCap platform(10). Graphpad Prism 5.0 and Graphpad Prism 8.4.2 (Graphpad Software, San Diego, CA) as well as JMP (v15, SAS Institute, Cary, NC) statistical software packages were used for formal analysis of data.

In order to combine data from the two reviewers, for craniocaudal gradient, anterior/posterior gradient, and central/peripheral gradient, diffuse distribution was assigned to a case if 1) at least 1 reviewer scored diffuse or 2) if both reviewers selected opposing categories (for example, if Reviewer 1 selected anterior distribution and Reviewer 2 selected posterior distribution). CXRs and chest CTs were characterized as having normal patterns only if both reviewers characterized it as normal. If the two reviewers selected different abnormal patterns, both patterns were scored for purposes of analysis; if one reviewer selected an abnormal pattern while the other selected ‘normal’, then only the abnormal pattern was scored. For severity and COVID-19 likelihood scores, the scores were averaged for the two reviewers.

For purposes of sensitivity and specificity, a CXR or CT scored as 3 (intermediate COVID-19 likelihood) or above in the COVID-19 likelihood score was classified as a ‘positive’ test.

The time intervals between symptom onset and chest radiographs were evaluated, and the quartiles were used to generate time interval bins for analysis of CXR and CT changes over time. Comparisons of categorical variables were done with Fisher’s exact test, one-tailed, for 2x2 tables or the chi-squared test for variables with multiple levels. Trend analysis of categorical variables across time interval bins was done using the Cochran-Armitage Test. Comparison of continuous variables across time interval bins was done with ANOVA. Multivariable logistic regression analysis was performed to evaluate factors associated with false negative chest x-rays. A p-value threshold of 0.05 was used for statistical significance. To adjust for multiple comparisons, we applied the Benjamini-Hochberg correction to the p-values for each table of univariable analyses, using a false-discovery rate of 0.05, and we named the p-values “Adjusted p-value” in these instances.

Confidence intervals and standard errors of the areas under the receiver operating characteristic curves (AUC) were obtained using bootstrapping with 2500 samples. The comparison of paired AUCs was performed using a previously described method(11).

Results

Patient characteristics

A total of 508 patients were included, 254 of whom were COVID-19+ with 254 matched COVID-19- controls. The demographics and clinical characteristics of the cohort are given in Table 1. 500 (98%) of patients had at least 1 CXR, 195 (38%) had at least 2 CXRs, 112 (22%) had 3 CXRs, and 169 (33%) had at least one chest CT. CXR were obtained at the following time intervals after onset of symptoms: 254 (32%) at ≤2 days, 192 (24%) at 3-6 days, 183 (23%) at 7-11 days, and 177 (22%) at >11 days. The median time from symptoms to CT was 4 days. The available clinical indications for CXR and CT between COVID-19 positive and negative patients are listed in Supplemental Tables 1-4.

Table 1:

Patient demographics

CXR evolution over time

The severity of CXR scores increased over time for COVID-19+ patients (from mean of 1.11 at ≤2 days to mean of 1.97 at >11 days, p<0.001, Figure 1A). However, the severity scores for COVID-19- patients did not change significantly over time (from mean of 0.93 at ≤2 days to mean of 1.10 at >11 days, p=0.21.

Figure 1a.

Evolution of COVID-19 chest X-ray severity, findings, and distribution after onset of symptoms. (a) Chest X-ray severity scores in COVID-19 infected patients (red) and COVID-19 negative (blue) patients after onset of symptoms. Graph represents averages from patients in each time interval ± standard error. (b) Frequencies of COVID-19+ chest X-rays positive for diffuse airspace opacities (DA) pattern, interstitial and airspace opacities pattern (IA) alone, or neither, in each time interval. (c) Frequencies of COVID-19+ chest X-rays positive for diffuse parenchymal involvement, lower lobe predominance, upper lobe predominance, or distribution characteristic of a normal chest radiograph, in each time interval.

The most common CXR finding in COVID-19+ patients was interstitial and airspace opacities (IA pattern), present in 62% of cases. IA pattern increased over time, present in 51% at ≤2 days versus 73% at >11 days (p<0.001 for trend, Figure 1B). CXR disease distribution also changed over time, with ncreasing prevalence of diffuse opacities (3% at ≤2 days vs 25% at >11 days, p<0.001 for trend, Figure 1C). Examples of evolving CXR in COVID-19+ patients are given in Figure 2.

Figure 1b.

Evolution of COVID-19 chest X-ray severity, findings, and distribution after onset of symptoms. (a) Chest X-ray severity scores in COVID-19 infected patients (red) and COVID-19 negative (blue) patients after onset of symptoms. Graph represents averages from patients in each time interval ± standard error. (b) Frequencies of COVID-19+ chest X-rays positive for diffuse airspace opacities (DA) pattern, interstitial and airspace opacities pattern (IA) alone, or neither, in each time interval. (c) Frequencies of COVID-19+ chest X-rays positive for diffuse parenchymal involvement, lower lobe predominance, upper lobe predominance, or distribution characteristic of a normal chest radiograph, in each time interval.

Figure 1c.

Evolution of COVID-19 chest X-ray severity, findings, and distribution after onset of symptoms. (a) Chest X-ray severity scores in COVID-19 infected patients (red) and COVID-19 negative (blue) patients after onset of symptoms. Graph represents averages from patients in each time interval ± standard error. (b) Frequencies of COVID-19+ chest X-rays positive for diffuse airspace opacities (DA) pattern, interstitial and airspace opacities pattern (IA) alone, or neither, in each time interval. (c) Frequencies of COVID-19+ chest X-rays positive for diffuse parenchymal involvement, lower lobe predominance, upper lobe predominance, or distribution characteristic of a normal chest radiograph, in each time interval.

Figure 2.

Representative serial chest X-ray imaging of COVID-19 infected patients. (a) 29-year-old man with rapid respiratory deterioration after symptom onset showing progression from lower lung predominant interstitial and airspace opacities on day 3 to diffuse involvement with extensive airspace disease on days 4 and 6. (b) 56-year-old-man with COVID-19, presenting initially with normal CXR, which then progressed to lower lung predominant interstitial and airspace opacities at day 9, which mildly worsened by day 15.

Sensitivity of imaging over time

The rise of CXR severity over the course of COVID-19 infection was reflected in the sensitivity CXR COVID-19 detection over time (Figure 3A). The sensitivity of CXR at ≤2 days is 55%, increasing to 79% at >11 days (p<0.001 for trend). Specificity declined slightly over time, 83% at ≤2 days to 70% at >11 days (p=0.02 for trend).

Figure 3a.

Sensitivity and specificity of chest X-ray and chest CT in COVID-19 infected patients. (a) Sensitivity and specificity by days after symptom onset. (b) Receiver operating characteristic curves for the 1st chest x-ray (CXR1), 2nd chest x-ray (CXR2), and chest-CT in patients with all three studies (n = 85). Areas under the curve (AUC) of CXR1 = 0.787, AUC of CXR2 = 0.875, and AUC of CT1 = 0.916.

We evaluated sequential imaging tests per patient for diagnosis of COVID-19. The first chest radiographs (CXR1) were obtained a median of 3 days after symptoms, the second chest radiographs (CXR2) were obtained a median of 8 days after symptoms, and CTs were obtained a median of 4 days after symptoms. In patients with both CXR1 and CXR2 (n=195), areas under the receiver operating characteristic curve (AUC) were 0.80 for CXR1 and 0.871 for CXR2. In patients who had both CXR1 and a CT study performed (n=160), CXR1 AUC was 0.80 and CT AUC was 0.90. In patients who had CXR1, CXR2, and CT (n=85), AUC was 0.79 (95% CI 0.67 – 0.88) for CXR1, 0.88 (95% CI 0.79 – 0.93) for CXR2, and 0.92 (95% CI 0.84 – 0.97) for CT (Figure 3B). CXR2 and CT AUCs were both significantly higher than the CXR1 AUC (p=0.02 and p=0.003, respectively), whereas comparison of CXR2 and CT AUCs found no significant difference (p=0.11). In this group, CXR1 had a sensitivity of 73% and specificity of 80%; CXR2 had sensitivity of 83% and specificity of 73%; and CT had sensitivity of 88% and specificity of 77%.

Figure 3b.

Sensitivity and specificity of chest X-ray and chest CT in COVID-19 infected patients. (a) Sensitivity and specificity by days after symptom onset. (b) Receiver operating characteristic curves for the 1st chest x-ray (CXR1), 2nd chest x-ray (CXR2), and chest-CT in patients with all three studies (n = 85). Areas under the curve (AUC) of CXR1 = 0.787, AUC of CXR2 = 0.875, and AUC of CT1 = 0.916.

Specific CXR1 findings for COVID-19

Frequencies of CXR1 findings in COVID-19 cases and controls are shown in Table 2 (with frequencies from total CXR shown in Supplemental Table 5). As noted above the most common CXR1 finding in COVID-19 was the IA pattern, present in 51% of COVID-19+ cases, versus 26% of COVID-19- cases (p<0.001). Peripheral airspace opacities and diffuse airspace disease were also more common in COVID-19+ cases than controls (21% versus 4%, p<0.001, and 6% versus 2%, p = 0.03). Conversely, COVID-19+ CXR1s demonstrated a reduced frequency of atelectasis, lobar consolidation, and pleural effusions compared to COVID-19- CXR1s (p=<0.001, p=0.03, p<0.001, respectively).

Table 2:

Frequency of CXR1 findings in COVID versus non-COVID patients

Specific CT findings for COVID-19

Frequencies of single CT findings in COVID-19+ cases and controls are given in Table 3. Diffuse involvement across craniocaudal and anterior/posterior spatial distributions of chest-CT findings predominated in COVID-19+ patients relative to COVID-19- patients (58% versus 28%, p<0.001, and 69% versus 47%, p=0.01, respectively), as did ground glass opacities (88% versus 49%, p<0.001). Additionally, the reverse halo sign was significantly more prevalent in COVID-19+ patients (33% in COVID-19+ cases versus 5% COVID-19- cases, p<0.001). Among these COVID-19+ characteristics in total CTs, only the frequency of consolidation significantly changed over the course of COVID-19 infection (Figure 4), increasing from 32% at ≤ 2 days to 91% at >11 days (p<0.001 for trend).

Table 3:

Frequency of single CT Findings in COVID versus non-COVID patients

Figure 4.

Frequencies of COVID-19+ chest-CT studies with consolidation, ground glass opacities (GGO) alone, or neither, across each time interval.

Features associated with false negative imaging.

There were 113 false negative (FN) initial CXRs and 23 FN second CXRs in COVID-19+ patients. Normal CXRs comprised 51 (45%), 3 (14%), and 54 (40%) of FN initial CXRs, second CXRs, and the summed initial and second CXRs. The combined normal and mild severity FN CXRs for these three respective groups were 106 (94%), 9 (39%), and 118 (88%). Of the 113 patients with FN CXR1 studies, 35 had a CXR2, of which 16/35 were positive. Analysis of patient demographics and comorbidities initially identified female sex, chronic lung disease, absence of hypertension, ethnicity, and young age to be associated with FN initial CXR (Table 4). A multivariable logistic regression analysis showed young age and African-American ethnicity increased FN rates whereas Hispanic-American ethnicity decreased FN rates.

Table 4:

Patient demographics and Comorbidities that influence CXR False Negative Rates

Of 10 FN CTs, 2 were normal and 7 (70%) were either normal or mild severity. Female sex (OR 5.44, p=0.03) and chronic lung disease (OR 5.56, p=0.02) also contributed to FN chest-CT studies.

Discussion

In summary, we evaluated the imaging characteristics, evolution of severity, sensitivity, and specificity of Chest X-Rays (CXRs) and chest CT in lab-verified novel 2019 coronavirus (COVID-19) positive and matched negative patients from two academic medical centers in two major US cities. Our results indicate that the sensitivity of CXR imaging increases over the course of COVID-19 infection, from as low as 55% at ≤2 days to 79% at >11 days after symptom onset, and serial CXR imaging approaches the diagnostic accuracy of chest CT (AUC of 0.875 versus 0.916). Normal imaging was the most prevalent factor underlying false negative (FN) CXRs and chest CTs. Patient characteristics including young age and certain ethnicities were associated with greater rates of FN exams. Finally, we found a number of specific imaging findings in CXR and CT that were helpful in the diagnosis of COVID-19, including interstitial and airspace opacity pattern (OR 2.84) and peripheral airspace opacities (OR 7.94) on CXR, and reverse halo sign on chest-CT (OR 9.36).

We showed that the severity and sensitivity of COVID-19+ CXR findings increases over time, with sensitivity markedly increasing after Day 6 following symptom onset. This coincides with the known time course of pathophysiologic deterioration in patients, as the time from illness onset to dyspnea, sepsis, and acute respiratory distress syndrome in COVID-19+ patients has been reported as 7 days, 9 days, and 12 days, respectively(12). A previous study analyzed baseline CXRs from 64 COVID-19+ patients and found a sensitivity of 44/64 (69%) in COVID-19 detection, comparable to our results(13). In our study, we stratified CXRs based on time since the onset of symptoms and were able to identify when imaging is expected to be most useful. Due to inadequate number of CTs in our patient population, we could not perform a similar analysis for CT sensitivity.

From our study population, we determined that the overall sensitivity of chest-CT for COVID-19 infection was 88%, which is comparable to 3 other studies reporting 86%, 86%, and 88% but lower than studies reporting sensitivities as high as 100%(1,3,4,14–16). Of note, our article specifically addressed diagnostic accuracy within a relevant clinical scenario (distinguishing COVID-19+ from COVID-19- patients), rather than assessing sensitivity alone without a control group or with artificially generated control groups.

Our finding that serial CXR imaging approaches the accuracy of CT imaging in COVID-19 has potential value to enhancing bio-containment and workflow in hospitals. The American College of Radiology has recommended decontamination of CT-imaging rooms after each use for COVID-19 evaluation to prevent viral transmission from CT scanners(17). In addition to the significant cleaning duties and personal protective equipment donning/doffing required of radiology technicians and janitorial staff, the process of transporting inpatients to a commonly utilized CT unit accumulates risk for viral transmission to other patients and healthcare staff. These concerns potentially limit widespread adoption of CT imaging for COVID-19 detection. Conversely, CXRs uses less radiation and portable CXR machines can be brought to patient bedsides to perform CXRs and require far less time to decontaminate, making it an attractive option to employ.

The majority of false negative studies were caused by CXRs or CTs characterized as normal, which likely results from imaging taken during early viral infection when infectious changes are not yet visible in lung parenchyma. We observed that older COVID-19+ patients had a higher frequency of true positive CXRs, which corresponds to reports that COVID-19 has greater severity in older patients(12,18). We noted increased FN rates in African Americans and a decreased FN rate in Hispanic-Americans, relative to Caucasian-Americans. This may be related to differential testing in these minority populations, with some patients having access to testing for more mild cases and others not; alternatively, there may be true biological differences (or different rates of comorbidities) that contribute to different CXR finding severities. Regardless, these findings may have implications for equity of care across minority populations.

Our study has several limitations, including its retrospective nature, as there was not a consistent clinical or imaging algorithm for these patients. Other limitations lie in patient selection, specifically in that the reasons for patient imaging were unknown and that serial chest X-ray imaging and CT imaging is likely biased towards patients with more severe disease. However, the ROC comparisons performed in this study were done among patients who had the same set and number of imaging tests, so those patients should be comparable to each other.

In conclusion, the results from our study imply that strategic serial CXR imaging can detect COVID-19 infection in symptomatic patients and may be a helpful adjunct to RT-PCR testing. In an inpatient hospital setting in which testing kit availability, laboratory resources, and/or laboratory staff are compromised and threaten delay to patient care and hospital workflow, serial CXRs could potentially be utilized in an adjunct diagnostic role in patients with suspected COVID-19.