Structured reporting in portable chest radiographs: An essential tool in the diagnosis of COVID-19.

PURPOSE: To date, the majority of chest imaging studies in COVID-19 pneumonia have focused on CT. Evidence for the utility of chest radiographs (CXRs) in this population is less robust. Our objectives were to develop a systematic approach for reporting likelihood of COVID-19 pneumonia on CXRs, to measure the interobserver variability of this approach and to evaluate the diagnostic performance of CXRs compared to real-time reverse transcription polymerase chain reaction (RT-PCR).
METHOD: Retrospective review of patients suspected of having COVID-19 pneumonia who attended our emergency department and underwent both CXR and a RT-PCR were included. Two radiologists reviewed the CXRs, blind to the RT-PCR, and classified them according to a structured reporting template with five categories (Characteristic, High Suspicion, Indeterminate, Unlikely and Normal) which we devised. For analysis of diagnostic accuracy, Characteristic and High Suspicion CXRs were considered positive and the remaining categories negative. Concordance between the two assessors was also measured.
RESULTS: Of 582 patients (51 +/- 20 years), 143/582 (24.6 %) had a positive RT-PCR. The absolute concordance between the two assessors was 71.1 % (414/582) with a Fleiss-Cohen-weighted Cohen's κ of 0.81 (95 % confidence interval, 0.78-0.85). A patient with a positive CXR had an 88 % (95 % CI 80-96 %) probability of having a positive RT-PCR during a period of high incidence, early in the COVID-19 pandemic.
CONCLUSION: Using a structured approach, a positive CXR had a high likelihood of predicting a positive RT-PCR, with good interrater reliability. CXRs can be useful in identifying new cases of COVID-19.

Introduction

The Coronavirus Disease 2019 (COVID-19) pandemic is an ongoing global health emergency with persistently high mortality [1]. The Fleischner Society have provided guidance on the role of chest imaging in patients with suspected COVID-19, but they have not issued explicit guidance on the best imaging modality [2]. The majority of chest imaging studies in COVID-19 to date have focused on CT and have identified consistent patterns in COVID-19 pneumonia including bilateral, peripheral lower lobe predominant ground glass opacities and consolidation as well as a low incidence of pleural effusions [[3], [4], [5], [6], [7], [8], [9]]. Evidence for the utility of CXRs in the initial diagnosis of COVID pneumonia is much less robust and their diagnostic value has been questioned [4].

A prior meta-analysis of the diagnostic performance of CT and RT-PCR in COVID-19 found that CT had a pooled sensitivity of 94 % and a pooled specificity of 37 % [10], however the disease prevalence was not always considered in these studies [11,12]. A pattern of peripheral opacities has been shown on both chest radiography and CT [13,14]. We have anecdotally found this pattern to have a high correlation with a positive real time reverse transcription-polymerase chain reaction (RT-PCR) diagnosis of severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) infection. A number of standardized assessments of pulmonary involvement of COVID-19 have been proposed [9,15,16]. These are designed to rank the suspicion of pulmonary involvement of COVID-19 in patients who have signs and symptoms suggestive of COVID-19 pneumonia and are intended to be interpreted with clinical and laboratory findings. The Dutch Radiological Society have proposed the COVID-19 Reporting and Data System (CO-RADS), which is modelled on established standardized reporting systems such as BI-RADS and Lung-RADS, and have demonstrated moderate to substantial interobserver agreement in application of these categories between multiple readers [16]. Expert consensus opinion has advised the use of structured reporting in CT when COVID-19 pneumonia is suspected in order to standardise the description of lung findings as well as to decrease reporting variability [15]. Available standardised assessments for pulmonary involvement in CXR are less descriptive than those that have been established for CT [17].

These uncertainties surrounding the role and clinical utility of chest radiography in the early evaluation of patients with suspected COVID-19 pneumonia, and how structured reporting can be applied to CXR, have prompted us to examine this issue in more detail. The purposes of this study were multifold: 1) to construct a detailed description of a structured approach to reporting CXR in patients with suspected COVID-19 pneumonia, 2) to measure interobserver agreement in the application of this approach and 3) to evaluate the diagnostic performance of chest radiography compared to RT-PCR.

Materials & methods

Setting

The study was performed in a 600-bed tertiary referral hospital with an average annual emergency department attendance of 50,000. The hospital is the national referral centre for Extracorporeal Membrane Oxygenation (ECMO) and contains the National Isolation Unit for infectious diseases. The study was approved by the National Research Ethics Committee (NREC) for COVID-19 and consent was waived.

Patients and design

All patients who attended the emergency department from March 10th – April 4th 2020, who were suspected of COVID-19 and underwent both a portable chest radiograph (CXR) and a RT-PCR test for presence of SARS-CoV-2 on initial assessment, were included for analysis. Patients were suspected of COVID-19 if they presented with symptoms of an acute respiratory infection, (including fever, shortness of breath and cough) or a fever of unknown cause. Fever was defined as a temperature >38.0 °C. RT-PCR of oropharyngeal and nasopharyngeal swab was completed as part of the initial assessment of all patients suspected of having COVID-19. Serial RT-PCR was completed if the initial result was negative, at intervals of 1 day or more, if deemed required by clinicians. For patients with multiple RT-PCR assays, a positive result on an assay within 7 days of the initial swab was adopted as confirmation of diagnosis. Patients who had a positive RT-PCR >7 days after their initial assay were excluded from analysis. Patients underwent a CXR if they were suspected of COVID-19 and had moderate to severe symptoms requiring admission for treatment.

Chest radiographs

All CXRs were acquired as computer or digital radiographs following local protocols. CXRs were acquired in anteroposterior (AP) projection using portable x-ray units in isolation units. Patients were positioned erect, semi-erect or supine for radiograph acquisition depending on the patients’ clinical state. Radiographs were uploaded to the hospital picture archiving and communication system.

Blind review of chest radiographs

Two radiologists (BH with 8 years and PMM with 13 years of experience in interpreting chest radiographs) retrospectively reviewed the baseline CXR of 592 patients, blinded to the RT-PCR result, initial CXR report and subsequent radiological tests. The clinical symptoms, prior radiological tests and prior biochemical test results were available to the radiologists to simulate real-life practice. The two radiologists categorised each CXR as Normal, Unlikely, Indeterminate, High-Suspicion or Characteristic using the reporting template described below (Fig. 1 ). These initial categorisations were used to calculate interobserver agreement. Where there was a discordance in initial categorisation of the CXR the result was decided by consensus. Final categorisations were compared to the RT-PCR to assess diagnostic accuracy.

Fig. 1

CXR Structured Assessment: COVID-19.

Reporting template

Prior to the commencement of this study, a structured approach to the interpretation of CXR in suspected cases of COVID-19 was adopted by many in our radiology department based on available consensus [9,[15], [16], [17]]. We decided to build on these resources to make a more prescriptive template due to anecdotal lack of consensus on how to categorise radiographs. Our template demonstrates a number of similarities to the recently published CO-RADS structured approach for CT [16]. We have described five categories (Fig. 1) of radiographic appearance (Negative, Atypical, Indeterminate, High Suspicion and Characteristic) which are similar to CO-RADS 1–5. Figs. Fig. 1, Fig. 2, Fig. 3, Fig. 4–5 shows an example of a chest radiograph for each category. For analysis of diagnostic accuracy, Characteristic and High Suspicion CXRs were considered positive and the results were compared to RT-PCR.

Fig. 2

An example of a Characteristic chest radiograph (CXR). 60 year old man presented to the emergency department with a history of cough, dyspnoea and pleuritic chest pain. On the baseline CXR there were bilateral subpleural opacities that were relatively symmetrical and of large volume consistent with a Characteristic CXR. Real-time reverse transcription polymerase chain reaction RT-PCR of a swab taken on the same day as the CXR was positive for COVID-19.

Fig. 3

An example of a High Suspicion chest radiograph (CXR). 79 year old man presented to the emergency department with a history of cough and dyspnoea. On the CXR there were bilateral large volume patchy opacities consistent with a High Suspicion CXR. Real-time reverse transcription polymerase chain reaction RT-PCR of a swab taken on the same day was positive for COVID-19.

Fig. 4

An example of an Indeterminate chest radiograph (CXR). 71 year old woman presenting with increasing dyspnoea on a background of chronic obstructive pulmonary disease. On CXR there were unilateral large volume, multilobar opacities consistent with an Indeterminate CXR. Real-time reverse transcription polymerase chain reaction RT-PCR of a swab taken on the same day was negative for COVID-19.

Fig. 5

An example of an Unlikely chest radiograph (CXR). A 44 year old woman presented to the emergency department with cough, dyspnoea and left side pleuritic chest pain. On CXR there was dense consolidation throughout the left lower lobe consistent with a lobar pneumonia pattern. This was classified as an Unlikely CXR. Reverse transcription polymerase chain reaction RT-PCR of a swab taken on the same day was negative for COVID-19.

A Characteristic CXR must have bilateral subpleural opacities that are relatively symmetrical. They should be of large volume (subjectively >20 % of lung) and there should be no doubt that the opacities are not artefactual and related to the chest wall.

A High Suspicion CXR must have either unilateral subpleural opacity or bilateral large volume patchy or ill-defined opacities with a non-upper lobe predominance. The bilateral opacities must be non-apical predominant and not of a perihilar distribution.

An Indeterminate CXR may include any of the following elements: unilateral large volume patchy or multilobar opacities; unilateral or bilateral small volume opacity (<20 % of lung); suspected superimposed opacity overlying significant underlying parenchymal disease which may limit evaluation for acute disease; difficulty distinguishing a peripheral opacity from an artefact related to the chest wall.

An Unlikely CXR has findings that are not suggestive of COVID-19 including pleural effusion, heart failure pattern, upper lobe predominant opacities with involvement of the lower zones, pneumothorax and lobar pneumonia pattern (dense, well-defined consolidation or small volume ill-defined opacity clearly confined to single).

A Normal CXR must have either normal lungs on chest radiograph or definitively no interval change if there is a long-standing chronic finding such as linear scarring, goitre, healed rib fractures, cardiomegaly with chronic congestion, chronic airways inflammation or hyperinflation).

Data collection and analysis

Patient demographics (age, gender) and RT-PCR results were extracted from patients’ electronic records. For each level of the radiological assessment, concordance between the two assessors was evaluated against the consensus diagnosis and calculated as percentage positive agreement (both assessors concord with consensus diagnosis) and percentage negative agreement (both assessors discord with consensus). Overall inter-rater reliability was expressed as Fleiss-Cohen-weighted Cohen’s κ.

To assess the performance of radiological assessments to diagnose RT-PCR-confirmed COVID-19, the probability of a positive RT-PCR for each level of the radiological assessment was calculated and the overall diagnostic performance was expressed as the C-index.

Strength of association between methods was summarised as the Somers’ D statistic.

As a supplementary analysis, using the RT-PCR result as a reference, the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of baseline portable chest radiography were calculated. Characteristic and High Suspicion chest radiographs were considered positive. Chest radiographs classified as Indeterminate, Unlikely and Normal were considered negative. 95 % confidence intervals were provided. Statistical analysis was performed using R v3.6.5 (R Core Team). Analysis code can be found in Appendix 1 in Supplementary materials.

Results

590 patients attended the emergency department that were initially suspected of COVID-19 underwent RT-PCR and a baseline CXR during the study period. Eight patients had a positive RT-PCR >7 days after their initial assay and were excluded from analysis. Table 1 depicts the baseline characteristics of the remaining 582 patients included for analysis (51 +/- 20 years, 300 men). In total 143 patients (143/582, 24.6 %) had a positive swab within 7 days of presenting to the Emergency Department. 134 patients had a positive RT-PCR at baseline (134/582, 23.0 %). An additional 9 patients (9/582, 1.5 %). had an initial negative RT-PCR but were positive on repeat swab within 7 days. The COVID-19 positive group contained relatively more men than women, compared to the COVID-19 negative group (odds ratio = 1.59 (95 % CI: 1.08–2.33), χ2 = 5.16, p = 0.023).

Table 1

Characteristics of the included 582 patients.

Characteristic	PCR Positive	PCR Negative
Total	143	439
Age, (years), Median (IQR)	48 (34–64)	52 (35–68)
Men, n (%)	86 (60.1)	214 (48.7)
Time interval of RT-PCR to CXR, (days), Median (IQR)	0 (0-0)	0 (0-0)
CXR Results, n (%)
Characteristic	17 (11.9)	0 (0.0)
Highly Suspicion	35 (24.5)	7 (1.6)
Indeterminate	22 (15.4)	40 (9.1)
Unlikely	15 (10.5)	127 (28.9)
Normal	54 (37.8)	265 (60.4)
Position, n (%)
Erect	89 (62.2)	264 (60.1)
Semi-Erect	52 (36.4)	160 (36.4)
Supine	2 (1.4)	15 (3.4)

Of the patients with a positive RT-PCR, on the baseline CXR: 54/143 (37.8 %) were categorised as Normal, 15/143 (10.5 %) as Unlikely for COVID-19, 22/143 (15.4 %) as Indeterminate, 35/143 (24.5 %) as Highly Suspicion and 17/143 (11.9 %) as Characteristic. In both RT-PCR positive and negative groups RT-PCR testing and CXR occurred on the same day for the large majority of cases (for both groups: median = 0 days. IQR: 0 – 0, with a range of the RT-PCR test occurring 2 days prior to the CXR to occurring 6 days afterward the CXR).

Diagnostic performance

The performance of structured CXR assessments to diagnose COVID-19 was evaluated. To produce reliable estimates, the Characteristic level, which had low case frequencies and no cases with an RT-PCR negative status, was merged with the High Suspicion level to create a new level called High Suspicion-Characteristic. The probabilities of a patient having a positive RT-PCR status were calculated for each CXR assessment level and are shown in Table 2 . High Suspicion-Characteristic cases showed a high probability 0.88 (95 % CI 0.80 - 0.96) and a high positive likelihood ratio 22.8 (95 % CI 10.6–49.1) for the presence of RT-PCR-confirmed COVID-19, indicating the structured CXR assessment performs well in diagnosing COVID-19 when it is indeed present. The negative likelihood ratios for Unlikely 0.4 (95 % CI 0.2 - 0.6) and Normal CXR 1.3 (95 % CI 1.2–1.4) assessments indicate a moderate performance for ruling out COVID-19. As expected, Indeterminate CXR assessments do not significantly shift the diagnosis of COVID-19.

Table 2

Probabilities of a positive RT-PCR for each radiological assessment level. Includes 95 % confidence interval.

CXR	Probability of a positive RT-PCR	Positive Likelihood ratio	Negative Likelihood ratio
High Suspicion- Characteristic	0.88 (0.80 - 0.96)	22.8 (10.6–49.1)	0.6 (0.6 - 0.7)
Indeterminate	0.35 (0.24 - 0.47)	1.7 (1.0–2.7)	0.9 (0.9–1.0)
Unlikely	0.11 (0.06 - 0.16)	0.4 (0.2 - 0.6)	1.3 (1.2–1.4)
Normal	0.17 (0.13 - 0.21)	0.6 (0.5 - 0.8)	1.6 (1.3–1.9)

Somers’ D was 0.49 (95 % CI 0.36-0.62), indicating a moderate overall strength of association between CXR and RT-PCR. The closely related C-statistic was 0.74 (95 % CI 0.70-0.79), indicating that radiological assessment has a moderate probability of predicting RT-PCR status.

Additional analysis was performed to assess the diagnostic performance of CXR as a binary diagnostic for COVID-19. CXRs categorised as Characteristic and High Suspicion were considered positive CXRs and CXRs classified as Indeterminate, Unlikely and Normal were considered negative CXRs for COVID-19. This binary CXR assessment had a sensitivity of 43 % (95 % CI 34–52 %), specificity of 98 % (95 % CI 96–99 %), positive predictive value of 88 % (95 % CI 77–95 %), and negative predictive value of 85 % (95 % CI 81–88 %). These show that the accuracy of baseline CXR in the diagnosis of COVID-19 is moderate to good, with a high specificity and a low sensitivity (Table 4).

Table 4

The performance of CXR for COVID-19 infection with RT-PCR result as reference.

Statistic	Estimate
True prevalence	0.23 (0.20 - 0.27)
Sensitivity	0.43 (0.34 - 0.53)
Specificity	0.98 (0.96 - 0.99)
Accuracy	0.85 (0.82 - 0.88)
PPV	0.88 (0.77 - 0.95)
NPV	0.85 (0.81 - 0.88)
Positive likelihood ratio	24.5 (11.4–52.5)
Negative likelihood ratio	0.6 (0.5 - 0.7)

Interobserver concordance

Table 3 displays the interobserver concordance between two assessors for each CXR level. Overall agreement of initial interpretation for all levels was 71.1 %, with a Fleiss-Cohen-weighted Cohen’s κ for interrater reliability of 0.81 (95 % CI: 0.78 - 0.85), indicating moderate to good overall interobserver concordance. By CXR level, only Normal (249/319, 78.1 %) and Unlikely (105/142, 73.9 %) CXRs showed good concordance. Indeterminate (31/62, 50.0 %) and High Suspicion (22/42, 52.4 %) CXR assessments had moderate concordance, while Characteristic (7/17, 41.2 %) CXR assessments had relatively poor concordance.

Table 3

Concordance between two assessors by level. For each category of the radiological assessment, concordance between the two assessors was evaluated against the consensus diagnosis and calculated as percentage positive agreement (both assessors concord with consensus diagnosis) and percentage negative agreement (both assessors discord with consensus).

	Positive agreement		Negative Agreement
Assessment	N	%	N	%
Characteristic	7	41.2 (17.6–64.7)	0	0.0 (0.0 - 0.0)
High Suspicion	22	52.4 (38.1–66.7)	7	16.7 (7.1–28.6)
Indeterminate	31	50.0 (37.1–61.3)	1	1.6 (0.0–4.8)
Unlikely	105	73.9 (66.2–81.0)	3	2 (0.0–4.9)
Normal	249	78.1 (73.3–82.4)	1	0 (0.0 - 0.9)
Overall	414	71.1 (67.5–74.6)	12	2.1 (1.0–3.3)

Discussion

The majority of chest imaging studies in COVID-19 to date have focused on CT and have identified consistent imaging findings. This study found that when using a structured template reporting system, CXRs demonstrated a high positive predictive value and specificity of 88 % (95 % CI 80–96 %) and 98 % (95 % CI 96–99 %) respectively making it a useful tool in the diagnosis of COVID-19. This approach had a high concordance between two assessors with an absolute concordance of 71.1 % (414/582) and a Fleiss-Cohen-weighted Cohen’s κ of 0.81 (95 % confidence interval, 0.78-0.85). Overall, our agreement was higher than that seen in the CO-RADS study [16] but this study involved multiple readers across multiple sites which is likely to account for the difference.

Numerous studies have described imaging findings in an entirely positive cohort [3,5,13,18,19] but we chose a mixed RT-PCR positive and negative cohort so that our findings might be more generalisable to daily practice. Previous studies have evaluated sensitivity and specificity of CT for the diagnosis of COVID-19 pneumonia by assessing for the presence of pulmonary findings that have been reported to be either commonly present or absent in the disease [6,8,18,20]. However in the clinical setting, how likely the patient is to have COVID-19 given a positive or negative CXR is more useful than the sensitivity or specificity [19]. Sensitivity of CT has been repeatedly shown to be high, ranging from 93 % to 98 % with a pooled sensitivity of 94 % on meta-analysis [10]. While false negatives in CT evaluation for COVID-19 are low, to date, CT has not shown a good ability to accurately rule in COVID-19 with pooled specificity of 37 % and positive predictive value (PPV) ranging from 1.5%–30.7% [5,10,14,21]. In this study, we have found CXRs to have an almost inverse performance relative to CT with a high specificity of 98 % but a low sensitivity of 43 %. This result can be partially explained by the inherent insensitivity of chest radiographs compared with CT. Several studies have demonstrated that early airspace opacities seen on initial CT are significantly less conspicuous on contemporaneous chest radiographs [5,13,18]. Furthermore, sensitivity and specificity are statistics based around positive and negative results rather than levels of confidence. By designating High Suspicion pattern and Characteristic pattern as positive and the remaining groups negative, this approach assumes a high false-negative rate. This strategy was felt to be appropriate in the patient presenting to the Emergency Department with suspected COVID-19 pneumonia as chest imaging studies in general have been shown to be frequently normal in early stages of COVID-19 infection [3,5,19]. This is particularly true of patients with RT-PCR confirmed COVID-19 who have no abnormalities on their chest radiograph, which accounts for 41–58.3 % of patients [6,19] Similarly 37.7 % of our RT-PCR-positive patients had normal initial radiography. One study has shown a sensitivity of 69 % [13] for CXRs in COVID-19 but this was in a smaller cohort of patients all of whom had RT-PCR confirmed COVID-19 prior to imaging analysis, which may account for some of the difference.

The Fleischner Society has advocated for imaging in all symptomatic patients suspected to have COVID-19 with moderate/severe clinical severity, with mild respiratory disease but with risk factors for progression, and for any patient with clinically worsening respiratory disease [15]. In general, portable chest radiography is predicted to be the most commonly utilised modality for the identification and follow up of lung abnormalities in suspected cases of COVID-19 pneumonia [14]. This is mainly because portable radiography avoids the need for patient movement through the hospital and simplifies cleaning procedures [22]. Turnaround times for RT-PCR for COVID-19 vary but can take over 24 h. Given that there are often constraints on resources (such as personal protective equipment, ventilators and healthcare staff), it is important to optimise decision-making on whether to manage a patient in a COVID-specific pathway. We have found that, using a COVID-specific structured approach to interpretation, a CXR designated as High Suspicion or Characteristic in a patient presenting to the Emergency Department with clinically suspected COVID-19 pneumonia correlates with a very high likelihood of a positive RT-PCR. Confirmation with RT-PCR is always recommended [22] but we believe the positive predictive value and specificity of a positive CXR merit a presumptive diagnosis of COVID-19 until the RT-PCR result becomes available. Furthermore, we would argue that if initial RT-PCR is negative, in those with a positive CXR, particularly those with a Characteristic appearance, that precautions should still be maintained and RT-PCR repeated. We have demonstrated very good agreement in the application of this structured approach in a single institution and feel that this practice is generalisable to other institutions. CXRs that are designated as Normal, Unlikely or Indeterminate are less useful in the initial patient evaluation but awareness of this finding is essential for clinicians. Given that CT of the chest has shown high sensitivity but low specificity, radiographs and CT may be complementary tests in diagnostically difficult cases. Lastly, CXRs have been shown to be useful in prognostication and in tracking disease progression [13,23].

The primary limitation of this study is the validation of our structured approach to radiograph interpretation. We have demonstrated very good concordance between two senior radiologists within the same institution but we have not demonstrated agreement between radiologists working in different environments, with different levels of experience and between radiologists and non-radiologist clinicians. We consider this study a proof-of-concept and hope to validate this approach on a wider scale in the future. Additionally, we have evaluated CXRs against COVID-19 RT-PCR positivity within 7 days but are aware that this may not reflect the true presence of the infection in our cohort and cannot accurately account for why some patients had more than one swab, however we do feel that this reflects everyday practice and this would not reduce our measured specificity or positive predictive value. We do not have a record of the timing of onset of symptoms compared with timing of radiograph and analysis of this may improve sensitivity. Lastly, we are aware that this study was carried out during a pandemic at a time of very high incidence of COVID-19 and many of the imaging findings will have a lower specificity at a time when this disease has a low incidence, however the Characteristic appearance has a narrow differential and could prove useful in identifying patients with COVID-19 in the future.

Conclusion

A structured approach to CXRs provides a template to describe the changes likely attributed to COVID-19. This study found that by using a structured approach, a positive chest radiograph had a high likelihood of predicting a positive RT-PCR with good interrater reliability.

Funding

None.

CRediT authorship contribution statement

Andrew Yates: Conceptualization, Methodology, Resources, Data curation, Writing - original draft, Writing - review & editing, Visualization, Project administration. Philip J. Dempsey: Data curation, Writing - original draft, Writing - review & editing. Sebastian Vencken: Data curation, Formal analysis, Software, Writing - original draft, Writing - review & editing. Peter J. MacMahon: Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization, Project administration, Supervision. Barry D. Hutchinson: Conceptualization, Methodology, Resources, Data curation, Writing - original draft, Writing - review & editing, Visualization, Project administration, Investigation, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.