Deployment of artificial intelligence for radiographic diagnosis of COVID-19 pneumonia in the emergency department.

OBJECTIVE: The coronavirus disease 2019 pandemic has inspired new innovations in diagnosing, treating, and dispositioning patients during high census conditions with constrained resources. Our objective is to describe first experiences of physician interaction with a novel artificial intelligence (AI) algorithm designed to enhance physician abilities to identify ground-glass opacities and consolidation on chest radiographs.
METHODS: During the first wave of the pandemic, we deployed a previously developed and validated deep-learning AI algorithm for assisted interpretation of chest radiographs for use by physicians at an academic health system in Southern California. The algorithm overlays radiographs with "heat" maps that indicate pneumonia probability alongside standard chest radiographs at the point of care. Physicians were surveyed in real time regarding ease of use and impact on clinical decisionmaking.
RESULTS: Of the 5125 total visits and 1960 chest radiographs obtained in the emergency department (ED) during the study period, 1855 were analyzed by the algorithm. Among these, emergency physicians were surveyed for their experiences on 202 radiographs. Overall, 86% either strongly agreed or somewhat agreed that the intervention was easy to use in their workflow. Of the respondents, 20% reported that the algorithm impacted clinical decisionmaking.
CONCLUSIONS: To our knowledge, this is the first published literature evaluating the impact of medical imaging AI on clinical decisionmaking in the emergency department setting. Urgent deployment of a previously validated AI algorithm clinically was easy to use and was found to have an impact on clinical decision making during the predicted surge period of a global pandemic.

INTRODUCTION

The surge of patients in acute respiratory distress during the coronavirus disease 2019 (COVID‐19) pandemic has inspired new innovations in diagnosing, treating, and dispositioning patients during high census conditions with constrained resources. Newer technologies have been widely and rapidly deployed to respond to the pandemic with examples such as telemedicine, wearable health sensors, and digital contact tracing among others. , , Many of these technologies can be used to scale up constrained healthcare resources to meet increasing healthcare demands of the pandemic response, whereas some such as telemedicine promise remote access to clinical expertise.

Artificial intelligence (AI) is an increasingly prevalent computational approach used to help automate pattern recognition in fields such as image evaluation, natural language processing, large‐scale data review, and others. The broad adoption of AI in other industries has been the impetus for its investigation in the clinical space. AI has been broadly theorized to impact nearly every aspect of operational clinical medicine. , Examples of AI applications in the emergency department (ED) have been ambulatory patient volume prediction, prediction of delayed cardiac complications, electronic triage, detection of papilledema from ocular fundus photographs, prediction of sepsis, and abstraction of common diagnoses through natural language processing of physician notes. , Furthermore, medical imaging has been a target application for AI in the ED by showing promise in automated detection of life‐threatening conditions on imaging studies including hemorrhage, mass effect, hydrocephalus, or suspected infarct on computed tomography imaging. Emerging research abroad has shown promise in the use of AI in fighting the COVID‐19 pandemic.

During the first wave of the pandemic, researchers in the department of radiology developed and deployed an AI algorithm in our radiology picture archiving and communication system to provide computer‐assisted interpretation of chest radiographs for use by radiologists and emergency physicians. Multiple open‐source and early‐stage algorithms have been reported to assist in the radiographic diagnosis of COVID‐19 in the clinical setting; however, this is the first reported and peer‐reviewed clinical deployment of such a strategy in the United States. , We report the first experiences of emergency physician interaction with this novel AI algorithm designed to enhance physician abilities to identify ground glass and consolidation on chest radiographs suspicious for COVID‐19 pneumonia.

METHODS

We evaluated initial impressions of a previously developed deep‐learning AI algorithm that provides AI augmentation of anterior‐posterior/posterior‐anterior projection (AP/PA) chest radiograph images. The algorithm was deployed in a fully automated digital data pipeline in the clinical ED environment leveraging a commercially available cloud computing service (Amazon Web Services, Seattle, WA). The algorithm overlays radiograph images with color‐coded “heat maps” that indicate pneumonia probability using semantic segmentation deep learning. This algorithm was developed using a publicly available and widely used data set of 25,684 annotated radiographs from the National Institutes of Health frontal‐chest radiograph database.

The convolutional neural network was implemented and trained in Python (version 3.5; Python Software Foundation, Wilmington, DE) using Keras 2.2 and Tensorflow 1.8. No transfer learning or image optimization was undertaken. Overall area under the receiver operating curve for the convolutional neural network was 0.854. At the optimal operating point (Youden J‐index threshold), this corresponded to an accuracy of 81.6%, sensitivity of 82.8%, and specificity of 72.6%.

The generated AI heat map overlay was provided alongside the patient's corresponding AP or PA chest radiograph images for physician use in real time at the point of care with existing imaging software (IMPAX 6, AGFA Healthcare IT, Mortsel, Belgium; Figure 1). The heat map overlaid a probability of consolidation at each pixel on a discrete, color‐coded intensity from 0% to 100%. The tool labeled images “for investigational use only” to avoid inadvertent miscommunication or claims about the accuracy of the algorithm during our evaluation of the algorithm's performance. The development of the automated clinical pipeline required that the algorithm could be viewed on any workstation in the ED through the picture archiving and communication system so that every physician could use the algorithm in their existing workflow. This prospective survey study was performed at 2 large, urban academic health centers with 70,000 estimated annual ED visits in Southern California (UC San Diego Health). The tool was approved for use by the institutional review board at UC San Diego Health (Institutional Review Board 191759). US Food and Drug Administration approval was not required for this study.

FIGURE 1

Example heat map of the artificial intelligence algorithm, as applied to a patient with the novel coronavirus disease 2019. As the chest radiograph is captured (A), the augmented image is generated in real time and displays a color‐coded heatmap of consolidation probability as shown (B). The corresponding computed tomography with angiography (CTA) that was performed contemporaneously to the chest radiograph with extensive ground‐glass opacities is shown (C)

We developed a 3‐point survey to characterize experiences with the tool regarding ease of use and impact on clinical decisionmaking. The survey was validated using best practices and developed in an iterative approach. Because physicians were queried in real time, the 3 questions were chosen to maximize clinically important impressions while balancing the need for brevity to respect physician time and to not interrupt patient care. Because of the necessarily brief survey instrument and because there is a lack of data reporting emergency physician interaction with clinically deployed AI tools, the study was designed to be hypothesis generating for further investigation into trends observed in this initial study. Given the ethical constraints of exposing research staff to the clinical environment into the pandemic, we created a remote research assistant program that allowed for real‐time query of physicians across 2 affiliated busy academic EDs. A federal declaration of emergency occurred March 13, 2020, and the tool was urgently deployed on March 25. Surveys were conducted during a 1‐month period surrounding the projected COVID‐19 surge locally (April 8–May 9).

RESULTS

Of the 5125 total visits and 1960 chest radiographs obtained in the ED during the study period, 1855 were analyzed by the algorithm. Among these, emergency physicians were surveyed for their experiences on 202 radiographs. During the 8‐hour daily research assistant shifts, 38% of all scheduled attendings in the health system were queried as well as 53% of all resident physicians.

Overall, 86% either strongly agreed or somewhat agreed that the tool was easy to use in the existing workflow. Of all respondents, 20% reported that the algorithm impacted their clinical decisionmaking. In general, resident physicians responding to the survey reported they found the AI implementation easier to use than attendings (Mann‐Whitney U; P = 0.002; 95% confidence intervals 1.06–1.29 and 1.42–1.80, respectively).

Notably, 43% of the attending physicians surveyed who felt that the tool influenced management indicated that it changed disposition times (longer or shorter time in the ED). In the overall cohort of physicians who felt that the AI‐augmented overlay contributed to their medical decisionmaking, 27% felt it contributed to decisions regarding diagnostic testing, 15% in decisions surrounding final diagnosis, 30% on treatment plan, 27% on disposition time, and 10% on disposition location (admission vs discharge). Further breakdowns of response differences between attendings and resident physicians are summarized in Table 1.

TABLE 1

Survey demographics and survey data as stratified by respondent level of training (resident cohort, attending cohort, and overall cohort)

Survey information
Total X‐rays in ED during study period	Total X‐rays with colormap applied during study period	Total number of surveys	Total physicians in ED	Attendings surveyed	Residents surveyed	Average time to generate heat map (minutes)
1960	1885	202	63 attendings, 49 residents	24 (38% of all scheduled)	21 (53% of all scheduled)	4

Note that question 3 rows do not sum to 100% (multiple choice). ED = emergency department.

DISCUSSION

The algorithm was created and reported in late 2019, and the tool was urgently deployed once the IT infrastructure was created and the local state of emergency was declared in March 2020 during the first wave of the global pandemic. Surveyed emergency physicians found this implementation easy to use within existing workflows. Of the physicians, 20% reported that the tool changed clinical decisionmaking, and approximately one third of those found that it impacted diagnostic testing decisions and treatment plans. Although the degree and nature of impact were not directly assessed in our survey instrument, several physicians reported ordering COVID‐19 polymerase chain reaction testing as a direct result of the AI, resulting in positive tests and subsequent quarantining of patients who otherwise might not have been appropriately diagnosed.

Approximately 75% of COVID‐19–positive hospitalized patients develop viral pneumonia, and 17% progress to acute respiratory distress syndrome and often fatal lung injury representing diffuse alveolar damage. Chest radiographs are often used in the ED as an easily obtainable, rapid, and inexpensive diagnostic tool for the undifferentiated patient in respiratory distress. Current guidelines recommend initial chest imaging as an initial step in the diagnosis of COVID‐19, especially in a resource‐limited setting in patients with high pretest probability and moderate to severe clinical features.

In general, AI has been used to facilitate pattern detection in medical imaging through disease detection, classification, image optimization, radiation reduction, and workflow enhancement. Medical research is making AI applications more explainable, safe, effective, and integrated into physician workflows. However, there has been no published literature on the impact of using AI on clinical decisionmaking in ED settings.

Emergency physicians read these films rapidly using pretest probability and test sensitivity to assign a post‐test probability of medical conditions before final radiologist interpretation but are subject to interpretation error attributed to fatigue, stress, task switching, caseload, and cognitive bias. As the pandemic continues, caseload may overwhelm available expert diagnostic oversight, and subtle imaging findings may be overlooked. Currently, the final interpretation is provided by an on‐site attending radiologist during overnight hours in only 27% of all academic hospitals in the United States. Although this figure is much higher in dedicated pediatric academic hospitals (81%), not all community emergency physicians have access to contemporaneous radiographic interpretation by a radiologist overnight. With increasing caseload, use of AI adjuncts to assist the emergency physician with clinical decisionmaking may help to offload some of the burden experienced by the physician during the global pandemic. Anecdotally, several respondents noted that use of the tool identified subtle consolidations that were initially missed by the emergency physician.

Deploying this tool in the existing clinical radiograph viewing software alongside the traditional radiograph viewing software required physicians to interpret the images independently. Whereas the output of the AI algorithm could be displayed as a numeric probability of consolidation, the heat map graphic was a design decision to ensure that the output was evaluated independently with the patient's clinical picture in mind. In short, the algorithm did not replace physician decisionmaking but, rather, was intended to be another datapoint for the emergency physician to consider in context.

We acknowledge several limitations to this brief research report. Current literature with respect to randomized control trials regarding AI deployment is a relatively underexplored with only 2 published studies reported, and those were limited by high rates of bias. As an initial survey, this report is hypothesis generating and will be expanded to a randomized controlled trial with human comparator groups. The brief 3‐question survey was a compromise to obtain key information regarding clinical application and first experiences with this tool while respecting the time of the physicians managing the first wave of the pandemic; however, more detailed questions would be helpful to further understand and improve the tool. It is important to note that because physicians were free to decline to respond to the survey evaluating this new tool, it is possible that confirmation bias may be present. The convenience nature of the sample selection does also mean that the responding group was non‐random, and there is potential for bias that should be taken into account while interpreting the survey responses. That is to say that new technology often is used by an “early‐adopter” user group who may be more enthusiastic about evaluating new diagnostic tools or conversely user groups that may be more critical of the tool. At the time of development, there were no publicly available annotated data sets that were specific to radiograph abnormalities associated with COVID‐19 infection. Further study using transfer learning to calibrate and tune the algorithm could lead to even better test characteristics. Finally, as with all AI algorithms in clinical medicine, testing the intervention in a multisite study would be helpful to improve external validity.

A significant criticism of the field of AI in the clinical space has been the overstatement of benefit and lack of prospective, well‐designed clinical trials with regard to implementation. Many AI algorithms proposed for clinical use have been validated retrospectively using ground truth data compared to algorithm output. , , , , , There is limited published research on how physicians interpret this information and the utility and impact the tools are found to have when deployed. , , , This article presents the deployment and perception of benefit from using deep‐learning algorithms in the clinical space during the infancy of AI deployment. Previously, the tool was evaluated during its development and validation to show excellent test characteristics. In this article, we report that clinical application of the tool in our health system with minimal training of physician users shows perceived benefit. Compared with the artificial in silico context in which many AI tools are reported, the modest survey responses regarding this tool's perceived impact on clinical decisionmaking is an important reminder that model performance is not the only factor in physician implementation and trust in AI.

In closing, we report the deployment and initial impressions of an AI tool that may assist physicians detect abnormalities in chest radiographs during a global pandemic. This brief research report is hypothesis generating and helps to bring the advances in deep learning from the laboratory to the ED, where physicians must make life or death decisions on often incomplete data sets. We have shown that despite rapid and emergent deployment into a clinical environment, emergency physicians found the tool easy to use within existing workflows and had clinical utility. To our knowledge, this is the first published literature evaluating the impact of medical imaging AI on clinical decisionmaking in the ED setting.

CONFLICT OF INTEREST

A.H. receives research grant support from GE Healthcare and Bayer AG, unrelated to the presented work. A.H. is also a co‐founder and shareholder of Arterys, Inc. The remaining authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Christian Dameff and Morgan Carlile conceived the study and designed the trial. Albert Hsiao and Brian Hurt developed the algorithm used in this study. Michael Hogarth and Christopher A. Longhurst provided institutional guidance and led operational deployment. Morgan Carlile performed statistical analysis. Morgan Carlile and Christian Dameff drafted the article, and all authors contributed substantially to its revision. Morgan Carlile takes responsibility for the article as a whole.