Artificial intelligence in emergency medicine: A scoping review.

INTRODUCTION: Despite the growing investment in and adoption of artificial intelligence (AI) in medicine, the applications of AI in an emergency setting remain unclear. This scoping review seeks to identify available literature regarding the applications of AI in emergency medicine.
METHODS: The scoping review was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for scoping reviews using Medline-OVID, EMBASE, CINAHL, and IEEE, with a double screening and extraction process. The search included articles published until February 28, 2020. Articles were excluded if they did not self-classify as studying an AI intervention, were not relevant to the emergency department (ED), or did not report outcomes or evaluation.
RESULTS: Of the 1483 original database citations, 395 were eligible for full-text evaluation. Of these articles, a total of 150 were included in the scoping review. The majority of included studies were retrospective in nature (n = 124, 82.7%), with only 3 (2.0%) prospective controlled trials. We found 37 (24.7%) interventions aimed at improving diagnosis within the ED. Among the 150 studies, 19 (12.7%) focused on diagnostic imaging within the ED. A total of 16 (10.7%) studies were conducted in the out-of-hospital environment (eg, emergency medical services, paramedics) with the remainder occurring either in the ED or the trauma bay. Of the 24 (16%) studies that had human comparators, there were 12 (8%) studies in which AI interventions outperformed clinicians in at least 1 measured outcome.
CONCLUSION: AI-related research is rapidly increasing in emergency medicine. There are several promising AI interventions that can improve emergency care, particularly for acute radiographic imaging and prediction-based diagnoses. Higher quality evidence is needed to further assess both short- and long-term clinical outcomes.

INTRODUCTION

The study of artificial intelligence (AI) in medicine has become increasingly popular over the last decade. , The field of AI refers to a broad subset of computer science that simulates human intelligence, including speech recognition, predictive modeling, and problem solving. Machine learning (ML), a subset of AI, has recently gained popularity in medicine because of its ability to improve algorithms autonomously. Because of rapid advances in computing power and processing techniques, sophisticated ML subtypes such as deep learning have shown potential to improve patient care. As such, there is growing literature that has shown that AI‐based interventions can match or even outperform physician expertise.

The emergency department may be uniquely situated to benefit from AI because of its potential value in prediction during triage, as well as its versatility in analyzing diverse patient factors. Patients are assessed in the ED with limited information, and physicians often find themselves balancing probabilities for risk stratification and decisionmaking. Furthermore, there is a potential opportunity for ED flow metrics and resource allocation to be optimized through algorithm support and computerized decisionmaking. This is otherwise difficult to perform with conventional computing because of the sheer number of variables involved and the constant flux of metrics.

However, there remain concerns regarding the use of AI and its implications for patient safety considering the limited body of evidence to support its implementation. , , For example, unintended patient outcomes may occur if an automated system cuts corners to meet data targets, also known as reward hacking. , Additionally, it can be difficult for clinicians and medical researchers to understand the available AI‐related interventions, because of the interdisciplinary nature of the field and the lack of readily available peer‐reviewed research. Although there have been narrative literature reviews exploring the relevance of AI in the ED, no systematic study of the research exists. , , , We sought to systematically search the available literature for AI interventions relevant to the ED through a scoping review and provide a snapshot of how AI can be conceptualized in contributing to emergency medicine.

METHODS

The scoping review was conducted according to the standards and guidelines established in the Preferred Reporting Items for Systematic Reviews and Meta‐Analysis with the associated extension for Scoping Reviews (PRISMA‐ScR), in addition to the fourth edition of the Joanna Briggs Institute Reviewer's Manual. , In comparison to a conventional systematic review, the scoping methodology was preferable to a conventional systematic review because of the breadth of the field and the high degree of heterogeneity of the included research. We conducted a systematic literature search of Medline‐OVID, EMBASE, CINAHL, and IEEE to inform our scoping review. Our search strategy was created in consultation with a research librarian and is included in Table 1.

TABLE 1

Search strategy. Database: Ovid MEDLINE: Epub Ahead of Print, In‐Process, and Other Non‐Indexed Citations, Ovid MEDLINE Daily and Ovid MEDLINE. Adapted for EMBASE, IEEE, and CINAHL

#	Searches
1	(Artificial adj2 intelligen*).tw,kf.
2	((Machine OR deep) adj0 learn*).tw,kf.
3	(Artificial neural network*).tw,kf.
4	exp Artificial intelligence/
5	1 OR 2 OR 3 OR 4
6	Emergency Treatment/or Emergency Medicine/or emergency medical services/or emergency service, hospital/or trauma centers/or triage/or exp Evidence‐Based Emergency Medicine/or exp Emergency Nursing/or Emergencies/or emergicent* or casualty department* or ((emergenc* or ED) adj1 (room* or accident or ward or wards or unit or units or department* or physician* or doctor* or nurs* or treatment*or visit*)).mp. or (triage or critical care or (trauma adj1 (cent* or care))).mp
7	5 AND 6

Eligibility criteria

Our inclusion criteria for articles were as follows:

Population

Treated in the out‐of‐hospital setting through emergency medical services (EMS), EDs, urgent care centers, and trauma bays in any country.

Intervention

Any computer science intervention classified as AI by the study authors, which included both supervised and unsupervised ML.

Comparator

No intervention, standard of care, another computer science intervention, or any other comparator.

Outcomes

Any outcome reported in the literature.

We examined only original, peer‐reviewed literature published in the English language. The search was initially conducted on July 10, 2019 and then updated to include articles published up until February 28, 2020.

Although published conference posters, papers, and abstracts were initially eligible for inclusion, they were later excluded in a second round of screening by 2 reviewers (AK, SK). Articles were excluded if they did not self‐classify as studying an AI or ML intervention within either the study title or abstract. Studies were also excluded if location or context was not the ED or out‐of‐hospital setting. Studies were not eligible if they used ED patient data sets but were not directly relevant to emergency medicine (eg, if ED patient visits were used to generate public health predictions). Finally, papers that did not include outcomes or evaluations were excluded.

A hand search of citations of relevant reviews was performed to ensure comprehensiveness of the search. Grey literature was not formally searched.

Study selection and extraction

Study selection was completed by 2 independent, parallel reviewers (AK, AT) for both title and abstract screening and then subsequent full‐text screening. A pilot test of screening was conducted for the first 100 search results. Each abstract underwent 2 rounds of evaluation by a separate reviewer. Eligible abstracts underwent full‐text screening again by the 2 separate reviewers. Discrepancies in screening were resolved through consensus between the 2 reviewers (AK, AT).

Data extraction was performed independently by 2 reviewers (AK, SK), with a third (AT) resolving discrepancies via consensus. Risk of bias (ROB) for individual studies was graded using an adapted rating scale based on the Cochrane ROB tool and the ROB Assessment tool for Non‐randomized Studies (RoBANS). ,

Categorization and analysis

We categorized the AI interventions into ML (further subdivided into supervised, non‐supervised, and reinforcement ML) and natural language processing (NLP). A simplified explanation of the different AI types is provided in Table 2.

TABLE 2

Descriptions of the broad categories of artificial intelligence (AI) included in the review

Type of AI	Common labels	Example	Definition	Study Citation
Supervised Machine Learning (ML)	Support vector machine (SVM)	Abedi et al 21	ML that learns a function based on examples and previous input	Evaluation of supervised learning algorithm in emergency department using retrospective data
	K‐nearest neighbor (KNN)
	Naive Bayes (NB)
	Regression techniques
	Random forest (RF)
	Gradient boosting (GB)
Unsupervised ML	K principle	Farahmand et al 63	ML that does not require human input or labeled responses to generate inferences	Artificial Intelligence‐Based Triage for Patients with Acute Abdominal
	Linear discriminant analysis			Pain in Emergency Department; a Diagnostic Accuracy Study
	Neural networks
	Hierarchical clustering
Reinforcement ML	Q‐learning	None included	Machine learning that trains models to make decisions based on incentives	N/A
	Apprenticeship learning
Natural Language Processing (NLP)	Sentiment analysis	Pestian et al22	Artificial intelligence (AI) regarding human language (including language recognition, understanding, and generation)	Randomized controlled trial of natural language processing software in ED
	Optical character recognition
	Natural language generation

We then categorized studies by the purpose of the intervention, which were identified based on the study's reported objective. Although the list of purposes is not exhaustive, we felt that they were able to best map the intentions of the studied interventions. Purpose categories were determined through an iterative process based on frequency of results and suggestions from the literature. ,

Data were further extracted regarding the purpose of the intervention, study type (eg, retrospective, prospective), sample size, and type of comparator (eg, human comparator, statistical model, standard of care). All outcomes were summarized descriptively.

RESULTS

Search yield

Results of the study screening process are available in the PRISMA diagram in Figure 1. Of the 2933 original database citations, 999 duplicates were removed using the Covidence platform. After screening the 1888 remaining studies, 395 were eligible for full‐text evaluation. Following screening, 105 conference abstracts, posters, and papers were removed. Of the remaining articles, a total of 150 were included in the scoping review. After a hand search of relevant journals and citations, no additional studies were added. Interrater reliability for study screening for titles and abstracts was 92.1%, and for full‐text review was 94.1%. The authors were in substantial agreement with a calculated kappa of 0.77 and 0.82 for abstract and full‐text screening respectively.

FIGURE 1

Preferred Reporting Items for Systematic Reviews and Meta‐Analysis (PRISMA) flow chart. AI, artificial intelligence; ED, emergency department

Article characteristics

Details of the included interventions are available in Table 3 and details of the included studies are available in Table 4. The majority of studies were rated as low ROB (n = 139, 92.6%) with the remainder rated as medium ROB (n = 11, 7.3%) (Table 5).

TABLE 3

Overview of included interventions

Type of artificial intelligence (AI) intervention	Number of studies (N)
Supervised	N = 70
Random forest	15
Support vector machine	11
Fuzzy logic	1
K‐nearest neighbor	1
Decision tree	8
Supervised artificial neural network	23
Gradient‐boosted algorithm	5
Classification tree	1
Other/unspecified	6
Unsupervised	N = 2
Clustering	1
Neural network	1
Reinforced	N = 0
Unspecified	N = 41
Artificial neural network	34
Deep learning	3
Other	4
Mixed models	N = 23
Natural language processing	N = 14
Categories of purpose
Diagnosis	37
Triage	18
Prediction	72
Decisionmaking	3
Operations	18
Other	2

Note: ***(numbers do not add to 150 as several studies evaluate multiple interventions)***

TABLE 4

Overview of included studies and research designs

Study setting
Out‐of‐hospital	16
Emergency department/trauma bay	134
Study design
Retrospective diagnostic accuracy	124
Prospective cohort	16
Case‐control	1
Controlled trial	3
Before‐and‐after	4
Simulation modeling	2
Type of comparator
None	48
Human	24
Clinical decisionmaking tool	17
Other non‐AI statistical model	27
Other AI intervention	27
Standard of care (eg, before and after, set values)	4
Other (eg, cutoffs from literature, simulation)	3
	Total number of studies: 150

AI, artificial intelligence.

TABLE 5

Description of studies with human comparator (n = 24)

Author	Year	Study design	Setting/Context	Intervention	Purpose	Details	Sample size	Outcomes	AI superiority?
Blomberg et al	2019	Retrospective diagnostic accuracy	EMS calling center	ML algorithms	Prediction; Out‐of‐hospital	Predicting out‐of‐hospital cardiac arrests	108 607	Sensitivity, specificity, PPV, NPV	Y
Chilamkurthy et al 57	2018	Diagnostic accuracy	ED	Deep learning algorithms	Diagnosis; Imaging	Diagnosis of intracranial hemorrhages	313 318	AUC	N/A (physician used as gold standard)
Cicero et al 58	2017	Diagnostic accuracy	ED	Deep convolutional neural networks	Diagnosis; Imaging	Classification of abnormalities on frontal CXRs	35 038	Sensitivity, specificity, AUC	N/A (physician used as gold standard)
Clarke et al 59	2002	Mixed methods (blinded controlled observational diagnostic accuracy)	ED	Unspecified AI	Decisionmaking	Judge preference of trauma management	97	Ratio of judge preference	Y
Deleger et al 60	2013	Retrospective diagnostic accuracy	ED	NLP and ML	Triage	Risk stratify appendicitis	2100	Recall, precision	N
Dipaola et al 61	2018	Observational (Diagnostic accuracy study)	ED	NLP	Diagnosis; Imaging	Detecting traumatic pediatric elbow joint effusions	901 images (882 patients)	Sensitivity, specificity, AUC	N (0.915 accuracy for fellow versus 0.907 accuracy for model)
England et al 62	2018	Diagnostic accuracy	ED	Deep convolutional neural network	Diagnosis; Imaging	Detecting traumatic pediatric elbow joint effusions	901 images	Sensitivity, specificity, AUC	N/A (physician used as gold standard)
Farahmand et al 63	2017	Prospective observational accuracy	ED	Multiple ML algorithms including association rules, decision trees, clustering, LR, NN, naive Bayes	Triage	Acute abdominal pain	215 (150 training, 75 test)	Accuracy	N/A (physician used as gold standard)
Forberg et al 64	2012	Retrospective diagnostic accuracy	Ambulance/critical care unit	ANN	Prediction	Predicting ST‐elevated myocardial infarction with ambulance ECGs	560	PPV, NPV, AUC	Y
Heden et al 65	1997	Retrospective diagnostic accuracy	ED	ANN	Diagnosis: Non‐imaging	Diagnose acute myocardial infarction	11,572	Sensitivity, specificity	Y
Huesch et al 66	2018	Diagnostic accuracy	ED	NLP	Diagnosis; Imaging	Diagnosis of PE	1133 images	Sensitivity, specificity, PPV, misclassification rate	N/A (physician used as gold standard)
Hwang et al 67	2019	Diagnostic accuracy	ED	DL	Diagnosis, Imaging	Diagnosis of chest X‐rays	1135 patients	AUC, sensitivity, specificity	Undetermined (radiology residents showed lower sensitivity but higher specificity)
Jenny et al 68	2015	Retrospective diagnostic accuracy	ED	Multiple ML algorithms including RF, LDA, CART, FDA, BA, NDA	Prediction	Prediction of mortality, acute morbidity, and acute infections	1278	Predictability, AUC	Y
Lammers et al 69	2003	Prospective cohort	ED	ANN	Prediction	Prediction of traumatic wound infection	5084	Sensitivity, specificity, PPV, NPV	Y
Lindsey et al 45	2018	Prospective diagnostic accuracy	ED	Deep neural network	Diagnosis: Imaging	Detection of fractures	135,409	Sensitivity, specificity	Y
Livingstone et al 70	2019	Diagnostic accuracy (cross‐sectional)	ED	Unspecified ML with ANN training	Diagnosis; Imaging	Diagnose otoscopic images	1366 images	Accuracy	Y
Ni et al 54	2019	Retrospective diagnostic accuracy	ED	NLP	Other: Research	Eligible patient identification	202,795	Workload; efficiency	Y
Olsson et al 71	2006	Retrospective diagnostic accuracy	ED	ANN	Diagnosis: Non Imaging	Diagnosing acute coronary syndrome	4000 (3000 training, 1000 test)	Sensitivity, specificity	Y
Pruitt et al 72	2019	Diagnostic accuracy	ED	NLP	Diagnosis; Imaging	Extract characteristics of subdural hematoma from head CT reports	643 images	Accuracy	N/A (physician used as gold standard)
Sinha et al 44	2001	Prospective	ED	ANN	Diagnosis: Imaging	Diagnosis intracranial hemorrhages in closed head injuries	351	Sensitivity, specificity	Y
Sjogren et al 73	2016	Controlled trial	ED	SVM	Diagnosis; Imaging	Diagnose abdominal free fluid	20	Sensitivity, specificity	N/A (physician used as gold standard)
Somoza et al. 74	1993	Prospective cohort	ED	ANN	Prediction	Prediction to admission to inpatient psychiatry	658	Percent agreement	N/A (physician used as gold standard)
Spangler et al 75	2019	Prospective cohort	Out‐of‐hospital	Multiple ML algorithms including regularized LR, SVM, RF, GB, deep neural networks	Triage	Triage out‐of‐hospital patients into risk scores	38203	Concordance	Y
Xue et al 18	2004	Before‐and‐after	Out‐of‐hospital	Supervised ML–ANN	Diagnosis, Non‐Imaging	Diagnose acute MI	1902 ECGs	Sensitivity and specificity of a physician's judgment	Y in conjunction with physician's judgment

ANN, artificial neural network; AUC, area under the curve; ECG, electrocardiogram; ED, emergency department; MI, myocardial infarction; ML, machine learning; N/A, not applicable; NLP, natural language processing; NPV, negative predictive value; PPV, positive predictive value.

The majority of included studies were retrospective diagnostic accuracy designs (n = 124, 82.7%), with several prospective cohorts (n = 16, 10.7%) and before‐and‐after implementation designs (n = 4, 2.7%). Only 3 of the included interventions were evaluated using prospective controlled trials, 2 used simulation modeling, and one used a case‐control design.

We found that 48 (32.0%) studies did not have a comparator to their AI intervention, and 27 (18.0%) studies used non‐AI statistical models. Other studies compared against clinical decision tools (n = 17, 11.3%), standard of care (n = 4, 2.6%), or cutoffs from literature or simulations (n = 3, 2.0%). In 27 (18%) studies, AI tools were compared against one another. A total of 24 (16.0%) studies directly used humans (eg, physicians; EMS staff) as comparators.

Intervention characteristics

AI utilization categories in emergency medicine are outlined in Figure 2. The majority of interventions centered around prediction, with 72 (48.0%) studies analyzing the predictive capabilities of AI (Table 3). Thirty‐seven (24.7%) interventions aimed at improving diagnosis within the ED. Eighteen (12.0%) interventions focused on triage of emergent conditions and another 3 (2.0%) interventions focused on medical decisionmaking. Seventeen (11.3%) studies demonstrated that AI can assist with organizational planning and management within the ED. An additional 2 (1.3%) studies used AI within a research recruitment context. Among the 150 studies, 19 (12.7%) focused on diagnostic imaging within the ED. A total of 16 (10.7%) studies were conducted in the out‐of‐hospital environment (eg, EMS, paramedics) with the remainder occurring either in the ED or the trauma bay. The breakdown of utilization categories is outlined in Figure 3. Furthermore, we found that 70 (46.7%) studies evaluated supervised ML interventions, 2 (1.3%) studies evaluated unsupervised ML interventions, and 14 (9.3%) studies evaluated NLP interventions (Figure 4).

FIGURE 2

Purpose of intervention in emergency medicine. ED, emergency department; EMS, emergency medical services

FIGURE 3

Artificial intelligence utilization category breakdown in emergency medicine

FIGURE 4

Types of artificial intelligence interventions in emergency medicine. NLP, natural language processing

Of the 24(16%) studies that had human comparators (Table 5), 8 (33.3%) studies defined the performance of clinicians as the gold standard of care. The remaining 16 (66.7%) studies directly compared human versus AI performance. There were 12 (50%) studies in which AI interventions outperformed clinicians. In these studies, AI interventions were better able to diagnose acute cardiac events (including out‐of‐hospital cardiac arrest and myocardial infarct), identify hyperkalemia, risk stratify patients in triage, identify participants, predict wound infection, predict mortality, predict patients for clinical trials, and read imaging (including intracranial hemorrhages, otoscopic imaging, and fractures).They were non‐superior to humans in 3(12.5%) studies that investigated triaging acute abdominal pain, detecting traumatic elbow effusions, and diagnosing chest X‐rays. In one study, AI combined with physician judgment was superior to physician judgment alone when diagnosing myocardial infarctions via electrocardiograms (ECGs).

Limitations

Limitations of our review are namely due to the emerging nature of the evidence base. It is difficult to synthesize conclusions because of the heterogeneous study designs and interventions included in our review. In addition, because of the selective reporting of some studies and the lack of transparency regarding the modeling, it was not often possible to adequately critique the methodology of the studied interventions. Once more AI‐related research is established, future systematic reviews may wish to assess more specific interventions in order to determine superiority and areas of improvement. Other limitations of our scoping review were that we examined only studies published in the English language, and we did not analyze patents or grey literature. Only studies that were explicitly self‐classified as AI by the study authors were eligible for our analysis, and as such, studies that do not directly identify themselves as AI in their titles and abstracts may have been missed by our search strategy.

DISCUSSION

Overall, we found that AI interventions in the ED are heterogeneous in both purpose and design. For example, supervised ML interventions included prediction models for pediatric asthma exacerbation, prediction of return visits, and stroke diagnosis. , , NLP models were used to optimize resource allocation in low‐resource settings, classify computed tomography (CT) imaging, and predict hospital admission using electronic medical records (EMR). , , There also appears to be rapidly growing interest in the varied opportunities for AI, as most studies were published in the last 5 years. We can expect an additional surge of AI‐related publications, as over 100 conference proceedings were excluded from our analysis.

The majority of interventions (72, 48.0%) centered around prediction, which aligns with the proposed superiority of AI in prediction modeling. Several studies showed AI outperformed existing decision tools and scoring systems that were originally derived using traditional statistical modeling. Examples include the superior ability of AI to predict mortality in pneumonia as well as calculate syncope risk based on clinical criteria. , One explanation is that AI may be superior to humans in predictive modeling because of the ability to process multiple variables simultaneously across large data sets. Several of our included studies showed superiority to human comparators when balancing different data points to predict complex outcomes.

As previous studies have shown, human decisionmaking is subject to potential biases and heuristics. , AI could mitigate illusory correlations and metacognition errors in medicine. For example, AI superiority in predictive modeling is hypothesized to be particularly useful in diagnosis of sepsis, a syndrome that can result in widespread organ dysfunction and high morbidity and mortality. In our review, we found that 6 large cohorts took advantage of big data to predict sepsis and mortality using ML, 4 of which were multicenter studies. , , , , , The 5 studies found that ML models improved prediction among suspected sepsis patients in the ED compared to traditional tools, such as the Quick Sequential Organ Failure Assessment (qSOFA) or other early warning scores.

A total of 17(9.4%) studies used AI in the out‐of‐hospital setting. Examples of studies in the out‐of‐hospital environment included demand forecast for allocation of ambulances, classification of out‐of‐hospital ECGs, and screening of EMS calls to recognize cardiac arrest. , , Several ML algorithms used EMS data to predict outcomes for out‐of‐hospital cardiac arrest. , , , Another intervention used supervised ML to automatically link EMS electronic patient care reports to ED records. The out‐of‐hospital setting presents a unique setting where limited clinical variables are used to make prompt decisions (for example, whether or not to transport to hospital). This is well suited to be tackled by AI given its predictive power and ability to use various data points and predict outcomes. As such, we feel that AI will likely have a significant impact in the out‐of‐hospital setting. Further, interventions in the out‐of‐hospital environment could promote interdisciplinary communication, reduce inconsistencies, and improve outcomes upon arrival to hospital.

An emerging area of interest was radiology‐focused AI interventions, with 19 (12.7%) studies evaluating methods to improve imaging‐based diagnosis in the ED. Of the 19 radiology studies, a total of 11(57.9%) had human comparators. For example, Sinha et al (2001) used artificial neural networks to detect intracranial hemorrhage on CT, which was more sensitive (82.2%) compared with physician prediction (62.2%). In contrast, Lindsey et al used an implementation approach to combine their deep learning model with clinician readings that improved fracture detection in comparison to clinicians working alone. Previous literature has noted that radiology is particularly amenable to AI interventions because of its technology‐driven interface, reliance on pattern recognition, and relative wealth of data sets. , As advances in automatic lesion detection and segmentation continue, we can expect further radiology studies with a focus on emergency medicine applications.

Several studies had implications for the ED beyond clinical decisionmaking. Although much of the current speculation regarding AI has centered on direct patient care, our review shows an emerging interest in ED operations planning, research, and medical education. A total of 18 (12.0%) of our included studies demonstrated that AI can assist organizational planning and management within the ED, including optimization of nursing staff hours, patient satisfaction, and resource planning. Six studies used ML to predict daily patient volume and flow within the ED, including daily trauma volume. , , , , , These interventions have the potential to reduce ED wait‐times through improved resource allocation and policy planning. Another 2 (1.3%) studies used ML to increase the efficiency of patient identification for ED clinical trials and research, with the goal of allowing research to be more accessible and standardized in an often fast‐paced environment. ,

To our knowledge, this is the first study to systematically review the body of literature regarding emergency medicine and AI. Our scoping review complements the previous literature reviews that have been conducted regarding AI in the ED. For example, Stewart et al (2018) noted that AI had diverse applications in the ED, including use for clinical image analysis, monitoring, and outcome predictions. Their narrative assessment noted that most studies used retrospective data sets, which was corroborated by our systematic search. Stewart et al also noted the lack of universally recognized and standardized reporting guidelines for ML, providing a rationale for why we opted to use a scoping review to better characterize the heterogeneous literature.

Although AI appears superior to clinicians regarding predictive modeling, the current body of evidence still remains uncertain. Most studies identified did not involve a human comparator and lacked information on safety‐oriented outcomes. The majority of included studies were retrospective analyses of data sets and require further validation in controlled clinical trials. In addition, most studies do not discuss the practical components associated with technology implementation, such as the convenience, training, or costs required. One obstacle to reproducibility and reliability of results is the proprietary nature of algorithms, as many interventions are not publicly available or transparently described.

Owing to the diverse and rapidly changing field of AI, a scoping review was the most suitable methodology to systematically search and evaluate the available literature. The flexible nature of the search allowed us to include a heterogeneous array of study designs and broad intervention types in order to best map the available evidence. Our scoping review followed PRISMA‐ScR guidelines and systematically examined a combination of medicine, allied health, and computer science databases for a concise snapshot. Whereas previous literature reviews have hypothesized different opportunities for AI implementation in emergency medicine, ours is the first to examine the available literature at a glance for physicians to understand the current field. Given the rapid development of AI technology, this review is a timely cross‐section of available research. Other strengths include our 2 independent parallel reviewers, our high interrater agreement, and our analysis of interdisciplinary databases.

In comparison to prior literature reviews, our study is the first to systematically examine the scope of AI‐related research in the ED. In addition, previous studies did not include out‐of‐hospital interventions, such as the role of AI in improving EMS or paramedic services. Other gaps in prior work include lack of information regarding the role of AI in medical education or research, as well as operations focused studies such as prediction of ED volume. Lastly, we are also the first study to categorize studies where AI interventions were balanced against human comparators in ED.

By examining the breadth of the literature, it is clear that AI shows strong promise in improving outcome prediction in the ED. AI showed superiority over human comparators in several areas, particularly when analyzing large data sets and rapidly fluctuating variables. Further research should be conducted to determine further opportunities for predictive modeling within the ED, and particularly with comparisons to existing standards of care. Studies should also consider comparing different types of ML for improved accuracy.

For effective AI implementation within hospital systems, AI‐related research must progress beyond proof‐of‐concept. It remains difficult to determine how AI will be adopted within existing systems, as most studies do not compare their interventions to existing standards of care or human comparators. Additional challenges surrounding AI include whether physicians and healthcare staff will have difficulty interfacing with AI‐based tools, and whether errors will occur as a result of poor technological literacy. Similarly, there have been concerns regarding both physician and patient uptake of AI, particularly the lack of trust in “black box” technologies that are not clearly understood. , , Specific challenges include mistrust in external validity of data sets, inability of computerized tools to understand clinical context, or mistrust in programmed correlations. , , , As such, further research must be conducted regarding both physician and patient perspectives towards implementation and ethics of AI. Future research must involve prospective controlled trials in order to determine true superiority, in addition to assessing costs, feasibility, and integration.

CONCLUSION

AI‐related research is rapidly increasing in emergency medicine. Studies show promising opportunities for AI in diverse contexts, particularly regarding predictive modeling for patient outcomes. However, there remains uncertainty regarding their superiority over standard practice, and further research is needed before clinical implementation.

CONFLICTS OF INTEREST

The authors have no conflicts of interest to disclose.

FUNDING INFORMATION

The authors have no sources of funding to disclose.

AUTHOR CONTRIBUTIONS

AK, AT, and SM conceived the study. SM supervised the conduct of the scoping review and data collection. AK and AT completed both abstract and full‐text screening. AK and SK completed data extraction and analysis. AK, AT, SK drafted the article, and all authors contributed substantially to its revision.