Under-triage: A New Trigger to Drive Quality Improvement in the Emergency Department.

The emergency department (ED) is a care setting with a high risk for medical error. In collaboration with our nursing colleagues, we identified a new trigger, under-triage, and demonstrated how its implementation could detect and reduce medical errors in the ED.
Methods: We defined under-triage as patient visits with an Emergency Severity Index (ESI) score of 4 or 5 (ie, low acuity), and the patient was admitted to the hospital during the same visit. We defined mistriage, or medical error, when nurse-physician dyad reviewers determined that a different ESI level should have been assigned based on the information available at triage. A multidisciplinary team used nominal group technique to build consensus on key drivers and outcome metrics for this new trigger. We randomly selected 267 charts for review utilizing the under-triage trigger.
Results: Of the 125,457 patients triaged as level 4 or 5 in 2019 and 2020, 1.1% (n = 1,423) were under-triaged. Of the 267 charts reviewed, 127 were categorized as mistriage, making the under-triage's positive predictive value trigger 48%. Reviews took 2-10 minutes per chart. We identified 10 categories of under-triage. Nine themes emerged, with four specific and measurable action items mapped to process and outcome metrics. Conclusions: We identify a new, feasible ED trigger, under-triage, that identifies medical error with a high positive predictive value. We identify process and outcome metrics and interventions to improve triage for future patients.

INTRODUCTION

One hundred thirty million people sought care in the US emergency departments (EDs) in 2018.[1,2] Emergency care is complex and characterized by nonlinear processes, frequent interruptions, and constant re-prioritizing tasks. Information on which to base diagnosis and therapies may be limited and change over time during a patient visit. As a result, the ED is a care setting with a high risk of medical errors.[3]

Medical error is an act of omission or commission in planning or execution that contributes or could contribute to an unintended consequence.[4] It is a deviation from the care process, which may or may not harm the patient. Because medical error includes faulty processes that can lead to error, whether or not that error occurred, medical errors are difficult to detect. Yet, reliably detecting a medical error is important to improve patient safety systematically.[5-8]

One method of discovering medical errors is to use specific clinical events to trigger chart review. Triggers are clinical events that make it more likely that a medical error has occurred. Commonly used triggers, such as ED revisits with admission, and transfers to intensive care units (ICUs) after inpatient admission, are sensitive but not specific and lack positive predictive value (PPV) in identifying medical errors.[9,10] For example, the PPV of ED revisits with admission is approximately 4%; thus, reviewing 24 charts may detect one medical error.[11] The ED lacks an efficient, high-yield trigger for identifying medical errors.

In collaboration with our Nursing colleagues, we propose a new trigger, under-triage. Triage is “the initial assessment and sorting of patients in an emergency setting to determine the clinical priority of need.”[12] Under-triage is the underestimation of the severity of illness during the triage process. At the start of the patient encounter, triage sets the stage for the entire clinical visit. Patients triaged to lower acuity pods, often in locations distant from the main ED, can be at risk of harm. “Geography is destiny”[13] and can contribute to several types of cognitive bias, including anchoring bias, premature closure, and overconfidence.

We performed this study to assess qualitatively and quantitatively the under-triage trigger as a potential quality improvement tool. The qualitative assessment included identifying broad themes and actionable errors to reduce under-triage. The quantitative assessment included a determination of the feasibility of the under-triage trigger and the PPV in identifying a medical error in the dynamic ED setting.

METHODS

Context

Study Setting

We conducted the study at an urban, academic, tertiary pediatric ED with approximately 90,000 annual visits. Cerner (Cerner Corporation, Kansas City, Mo.) is the primary electronic health record (EHR) in the ED and across the hospital system. Similar to many EDs, we use a two-stage triage process. First, a nurse employs the Emergency Severity Index (ESI) to assign an initial triage score based on a brief history and a rapid assessment of clinical appearance, the ABCs (airway, breathing, and circulation), and the presence or absence of high-risk chronic conditions, such as sickle cell disease or cancer.[14] This initial “sort” takes approximately two minutes. A second nurse subsequently completes a full nursing assessment, including an expanded history, vital signs, medical history, current medications, and targeted physical exam. Finally, the second nurse confirms or changes the ESI score based on the information obtained. Time from pivot to assessment generally ranges from 5 to 15 minutes. However, the time-lapse can be up to one hour during volume surges.

Patients with low-acuity illnesses who require 0 or 1 resource are assigned an ESI score of 5 or 4, respectively. A resource is an intervention or diagnostic tool excluding oral medication or prescriptions. Patients are upgraded to ESI 3 if they need more than one resource (eg, blood draw plus x-ray) or ESI 2 if they need urgent attention (eg, severe pain or physiologic instability). Patients with an ESI 1 are in extremis, and these patients bypass the second stage of the triage process.

Approximately 55% of our ED patients receive an ESI score of 4 or 5 on arrival.[15] This proportion is typical in urban, academic tertiary care pediatric EDs, where the fraction of low-acuity patients ranges from 30 to 60%.[16,17] These patients are assigned to a separate location in the ED, conceptualized as a lower acuity pod. A lower ESI score often results in longer waiting times as patients compete with higher acuity ESI patients in the queue.[2,15,18,19]

Study Population

We defined under-triage as a patient visit in which the patient received an ESI score of 4 or 5 and was admitted to the hospital. We queried the EHR for under-triaged patients under 21 years of age presenting to the ED from January 1, 2019, to December 31, 2020. We excluded from chart review those patients with missing nursing assessment notes or missing physician documentation.

Intervention

We formed a multidisciplinary panel of ten physicians and nurses with expertise in pediatric emergency medicine, medical education, quality improvement methodology, and informatics. Two of our investigators with formal experience in nominal group technique introduced the technique to the panel. The nominal group technique is a well-established, multistep, structured, facilitated technique for group brainstorming to reach a consensus.[20] The nominal group technique can improve agreement in clinically complex cases with varying provider approaches.[21]

The multidisciplinary team met monthly. Reviews were conducted asynchronously in nurse-physician dyads in preparation for the monthly meetings. With nurse educators and junior nurses on the team, the meetings allowed senior nurses to share clinical pearls with junior nurses and for physicians to better understand the triage process.

Team members initially individually reviewed a total of 100 under-triaged patient charts. Using the nominal group technique, the team identified general themes and specific categories of under-triage. We further refined key clinical categories at group meetings. The dyad discussed discordant reviews until reaching a consensus. We recoded charts where appropriate. We intended to apply this trigger tool technique to an additional 300 charts; random selection using the “random” module in Python v3.7 (Python Software Foundation at python.org) included charts from all months of the year. The computerized selection process included some duplicate charts; after exclusion, 267 charts remained. Team members independently coded these charts in nurse-physician dyads using the agreed-upon categories. Again, the dyad discussed discordant reviews until reaching a consensus.

We defined mistriage or medical error when reviewers determined that a different ESI level was more appropriate based on the information available during the triage process. This definition specifically excluded cases in which the medical provider elicited more detailed medical information, and reviewers felt this was an appropriate difference between nursing assessment and medical assessment or when reviewers determined that hospital admission was highly discretionary and, therefore, unexpected by a reasonable and prudent triage nurse. To define medical error conservatively, we excluded cases in which the patient’s clinical condition changed between triage and medical assessment since we could not determine post hoc whether this was an inaccurate initial nursing assessment or simply evolution of the condition (eg, wheezing). Therefore, we did not include patients with an ESI change from 5 to 4 among the mistriaged. Team members identified themes related to the underlying reasons for under-triage and then identified potential interventions to address these themes. Interventions were discussed initially by each physician-nurse dyad and then presented to the larger group. The nominal group technique resulted in four high-yield, actionable interventions to create the key drivers.

Ethical Considerations

The Institutional Review Board approved this study.

RESULTS

Qualitative Results

Themes

Recurrent themes emerged during consensus reviews, including the need for heuristics in triage, failure to incorporate recent medical encounters into triage decisions, and failure to synthesize historical or objective patient information when assigning ESI scores in triage.

Experienced nurses shared heuristics with the panel, reinforcing the value of experienced nurses in triage. For example, a senior nurse educator noted that a patient with RLQ pain and fever should be assigned an ESI level 3 for a more extensive workup despite the high prevalence of gastroenteritis in the pediatric population.

Another theme emerged from the lack of review of referrals from the medical home or transfers from outside hospitals when collecting medical history in triage. Because referred patients have been prescreened to need an ED visit, they often require a higher level of care. Closely related is a failure to leverage the electronic medical record (EMR) information before determining the ESI score. Most EMR systems allow users to see a snapshot of pre-existing medical conditions.

A recurrent theme in several charts was a failure to synthesize the history of objective patient information when assigning ESI scores. Triage occurs in a busy environment with rapid patient processing. The EMR is a double-edged sword; precompleted sentences and checkboxes may allow easier documentation, but these tools, coupled with the need for extended documentation and the pressure to move to the next patient, may hinder critical thinking. Several charts included grossly abnormal vital signs obtained and unappreciated in triage.

See Table 1 for a full list of themes.

Table 1.

Themes of Under-triage

Theme of Under-triage	Examples
Anchoring bias	An 11 y old with multiple prior visits for encopresis, presented with abdominal pain, assigned ESI 4, admitted for pancreatitis
Need for heuristics in triage	Rules of Thumb, or “If I see this, then I do this” may benefit triage nurses. eg, “If patient failed 7-day course antibiotics, assign no less than ESI 3.” Additional examples include: “RLQ abdominal pain with fever should be no less than ESI 3” despite high prevalence of gastroenteritis in the PED
Recent medical history	Examples include a patient who failed outpatient oral antibiotics for facial infection sent into ED with an unimpressive physical presentation assigned a low-acuity ESI status. Another example of medical history is leveraging the information in the EMR before determining ESI. Most EMR systems allow user to see a snapshot of pre-existing medical conditions
Nurse experience	Senior nurses have well developed heuristics and critical thinking skills
Staffing challenges	Patients are better served by an initial quick look triage and second, more thorough exam, by a different assessment nurse to confirm ESI level. Staffing challenges may require assessment by the same nurse and thus may contribute to confirmation bias
Knowledge gaps	The triage role provides for many educational opportunities for new staff. Novice nurses in triage may be more likely to under-triage patients at the start of their professional career
Lack of complete physical exam in assessment	Conducting a thorough physical examination in triage can be challenging in the ED triage setting and therefore may be deferred. Lacerations should be unwrapped and examined for bone exposure. Foreign body sites should be carefully examined. Deferred GU physical exam can cause care delays
Challenge of critical thinkingin distracting environment	EMR contributes to lack of critical thinking and information synthesis. The need for rapid patient processing, the prevalence of checkboxes, and precompleted text leads to automated documentation. Task burden in marking required checkboxes contributes to failure to notice abnormal vital signs. An example of error is entering the patient’s temperature as the weight
Language barriers	Language barriers create miscommunicationsin triage

Categories of Under-triage

The expert panel identified twelve initial under-triage categories clustered into ten categories on review. We aimed to keep categories as specific as possible to facilitate actionable process metrics. The panel immediately recognized that some under-triages are not medical errors and do not involve mistriage.

See Table 2 for under-triage categories.

Table 2.

Categories of Under-triage

	Categories of Under-triage	Explanation
Nonmedical error
1.	Disease progression	Patient’s clinical status evolved between the time of triage and the time of physician assessment of patient
2	Provider obtains more information	The provider may obtain more information as part of the medical history, physical examination, and ancillary tests. (ie, x-ray results that impact disposition). This category is distinct from error because the goal of triage is distinct from that of the provider
3	Disclosure	Patient does not disclose important information relevant to medical visit during triage. An example is failure of caregiver to disclose that patient is not immunized
4	Practice pattern variation	Discretionary patient admission that could not be predicted by a prudent triage nurse. For example, a 5-month-old presents with caretaker stating child will not move arm. Patient symptoms resolve in the ED. Provider admits patient for rule out transient ischemic attack
5	Other	Insufficient information in chart to determine cause of under-triage
Medical error
6	Discrepancy in examination or history between sort RN and assessment RN	A discrepancy during the triage process. The assessment nurse downgrades the ESI level inappropriately
7	Discrepancy between chief complaint and physical examination	Mismatch between patient complaint and physical exam findings. For example, 3-year-old presenting with complaint of bruise to arm. MD noted multiple bruises and diagnoses nonaccidental trauma
8	English language proficiency	Language barriers contributing to missing information in triage and no documented evidence of interpreter
9	Recent medical encounter	This includes revisit within 72 h, several revisits within the week, and referrals from primary providers or ambulatory clinics
10	Failure to synthesize historic or objective information	Relevant past medical history not obtained in triage or was obtained but the significance not appreciated when assigning ESI score. Vital signs or relevant physical examination findings not obtained or were obtained but significance not appreciated when assigning ESI score

Quantitative Results

During the 2-year study period, 125,457 patients with ESI 4 or 5, of whom 1,423 (1.1%) had a disposition of hospital admission (Fig. 1). Mistriage (medical error) was detected in 127 of 267 reviewed charts, producing a PPV of 47.6%. See Table 3 for patient and visit characteristics.

Fig. 1.

Flow diagram of the study population.

Table 3.

Patient and Visit Characteristics

Factor	Characteristic	n (%)
Age	Mean (SD) y	6 (5.6)
Race/ethnicity	NH –Black	143 (55.9%)
	Hispanic	70 (27.3%)
	Other/unknown	29 (11.3%)
	NH-White	14 (5.5%)
Season	Winter	105 (41%)
	Spring	63 (24.6%)
	Fall	48 (18.8%)
	Summer	40 (15.6%)
Diagnosis	Pneumonia and bronchiolitis	26 (10.2%)
	Other gastrointestinal	24 (9.4%)
	Fever and upper respiratory infection	22 (8.6%)
	Other infections	22 (8.6%)
	Musculoskeletal	19 (7.4%)
	Dehydration and malnutrition	16 (6.3%)
	Infections of eye and ear	16 (6.3%)
	Appendicitis	15 (5.9%)
	Neurologic	15 (5.9%)
	Genitourinary	14 (5.5%)
	Fractures	10 (3.9%)
	Psychiatric disease	10 (3.9%)
	Asthma	9 (3.5%)
	Foreign body	9 (3.5%)
	Immunologic	9 (3.5%)
	Other	9 (3.5%)
	Skin and soft tissue	8 (3.1%)
	Blood disorders	3 (1.2%)

Of the 10 categories comprising mistriage, the most common were failure to synthesize historical or objective information (eg, vital signs) (35.7% of charts), discrepancies between the chief complaint and medical exam (7.1%), failure to note recent medical encounters (2.1%), and discrepancies between the sort nurse and the assessment nurse (1.5%). In addition, a lack of English language proficiency associated with mistriage occurred in 0.8%. Additional causes of under-triage, not deemed medical error, included providers obtaining additional relevant information (17.7%), practice pattern variation (13.3%), progression of disease (12.9%), and others (7.6%).

Team members reported it took 2 to 10 minutes to review each chart, making this process feasible for multiple chart reviews. The programming required to identify under-triaged patients from the EMR took less than 60 minutes.

Development of Quality Improvement Targets

We designed the following key driver diagram based on qualitative and quantitative analysis to target QI improvements moving forward (Fig. 2).

Fig. 2.

Key driver diagram. Mistriage definition: experienced reviewers determine a different ESI level should have been assigned based on information available during triage process.

The global aim is to decrease mistriage, and the SMART aim is to decrease the proportion of mistriaged patients by 50% in 12 months. Statistical process control charts, such as a P chart, can track improvements over time. This methodology is possible because the team codes each medical chart by type of under-triage error, so the proportion of charts with errors by category is monitored over time.

The themes provided the content for creating the key drivers. We identified four high-yield, actionable themes of contributions to medical error: incorporation of recent medical encounters into triage decisions, synthesis of historical or objective patient information when assigning ESI scores, heuristic teaching, and addressing barriers related to limited English proficiency.

Identified Interventions to address each key driver:

#1 Incorporate Recent Medical Encounters into Triage Decisions

The team devised several interventions to improve the process of incorporating recent medical encounters into triage decisions. Triage education can be revised based on chart findings.[22] Email reminders can be sent out bi-weekly and include examples from real patients. Different prompts introduced into the EMR can encourage the review of prearrival data.[23] One example is a prearrival checkbox labeled “prearrival reviewed or not applicable.”

#2 Synthesize Historical or Objective Information when Assigning ESI Score in Triage

The ability to synthesize important medical history or vital signs in triage can be challenging with rapid patient processing in the busy triage environment. One intervention reinforces the STAR (stop, think, act, review) safety behavior through reminders at team meetings.[24] Another intervention is to share examples of missed critical vital signs or missed chronic medical conditions from chart review regularly through weekly newsletters.[25] Interventions should include efforts to remove distractions from the triage process. Because the task burden of assessment in triage is diffuse, standardizing the information required in triage would reduce the work burden and allow for more critical thinking.[26]

#3 Heuristic Teaching

Interventions to improve the use of heuristics among triage nurses include partnering senior nurses with training nurses.[27] Additionally, a list of common heuristics can be developed over time and shared with orienting staff.

#4 Address Limited English Proficiency

Interventions to decrease mistriage caused by limited English proficiency include professional interpreter staffing in triage and increased use of interpreter phones when professional in-person interpreters are unavailable.[28] “Swarming” behavior is a promising intervention, for example, physician and nurse going in together to obtain vital signs, history, and exams on low-acuity patients. Swarming is more efficient, but it makes it less likely for triage to occur without an interpreter.[29]

DISCUSSION

Summary

This study introduces a new trigger tool to detect a medical error in the pediatric ED. This report is the first to describe under-triage as an indicator of medical error and the first use of under-triage to direct improvement efforts. We believe the under-triage trigger is a promising trigger tool with a PPV of 48% in identifying mistriage. Our qualitative analysis identified key drivers and outcome metrics to demonstrate how the trigger can direct improvement efforts

Interpretation

Measuring medical error is challenging. Unlike diagnostic error, which is a failure to establish an accurate and timely explanation of the patient’s health problems or communicate that explanation to the patient, medical error captures a broader framework of potential adverse outcomes, whether or not those outcomes occur.[4,30] This fact makes medical errors challenging to find and measure. But capturing medical error is important because early identification of faulty processes is an opportunity to avoid potentially adverse outcomes and improve patient safety. The under-triage trigger identifies instances of mistriage, a type of medical error, and facilitates its measurement.

Mistriages are medical errors. Triage occurs at the start of the patient encounter and sets the stage for the entire visit. Mistriaged patients are sent to lower acuity pods, often in locations far from or external to the ED, where they are at risk of misdiagnosis or delayed diagnosis. Our work was driven in part by recent cases at our hospital. On arrival, several patients were mistriaged, sent to a distal pod, and experienced delayed diagnosis and treatment. Learning how these mistriages occurred is important if we are to prevent their re-occurrence.

Mistriage can contribute to several types of cognitive bias, including anchoring bias, premature closure, and overconfidence. Mistriaged adult patients have significantly longer ED stays than those correctly triaged.[31]

Our work builds on previous literature exploring triggers to improve patient safety in the ED and, specifically, advance medical error detection methods. The PPV of 48% for under-triage compares favorably to other ED-based triggers. The trigger of ED revisits with admission within 72 hours of index visit demonstrates a PPV of 4.6%, according to DePiero et al.[32] Similar results were reported by Aaronson et al.[33] They concluded that 72-hour ED returns have a low yield in identifying suboptimal care, as less than 3% of cases represent deviations from standard care.[33] Alessandrini et al[11] noted a 3.5% return visit rate and the inefficiency of admissions after revisits as a trigger for evaluating ED performance. Radiology callbacks, a potential quality indicator, are a subset of revisits and a potential ED quality indicator. Rajan et al evaluated the clinical impact of discordant radiology reads. They reported 6.6% to 34% in the literature for discrepancies resulting in a negative impact or major alteration in the treatment plan.[34]

Other ED-based triggers lack quantifiable metrics. For example, transfer to an ICU within 6 hours of inpatient ward admission is a common trigger, but to our knowledge, the usefulness of this trigger is unpublished. Furthermore, the criteria for upgrading from floor to ICU vary widely. For example, in our ED, transfer to the ICU within 12 hours has a PPV of 8%–12%. (Brown KM, unpublished data). Other examples include cases referred to division chiefs or medical directors, cases from risk management, or cases discussed at morbidity and mortality conferences. Other promising triggers, such as high-risk conditions based on symptom-disease dyads, await further development.

Our trigger tool has quantifiable metrics. The outcome measure is the decreased proportion of mistriaged patients. Each category of mistriage is a process measure monitored on control charts for improvement(see Fig. 2). Several themes underlying mistriages yielded practical interventions.

Limitations

Our study has several limitations. First, findings from consensus methods rely on panel members’ opinions. We mitigated this limitation by creating a multidisciplinary panel, meeting iteratively to improve learning, and purposely defining mistriage conservatively. We ensured that experienced triage nurses and nurse educators comprised half the panel. Second, retrospective reviews are subject to hindsight bias. We sought to mitigate this by meeting as nurse-physician dyads and as one committee to voice all hindsight opinions. Third, we explored the under-triage trigger at a single tertiary care institution. Our institution has an established EMR and access to EMR analysts to extract data and create clinical decision support tools. We have a quality culture that enabled us to turn the multidisciplinary panel into a standing committee that meets on an ongoing basis. Furthermore, we enlisted data analysts to pull chart demographics into a redcap survey to facilitate ongoing chart review. These interventions may not be generalizable to institutions with limited resources. Fourth, we could not find other published data quantifying the cost of mistriage to the patient or healthcare system. We suspect this is due to mistriage being a new trigger and intend to conduct further studies to investigate the cost.

We created a reliable methodology to capture medical errors occurring at the start of the patient encounter in the ED. Our approach is feasible and can be easily translated, with center-specific modifications, to other EDs. Although chart review is required, the process is manageable and takes under five minutes per chart.

Our next steps include ongoing monthly chart review by nurse-physician dyads and monthly team meetings to identify the mistriage errors, focusing on implementing the directed interventions to improve patient safety. We intend to explore the cost of mistriage to patients and the healthcare system.

CONCLUSIONS

In this single-site study, under-triage is a useful trigger for chart review and better identifies medical errors than other common ED-based triggers. Systemic investigation of under-triaged patient charts provides an innovative and high-yield approach to identifying knowledge gaps and quality improvement opportunities. Moreover, the methodology is conducive to the measurement of changes over time. The under-triage trigger could be leveraged to monitor and improve ED performance with additional development and implementation.

Disclosure

The authors have no financial interest to declare in relation to the content of this article.