Defining quality indicators for emergency care delivery: findings of an expert consensus process by emergency care practitioners in Africa.

Facility-based emergency care delivery in low-income and middle- income countries is expanding rapidly, particularly in Africa. Unfortunately, these efforts rarely include measurement of the quality or the impact of care provided, which is essential for improvement of care provision. Our aim was to determine context-appropriate quality indicators that will allow uniform and objective data collection to enhance emergency care delivery throughout Africa. We undertook a multiphase expert consensus process to identify, rank and refine quality indicators. A comprehensive review of the literature identified existing indicators; those associated with a substantial burden of disease in Africa were categorised and presented to consensus conference delegates. Participants selected indicators based on inclusion criteria and priority clinical conditions. The indicators were then presented to a group of expert clinicians via on-line survey; all meeting agreements were refined in-person by a separate panel and ranked according to validity, feasibility and value. The consensus working group selected seven conditions addressing nearly 75% of mortality in the African region to prioritise during indicator development, and the final product at the end of the multiphase study was a list of 76 indicators. This comprehensive process produced a robust set of quality indicators for emergency care that are appropriate for use in the African setting. The adaptation of a standardised set of indicators will enhance the quality of care provided and allow for comparison of system strengthening efforts and resource distribution.

The need for emergency care in low-income and middle-income countries (LMICs) has never been greater.

Quality assessment of emergency care delivery is essential for improvement of care provision.

Measurable indicators for the provision of emergency care in LMICs are lacking, and there is no formalised set of clinical quality indicators established and agreed on by providers and policy makers in these settings.

This manuscript proposes a set of 76 quality indicators for emergency care provision in LMICs.

These indicators were selected and agreed on by expert clinicians practising emergency care in Africa using a multiphase consensus process.

These indicators will facilitate intranational and international comparison of emergency care delivery and development.

This comparison should lead to enhanced quality and safety of care provided.

Local adaptation specific to burden of disease and feasibility of measurement will be a crucial next step.

Introduction

In low-income and middle-income countries (LMICs), the need for quality emergency care has never been greater. It is estimated that 54% of worldwide morbidity and mortality can be attributed to emergency conditions.1–3 Emergency care systems are increasingly recognised as an essential delivery platform by which to prevent a substantial portion of death and disability.4 5 However, in many locations the delivery of coordinated, quality emergency care is still in its infancy.

Measurable indicators for the provision of emergency care in LMICs are lacking; the quality of such care delivered at health facilities in these settings has yet to be addressed. In an effort to promote emergency care, the World Health Assembly passed resolution 60.22 in 2007 requesting that WHO ‘provide support to Member States for design of quality-improvement programmes and other methods needed for competent and timely provision of essential trauma and emergency care’. This resolution further urged Member States to ‘identify a core set of trauma and emergency-care services, and to develop methods for assuring and documenting that such services are provided appropriately to all who need them’.6 In support of these efforts, WHO recently established the Emergency, Trauma and Acute Care programme.7

Facility-based emergency care delivery in LMICs is expanding rapidly, but such developments rarely include measurement of the quality or the impact of the care provided. Quality assessment of emergency care delivery is the essential foundation for improvement of care provision.8 9 There is no formalised minimum set of clinical quality indicators established and agreed on by providers and policy makers in LMICs.10 National health systems seeking to monitor performance of newly established emergency care service delivery are left without context-appropriate indicators. The lack of standardisation impedes evaluation of the impact of emergency care service delivery initiatives and comparison within and between regions.

Well-established emergency care systems in high-income countries have quality improvement programmes that have had significant impact on standardisation of service delivery.11–14 Yet even in the most developed health systems, emergency care quality indicators have been difficult to link to patient outcomes.8 Where such indicators have been developed, they are usually not appropriate for low-income settings. The most frequently used indicators in high-income countries are dependent on time intervals that are difficult to capture, diagnostics (eg, time to ECG) and therapeutics (eg, tissue plasminogen activator for stroke) not readily available in LMICs, or robust systems that require multiple levels of coordination, resources and specialisation (eg, door to balloon time for acute myocardial infarction).9 15 In addition, the vast majority of quality indicators for emergency care are process indicators (activities and outputs), and are not proximally linked with improved clinical outcomes.10 12 16–19

Performance metrics with substantial underlying assumptions cannot be included in isolation as a composite indicator of quality. For example, a commonly used high-income country metric is ‘door-to-doctor time’, or the time it takes from a patient’s arrival to being seen by a decision-making provider. A number of assumptions are embedded in this indicator: (1) A provider is present with the necessary training in emergency care (2) The patient’s emergent clinical syndrome is recognised (3) The patient is resuscitated according to an established standard of care based on best evidence (4) The provider arrives at a correct preliminary diagnosis and disposition. However, the reality is often drastically different; many facilities in LMICs do not have a physician present at all hours, these physicians rarely have specific training in emergency care, clinical protocols relevant to the setting are lacking, and the evidence base for interventions in such resource-constrained settings is limited. In these settings, prioritisation of such an indicator in isolation could lead to false reassurance of appropriate clinical quality while missing the substantial morbidity and mortality that occurs due to undertrained providers and lack of physical resources.

Some LMICs have attempted to define their own quality indicators. A recent systematic review identified 34 articles using indicators to measure the quality of emergency care in resource-limited settings.10 These publications generally describe indicators assessing care delivered for one specific disease process such as asthma or trauma, rather than addressing the emergency care system as a whole. South Africa has identified performance indicators for its emergency units (EUs), however, the uptake and feasibility of these performance indicators has been extremely limited.20 In addition, South Africa is an upper-middle-income country with a developed emergency care system and thus selected indicators may not be applicable in a true low-income setting.21

The methodology for the development of quality indicators is quite variable and not well defined.22 Consensus-based strategies including the Delphi technique have been used in the literature to produce emergency-specific indicators.23 24 The African Federation for Emergency Medicine (AFEM) hosts a series of annual consensus conferences to address current gaps impeding the development of emergency care; the 2016 conference aimed to develop a minimum set of context-appropriate clinical quality indicators for facility-based emergency care. Such indicators should be pragmatic, measurable and centred around current health priorities. This effort supports two primary ends: (1) To allow uniform measurement and objective data collection to enhance a facility’s emergency care delivery and facilitate comparison across a national health system or region, and (2) To serve as an input to a larger WHO process on the standardisation of quality indicators for emergency care in LMICs.

We undertook a multiphase expert consensus process to identify, rank and refine quality indicators.

Phase 1: literature review for quality indicators

We searched peer-reviewed publications and the grey literature to determine commonly cited quality indicators for emergency care. We searched PubMed, MEDLINE, EMBASE and the Cochrane Library using the search term ((quality(Title) AND emergency(Title/Abstract)). Fifty-three articles were selected which included publications that both addressed emergency care quality indicators and enumerated a list of indicators. We also reviewed citation lists of included papers. We then further searched indicators for different specialty domains relating to emergency care, those published by professional societies in the UK, USA, Canada and Australia, and those from the Columbia University sidHARTe Programme11–14 16 (Systems Improvement at District Hospitals and Regional Training of Emergency Caredx - diagnosis (sidHARTe) Indicators for Acute Care in Low and Middle-Income Countries Toolkit, unpublished). All indicators listed were extracted from the articles and compiled.

After literature review, 2307 indicators were returned. The initial list of 2307 indicators contained many indicator duplicates, as well as indicators that had slightly different wording but were synonymous. Duplicates were removed, synonymous indicators were represented by a single indicator and final list of indicators (137) were categorised by the authors (MCB and EJCH) into Donabedian’s categories of structure, process and outcome.25 The categorised indicators were mapped to specific components of the patient encounter as defined by the WHO Emergency Care System Framework in order to break the indicators into more manageable blocks for discussion, and provide context and organisation for consensus conference participants.26

Phase 2: selection of clinical conditions and initial indicators

A diverse group of 32 physicians, clinical officers, nurses and administrators from 21 countries participated in the 2016 AFEM Consensus Conference. Participants were provided with a description of the working group goals and objectives, a briefing document describing the core tenets of clinical quality indicators and a template of the indicator matrix prior to the initial encounter day. Patient safety metrics in emergency care are being addressed in a separate process by WHO, and thus were not specifically addressed during this consensus session.

Priority clinical conditions associated with emergent presentations were identified and informed by the Global Burden of Disease project.27 Explicit principles were established for the selection of clinical conditions: the condition must occur with significant frequency, be associated with high morbidity and mortality, fall within the scope of emergency care, and with timely intervention lead to improved clinical outcomes.

Indicators associated with these conditions were then reviewed by the group according to the inclusion criteria described in table 1. Borderline indicators were marked and noted for clarification in further rounds.

Table 1

Criteria for emergency care clinical quality indicators in Africa

Inclusion criteria	Explanation
Represents conditions which:
Occur with significant frequency
Are epidemiologically significant	The acute condition has high morbidity and mortality if appropriate care is not received in a timely manner.
Fall within the scope of emergency care	The scope of emergency care includes all acute presentations within cadres of medical specialty across the life-cycle; it does not include longitudinal chronic care for ongoing morbid conditions. Example: While emergency care plays an important role in acute services for psychotic behaviour or suicidal ideation, ongoing mental health provision is best served by longitudinal chronic care.
Have improved outcomes with optimal emergency care	Some acute presentations are more influenced than others by the emergency care provided. Example: Sepsis mortality is significantly impacted by early appropriate resuscitation.36
Represents conditions whose outcomes:
Depend on currently available diagnostic test or equipment	The diagnostic study must be readily available in the majority of emergency units. Example: CT scans are unavailable in the majority of emergency units in Africa and even when available at a tertiary referral level, are difficult to access due to cost, maintenance and expertise.
Depend on currently available therapeutics	The therapeutic must be commonly available on national formularies, often included on the WHO Essential Medications List.37 Example: Tissue plasminogen activator would not be readily available in most African settings for treatment of acute stroke.
Are feasible to collect	Most charting systems in LMICs are paper-based and documentation varies significantly.
Are applicable to the current reality of emergency care delivery in Africa	While there is need for ongoing development of emergency care, the proposed quality indicators must address the current status of care delivery.

LMICs, low-income and middle-income countries.

Through consensus, the working group identified seven clinical conditions to prioritise in their indicator development (table 2). Overall, the identified conditions addressed nearly 75% of the mortality in Africa.28 Of note, we deliberately chose not to include prehospital services: they will be the subject of a separate AFEM indicator development process.

Table 2

Selected emergency clinical conditions and representative disease28

Condition	Disease examples	Associated regional percentage of total death according to Global Burden of Disease
Trauma	Falls, road injury, violence, burn	7.15%
Sepsis	Malaria, PNA, neonatal sepsis, measles, HIV, meningitis, maternal sepsis	32.6%
Acute respiratory compromise	Lower respiratory infections, COPD, asthma	10.97%
Shock	Diarrhoeal disease	6.82%
Altered mental status	Seizure, ingestion, diabetes	2.68%
Pain	Headache (stroke) Abdominal pain (PUD, gastritis, appendicitis, gall bladder disease, ileus) Chest pain (ischaemic heart disease, hypertensive heart disease, cardiomyopathy, rheumatic heart disease)	5.15% 1.3%  6.8%
Obstetrical bleeding	Postpartum haemorrhage, ectopic pregnancy	0.79%
Total % of African Region Global Burden of Disease mortality addressed by emergency care		74.26%

COPD, chronic obstructive pulmonary disease; PNA, pneumonia; PUD, peptic ulcer disease.

Indicators compiled from the literature review were discussed and modified based on the targeted conditions and pre-established inclusion criteria (table 1). We did not generate any de novo indicators in phase 1. The group started with a list of 137 indicators: 101 (74%) reached agreement and were included in the next phase. The working group identified that timeliness indicators must be specific to only the EU encounter. The group deemed quantitative ‘time to’ indicators could not be feasibly collected with current charting methods in Africa.

It was decided that while critical incident rates are important for quality improvement, they are difficult to define and measure within a minimum set of clinical quality indicators. Additionally, patient perception of care was also identified as an essential quality metric, but standardisation of patient satisfaction scores across various cultures and countries within Africa was beyond the scope of the recommendations of this study.

Phase 3: formal survey—expert clinician selection of indicators

Conditions and quality indicators were sent to a convenience sample of 38 experts from 17 countries, with extensive experience in the provision of emergency care across Africa: as in the prior round, participants were given instructions to follow criteria established in table 1 and to reflect the conditions selected in table 2. All participants completed the survey. Respondents ranked each indicator according to a 5-point Likert Scale (ranging from ‘strongly disagree’ to ‘strongly agree’). During this phase, respondents were given the opportunity to suggest additional de novo indicators they felt appropriate. Quality indicators with greater than 70% agreement of ‘agree’ or ‘strongly agree’ were selected to proceed to the next round.

Of the 101 indicators, 89 (88%) were selected for inclusion in the next phase.

Phase 4: expert review and ranking

A subsequent expert panel sampled from senior delegates at the 2016 African Conference on Emergency Medicine was convened to review and provide further refinement of the quality indicators selected by the formal survey. This diverse panel was composed of 22 clinicians practising emergency care in Africa. The panel first reviewed the indicator list, and removed those that did not meet group consensus. The panel was then asked to rank the different indicators in three domains: correspondence with level of emergency system development, feasibility of data collection and value to patients (1—lowest priority, 3—highest priority). ‘Correspondence with level of emergency system development’ was described as whether the indicator was valuable as a measure of the overall development of the emergency care system; for example, mortality in the first 24 hours from trauma might be a good indicator of how well the system is functioning whereas documentation of disposition is particular to the facility and would likely be a less reliable indicator of system development. A composite score for each indicator was generated, and they were ranked according to their importance as defined by the three domains.

Seventy-six of the 89 indicators achieved consensus; due to time constraints, indicators deemed most important for measuring facility processes (55) were subsequently rated in three domains and a composite score used for generation of a prioritised master list (online supplementary appendix 2).

Phase 5: indicator definition and refinement

Each indicator was assigned a standard profile that included an operational definition, numerator and denominator, and reference data sources for data capture. Standard definitions for each term are included in the meta-data, which will be provided to users of the indicators. Further required detailed clarification was accomplished via expert review. The profiles for selected outcome clinical quality indicators are described in table 3. Process, time-based and structure indicators are included as online supplementary tables A–D. Summary of selected clinical conditions and their associated outcomes are available in table 4.

Table 3

Outcome clinical quality indicators

Rank	Indicator	Definition
1	Mortality from trauma	% of patients with trauma-related chief complaint who die within 24 hours of EU presentation Numerator: # of deaths from trauma within 24 hours of EU presentation Denominator: # of patients with trauma-related chief complaints
2	Mortality from lower respiratory tract infection (adult)	% of adult patients with diagnosis of LRTI who die within 24 hours of EU presentation Numerator: # of adult patients who die from LRTI within 24 hours of EU presentation Denominator: # of adult patients who have a diagnosis of LRTI
3	Mortality from lower respiratory tract infection (child)	% of patients <5 years with diagnosis of LRTI who die within 24 hours of EU presentation Numerator: # of patients <5 years who die from LRTI within 24 hours of EU presentation Denominator: # of patients <5 years who have a diagnosis of LRTI
4	Mortality from asthma	% of patients with diagnosis of asthma who die within 24 hours of EU presentation Numerator: # of patients who die from asthma within 24 hours of EU presentation Denominator: # of patients who have a diagnosis of asthma
5	Left without being seen	% of patients registered in the EU who left without being seen by a provider Numerator: # of patients registered in the EU who left without being seen by a provider Denominator: # of patients registered in the EU

EU, emergency unit; LRTI, lower respiratory tract infection.

Table 4

Summary table of clinical conditions and associated indicator list

Condition	Outcome category	Indicator
Trauma Includes: Falls, road injury, violence, all penetrating and blunt trauma	Mortality	% of patients with trauma-related chief complaint who die within 24 hours of emergency unit (EU) presentation
	Morbidity	% of patients meeting local transfusion criteria who receive blood transfusion % of patients with active external bleeding who have action taken to control haemorrhage % of patients with extremity fracture who are splinted % of patients with documentation of pain who receive pain medications
	Assessment	% of patients with chief complaint of trauma with documented primary survey % of patients with highest triage priority who have VS recorded every 15 min in the first hour of presentation
	Diagnostics	% of patients with chief complaint of blunt abdominal trauma who have documented finding of FAST exam
	Structure	% of EUs with trauma protocol
Sepsis Communicable disease Includes: Malaria, PNA, neonatal sepsis, measles, HIV, diarrhoeal disease	Morbidity	% of patients with diagnosis of sepsis given antibiotics during EU length of stay Mean amount of IVF given per adult patient with dx of sepsis % of adult patients with SBP <90 mm Hg given IVF
	Assessment	% of patients with highest triage priority who have VS recorded every 15 min in the first hour of presentation
	Structure	% of EUs with sepsis protocol
Lower respiratory tract infection (LRTI) Communicable disease Includes: Adult PNA Paediatric PNA	Mortality	% of adult patients with dx of LRTI who die within 24 hours of EU presentation % of patients <5 years with diagnosis of LRTI who die within 24 hours of EU presentation
	Morbidity	% of adult patients with pneumonia given antibiotics during EU length of stay % of patients <5 years with pneumonia given antibiotics during EU length of stay % of patients with SaO2 <92% who had supplemental oxygen given
	Assessment	% of patients with highest triage priority who have VS recorded every 15 min in the first hour of presentation
Asthma Non-communicable disease	Mortality	% of patients with diagnosis of asthma disease who die within 24 hours of EU presentation
	Morbidity	% of patients with SaO2 <92% who had supplemental oxygen given % of patients with documentation of wheezing who receive bronchodilator treatment % of patients with documentation of wheezing who receive corticosteroid treatment
	Assessment	% of patients with highest triage priority who have VS recorded every 15 min in the first hour of presentation
	Structure	% of EUs with asthma protocol
Obstetric emergencies Maternal health Includes: PPH Eclampsia/pre-eclampsia Ectopic pregnancy	Morbidity	% of patients with diagnosis of eclampsia or pre-eclampsia receiving magnesium % of adult patients with SBP <90 mm Hg given IVF % of patients with active bleeding who have physical manoeuvres applied to control haemorrhage % of patients with postpartum haemorrhage who receive pharmacological manoeuvres to control bleeding % of patients with diagnosis of sepsis given antibiotics during EU length of stay Mean amount of IVF given per adult patient with dx of septic shock
	Assessment	% of patients with highest triage priority who have VS recorded every 15 min in the first hour of presentation
	Diagnostics	% of female patients age 15–49 years with pregnancy test checked

FAST, Focused Assessment with Sonography in Trauma; IVF, intravenous fluid; PPH, post partum haemorrhage; PNA, pneumonia; SaO2, oxygen saturation; SBP, systolic blood pressure; VS, vital signs.

Discussion

The initial stages identified clinical conditions which represent much of the burden of disease amenable to emergency care (table 1); this finding independently reflects other lists of life-threatening conditions requiring emergency care that have been produced through separate processes.29–31 The final list of clinical conditions with associated indicators represents the diversity of acute presentations that are amenable to emergency care. These indicators are also translatable across a range of clinical conditions (eg, % of adult patients with SBP <90 mm Hg given intravenous fluid applies to trauma, sepsis and obstetric emergencies). This emphasises the cross-cutting efficiency of emergency care interventions in an EU (or emergency receiving areas, in the absence of a dedicated EU) within the hospital.

This process produced a robust set of clinical quality indicators appropriate to the African context (online supplementary appendix 1). These indicators should be readily transferrable to most low-income countries with limited resources for emergency care, and may be used: for quality improvement projects within a single facility; to allow comparison and benchmarking across the facilities within or between emergency care systems; and to allow for evaluation of a targeted intervention (eg, an educational programme). In addition, these indicators intersect with current priorities in emergency care development, such as minimum data sets, clinical and operational protocols and a standardised minimum package of emergency care services.32 Some of the indicators, inevitably, have underlying assumptions which may not hold in all settings (such as those which are in part dependent on provider availability). We did not aim to produce a list of indicators which could only be used at the lowest resource level, and as such not all indicators may be applicable at all settings.

Three notable points regarding the selection of indicators deserve discussion. First, time-bound indicators for interventions were defined as occurring during the EU encounter. This decision eliminated many commonly used indicators in high-income countries such as ‘door to balloon’ for acute myocardial infarction.33 Recording time-stamped interventions is not feasible when accounting for the reality of clinical documentation in LMICs (eg, paper-based, inconsistency, missing data).20 Thus, all time-based indicators report whether the intervention occurred during the EU length of stay, although more granular data would be preferable. Second, these indicators do not address the significant contribution that prehospital services may have on outcomes, and do not include indicators related to the interface of prehospital care with the facility (eg, amount of time a facility diverts ambulances).34 These particular indicators are under consideration in a separate project by the AFEM Scientific Committee. Lastly, this work did not capture the importance of the patient’s perception of care as essential to the quality of emergency care delivered, as little multicultural data were available to inform such a process.

This multiphase consensus process was not intended to be systematic: we aimed to produce a pragmatic and feasible list of indicators for the African setting. It is possible that potential indicators were missed in the initial processing after the literature review, but the multiple phases gave a diverse group of expert clinicians repeated opportunity to flag any significant deficiencies, and none were noted. While expert input addressed feasibility via the indicator ranking scheme, true feasibility will only be determined by formalised studies investigating what data are currently being collected in EUs at a variety of levels across the health system.

Limitations

There are a number of limitations that deserve mention. There is a widespread lack of published indicators for emergency care in LMICs, especially in Africa. This may be due to the fact that emergency medicine as a specialty either does not exist or has been recently developed in most countries. In addition, in many LMICs robust emergency care systems are in their infancy, so they may not yet have agreed on national indicators for quality at facilities.

Another limitation relates to the search strategy used to identify indicators. While we attempted to be as inclusive as possible, a formal systematic review was not employed for the purposes of this consensus process. Thus, we may have missed some possible indicators for inclusion. In addition, the authors grouped indicators by structure, process, and outcome and mapped them to the WHO emergency care system (ECS) Framework to facilitate discussion and provide a contextual reference point. These indicators were then subsequently disassociated from the framework and Donabedian categories in the final ranking. Grouping them in such a manner could have influenced the way that participants selected and subsequently ranked indicators.

The clinical conditions prioritised in this process may not be representative of the most important clinical conditions affecting a specific region or area. The local burden of emergent conditions is still unknown in much of Africa;3 thus, Global Burden of Disease data were used as a proxy for the burden of emergency conditions to guide the process.25 Further work is necessary to define the essential emergency conditions requiring evaluation with quality metrics for a given region.

Lastly, while the authors attempted to be as inclusive of those in emergency clinical practice in Africa as possible, convenience sampling of participants at the 2016 AFEM Consensus Conference and then the subsequently engaged expert panel could have skewed results so as not to be representative of the larger experience of emergency care on the African continent. Participants in this phase represented 21 countries but, as with any consensus process, not all views may have been included. Group discussion was elicited and facilitated but this may have led to some participants conforming with the majority view and may not represent the full spectrum of opinions.35

Conclusion

Expanding interest in the development of emergency care by clinicians and policy makers in resource-limited settings underscores the need to measure and improve the quality of care delivered.28 We propose a minimum set of clinical quality indicators for facility-based emergency care in Africa, and provide a common language by which to facilitate intranational and international comparison; harmonisation on reported performance indicators will allow direct comparison of development efforts and should lead to enhanced quality and safety of care provided. Local adaptation specific to burden of disease and feasibility of measurement are crucial components of operationalising quality metrics. Feasibility studies are required to test the functionality of our proposed indicators; however, these indicators bring us one step closer to measuring quality emergency service delivery in low-resourced settings.