Evaluating emergency care capacity in Africa: an iterative, multicountry refinement of the Emergency Care Assessment Tool.

Healthcare facilities in low-income and middle-income countries lack an objective measurement tool to assess emergency care capacity. The African Federation for Emergency Medicine developed the Emergency Care Assessment Tool (ECAT) to fulfil this function. The ECAT assesses the provision of key medical interventions (signal functions) that emergency units (EUs) should be able to perform to adequately treat six common, life-threatening conditions (sentinel conditions). We describe the piloting and refinement of the ECAT, to improve usability and context-appropriateness. We undertook iterative, multisite refinement of the ECAT. After pilot testing at a South African referral hospital, subsequent studies occurred at district, regional and central facilities across four countries representing the major regions of Africa: Cameroon, Uganda, Egypt and Botswana. At each site, the tool was administered to three participants: one senior physician, one senior nurse and one other clinical provider. Feedback informed refinements of the ECAT, and an updated tool was used in the next-studied country. Iteratively implementing refined versions of the tool in various contexts across Africa resulted in a final ECAT that uses signal functions, categorised by sentinel conditions and evaluated against discrete barriers to emergency care service delivery, to assess EUs. It also allowed for refinement of administration and data analysis processes. The ECAT has a total of 71 items. Advanced facilities are expected to perform all 71 signal functions, while intermediate facilities should be able to perform 53. The ECAT is the first tool to provide a standardised method for assessing facility-based emergency care in the African context. It identifies where in the maturation process a hospital or system is and what gaps exist in delivery of care, so that a comprehensive roadmap for development can be established. Although validity and feasibility testing have now occurred, reliability studies must be conducted prior to amplification across the region.

The development of emergency care systems is rapidly accelerating across low-income and middle-income countries (LMICs), particularly those in Africa.

Although some tools exist to assess facility-based management of specific acute clinical conditions, most are targeted towards high-income countries and none allow for comprehensive assessment of the full breadth of emergency care provision at LMIC facilities.

Through consensus, the African Federation for Emergency Medicine developed the Emergency Care Assessment Tool (ECAT), which aims to evaluate LMIC emergency unit capacities in a contextually appropriate manner.

This study puts forth a modified version of the ECAT that has undergone iterative refinements across the four major geographic regions of Africa.

The ECAT allows facilities to identify gaps in service delivery and make targeted improvements to meet essential emergency care needs at various stages of systems development.

Introduction

Although over a decade has passed since the 60th World Health Assembly called on all member states to implement ‘formal, emergency medical-care systems’,1 and despite the fact that emergency care systems can directly impact over half of mortality in low-income and middle-income countries (LMICs),2 development of these systems is still in its early stages throughout Africa.3 4 Most countries have no organised prehospital care services and, if patients do reach hospitals, many facilities lack an area dedicated to providing emergency care or staff with adequate training in the field.4–6 Even the most advanced referral facilities across the continent differ greatly in capacity to manage patients with acute conditions, with wide variations in equipment, infrastructure and services provided.7

Contained within the same World Health Assembly resolution was a call for member states to ‘assess comprehensively the prehospital and emergency-care context’.1 Both baseline and longitudinal evaluations are critical steps in properly improving any facet of a healthcare system; they are particularly valuable in the earlier stages of development, as is the case with emergency care across Africa.8 These tools have potential utility during the initial stages of developing emergency units (EUs) and other aspects of the emergency care system, as they can help to identify existing barriers to care and gaps in the system. Longitudinally, as these systems mature, the tools can continue to inform decision-making both within facilities and at-large through Ministries of Health (MoHs) and other engaged stakeholders.8

However, no standardised assessment tool has yet been developed that covers the breadth and scope of emergency care delivery across health facilities, particularly in African settings.6 A recent publication has highlighted that approximately 30% of the population in sub-Saharan Africa lives greater than 2 hours from any hospital services. However, this same study was unable to ascertain which emergency services were provided at the hospitals studied, highlighting the urgent need for emergency capacity evaluatory tool development.9 There are many international tools, but most are targeted at well-developed systems in high-income countries that differ vastly from the resource-constrained and disease-burdened situations seen in most LMICs.10 WHO and other organisations have developed tools that assess availability of infrastructure and capacity to provide care within the context of specific clinical conditions in LMICs, but these tend to be narrowly focused checklists. As such, they do not provide an accurate representation of the spectrum of emergency care provision.4 10–12

Perhaps the most noteworthy tool is the WHO’s Monitoring Emergency Obstetric Care Handbook (EmOC).13 EmOC is unique because, although it is used to assess a very particular part of healthcare—maternal health—it uses signal functions, life-saving procedures that are based on function, not individual components, and encompass both skills and resources.3 5 13 Signal functions focus on the practical capacity and delivery of a service, rather than cumbersome checklists about specific supplies and equipment; however, their sensitivity still allows for the detection of serious deficits in the provision of care.13 They represent the sequence of events required for service delivery and are directly translatable to informing and modifying health policies and programmes.3 13

An example of a signal function is the ability to administer antibiotics intravenously, which simultaneously assesses provider knowledge, intervention capacity and availability of supplies (online supplementary appendix 1). Signal functions proved particularly useful in determining capacity for managing emergency care in the context of emergency obstetric complications, suggesting that they likely have applications to broader emergency care provision as well.14 EmOC was also revolutionary in its efforts to capture barriers to the delivery of care, where signal functions could not be performed. Although limited to obstetric patients presenting with acute emergencies, these data points provide substantial evidence from which facilities and MoHs can make informed decisions about improving care at multiple levels of the health system.

Development of the Emergency Care Assessment Tool

In 2013, the African Federation for Emergency Medicine (AFEM), an umbrella organisation representing a broad coalition of African emergency care societies, organisations and individuals, brought together 135 experts from 32 countries in an effort to develop a framework to meet the broad assessment needs of budding African emergency care systems.3 Through consensus, it was decided that the first portion of the tool should be based off EmOC’s evaluation strategy: using signal functions as a means of identifying gaps in resource availability or provider knowledge and the entire breadth of care provision. The group set forth to first derive a core set of sentinel conditions (specific pathophysiological syndromes that are commonly seen before death, that should be amenable to emergency care) and then derive context-appropriate emergency care signal functions for sentinel conditions. In short, AFEM identified the main life-threatening conditions (sentinel conditions), that should be able to be handled by a fully operational EU, and then identified the necessary functions that an EU must be able to perform to handle them well (signal functions). It would be these signal functions that would be used as the foundation for a facility-based emergency care assessment framework in Africa.3 It was crucial that included sentinel conditions were reflective of time-sensitive life threats, as those are most likely to be remedied by effective emergency care provision and yield improved patient outcomes. Based on information from the WHO Integrated Management of Adolescent and Adult Illness,12 six sentinel conditions were chosen to represent the final common pathways for the majority of aetiologies causing death and disability which were amenable to emergency intervention: respiratory failure, shock, altered mental status, severe pain, trauma and dangerous fever.3 Over 100 signal functions were derived to cover the provision of emergency care for these six sentinel conditions; examples of these signal functions can be found in table 1.3

Table 1

Sentinel conditions and example signal functions

Sentinel condition	Example signal functions
Respiratory failure	Oxygen administration Relieve obstruction of airway Bag valve mask ventilation
Shock	Packing and suturing for control of haemorrhage Peripheral percutaneous venous access Administration of epinephrine for anaphylactic shock
Altered mental status	Check and/or administer glucose if required Administer benzodiazepine for seizure Perform lumbar puncture
Severe pain	Administer opiate-based analgesia Perform therapeutic paracentesis Perform point of care ultrasound in the EU
Trauma	Perform initial appropriate wound care Immobilise fracture (basic) Administer tetanus vaccine and IVI as indicated
Dangerous fever	Perform source control with bedside techniques, including abscess and empyema drainage Perform rapid cooling Measure serial lactate

EU, emergency unit; IVIG, intravenous immunoglobulin.

The original framework included three sections: (1) assessment of ability to perform signal functions for the six sentinel conditions, (2) checklist of essential/desired physical equipment/infrastructure and (3) identification of barriers to delivery of signal functions. The checklist (section 2) comprised a list of specific materials that were to be evaluated for physical availability and functionality. The barrier-identification portion (section 3) was based on well-validated service delivery assessment tools in other fields.10–13 15 16 These ‘barriers to delivery’ were evaluated for each signal function and defined as issues surrounding:

Policies: lack of policies and processes that facilitate optimal patient care (eg, no triage system, lack of timely patient movement to definitive care).

Human resources: insufficient number of empowered healthcare workers to perform the desired function.

Healthcare worker training: cadre is available but not trained or there is a lack of confidence in providers’ skills.

Supplies/equipment/medication: supplies and equipment are not available, not functional or broken, or needed medications are unavailable.

Infrastructure: critical facility-based infrastructure is not available or not functional (eg, electricity, lab, X-ray).

No indication: no indication can be identified as to why the signal function cannot be performed at least 90% of the time.

The AFEM meeting resulted in a regionally relevant framework of the essential components of emergency care delivery, with expected levels of provision stratified by facility level (eg, a basic facility would not be expected to meet some of the more complex standards that an advanced one should provide).3 The initial framework included 280 items across its three sections; each item was considered to be either essential or desirable for each facility level—basic, intermediate and advanced.3

From this framework came a draft assessment tool for emergency care provisions: The Emergency Care Assessment Tool (ECAT). The ECAT’s explicit purpose is to identify what the targeted EUs in disparate African settings have and do not have, and thus identify the gaps in their emergency care service delivery that need to be closed. It is intended to evaluate service delivery for patients of all ages presenting with sentinel conditions. Thus, the identified gaps can serve as a roadmap to guide facilities and policy-makers to develop their emergency care and services.

In this study, we aimed to conduct broad testing and refinement of the ECAT, so as to improve both usability and contextual-appropriateness. We undertook iterative, multisite refinement of the ECAT. The tool was first piloted in South Africa, after which it was sequentially implemented in a range of health facilities across four African nations: Cameroon, Uganda, Egypt and Botswana. Due to the study’s nature, exploring ECAT feasibility prior to validity and reliability testing, it was agreed on with MoHs that study sites and all associated results would not be published.

Phase 1: Pilot study

The ECAT is intended for any facility; however, only public hospitals were assessed in this development process, as the majority of Africans seek care in public hospitals.17 The initial draft ECAT was pilot tested at one urban referral-level EU in South Africa. The purpose of this pilot study was to understand the survey’s overall feasibility and contextual appropriateness prior to conducting large-scale refinement.

The EU manager at the South African facility identified three emergency care providers to participate in the initial pilot process. This included one senior doctor, one senior nurse and one other clinical provider, so as to include the breadth of cadres represented in EUs while also surveying staff that are likely to be well-informed of the reality of their EU. Participants were verbally informed of the study, after which written informed consent was obtained. Participation was entirely voluntary, and participants were informed that they could decline participation or withdraw from the survey at any time without negative consequences.

The ECAT was administered prospectively on-site, with one-on-one meetings held between trained ECAT administrators and participants. It was administered verbally and primarily in English (as the tool was only available in English at that stage). The administrator explained the ECAT in depth and provided examples for each barrier to delivery. As a prospective survey, the ECAT was administered using appropriate language. For example, the administrator would ask ‘If a patient with [sentinel condition] entered your health facility right now, would you be able to [signal function]?’. The tool administration included a series of yes/no questions with possible follow-up questions based on the response. The administrator explained that answering ‘yes’ to a question was under the assumption that the signal function was available and could be performed correctly at least 90% of the time it is required. Negative (‘no’) responses indicate that it could not be and required clarification under the ‘Barriers to Delivery’ section. Any further comments offered by respondents were collected, to be taken into consideration during the refinement process.

Deidentified data collected by administrators on paper forms were entered into encrypted Microsoft Excel (© Microsoft, Richmond, Washington, USA) spreadsheets; Excel was also used to generate basic statistics. All data points from each survey were then checked by two others on the research team to ensure accuracy. Written qualitative feedback was collated into Microsoft Word (© Microsoft) documents.

Pilot testing generated substantial feedback from participants that allowed the research team to make further revisions to improve usability, feasibility and clarity. Critical modifications include the following: (1) Signal function structure was selected for brevity and ease of administration, thus eliminating the other two sections of the previously-discussed initial framework—facility infrastructure and material (as it was clear that they were already being indirectly captured by signal function performance in the first section) and (2) Only signal functions designated as essential (not desirable) at the consensus conference were included.

Phase 2: Multicountry refinement process

For refinement, the study sampled countries in all four major geographic African regions: Cameroon (Central/West), Uganda (East), Egypt (North) and Botswana (Southern) (figure 1). South Africa was specifically not chosen to represent Southern Africa in this portion of the study as to allow for a more accurate representation of the general economic development in the Southern region.18 These countries are all LMICs, ranging from upper-middle-income (Botswana) to low-income (Uganda). While emergency medicine is a recognised specialty in two of the four countries (Botswana and Egypt),19 20 there remains a paucity of adequately trained providers to manage the heavy burdens of injury and illness that these nations face.21

Figure 1

Geographical distribution of piloting and refinement sites.

AFEM has local representatives in each of the chosen countries who facilitated the selection of appropriate facilities via convenience sampling. This was done in place of random sampling, because the primary aim of this study was not intended to be an analysis of facility capacity, but rather, tool usability. A total of three facilities (one district, one regional and one referral) were surveyed per country. These hospitals are the local equivalents of intermediate (district) and advanced (regional and referral) facilities, as defined by AFEM.3 Basic facilities—health posts and health centres—were not included in this study, as they typically are one-room facilities lacking a formalised EU or receiving area.

As with the pilot study, local ethics approval was also obtained from each site. The hospital manager at each facility in the refinement countries helped to identify three personnel in their designated EU: one senior physician, one senior nurse and one other clinical provider. Participants gave both verbal and written consent, after which the ECAT was administered prospectively on-site. Where needed, the administrator provided clarifications in the native language. The ECAT took an average of 30 min to complete. It was first administered in Cameroon, then Uganda and Egypt and lastly in Botswana.

All quantitative data were again analysed in Excel; qualitative feedback was collated in Word. After implementation in each country, the research team convened to discuss outcomes and potential refinements. If consensus was reached regarding a particular change, then it was included in the next iteration of the tool. Stepwise refinements of the survey, generated from these discussions, are described below.

With a total of three respondents per facility (similar to the pilot study, one senior doctor, one senior nurse and one other clinical provider) across three facilities in each of four nations, we conducted a total of 36 complete ECAT assessments. Refinement occurred primarily via qualitative feedback captured on the ECAT by administrators during survey administration. While quantitative data were necessary to capture in order to assess usability and could display preliminary trends, they were not of direct use in refinement. The following were the primary changes made to the tool at each stage of the phase 2 refinement process.

For items regarding administration of medications, the original ECAT only allowed for participants to indicate a single route of administration. These items were changed so participants could identify all of the routes the facility was capable of, instead of being limited to only one of them.

Development of a ‘barriers to delivery’ document with definitions and examples for referencing, to be used by the participant.

Development of a set of easily-referenced ECAT administration instructions, to support initial administrator training and ensure that ECAT administration was consistent across all sites.

Changing terminology from ‘district’, ‘regional’ and ‘referral’ to describe facility levels to:

‘mid-level’ facility to include regional and district level facilities, and

‘referral-level’ facility to include tertiary referral hospitals and other academic/university facilities.

Adding clarity to the administration instructions that the availability of a signal function is in the emergency care area of the facility, not at another site within the facility (eg, the operating theatre or a ward).

Allowing for participants to select multiple barriers to delivery as problematic in delivering a signal function.

Addition of the ‘rectal’ route for administration of medication, in addition to ‘PO’, ‘IM’ and ‘IV’.

Botswana

No changes, other than basic formatting, were made to the survey after administration in Botswana.

Phase 3: Final consensus process

After completion of the refinement process, a meeting was held by the research team to finalise the ECAT tool based on the comprehensive feedback gathered from the four target countries. In the initial creation of ECAT, obstetric emergencies were believed to generally bypass the EU and go straight to the obstetric unit in African settings.22 Feedback throughout the study showed a significant volume of emergency care delivery related to obstetric emergencies within the EU at each of the facilities involved. Thus, although haemorrhage and sepsis were covered in other sentinel conditions, obstetric emergencies was added as a sentinel condition in the consensus phase, along with the original consensus-based signal functions. Signal functions for emergency obstetrics were adapted from EmOC.13

Feedback from phase 2 also suggested that signal functions related to dangerous fever only occurred in the context of one of the other sentinel conditions: for example, in septic shock, there is fever and shock or altered mental status. As such, dangerous fever was deemed redundant and removed as a sentinel condition during consensus.

The language in many of the signal functions was non-specific in the initial draft—for example, ‘clearing of airway’ left respondents uncertain about the exact procedure the ECAT was referencing—but this iterative process allowed for reductionism and clarity in phrasing of statements. In addition, the final ECAT is clear that the assessment should be viewed prospectively, meaning that participants should consider the signal functions in the context of managing a patient with the given sentinel condition in the immediate moment.

The final advanced facility ECAT (online supplementary appendix 1) had a total of 71 items, reduced from the original 280. These items are stratified across six sentinel conditions: respiratory failure (14 signal functions), shock (17), altered mental status (10), severe pain (9), trauma (17) and emergency obstetrics (4).

While advanced facilities were expected to perform the full breadth of signal functions (71), some of the more complex and resource-intensive signal functions were removed for intermediate facilities based on original AFEM consensus recommendations.3 Although still stratified across the same six signal functions, the final intermediate facility ECAT (online supplementary appendix 2) had a total of 53 items: respiratory failure (11 signal functions), shock (13), altered mental status (7), severe pain (8), trauma (10) and obstetric emergencies (4).

Strengths and limitations of the Emergency Care Assessment Tool

Unlike many other tools, the ECAT assesses facilities ability to perform crucial emergency interventions based on current level designations (eg, being assigned the title of ‘district hospital’).11 Defined objective emergency capacity combined with geospatial location data, population densities and mortality mapping, could greatly enhance prospective, rational emergency health system planning. The breadth of the ECAT’s evaluation methodology, including barriers to delivery both internal to EUs and in the healthcare system at-large, allows for validation of these designations and meaningful and contextually appropriate feedback that can be used to quickly enact positive changes.

The original ECAT included three sections, evaluating: signal function performance, availability of resources and physical infrastructure and functionality. By removing the two latter portions of the framework, the pilot study substantially shortened the tool. In originally setting goals for emergency care development, the 2013 consensus group identified signal functions, infrastructure items and equipment that are either essential or desirable at the three tiers of facilities.3 While this is useful information for systems developers, pilot feedback suggested, it was unnecessary to include items that are only desirable, if the goal of the tool is to determine functional capacity. Again, this commentary helped to simplify the tool prior to the refinement process, likely saving significant amounts of time during administration.

Because the refinement process took place across a wide range of sites with varying clinicians and administrators, both the administration process and tool itself were significantly improved. The refinement process helped to solidify the language and style of questions, while clarifying guidance documents for administrators. This should ensure that, in future studies, the ECAT is administered and understood by participants uniformly. But, its iterative refinement over a variety of countries could have some impact on translatability; future studies will be required to evaluate this.

As a prospective survey, validity studies are required to ensure that the responses collected through the ECAT are an accurate representation of ability. Prior to more expansive roll out, the ECAT will have undergone more rigorous evaluation and testing.5 16 A French version of the tool has already been developed,23 and other alternate language versions are being considered to achieve wider uptake. The final steps in the tool’s development include larger studies on inter-rater and intra-rater reliability.

As previously discussed, there are many tools that already exist and are in use; however, the ECAT is intended to fill a much-needed role that other tools have not been able to achieve: comprehensively assessing facility-based emergency care capacity across a breadth of time-sensitive, high-mortality emergency conditions that are amenable to emergency care interventions. The tool assesses what facilities can and cannot do in their current states; the identification of these gaps goes on to provide a roadmap for what the facilities need to improve. Its stratification by facility level allows for realistic assessments of capacity and can help stakeholders in determining if their facilities are meeting appropriate emergency care standards. The ECAT can be leveraged by facilities or countries during any point of the emergency care systems development process, and repeated administrations can assess potential improvements. The simplistic tool and its accompanying training documents allow for efficient, cost-effective assessments to occur. Easily interpretable data means that systems developers at all levels—from hospital administration to MoH leadership—can rapidly translate results into feasible, implementable action items. A hospital or region’s ECAT results do not just serve to asses where their emergency care services are, or track improvements over time, but also create a roadmap for development by identifying the gaps that the system needs to close.

The ECAT was developed to serve as a broad, general assessment, to quickly determine gaps in facility-based emergency care provisions and the system at-large; it was not intended to collect nuanced details. While it can successfully identify a range of barriers to the delivery of care, it does not provide detailed follow-up questions to further explore these issues. However, it can easily be supplemented with existing equipment and care checklists after a first-pass. For example, for a facility that does poorly on the ‘trauma’ section of ECAT, could then be directed towards using the WHO Trauma checklist.11 A facility that needs improvement with emergency obstetric care according to ECAT could then complete the WHO EmOC tool,13 and a facility that is struggling with the supply chain of certain medications might be referred to the WHO Model List of Essential Medicines and the AFEM Essential Medicines for Emergency Care in Africa List.15 24 Recommendations for follow-up assessments will vary based on site needs and priorities, but it is clear that currently existing tools can serve as strong support mechanisms for further exploring barriers identified in the ECAT.

The ECAT tool intentionally designed signal functions with medication requirements to be open-ended. For example, ‘administer locally appropriate antidote for toxic cause, for example, anti-venom’ and ‘administer critical therapeutics for reactive airway disease’ do not specify a medication that must be used to successfully perform the intervention. Medication supply chains, cost and availability issues in LMICs often prevent hospitals from obtaining very specific drugs.25 Stocks and uses of these drugs are context-specific, especially in Africa.24 A prime example of this regards antivenom: different snakes are endemic to different areas of the continent and so different antivenoms are indicated by region. No single antivenom would adequately assess the capacity of African facilities at-large to manage this scenario. Leaving room for interpretation of what medication each unique facility might use in these cases was the most feasible way of ensuring the ECAT remained both context-appropriate and applicable continent-wide.

It is important to note that this study did not test for validity. Since the tool was changed between countries, sample sizes were not sufficient for statistical testing. As such, the preliminary data are not robust enough to infer any conclusions about the tool’s reliability and validity. In addition, this study was based on convenience samples of respondents from a select number of hospitals, so may not fully represent feasibility across all of Africa. However, the point of this study was to gain as wide of a breadth as possible of African settings and personnel, so that the final ECAT could be of use in as much of Africa as possible. It was more important to receive feedback from countries throughout the major regions of Africa than to develop a study with random sampling that might limit the availability and quality of feedback. Further implementation and refinements of the ECAT can serve to further improve the tool.

Conclusion

The intention of the ECAT was to serve as a sensitive predictor of emergency care capacity at EUs in scenarios where early intervention and proper management of sentinel conditions would improve patient outcomes. Our study successfully refined a preliminary consensus output framework into a tool that meets this need, rapidly assessing emergency care provisions at various facilities levels via the use of representative signal functions. The ECAT underwent major revisions, taking into account usability, specific terminology, its use within the context of existing tools and the ability for rapid administration, while still capturing the capacity of service delivery accessible to EU patients. In addition to informing changes at these individual facilities, ECAT results can also allow for insights into the broader emergency care system.