Development of a simple, practice-based tool to assess quality of paediatric emergency care delivery in resource-limited settings: identifying critical actions via a Delphi study.

OBJECTIVE: Provision of timely, high-quality care for the initial management of critically ill children in African hospitals remains a challenge. Monitoring the completion of critical actions during resuscitations can inform efforts to reduce variability and improve outcomes. We sought to develop a practice-based tool based on contextually relevant actions identified via a Delphi process. Our goal was to develop a tool that could identify gaps in care, facilitate identification of training and standardised assessment to support quality improvement efforts.
DESIGN: Six sentinel conditions were selected based on disease epidemiology and mortality at rural and urban African emergency departments. Potential critical actions were identified through focused literature review. These actions were evaluated within a three-round modified Delphi process. A set of logistical filters was applied to the candidate list to derive a practice-based tool. SETTING AND PARTICIPANTS: Attendees at an international emergency medicine conference comprised an expert panel of 25 participants, with 84% working primarily in African settings. Consensus rounds allowing novel responses were conducted via online and in-person surveys.
RESULTS: The expert panel generated 199 actions that apply to six conditions in emergently ill children. Application of appropriateness criteria refined this to 92 candidate actions across the following seven categories: core skills, active seizure, altered mental status, diarrhoeal illness, febrile illness, respiratory distress and polytrauma. From these, we identified 28 actions for inclusion in a practice-based tool contextually relevant to the initial management of critically ill children in Africa.
CONCLUSIONS: A group consensus process identified critical actions for severely ill children with select sentinel conditions in emergency paediatric care in an African setting. Absence of these actions during resuscitation might reflect modifiable gaps in quality of care. The resulting practice-based tool is context relevant and can serve as a foundation for training and quality improvement efforts in African hospitals and emergency departments. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Simple, practice-based tool developed to evaluate paediatric emergency medical care in resource-limited settings, with particular focus on African countries.

Developed by expert consensus using an iterative, self-validating process.

Tool developed for use by observers with limited medical training to assess quality of emergency medical care for children in real time.

Expert panel represents significant practice experience within African settings.

Practice recommendations are not exhaustive; they are selected based on the ability to widely apply across varied practice environments.

Introduction

Over the past decades, there has been increasing awareness of the importance of monitoring clinical practice to ensure delivery of high-quality clinical care. Standardised assessment of care delivery can highlight areas of deficiency, identify potential targets for process improvement and ultimately lead to improved patient outcomes. This is nowhere more important than in paediatric emergency care where timely recognition and management is essential to improving patient outcomes.1

A recent study exploring minimum standards of emergency care for children in resource-limited settings identified training and policy priorities over structural needs.2 While there exist some standard instruments for monitoring the quality of emergency care training and delivery, few focus on paediatric resuscitation3–9 and most have only limited relevance to resource-constrained settings.10 11 There is evidence that establishment of paediatric-specific standards of care can improve the emergency care of children in these settings.1 8 9 Yet even where there is context-relevant clinical guidance, such as the WHO guidelines for the management of sick and injured children,12 13 there is no standard tool for assessing adherence to these recommendations during initial resuscitation.

The Delphi process is a group consensus method allowing the collection of known and published data to be aggregated and presented to a panel of experts for review.14 By using facilitated evaluation and refinement of group opinion, the method provides robust guidance even when context-relevant experimental data are not available.

We sought to develop a consensus-based list of context-relevant critical actions for the management of sentinel emergency presentations in children, in order to derive a simple, practice-based quality assessment tool for resource-limited settings. Of note, our goal was not to develop comprehensive algorithms to guide care, but to identify a short list of actions that: are consistent with existing guidelines, are near-universally indicated for a given clinical presentation, and for which there is clear consensus among relevant regional experts that the actions are appropriate and feasible within regional context. Our goal was to select actions whose absence would clearly reflect a modifiable gap in the quality of care delivery, not merely an acceptable variation in practice, nor a common regional resource constraint.

Methods

Identification of sentinel presentations

Sentinel presentations were identified by review of the top causes of death among children in sub-Saharan Africa,15 review of published data on common paediatric presentations to urban and rural emergency departments in several countries in the region16–20 and review of the top conditions addressed by existing WHO and international society guidelines on paediatric emergency care.13 21 22 In order to ensure that the resulting tool would support robust quality monitoring, we selected conditions with both a high burden of associated mortality in the region and a high frequency of presentation at relevant clinical sites. In addition, because our goal was to generate an instrument to monitor condition-specific management actions, we also considered the ease of initial identification of the clinical presentation by an observer and chose presentations for which the benefit of early intervention is well established. Ultimately, we sought to identify a few common, life-threatening and intervention-responsive conditions with the potential to reflect the overall quality of paediatric resuscitation. We did not purport to include all, or only, the top conditions at any particular site. Based on these criteria, we selected six presentations: acute diarrhoeal illness, acute febrile illness, respiratory distress, active seizure, altered mental status and polytrauma.

Identifying candidate critical actions by literature review

We conducted a scoping review to identify published articles and international society guidelines that include management recommendations for the selected sentinel conditions (see figure 1). We also referred to training resources and major textbooks to identify commonly recognised standards of care in resource-limited settings.13 21 23 24 Two reviewers (RKD, BLM) extracted and sorted potential actions by presenting condition. Candidate actions were compiled into a master list (see figure 1).

Figure 1

Numbers represent total actions considered in each step. Percentages indicate the proportion of actions, of the total considered at each step, that met a priori inclusion criteria.

Modified Delphi process

An expert panel was derived from registered attendees of the joint World Association of Disaster and Emergency Medicine Conference and African Federation of Emergency Medicine (AFEM) Consensus Conference held in Cape Town, South Africa, in April 2015. Criteria used to select experts included: clinical practice experience in an emergency unit in Africa, authorship of publications addressing clinical practice in global emergency care and active leadership within emergency care organisations focused on Africa. Extended clinical practice experience in a resource-limited setting was essential.

Candidates were invited by email to participate, and in round 1, those agreeing were informed of the purpose of the study and emailed a link to an online survey (Qualtrics, Provo, Utah, USA, 2015). Participants were asked to review the list of candidate actions, identify any that should be deleted and provide any others critical to the management of an acutely ill child presenting with the specified condition. Responses were compiled and redundant responses eliminated.

In round 2, the expert panel met in-person and reviewed the purpose of the study and the intended use of the outputs. Each participant was given a choice of an online or paper survey listing actions within each condition, and then asked to anonymously rate each action on whether it was a critical action to perform for a given condition. Actions were rated on a 9-point Likert scale. A score of 1 indicated ‘Strongly Disagree’, 5 indicated ‘Neutral’ and 9 indicated ‘Strongly Agree’. The expert panel was asked to consider the importance, validity, usability and feasibility of each action during rating.25 A small subset of participants provided advance notification that they would not be able to attend the first in-person meeting and completed the round 2 survey online. All actions with >80% of responses of ≥7 met consensus for inclusion. Those with 80% of responses of ≤3 met consensus for exclusion. (When the number of participants was an odd number, the percentage closest to 80% was used as the threshold.) This threshold is similar to that used in other studies.3 8 10 11 26 Actions not meeting consensus for either inclusion or exclusion were advanced to round 3 for additional review.

In round 3, the expert panel was reconvened. All actions that had not met consensus in round 2 were re-presented, with the median score from the prior round, and anonymously rated again (via online or paper survey at participant preference) using the same Likert scale. After round 3, actions meeting consensus as defined above were included in a final list of consensus-based critical actions.

We then applied filters based on logistical considerations, given our goal of deriving a simple practice-based tool (PBT) for use in acute care settings. The goal of this phase was to remove actions that might be critical in clinical practice, but would not serve well for the purposes of a tool intended for use during initial resuscitation.

We eliminated actions that could not be verified by an observer standing at a distance from a patient, those not applying to all presentations of a condition and those not necessarily indicated within the first hour of care or where an equally acceptable alternate management action exists (such that the failure to perform the action under consideration would not necessarily constitute a gap in care). We also excluded contingent actions that would only be considered critical on recognition of a particular diagnosis (eg, give antidote for a specific toxidrome) rather than a general clinical presentation, since such diagnosis would not always be obvious to an observer.

Two fellowship-trained experts in paediatric emergency medicine (RKD, BLM) conducted the above process. A senior emergency medicine specialist (TAR) reviewed the classifications. We used consensus discussion to resolve any discrepancies.

The remaining actions were compiled into the PBT, and duplicate actions common to all conditions were extracted and classified as ‘core’.

Patient and public involvement

Patients and the general public were not directly involved in the development of this research question or in any portion of critical action development. Results of this study will be distributed via direct correspondence to participants in the expert panel.

Results

The flow of the study is outlined by the figure 1. We sent email invitations to 46 potential participants. Of those, 29 agreed to participate, and 20 initiated the first round. Seventeen participated in round 2, including 12 who had participated in round 1. Fifteen of 17 round 2 participants completed round 3 (table 1). Of the 25 participants who participated in any round, 84% actively practice paediatric emergency care in an African setting (Ethiopia, South Africa, Tanzania, Uganda).

Table 1

Composition of expert panel

	Invited	Accepted	Round 1	Round 2	Round 3
African	36 (78%)	21 (72%)	18 (90%)	14 (82%)	12 (80%)
Non-African	10	8	2	3	3
Total	46	29	20	17	15

Number of participants recruited or active in each round are noted above. The primary region of practice is also noted.

The initial literature review generated a total of 265 actions for the six identified conditions (see figure 1). Round 1 produced an additional 372 free-text responses that were consolidated into 62 discrete actions. In round 2, 194 (59.3%) measures achieved inclusion consensus and immediately graduated to the final action list (bypassing round 3). No actions met exclusion consensus. One hundred and thirty-three actions did not meet either inclusion or exclusion consensus. We submitted these actions into round 3. There, five actions (3.8%) met inclusion consensus. Thus, a total of 199 actions met inclusion consensus for the final list of consensus-based actions, though some actions applied to multiple sentinel conditions.

After removal of non-critical and contingent actions, we refined this list to 92 unique critical actions (online supplementary appendix A—Candidate List). The bulk of these actions represent interventions relevant to the first 15 min of care including airway, breathing and circulation assessment and stabilisation.

Application of the logistical filters described above left 24 unique actions for use in the PBT (39 total actions across all categories) with the number of actions per diagnosis ranging from two to seven (table 2, online supplementary appendix B).

Table 2

Actions included in practice-based tool

Category	Action
Core skills	Assess breathing (auscultate lungs)
	Assess pulse
	Assess capillary refill
	Obtain weight or estimate using standardised technique
	Measure temperature
	Obtain history
	Perform physical examination (of at least three systems)
Active seizure	Obtain oxygen saturation
	Give oxygen
	Assess pupillary response
	Obtain intravenous or ensure intravenous access, or obtain intraosseous if intravenous not available
	Check glucose or administer dextrose if unable to check
	Give benzodiazepines as first-line anticonvulsant—intravenous, intraosseous or rectal
Altered mental status	Obtain oxygen saturation
	Expose patient
	Measure blood pressure
	Check for signs of head injury/trauma
	Obtain intravenous or ensure intravenous access, or obtain intraosseous if intravenous not available
	Check glucose or administer dextrose if unable to check
	Test for malaria
Diarrhoeal illness	Assess skin turgor
	Obtain intravenous or ensure intravenous access, or obtain intraosseous if intravenous not available
	Check glucose or administer dextrose if unable to check
Febrile illness	Obtain oxygen saturation
	Measure blood pressure
	Obtain intravenous or ensure intravenous access, or obtain intraosseous if intravenous not available
	Check glucose or administer dextrose if unable to check
	Test for malaria
	Full septic work-up for children <28 days old
	Administration of broad-spectrum antibiotics for children <28 days old
Respiratory distress	Obtain oxygen saturation
	Give oxygen
Polytrauma	Expose patient
	Measure blood pressure
	Assess pupillary response
	Visualise back
	Obtain intravenous or ensure intravenous access, or obtain intraosseous if intravenous not available
	Obtain blood type and cross-match
	Give analgesia

Actions that met all inclusion criteria and can be monitored by a non-participant observer during resuscitation. See online supplementary appendix B for actual tool.

Discussion

Our practical aim was a tool that might be used to monitor quality of care delivery and adapted to provide real-time feedback following resuscitations.

This study identifies critical actions important in the management of ill children presenting to an emergency department in the African setting. These actions should be performed in the first hour of care when resuscitation and stabilisation are especially important. With the use of the PBT, adherence to these actions can be assessed in real time during provision of patient care. Omission of these actions could suggest a need for focused training in disease recognition and management or evaluation of underlying processes impeding patient care.

In evaluating individual patient encounters, the PBT enables data to be gathered about individual practitioners. Such data can be aggregated to evaluate overall practices within an emergency department. This information could be used to measure change in practice following an education or policy intervention within a department. Given variability across providers and emergency departments, it is likely to have limited application in comparison between institutions.

Neither the candidate list nor the PBT are meant to be used as prescriptive guidelines for patient care. They are not comprehensive—many additional critical and non-critical actions would be required in the management of each of these conditions. The included actions here do not constitute even a minimum standard of care, nor are they necessarily more clinically important than actions that were not chosen since our selection was informed by a series of practical considerations, including challenges to implementation, staffing and resources.

We have merely identified a short list of actions that are consistent with existing guidelines, and for which there is clear consensus among relevant regional experts that the actions are solidly within a context-relevant minimum expectation for care. Our ultimate goal was to select actions whose absence would clearly reflect a modifiable gap in the quality of care delivery, not merely an acceptable variation in practice, and whose absence would not inevitably result from common regional resource constraints.

The core skills category included items similar to the Paediatric Assessment Triangle and Paediatric Emergency Assessment standards in prehospital, trauma and emergency education.27 These actions emphasise immediate evaluation of the airway, breathing and circulation and a systematic approach to life-saving interventions. Beyond that, most categories of illness had, at most, seven actions per category. Again, this relatively small number of actions should only be seen as a subset of the actions required for care of a given patient.

Many of the measures not meeting early inclusion criteria were conditional actions (eg, initiate vasopressor support after 60 mL/kg intravenous fluid bolus if circulation abnormal), specific to certain clinical scenarios (eg, measure opening pressure during lumbar puncture) or subject to resource availability (eg, obtain a head CT or MRI). Others did not meet the very high standard (80% agreement) required for consensus. Exclusion of such actions may have come as a result of selection of other actions that accomplished the same ends. For example, measuring blood pressure did not meet consensus threshold for the management of diarrhoeal illness, but assessing pulse, capillary refill and skin turgor did, and may supplant blood pressure as a test of perfusion in such patients. Participants may have preferred less specific actions to allow application of the tool to a broader variety of settings.

The expert panel nominated some actions not essential to care in all situations or environments (test for typhoid for altered mental status, administer antipyretic for active seizure, provide fluid maintenance for febrile illness). In development of the candidate list, we opted to include an action if it met consensus criteria, so as to accurately represent the opinions of the expert panel. This allows adopters of these recommendations to customise care based on common presentations within their setting. However, this product required further refinement in order to achieve the intended goal of a widely adaptable PBT.

Development of the PBT subjected these actions to more rigorous criteria. Because the Delphi model produces limited benefit with more than three rounds or when consensus begins to converge,14 28 we developed the PBT using author input instead of reconvening the expert panel. We limited introduction of bias by drawing from actions only already meeting consensus criteria. Therefore, reintroduction of excluded actions, such as measurement of blood pressure for diarrhoeal illness, was not possible. Many actions were excluded because they would not be able to be verified by an observer standing at distance (ensure airway patency, assess Glascow Coma Scale, assess for malnutrition, assess mental status), or were not applicable to every patient. Such actions are still important in the emergency care of ill patients, and exclusion reflects the challenges of creating and using such a tool. We have presented the final list of critical actions and the PBT so that institutions may use either list that best fits their needs.

All experts who received an invitation to participate were identified as having expertise in emergency medicine in an African setting, and a large majority of the expert participants were identified as working primarily in an African setting. Thus, these actions were developed with consideration of the disease burden cared for in African emergency departments, the challenges of provision of care in these settings and the level of care necessary to care for children presenting with the selected sentinel conditions. As the majority of participants work, or have experience in, African emergency departments in larger, urban hospitals some of these actions may not be feasible in smaller hospitals, particularly in rural settings where a large proportion of mortality occurs.29

Further, the majority of actions meeting inclusion were based on care guidelines with international acceptance at the time of investigation. Newly developed standards may not be represented in the results. For example, recent studies have identified the limitations of using length-based tape to estimate weight in areas with high prevalence of malnutrition.30 Despite this, the decision to use this method by the expert panel may reflect the challenges of knowledge translation and modifying entrenched practices, or the practical limitations of implementing novel methods. The PBT represents an interpretation of the candidate actions list and an attempt to address such discrepancies (use of length-based tape was modified to ‘estimate using standardised technique’). Local experts may choose to tailor the PBT prior to utilisation based on setting and resources.

We identified limitations to our study. We used input from a group of key informants identified within constraints of availability within an in-person forum. The opinion of the expert panel may not be representative of all experts within the field, but we did achieve a range of practitioners from a number of African countries representing differing disease burdens and resources.

Only a small number of those participants in round 1 attended the in-person meeting in round 2. This resulted in a different group of participants engaging in the latter half of the study, thus limiting the opportunity to submit additional novel actions. The impact of this is probably minimal as a robust number of participants was maintained for each round of this group consensus exercise.26

The actions were sorted based on the recommendations of the authors. These actions are not feasible in all settings or applicable in all presentations of a sentinel condition, hence the refinement into subsequent candidate actions and a further PBT.

Despite the above-mentioned limitations, we believe the results are supported by this process and existing literature and that the resulting tool could be adapted to individual practice environments. Additional work is needed to study implementation of these products within African emergency departments. Performance as measured by the PBT should be compared with clinical outcomes such as 48-hour survival, so as to determine the meaningfulness of collecting such information. If a consistent correlation is found between high performance and survival, the PBT could be used as a proxy to determine the benefit of quality improvement efforts in individual emergency departments.

Conclusion

By generating a consensus-based select list of critical actions for the care of severely ill children, we derived a simple, context-relevant instrument to facilitate quality assessment. These targets may be of particular use to clinicians and administrators seeking to assess the impact of educational and process interventions in the context of quality improvement efforts for the care of acutely ill children presenting for emergency care in resource-constrained settings. Further work is needed to validate the PBT and link it to process and clinical outcomes.