Mixed methods process evaluation of pilot implementation of the African Federation for Emergency Medicine trauma data project protocol in Ethiopia.

INTRODUCTION: The African Federation for Emergency Medicine Trauma Data Project (AFEM-TDP) has created a protocol for trauma data collection in resource-limited settings using a clinical chart with embedded standardized data points that facilitates a systematic approach to injured patients. We performed a process evaluation of the protocol's implementation at Tikur Anbessa Specialized Hospital in Addis Ababa, Ethiopia to provide insights for adapting the protocol to our setting.
METHODS: During the pilot implementation period, the quality of collected data was assessed. Structured key informant interviews about participant experiences and perceptions of the protocol implementation were then conducted. Interviews were analysed using a SWOT model.
RESULTS: During pilot data collection, the overall capture rate was 21%. Variables collected with high frequency included demographics, vital signs and ED diagnosis, while mechanism of injury and ED disposition were often missed. Key informant interviews identified Strengths, Weaknesses, Opportunities and Threats to the protocol. Strengths included improved patient care, enhanced training for junior providers and facilitated data collection. Weaknesses included inadequate supervision and challenges relating to the physical size of the form, which resulted in missing data. Opportunities included retrospective research and quality improvement work. Threats included perceived lack of a local champion, poor buy-in from other hospital departments and need for ongoing financial support.
CONCLUSION: A mixed methods process evaluation is an invaluable tool when implementing novel data collection protocols, especially in resource-limited settings. We determined early successes and challenges of the implementation of the AFEM-TDP protocol and generated strategies to adapt the protocol to better suit our setting. Lessons from this process evaluation may be informative for other researchers designing and implementing similar data collection protocols.

African relevance

Injury accounts for a significant portion of the burden of disease in sub-Saharan Africa.

Systematically collecting data on injured patients is essential for surveillance and quality improvement.

A mixed-methods process evaluation of a trauma data project, as described here, is crucial during implementation of a novel data collection protocol and can provide invaluable insights for protocol improvement.

Introduction

Injury results in five million deaths per year, over a third of which could be prevented with improved trauma care in low- and middle-income countries [1], [2], [3]. Trauma registries that compile data about the demographics, injuries and care of trauma patients have been used across Africa for injury prevention, quality improvement and research endeavours [4], [5], [6], [7].

Members of the African Federation for Emergency Medicine (AFEM) developed the AFEM Trauma Data Project (AFEM-TDP) in hopes of improving both clinical care and trauma research capacity in Africa. It uses a standardized data collection form that serves as both a clinical chart for emergency department (ED) trauma patients and a research instrument. The AFEM-TDP protocol, which defines practices for collecting data with the AFEM-TDP form and entering them into a database, was developed at Muhimbili National Hospital in Tanzania and has been implemented at hospitals in five countries in sub-Saharan Africa (SSA) to date.

This study presents a mixed-methods process evaluation of the pilot implementation of the AFEM-TDP protocol in Ethiopia. The goals of the study were to identify early successes and challenges of the implementation of the protocol and to develop strategies for site-specific protocol adaptation.

Methods

Tikur Anbessa Specialized Hospital (TASH) is the quaternary trauma referral centre in Ethiopia [8]. It is staffed by attending physicians, residents specializing in emergency medicine (EM) and medical students in their last year of undergraduate training, who are called interns in Ethiopia.

The AFEM-TDP was introduced at TASH by AFEM staff and pilot data collection using the AFEM-TDP protocol started in June 2014. EM residents and interns rotating in the TASH ED were tasked with completing the AFEM-TDP forms. They received an orientation to the protocol as well as ongoing feedback and reinforcement from attendings and ED research nurses. The original copies of completed AFEM-TDP forms were placed in patients’ charts for clinical documentation. Carbon copies of the forms were collected and digitalized by specially trained data clerks using Adobe Acrobat forms. Data were compiled and analysed using Microsoft Excel.

The quantitative component of this evaluation consists of analysis of the quality of the first two months’ pilot data, quantified by overall capture rate and form completeness. The qualitative component of the evaluation consists of a series of structured key informant interviews with TASH ED staff, conducted four months after pilot implementation of the protocol. Interviews evaluated participant experiences of the protocol implementation and addressed data quality issues identified in the quantitative component of the evaluation. Responses were recorded, transcribed and coded in a SWOT framework to address Strengths, Weakness, Opportunities and Threats of the protocol implementation, which is widely used in health services research to promote awareness and foster program improvement [9].

The study was reviewed and approved by the institutional review boards of Addis Ababa University and the University of California San Francisco.

Results

A total of 174 forms were completed and entered into the AFEM-TDP database over a two-month period shortly after pilot implementation of the protocol at TASH. Compared with 821 trauma-related visits recorded in ED clinical logbooks, this yields a 21% overall capture rate. Among the forms that were filled, completeness varied widely by variable. The completeness rate was highest for data points that were recorded at the time of patient arrival, including age (96%), sex (99%), systolic blood pressure (77%), pulse (86%) and Glasgow Coma Score (98%). Most patients also had an ED diagnosis documented (97%). However, injury mechanism (40%) and ED disposition (29%) were often left blank.

Thirteen respondents were interviewed about their early experiences with the AFEM-TDP including four interns, five EM residents, one EM attending, two ED nurses and one ED data clerk. The results of these interviews are summarized in Table 1, with representative quotations from the interview transcripts.

Table 1

SWOT analysis of interview data about implementation of the AFEM-TDP protocol at Tikur Anbessa Specialized Hospital with representative quotations.

Strengths
1. Improves quality of patient care	“It is very helpful—detailed, exhaustive, systematic.”
2. Tool for resident education	“When we go to the countryside, we will remember this and use what we learned there.”
3. Tool for data collection	“It is useful for many research projects for everyone in the department.”
Weaknesses
2. Not self-explanatory	“It was a bit uncomfortable initially since it mostly uses check boxes, and we are used to writing a history note.”
1. Cumbersome size	“It’s difficult to read on rounds because it is folded in four—you can’t open it easily.”
3. Often missed at high volume times	“When there is a mass casualty we evaluate all of the patients first so when we sit to fill the form it may be 15 patients that we have seen by then.”
Opportunities
1. Can provide a dataset for retrospective research	“With this we can analyze morality, severity, causes, geographical locations of injuries.”
2. Can identify opportunities for quality improvement	“It will help us to identify things that we aren’t doing properly. If we can identify our problems, we can fix them.”
3. Can be expanded to other sites	“We can replicate it at other hospitals in the region, and when we go to work in the countryside we can use it there, too.”
Threats
1. Lack of a clear local champion	“We need commitment from all members of the department, but everyone has other duties and responsibilities.”
2. Poor buy-in from other departments	“If consulting residents see nothing in the card except the trauma registry form they will complain and even leave without seeing the patient.”
3. Need for ongoing funding and support	“I have a fear that this will not persist. Who will pay for this in the future?”

Regarding its strengths, many respondents believed the AFEM-TDP protocol helped to improve the quality of patient care by encouraging a standardized approach to trauma patients and guiding junior providers through a thorough exam. Multiple trainees remarked that the form helped them learn how to assess trauma patients. One intern mentioned that it was especially useful on nights and weekends when many trauma patients came in and there was less supervision or free time for teaching. Several respondents saw the AFEM-TDP protocol’s ability to facilitate data collection as an important asset.

Regarding its weaknesses, several respondents noted that the form, which heavily utilized check boxes, was confusing for junior providers who did not use similar forms elsewhere and did not always receive adequate orientation. The size of the form, which was printed on one 40 × 60 cm sheet, made it cumbersome and difficult to bring to the bedside. Several respondents suggested that this resulted in patients being missed, especially when unstable patients or mass casualties arrived. Once folded in quarters and placed in the medical chart, the form was often overlooked by consultants. Finally, several respondents thought a single form was insufficient to document all the important details of an ED stay, which commonly lasts for several days at TASH.

Regarding its opportunities, multiple respondents mentioned the possibility of using data collected through the protocol for retrospective research to address patterns of injuries and clinical outcomes of trauma patients at TASH. Several respondents suggested that data gathered through the protocol could help to identify patients with poor outcomes and inform quality improvement projects. Some respondents hoped the AFEM-TDP protocol would be implemented in other hospitals in Ethiopia, leading to a fuller picture of the burden of injury nationally.

Regarding threats to the AFEM-TDP protocol, several respondents felt like the project lacked a clear local champion who was responsible for overseeing implementation of the protocol. Others mentioned poor buy-in from consulting surgical services, who discouraged ED staff from using the AFEM-TDP form or demanded free-form History and Physical notes in the chart before staffing consults or admitting patients. Finally, some respondents voiced scepticism about long-term sustainability of the protocol given its ongoing costs.

Discussion

By using complementary quantitative and qualitative data sets, this study was able to assess the quality of data collected during pilot implementation of the AFEM-TDP protocol at TASH and to gain insight into the protocol’s strengths and challenges. A mixed methods approach can provide more nuanced insights than either methodology could alone [10]. By conducting a process evaluation early in protocol implementation, we were able to obtain timely feedback and identify opportunities to adapt the AFEM-TDP protocol for TASH.

The quantitative component of the evaluation demonstrated a troublingly low overall capture rate, which could result in a distorted picture of trauma at TASH if not addressed. While demographic data and triage vital signs were frequently captured, variables relating to injury mechanism and ED disposition were often missed. These data points have significance for epidemiologic research and injury prevention efforts.

The qualitative component of the evaluation used a SWOT framework proved to synthesize its results for analysis. Strengths included the protocol’s potential to improve the quality of patient care, its role in training junior providers and its ability to facilitate data collection. Weaknesses included the form’s unfamiliar layout, its large physical size, and challenges with patients who had prolonged ED stays. Opportunities included the potential for epidemiologic research and quality improvement work and the possibility of expanding the protocol to other sites. Threats included perceived lack of a local champion for the project, poor buy-in from other departments in the hospital and the need for ongoing financial support.

Study results and suggestions to adapt the protocol to better suit TASH were communicated to key stakeholders including the local trauma registry team and international partners at a meeting convened for this purpose. Suggestions included changing the layout of the AFEM-TDP form to improve clarity and strengthening the orientation and supervision processes. Growth of the EM residency now allows for increased presence of senior providers at triage and more staffing at times of high trauma volume. These changes may result in higher capture rates and improved data quality moving forward. A new trauma follow-up form has also been developed to better capture data about prolonged ED stays.

While several respondents in this study said they felt using the AFEM-TDP form helped them to be more thorough in their patient assessment, a study by Laing et al. in South Africa found that a similarly designed admission form that relied heavily on check boxes did improve the quality of documentation but did not clearly impact the quality of patient care [11]. An important next step in this project will address changes in both data quality and patient care over time following the implementation of the AFEM-TDP protocol.

Since the pilot implementation of the AFEM-TDP protocol at TASH, the project has evolved into a collaboration between AFEM and WHO that uses a new data form based on the standardized elements of the WHO Data Set for Injury and linked to the WHO Emergency Care Registry. Experiences at TASH were incorporated into this process, amplifying the impact of this study. The preliminary experiences with the AFEM-TDP at TASH were also pivotal in cultivating support for a sustainable trauma registry in the Ethiopian Federal Ministry of Health. A research partnership between academic clinicians and the Ministry of Health is a powerful strategy to address several of the threats identified in this study, including poor buy-in from other departments in the hospital and the need for ongoing financial support.

This study adds to a small but growing body of literature of implementation research to evaluate novel clinical protocols in SSA. Rujumba et al. used qualitative interviews with health workers to obtain feedback to strengthen a program to prevent mother-to-child HIV transmission in Uganda [12]. Mash et al. used qualitative interviews with clinic staff that were analysed using a SWOT framework to evaluate the pilot of a diabetic retinopathy screening protocol in South Africa [13]. Laing et al. used data audits and user satisfaction surveys to evaluate a hybrid electronic medical record system in South Africa [14]. Stone and Ndagijimana also used a mixed methods approach, supplementing pre-/post-intervention quantitative data about disease rates with qualitative interviews to evaluate the feasibility, acceptability and effectiveness of an educational intervention addressing basic hygiene in Rwanda [15]. While the use of qualitative and mixed methods process evaluations and SWOT analyses remain rare in SSA, these studies demonstrate that they can help to ensure that protocol implementation is effective and well-adapted to local clinical contexts.

Conclusions

Our study highlights the benefits and pitfalls of implementing a new clinical documentation and data collection protocol at a busy teaching hospital in SSA. The AFEM-TDP protocol has the potential to be a powerful tool for improving patient care and collecting data for research and quality improvement. An early process evaluation identified strengths and challenges of the protocol, and elucidated opportunities to adapt it to site-specific needs. This evaluation process and the lessons it generated may be informative for researchers developing and implementing similar data collection protocols.

Dissemination of results

Results this study were shared locally with key stakeholders in a meeting to discuss modifications to the AFEM-TDP protocol in Ethiopia. Findings were presented at the World Congress on Disaster and Emergency Medicine in Cape Town, South Africa in 2015.

Author contribution

Authors contributed as follows to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: AL contributed 50%; AA and TAR each contributed 10%; and BG, FD, LB, HS, ML and JW each contributed 5%. All authors approved the version to be published and agreed to be accountable for all aspects of the work.

Conflicts of interest

The authors declare no conflicts of interest.