Designing and evaluating a novel mobile application for Helping Babies Breathe skills retention in Uganda: comparative study protocol.

INTRODUCTION: Over 600 000 newborns die each year of intrapartum-related events, many of which are preventable in the presence of skilled birth attendants. Helping Babies Breathe (HBB) is a neonatal resuscitation training programme designed for low-resource settings that can reduce both early neonatal mortality and stillbirths. However, as in other similar educational programmes, knowledge and skill retention deteriorate over time. This trend may be counteracted by strategies such as regular simulated exercises. In this study, a mobile application (app) 'HBB Prompt' will be developed to assist providers in retaining HBB knowledge and skills. METHODS AND ANALYSIS: This is a comparative study in Uganda with two phases: an app development phase and an assessment phase. In the first phase, HBB trainers and providers will explore barriers and facilitators to enhance learning and maintenance of HBB skills and knowledge through focus group discussions (FGDs). The FGDs are designed with a human factors perspective, enabling collection of relevant data for the prototype version of HBB Prompt. The app will then undergo usability and feasibility testing through FGDs and simulations. In the second phase, a minimum of 10 healthcare workers from two district hospitals will receive HBB training. Only the intervention hospital will have access to HBB Prompt. All participants will be asked to practise HBB skills every shift and record this in a logbook. In the intervention site, app usage data will also be collected. The primary outcome will be comparing skills retention 12 months after training, as determined by Objective Structured Clinical Examination B scores. ETHICS AND DISSEMINATION: This study received ethics approval from The Hospital for Sick Children and Mbarara University of Science and Technology. The authors plan to publish all relevant findings from this study in peer-reviewed journals. TRIAL REGISTRATION NUMBER: NCT03577054.

Helping Babies Breathe (HBB) is an effective neonatal resuscitation educational programme that is designed for use in low-resource settings.

After initial training, HBB provider knowledge and skills decrease over time.

Low-dose high-frequency practice can improve HBB skills retention, but it is often expensive to implement.

Through user-centred design, we will develop a novel prototype mobile application, HBB Prompt, using focus group discussions with frontline neonatal providers in Southwestern Uganda.

Through an iterative process of feasibility and usability testing, we will upgrade the app to create a final version designed for use in hospitals.

By assessing educational outcomes over 12 months, we hope to demonstrate improved scores in providers who used HBB Prompt compared with those who did not.

Introduction

In 2015, nearly half of the 5.9 million children who died under the age of 5 were newborns.1 This is after substantial progress made between 2000 and 2015 with the Millennium Development Goals by decreasing neonatal mortality in 81 countries.2 The next target for the Sustainable Development Goals of 2030 is for this number to be halved, to 12 per 1000 live births.2 For this goal to be met, large-scale proven interventions will need to be implemented. The three leading causes of neonatal death include preterm birth complications, intrapartum-related events and infection. It is estimated that 637 000 (uncertainty range 550 000–723 000) newborns die each year of intrapartum-related deaths, formerly known as ‘birth asphyxia’.1

To address the problem of intrapartum-related neonatal mortality, WHO recommends a skilled birth attendant at every delivery.3 Although the majority of newborns require only routine care, 1 out of every 10 babies require some neonatal resuscitation.4 Neonatal resuscitation has been shown to decrease neonatal mortality for births at home and in facilities.5 Helping Babies Breathe (HBB) is a neonatal resuscitation training programme developed by the American Academy of Pediatrics for providers who practise in low-resource settings. The training is simulation-based using reusable, low-cost NeoNatalieTM mannequins, resuscitators and suction devices. Since launching in 2010, the course has been taught in more than 80 countries with more than 500 000 providers trained.6 In a large-scale implementation trial in Tanzania, HBB helped reduce early neonatal mortality by 47% and fresh stillbirths by 24%.7 Significant reductions in intrapartum stillbirths and first-day mortality have also been demonstrated in other low-income and middle-income countries.8 9 In 2016, the second edition of HBB was launched to complement the 2015 International Liaison Committee on Resuscitation Consensus on Science with Treatment Recommendations and the 2012 WHO Basic Newborn Resuscitation Guidelines.6

Although HBB has been shown to be successful, one of the most significant barriers to its sustained clinical impact is provider retention of knowledge and skills. A recent systematic review of neonatal resuscitation in low-resource settings found that 5 of 10 studies examined demonstrated significant fall-off in knowledge and skills, three demonstrated no fall-off and two demonstrated mixed results among different level providers.10 Importantly, four of the five studies demonstrating no fall-off or mixed results included refresher training or regular meetings.10 Several studies have now demonstrated the importance of low-dose high-frequency (LDHF) skills practice through simulation to improve neonatal resuscitation skills retention.11–13 Importantly, Mduma et al14 have also shown that frequent, brief, on-site simulation with the HBB programme can have a sustained impact on neonatal mortality over 1 year.

Smartphone applications are used in both high-income and low-income countries for a variety of purposes in medicine including education, research and diagnostics.15 Our group has previously developed and tested a smartphone application for the Neonatal Resuscitation Program,16 the newborn stabilisation algorithm used in North America.4 Mobile technology is widely available in low-resource settings and may provide the most practical platform for an educational programme at relatively low cost.17 The goal of this project is to develop and test a novel mobile application based on the HBB programme, entitled ‘HBB Prompt’. This application will facilitate individual and group simulation-based training in health facilities using the LDHF model for resuscitation skills retention. We hypothesise that applying a user-centred design approach of collecting iterative qualitative data from end-users regarding learning needs, and facilitators and barriers to HBB skills maintenance, will enable creation of a mobile app that will improve skills retention in an intervention group versus a control group.

Methods and analysis

Study overview and population

This will be a comparative study with a development phase (phase I) and an assessment phase (phase II). In phase I, we will develop the HBB prompt app through user-centred design, eliciting input from frontline birth attendants in focus group discussions (FGDs). Once the prototype app is built, it will undergo an iterative process of usability and feasibility testing. In phase II, we will train a minimum of 20 healthcare providers from two district hospitals in HBB. Subsequently, one hospital will have access to HBB Prompt (intervention), while the control hospital will not. We will then measure app analytic data at the intervention hospital and assess skills retention at both hospitals over the course of 12 months. This study commenced on 16 May 2018.

The specific objectives of this study will be to:

Develop a mobile application, ‘HBB Prompt’, through user-centred design involving input from frontline birth attendants in Uganda.

Train a minimum of 20 health workers from two district hospitals in Southwestern Uganda in HBB.

Determine the impact of HBB Prompt on HBB skills retention of providers in the intervention site compared with providers in the control site as measured by average Objective Structured Clinical Examination (OSCE) B test scores and quality of ventilation captured by the Augmented Infant Resuscitator (AIR) at various time points within 12 months after training.

The study will be conducted at Mbarara University of Science and Technology (MUST) and two district hospitals in Southwestern Uganda: Itojo Hospital and Kitagata Hospital.

Recruitment and eligibility criteria

Participants for phase I of the study will be composed of HBB trainers and providers. Participants will be recruited from various disciplines including consultant physicians, nurses, midwives and medical officers. Inclusion criteria for phase I is prior experience with newborn resuscitation and HBB. An informed consent document will be reviewed with eligible subjects prior to their participation. Participants for phase II of the study will be recruited from Itojo Hospital and Kitagata Hospital, which are two district hospitals in Southwestern Uganda that have HBB providers but not trainers. We will include frontline birth attendants involved in the care of newborns. Once consent is obtained from the medical superintendent, individual birth attendants will be consented to participate in the study before training sessions are conducted.

Phase I: mobile app development

The objective of phase I is to develop the HBB Prompt app through user-centred design.18 We will solicit qualitative feedback regarding perceived facilitators and barriers to learning and maintaining HBB knowledge and skills through FGDs organised by provider level of HBB experience. We aim to recruit four to six participants per group to reach saturation of themes.

FGDs will include both open and closed ended questions to ensure all information that could contribute to optimising HBB Prompt function can be incorporated into its design. The questions will aim to address:

Personal experiences regarding successes and challenges in stabilisation of the newborn.

Barriers encountered and perceived needs to enable optimal maintenance of skills.

Personal experiences regarding successes and challenges of HBB training sessions.

Participant ideas on how to overcome barriers and/or to enhance their ability to maintain HBB knowledge and skills.

Whether participants have taken any refresher courses and their perspectives on refresher training.

Initially, the investigators will work together with a team of engineers from MUST to create a prototype app, HBB Prompt version 0 (v0). The app will be designed on the Android platform because it is open source and accessible on inexpensive devices. With input from the team’s specialist in human factors and user-centred design, data gathered from the FGDs will be integrated into HBB Prompt v0 to create HBB Prompt v1.

Starting with HBB Prompt v1, each version of the app will undergo a cycle of usability and feasibility testing to optimise the final version of the app to be used in phase II (see figure 1). Participants who initially provided feedback on perceived facilitators and barriers to learning and maintaining HBB knowledge and skills will be given the opportunity to explore each upgraded version of the app and then use it in simulated neonatal resuscitation training. Afterwards, they will provide feedback in FGDs. FGDs will be recorded, transcribed and analysed to inform further app development in an iterative cycle. There will be a minimum of two cycles of feedback prior to launching phase II of the study.

Figure 1

Schematic for phase I: development of Helping Babies Breathe (HBB) Prompt app through user-centred design. Iterative process of user-centred design that each new version of HBB Prompt goes through, starting with specifying context of use, specifying user requirements, then co-designing of solutions to address needs identified, followed finally by evaluating whether the requirements were met through simulations and focus group discussions (FGDs). The final version of HBB Prompt to be used in phase II will be available once all the identified requirements are addressed to satisfaction. Figure adapted from the International Organization of Standardization.18

In this part of phase I, FGD questions will address:

App content, interface, navigation, functionality, customisability and usability.

Where and how participants see the use of the app taking place.

How can the app be further modified or improved to achieve their desired functionality or fill the gaps that are identified with HBB learning.

At the time of publication of this article, we had recruited six HBB facilitators and seven frontline birth attendants (HBB providers) in developing the mobile app, which is now complete.

Phase II: assessment

In the second phase of the study, we will evaluate the impact of HBB Prompt in improving knowledge and skills maintenance. Initially, a minimum of 20 frontline health providers (10 per hospital) will be trained in HBB second edition at Itojo and Kitagata Hospitals. Following the training, one hospital (intervention arm) will be randomly selected and its providers will have access to the most updated version of HBB Prompt. The control group will not have exposure to the app post training. The intervention hospital will be provided with four Android tablet devices that will have HBB Prompt installed. Contamination between the two arms will be avoided since HBB Prompt is not available for download by the public, and the tablet devices will remain in the intervention hospital.

Participants in both centres will be asked to practise their HBB skills in pairs at least once per shift. For the intervention arm, participants will be asked to use HBB Prompt as a part of their practice routine. Both sites will receive HBB materials including practice mannequins, bag-valve mask resuscitators, bulb suctions, flip charts and an algorithm poster. Additionally, each site will have a standard logbook to record frequency of practice.

The trends and frequency of HBB practice (individual and group) will be presented to each site at quarterly debriefing meetings that will take place after participant skills assessments. These data will be captured by app analytics at the intervention site and from practice logbooks at the control site.

Measurement of outcomes

In this comparative study, we will measure the following outcomes in phase II (see table 1).

Table 1

Timeline of outcome measures during phase two and analysis plan

	0 months	3 months	4 months	6 months	9 months	12 months	Analysis plan
OSCE B	Pre/post HBB training	X		X		X*	Mann-Whitney U test for comparison of intervention vs control for each measure at each time pointWilcoxon signed-rank test for comparison of change in OSCE B scores from post-training 0 to 12 months in intervention vs control (to assess relative decay of skills)
OSCE A			X		X
Knowledge check			X		X
Bag-mask skills			X		X
AIR device measurements	X	X	X	X	X	X
Focus group discussions						X	Descriptive statistics
HBB Prompt analytic data		X		X	X	X

Summary of outcome measures throughout phase II. X marks the outcomes to be measured at each of the designated time points, which are months after initial HBB 2.0 training of participants in phase II. The primary outcome of comparing OSCE B scores between the intervention and control group 12 months after initial training is denoted by *. The remainder are secondary outcomes that will be collected and analysed.

AIR, Augmented Infant Resuscitator; HBB, Helping Babies Breathe; OSCE, Objective Structured Clinical Examination.

Primary outcome

Comparison of OSCE B scores in intervention versus control group 12 months after training.

Secondary outcomes

Comparison of OSCE B scores at 0 (before and after training), 3 and 6 months.

Knowledge and skills assessments at two unannounced visits (4 months, 9 months).

Quality of newborn resuscitation during ventilation assessments using the AIR19 at 0, 3, 6 and 12 months and at the unannounced visits at 4 and 9 months.

App analytics including pattern of usage, frequency of use, scores in simulation and quizzes and other additional features as determined by phase I.

Exit FGDs and interviews at both sites to better understand the impact of the intervention.

During phase II, there will be two unannounced visits at approximately 4 and 9 months at both sites. During these visits, we will use the following HBB second edition tools to objectively assess participants: (1) Knowledge check, (2) Bag and mask ventilation (BMV) skills check, and (3) OSCE A. The AIR device can be attached to BMV resuscitators to assess ventilation effectiveness parameters including rate, face-mask seal integrity, airway status and appropriate pressure. AIR measurements on quality of ventilation will be collected immediately before and after training, and at each announced and unannounced assessment visit. Although the AIR device can provide feedback regarding ventilation effectiveness, this feature will be made unavailable to participants. At the conclusion of phase II, there will be exit FGDs and interviews at both sites. At the intervention site, questions will address helpfulness of HBB Prompt, whether HBB Prompt meets end-user needs, app functionality, suggestions for improvement, and feasibility and sustainability of scale-up. Control site participants will be given an opportunity to provide suggestions on ways to facilitate HBB skills maintenance. Participants in both sites will be asked about facilitators and barriers to implementing and maintaining HBB skills.

At the time of publication of this article, we had recruited a total of 30 health providers for phase II. Kitagata Hospital was selected as the intervention hospital where 17 participants were recruited, and Itojo Hospital was selected as the control hospital where 13 participants were recruited. Data collection is ongoing, and we aim to complete phase II by December 2019.

Data management

Data collection in phases I and II will consist of audio recordings and transcripts from FGDs. Additionally, in phase II, we will collect a multitude of anonymous data pertaining to participants, the hospital site and the infants cared for during the study period in order to better understand potential confounders. At the provider level, we will collect data on the volume of deliveries, workload, number of patients and areas each provider is responsible for, baseline supervision, provider level, experience and previous training. At the hospital level, we will collect data on the frequency of practice by healthcare workers, availability of equipment and an HBB algorithm, number of deliveries, number of infants requiring ventilation, fresh stillbirths and neonatal mortality. For patients, we will collect data on date and time of birth, birth weight, death before discharge, admission to nursery, Apgar scores, resuscitation status at birth and outcome. This clinical data will be collected from the national birth register.

Data processing and analysis

Phase I

Transcripts from FGDs will be reviewed for themes using a grounded theory approach. Data analysis will take place after each of the iterations in phase I to incorporate feedback and improve each version of HBB Prompt. At least two investigators will independently read all transcripts to obtain an overall understanding, identify data codes and ensure that all comments are carefully considered and included. Categories within the themes will then be identified and any relationships among themes determined. After each of the iterations, categories and themes will be reviewed together with the software developers to determine and design feasible improvements to HBB Prompt. A similar approach has previously been successfully used in the development of an osteoporosis clinical decision tool.20

Phase II

Comparison of intervention versus control for the following outcomes will be done using the Mann-Whitney U test (independent samples): OSCE A scores, OSCE B scores, knowledge check scores, BMV skills check scores and AIR data. Additionally, we will compare OSCE B scores after training at 0 months and at 12 months within each arm using the Wilcoxon signed-rank test (dependent samples) to determine whether there is a difference in the expected decay in OSCE B scores. In the intervention arm, app analytic data on practice frequency will be compared with logbook data using the Kappa statistic. Multivariate linear regression models will be used to assess variables that may confound the primary outcome of OSCE B score at 12 months and ventilation quality data from AIR.

Descriptive statistics will be used to compare intervention and control sites for provider level data, facility data and patient data. For the final exit FGDs and interviews, audio recordings will be transcribed and reviewed using a grounded theory approach, similar to phase I. Coded themes and categories will be compared descriptively between participants in the intervention versus the control groups.

Patient and public involvement statement

This protocol was designed without patients and or public involvement.

Ethics and dissemination

Data stored in electronic databases will be anonymised by attributing a study ID number to each participant. All protocol modifications will be submitted as formal amendments at both sites.

The datasets generated during the current study will be available from the corresponding author on reasonable request. The authors plan to publish all relevant findings from this study in peer-reviewed journals.