Mobile application adjunct to the WHO basic emergency care course: a mixed methods study.

OBJECTIVES: The WHO developed a 5-day basic emergency care (BEC) course using the traditional lecture format. However, adult learning theory suggests that lecture-based courses alone may not promote long-term knowledge retention. We assessed whether a mobile application adjunct (BEC app) can have positive impact on knowledge acquisition and retention compared with the BEC course alone and evaluated perceptions, acceptability and barriers to adoption of such a tool.
DESIGN: Mixed-methods prospective cohort study. PARTICIPANTS: Adult healthcare workers in six health facilities in Tanzania who enrolled in the BEC course and were divided into the control arm (BEC course) or the intervention arm (BEC course plus BEC app). MAIN OUTCOME MEASURES: Changes in knowledge assessment scores, self-efficacy and perceptions of BEC app.
RESULTS: 92 enrolees, 46 (50%) in each arm, completed the BEC course. 71 (77%) returned for the 4-month follow-up. Mean test scores were not different between the two arms at any time period. Both arms had significantly improved test scores from enrolment (prior to distribution of materials) to day 1 of the BEC course and from day 1 of BEC course to immediately after BEC course completion. The drop-off in mean scores from immediately after BEC course completion to 4 months after course completion was not significant for either arm. No differences were observed between the two arms for any self-efficacy question at any time point. Focus groups revealed five major themes related to BEC app adoption: educational utility, clinical utility, user experience, barriers to access and barriers to use.
CONCLUSION: The BEC app was well received, but no differences in knowledge retention and self-efficacy were observed between the two arms and only a very small number of participants reported using the app. Technologic-based, linguistic-based and content-based barriers likely limited its impact. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Mixed-methods study design enables both quantitative and qualitative assessment of outcome measures of interest.

Two-armed intervention-based prospective cohort study with relatively large number of participants and high follow-up rate provides a solid ground for drawing conclusions.

Significant difficulties were encountered in both installation and subsequent usage of the app, despite attempts to mitigate issues. This limited the study’s ability to assess impact of the app on knowledge retention.

Study was limited to a single country in Eastern Sub-Saharan Africa, thus limiting generalisability of results.

Introduction

Well-organised emergency care systems play an important role in the delivery of emergency services and health outcomes of patients in low-resource settings.1–6 However, healthcare workers in low and middle-income countries (LMICs) often lack basic training to recognise and treat life-threatening conditions,7 8 and access to adequate emergency care in low-resource settings is limited.9 Although over half the deaths in LMICs are potentially addressable by prehospital emergency care,10 the burden of acute diseases, coupled with the lack of emergency care training in LMICs, leads to unnecessary deaths. Additionally, staff working in LMICs frequently site lack of training in emergencies as a frustration, leading to lack of satisfaction with their work and desire to work in other parts of the hospital.11

To give countries the tools to reduce mortality from emergency conditions, the WHO, in collaboration with the International Committee of the Red Cross and with endorsement by the International Federation for Emergency Medicine, developed a 5-day, open-access basic emergency care (BEC) course to provide standardised training in basic assessment and life-saving techniques using a traditional lecture format. The BEC course materials focus on the primary survey, or airway, breathing, circulation, disability and exposure (ABCDE) approach to emergencies. The course was developed using a robust consensus process, and initial pilot studies showed that the course was both effective in knowledge transfer and locally acceptable in varied communities of Tanzania, Uganda and Zambia.12

Given the unprecedented increase in mobile phone and internet users as well as a decline in the price of devices and services in LMICs,13 supplementing the BEC course with technology-based, ‘just-in-time’ content that can be used at the patient bedside was suggested by end users. Indeed, adult learning theory suggests that lecture-based courses alone may not promote long-term knowledge retention and that just-in-time training might be preferable.14 Just-in-time training, defined as training that is done immediately prior to the procedure, has been shown to improve provider performance in a variety of surgical and non-surgical procedures across multiple specialties and practice settings.15 However, it is unknown if these newer techniques are acceptable to, or useful for, healthcare workers in LMICs who are presented with medical emergencies.

In response to this request for additional support, a mobile application was developed as an adjunct to the BEC course in 2017. The application walked the user through a decision-making process and is based on the principles of the BEC course. It was intended to enable learners to access content before the course through simple links as well as basic yes/no html pathways through primary survey (ie, ABCDE) elements, which could serve as a real-time clinical, bedside reference after course completion. We conducted a small pilot study in Dar es Salaam, Tanzania in 2017 to compare knowledge acquisition and retention between participants who received the standard BEC course alone and participants who received the course with educational adjuncts (a suite of clinical cases to simulate patient care and the mobile application). Preliminary findings were promising, indicating that such tactics may improve knowledge and retention. However, the study was small, results did not reach statistical significance, and the study had several limitations.16

The purpose of this study was to assess if a mobile application adjunct to the BEC course, specifically, can impact knowledge acquisition, knowledge retention and work experience compared with the BEC course alone. We hypothesised that the application would improve knowledge retention, knowledge acquisition and work experience. In addition, we aimed to evaluate perceptions, acceptability and barriers to adoption of such a tool.

Methods

Development of second-generation mobile BEC application

A second-generation mobile application adjunct to the BEC course (BEC app) was developed for this study. Similar to the first-generation application that was used in the 2017 pilot, the second-generation BEC app was designed to be a point-of-care reference tool that guides healthcare workers through the ABCDE approach and the evaluation and treatment of seriously ill patients at the bedside (online supplemental appendix A). The second-generation BEC app incorporated extensive feedback from the first-generation application and addressed reported limitations with that version, including the ability to function without internet access and track application use centrally.17 The second-generation BEC app was designed as an expanding logic framework that could be deployed within a self-contained app on any mobile operating system or accessed through chatbot services across multiple platforms like WhatsApp or Facebook Messenger as well as short message service (SMS) for the greatest possible reach. Additionally, a chat-based approach was believed to allow for broader translation from English to other languages as it does not always require completely rewriting the app software, but rather may allow for common translational services to be employed.

The second-generation BEC app was developed through a collaboration between study investigators and a software engineer. It underwent extensive prerelease testing with emergency medicine doctors and nurses across the world, including Tanzania, Rwanda, Cameroon, Sierra Leon, South Africa, United Arab Emirates and the USA. Reviewers were asked to recall their last five emergency cases, run them through the app and provide feedback on the algorithm and application interface based on this process. Updates to the application were made accordingly.

Study design

We conducted a prospective, cohort study with adult (over 17 years of age) healthcare workers without prior formal emergency care training in six health facilities in Tanzania. The six facilities include Mt. Meru Regional Hospital in Arusha Region; Mbagala Rangi Tatu Health Center in Temeke, Dar es Salaam; Sinza Health Centre in Kinondoni, Dar es Salaam; Vijibweni Health Center in Kigamboni Dar es Salaam; Mkuranga District Hospital in Coastal Region and Mbeya Zonal Referral Hospital in Mbeya. These are considered busy hospitals with high-acuity patients who currently receive evaluation and care by providers who have not undergone emergency care training.

The onboarding, courses and follow-up—including all associated assessments and focus groups (FGs)—were delivered at or near these six facilities. Courses at Sinza Health Center; Mbagala Rangi Tatu Health Center and Mbeya Zonal Referral Hospital consisted of participants from these sites only, while the courses at Mt. Meru Regional Hospital; Vijibweni Health Center and Mkuranga District Hospital were comprised of participants from multiple sites within the district.

The study consisted of two arms. The first arm was the control arm with participants in this arm undergoing the traditional BEC course, but without access to the mobile application. The second arm was the intervention arm. Participants in this arm were given access to the BEC app in addition to completing the BEC course. Mkuranga District Hospital, Mbagala Zakheim and Mt. Meru Hospital were arbitrarily assigned to be in the control arm (arm 1, non-app arm) and Sinza Health Center; Mbeya Zonal Referral Hospital and Vijibweni Health Centre to be in the intervention arm (arm 2, app arm).

Participants from each site were selected from a convenience sample from these six hospitals based on voluntary enrolment in July 2019. Participants were initially approached and offered the opportunity to volunteer by Medical Officers Incharge at each site. There were no exclusion criteria. Prior to enrolment in the course, informed consent to participate in the study was obtained from all participants (online supplemental appendix B). Participants received no financial incentives for participating in the course or enrolling in the study. Ninety-seven participants were enrolled across the six sites 1.5–2 weeks prior to course initiation.

On enrolment in the study (T1), participants were given a multiple-choice knowledge assessment to assess baseline knowledge, Likert scale self-efficacy survey (online supplemental appendix C) and BEC course materials. Knowledge assessments used at all phases of the study were developed by WHO and are a standard part of the BEC course. Recognising the inherent knowledge acquisition through test-enhanced learning,18 the same knowledge assessments were given to participants in both arms at the same time points. In the absence of existing tools with validity evidence in our study population, the self-efficacy survey was designed by our team to assess key aspects of the workplace experience, including participants’ comfort with, satisfaction with and enjoyment of their work.

Basic information about demographics (including age, sex) and professional background (including role, years of experience) were gathered at enrolment. Participants in the app arm who had smart phones compatible with the mobile application were given the opportunity to download the BEC app with moderator support.

Basic emergency care

BEC courses were held at each healthcare site over a 2-week period, with three courses running simultaneously during the first week and the second week. Courses at non-app arm facilities ran during the first week, while courses at app arm facilities ran during the second week.

Of the 97 participants who enrolled, 92 participants (46 in each arm) presented on the first day of the course (T2). Participants who were not present for the initial enrolment were not permitted to join the course. All participants were given a second multiple-choice knowledge assessment on the first day of the course prior to any course instruction. Participants in the app arm who were not able to successfully access the BEC app during the initial onboarding were given another opportunity to download the BEC app during the course. Of the 46 participants within the app arm, 42 had smartphones. These participants were assisted with BEC app downloading and troubleshooting of technical issues, with no discussion of course content. Of these participants, 29 were able to download and use the mobile application.

Swahili is participants’ native language and English is spoken as a second language. Each of the BEC courses was taught by local instructors fluent in both English and Swahili. Each course had at least four instructors, with two instructors from the site at which the course was held and the rest from an outside site. At the end of the course (T3), participants completed a third multiple-choice knowledge assessment and the same Likert-scale self-efficacy survey they filled out at onboarding. All 92 participants who initiated the course completed the postcourse assessment.

Written course materials, the BEC app, course assessments and self-efficacy surveys were provided in English. If a participant did not meet the 75% passing threshold for the final examination, questions answered incorrectly were translated to Swahili to address language barriers. The new score was used as the basis for successful completion of the course, but the initial score was used for study purposes. All data were collected at the location of training at each site.

Focus groups

After completion of the BEC courses, participants were invited to participate in FGs to elicit perspectives of accessibility, applicability and opportunity for improvement of the BEC course and BEC app without course instructors or study moderators. Each 1-hour FG was moderated by two Swahili-speaking facilitators, who were male or female doctors (MD) or nurses (RN) at the Muhimbili National Hospital Emergency Medicine Department and who underwent training on how to facilitate FGs led by an experienced FG facilitator. FGs were conducted in Swahili with a written facilitator guide (online supplemental appendix D). Field notes were taken during the FG by one of the FG facilitators. In order to minimise potential bias, participants had no prior relationship with FG facilitators and understood them to be external parties (outside of the course structure) present only to facilitate the FGs. FGs were audio recorded, content was transcribed and translated into English, and identifying information was redacted by the FG facilitators. Original recordings were subsequently deleted.

Study follow-up

Approximately 4 months after completion of the BEC courses (T4), participants in both arms were reassessed with a fourth multiple-choice knowledge test and another Likert-scale self-efficacy survey. Participants were also invited to participate in another FG discussion (online supplemental appendix E) that was structured similarly to the initial. Seventy-one out of the 92 enrolees participated in the follow-up. US$25-equivalent stipend was provided for their participation in the follow-up. As with the initial pilot, following the conclusion of the study, access to the BEC app was made freely available to participants in the non-app arm.

Data analysis

Outcomes of interest in this study were participant scores on the knowledge assessments at four time points; scores from the Likert-scale self-efficacy survey at three time points and qualitative FG data on user perspectives of the BEC app. An intention-to-treat analysis was performed to compare the two arms.

Descriptive statistics were used to compare demographic and professional characteristics between the two arms. We used ANOVA models to compare mean knowledge assessment scores by time point and further analysed differences by site using analysis of variance (ANOVA)-based mixed-effect models, accounting for repeated measures within the same individual. Responses to the self-efficacy survey in the group of study completers were compared at three time points with either χ2 or Fisher exact tests, as appropriate. All analyses were performed using R V.3.6 (R Project for Statistical Computing).

Two investigators (ALG and AF) conducted thematic analysis of the FG transcripts.19 Transcripts were independently read by both investigators and codes were identified using an inductive approach. The investigators then came to a consensus on the codebook based on joint discussion and recoded all transcripts accordingly. Codes were reviewed to generate themes and subthemes through iterative discussion and refinement by the two investigators. The process was repeated for the follow-up FGs. Codes generated from the initial phase, as well as emerging codes, were applied. Themes and subthemes were identified. One investigator used the qualitative analysis software Atlas.ti V.8, whereas the other used Microsoft Word.

Results

A total of 92 healthcare workers (table 1) from the six participating sites, 46 (50%) in each arm, completed the BEC course and the first phase of the study, including a knowledge assessment and self-efficacy survey at onboarding (T1), a second knowledge assessment on the first day of the course (T2) and a third knowledge assessment and second self-efficacy survey on completion of the course (T3) (figure 1). Per intention to treat analysis, participants were compared according to their initial arm allocation; all 46 participants in arm 1 were compared against all 46 participants in arm 2 (including those who were unable to download the app).

Table 1

Demographic and professional characteristics of participants by study arm

	All sites n=92		Arm 1 (Non-App) n=46		Arm 2 (App) n=46
	n	%	n	%	n	%
Age group
20–29	31	34	14	30	17	37
30–44	48	52	25	54	23	50
45–59	12	13	7	15	5	11
Sex
Female	56	61	29	63	27	59
Male	36	39	17	37	19	41
Role
Doctor	36	39	21	46	15	33
Nurse	56	61	25	54	31	67
Experience
<5 years	47	51	19	41	28	61
5 or more years	45	49	27	59	18	39

Figure 1

Flow diagram of study participants.

No significant differences in demographic or professional characteristics were observed between the two arms. While compared with the non-app arm, participants in the app arm had a lower absolute mean age (32.8 years vs 36.1 years), this difference did not reach statistical significance (p=0.71). Participants in both arms were more likely to be woman (61%) and more likely to be nurses (61%), with no difference in proportion of women (p=0.83) or nurses (p=0.29) between the two arms. A trend towards statistical significance was seen in years of experience, where compared with the non-app arm, participants in the app arm were more likely to have less than 5 years of experience (p=0.09).

Knowledge assessments

Knowledge assessments were administered at all four time points, T1–T4. Mean knowledge assessment scores at T1 (prior to any exposure to BEC course materials or the BEC app) were not statistically different between the two arms (62.9% arm 1, 61.5% arm 2) (table 2). Mean scores at T2 (after the distribution of BEC course materials for all enrolees and download of the BEC app for a small minority of participants in arm 2), were significantly higher than T1 (p<0.001) for both arms, but no difference was observed in the mean scores at T2 between the two arms. Mean knowledge assessment scores at T3 (after completion of the BEC course) were significantly higher than T1 (p<0.001) and T2 (p<0.001) for both arms, but again no difference was observed in the mean scores at T3 between the two arms.

Table 2

Knowledge assessment scores at onboarding (T1), first day of course (T2), at course completion (T3) [P1] and 4-month follow-up (T4) by study arm [P2]

Phase		All enrolees (n=92)
	Time	All sites			Arm 1 (Non-App)			Arm 2 (App)			P-value
	Period				n=46			n=46
		Mean	SD	95% CI	Mean	SD	95% CI	Mean	SD	95% CI
P1	T1	62.2	14.6	60.7 to 63.7	62.9	17.0	60.4 to 65.4	61.5	11.9	59.7 to 63.3	0.64
P1	T2	80.1	12.3	78.8 to 81.4	80.7	13.6	78.7 to 82.7	79.6	10.9	78.0 to 81.2	0.67
P1	T3	86.5	12.1	85.2 to 87.8	86.6	13.7	84.6 to 88.6	86.4	10.4	84.9 to 87.9	0.95
Phase		Enrolees present at follow-up (n=71)
	Time	All Sites			Arm 1 (Non-App)			Arm 2 (App)			P-value
	Period				n=32			n=39
		Mean	SD	95% CI	Mean	SD	95% CI	Mean	SD	95% CI
P1	T1	62.0	14.6	60.3 to 63.7	60.9	17.4	57.8 to 64.0	62.8	12.0	60.9 to 64.7	0.58
P1	T2	81.3	11.8	79.9 to 82.7	81.9	13.0	79.6 to 84.2	80.8	10.9	79.1 to 82.5	0.7
P1	T3	87.3	11.5	85.9 to 88.7	87.9	12.5	85.7 to 90.1	86.9	10.7	85.2 to 88.6	0.72
P2	T4	84.1	11.9	82.7 to 85.5	84.5	12.0	82.4 to 86.6	83.7	11.9	81.8 to 85.6	0.76

Of the 92 enrolees who completed the BEC course and assessments in the first phase (P1), 71 (77%) returned for the second phase (P2) while 21 (23%) were lost to follow-up. The trends in knowledge assessments during P1 for this subgroup are consistent with those seen for the overall group of enrolees who completed the BEC course. For this subgroup, mean scores at T2 were significantly higher than those at T1 (p<0.001) and mean scores at T3 were significantly higher than those at T1 and those at T2 (p<0.001) for both arms.

While there was some drop-off in mean scores from T3 to T4 for both arms, this change was not significant for either arm (p=0.78 arm 1, p=0.60 arm 2) and mean scores for both arms at T4 remained significantly higher than those at T2 (p<0.001) and T1 (p<0.001). No difference in mean scores was observed between the two arms at any of the four time periods (figure 2).

Figure 2

Mean knowledge assessment scores at time points T1–T4 by study arm, enrolees present at follow-up only.

Self-efficacy surveys

Four-point Likert scale self-efficacy surveys were administered at three time points: at onboarding (T1), at completion of the BEC course (T3) and at the 4-month follow-up (T4). The self-efficacy survey was not administered at T2. Responses were compared among the 71 participants who completed the surveys at all three time points and are seen in table 3.

Table 3

Self-efficacy survey scores at onboarding (T1), course completion (T3) and 4-month follow-up (T4) for both study arms combined, enrolees present at follow-up only

n=71	T1		T3		T4		P-value
	n	%	n	%	n	%
Q1: Do you see patients in your setting who present with emergency conditions?
Never	1	1	0	0	0	0	0.05
Almost never	0	0	0	0	0	0
Sometimes	22	31	11	16	21	30
Frequently	48	68	60	85	49	69
Q2: How comfortable do you feel taking care of patients with emergency conditions?
Very uncomfortable	6	9	0	0	1	1	<0.001
Somewhat uncomfortable	13	18	7	10	3	4
Comfortable	28	39	20	28	40	56
Very comfortable	23	32	44	62	26	37
Q3: Do you feel like you have the resources that you need to deliver basic care to patients with emergency conditions?
Never	3	4	1	1	1	1	0.35
Almost never	5	7	1	1	5	7
Sometimes	49	69	48	68	50	70
Frequently	14	20	21	30	14	20
Q4: How comfortable are you with your knowledge of emergency conditions?
Very uncomfortable	5	7	0	0	2	3	<0.001
Somewhat uncomfortable	19	27	3	4	4	6
Comfortable	32	45	24	34	36	51
Very comfortable	15	21	44	62	28	39
Q5: How satisfied are you with your ability to do your work well?
Unsatisfied	2	3	0	0	0	0	0.01
Somewhat unsatisfied	5	7	2	3	2	3
Somewhat satisfied	44	62	32	45	46	65
Very satisfied	19	27	37	52	22	31
Q6: Do you enjoy your work?
Never	0	0	0	0	0	0	0.74
Almost never	0	0	0	0	0	0
Sometimes	27	38	24	34	28	39
Frequently	44	62	47	66	42	59
Q7: Do you feel like you are able to make a difference in your patients’ lives?
Never	0	0	0	0	0	0	0.14
Almost never	0	0	0	0	0	0
Sometimes	27	38	17	24	26	37
Frequently	44	62	54	76	44	62

Significant decreases from T1 to T3 in the proportion of responses of ‘1’ or ‘2’ were observed for question 2 (‘very uncomfortable’ or ‘somewhat uncomfortable’ taking care of patients with emergency conditions; p=0.015) and question 4 (‘very uncomfortable’ or ‘somewhat uncomfortable’ with knowledge of emergency conditions; p=<0.001). A decrease from T1 to T3 in the proportion of responses of ‘1’ or ‘2’ for question 5 (‘unsatisfied’ or ‘somewhat unsatisfied’ with ability to do work well) was also observed, but it did not reach significance (p=0.16). Significant differences in the proportion of responses of ‘1’ or ‘2’ between T3 and T4 were not observed for question 2 (p=0.54), question 4 (p=0.47) or question 5 (p>0.99).

There was a significant increase from T1 to T3 in the proportion of responses of ‘4’ for question 1 (‘frequently’ see patients who present with emergency conditions; p=0.03), question 2 (‘very comfortable’ taking care of patients with emergency conditions; p<0.001), question 4 (‘very comfortable’ with knowledge of emergency conditions; p<0.001), question 5 (‘very satisfied’ with ability to do work well; p=0.004), and question 7 (‘frequently’ able to make a difference in patient’s lives; p=0.004). The proportion of responses of ‘4’ decreased significantly from T3 to T4 for these questions, returning to T1 levels for question 1, 2, 5 and 7 but to a level still significantly higher than T1 for question 4 (p=0.02).

The same trends were observed for each individual arm, with no differences in responses between the two arms for any question at any time point (table 4).

Table 4

Self-efficacy survey scores at onboarding (T1), course completion (T3) and 4-month follow-up (T4) by study arm, enrolees present at follow-up only

	T1			T3			T4
	Arm 1	Arm 2		Arm 1	Arm 2		Arm 1	Arm 2
n=71	n (%)	n (%)	P-value	n (%)	n (%)	P-value	n (%)	n (%)	P-value
Q1: Do you see patients in your setting who present with emergency conditions?
Never	0 (0)	1 (3)	0.13	0 (0)	0 (0)	0.10	0 (0)	0 (0)	0.44
Almost never	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Sometimes	7 (22)	15 (38)		2 (6)	9 (23)		8 (25)	13 (33)
Frequently	25 (78)	23 (59)		30 (94)	30 (77)		24 (75)	25 (64)
Q2: How comfortable do you feel taking care of patients with emergency conditions?
Very uncomfortable	3 (9)	3 (8)	0.81	0 (0)	0 (0)	0.52	0 (0)	1 (3)	0.42
Somewhat uncomfortable	6 (19)	7 (18)		2 (6)	5 (13)		1 (3)	2 (5)
Comfortable	11 (34)	17 (44)		8 (25)	12 (31)		16 (50)	24 (62)
Very comfortable	12 (38)	11 (28)		22 (69)	22 (56)		15 (47)	11 (28)
Q3: Do you feel like you have the resources that you need to deliver basic care to patients with emergency conditions?
Never	3 (9)	0 (0)	0.06	0 (0)	1 (3)	0.17	0 (0)	1 (3)	0.84
Almost never	3 (9)	2 (5)		1 (3)	0 (0)		3 (9)	2 (5)
Sometimes	23 (72)	26 (67)		19 (59)	29 (74)		22 (69)	28 (72)
Frequently	3 (9)	11 (28)		12 (38)	9 (23)		7 (22)	7 (18)
Q4: How comfortable are you with your knowledge of emergency conditions?
Very uncomfortable	3 (9)	2 (5)	0.16	0 (0)	0 (0)	0.23	1 (3)	1 (3)	0.84
Somewhat uncomfortable	10 (31)	9 (23)		0 (0)	3 (8)		2 (6)	2 (5)
Comfortable	16 (50)	16 (41)		13 (41)	11 (28)		18 (56)	18 (46)
Very comfortable	3 (9)	12 (31)		19 (59)	25 (64)		11 (34)	17 (44)
Q5: How satisfied are you with your ability to do your work well?
Unsatisfied	1 (3)	1 (3)	0.93	0 (0)	0 (0)	0.81	0 (0)	0 (0)	0.81
Somewhat unsatisfied	3 (9)	2 (5)		1 (3)	1 (3)		1 (3)	1 (3)
Somewhat satisfied	20 (62)	24 (62)		13 (41)	19 (49)		22 (69)	24 (62)
Very satisfied	8 (25)	11 (28)		18 (56)	19 (49)		9 (28)	13 (33)
Q6: Do you enjoy your work?
Never	0 (0)	0 (0)	0.81	0 (0)	0 (0)	0.62	0 (0)	0 (0)	0.22
Almost never	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Sometimes	13 (41)	14 (36)		12 (38)	12 (31)		10 (31)	18 (46)
Frequently	19 (59)	25 (64)		20 (62)	27 (69)		22 (69)	20 (51)
Q7: Do you feel like you are able to make a difference in your patients’ lives?
Never	0 (0)	0 (0)	0.81	0 (0)	0 (0)	0.99	0 (0)	0 (0)	0.46
Almost never	0 (0)	0 (0)		0 (0)	0 (0)		0 (0)	0 (0)
Sometimes	13 (41)	14 (36)		8 (25)	9 (23)		10 (31)	16 (41)
Frequently	19 (59)	25 (64)		24 (75)	30 (77)		22 (69)	22 (56)

FG discussions

Two FGs were held at each study site; the first immediately after completion of the BEC course (T3) and the second at the 4-month follow-up (T4). Comparison of recurrent codes across app sites and both time periods revealed five major emerging themes related to the perceptions, acceptability and barriers to adoption: educational utility, clinical utility, user experience, barriers to access and barriers to use (table 5).

Table 5

Major themes, subthemes, concepts and considerations raised at phase I and phase II focus groups

Major theme	Subtheme	Concepts raised		Considerations raised	Example quote
		Anticipated(P1, T3)	Actual(P2, T4)
Educational Utility	Knowledge retention	Reinforcement of learnings from lecture-based BEC course and easy reference (given phone is always nearby) without need to carry full BEC manual	App was found to have been a helpful resource in retaining information from BEC course		‘I like it because I normally don’t like reading from a hardcopy first of all, it also helps me with summaries of conditions such as allergic reactions, what to do when it happens, if you encounter a trauma patient…what you are supposed to do.’
	Access to content	Access to concise summaries of key topics of BEC content	Pictures for skills were found to be helpful, but summaries were sometimes found to be too high-level and topics of interest missing	Addition of videos and direct links to society guidelines; expansion of summaries
	Knowledge spread		App was used for teaching purposes, to share information with others who did not attend the BEC course
Clinical Utility	Practical guidance	Real-time stepwise guidance on patient care, confirmation that what was done was correct and that no life-saving steps were missed	Stepwise guidance was found to be helpful, but there was uncertainty about when exactly to use—that is, at patient arrival or after initial intervention (to confirm)		‘When you get any patient with different cases and you apply ABCDE as per the standard, I am sure you will manage and you will not be needed to wait for the Doctor in case you are a nurse in the ward. When the Doctor comes, he/she will find a patient doing well and he finishes other remaining things, but for us, we are concentrating in saving lives of the patients who are in life threatening situation.’
	Scope expansion	Ability for nurses to provide faster, more timely care independently, without need to wait for a doctor
	Setting	Concern that app would be less helpful for referral hospitals since it gives guidance to refer rather than definitive management	App was felt to be less helpful for referral hospitals since final guidance indicates to refer and does not get into definitive management	Different algorithms for referring (stabilisation/ transfer) vs receiving (definitive care) hospitals
User Experience	Application interface	User-friendly, clear, easy to navigate interface (both between and within chat-bot and resource pages)	App was felt to be user-friendly, clear, and easy to navigate		‘There are couple of things that personally I have seen in the application, number one is it’s easy to use, number two it’s like a reference notes or a summary. But also, they have those highlighted words that are useful, clicking on them transferred you to a page where you can learn these skills like how to give oxygen and the like.’
	Chat-bot algorithm	Like the idea of chat-bot, but algorithm not allowing bypassing of steps already performed may result in delays in administration of life-saving interventions
	Search function	Concern that search function limited by need to input specific words or phrases would make finding specific references difficult	Search function was felt to be limited by need to input specific words or phrases	Iteration of BEC app search function to include related results
Barriers to Access	Device-related	Numerous reasons individuals were unable to access to app initially (ie, lack of smartphone, outdated smartphone, insufficient storage on smartphone)	Inability to access app was reiterated to be disappointing	Central phone with app in the clinic for shared use	‘It is not friendly to people whose device cannot accept the app. That is a very big challenge. We are planning to take this to peripheral areas where smart phones are unlikely to be present. The app should be improved so that it can be accessed regardless of the type of device.’
	Data-related	Those who were able to download app appreciated ability to use is offline given limited access to and high-cost of mobile data	Ability do use app offline continued to be identified as an important feature
	Need to re-download	Concern about losing access to app (eg, if get new phone) and latest content (eg, if there are app updates)	Obtainment of new phone resulted in loss of access to the app for several participants
Barriers to Use	Perception	Concern about what it will look like to take out phone in front of patients and families	Concern about taking phone out in front of patients and families limited its use		‘(The app] should be in different languages, it is in a foreign language, English. It should be improved into Swahili so that it can be used by people in areas involved.’
	Logistics	Concern about how to use if hands are gloved and that will be pulled in too many directions during an emergency to be able to stop and use the app
	Language	Only available in English, not Swahili	Lack of availability of app in Swahili limited its use	Translate app content into languages outside of English

BEC, basic emergency care.

Educational utility

Described ways in which participants expected or found the BEC app to support BEC knowledge development as an extension of the BEC course. Three subthemes emerged: knowledge retention, access to content and knowledge spread. At T3, participants anticipated that the BEC app would reinforce learnings from the BEC course and serve as an effective reference. This sentiment was reiterated at T4, whereby participants expressed that they indeed found the app to be a helpful resource in retaining information from the course. Participants at T3 identified access to content via topic summaries to be especially helpful in reinforcing BEC course content. This sentiment was similarly reiterated at T4, with opportunity for further build-out of this component of the app identified. Finally, several participants at T4 noted that they used the app to share information with colleagues who did not attend the BEC course.

Clinical utility

Described ways in which participants expected or found the BEC app to support BEC delivery in their place of work. Three subthemes emerged: practical guidance, scope expansion and setting applicability. At T3, it was widely expected that the BEC app would be a helpful resource for real-time stepwise guidance on patient care (via the chat-bot feature) as well as confirmation that the correct interventions were given and no life-saving steps were missed. This utility was reiterated at T4. However, some participants expressed uncertainty about when the tool should be used. Nurses in particular, at T3, raised the notion that the BEC app would allow for them to provide faster, more timely care independently, without need to wait for a doctor. This was not raised again at T4. Finally, a concern about setting applicability was raised at T3 and reiterated at T4. Workers in referral centres suggested that the app is not as useful to them as it could be because the algorithm ends with guidance to transfer the patient. It was suggested to build two different algorithms into the BEC app: one for referring facilities (with ultimate goals of stabilisation and transfer) and another for receiving facilities (with ultimate goal of definitive care).

User experience described ways in which participants experienced the application itself. Three subthemes emerged: application interface, chat-bot algorithm and search function. Both at T3 and T4, the app was described as user-friendly, clear and easy to navigate. At T3, participants stated that they liked the concept of a chat-bot, but expressed concern that inability to bypass steps already performed could lead to delays in life-saving care. This was not raised again at T4. Finally, initially raised as a concern at T3 and then reinforced at T4, participants felt that the search function was not optimised.

Barriers to access described challenges identified in obtaining and retaining the mobile application. Three subthemes emerged: device-related, data-related and need to redownload. Individuals at T3 and T4 who were not able to access the mobile application (eg, due to lack of smartphone, outdated smartphone, insufficient storage on smartphone) expressed disappointment for missing out on this resource. Those who were able to download app at T3 appreciated ability to use is offline given limited access to and high-cost of mobile data. This was reiterated as an important feature at T4. Finally, some individuals who were able to download the app at T3 raised concern about losing access to the app due to phone turnover and not being able to have the latest content if there are app updates. Loss of app due to phone turnover was identified as an issue in T4.

Barriers to use described challenges identified in using the mobile application. Three subthemes emerged: perception, logistics and language. At T3 participants expressed concern about what message it would send patients and their families if they were to take out their phone during patient care. This concern was reiterated at T4 and sited as a reason that use of the BEC app as a bedside reference was limited. Logistical concerns such as how could the app be used with gloves or how could app be used in a very busy setting were cited as concerns at T3, but not raised again at T4. Finally, contents of the app not being in the participants’ native language (Swahili) were cited as a concern in T3 and a limiting factor to app adoption in T4.

Discussion

In this study, we examined the impact of a mobile application adjunct to the BEC course on knowledge acquisition and retention, workplace experience and user perceptions, acceptability and barriers to adoption of this tool. Together, these data provide important insight into the use of a mobile application adjunct as a training strategy for improving healthcare worker knowledge and self-efficacy as an ultimate means to improve emergency care access and delivery in low-resource and remote settings throughout the world.

As exhibited by significant increases in knowledge assessment scores from T1 to T2, and T2 to T3 in both arms, our findings support prior studies that have shown the BEC course to improve participants’ knowledge of emergency care.12 Our findings similarly support prior studies that have shown the BEC course to be associated with significantly increased confidence in managing emergency conditions.12 However, while the previous feasibility study suggested that online open access educational resources (ie, suite of clinical cases and first generation mobile phone application) may lead to greater knowledge acquisition and retention compared with the BEC course alone,16 our study did not find a significant difference in knowledge assessment scores between the app and non-app groups at any time point. It is unclear if this was due to lack of efficacy of the app or insufficient use of the app across study participants, for the reasons listed above. Also, of note, though the notion of test-enhanced learning may have played a role in the increases in knowledge at each time point for participants in the two arms,18 we can be confident that the effect of the BEC app would have been isolated since the same course content and knowledge assessments were delivered to both arms.

Furthermore, self-efficacy surveys in the present study revealed that participant comfort in emergency care settings and satisfaction with ability to do work well increased over the study period, but enjoyment of work and ability to impact patient lives did not differ. Though further exploration of these domains is needed, these findings suggest that the BEC course may impact select elements of the workplace experience. However, no differences in self-efficacy between the app and non-app groups were observed at any time point. Finally, FG responses suggested that the BEC app was user-friendly, clear and easy to navigate and a helpful resource in retaining information from BEC course. However, a number of challenges identified both by moderators during the implementation of the BEC app20 and by participants during the FGs discussions may have limited the impact of the BEC app, resulting in the negative findings of this study.

First, of the 46 participants in the app arm, only 29 (63%) were able to successfully download and use the app by the end of the BEC course with a subset of this group no longer having access to the BEC app at follow-up. Based on a prior analysis of implementation factors, a number of reasons, including accessibility (eg, no smartphone), technical (eg, outdated operating system, insufficient storage space on device) and participant related (eg, smartphone turnover) were identified by moderators20 and reiterated through the FGs. This limitation in reach of the application was likely a key driver that no group-based differences were seen in knowledge assessments and self-efficacy scores. In addition to maximising the reach through technologic (eg, limiting data needed) and implementation-based (eg, easing of download process) solutions, participants raised system-level solutions, such as having a shared departmental phone with BEC app installed accessible to all healthcare workers.

Second, language was a key barrier to maximising the app’s effectiveness. While there was consensus that a mobile application is a desirable resource, participants noted that having the app in English, rather than in their native language of Swahili, limited its usability. This was potentially another key driver of our findings. Future iterations of the BEC app could consider translation into languages commonly used in healthcare settings within LMICs.

Finally, several opportunities for improvement of the BEC app content were identified through the FG that may further explain results. For example, participants in referral hospitals expressed concern in the initial FG that the app was not as useful to them since the final step indicates to refer and does not provide guidance for definitive management. This concern at the onset may have deterred this subgroup from using the app altogether, thus limiting the effectiveness of the app as a course retention tool overall. Further updates to the application content could incorporate these and other enhancements to ensure that the app is as broadly applicable as possible and does not deter users.

With the increase in cell phone and internet technology users in LMICs,13 mobile health interventions are looked to as potential interventions to improve care in these areas.21 22 However, findings of the present study reinforce the importance of careful exploration of the context of these solutions. While our study did not find differences in knowledge retention or self-efficacy between the two arms, we were able to elucidate a number of factors that may have obscured our results and opportunities for iteration to further effectiveness of our intervention. Importantly, while participants in general expressed desire for a mobile app, there were several operational limitations that prohibited participants from accessing and using the app, despite significant mitigation efforts. Addressing these key issues will be critical before a final assessment is made on whether a mobile application is an appropriate intervention in this setting.

Limitations

Our findings should be considered in the context of several limitations. First, our study was limited to workers in healthcare settings in one country in Eastern Sub-Saharan Africa, potentially limiting the generalizability of our results. However, representation of multiple facility types (health centres, district facilities and referral hospitals) and different geographies (urban, rural) offsets this limitation to some degree.

Second, for the purposes of this study, extensive moderator support was provided to facilitate the BEC app download process. This level of resource may not be feasible for mass distribution of such an intervention and barriers to access may be even greater than reported here. However, this enabled a comprehensive understanding of implementation challenges,20 which can be used to further refine the intervention and its roll-out.

Third, we did not track at the individual level who had access to the BEC app for the entirety of the study (ie, those who were able to download initially and did not have phone turnover). Moreover, among individuals who were able to access the BEC app, we did not track app utilisation among those who were able to successfully download. Future studies could consider this level of tracking to allow for subanalyses to isolate the effect of utilising the BEC app on knowledge retention.

Finally, while the primary outcomes of the present study were knowledge retention, self-efficacy and perspectives of the BEC app, this study was in support of our bigger-picture objective to understand effectiveness of a mobile application as a real-time bedside reference. As this bigger-picture objective was not feasible as part of the present study, the primary outcomes of this study were used as proxies. Nonetheless, our study elucidated critical considerations in the implementation of a mobile application intervention. Future studies should aim to directly assess utility of a mobile application at the point-of-care.

Conclusions

The BEC app was generally well received as a reference tool and guide for clinical are. However, no differences between knowledge retention and self-efficacy were observed between the app and non-app arms. Technologic-based, linguistic-based and content-based barriers likely limited the impact of the app. Ongoing perspectives from end users will be critical as the app continues to be refined and further rolled-out.