Knowledge acquisition and retention following Saving Children's Lives course for healthcare providers in Botswana: a longitudinal cohort study.

OBJECTIVES: Millions of children die every year from serious childhood illnesses. Most deaths are avertable with access to quality care. Saving Children's Lives (SCL) includes an abbreviated high-intensity training (SCL-aHIT) for providers who treat serious childhood illnesses. The objective of this study was to examine the impact of SCL-aHIT on knowledge acquisition and retention of providers.
SETTING: 76 participating centres who provide primary and secondary care in Kweneng District, Botswana. PARTICIPANTS: Doctors and nurses expected by the District Health Management Team to provide initial care to seriously ill children, completed SCL-aHIT between January 2014 and December 2016, submitted demographic data, course characteristics and at least one knowledge assessment.
METHODS: Retrospective, cohort study. Planned and actual primary outcome was adjusted acquisition (change in total knowledge score immediately after training) and retention (change in score at 1, 3 and 6 months), secondary outcomes were pneumonia and dehydration subscores. Descriptive statistics and linear mixed models with random intercept and slope were conducted. Relevant institutional review boards approved this study.
RESULTS: 211 providers had data for analysis. Cohort was 91% nurses, 61% clinic/health postbased and 45% pretrained in Integrated Management of Childhood Illness (IMCI). A strong effect of SCL-aHIT was seen with knowledge acquisition (+24.56±1.94, p<0.0001), and loss of retention was observed (-1.60±0.67/month, p=0.018). IMCI training demonstrated no significant effect on acquisition (+3.58±2.84, p=0.211 or retention (+0.20±0.91/month, p=0.824) of knowledge. On average, nurses scored lower than physicians (-19.39±3.30, p<0.0001). Lost to follow-up had a significant impact on knowledge retention (-3.03±0.88/month, p=0.0007).
CONCLUSIONS: aHIT for care of the seriously ill child significantly increased provider knowledge and loss of knowledge occurred over time. IMCI training did not significantly impact overall knowledge acquisition nor retention, while professional status impacted overall score and lost to follow-up impacted retention. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first longitudinal cohort study to describe the impact of an abbreviated high-intensity training for serious childhood illness on doctors and nurses working in community clinics and district hospitals in a middle-income country.

Data collected allowed the examination of demographic and training factors that impact training with knowledge acquisition and retention.

There was significant lost to follow-up during the study period, and those doctors and nurses had significantly increased loss of knowledge over time.

Outcomes are limited to provider knowledge, not actual or reported performance.

Introduction

Each year, severe pneumonia, shock from diarrhoeal dehydration and sepsis are responsible for 25% of 5.1 million child deaths that occur worldwide.1 2 Over 1 million children die each year due to lack of effective, low-cost interventions being available and used appropriately.3 Access to quality healthcare is a global challenge, and timely and effective treatment for pneumonia and diarrhoea are essential components.4–6

A child mortality audit in Botswana between 2011 and 2013 demonstrated that 46% of paediatric in-hospital deaths were due to severe pneumonia, diarrhoeal dehydration and sepsis.7 Thirty-three per cent of in-hospital paediatric deaths occurred within the first 24 hours, an indication that children arrived critically ill. Twenty-six per cent of all in-hospital deaths were considered avoidable, with an average of 2.6 modifiable factors contributing to each death.7 Delayed or inadequate recognition and treatment of serious illness were major modifiable factors, and over 50% of factors were attributed to provider performance.

Healthcare providers in Botswana are trained to care for ill children using the Integrated Management of Childhood Illness (IMCI). IMCI is a training programme endorsed by WHO to train healthcare providers to care for children in low-income and middle-income countries (LMICs). However, studies have demonstrated that after health providers receive IMCI training, one-third to one half of seriously ill children are not identified and do not receive correct treatment for potentially life-threatening conditions.8–10

Saving Children’s Lives (SCL) programme is a collaboration between the Botswana Ministry of Health, the University of Botswana, Botswana University of Pennsylvania Partnership, Children’s Hospital of Philadelphia Center for Global Pediatric Critical Care, and the American Heart Association to improve the quality of care for seriously ill or injured children. SCL abbreviated high-intensity training (SCL-aHIT) is a 2-day training focused on the knowledge and skills a healthcare provider needs to optimally recognise and initiate stabilising treatment in the community clinic, primary or district hospital setting.

We hypothesised that SCL-aHIT would lead to significant knowledge acquisition and retention by healthcare providers. We also hypothesised that IMCI training would not have significant impact on knowledge acquisition or retention. Further, we hypothesised that provider, training or work environment characteristics may impact knowledge acquisition and retention. Finally, we hypothesised that SCL-aHIT may impact knowledge of pneumonia and diarrhoea scores may have differential acquisition and retention.

Methods

Study design

This retrospective cohort study was conducted to examine the impact of district-level SCL-aHIT on provider knowledge in Kweneng District, Botswana. All components of the SCL programme were active during the study period. Data were extracted from the SCL administrative database and included participant demographics and knowledge assessments. Our primary outcome was total score acquisition with secondary outcomes of total score retention, and pneumonia and diarrhoea subscores of both acquisition and retention.

Setting

Kweneng District, Botswana, has a population of 304 000, with 83% people living within 8 km of a health facility (100% within 15 km).11–13 There are 1 district hospital, 2 primary hospitals, 9 clinics with beds and 64 health posts and clinics without beds in the district. The estimated doctor/population ratio is 1:550 and nurse/population ratio of 1:80.

Cohort description

Cohort consisted of a convenience sample of physicians and nurses from community clinics, health posts, primary and district hospitals. Providers were identified for training by the Kweneng District Health Management Team (DHMT) based on if they were expected to provide initial stabilising care to seriously ill children in their position. All subjects who participated in SCL-aHIT, completed demographic data and at least one knowledge assessment were eligible for inclusion. To minimise selection bias, follow-up assessments were attempted all providers identified in Kweneng District completed training. The SCL implementation team attempted to follow up with participants of the training sessions in person or by phone. There were two attempts to complete the assessment at each time point for all course participants. More attempts were not possible due to programme limitations.

Saving Children’s Lives

The SCL programme employs five major implementation strategies:

Implementation strategy 1: aHIT in-service training

SCL-aHIT is a contextualised version of the American Heart Association’s Pediatric Emergency Assessment Recognition and Stabilization programme. The contextualisation process and initial training programme has been described previously.14 It is a combination of didactics, skills stations and simulated patient scenarios. To increase peer to peer learning and instructor situational awareness, didactics and final examination employed audience response software. While Rowe et al defined high-intensity training as having a duration >5 days, which included interactive sessions (eg, role play),15 we defined aHIT as having interactive sessions but with a training duration <5 days.

Implementation strategy 2: serial facility readiness assessments

The implementation team conducted bimonthly facility readiness assessments (FRA). The FRA is a focused inventory of personnel, equipment and supplies relevant to treatment of seriously ill or injured children who present to clinics, health posts or wards (hospitals). Estimated time to completion was 1 hour, and findings were immediately reported to the on-duty leadership, active issues reviewed and solutions identified.

Implementation strategy 3: longitudinal provider knowledge assessment

The knowledge assessment is a 6-item multiple-choice questionnaire was administered pre-SCL-aHIT training, immediately following, and at 1, 3 and 6 months. The assessment targeted to basic content regarding recognition and treatment of severe dehydration and moderate-severe pneumonia. Question types include ‘select all that apply’ and single best answer. Correct volume and rate of fluid administration for severe dehydration were consistent with current WHO and PALS guidelines. Choice of antibiotics for pneumonia was dependent on reported location of work and aligned with national guidelines.

Implementation strategy 4: bidirectional active feedback between front-line providers and health system leadership

The implementation team actively reported to health system leadership biannually and received feedback on programme implementation strategies (training, facility readiness and provider knowledge). Reports were conducted in person with DHMT leadership (district hospital superintendent, matron and chief medical officer) as well as Ministry of Health (Deputy Permanent Secretary of Clinical Services, Public Health, Permanent Secretary to the Minister of Health). These results, as well as the feedback from health system leadership, were reported back to the instructor group biannually at instructor ‘bootcamps’.

Implementation strategy 5: development and maintenance of a clinically relevant instructor core

Instructor candidates were identified by instructors based on course performance, interest in subject matter and interpersonal skills. After receiving approval from the DHMT, instructor candidates underwent a 2-day instructor training focused on adult learning strategy, simulated patient scenario facilitation, and roles and responsibilities of being an active instructor. Then, they were monitored with structured feedback by senior SCL faculty for a minimum of two courses.

Outcomes

Data extracted from the SCL administrative database for analysis include self-reported provider demographics and serial knowledge assessment responses. Total and subscores were treated as continuous variables (potential range from 0 to 100). Provider demographics included: professional status, work location, type of personal mobile phone (smart vs other), language most commonly spoken, IMCI subtypes (time since training, training duration—short vs long), other previous resuscitation training, perception of resuscitation and course multiple choice question score. A smartphone was defined as a mobile phone that had applications, access to internet and email. As SCL-aHIT employed audience response software as an education tool, the SCL programme tracked smartphone ownership as a surrogate for comfort with technology to monitor possible impact on knowledge acquisition. The SCL programme was initiated using highly experienced paediatric resuscitation education experts and transitioned to newly trained local instructors during the study period. To control for confounding due top variation in instructor experience, training characteristics included year of training and instructor mix. We defined the instructor mix of the initial training to be of four types: international faculty only, <70% local faculty (LT70LF), >70% local faculty (GT70LF) and local faculty only. We also defined training by the year initial SCL training was conducted: January 1 to December 31 2014 (IF led, full programme support), 2015 (LF led, high degree of IF supervision, full programme support) and 2016 (LF led, minimal IF supervision, minimal programme support).

Statistical approach

The statistical analysis was performed using SAS software, V.9.4. We conducted exploratory analyses to test for potential confounding between IMCI training and knowledge acquisition and retention. Means and SD were presented for continuous variables, while frequency and per cent were presented for discrete variables. Differences in participant or course characteristics between IMCI and non-IMCI groups were tested with X2 test for discrete variables. Differences in immediate post-training assessment score among participant or course characteristics were tested with independent-samples t-tests (professional status, English spoken most commonly, perceived frequency of resuscitation >1 month, I am comfortable with the initial steps of stabilising a paediatric patient with severe pneumonia/severe dehydration, year of the programme, previous resuscitation training, smartphone usage,) or one-way analysis of variance (location of work, resuscitation success (perceived), instructor type), as appropriate.

To answer the study’s primary hypothesis, that participation in the SCL training would lead to significant increases in knowledge from baseline to post-course assessments and that the knowledge would be retained over the study period, we used a linear mixed model approach. Best-model fit was achieved with two linear segments, with baseline to immediate post-training as the first segment (knowledge acquisition), and immediate post-training to 6-month follow-up as the second segment (knowledge retention). Random intercepts were fit to allow for subject-specific baseline scores and random slopes were fit to the initial piece-wise segment to allow for subject-specific knowledge acquisition scores, as well as the second piece-wise segment to allow for subject-specific knowledge retention. The first model fit was the unconditional means model which includes only the random intercept. Model 2 was the unconditional growth model which included the fixed effects for each time segment, the random intercept and the random slope for time segment 1 and 2. We show the proportion of variance in knowledge change over time that is explained by the complete Knowledge Assessment at 1, 3 and 6 months after SCL-aHIT training (and subsequently by IMCI training and then the covariates) by examining the decrease in the within person residual variance from one model to the next. To answer hypothesis 2, model 3 adds IMCI training to the unconditional growth equation. The main effect of IMCI assessed the difference in baseline knowledge level between the IMCI versus non-IMCI group. An interaction effect between previous IMCI training and the piece-wise time effects was also added into the model to assess whether IMCI training enhanced or diminished knowledge acquisition and/or retention. Model 4 presents the confounder-adjusted model. Several covariates were included in this model a priori (year of training, location of work). To maintain a relatively parsimonious model yet still use a conservative cut-off for issues of confounding, we retained any variable that was significantly different between IMCI and non-IMCI participants or had significantly different course assessment scores in bivariate analysis (p<0.10). To evaluate for non-random lost of follow-up (non-response bias), we conducted a sensitivity analysis that involved creating a variable for those missing 6-month assessments and those with 6-month assessment. Since these analyses revealed patterned missingness (drop-outs had lower knowledge retention), we added this variable as a fixed factor to further control for lost-to-follow-up. We performed model diagnostics including testing for multivariate normality of residuals and testing for linearity of the trend in each time segment. Multicollinearity was assessed with Pearson correlation coefficient (r<0.8) among the potential confounders.

We used the Strengthening the Reporting of Observational Studies in Epidemiology cohort checklist when writing our report.16

Patient and public involvement statement

This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy.

Results

Description of cohort

Between January 2014 and December 2016, 211 providers had data available for analysis. Ninety-one per cent (187) were nurses, and 61% (127) were clinic/health postbased and 25% (52) hospital based (table 1). Ninety-eight per cent (206) of providers had a mobile phone and 53% (111) reported owning a smartphone. Twenty-four per cent reported English was the most commonly used language. Sixty-seven per cent self-reported that they resuscitated a seriously ill child at least once a month, and 30% and 20% of participants were not comfortable with the initial steps of stabilising a child with severe pneumonia or diarrhoea, respectively. Forty-one per cent (84) of providers perceived resuscitation to be successful in less than 25% of cases where they work. Only 45% (95) reported previous IMCI training. Of providers with previous IMCI training, 74% (70) reported that the duration of IMCI training was less than 7 days (table 2). Thirty-eight (40%) received IMCI training >5 years ago and 32 (34%) <2 years ago. Paediatric, neonatal or trauma resuscitation training was less than 12%, while 35% (73) had received cardiopulmonary resuscitation training. Seventy-eight per cent (162) of participants received training in 2014, while 29% (60) were taught by an instructor group with 70% local instructors or only local instructors.

Table 1

Provider characteristics

N	Overall	IMCI trained	No IMCI	P value
	211	95	116
Professional status*		n (%)	n (%)
Nurse	187 (90.8)	91 (48.7)	96 (51.3)	0.0061
Physician	19 (9.2)	3 (15.8)	16 (84.2)
Location of work‡
Clinic or health post	127 (61.10)	62 (48.8)	65 (51.2)	0.2186
Hospital	52 (25.0)	23 (44.2)	29 (55.8)
Other	29 (13.9)	9 (31.0)	20 (69.0)
Mobile phone†
Smart	111 (53.1)	57 (51.4)	54 (48.6)	0.0684
Text and Voice only or no cell phone	98 (46.9)	38 (38.8)	60 (61.2)
English spoken most commonly
Yes	51 (24.2)	15 (29.4)	36 (70.6)	0.0101
No	160 (75.8)	80 (50.0)	80 (50.0)
Perceived frequency of resuscitation >1 month§
Yes	138 (66.7)	65 (47.1)	73 (52.9)	0.4896
No	69 (33.3)	29 (42.0)	40 (58.0)
I am comfortable with the initial steps of stabilising a paediatric patient with severe pneumonia
Agree	147 (69.7)	78 (53.1)	69 (46.9)	0.0004
Disagree/Neutral	64 (30.3)	17 (26.6)	47 (73.4)
I am comfortable with the initial steps of stabilising a paediatric patient with severe dehydration
Agree	168 (79.6)	80 (47.6)	88 (52.4)	0.1342
Disagree/neutral	43 (20.4)	15 (34.9)	28 (65.1)
Resuscitation success (perceived)*
0%–25%	84 (40.8)	42 (50.0)	42 (50)	0.2832
26%–50%	32 (15.5)	13 (40.6)	19 (59.4)
51%–75%	37 (18.0)	12 (32.4)	25 (67.6)
76%–100%	53 (25.7)	26 (49.1)	27 (50.9)
Previous resuscitation training
Paediatric§	23 (11.1)	12 (52.2)	11 (47.8)	0.4
Neonatal*	21 (10.2)	8 (38.1)	13 (61.9)	0.5538
Trauma*	21 (10.2)	7 (33.3)	14 (66.7)	0.2911
CPR‡	73 (35.1)	29 (39.7)	44 (60.3)	0.3361
Year of the programme‡
2014	162 (77.9)	77 (47.5)	85 (52.5)	0.3128
2015 or 2016	46 (22.1)	18 (39.1)	28 (60.9)
Instructor type‡
IFO	52 (25.0)	28 (53.9)	24 (46.1)	0.3872
LT70LF	96 (46.2)	44 (45.8)	52 (54.2)
GT70LF	39 (18.8)	16 (41.0)	23 (59.0)
LFO	21 (10.0)	7 (33.3)	14 (66.7)

*5participants did not report profession, resuscitation success (perceived), previous neonatal resuscitation or trauma training

†2 did not report cellphone access.

‡3did not report location of work, previous CPR training, year of training or instructor type.

§4did not report previous paediatric resuscitation training or perceived frequency of resuscitation.

Previous Paediatric Resuscitation=Paediatric Advanced Life Support, Emergency Triage Assessment and Treatment.

Previous Neonatal Resuscitation=Neonatal Resuscitation Training, Helping Babies Breathe. Previous CPR:

Other includes hospital based (administrative/‘other’).

CPR, cardiopulmonary resuscitation; IFO, international faculty only; IMCI, Integrated Management of Childhood Illness; LFO, local faculty only.

Table 2

Characteristics of provider previous Integrated Management of Childhood Illness (IMCI) training

	n=95
Time since training	% (N)
<6 months	14 (13)
>6 months to 2years	20 (19)
2–5 years	26 (25)
>5 years	40 (38)
IMCI course duration	% (N)
<7 days	74 (70)
≥7 days	26 (25)

Sensitivity analysis

To determine whether there were biases due to lost to follow-up, we created two groups: one group that had 6-month follow-up score and one group that did not have 6-month follow-up. We compared the acquisition of knowledge trajectory and retention of knowledge trajectory to ensure they were similar. Analysis showed differences in knowledge retention between the groups: the group with 6-month follow-up did not have a significantly better knowledge acquisition (+1.26, SE=3.69, p=0.7329), but demonstrated significantly better retention (+3.03/month, SE=0.88, p=0.0007). To control for this bias, we entered this variable into the confounder-adjusted piecewise regression models described below (model 4, tables 3–5). See online supplementary table and figures for results of the sensitivity analysis.

Description of model

The assumption of multivariate normality was adequately met. Linearity was satisfied within each time segment. No multicollinearity issue was found. Covariates included in the final model included: year of initial training, professional status, smartphone usage, language spoken most commonly, degree of comfort with treatment of severe pneumonia, location of work, perceived frequency of resuscitation and presence/absence of 6-month follow-up.

A strong and significant main effect was seen for knowledge acquisition due to SCL-aHIT (time: pre to post) (b=+24.56±1.94, p<0.0001), and loss of knowledge over time (b=−1.60±0.67/month, p=0.018). The proportion of variance in total scores knowledge change over time explained by the SCL education was 56.17% (R2, table 3). For dehydration subscores, a strong and significant main effect was seen for both knowledge acquisition (b=+14.58±1.29, p<0.001) and loss of knowledge over time (b=−1.10±0.39/month, p=0.0055). The proportion of variance in dehydration subscores knowledge change over time explained by the SCL education was 51.90% (R2, table 4). For pneumonia subscores, a strong and significant main effect was also seen for knowledge acquisition (b=+9.83±1.48, p<0.001), and no significant change in knowledge over time (b=−0.34 ±0.42/month, p=0.4229). The proportion of variance in pneumonia subscores knowledge change over time explained by the SCL education was 47.73% (table 5).

Table 3

Models for total score acquisition and retention in relation to previous IMCI training

Fixed effects	Model 1 (n=679)		Model 2 (n=679)		Model 3 (n=679)		Model 4 (n=655)
	γ (SE)	P value	γ (SE)	P value	γ (SE)	P value	γ (SE)	P value
	Unconditional model		Unconditional growth		IMCI training added		Confounder adjusted
Initial knowledge status
Intercept	60.20 (0.90)	<0.0001	43.08 (1.32)	<0.0001	44.16 (1.78)	<0.0001	45.70 (3.82)	<0.0001
Previous IMCI training (yes vs no)					−2.41 (2.65)	0.3641	−0.52 (2.45)	0.8336
Location of work (clinic or health post vs hospital)							−2.72 (1.96)	0.1677
Location of work (other vs hospital)							−7.04 (2.77)	0.0125
Profession status (physician vs nurse)							19.39 (3.30)	<0.0001
Perceived frequency of resuscitation >1 month (yes vs no)							0.91 (1.74)	0.6022
English spoken most commonly (yes vs no)							1.93 (2.28)	0.3989
Comfortable with treatment of severe pneumonia (agree vs disagree/neutral)							0.02 (1.77)	0.9903
Smartphone usage (yes vs no)							0.72 (1.57)	0.6453
Year of programme (2014 vs 2015 or 2016)							−4.04 (2.33)	0.0852
Had 6-month assessment (yes vs no)							1.91 (1.89)	0.313
Rate of change (slope for timepiece one)
Knowledge acquisition			26.37 (1.32)	<0.0001	24.61 (1.88)	<0.0001	24.56 (1.94)	<0.0001
Previous IMCI training (yes vs no)					3.89 (2.79)	0.1655	3.58 (2.84)	0.2113
Rate of change (slope for timepiece two)
Knowledge retention per month			−1.49 (0.46)	0.0014	−1.59 (0.66)	0.0172	−1.60 (0.67)	0.018
Previous IMCI training (yes vs no)					0.17 (0.92)	0.8574	0.20 (0.91)	0.824
Variance components
Level 1	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value
Within person	399.07 (25.19)	<0.0001	174.92 (18.14)	<0.0001	174.79 (18.15)	<0.0001	172.68 (17.89)	<0.0001
Level 2	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value
Intercept	41.59 (16.39)	0.0056	180.80 (39.60)	<0.0001	181.40 (39.72)	<0.0001	99.19 (34.29)	0.0019
Slope for knowledge acquisition			93.17 (48.81)	0.0281	91.31 (48.69)	0.0304	96.19 (48.85)	0.0245
Slope for knowledge retention			5.68 (3.36)	0.0452	5.95 (3.42)	0.041	5.43 (3.24)	0.0469
Goodness-of-fit statistics
−2 log likelihood	6051.2		5736.6		5725.9		5428.3
Akaike’s information criterion	6055.2		5750.6		5739.9		5442.3

Intercept for model 1 is the grand mean in knowledge status across all time points; Intercept in model 3 is the baseline knowledge for the group with no IMCI training. The intercept in model 4 represents the baseline knowledge for the person who had no IMCI training, had SCL initial training in 2015/2016, were nurses, did not use smartphones, Setswana spoken most commonly, were discomfort with treatment of severe pneumonia, work in hospital, perception of frequency of resuscitation ≤1 month, and did not complete a 6-month assessment.

IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

Table 4

Models for dehydration subscore acquisition and retention in relation to previous IMCI training

Fixed effects	Model 1 (n=679)		Model 2 (n=679)		Model 3 (n=679)		Model 4 (n=655)
	γ (SE)	P value	γ (SE)	P vlaue	γ (SE)	P value	γ (SE)	P value
	Unconditional model		Unconditional growth		IMCI training added		Confounder adjusted
Initial knowledge status
Intercept	26.37 (0.50)	<0.0001	16.82 (0.77)	<0.0001	16.77 (1.03)	<0.0001	17.16 (2.24)	<0.0001
Previous IMCI training (yes vs no)					0.13 (1.54)	0.9304	1.06 (1.57)	0.5026
Location of work (clinic or health post vs hospital)							−2.24 (1.12)	0.0481
Location of work (other vs hospital)							−1.65 (1.59)	0.3025
Profession status (physician vs nurse)							7.21 (1.89)	0.0002
Perceived frequency of resuscitation >1 month (yes vs no)							0.76 (1.00)	0.4473
English spoken most commonly (yes vs no)							0.71 (1.31)	0.5901
Comfortable with treatment of severe pneumonia (agree vs disagree/neutral)							−0.97 (1.01)	0.3396
Smartphone usage (yes vs no)							0.27 (0.90)	0.7676
Year of programme (2014 vs 2015 or 2016)							−0.19 (1.34)	0.8905
Had 6-month assessment (yes vs no)							1.04 (1.08)	0.3361
Rate of change (slope for timepiece one)
Knowledge acquisition			14.74 (0.91)	<0.0001	14.73 (1.23)	<0.0001	14.58 (1.29)	<0.0001
Previous IMCI training (yes vs no)					0.04 (1.83)	0.9843	0.12 (1.90)	0.9513
Rate of change (slope for timepiece two)
Knowledge retention per month			−0.92 (0.26)	0.0005	−1.16 (0.38)	0.0023	−1.10 (0.39)	0.0055
Previous IMCI training (yes vs no)					0.46 (0.52)	0.3795	0.39 (0.54)	0.4681
Variance components
Level 1	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value
Within person	137.48 (8.57)	<0.0001	66.13 (6.78)	<0.0001	66.02 (6.78)	<0.0001	65.42 (6.75)	<0.0001
Level 2	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value
Intercept	9.55 (4.98)	0.0275	53.95 (13.70)	<0.0001	54.64 (13.77)	<0.0001	42.28 (13.51)	0.0002
Slope for knowledge acquisition			55.47 (20.16)	0.003	56.40 (20.26)	0.0304	63.18 (20.97)	0.0013
Slope for knowledge retention			1.23 (1.10)	0.1322	1.28 (1.11)	0.1256	1.34 (1.12)	0.1159
Goodness-of-fit statistics
−2 log likelihood	5310.7		5032.1		5025.9		4798.2
Akaike’s information criterion	5314.7		5046.1		5039.9		4812.2

Intercept for model 1 is the grand mean in knowledge status across all time points; Intercept in model 3 is the baseline knowledge for the group with no IMCI training. The intercept in model 4 represents the baseline knowledge for the person who had no IMCI training, had SCL initial training in 2015/2016, were nurses, did not use smartphones, Setswana spoken most commonly, were discomfort with treatment of severe pneumonia, work in hospital, perception of frequency of resuscitation ≤1 month, and did not complete a 6-month assessment.

IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

Table 5

Models for pneumonia subscore acquisition and retention in relation to previous IMCI training

	Model 1 (n=679)		Model 2 (n=679)		Model 3 (n=679)		Model 4 (n=655)
	γ (SE)	P value	γ (SE)	P value	γ (SE)	P value	γ (SE)	P value
Fixed effects	Unconditional model		Unconditional growth		IMCI training added		Confounder adjusted
Initial knowledge status
Intercept	33.84 (0.60)	<0.0001	26.23 (1.00)	<0.0001	27.36 (1.34)	<0.0001	28.31 (2.77)	<0.0001
Previous IMCI training (yes vs no)					−2.52 (2.00)	0.2113	−1.74 (1.94)	0.3711
Location of work (clinic or health post vs hospital)							−0.78 (1.39)	0.575
Location of work (other vs hospital)							−6.09 (1.96)	0.0024
Profession status (physician vs nurse)							10.20 (2.30)	<0.0001
Perceived frequency of resuscitation >1 month (yes vs no)							−0.16 (1.23)	0.8943
English spoken most commonly (yes vs no)							2.26 (1.59)	0.1568
Comfortable with treatment of severe pneumonia (agree vs disagree/neutral)							0.72 (1.25)	0.5655
Smartphone usage (yes vs no)							0.49 (1.10)	0.6574
Year of programme (2014 vs 2015 or 2016)							−3.06 (1.66)	0.0676
Had 6-month assessment (yes vs no)							2.02 (1.31)	0.1287
Rate of change (slope for timepiece one)
Knowledge acquisition			11.56 (1.09)	<0.0001	9.76 (1.47)	<0.0001	9.83 (1.48)	<0.0001
Previous IMCI training (yes vs no)					3.96 (2.18)	0.0717	3.65 (2.17)	0.096
Rate of change (slope for timepiece two)
Knowledge retention per month			−0.42 (0.28)	0.1422	−0.21 (0.41)	0.6084	−0.34 (0.42)	0.4229
Previous IMCI training (yes vs no)					−0.45 (0.57)	0.4384	−0.39 (0.55)	0.4829
Variance components
Level 1	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value	σ2 (SE)	P value
Within person	151.45 (9.83)	<0.0001	79.17 (8.46)	<0.0001	79.08 (8.46)	<0.0001	78.49 (8.22)	<0.0001
Level 2	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value	τ (SE)	P value
Intercept	25.91 (7.80)	0.0004	124.93 (21.90)	<0.0001	124.51 (21.90)	<0.0001	94.07 (19.96)	<0.0001
Slope for knowledge acquisition			106.84 (27.75)	<0.0001	104.01 (27.56)	<0.0001	95.09 (26.71)	0.0002
Slope for knowledge retention			2.09 (1.43)	0.0713	2.17 (1.45)	0.0671	2.06 (1.33)	0.0617
Goodness-of-fit statistics
−2 log likelihood	5424.9		5266.9		5257.3		4986.7
Akaike’s information criterion	5428.9		5280.9		5271.3		5000.7

Intercept for model 1 is the grand mean in knowledge status across all time points; Intercept in model 3 is the baseline knowledge for the group with no IMCI training. The intercept in model 4 represents the baseline knowledge for the person who had no IMCI training, had SCL initial training in 2015/2016, were nurses, did not use smartphones, Setswana spoken most commonly, were discomfort with treatment of severe pneumonia, work in hospital, perception of frequency of resuscitation ≤1 month, and did not complete a 6-month assessment.

IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

To test the second hypothesis, IMCI training had no effect on knowledge at baseline (b=−0.52±2.45, p=0.834), knowledge acquisition (b=+3.58±2.84, p=0.211), knowledge retention (b=+0.20±0.91/month, p=0.824), for total scores. IMCI training explained 0.07% of additional variance in total score change. As for dehydration subscores, IMCI training had no effect on knowledge at baseline (b=+1.06±1.57, p=0.5026), knowledge acquisition (b=+0.12±1.90, p=0.9513) or knowledge retention (b=+0.39±0.54/month, p=0.4681). IMCI training explained 0.17% of additional variance in dehydration score change. For pneumonia subscores, IMCI training had no effect on knowledge at baseline (b=−1.74±1.94, p=0.3711). There was no difference in knowledge acquisition (b=3.65±2.17, p=0.096) or knowledge retention (b=−0.39 ±0.55/month, p=0.4829). IMCI training explained 0.11% of additional variance in pneumonia score.

Our final hypothesis was examined in the confounder-adjusted models (see model 4 in tables 3–5). On average, nurses scored significantly lower than physicians at all time points: (b=−19.39±3.30, p<0.0001) on total score, (b=−7.21±1.89, p=0.0002) on dehydration subscore and (b=−10.20±2.30, p<0.0001) on pneumonia subscore. Compared with those who worked in hospitals, participants who worked in clinics/health posts scored significantly worse on dehydration: (b=−2.24±1.12, p=0.0481). Perceived frequency of resuscitation, language, perceived comfort with treatment of pneumonia, smartphone usage, year of training and completeness of follow-up had no significant effect on total scores or the dehydration or pneumonia subscores.

Model-based mean scores for each assessment were calculated based on populations that represented the majority of the cohort: those who had SCL initial training in 2014, were nurses, used smartphones, spoke non-English most commonly, were comfortable treating of severe pneumonia, worked in clinic/health post, reported frequency of resuscitation >1/month, and did not complete a 6 month assessment were plotted (figures 1–3).

Model-based marginal total score by IMCI training over time (adjusted). Plots were painted based on populations who had SCL initial training in 2014, were nurses, use smartphones, Setswana spoken most commonly, comfort with treatment of severe pneumonia, work in clinic/health post, perception of frequency of resuscitation >1 month and did not complete a 6-month assessment. IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

Model-based dehydration subscore by IMCI training over time (adjusted). Plots were painted based on populations who had SCL initial training in 2014, were nurses, use smartphones, Setswana spoken most commonly, comfort with treatment of severe pneumonia, work in clinic/health post, perception of frequency of resuscitation >1 month and did not complete a 6-month assessment. IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

Model-based pneumonia subscore by IMCI training over time (adjusted). Plots were painted based on populations who had SCL initial training in 2014, were nurses, use smartphones, Setswana spoken most commonly, comfort with treatment of severe pneumonia, work in clinic/health post, perception of frequency of resuscitation >1 month and did not complete a 6-month assessment. IMCI, Integrated Management of Childhood Illness; SCL, Saving Children’s Lives.

Discussion

This study demonstrates for the first time that SCL-aHIT significantly increases provider knowledge acquisition in the recognition and treatment of serious childhood illness. This is the largest study to our knowledge to report knowledge retention outcomes of providers who care for seriously ill children outside of academic centres in the LMIC. While previous IMCI training did not decrease knowledge acquisition, professional status and completing follow-up assessments impacted scores significantly. There was significant lost to follow-up during the study period, and while the adjusted model demonstrated worse knowledge retention than those who completed 6-month follow-up, we are limited in our ability to draw strong conclusions regarding knowledge the true rate of loss of retention.

This increase in knowledge may be due to the characteristics of training, and our study is consistent with previous studies that demonstrate high-intensity training being the most effective single implementation strategy to improve healthcare worker performance.15 17 Rowe et al found that high-intensity training had the greatest median training effect (11, IQR 8–15) compared with low-intensity training only (8, IQR 2–22), supervision (8, IQR 3–17), group problem solving (8, IQR 6–21), regulation/governance (5, IQR −1–20) or job aids (−3, IQR −7to 7). This is a similar increase Tuyisenge et al found when examining paediatric resuscitation knowledge acquisition and retention of final year medical students after a high-intensity training in an academic referral hospital setting in Malawi.18 Further, the high impact of a (2-day) aHIT is notable as shortened (5–10 days) IMCI training has been associated with a 2–16 point loss of treatment effect over standard (11 days) training.9 While SCL-aHIT demonstrated a larger effect than the range in the systematic review, our outcomes were limited to knowledge assessment and the difference in magnitude needs to be interpreted with caution.

There was significant loss of knowledge after SCL’s aHIT. At the estimated rate found in this study, knowledge would return to baseline in under 2 years for those who completed 6-month follow-up, while those who did not complete follow-up would return to baseline in less than 8 months. While Tuyisenge et al demonstrated retention up to 9 months after training final year medical students in paediatric resuscitation,18 our study is consistent with other studies in resuscitation training that demonstrate rapid loss of knowledge or skills, often in as little as in 6–12 weeks after training.19–23 This has also been seen with clinical management of malaria24 as well as with IMCI.10 While the lost to follow-up in this study only allows us to estimate a range of the rate of knowledge loss, is notable that the most rapid estimate is not as rapid as other studies. Several reasons may account for this. It may be due to the contextualisation process to ensure training was relevant to disease epidemiology and health system resources in Botswana. It may have been due to other components of SCL programme besides aHIT. The SCL programme integrates support in inventory of relevant medication and functioning equipment as part of its training and it provides immediate individual education on follow-up assessments by a master trainer. It is possible that similar results could have been obtained just from inventory support. The SCL programme uses active reporting of training results to local health leadership—this may have stimulated additional feedback and support through administrative communication independent of the SCL programme. Finally, it may be due to regression to the mean, as baseline knowledge scores were low and thus could only improve.

In this study, previous IMCI training did not significantly impact provider knowledge gained or retained from SCL-aHIT. That overall knowledge gained from SCL-aHIT was not negatively impacted supports the theory that programmes such as SCL that focus on serious childhood illness may be an added value and not redundant to IMCI training. This may be especially important in environments where quality of pneumonia and diarrhoea care is poor despite IMCI implementation. While IMCI-trained workers are more likely to correctly classify illnesses, administer oral therapies, employ rational antibiotic use, vaccinate children and counsel families on adequate nutrition for moderate illness,8 25 IMCI has limited impact on care delivery of the seriously ill child.8–10 If there was significant overlap in content between SCL and IMCI, we might expect higher baseline scores and decreased acquisition. Alternatively, it may be that current existing IMCI training is not optimally effective.

Nurses, on average, scored significantly lower than physicians. This may be due to differences in preclinical education, in-service training or unmeasured provider and environment characteristics that are highly correlated with professional status. Nevertheless, as nurses are the major training target for SCL-aHIT, further modifications to course content, structure or follow-up training may be needed.

Access to quality treatment of pneumonia and diarrhoea are major contributors to avertable mortality worldwide.4–6 Studies of provider performance show that standard guidelines were only followed 30%–40% of the time, and often led to misallocation of resources.26 Further, studies have shown that children with complex serious illness often receive worse care than those with milder, straightforward presentations.27 28 This poor quality of services for treatable conditions is directly responsible for over 5 million deaths each year and contributes to decreased utilisation of services, which accounts for another 3.6 million deaths.6 A sustained and integrated improvement of provider knowledge and resource awareness is needed to address these gaps that currently limit systems to provide quality care.

Limitations

As with any study there were several limitations. Use of an administrative database and infrastructure for the SCL programme may have contributed to non-random lost to follow-up. Although effect was minimised through the conducted sensitivity analyses (see online supplementary appendix 1), the study should be repeated with stronger support for follow-up data collection as well as training. Our outcome data were limited to knowledge assessments, and future studies that examine operational performance or patient outcomes are needed. The knowledge assessments have not been previously validated, and future studies should have multiple versions to better discriminate retention of test knowledge versus content knowledge. Finally, use of two time piece model assumes linearity throughout the follow-up period, and the true slope of knowledge retention may be non-linear.

Conclusion

aHIT focused on the seriously ill child significantly increases provider knowledge for both clinic and hospital-based providers. There appears to be significant loss of knowledge after initial training. IMCI training did not significantly impact overall knowledge acquisition or retention, but professional status impacted overall scores and lost to follow-up impacted retention of knowledge. In health systems where access to quality care for the seriously ill child is poor, programmes such as SCL may have a significant impact if knowledge retention can be addressed.