Effectiveness of an electronic clinical decision support system in improving the management of childhood illness in primary care in rural Nigeria: an observational study.

OBJECTIVES: To evaluate the impact of ALgorithm for the MANAgement of CHildhood illness ('ALMANACH'), a digital clinical decision support system (CDSS) based on the Integrated Management of Childhood Illness, on health and quality of care outcomes for sick children attending primary healthcare (PHC) facilities.
DESIGN: Observational study, comparing outcomes of children attending facilities implementing ALMANACH with control facilities not yet implementing ALMANACH.
SETTING: PHC facilities in Adamawa State, North-Eastern Nigeria. PARTICIPANTS: Children 2-59 months presenting with an acute illness. Children attending for routine care or nutrition visits (eg, immunisation, growth monitoring), physical trauma or mental health problems were excluded.
INTERVENTIONS: The ALMANACH intervention package (CDSS implementation with training, mentorship and data feedback) was rolled out across Adamawa's PHC facilities by the Adamawa State Primary Health Care Development Agency, in partnership with the International Committee of the Red Cross and the Swiss Tropical and Public Health Institute. Tablets were donated, but no additional support or incentives were provided. Intervention and control facilities received supportive supervision based on the national supervision protocol. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was caregiver-reported recovery at day 7, collected over the phone. Secondary outcomes were antibiotic and antimalarial prescription, referral, and communication of diagnosis and follow-up advice, assessed at day 0 exit interview.
RESULTS: We recruited 1929 children, of which 1021 (53%) attended ALMANACH facilities, between March and September 2020. Caregiver-reported recovery was significantly higher among children attending ALMANACH facilities (adjusted OR=2·63, 95% CI 1·60 to 4·32). We observed higher parenteral and lower oral antimicrobial prescription rates (adjusted OR=2·42 (1·00 to 5·85) and adjusted OR=0·40 (0·22 to 0·73), respectively) in ALMANACH facilities as well as markedly higher rates for referral, communication of diagnosis, and follow-up advice.
CONCLUSION: Implementation of digital CDSS with training, mentorship and feedback in primary care can improve quality of care and recovery of sick children in resource-constrained settings, likely mediated by better guideline adherence. These findings support the use of CDSS for health systems strengthening to progress towards universal health coverage. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

To the best of our knowledge, this is the first study to evaluate the impact of Integrated Management of Childhood Illness-related digital Clinical Decision Support Systems on health outcomes when implemented at scale in a programmatic context in resource-constrained settings.

Large observational study, recruiting 1021 children from 45 intervention primary healthcare facilities and 908 children from 44 control facilities with high rates of follow-up completion at day 7 for primary outcome assessment.

Though we adjusted for important potential confounders within the analysis, the nature of the evaluation in the context of large-scale implementation meant that it was not possible to randomise facilities; therefore, contextual differences may have influenced our findings.

Despite the use of standardised tools and procedures, performance or detection bias could have occurred given that the intervention could not be blinded.

Introduction

Global health initiatives place high expectations on digital technology to improve quality of care (QoC) in low and middle-income countries (LMICs).1 2 One promising approach is the implementation of digital Clinical Decision Support Systems (CDSS) for healthcare providers (HCPs) in remote regions and resource-constrained settings.3 4 CDSS guides HCPs through clinical consultations with simple, structured, step-by-step decision logic, providing them with evidence-based diagnostic and treatment recommendations, displayed on a digital device (eg, tablet computers).

Good quality of care (safe, effective, timely, efficient, equitable and people-centred) is essential to achieve universal health coverage.5 To improve QoC for children under 5 years of age, WHO developed the Integrated Management of Childhood Illness (IMCI) guidelines,6 now a standard for primary healthcare (PHC) consultations in over 100 LMICs. It focuses on diagnosis, classification and treatment of conditions responsible for 70% of child mortality (malaria, pneumonia, diarrhoea, measles and malnutrition). Evidence suggests that IMCI can improve QoC and reduce under-five mortality,7–9 but its roll-out over the past two decades has not yielded the anticipated effect.10 The reasons are manifold,11–13 but non-adherence of HCPs to guidelines, possibly due to the difficulty of practical integration into the clinical workflow, plays a central role.14–16

While adherence to guidelines may be improved by providing them in a digital CDSS format,17–21 systematic reviews assessing the impact of such digital tools on health outcomes show heterogeneous results: some report improvements in certain QoC indicators3; others have shown little to no effect of CDSS use on morbidity and mortality.22 Several IMCI-based CDSS are now available on smartphones or tablets.23 Their efficacy has been demonstrated in clinical trial settings,24 25 but evidence on effectiveness at larger scale, particularly in programmatic settings (ie, under real-world conditions) is limited.26

In this study, we explore the impact of a digital IMCI-based CDSS, the ALgorithm for the MANAgement of CHildhood illness (ALMANACH), on clinical outcomes and QoC in the programmatic setting. ALMANACH was first developed and evaluated from 2010 to 2014 to address clinical management of febrile children (age 2–59 months) in Tanzania.27 Controlled trials have demonstrated acceptance among end-users28 and clinical efficacy.25 29

ALMANACH was then further adapted for the programmatic setting of PHC clinics in Afghanistan and Nigeria, respecting national IMCI protocols, latest evidence, local epidemiology and the daily work reality of the health facilities (figure 1).30 Evaluation of ALMANACH after one year of implementation showed good acceptance by HCPs, improved completeness of clinical assessment, better adherence to treatment recommendations and reduced antibiotic prescription rates.31

Figure 1

Clinical decision support algorithm flow in ALMANACH Nigeria. ALMANACH, ALgorithm for the MANAgement of CHildhood illness.

The objective of this study was to evaluate the hypothesis that ALMANACH implementation improves recovery of children (age 2–59 months) from acute illness and QoC outcomes when implemented at scale in routine practice at primary care facilities.

Methods

Study design and participants

We conducted an observational study within the programmatic context of ALMANACH in Adamawa State, a conflict-torn region in North-Eastern Nigeria. In 2016, the International Committee of the Red Cross (ICRC) in partnership with the Adamawa State Primary Health Care Development Agency and technical support from the Swiss Tropical and Public Health Institute initiated the stepwise roll out of ALMANACH to the State’s PHC facilities. One HCP per facility received a 3-day introduction training on how to use the tablet and a refresher on basic clinical concepts included in the CDSS. These HCPs were responsible for cascading their acquired knowledge to their colleagues. HCPs at all accessible ALMANACH facilities received a 1 day of start-up supervision, followed by supportive supervision and mentorship (checking correct use of the CDSS, technical trouble shooting, orientation for untrained HCPs) provided every 4–6 months by ICRC and Agency staff. Additionally, all PHC facilities received the routine monthly supportive supervision by the government health agency, based on a national supervision protocol. Tablets were donated through the project, but no additional support (eg, drug or consumables supply, or financial incentives) was provided. Further detailed description of the ALMANACH intervention is described elsewhere.31

The State’s PHC facilities are clustered into 21 Local Government Areas (LGAs). At the time of the study, six LGAs had not yet implemented ALMANACH of which two were excluded due to security constraints. Control facilities were, therefore, selected from the remaining four LGAs. Four LGAs were selected for the intervention group that had a similar epidemiological and sociodemographic profile, after excluding those with security issues or that were implementing another major child health intervention.

Children 2–59 months of age who presented to a study facility during the data collection period with an acute illness were eligible for inclusion. Children attending for routine care or nutrition visits only (eg, immunisation, growth monitoring), physical trauma or mental health problems were excluded as these consultations are not addressed by ALMANACH.

Patient and public involvement

Community engagement prior to and during the study was built on existing long-term relationships with community representatives from the LGAs and Ward Development Committees (WDCs). Representatives were consulted on the purpose and conduct of the study, with detailed consultation on the recruitment strategy, particularly in relation to informed consent, though patients were not specifically involved in the study design. Planned dissemination activities include sharing of the findings with LGA and WDC representatives, and patients and the communities through an information campaign at the health facilities.

Informed consent

Caregivers of eligible children were recruited following informed consent. Verbal informed consent was obtained if the caregiver was willing to participate but not willing to provide written/thumbprint consent. This approach was taken as local community leaders advised that signing documents is regarded with suspicion both due to illiteracy and the high proportion of internally displaced people with identity protection concerns. No incentives were provided for participation, neither to participants nor to HCPs.

Procedures

At each facility, trained non-clinical research assistants collected basic information about the facility amenities and services from the facility manager, and information on training and experience of HCPs consulting children under 5 years of age (after informed consent).

For both intervention and control group, research assistants obtained basic sociodemographic details, main symptoms and information about prior care sought from caregivers while awaiting consultation. Brief exit interviews were conducted to record medication (prescription or medication in-hand), diagnosis and management advice as understood by the caregiver.

In ALMANACH facilities, consultation data (diagnosis, measurements and investigations) were extracted from the CDSS database. In control facilities, consultation data were obtained from facility registries immediately after the consultation.

On day 7, research assistants conducted a standardised health outcome assessment by phone. If unsuccessful, further attempts were made on 3 consecutive days. If the caregiver was not reachable by phone, contact was attempted through a community representative. Children of caregivers not contactable by day 10 were considered lost to follow-up. For follow-up occurring after day 7, caregivers were asked to reflect the status of the child on day 7.

Data collection was conducted in ALMANACH and control facilities in parallel, one LGA at a time within each group. All eligible facilities in the first three LGAs in each group were included; in the last LGAs (Hong and Song, last due being the least accessible at the time of the study), the most accessible facilities out of those eligible were selected until the target sample size was reached. Data collection lasted on average three weeks per LGA, occurring from 3 March to 28 March and 17 July to 30 September 2020, with a forced interruption due to the SARS-CoV2 pandemic.

Outcomes

The primary outcome was day 7 caregiver-reported recovery, assessed at phone follow-up. Secondary outcomes were antibiotic and antimalarial prescription during the consultation, referral to hospital, and communication of diagnosis and follow-up advice to the caregiver, as assessed at exit interview.

Statistical analysis

The sample size to detect a difference in recovery from 60% in control to 70% in ALMANACH facilities was estimated with 85% power and 0.05 significance threshold, assuming a SD of 10 percentage points of cure rates between facilities. To allow flexibility according to the fluctuating security situation, we calculated sample size for a range of cluster numbers, estimating that we would require between 48 and 42 facilities per study arm.

We examined imbalances in patients’ and HCPs’ characteristics between ALMANACH and control facilities using appropriate statistical tests. For each endpoint, we calculated its frequency in ALMANACH and control facilities separately. The ratio of these frequencies (risk ratio, RR) was used as the measure of the association between being treated in an ALMANACH facility and the endpoint. We also calculated the corresponding ORs as well as the 95% CIs for both measures.

Mixed logistic regressions were used to estimate adjusted ORs for each endpoint. The prespecified set of adjustments for the analysis of the primary endpoint (recovery) consisted of child’s age, sex, the collection period (pre-Covid-19 and post-Covid-19-related restrictions), symptom duration, travel time to the facility, whether the child was accompanied by their mother, presence or absence of five main symptoms, the qualification of the HCP and three facility-level variables: distance from a paved road and from the referral hospital (less or more than 30 min) and the average monthly number of consultations. Due to the observed imbalance between the study arms, we adjusted for whether care was sought in the 2 weeks preceding the consultation. The same set of variables was used to adjust the association of ALMANACH with the prescription of parenteral and oral antimicrobial treatment. Owing to the small number of cases, the analyses of referral to hospital, antibiotic and antimalarial treatment were adjusted only for sex, age, collection period, HCP’s qualification and symptoms. Regressions included a random effect for the health worker nested in a random effect for the health facility.

Analyses of communication by the HCP to the child’s caregiver on diagnosis and follow-up were performed using negative binomial regression. The number of such consultations for every HCP was the dependent variable. The logarithm of the total number of consultations performed by the HCP during the study was included with its coefficient constrained to 1, thus the adjusted estimate can be interpreted as a rate ratio. All HCP and facility-level variables mentioned above were used to adjust.

To ensure that our estimates were not affected by loss to follow-up, we applied inverse probability weighting. The probability model to derive the weights was constructed using least absolute shrinkage and selection operator, 'LASSO'.32 All reweighted estimates were virtually identical (within 0.05) to those from unweighted regressions, and thus we report only the latter.

Finally, we conducted exploratory descriptive analysis of clinical records (from the ALMANACH database and from paper records in control facilities) to further assess care processes. The decision logic of ALMANACH prompts direct referral when criteria for very severe disease are met and differentiates certain assessments according to the presence or absence of fever and certain other symptoms and signs (figure 1). Subsequently, the denominators reported for variables in the analysis of care processes were considered as ‘indicated according to the algorithm’, whereas in control facilities, the denominator was considered as ‘indicated according to IMCI’.

All calculations were performed using Stata V.16. We used the Strengthening the Reporting of Observational Studies in Epidemiology cohort checklist when writing our report.33

Results

We recruited children from 89 PHC facilities (45 ALMANACH and 44 control). Out of 4148 children screened, 525 (12·7%) children were excluded due to their age, 1602 (38·6%) because attending for routine care or nutrition visits only, 60 (1·4%) for trauma. Five caregivers did not consent for their child to participate in the study and 27 (0·7%) were excluded for other reasons. Of 1929 children enrolled, 1021 (52·9%) were consulted in ALMANACH facilities and 908 (47·1%) in control facilities (figure 2).

Figure 2

Study flow. *45 facilities included, but at one facility in the routine care arm no children presented that were eligible for screening or recruitment. ALMANACH, ALgorithm for the MANAgement of CHildhood illness; LGA, local government area; PHC, primary healthcare.

About half of the enrolled children (48·4%) were under 2 years old, and 48·7% were female. The most commonly reported symptoms were fever (87·2%), cough/breathing problems (35·3%), diarrhoea (31·1%) and vomiting (30·8%). Children most frequently attended between 2 and 7 days from symptom onset (41·2%). In both groups, children had commonly received medication for their illness within 2 weeks prior to the consultation (53·2% in ALMANACH and 57·4% in control facilities), mostly from patent medicine stores or pharmacies (43·1% vs 43·7%, respectively). The majority of patients (76·0%) lived within 30 min travel of the health facility.

Provider cadre was most commonly Community Health Extension Workers (CHEWs) or Community Health Officers (49·3% ALMANACH vs 43·8% control) followed by Junior CHEWs (25·7% vs 20·3%, respectively). Over one-third of HCPs (34·0% in ALMANACH and 39·2% in control facilities) had never received IMCI training. Participant, facility and HCP characteristics are shown in table 1, with additional detail in online supplemental appendix 1.

Table 1

Study sample characteristics: sociodemographic data, illness characteristics, information on healthcare providers and health facilities

Children and care-seeking	ALMANACH	Control	P value	Total
	n (%)	n (%)		n (%)
Pre-SARS-CoV2 (March 2020)	343 (61.4)	216 (38.6)	<0.001*	559 (100)
During SARS-CoV2 (July–Sep 2020)	678 (49.5)	692 (50.5)		1370 (100)
Age in months
2–5	105 (10.3)	101 (11.1)		206 (10.7)
6–11	140 (13.7)	118 (13.0)	0.02*	258 (13.4)
12–23	256 (25.1)	213 (23.5)		469 (24.3)
24–59	508 (49.8)	453 (49.9)		961 (49.8)
Unknown	12 (1.2)	23 (2.5)		35 (1.8)
Sex
Female	483 (47.3)	456 (50.2)	0.02*	939 (48.7)
Male	526 (51.5)	429 (47.3)		955 (49.5)
Unknown	12 (1.2)	23 (2.5)		35 (1.8)
Presenting symptoms††
Fever	899 (88.1)	783 (86.2)	0.23*	1682 (87.2)
Cough/difficulty breathing	369 (36.1)	312 (34.4)	0.41*	681 (35.3)
Diarrhoea	317 (31.1)	283 (31.2)	0.96*	600 (31.1)
Vomiting	329 (32.2)	264 (29.1)	0.14*	593 (30.8)
Skin	41 (4.0)	55 (6.1)	0.04*	96 (5.0)
Other‡	333 (36.7)	371 (36.3)	0.88*	704 (36.5)
Onset of symptoms prior to consultation
Same or previous day	265 (26.0)	275 (30.3)	0.27*	540 (28.0)
2 days to <1 week	439 (43.0)	362 (39.9)		801 (41.5)
1–2 weeks	230 (22.5)	194 (21.4)		424 (22.0)
≥2 weeks	76 (7.4)	63 (6.9)		139 (7.2)
Unknown	11 (1.1)	14 (1.5)		25 (1.3)
Treatment in last 2 weeks
Yes	543 (53.2)	521 (57.4)	0.05*	1064 (55.2)
No	474 (46.4)	379 (41.7)		853 (44.2)
Unknown	4 (0.4)	8 (0.9)		12 (0.6)
Reported travel time to facility
<30 min	696 (68.2)	770 (84.8)	<0.001*	1466 (76.0)
≥30 min	307 (30.1)	128 (14.1)		435 (22.6)
Unknown	18 (1.8)	10 (1.1)		28 (1.5)
Healthcare providers	144 (48.5)	153 (51.5)		297 (100)
Qualification
CHEW/CHO	71 (49.3)	67 (43.8)	0.35§	138 (46.5)
Junior CHEW	37 (25.7)	31 (20.3)		68 (22.9)
Nurse/midwife	2 (1.4)	4 (2.6)		6 (2.0)
Other¶	34 (23.6)	51 (33.4)		85 (28.6)
Last IMCI training received (date)
<1 year ago (2020)	1 (0.7)	3 (2.0)	0.45§	4 (1.4)
1–2 years ago (2018, 2019)	55 (38.2)	41 (26.8)		96 (32.3)
3–4 years ago (2016, 2017)	17 (11.8)	22 (14.4)		39 (13.1)
≥5 years ago (prior to 2016)	9 (6.3)	13 (8.5)		22 (7.4)
Never	49 (34.0)	60 (39.2)		109 (36.7)
Unknown	13 (9.0)	14 (9.2)		27 (9.1)
Health facilities	45 (50.6)	44 (49.4)		89 (100)
Distance from referral hospital
<30 min	16 (35.6)	20 (45.5)	0.39§	36 (40.5)
≥30 min	29 (64.4)	24 (54.6)		53 (59.6)
Number of consultation children (U5) at facility/month
0–99	15 (33.3)	22 (50.0)	0.19§	37 (41.6)
100–199	16 (35.6)	15 (34.1)		31 (34.8)
≥200	11 (24.4)	4 (9.1)		15 (16.9)
Unknown	3 (6.7)	3 (6.8)		6 (6.7)
Health facility power supply
All day (no interruptions)	7 (15.6)	4 (9.1)		11 (12.4)
All day (interruptions)	10 (22.2)	15 (34.1)		25 (28.1)
≤Half a day	17 (37.8)	16 (36.4)	0.82§	33 (37.1)
No electricity source	6 (13.3)	8 (18.2)		14 (15.7)
Unknown	5 (11.1)	1 (2.3)		6 (6.7)
Health facility water supply††
Piped	2 (4.4)	2 (4.6)	1.0§	4 (4.5)
Pump/well	33 (73.3)	23 (52.3)	0.03§	56 (62.9)
None	7 (15.6)	18 (40.9)	0.01§	25 (28.1)
Unknown	5 (11.1)	1 (2.3)	0.20§	6 (6.7)
Outages	7 (15.6)	10 (22.7)	0.43§	17 (19.1)
Stock-outs of medicines for severe illness
Parenteral antibiotics	10 (22.2)	15 (34.1)	0.15§	25 (28.1)
Parenteral antimalarials	7 (15.6)	14 (31.8)	0.13§	21 (23.6)

*Χ2 test.

†Multiple answers possible.

‡Other symptoms: Child is feeling very weak/not drinking/not eating; Child has an ear problem; Belly problem; Pain; Painful urination; Runny nose; Blood in the stool; Sore throat, I don’t know; Other.

§Fisher’s exact test.

¶Other qualifications: Lab technician; EHO; Medical doctor (n=1); Other.

ALMANACH, ALgorithm for the MANAgement of CHildhood illness; CHEW, Community Health Extension Worker; CHO, Community Health Officer; U5, under 5 years of age.

Day 7 recovery rates differed markedly between ALMANACH and control facilities. In ALMANACH facilities, 849 (85·4%) of 994 children with complete follow-up interview were reported to have fully recovered. In control facilities, 603 (71·4%) of 845 children were reported fully recovered. The OR for day 7 recovery was 2.34 (95% CI 1.87 to 2.96) in ALMANACH compared with control facilities, and 2.63 (1.60 to 4.32) after adjusting as described above.

For most secondary outcomes, we observed large differences between ALMANACH and control facilities. HCPs referred patients to a higher level of care over three times more often in the ALMANACH group (RR=3·25 (2.12 to 4.96)). We saw more parenteral antimicrobial prescription and less oral antimicrobial prescription in ALMANACH facilities, with adjusted ORs of 2.42 (1.00 to 5.85) and 0.40 (0.22 to 0.73), respectively. Differentiating antimicrobials into antibiotic and antimalarial treatment revealed that in ALMANACH facilities 120 (12.4%) of 966 children received parenteral antimalarials compared with 59 (6.9%) of 855 in control facilities; for parenteral antibiotics, the respective difference was 56 (5.8%) of 966 versus 25 (2.9%) of 855. Oral antimalarial prescription rate was lower in ALMANACH facilities: 476 (49.3%) of 966 compared with 490 (57.3%) of 855 cases in control facilities. There was no significant difference in oral antibiotic prescription between the groups (ALMANACH: 290 (30.0%) of 966 vs control: 291 (34.0%) of 855).

The intervention affected the communication of diagnosis and follow-up advice to caregivers. In ALMANACH facilities, 811 (84·0%) of 966 caregivers reported that HCPs explained the child’s diagnosis to them compared with only 557 (65·1%) of 855 in control facilities. The likelihood of receiving follow-up advice by the HCP was also much higher in ALMANACH facilities, with 596 (61.7%) of 966 families advised versus 179 (20.9%) of 855 in facilities without ALMANACH. Adjusted associations were very similar.

Finally, we analysed differences in proportions of key diagnoses (according to IMCI protocols) made by HCPs in both groups. Pneumonia, diarrhoea, malnutrition, anaemia were markedly more often, and suspected malaria markedly less often, diagnosed in ALMANACH facilities. Only the rates of diagnosis of malaria confirmed through rapid diagnostic tests were similar between ALMANACH and control groups. All estimates are summarised in table 2.

Table 2

Primary and secondary outcomes

Outcome	ALMANACH facilities			Control facilities			Unadjusted effect estimate		95% CI for unadjusted effect	Adjusted effect estimate		95% CI for adjusted effect
	Cases	Total	%	Cases	Total	%
Recovery after 7 days	849	994	85.4%	603	845	71.4%	RR	1.20	1.14 to 1.26
							OR	2.34	1.87 to 2.96	OR	2.63	1.60 to 4.32
Referral to hospital	96	1009	9.5%	26	861	3.0%	RR	3.25	2.12 to 4.96
							OR	3.48	2.24 to 5.41	OR	3.93	1.89 to 8.14
Diagnosis communicated to caregiver	811	966	84.0%	557	855	65.1%	RR	1.29	1.22 to 1.36	RR*	1.27	1.13 to 1.42
Follow-up advice given to caregiver	596	966	61.7%	179	855	20.9%	RR	2.95	2.56 to 3.39	RR*	2.76	2.12 to 3.58
Parenteral antimicrobial treatment†	164	966	17.0%	80	855	9.4%	RR	1.81	1.42 to 2.33
							OR	1.98	1.41 to 2.63	OR	2.42	1.00 to 5.85
Oral antimicrobial treatment	620	966	64.2%	636	855	74.4%	RR	0.86	0.81 to 0.92
							OR	0.62	0.50 to 0.76	OR	0.40	0.22 to 0.73
Any antimicrobial treatment	745	966	77.1%	690	855	80.1%	RR	0.96	0.91 to 1.00
							OR	0.81	0.64 to 1.01	OR	0.67	0.39 to 1.17
Parenteral antimalarial treatment	120	966	12.4%	59	855	6.9%	RR	1.80	1.34 to 2.42		N/A
Oral antimalarial treatment	476	966	49.3%	490	855	57.3%	RR	0.86	0.79 to 0.94		N/A
Parenteral antibiotic treatment	56	966	5.8%	25	855	2.9%	RR	1.98	1.25 to 3.15		N/A
Oral antibiotic treatment	290	966	30.0%	291	855	34.0%	RR	0.88	0.77 to 1.01		N/A
Antibiotic and antimalarial treatment	189	966	19.6%	168	687	19.6%	RR	1.00	0.83 to 1.20
							OR	0.99	0.79 to 1.25	OR	0.87	0.41 to 1.85
Suspected malaria	70	811	8.6%	77	557	13.8%	RR	0.62	0.46 to 0.85		N/A
Confirmed malaria	525	811	64.7%	363	557	65.2%	RR	0.99	0.92 to 1.08		N/A
Pneumonia	59	811	7.3%	5	557	0.9%	RR	8.10	3.27 to 20.06		N/A
Diarrhoea	118	811	14.5%	46	557	8.3%	RR	1.76	1.28 to 2.43		N/A
Malnutrition	57	811	7.0%	7	557	1.3%	RR	5.60	2.57 to 12.17		N/A
Anaemia	31	811	3.8%	3	557	0.5%	RR	7.10	2.18 to 23.10		N/A

*Adjusted rate ratio from negative binomial regression (see statistical analysis).

†Eight observations not used due to separation (combinations of covariates predicting the outcome perfectly).

ALMANACH, ALgorithm for the MANAgement of CHildhood illness; N/A, not applicable; RR, risk ratio.

Comparative analysis of ALMANACH and record data indicated further differences between the groups in care process outcomes. In ALMANACH facilities, 850 (83.3%) of 1021 consultations were conducted using ALMANACH. When ALMANACH was used, 850 (100%) children were screened for IMCI danger signs, 847 (99.6%) had a recorded weight, MUAC was recorded in 718 (99.9%) of 719 indicated, temperature recorded in 714 (95.1%) of 751 indicated, pallor assessed in 745 (100%) of 745 indicated, and respiratory rate recorded in 192 (100%) of 192 indicated. In control facilities, we found no documentation of danger signs or respiratory rate, 197 (21.7%) of 908 had a recorded weight, MUAC was recorded in 52 (6.6%) of 784 children 6–59 months old and temperature in 469 (51.7%) of 908 cases. Using malaria assessment and treatment as an example of adherence to guidelines, we found lower effectiveness decay through the care pathway in ALMANACH compared with control facilities (figure 3).

Figure 3

Comparison of systems effectiveness decay for malaria assessment and treatment in ALMANACH and control facilities. Steps reflecting adherence to guidelines are shown in blue, non-adherence shown in red, and not applicable in grey. ALMANACH, ALgorithm for the MANAgement of CHildhood illness. RDT, Rapid Diagnostic Test.

Discussion

This observational study found significant improvements in caregiver-reported recovery of children 7 days after primary care consultation associated with the implementation of the ALMANACH digital CDSS with training, mentorship and data feedback in a programmatic setting. This impact on health outcomes is likely mediated by better adherence to evidence-based guidelines, supported by demonstrated improvements in QoC process outcomes across assessment, diagnosis and management.

In children for whom ALMANACH was used, we found almost complete adherence to key IMCI assessments. In contrast, in control facilities in this study, and other studies on IMCI-related quality of care, HCPs commonly complete assessments in fewer than 50% of consultations with sick children.34 35 We posit that this adherence to IMCI assessments, guided by ALMANACH, led to higher detection of severe illness and of diagnoses that are important causes of morbidity and mortality, particularly pneumonia, anaemia, diarrhoea and malnutrition. The significantly higher rate of these IMCI diagnoses (and lower rate of ‘suspected’ malaria diagnosis) made in ALMANACH facilities supports this assumption, which may also have contributed to more accurate antimicrobial prescription.

In turn, we observed higher rates of referral and of treatment with parenteral antimicrobials, indicative of increased recognition and management of severe illness. Inadequate identification and treatment of severe disease are major barriers to reducing child mortality in LMICs.36 37 While the proportion of children with severe disease varies substantially between and within countries, with estimates varying from 5% to 21% in studies from comparable settings,35 38 the 3.0% referral rate reported by caregivers in control facilities in this study is likely to be inappropriately low, and the 9·5% ALMANACH referral rate more consistent with the clinical need.

Furthermore, ALMANACH showed significantly better HCP communication with caregivers as evidenced by higher rates of awareness of diagnosis and follow-up advice at exit interviews in ALMANACH facilities. Communication about a child’s illness and treatment are key standards set out by WHO for improving QoC for children at health facilities and are associated with higher satisfaction with and intention to return to care.5 39 In ALMANACH facilities, caregivers were made aware of diagnosis in 84.0% and given follow-up advice in 61.7% of consultations, substantially higher than in control facilities (65.1% and 20.9%, respectively) and in multicountry IMCI studies in sub-Saharan African countries (43%–70% and 20%–57%, respectively).34 39

These findings of better adherence to key evidence-based practices are consistent with other studies of child health-related CDSS in resource-constrained settings, including ALMANACH.19–21 29–31 Lack of training and other knowledge gaps such as difficulty recalling specific criteria (eg, respiratory rate cut-offs) contribute to low adherence to IMCI, as do low motivation and physical and cognitive overload associated with working in such challenging settings.12 Qualitative feedback from HCPs in Burkina Faso and Tanzania suggests that IMCI-related CDSS can improve confidence in diagnoses and managements, strengthen motivation and address the issue of cognitive overload through the step-by-step nature of the guidance tailored to individual patients, particularly for severity classification and drug dosing.17 28 Though we did not collect qualitative data as part of this study, it is possible that similar factors were mediators of effectiveness in Adamawa. However, more research is needed to understand the relative importance of context and different components of the intervention package as enablers (including those mentioned above and others such as general investment in staff and commitment of the ADSPHCDA) and barriers (such as staff turnover and consultation length) to adoption and guideline adherence.

While increased adherence to guidelines has been consistently demonstrated, few studies have assessed the impact of IMCI-related CDSS on health outcomes and, to our knowledge, none in the context of a long-term, large-scale implementation. The ALMANACH controlled trial in Tanzania also found higher day 7 recovery rates, though also in the control recovery was higher (92·0% in control to 97·3% in ALMANACH facilities).25 This is relatively high compared with the day 7 recovery rates we saw in this study (71·4% in control and 85·4% in ALMANACH facilities). The contextual differences in sociodemographic, epidemiological and health system factors are likely to account for this, though a more substantial Hawthorne effect may also have been in effect as data were collected inside the consultation room in the Tanzania study.

This study demonstrates that implementation of ALMANACH can deliver impact on health outcomes at scale in remote, resource-constrained PHC facilities, where drug stock-outs are common, many facilities lack basic amenities, and one-third of HCPs have never received IMCI training. A recent systematic review and meta-analysis found only modest changes in health outcomes with CDSS implementation.40 Most of the CDSS studies included in this review were implemented in high-income countries, where HCP’s access to resources is substantially higher than most PHCs implementing IMCI. This may indicate that CDSS can deliver most value when used to support HCPs with limited skills and resources, provided that they are appropriately contextualised and implemented.

In contrast with earlier ALMANACH studies,29–31 we did not find significant lower antibiotic prescription rates. Oral antibiotics were prescribed to 30·0% of children in ALMANACH and 34·0% in control facilities, slightly lower than the 43.1% (33.2 to 50.5) found in a recent meta-analysis of reported antibiotic use for sick children in LMICs. However, these may not reflect true antibiotic consumption rates given that most caregivers also reported treatment prior to consultation, most commonly from patent medicine stores or pharmacies. With increasing antibiotic prescription rates for children under 5 years of age, rising fastest in low-income countries, antimicrobial stewardship remains a global health priority to mitigate individual adverse events, rising antimicrobial resistance and resource waste.41 42 Further gains in antimicrobial stewardship may be possible to achieve through integrating host biomarkers into IMCI-related CDSS.24

Our study has several limitations. Due to the nature of the evaluation in the programmatic setting, facilities were not randomised to ALMANACH or control. The presence of a large-scale child health intervention in some LGAs, combined with security-related and weather-related accessibility issues, limited the number of suitable LGAs for the control group. Contextual differences including in epidemiology, health-seeking behaviour and the health system may, therefore, have influenced our data, though we adjusted for important potential confounders within the analysis. Furthermore, the intervention could not be blinded, so performance or detection bias could have occurred, despite the use of standardised tools and procedures. The Hawthorne effect may have influenced the data, which has been found to increase patient-reported quality of care by 13%.43 The observer effect should be similar in both intervention and control facilities, although there is a possibility that a differential effect could occur if the tablets were used more than usual in ALMANACH facilities. To reduce the likelihood of influencing HCP performance, we avoided direct observation and only collected information from caregivers and records. We did not conduct repeat clinical assessments after consultations due to the potential need to modify treatment, which could have influenced the primary outcome. It was, therefore, not possible to have complete certainty about the content of clinical consultations, not to conduct a thorough analysis of correct assessment, classification and treatment (including of antimicrobial appropriateness and referral appropriateness) of all IMCI diagnoses, and, thus, we were only able to provide some insights into the differences in routinely documented quality of care indicators. Finally, given the complex nature of the intervention, incorporating the tablet-based CDSS along with training, mentorship and data feedback, we cannot determine the effect of the CDSS itself versus the entire intervention package. However, it is unlikely that training alone accounts for the difference in outcomes given that there was no major difference between groups in time since most recent IMCI training.

In conclusion, we found substantial impact of this IMCI-related CDSS on health and QoC outcomes, demonstrating that earlier findings in controlled or small-scale studies can be achieved at scale in a resource-constrained setting. Positive effects were seen across a range of process and outcome indicators, including assessment, diagnosis, treatment and communication. These findings support the implementation of digital CDSS, with training, mentorship and data feedback, in resource-constrained settings as a means to strengthen progress towards universal health coverage.