Usability and acceptability of an automated respiratory rate counter to assess children for symptoms of pneumonia: A cross-sectional study in Ethiopia.

AIM: Manually counting respiratory rate (RR) is commonly practiced by community health workers to detect fast breathing, an important sign of childhood pneumonia. Correctly counting and classifying breaths manually is challenging, often leading to inappropriate treatment. This study aimed to determine the usability of a new automated RR counter (ChARM) by health extension workers (HEWs), and its acceptability to HEWs, first-level health facility workers (FLHFWs) and caregivers in Ethiopia.
METHODS: A cross-sectional study was conducted in one region of Ethiopia between May and August 2018. A total of 131 HEWs were directly observed conducting 262 sick child consultations after training and 337 after 2 months. Usability was measured as adherence to the WHO requirements to assess fast breathing and device manufacturer instructions for use (IFU). Acceptability was measured through semi-structured interviews.
RESULTS: After 2 months, HEWs were shown to adhere to the requirements in 74.6% consultations; an increase of 18.6% after training (P < .001). ChARM is acceptable to users and caregivers, with HEWs suggesting that ChARM increased client flow and stating a willingness to use ChARM in future.
CONCLUSION: Further research on the performance, cost-effectiveness and implementation of this device is warranted to inform policy decisions in countries with a high childhood pneumonia burden.

Acute respiratory infection

Acute Respiratory Infection Diagnostic Aid

Children's Respiration Monitor

Community health worker

Confidence interval

First‐level health facility worker

Health extension worker

Instructions for use

Integrated community case management

Integrated management of childhood illnesses

Respiratory rate

Southern Nations, Nationalities, and Peoples' Region

Standard deviation

United Nations Children's Fund

World Health Organization

Correctly counting breaths manually to detect fast breathing, an important sign of childhood pneumonia, is challenging and often leads to inappropriate treatment.

These results suggest that health extension workers in Ethiopia can use a new automated respiratory rate counter to manage under‐five children according to requirements.

Further research on the performance, cost‐effectiveness and implementation of this device is warranted to inform policy decisions in high childhood pneumonia burden countries.

INTRODUCTION

In the last two decades, considerable progress has been made in reducing child mortality, with an estimated 5.5 million deaths in children under‐five occurring in 2017.1 However, acute respiratory infections (ARIs), primarily pneumonia, remain the leading infectious cause of death amongst children under‐five globally, accounting for an estimated 0.9 million deaths in 20152 with over 75% of these deaths clustering in sub‐Saharan Africa and south‐east Asia.1 With deaths from pneumonia in children resulting mostly from delayed presentation to appropriate health care providers and inappropriate treatment,3 there is a need to improve delivery of interventions to expand their reach and improve access for the most vulnerable children.4, 5 Community‐based care through integrated community case management (iCCM)4 of childhood illness by community health workers (CHWs) or integrated management of childhood illnesses (IMCI)6 by first‐level health facility workers (FLHFWs) trained in assessment and treatment of suspected cases can help in the early detection and prompt treatment, thus significantly contributing to the reduction of pneumonia‐associated morbidity and mortality. However, increasing access to community‐level interventions must be integrated with efforts to ensure children are receiving appropriate diagnosis and treatment for their condition.

In low‐income settings, CHWs and FLHFWs (collectively known as front‐line health workers) rely on clinical assessment for symptoms of pneumonia based on counting the number of breaths in 60 seconds in under‐five children with cough and/or difficulty breathing and assessing whether the respiratory rate (RR) is high enough to be considered ‘fast breathing’, one of the signs used to detect suspected pneumonia and a trigger for the prescription of life‐saving antibiotics.

Standard practice is to manually count RR by observing chest movements. In practice, defining a breath and counting RR can be difficult, as children breathe irregularly and faster than adults, and the child may not be calm and still for a full minute. Front‐line health workers use ARI timers, stopwatches, clocks and smartphone timers to help count the number of breaths in 60 seconds, but a precise estimate of the RR remains challenging, even for experienced health workers. Misclassification of the observed rate remains high,7, 8 often leading to inappropriate treatment.9

The Acute Respiratory Infection Diagnostic Aid (ARIDA) project,10 was initiated to address calls for better devices that diagnose symptoms of pneumonia.11, 12 Several ARIDA devices have been developed in response to UNICEF's Request for Proposals,13 and field trials have been conducted in Ethiopia and Nepal to assess usability and acceptability of these devices. The study reported in this paper is one of the field trials in Ethiopia for the Philips Children's Respiration Monitor (ChARM), an accelerometer‐based system to measure the RR in children 0‐59 months old and automatically classifies the breathing rate according to the iCCM/IMCI algorithms.14

Since 2010, Ethiopia has scaled up iCCM in all regions through CHWs locally known as health extension workers (HEWs).15 All HEWs are literate women with at least tenth grade education who are trained for 1 year in iCCM and equipped to assess, classify and manage uncomplicated pneumonia, malaria, diarrhoea and severe acute malnutrition and provide preventive and curative health services. Trained HEWs are deployed, usually in pairs, to a health post to work at the sub‐district (kebele) level as a government employee and serve a population of approximately 5000 people. Through iCCM, any child 2‐59 months old with fast breathing pneumonia and no danger signs can be treated at the health post; sick children under 2 months are referred to a health centre. Each health centre is staffed by around 20 FLHFWs and provides services to approximately 25 000 people and serves as a referral centre and training institution for HEWs.

A front‐line health workers’ intention to adhere to requirements can be affected by facets of acceptability16: affective attitude, burden, intervention coherence, perceived effectiveness and self‐efficacy. These acceptability facets, combined with front‐line health workers’ skills and abilities (level of education, knowledge of WHO requirements to assess fast breathing, understanding of how to use the device and the device manufacturer instructions for use [IFU]) and other constraints (child behaviour, caregiver behaviour, context and setting) will affect their adherence behaviour now and in future. The aim of this study was therefore to understand the usability of ChARM to HEWs, measured as their adherence to WHO requirements to assess fast breathing and device manufacturer IFU17 and its acceptability amongst HEWs, FLHFWs and caregivers measured through semi‐structured interviews.

METHODS

Study design

This was a cross‐sectional study using mixed methods design. Only HEWs were observed, under the assumption that FLHFWs would be able to use ChARM as well or better than HEWs due to their higher education level and longer training. The methods are described in full elsewhere.18

Study setting

Data were collected from HEWs in health posts and FLHFWs in health centres in Shebedino, Dale and Boricha districts in Southern Nations, Nationalities, and Peoples' Region (SNNPR), Ethiopia, between May and August 2018. Ethiopia was selected due to the high burden of ARIs in under‐fives (responsible for 16% deaths in under‐fives in 2016),19 availability of front‐line health workers trained in and delivering community‐based management of pneumonia who may benefit from an ARIDA device, availability of first‐line amoxicillin dispersible tablets and availability of oxygen at the district hospital. SNNPR also has sufficient number of HEWs with experience and availability to participate and logistical and operational feasibility for data collection and quality assurance.

Sample size

The study was powered for the primary outcome that is to measure the proportion of under‐five child consultations where HEWs using ChARM adhered to WHO requirements to assess fast breathing and device manufacturer IFU, after 2 months. Using the sample size formula for a prevalence study with a 95% level of confidence and 7.5% precision, a sample size of n = 264 child consultations is required to estimate the true proportion of the outcome, assuming that the proportion of HEWs completing all the steps correctly is 71% (75% conducting the RR steps correctly and 95% of these classifying the RR correctly), a design effect of 1.7 to account for clustering at CHW level and a dropout rate of 10%. Thus, 132 HEWs would need to be observed completing two sick children assessments twice (totalling 528 consultations), once after training and once after 2 months.

Data collection methods and sampling

All available HEWs in Dale and Shebedino were selected for the study and a further 17 in Boricha, in consultation with the regional health bureau. A total of twenty FLHFWs working in the under‐five clinic of their health centre (one per health centre) were selected in consultation with the heads of the three district health offices.

ChARM devices were provided to 133 HEWs and 20 FLHFWs with refresher training on iCCM/IMCI and training on how to use ChARM. They completed pre‐ and post‐training assessments to test knowledge of iCCM/IMCI and ChARM using a 12 question questionnaire. The pass mark for HEWs was 75% to ensure they had reached a sufficient level to be assessed in the study. Immediately after the training, research assistants observed 131 HEWs using ChARM during two consecutive child consultations in the health post (observation one). HEWs were observed completing 11 steps comprising WHO requirements on how to prepare a child for fast breathing assessment and how to count and classify RR, plus device manufacturer IFU on how to position the child and ChARM to get a valid RR reading (Appendix S1). HEWs had up to three attempts to obtain a RR classification with ChARM; if these attempts were unsuccessful, the HEW reverted to using standard practice. HEWs’ ability to treat and refer children was also recorded. If research assistants noted the HEW making an incorrect classification or giving incorrect treatment or referral advice, they were instructed to wait for the HEW to complete the assessment, complete the data collection form based on the initial consultation and then advise on the correct course of action. HEWs then used ChARM routinely for 2 months before being observed a second time (observation two) conducting two child consultations using an identical methodology.

Research assistants (either medical officers or degree‐qualified nurses) worked in pairs to screen under‐five children for inclusion criteria, and then directly observe the HEW conducting the consultation. Inclusion criteria were any child aged 0‐59 months presenting to the health post with caregiver consent. For 2‐ to 59‐month‐olds, the child also needed to have had cough and/or difficulty breathing. Exclusion criteria for children in all elements of the study were caregiver's age <16 years, no caregiver consent or device manufacturer safety exclusion criteria,17 those with convulsions (0‐<2 months) and those with iCCM general danger signs or referral signs for severe disease (2‐59 months).4

Each step of the consultation was recorded in a digital tablet‐based data collection form using CommCare (version 2.38.1, Dimagi). Research assistants photographed the RR result, classification and age displayed on ChARM to provide source documents for verification purposes. They synced data daily to a protected cloud server. Between the first and second observations, HEWs were encouraged to use ChARM, but were not disallowed to revert to standard practice and instructed to record which device they used in their patient register using coloured stickers (one patient register per health post). FLHFWs using ChARM were not observed, but they were encouraged to use ChARM at the health centre for 2 months prior to their semi‐structured interview.

Semi‐structured interviews were conducted with a subsample of HEWs immediately after observation two. HEWs with a range of years of experience qualified as a HEW were purposefully selected, and caregivers of children who were assessed by this subsample of HEWs were also interviewed. A convenience sample of FLHFWs available on the day of the visit to the health facilities was used. The topic guides were developed using a comprehensive conceptual framework of acceptability of healthcare interventions and translated into the local languages (Amharic and Sidaminya).16 A total of six research assistants (all Ethiopian nationals with prior experience of qualitative methods) were briefed on the purpose of the study and trained to conduct interviews. All semi‐structured interviews were conducted in the local language, audio recorded, and subsequently translated and transcribed to English.

Data analysis

Continuous data were summarised as percentages, and means with standard deviation (SD) and categorical data were presented as percentages with 95% confidence intervals (CIs). For the main outcomes, the most conservative estimates were used, that is if the two research assistants disagreed on how the HEW performed a step in the consultation, the one that recorded an inconsistency/error for that step was used over the one who recorded that the step was performed correctly. A sensitivity analysis was also conducted using the less conservative estimates. Multilevel logistic regression analysis was performed to assess whether the odds of a child being assessed and classified according to WHO requirements to assess fast breathing, and device manufacturer IFU was different between the two time points. Univariable logistic regression analyses were performed to assess for a relationship between the proportion of consultations where HEWs adhered to WHO case management and device manufacturer IFU and three variables: time since (a) a HEW's last routine iCCM integrated refresher training; (b) a HEW's last routine supervision; or (c) qualification as a HEW. The mean time taken to complete the full assessment, defined as the difference between the time when the HEW starts to strap on ChARM to when it displays a RR reading, inclusive of multiple attempts, was calculated at both time points. The number of children who were assessed for respiratory signs and symptoms by HEWs with ChARM or standard practice at the health post was calculated, excluding data from the direct observations (after training and after 2 months). All quantitative data were analysed using Stata version 13 (StataCorp).

A thematic analysis of the qualitative data was conducted, using MAXQDA (VERBI Software, 2016) to manage data coding, searching and retrieval. HS developed initial coding frames for caregiver and front‐line health worker transcripts, which were discussed with CW, KB and AM and then used to code all transcripts. Frequently used codes were explored, and coded data extracted into matrices; HS further collated coded data into broad categories and then emerging themes. Summaries of each theme were reviewed and discussed before final consolidation.

Ethical approval

Liverpool School of Tropical Medicine Research Ethics Committee (ref 18‐026) gave favourable ethical opinion to the protocol on July 10, 2018 and the SNNPR Regional Health Bureau Ethics Committee (ref እድ‐241/20852) gave ethical approval on May 4, 2018. Research assistants obtained written consent for observations and interviews from each HEW and from each caregiver whose child was assessed by a HEW during an observation. Written consent was obtained from FLHFWs selected for interview.

Patient and Public Involvement

Patients and the public were not involved in the design, conduct or reporting of the research.

RESULTS

Participant characteristics

One hundred and thirty‐three female HEWs and twenty female FLHFWs completed the training. The average test score before and after training for HEWs was 61.0% and 88.0%, respectively. Characteristics of the 131 HEWs who were observed using ChARM and characteristics of HEWs, FLHFWs and caregivers who participated in semi‐structured interviews are shown in Table 1.

Table 1

Characteristics of front‐line health workers and caregivers, by district

	District			Overall
	Shebedino	Dale	Boricha
Number (%) HEWs observed using ChARM	52 (39.7)	62 (47.3)	17 (13.0)	131a (100)
Number (%) HEWs received iCCM integrated refresher training ≤3 y agob	22 (42.3)	36 (58.1)	6 (35.3)	64 (48.9)
Number (%) HEWs received last supervision ≤3 mo agob	30 (57.7)	49 (79)	16 (94.1)	95 (72.5)
Mean (SD) years' experience as a HEW	8.6 (4.6)	7.7 (4.5)	7.5 (4.2)	8.0 (4.5)
Number (%) HEWs participating in a semi‐structured interview	3 (21.4)	8 (57.1)	3 (21.4)	14 (100)
Mean (SD) years' experience as a HEW	11.7 (1.5)	9.3 (4.0)	5.5 (5.1)	8.9 (4.2)
Number (%) FLHFWs participating in a semi‐structured interview	5 (38.5)	6 (46.2)	2 (15.4)	13 (100)
Mean (SD) years' experience as a FLHFW	5.8 (0.4)	c	7.0 (1.4)	6.1 (0.9)
Number (%) caregivers participating in a semi‐structured interview	3 (23.1)	8 (61.5)	2 (15.4)	13 (100)

N = 2 lost to follow‐up between training and the first observation due to sickness.

Training and supervision coordinated by the SNNPR Regional Health Bureau, independently and prior to this study.

Missing.

About two hundred and sixty‐eight children were enrolled for the first and 339 children for the second observation (Figure 1). This increase was due to a recruitment drive where research assistants encouraged HEWs to ask caregivers of the youngest children to bring their sick child to the health post for a pneumonia consultation, which increased enrolment for 0‐ to <2‐month‐old children by 17.2% for observation two. Of these, 262 (observation one) and 337 (observation two) consultations were started, the reason for not starting was the child not being calm enough. For observations at both time points, just over half the children were male (Figure 1).

Figure 1

Participant study flow for observations one (after training) and two (after 2 mo routine ChARM use)

Most consultations were completed with ChARM on the first attempt (87.3% and 91.2% for the first and second observations, respectively) (Figure 1). The proportion of consultations (observation one and two) completed on attempt one by age group was 92.8% (0‐<2 months); 94.5% (2‐<12 months) and 88.4% (12‐59 months). There were 35 (11.9%) and 34 (9.2%) unsuccessful ChARM attempts out of all attempts, for observations one and two, respectively. The most common reason being that ChARM displayed ‘‐‐‐’(74.3% and 88.2% for observations one and two, respectively), indicating that there was excess motion or a loosely attached belt. A total of three (1.1%) and two (0.6%) consultations could not be completed with ChARM out of all consultations, for observations one and two, respectively (Figure 1).

Usability of ChARM to HEWs

In the second observation, HEWs fully adhered to WHO requirements to assess fast breathing and device manufacturer IFU in 74.6% (95% CI 69.9‐79.3) of the consultations, an increase of 18.6% from the first observation immediately after training (Table 2) (P < .001).

Table 2

Number and proportion of child consultation steps correctly performed by HEW with ChARM after training (observation one) and after 2 mo of routine use (observation two)

No	Consultation step	Observation 1 (after training)			Observation 2 (after 2 mo)
		n	%d	95% CI	n	%d	95% CI
1	Correct child positiona	200	76.3	71.2‐81.5	273	81.0	76.8‐85.2
2	Correct device positiona	214	81.7	77.0‐86.4	319	94.7	92.3‐97.1
3	Correct belt positiona	259	98.9	97.6‐1.0	337	100.0	98.9‐1.0
4	Correct age groupb	248	94.7	91.9‐97.4	332	98.5	97.2‐99.8
5	Child calm before ChARM attempta	242	92.4	89.2‐95.6	326	96.7	94.8‐98.6
6	Child not eating/feeding during ChARM attempta	256	97.7	95.9‐99.5	336	99.7	99.1‐1.0
7	Child calm during ChARM attempta	249	95.0	92.4‐97.7	332	98.5	97.2‐99.8
1‐7	Cumulative assessment (steps 1‐7)	186	55.3	49.9‐61.4	251	74.5	69.8‐79.1
8	Correct classification using ChARM (yes/no?)b	256	98.8	97.5‐1.0	333	99.4	97.5‐1.0
1‐8	Correct assessment and classification (steps 1‐8) – primary outcome	145	56.0	49.9‐62.0	250	74.6	69.9‐79.3
9	Correct treatment using ChARM ‐ did the HEW make the right choice of whether to treat (yes/no?)c	256	98.8	97.5‐1.0	331	99.1	98.1‐1.0
10	Correct course of treatment using ChARM and HEW's assessment of other symptoms (yes/no?)c	56	93.3	87.0‐99.6	109	98.2	95.7‐1.0
11	Correct referral using ChARM and HEW's assessment of other symptoms (yes/no?)	5	62.5	29.0‐96.0	32	88.9	78.6‐99.2
1‐3	Manufacturer instructions for use correctly performed (steps 1‐3)a	164	62.6	56.7‐68.5	259	76.9	72.4‐81.4
4‐8	WHO requirements to assess fast breathing correctly performed (steps 4‐8)a, b	222	84.7	80.4‐89.1	318	94.4	91.9‐96.8

Based on two research assistants observing the HEW. Where two research assistants disagreed, most conservative estimate was used.

Based on comparison of the age group recorded on the screening checklist and the photograph of the ChARM with result displayed.

Based on two research assistants observing the HEW. Where two research assistants disagreed, the project manager verified through retrospective patient register review.

Step nos. 1‐7, 1‐3, 4‐8: N = 262 (observation 1) and 337 (observation 2)—children whose consultation started. Step nos. 8, 1‐8 and 9: N = 259 (observation 1) and N = 335 (observation 2)—children with RR classification with ChARM. Step no. 10 N = 60 (observation 1) and N = 11 (observation 2)—children with fast breathing. Step no. 11 N = 8 (observation 1) and 111 (observation 2)—children with fast breathing and a referral sign.

In the second observation, HEWs adhered WHO requirements to assess fast breathing and device manufacturer IFU correctly for 91.7% (95% CI 85.3‐98.1) of the 0‐ to <2‐month‐olds compared to 76.6% (95% CI 69.6‐83.6) of the 2‐<12 month and 62.3% of the 12‐ to 59‐month‐olds (95% CI 53.7‐70.9). The largest variation in assessment steps between age group was seen for ‘correct child position’ (97.2% vs 85.8% vs 66.1% for the 0‐ to <2‐month‐olds, 2‐<12 months and 12‐59 months, respectively). HEWs’ adherence was higher during consultations of fast breathing children (79.6%; 95% CI 72.2‐87.1) compared to normal breathing children (72.1%; 95% CI 71.1‐82.9) (Table 3).

Table 3

Number and proportion of child evaluation steps correctly performed by HEW with ChARM after 2 mo of routine use, by child age group and breathing status

No.	Consultation step	Observation two (0‐<2 mo) N = 72			Observation two (2‐<12 mo) N = 141			Observation two (12‐59 mo) N = 124			Observation two (fast breathersb) N = 113			Observation two (normal breathersb) N = 222
		n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI
1	Correct child positiona	70	97.2	93.4‐1.0	121	85.8	80.1‐91.6	82	66.1	57.8‐74.5	102	90.3	84.8‐95.7	171	77.0	71.5‐82.6
2	Correct device positiona	70	97.2	93.4‐1.0	128	90.8	86.0‐95.6	121	97.6	94.9‐1.0	103	91.2	85.9‐96.4	214	96.4	93.9‐98.8
3	Correct belt positiona	72	100.0	95.0‐1.0	141	100.0	97.4‐1.0	124	100.0	97.1‐1.0	113	100.0	96.8‐1.0	222	100.0	98.4‐1.0
4	Correct age groupb	71	98.6	95.9‐1.0	140	99.3	97.9‐1.0	121	97.6	94.9‐1.0	111	98.2	95.8‐1.0	219	98.6	97.1‐1.0
5	Child calm before ChARM attempta	70	97.2	93.4‐1.0	137	97.2	94.4‐99.9	119	96.0	92.5‐99.4	109	96.5	93.1‐99.9	215	96.8	94.5‐99.1
6	Child not eating/feeding during ChARM attempta	72	100.0	95.0‐1.0	140	99.3	97.9‐1.0	124	100.0	97.1‐1.0	112	99.1	97.4‐1.0	222	100.0	98.4‐1.0
7	Child calm during ChARM attempta	72	100.0	95.0‐1.0	138	97.9	95.5‐1.0	122	98.4	96.2‐1.0	111	98.2	95.8‐1.0	219	98.6	97.1‐1.0
8	Correct classification using ChARM (yes/no?)b	72	100.0	95.0‐1.0	140	99.3	97.9‐1	121	99.2c	97.6‐1.0	112	99.1	97.4‐1.0	221	99.5	98.7‐1.0
1‐8	Correct assessment and classification (steps 1‐8)	66	91.7	85.3‐98.1	108	76.6	69.6‐83.6	76	62.3c	53.7‐70.9	90	79.6	72.2‐87.1	160	72.1	71.1‐82.9

Based on two research assistants observing the HEW. Where two research assistants disagreed, least conservative estimate was used.

Based on comparison of the age group recorded on the screening checklist and the photograph of the ChARM with result displayed.

Denominator = 122 due to two 12‐59 mo children not having RR classification with ChARM.

The mean time to get a reading with ChARM was 03:13 minutes (range, 01:09‐08:55 minutes, SD = 01:14 minutes) for observation one and 03:17 minutes (range = 01:12‐15:13 minutes, SD = 01:25 minutes) for observation two. In the second observation, HEWs took on average 03:31 minutes (SD = 01:08) to get a reading for the 0‐ to <2‐month‐olds, which on average was 23 and 13 seconds longer than the 2‐ to <12‐month‐olds and 12‐ to 59‐month‐olds, respectively.

In the first observation, HEWs who had had fewer months since their last iCCM integrated refresher training were more likely to adhere to requirements (P = .016). After 2 months, the logistic regression analysis found no significant association between (a) number of years the HEW had been qualified (P = .45); (b) number of months since last iCCM integrated refresher training (P = .10); and (c) number of months since last supporting supervision (P = .42) and whether the child was managed according to requirements (Table 4).

Table 4

Results from univariable logistic regression to explore whether characters of health extension workers affect their ability to adhere to required guidelines, after training and after 2 mo

	Adherence to requirements in observation 1 (after training)			Adherence to requirements in observation 2 (after 2 mo)
	Odds ratio	95% CI	P‐value	Odds ratio	95% CI	P‐value
No. years HEW has been qualified	1.02	0.96‐1.17	.594	0.98	0.93‐1.03	.449
No. months since last iCCM integrated refresher training	0.99	0.98‐1.00	.016	0.99	0.98‐1.00	.097
No. months since last supporting supervision	1.05	0.99‐1.11	.107	1	0.97‐1.08	.421

Results from the sensitivity analysis show that in 84.2% (95% CI 80.3‐88.9) of consultations, HEWs adhered to WHO requirements to assess fast breathing and device manufacturer IFU using ChARM, an increase of 9.3% from the first observation immediately after training. When comparing agreement between research assistants observing the same child, most disagreement was for determining child position, where pairs disagreed in 10.7% (observation one) and 9.5% (observation two) of child consultations (Appendix S2).

Between the first and second observation, 60 (89.6%) health posts reported data in their patient registers with stickers. There were 677 under‐five consultations for children presenting with respiratory symptoms completed between June and July 2018. Of these, 428 (63.2%) were completed with ChARM (mean 7.1 ChARM consultations per health post), 40 (5.9%) were completed with their standard practice device (ARI timer, smartphone timer or watch), and 209 (30.9%) were completed with an unknown device. Of the assessments completed with a known device, 91.5% were completed with ChARM.

ACCEPTABILITY OF CHARM

We identified four main themes relating to acceptability of ChARM amongst HEWs, FLHFWs and caregivers.

Attitudes towards ChARM

All HEWs and many FLHFWs compared ChARM to their previous RR counting method (stopwatch), for which they described potential for ‘miscounting’ or ‘carelessly’ counting RR and a frequent need to repeat counting. All of which they perceived led to a ‘wrong’ or ‘misleading’ RR count or led front‐line health workers to ‘misclassify’ the RR. Overall, both HEW and FLHFWs felt that ChARM overcame these difficulties (Appendix S3, box 1). Both described how they trusted the device only after checking it ‘several times’ with the same child or cross‐checking the result with the stopwatch. Some HEWs even described how they exchanged ChARM with a colleague to check if they produced the same result (Appendix S3, box 1).

Caregivers’ response to ChARM was overwhelmingly positive (Appendix S3, box 1). As commonly seen in the Ethiopia context, many described how they accepted ChARM because the government provided it ‘to support the community’ and how the government brings only ‘good things’ which ‘do not hurt us’. Many also accepted ChARM because they trusted the HEWs, and they would not do anything to cause ‘harm’. Many caregivers felt ChARM was safe, and some explained they did not see any ‘problems’ with the device and said it does not cause any ‘damage on the child's body’. Others described the short time taken to assess their child with the ChARM device, with some comparing this to the time spent before in other facilities.

Views on ChARM in supporting the classification of respiratory rate

Most HEWs described the red and green lights displayed on ChARM as ‘easy to read and understand’; FLHFWs were of the same opinion. Some compared using ChARM with the stopwatch, where they would have to ‘remember the cut off points’ from the job aid to be able to classify cases. Both HEWs and FLHFWs said that having the results displayed on the screen enabled them to show the mother and directly communicate the result (Appendix S3, box 2).

The majority of HEWs and FLHFWs referred to tying and adjusting the belt as the most difficult step, especially on ‘small’ or ‘skinny’ children. Both HEWs and FLHFWs felt that older children (above one or 2 years) were more likely to be distressed during the assessment; reasons included ‘fear of the device’, fear of the ‘white gown of the health worker’, fear related to ‘previous experience with injections’ and general unfamiliarity with ChARM and the environment. Distress was associated with seeing a new device, attaching ChARM to the abdomen, and sick children tended to be more distressed (Appendix S3, box 2).

Most caregivers felt comfortable for the device to be used on their child again because they felt that ChARM ‘shows the accurate result’ or ‘knows the problem of the child’. Many caregivers associated the device with treatment and were willing to recommend it to others on that basis (Appendix S3, box 2). For a few caregivers, the availability of the device at the health post ‘near my home’ made care seeking ‘easy’, and this was considered important for future use.

Burden or effort required to use ChARM

Almost all HEWs expressed that they could easily perform other tasks whilst waiting for the ChARM result, with some specifying that it is easier to multi‐task if the child is calm (Appendix S3, box 3). Some also perceived that assessment with ChARM took less time to complete than with the stopwatch. In contrast, FLHFWs felt that it was ‘not good’ or ‘impossible’ to accomplish other activities whilst ChARM is counting. They thought it was important to ‘closely follow up’ the device and focus ‘only on the diagnosis activity’. Some FLHFWs thought ChARM saved time compared with the stopwatch whilst others felt it took more time in order to adjust the belt and wait for the ‘children to calm’. A few FLHFWs spoke of the burden placed on them as the only health worker trained to use the device; this caused ‘pressure’, ‘high responsibility’ and increased workload for them.

Confidence in use of and demand for ChARM

HEWs talked about increased flow of clients to the health post due to the availability of ChARM (Appendix S3, box 4). Suggested reasons for this were ‘better community acceptance’ compared to the previous device, caregiver ‘belief’ in the device and the community ‘currently has interest in the device’. Both HEWs and FLHFWs discussed how information sharing about the device in the community helped, with some mothers ‘disseminating information’ about the device. Both HEWs and FLHFWs expressed how the ChARM device had positively affected their (or the service) credibility in the community and increased their ‘motivation to work more in the community’.

HEWs talked a lot about learning to use the device, including how their ability and confidence had improved over time (Appendix S3, box 4). Many talked about initial difficulties operating the device, from ‘adjusting the belt’ on the abdomen to ‘selecting the age of the child’ and even identifying the ‘top and bottom of the device’. More HEWs than FLHFWs expressed initial fear that the device was too complicated, ‘difficult to operate’, or that it was ‘hard to operate all the steps’, and confusion and discomfort during the training as to how to ‘apply the steps in a real situation’. However, these HEWs were in the minority and all added that their impression changed and confidence grew after they used the device in practice.

There was considerable demand from HEWs and FLHFWs to use ChARM in future (Appendix S3, box 4). Amongst FLHFWs, this seemed to be connected to their perception of results being ‘always accurate’ and the device providing ‘tangible information about pneumonia diagnosis’. Many HEWs expressed relief in the perceived accurate diagnosis provided by the device, for example it helped ‘minimise our mistakes’, avoids ‘false diagnosis’ and ‘correctly identifies and classifies’ disease. HEWs desired to use the device in future because of the burden of pneumonia in their community and ‘number of affected children’.

DISCUSSION

The results show that in almost three‐quarters of consultations, HEWs adhered to WHO requirements to assess fast breathing and device manufacturer IFU.

A study of the competence of CHWs when using the ARI timer in Uganda found a high ability to assess (96%) and classify (85%) symptoms of pneumonia compared to a gold standard paediatrician immediately after an 8 day iCCM training,7 although the assessment steps observed differed from our study. The high proportion of correctly classified children according to ChARM parameters (98.8%) in this study suggests that HEWs can interpret the RR and the colour of the light given by ChARM. This is also reflected in the qualitative findings where HEWs and FLHFWs expressed appreciation for the support from the red and green light. Another study has shown that CHWs in Zambia using the ARI timer adhered to treatment requirements for 92% of children seen.20 Our findings show that almost all children were treated according to WHO requirements (98.2%) based on HEWs’ classification using ChARM. In terms of CHWs’ perceptions on the usability and acceptability of the ARI timer, a recent study found that the ARI timer was rated highly in regards to usability by CHWs, compared to other automated RR timers and pulse oximeters, but the inability to produce an automated result, the ticking sound and its perceived low accuracy reduced acceptability in some groups.21

HEWs’ adherence to requirements increased over time, perhaps due to an increase in HEWs’ confidence in using ChARM, as suggested in the semi‐structured interviews. On average, each health post recorded using ChARM seven times during the routine data collection, but without individual HEW data it is hard to attribute this increase to the HEWs’ practice using the device. Furthermore, HEWs may have practised using ChARM on other children and not recorded it in the patient register. Whilst this study did not collect empirical data about the rate of ChARM adoption by HEWs, further implementation studies could measure uptake of the device over a longer time period. It would be interesting to capture the characteristics of early and late adopters to understand how to encourage new users of ChARM to accept, adopt and sustain device use in everyday practice.22

Children in the youngest age group had the highest proportion of consultations completed according to requirements. The largest adherence disparity between age groups was for child positioning. HEWs found correct child positioning harder for the older children, perhaps because the youngest children were assessed whilst lying in the caregivers’ arms, making it easier to ensure the back was fully supported for the duration of the assessment.

On average it took above 3 minutes to get a reading with ChARM, including belt strapping and multiple attempts. The qualitative results suggest that the time taken was acceptable, but also that adjusting the belt was challenging which may have extended the consultation time, particularly for the youngest and often smallest children where it took longest to get a reading. Whilst no similar studies have measured the time CHWs take to complete an assessment with the ARI timer, one study evidenced that CHWs often lose count with the ARI timer and have to repeat the count, increasing overall assessment time.21 HEWs also discussed the ease at which they could perform other tasks whilst ChARM was counting, whereas FLHFWs were more cautious. It is possible that the views differed between groups because only one FLHFW per health centre was trained to use ChARM, potentially increasing workload for the trained FLHFW. Feasibility studies at the health facility could be done to explore how introducing ChARM changes workload and also explore whether an assistant health worker could use ChARM in the waiting room as a screening tool, to reduce consultation time. Furthermore, the assumption that FLHFWs are more able to use ChARM correctly than CHWs due to their higher education and training could be explored in a future usability study. It is plausible that in some cases, CHWs will be more competent as they have a lower patient load and fewer tasks to complete and therefore place more value to measuring RR and have more time to conduct these assessments.

HEWs felt that having ChARM available encouraged caregivers to visit the health post and caregivers were accepting of the device and would be comfortable for it to be used on their children again. Caregiver acceptance of ChARM is likely to be partly due to their inherent trust in provisions from the Ethiopian government and partly due to the attributes of the device itself. An implementation evaluation to understand the impact of ChARM on patient flow and care seeking behaviour over a longer study period would add to the evidence.

A strength of this study is that two research assistants observed the HEWs’ assessment independently. We also performed a sensitivity analysis to understand how disagreement between research assistants affected adherence to different consultation steps. Source documents were also used to verify the age selected, RR displayed and classification of ChARM by the HEW. Another strength is that adherence was measured after 2 months without any refresher training.

There are a number of limitations to the study. There was no ‘gold standard’ review of the HEWs’ management of the child, as done in similar studies using the ARI timer.7, 20 It was not within the scope of this study to assess the accuracy of the RR counts made with either ChARM or the ARI timer. Based on UNICEF Supply Division's technical and commercial evaluation, the study team accepted the manufacturer's claim of accuracy based on their CE mark. It proved challenging to recruit the youngest children because HEWs often conduct home‐based postnatal check‐ups and caregivers do not visit the health post due to cultural barriers that discourage women taking their newborn out of the home for the first 60 days of life. Whilst the research team silently observed the HEW and did not interfere with the consultations, it is possible that the results are influenced by the ‘Hawthorne effect’. There is potential for courtesy bias in the qualitative data; where some interviewees may have responded in ways they felt were appropriate rather than reflecting their own views. Limitations of the patient register review are that data are not collected on an individual HEW level, rather at the health post level, and response bias may also be present.

CONCLUSIONS

This is the first study to explore the usability and acceptability of a new automated RR device for classifying the symptoms of pneumonia by front‐line health workers in a low‐resource setting. The findings from this study support the rationale for further studies on performance, cost‐effectiveness and implementation of this and other respiratory rate devices to inform policy decisions in countries with a high burden of childhood pneumonia.

CONFLICT OF INTEREST

The study was sponsored by UNICEF in partnership and with funding from ‘la Caixa’ Foundation. The manufacturer of the device (Philips) did not contribute any funding for this work and did not participate in any data analysis or interpretation. The Malaria Consortium research team has subsequently received funding from the Philips Foundation to conduct future work on developing an improved respiratory rate reference standard. We declare there is no conflict of interest as these are discrete pieces of work.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.