Usability and acceptability of an automated respiratory rate counter to assess childhood pneumonia in Nepal.

AIM: Pneumonia is the leading cause of child death after the neonatal period, resulting from late care seeking and inappropriate treatment. Diagnosis involves counting respiratory rate (RR); however, RR counting remains challenging for health workers and miscounting, and misclassification of RR is common. We evaluated the usability of a new automated RR counter, the Philips Children's Respiratory Monitor (ChARM), to Female Community Health Volunteers (FCHVs), and its acceptability to FCHVs and caregivers in Nepal.
METHODS: A cross-sectional study was conducted in Jumla district, Nepal. About 133 FCHVs were observed between September and December 2018 when using ChARM during 517 sick child consultations, 264 after training and 253 after 2 months of routine use of ChARM. Acceptability of the ChARM was explored using semi-structured interviews.
RESULTS: FCHV adherence to guidelines after 2 months of using ChARM routinely was 52.8% (95% CI 46.6-58.9). The qualitative findings suggest that ChARM is acceptable to FCHVs and caregivers; however, capacity constraints such as older age and low literacy and impacted device usability were mentioned.
CONCLUSION: Further research on the performance, cost-effectiveness and implementation feasibility of this device is recommended, especially among low-literate CHWs.

acute respiratory infection

acute respiratory infection diagnostic aid

children's respiration monitor

community health worker

confidence interval

female community health volunteer

integrated community case management

integrated management of childhood illnesses

respiratory rate

standard deviation

United Nations Children's Fund

World Health Organisation

Diagnosis of pneumonia based on counting respiratory rate is challenging, and misdiagnosis and inappropriate treatment of children are common.

These results show that the usability of a new automated RR counter, the Philips ChARM device, among community health workers (CHWs) in Nepal was limited.

ChARM alone does not support low‐literate CHWs sufficiently in assessing and classifying sick children with symptoms of pneumonia.

INTRODUCTION

Acute respiratory infections (ARI), including pneumonia, are leading causes of death of 0‐ to 59‐month‐old children, with an estimated 0.9 million pneumonia related deaths in 2015.1 Over 75 per cent of these deaths occur in sub‐Saharan Africa and South East Asia.2 In Nepal, an estimated 15% of all deaths in children under five were due to pneumonia in 2015.3 These deaths result mostly from delayed presentation to appropriate healthcare providers and inappropriate treatment.4 Diagnosis of pneumonia as defined by the World Health Organisation (WHO) is based on presence of chest indrawing or fast breathing in children with cough and/or difficult breathing and severe pneumonia as any general danger sign or stridor in a calm child.5, 6 Current standard practice for front‐line health workers is to count respiratory rate (RR) using an ARI timer, but health worker (especially those with low numeracy skills) often finds it hard to keep track of the count. As a result, misclassification of RR remains high,7, 8, 9 leading to incorrect diagnosis and inappropriate treatment.10

Since 1988, Nepal has provided community health services through the Female Community Health Volunteer (FCHV), with over 52 000 FCHVs working to deliver the Community‐Based Integrated Management of Neonatal and Childhood Illness (CB‐IMNCI) programme.11 FCHVs are trained for 18 days on family planning, maternal newborn child health and nutrition issues; they are supervised by health workers from the nearest health facility.12, 13 While the FCHVs’ has contributed to Nepal's reduction in child mortality rate,14 the revised CB‐IMNCI protocol from 2014 no longer allows FHCVs to treat pneumonia in the community.13

UNICEF’s Acute Respiratory Infection Diagnostic Aid (ARIDA) project 15 was a response to a call for better pneumonia diagnostic aids 16, 17 and aims to identify automated RR counting aids for classifying fast breathing pneumonia, for use by front‐line health workers in resource‐limited settings. Philips responded to the UNICEF request for proposals and developed the Children's Respiration Monitor (ChARM) which uses an accelerometer to measure the RR in children 0 to 59 months and classifies the RR according to the IMCI/iCCM guidelines.18 ChARM is intended to be used by health workers at all levels in low resource settings and is strapped around the belly of the child using an elastic belt.

Despite the attractiveness of this new device, evaluations of previous technology introductions among low literate health workers have shown mixed results.19, 20 A health workers’ intention to adopt a new innovation and adhere to user guidelines can be affected by several facets of acceptability: affective attitude, burden, intervention coherence, perceived effectiveness and self‐efficacy.21 Acceptability, combined with skills, abilities and environmental factors such as child and caregiver behaviour, availability of commodities, context and setting can affect use over time. Knowledge of how and why innovations are used appropriately (or not) to produce their observed effects can inform design improvements and changes in implementation procedures. The aim of our study was therefore to evaluate the usability, defined as the adherence to required WHO case management as well as the device manufacturer instructions for use (IFU) for the new automated RR device (ChARM) among FCHVs, and its acceptability to FCHVs and caregivers based on the acceptability facets.

METHODS

Study design

This is a cross‐sectional study of FCHVs with a convenience sample of children, using mixed method design. Prior to starting the first quantitative data collection, a training of trainers (ToT) and research teams and a cascade training for FCHVs were conducted. The methods are described in detail elsewhere.22

Study setting

The study was conducted in community settings in the three municipalities Tatopani, Tila and Chandannath in the mountain district of Jumla in the Karnali province, Nepal between September and December 2018. This area was selected because of the high burden of pneumonia, sufficient number of FCHVs with training in CB‐IMNCI, geographical remoteness yet with logistical and operational feasibility for data collection and quality assurance. All 179 FCHVs who operated in the selected municipalities had been trained in CB‐IMCI, but based on observations by the research team, many FCHVs never counted RR before the study because of lack of ability and lack of timer.

Sample size

The study was powered for the primary outcome, that is to measure the proportion of under‐five child consultations where FCHVs using ChARM adhered to required WHO case management guidelines and device manufacturer IFU after 2 months. Using the sample size formula for a prevalence study with a 95% level of confidence and 7.5% precision, a sample size of n = 264 child assessments was required to estimate the true proportion of the outcome, assuming that the proportion of FCHVs completing all the steps correctly is 71% (75% conducting the RR steps correctly and 95% of these classifying the RR correctly), a design effect of 1.7 to account for clustering at FCHV level between observation 1 and 2, and a dropout rate of 10%. Thus, 132 FCHVs would need to be observed completing two sick children assessments twice (totalling 528 sick child assessments), once after training and once after 2 months.

Data collection methods and sampling

Data sources included structured observations of FCHVs, routine FCHV ward registers and semi structured interviews with FCHVs and caregivers. All 179 FCHVs in the three municipalities were trained for the study. For the semi‐structured interviews, 15 FCHVs were purposefully selected. Included FCHVs were provided with refresher training on CB‐IMNCI and on how to use ChARM. They completed a post‐training assessment to verify their ability to use ChARM as part of a CB‐IMNCI assessment using a 12 question test. The pass mark was 75% to ensure they had reached a sufficient level to be assessed in the study. All data collection procedures and tools were pre‐tested before data collection started.

FCHVs were observed by trained research assistants conducting child consultations with ChARM. Children under five who were bought for care to the FCHVs in the study were enrolled prospectively if the child was described as sick (young infants), had cough and/or difficulty breathing (2‐59 month old) and where the guardian gave consent. Exclusion criteria for children were those with IMCI general danger signs or referral signs for severe disease,5 parent or guardian's age <16 years, no guardian consent or device manufacturer safety exclusion criteria.23 Sixteen trained research assistants (RAs), who worked in pairs, screened the child to ascertain eligibility and guardian's consent. Due to expected low patient attendance, research assistants were asked to call the selected FCHVs two days before the assessment visit to ask them to mobilise children for the day of the assessment. Immediately following training, research assistants observed 133 of the 179 FCHVs were purposively selected based on their location (ie being accessible during the winter season when snow often blocks road access) using ChARM during two child consultations (observation 1). FCHVs completed 9 steps comprising CB‐IMNCI guidelines on how to prepare a child for fast breathing assessment and how to count and classify RR, plus device manufacturer IFU on how to position the child and ChARM to get a valid RR reading (Table A1). Up to three attempts to obtain a RR classification with ChARM were allowed; if these were unsuccessful, the FCHV could reverted to using standard practice (ARI timer). FCHVs ability to refer children was also recorded. After training, FCHVs used ChARM routinely for 2 months before being observed a second time (observation 2). On completion of the assessment, the FCHV explained the classification to the caregiver and gave them referral or home care advice as appropriate.

Research assistants, who all had research experience and qualifications, silently observed the FCHVs conducting the consultation and entered their data independently in tablets (Samsung Tab 3) using CommCare recording forms (version 2.38.1, Dimagi). Data were synced daily to a protected cloud server, and the data manager validated and cleaned the data. Research assistants photographed the RR result, classification and age displayed on ChARM to provide source documents for verification purposes. Once the evaluation was completed, the RAs gave feedback to the FCHV if they had observed any incorrect actions and corrected the advice given to the caregiver.

Between the first and second observations, FCHVs were encouraged to use ChARM during routine practice, but could revert to standard practice if they preferred to. For each child, they assessed for respiratory signs and symptoms, and they were instructed to record which device they used in their patient register using coloured stickers.

A total of 15 FCHVs were purposefully sampled to maximise variation for location, age, years of experience and caste. These FCHVs asked one caregiver each to participate in the interviews. The first caregiver who was asked and agreed to participate was selected for the interviews. The first sets of transcripts from each pair of qualitative research assistants were reviewed for quality assurance, and feedback was provided. All semi‐structured interviews were conducted in Nepali, audio recorded, and subsequently translated and transcribed to English. Each English transcript was reviewed by HERD International before being finalised.

Data management

Malaria Consortium (UK) and the HERD International conducted quality assurance (QA) visits to the research site during data collection. Malaria Consortium created a QA form template which was completed by team members when shadowing the research assistants (RAs). All data collected by RAs from the FCHVs assessments were checked and verified by the Malaria Consortium research team.

Data analysis

Descriptive results are summarised as percentages, means and standard deviations, whereas categorial data are presented as percentages with 95% confidence intervals (CIs). For the main outcomes, the most conservative estimates were used, that is if two research assistants entered conflicting data on how the FCHV performed a stage in the assessment, the one that recorded an inconsistency/error for that step was used over the one who recorded that the step was performed correctly. Logistic regression was performed to assess whether exposure to refresher training, time since qualification as a FCHV, and literacy level was associated with ability to correctly adhere to the assessment guidelines at both observation points. A sensitivity analysis using the least conservative estimates is presented for the primary outcome. McNemar's test for matched pairs was used to analyse the difference in the proportion of FCHVs who were able to complete all the required WHO case management guideline steps correctly compared with the steps in the manufacturer IFU of ChARM. The mean time taken to complete the full assessment was recorded, starting from when the FCHV strapped on ChARM till when it displayed a RR reading, inclusive of multiple attempts. The number of children who were assessed for cough and/or difficult breathing by FCHV with ChARM or standard practice during routine care is presented. All quantitative data were analysed using Stata version 13 (Stata‐Corp LP) by CW and MdC. Qualitative data were managed using MAXQDA (VERBI GmbH) and analysed by thematic analysis. A coding frame for the FCHV and caregiver interviews was developed by HS.

Ethical approval

The Nepal Health Research Council (NHRC) gave approval on July 2, 2018 (ref 2334), and the Liverpool School of Tropical Medicine Research Ethics Committee (ref 18‐026) gave favourable opinion to the protocol on the July 10, 2018. Research assistants obtained written consent for observations and interviews from each FCHV and from each caregiver whose child was assessed by a FCHV during an observation.

RESULTS

Participant characteristics

The 133 FCHVs, who were included in the study were on average 42 years old (SD 10.4), had an average of 15.5 (SD 7.3) years’ experience as a FCHV, 71.4% had received CB‐IMNCI refresher training within 3 years, and about half (54.9%) were literate (could read and write simple language). Fifteen FCHVs participated in semi‐structured interviews; they had an average of 15.1 years (SD = 9.7) experience as a FCHVs; the mean age was 38.2 years (SD = 10.2), and 10 out of 15 (66.7%) were literate. Fifteen caregivers, whose mean age was 26.9 years (SD = 12.0) and of whom 50% were literate, participated in the semi‐structured interviews.

A total of 269 children were enrolled for observation 1 and 257 children for observation 2 (Figure 1). Of these, 264 and 253 ChARM assessments were completed, respectively; five were not completed because the child not being calm enough, and for 4, there was incomplete collection of data. The majority of children were above 2 months old.

Figure 1

Participant study flow for observations 1 (after training) and 2 (after 2 mo routine ChARM use)

FCHVs’ adherence to assessment guidelines (usability)

Most of the sick child assessments were completed with ChARM on the first attempt (Figure 1). The proportion of child assessments completed on attempt one by age group was 85.7% (0‐<2 months), 84.2% (2‐<12 months) and 80.1% (12‐59 months). The most common reasons for unsuccessful attempts were that the ChARM device displayed the error message ‘‐‐‐’ or that the child was not calm or was moving. A total of two (0.8%) and one (0.4%) assessments could not be completed with ChARM for observations 1 and 2, respectively.

The proportion of FCHVs using ChARM who adhered to the required WHO case management guidelines and device manufacturer IFU using ChARM for a CB‐IMNCI sick child assessment was 52.8% (95% CI 46.6‐58.9), an increase of 2.8% from the first observation after training (Table 1). In the sensitivity analysis (Table 2), 57.1% of FCHVs using ChARM adhered to all steps of the required WHO case management and device manufacturer IFU; only 4.3% higher than the conservative estimate indicating that the two observers agreed on most of the observations. Hence, for following analysis, we have used the most conservative estimate. Regression analysis showed no significant association between adherence to all steps in the assessment guidelines and number of years the FCHV had been qualified (Odds ratio (OR) 1.0; 95% CI 0.97‐1.04), time since last refresher training (OR 1.0; 95% CI 0.98‐1.02) or FCHV literacy (OR 0.59; 95% CI 0.36‐0.99), nor was there a significant difference in adherence between observations 1 and 2 (OR 1.13; 95% CI 0.79‐1.62).

Table 1

Number and proportion of child consultation steps correctly performed by FCHV with ChARM after training (observation 1) and after 2 mo of routine use (observation 2)

No.	Consultation step	Observation 1 (after training)			Observation 2 (after 2 mo)
		n	%c	95% CI	n	%c	95% CI
1	Correct child positiona	231	87.5	82.9‐91.0	215	85.0	80.0‐88.9
2	Correct device positiona	255	96.6	93.6‐98.2	238	94.1	90.4‐96.4
3	Correct belt positiona	261	98.9	96.5‐99.6	246	97.2	94.2‐98.7
4	Correct age groupb	236	89.4	85.0‐92.6	225	88.9	84.4‐92.3
5	Child calm before assessmenta	203	76.9	71.4‐81.6	213	84.2	79.1‐88.2
6	Child not eating/feeding during assessmenta	246	93.2	89.4‐95.7	231	91.3	87.1‐94.2
7	Child calm during assessmenta	207	78.4	73.0‐83.0	219	86.6	81.7‐90.3
1‐7	Cumulative assessment (steps 1‐7)	132	50.0	44.0‐56.0	133	52.6	46.4‐58.7
8	Correct classification using ChARM (yes/no)b	254	96.9	94.0‐98.5	244	96.8	93.7‐98.4
1‐8	Correct assessment and classification (steps 1‐8)—primary outcome	131	50.0	43.9‐56.1	133	52.8	46.6‐58.9
9	Correct referral using ChARM RR classification and FCHV assessment of other symptoms	54	83.1	71.6‐90.5	23	69.7	51.2‐83.4
1‐3	Manufacturer guidelines correctly performed (stages 1‐3)a	219	83.0	77.9‐87.0	200	79.1	73.6‐83.7
4‐8	WHO case management guidelines correctly performed (stages 4‐8)b	160	60.6	54.5‐66.4	168	66.4	60.3‐72.0

Based on two research assistants observing the FCHV. Where two research assistants disagreed, the most conservative estimate was used.

Based on comparison of the age group recorded on the screening checklist and the photograph of the ChARM with result displayed.

Step nos. 1‐7, 1‐3, 4‐8: N = 264 (observation 1) and 253 (observation 2)—children whose consultation started. Step nos. 8 and 1‐8: N = 262 (observation 1) and N = 252 (observation 2)—children with RR classification with ChARM. Step no. 9: N = 65 (observation 1) and 33 (observation 2)—children with fast breathing and/or a referral sign.

Table 2

Number and proportion of child consultation steps correctly performed by FCHV with ChARM after training (observation 1) and after 2 mo of routine use (observation 2)—sensitivity analysis

No.	Consultation step	Observation 1 (after training)			Observation 2 (after 2 mo)
		n	%c	95% CI	n	%c	95% CI
1	Correct child positiona	243	92.0	88.1‐94.8	221	87.4	82.6‐90.9
2	Correct device positiona	255	96.6	93.6‐98.2	242	95.7	92.3‐97.6
3	Correct belt positiona	262	99.2	97.0‐99.8	247	97.6	94.8‐98.9
4	Correct age groupb	n/ad			n/ad
5	Child calm before assessmenta	221	83.7	78.7‐87.7	224	88.5	84.0‐91.9
6	Child not eating/feeding during assessmenta	248	93.9	90.3‐96.3	233	92.1	88.0‐94.9
7	Child calm during assessmenta	222	84.1	79.1‐88.0	224	88.5	84.0‐91.9
1‐7	Cumulative assessment (steps 1‐7)	156	59.1	53.0‐64.9	144	56.9	50.7‐62.9
8	Correct classification using ChARM (yes/no)b	n/ad			n/ad
1‐8	Correct assessment and classification (steps 1‐8)—primary outcome	155	59.2	53.1‐65.0	144	57.1	50.9‐63.1
1‐3	Manufacturer guidelines correctly performed (stages 1‐3)a	232	87.9	83.3‐91.3	211	83.4	78.3‐87.5
4‐8	WHO case management guidelines correctly performed (stages 4‐8)b	179	67.8	61.9‐73.2	176	69.6	63.6‐75.0

Based on two research assistants observing the FCHV. Where two research assistants disagreed, the less conservative estimate was used.

Based on comparison of the age group recorded on the screening checklist and the photograph of the ChARM with result displayed.

Step nos. 1‐7, 1‐3, 4‐8: N = 264 (observation 1) and 253 (observation 2)—children whose consultation started. Step nos. 8 and 1‐8: N = 262 (observation 1) and N = 252 (observation 2)—children with RR classification with ChARM. Step no. 9: N = 65 (observation 1) and 33 (observation 2)—children with fast breathing and/or a referral sign.

A source document was available to verify these steps, negating a need for sensitivity analysis.

Most of the FCHVs attached the ChARM belt to the child correctly, and more than 90% adherence was achieved for correctly positioning the device and ensuring that the child was not feeding during the assessment. The most challenging step for FCHVs using ChARM was the correct child positioning and ensuring that the child was calm before the assessment. About 69.7% of FCHVs made the correct referral decision based on the ChARM RR classification and their assessment of other symptoms.

McNemar's test revealed that the percentage of the FCHVs that correctly followed manufacturer IFU (79.0%) significantly differed from the percentage who correctly followed the required WHO case management guidelines (66.7%), a difference of 12.3% (95% CI 4.2‐20.4; P = .002; Table 1). The proportion of FCHVs who conducted all eight steps correctly was significantly higher when they assessed 12‐ to 59‐month‐old children compared with those 2‐<12 month old (Table 3). The largest variation in assessment steps between age group was seen for ‘child calm during the assessment’. A significantly larger proportion of FCHVs completed all eight steps correctly for normal breathing children compared with fast breathing children (57.1% vs 24.2%; P = .0004). The mean performance time, that is the time from when the device started to be strapped to the child, to when the FCHV got a RR reading was 04:26 minutes (range 01:15‐25:26; SD = 03:00).

Table 3

Number and proportion of child evaluation steps correctly performed by FCHV with ChARM after 2 mo of routine use, by child age group and breathing status

	Assessment step	Observation 2 (0‐<2 mo) N = 7			Observation 2 (2‐<12 mo) N = 77			Observation 2 (12‐59 mo) N = 168			Observation 2 (fast‐breathers) N = 33			Observation 2 (normal breathers) N = 219
		n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI
1	Correct child position	6	85.7	25.7‐99.0	69	89.6	80.3‐94.8	142	84.0	76.1‐87.6	29	87.9	70.6‐95.6	186	84.5	79.1‐88.8
2	Correct device position	7	100.0	59.0‐1.0	73	94.8	86.7‐98.1	158	93.5	88.6‐96.4	31	93.9	77.4‐98.6	207	94.1	90.1‐96.6
3	Correct belt position	7	100.0	59.0‐1.0	75	97.4	89.9‐99.4	164	97.0	93.0‐98.8	32	97.0	79.7‐99.6	214	97.3	94.0‐98.8
4	Correct age group	5	71.4	21.5‐95.8	65	84.4	74.3‐91.0	155	91.7	86.4‐95.1	23	69.7	51.2‐83.4	202	91.8	87.3‐94.8
5	Child calm before assessment	6	85.7	26.7‐99.0	62	80.5	69.9‐88.0	145	85.8	79.6‐90.3	20	60.6	42.4‐76.3	193	87.7	82.6‐91.5
6	Child not eating/feeding during assessment	7	100.0	59.0‐1.0	68	88.3	78.8‐93.9	156	92.3	87.1‐95.5	29	87.9	70.6‐95.6	202	91.8	87.3‐94.8
7	Child calm during assessment	5	71.4	21.5‐95.8	61	79.2	68.5‐87.0	153	90.5	85.0‐94.1	19	57.6	39.6‐73.8	200	90.9	86.3‐94.1
8	FCHV classified the child's breathing status correctly using ChARM	6	85.7	25.7‐99.0	74	96.1	88.3‐98.8	164	97.6	93.8‐99.1	26	78.8	60.6‐90.0	218	99.5	96.8‐99.9
1‐8	Correct assessment and classification (steps 1‐8)	5	71.4	21.5‐95.8	33	42.9	32.1‐54.3	95	56.5	48.9‐63.9	8	24.2	12.1‐42.6	125	57.1	50.4‐63.5

In the period (median = 76 days) between observations 1 and 2, a total of 107 FCHVs reported data correctly in their ward register using stickers; a total of 571 child pneumonia assessments were documented. Of these, 430 (75.3%) were completed with ChARM, 33 (7.1%) were completed with their standard practice device, and 108 (23.3%) were completed with an unknown device.

Acceptability of ChARM

We identified three main themes related to ChARM acceptability among FCHVs and caregivers.

Constraints impacting device use

Many FCHVs were initially concerned about their ability to use ChARM, and several mentioned that they found it ‘really difficult’ to learn how to use it, that they felt ‘uncomfortable during the training’ or they were ‘intimidated’ by the device (Box 1)

Capacity constraints impacted on device use

‘When we were receiving the training, we found it really difficult. During the starting of the program, later, before 2 days, it was difficult to explain us what it was on the first day. Trainers had to work really hard. So, first all the FCHVs thought what is this, “Timer” was easier than this. We felt bad too’. (FCHV 04, Lamra)

‘We would find it uncomfortable during the training in the beginning. We had not learnt about it before and did not know how to measure… We had Sirs and Mams who taught us then. They used to tell me about the things that we were supposed to do. Some of them might have felt difficult as well. I found it to be all right’. (FCHV 05, Chandannath)

‘A number of women could not sleep at night wondering these things. They were not able to learn it despite learning it the whole night and day yesterday. We met a few of them who told us that they could not sleep all night because they were unable to learn using it’. (FCHV 05, Chandannath)

‘During the initial period of training, all the FCHVs were quite intimidated by the ChARM device. None of us have ever seen that device so, all of us were quite curious about the device…some of us were telling that the device is similar to the mobile phone but none of us seen the device till that time. In addition to the curiosity, we were concerned that how we would be able to use the device on our own’. (FCHV 12, Tatopani)

‘As, I have been telling that we might have forgotten the learning and made errors, so, training would help in recalling that information. Every time, illiterate person ends up like this. If, I was educated then, I would have answered your question fluently…’ (FCHV 10, Chandannanth)

‘There is no surety that acquiring training can lead to adequate comprehension of all the information. We have to engage ourselves in family, children, animal husbandry. So, we are not able to memorise all the information learned during the training…’ (FCHV 09, Chandannanth

‘First of all, sir taught us. Those who are educated they learned quickly. Those who are not educated they might have had some problem’. (FCHV 02, Kudari)

Device attributes supported illiterate FCHVs

‘For those who cannot read, it will be tough to read it. For those of us who are educated it is easy. Once I segregate the child age… three groups are displayed…’ (FCHV 02, Kudari)

‘Among them, we do not know how to read the numbers. Apart from reading the numbers, I sometimes get confused when determining the age group for the child…’ (FCHV 05, Chandannath)

‘And even though FCHVs is unable to recognise the numeric values displayed by the device… we can easily interpret the result of the monitoring through the help of light signals that get flashed from the device’. (FCHV 07, Chandannath)

‘Since, I have not acquired any formal education. So, I am unable to observe the numeric values displayed on the ChARM device’. (FCHV 09, Chandannath)

‘It is the same even if the letters are in Nepali. They should be able to read it in English or Nepali. Those who cannot read they should categorise the condition of the child through blue (red) or green lights. Those of us who can read can give the result on the basis of number as well as on the basis of the light glowing in the indicator’. (FCHV 08, Chandannath)

‘Another one is the result shown in red and yellow. Oh, it is green and red. Because the result is given in green and red, it is better for uneducated persons as well. Like us…’ (FCHV 04, Lamra)

‘Yes. Red translates to danger. Isn’t it?…Green translate to healthy state. And red translate to danger. Isn’t it? Actually, we were told that the red light signifies danger. I have been telling the same thing that was instructed during the training period’. (FCHV 09, Chandannath)

‘By that, I mean that red…Whether pneumonia is present or not…. If pneumonia is present then, red [Referring to red light signal flashed by the ChARM device] gets displayed and we will know about the presence of pneumonia. And green gets flashed in the absence of pneumonia’. (FCHV 10, Chandannath)

Initial hesitance to use the device

‘Yes, I was anxious. However, I got to know that all the procedural steps were same and weren’t altered through the people [Referring to quantitative team members]. So, getting anxious was worthless, right? We learned about the ChARM device however, sometimes we forgot the procedural steps and its quite usual for the elderly people like us’. (FCHV 09, Chandannath)

‘I was frightened when I saw ChARM thinking, “what might happen? How it [Referring to ChARM device] would be like?” Initially, we thought, “Will we be able to use that or not? Will we make mistakes or not?”. However, later along with the practice, it has been easier. It was quite obvious to get frightened while seeing the something new. Right?’ (FCHV 10, Chandannath)

‘It is quite obvious to encounter difficulties at the initial stage of learning phase. I used to get intimidated just thinking about using the device. But trainers taught us the procedures that needs to be followed while monitoring the child with ChARM device’. (FCHV 12, Tatopani)

‘We did not know how to use it, or what to do with this so we were uncomfortable’. (FCHV 04, Lamra)

‘How are we supposed to measure…? I wondered how we are supposed to measure. It was supposed to be put on the [child’s] belly. How are we supposed to measure it placing it on the belly?… A number of women could not sleep at night wondering these things’. (FCHV 05, Chandannath)

‘Charm device is better than the Timer. It is easier to take respiration rate of children using it. With timer what used to happen was we would not know when they time used to gets over. While the machine beeps, our thoughts wander and we would miss the count’. (FCHV 02, Kudari)

‘What it was earlier is, first one is those FCHVs, who are uneducated and old would not be able to count and another one is we had to count the breaths even when the child is crying or screaming. And sometimes we tend to forget the count. We are supposed to count either the breath‐ins or the breath‐outs, this made it more difficult. But with this, after we have put it on the child it counts respiration on its own’. (FCHV 04, Lamra)

‘While monitoring the child with timer, we often missed the count so, we have to make repetitive attempt in order obtain accurate result. However, with Charm device, the result can be obtained with single attempt. So, the Charm device is faster than the timer…. I don’t mean to say that the timer is inconvenient to use. We find timer a bit exhausting because of lacking in our skills regarding the counting of the respiratory rate of the child’. (FCHV 07, Chandannath)

However, for most, this initial reaction was short‐lived, and FCHVs said they got more comfortable using it with time and practice. The majority of FCHVs eventually stated that they preferred ChARM over the previously used timer, mainly because it avoids miscounting or forgetting the count with the timer, especially given that illiterate FCHVs found it difficult to count beyond 20.

Many FCHVs associated their ability to use the device with age or education level, admitting it was difficult for those who were ‘uneducated’ to read and understand the contents; however, they acknowledged that the pictures in the job aid were ‘easy to understand’. Literacy level also impacted on FCHVs ability to read the results displayed on the device, especially for those who do not know Arabic numeral system. However, the red and green indicators were found helpful, and caregivers mentioned how the FCHV would show the red/green light to communicate the result of the assessment. Caregivers were unconcerned about the use of the device on their child; many said they were curious when the device was attached to their child. They were more worried about their child being ill, or if pneumonia ‘would show up’ in the assessment.

Perceived effectiveness of the device

FCHVs were of the view that caregivers trusted them and the services provided and that the new device had improved that trust and made it more likely that caregivers bring sick children for check‐ups (Box 2)

Low service demand meant the device was not often used

‘Rather than bringing children to us, people here prefer taking them to Omgard (Name of place), for checkup. People here think medicine bought is better. [Laughing] They take their child there but when they are asked to come to visit FCHV, then they do come’. (FCHV 08, Chandannath)

‘Children… mothers too take the children elsewhere if the child is too sick. That many have not come to me. They might be coming now… in the winter… there are not much who are sick during the summer’. (FCHV 08, Chandannath)

‘But, till the date, I have never monitored using “timer”. I want to be honest with you “sister”…children haven’t come to me and I have never monitored any of them with ChARM device as well. So, we only monitor the children during the time of this type of visits. Sometimes, people make visit to FCHVs to receive “iron” tablet during pregnancy… However, children don’t come to us. I don’t want to lie, so, I want to tell why children don’t come to us. There are so many “hospital” at here, so, women often tell that, “we should visit FCHVs, we will take tour children to hospital”. Sometimes, under and over 5 years old children seek services such as demandings for ORS [Oral Rehydration Therapy], “zinc” tablets and “cetamol” [Referring to paracetamol], if they experience diarrhea. But monitoring with “timer” and “ChARM”, we haven’t monitored much children’. [Nodding her head] (FCHV, 10, Chandannath)

‘…However, we receive some children occasionally, as like today… Mostly, children are taken to hospital… Yes, children are very low in numbers in this village and they aren’t brought to us. Mostly, people take their children to hospital. Sometimes, visit to mothers group… ’ (FCHV 10, Chandannath)

‘They say that it would be better if we could also provide them medicines along with this [CHARM device]. They say that we will send them to the hospital after conducting their checkup’. (FCHV 05, Chandannath)

‘When measuring with the timer… I used to give the children cotrim before. So, people have said that it would be better if we would again start providing them cotrim to take home instead of taking the children all the way to the hospital… There have been one or two guardians who have asked me questions about that too. They have said that we used to provide cotrim when conducting checkups of the children using the timer in the past but now we leave the children after using this [CHARM] device. They have said that it used to help against the pneumonia. They ask us where the cotrim is now. We do not distribute it now. We can only give it if we get it from…’ [the health post] (FCHV 05, Chandannath)

‘We were allowed to give ‘Cotrim’ earlier to those who had pneumonia but we are not allowed to do that now. [Distribute medicine] So if we send the child to the hospital and they are told that the child is normal, then the child’s parents complained about it’. (FCHV 06, Jumla)

Referral and communicating results to caregivers

‘…But with ChARM on the other hand this is not the case is gives the respiration rate confidently and we can thus show it to the parents and tell them about their children’s condition’. (FCHV 02, Kudari)

‘If the red light glows on the device, then we tell the parents that their child’s respiration rate is high so not to keep the child home, instead I tell them to take the child to hospital as soon as possible. As the child will not be able to drink in any liquid or even mother’s milk during this condition, we insist them to take the child to hospital’. (FCHV 06, Jumla)

‘If the red light glows, then we have to tell the parents that their child has pneumonia, and should be taken to hospital. We have to counsel them’. (FCHV 08, Chandannath)

‘We do not tell the parents that their child has pneumonia, as they get scared. Now, we know…If we say this to the parents then they get scared. They start worrying about admitting the child, thinking about their financial conditions…’ (FCHV 02, Kudari)

‘We cannot tell people that they [the children] are suffering from pneumonia because they will be shocked. So, we suggest them to take their children to the hospital as we found that their children have high respiration rate. That is why it is easier for us’. (FCHV 05, Chandannath)

‘We advise them to take the children to the hospital when the yellow lights are lit because it means that their respiration rate is higher. We console them that there is nothing that they have to be afraid about and they simply need to take the child to the hospital. They get afraid if we tell them that their child is suffering from pneumonia because they think that pneumonia is a very dangerous disease. So we tell them that their child has to be taken to the hospital because they have higher respiration rates’. (FCHV 05, Chandannath)

Balancing FCHV role and household responsibilities

‘I find this [Referring to responsibilities of FCHVs] very tedious. I am required to skip my various personal and household activities quite often. You know that I have to collect woods from the forest for cooking. Right? But, I am investing my time here communicating with you…in addition, the government have delegated us with so much work load, which surpasses the work burden of office attendant. However, the added workload comes without any incentives. So, sometimes I feel like leaving my service after reaching the age of 60’. (FCHV 07, Chandannath)

‘ChARM device is added work load [Laughs]…Yes. ChARM device was introduced along with that work load is added up [Laughing]…Recently, I have been preoccupied with the thought training is synonymous to additional work load for us [Laughing]’. (FCHV 05, Chandannath)

‘What other work we might have… I had already left other work. I leave behind every kind of work when they [mother] bring children. We should perform our responsibility first. Children are our responsibility, Sir’. (FCHV 03, Lamra)

‘Now, if they were to bring in the child early in the morning then I have to cook a meal [for my family]. Right now… it is the not the season of work. There is a lot of work when we have it. We have to go collect the firewood. We have to go to the fields also. We have to be willing to conduct the children’s checkup using the ChARM device even if it means that we have to stop our work because we have the responsibility of working as a FCHV. We cannot make excuses. Even though there are some effects on our works, we stop the work that we are doing to perform our responsibility of conducting the children’s checkup’. (FCHV 05, Chandannath)

‘If the child moves, it shows red and something else. That’s why we should stay with mother. And mothers… Some of them would breastfeed their child thinking they would cry. That’s why we should stay with them’. (FCHV 03, Lamra)

‘No, we do not do that. [Moving head in denial] after we are done with household works… After turning the ChARM device off we move on to doing other activities. We do not work when we are using ChARM on children’. (FCHV 06, Jumla)

‘While monitoring the child with ChARM device, we have to observe whether the device is flashing red or green signal as well as we should be mindful regarding the respiratory rate of the children. So, we should never divert our attention from the ChARM device’. (FCHV 07, Chandannath)

However, the low numbers of children being brought to them for care weres a concern for many FCHVs.

Inability to provide treatment to children seemed to influence how often FCHVs use the device. Across all districts, FCHVs were concerned; they had to send mothers away ‘empty handed’ even though they had been assessed with the device and that it was difficult for caregivers to have to take children ‘all the way to the hospital’ to receive ‘cotrim’ [Cotrimoxazole]. Caregivers frequently asked FCHVs why they cannot provide ‘cotrim’ as they used to, and there was concern that caregivers linked the lack of treatment with the introduction of the ChARM device. A few FCHVs described that with ChARM, they could now more confidently ‘send the child to the health facility’, if the respiration rate was high.

Burden of using the ChARM device

Across all districts, FCHVs talked about balancing their community health responsibilities with personal household activities, and some described that it was ‘stressful’ to also serve the community. For others, their responsibility as an FCHV always came ‘first’. Most FCHVs said they were unable to carry out other work when using the ChARM device to assess children; many thought it was not right to divert attention away from the assessment until assessment was complete. A minority said they were able to do other minor tasks such as prepare food while the device was monitoring the child ( Box 3).

DISCUSSION

The findings show that among FCHVs adherence to the eight assessment and classification steps of a child with cough and/or difficult breathing using ChARM was low (52.8%). This adherence was lower than what was observed in a sister study using a similar study protocol, where ChARM was used by Health Extension Workers (HEWs) in Ethiopia.24 In a study of Ugandan CHWs using the ARI timer for manual RR counting, different assessment criteria were used to assess adherence, making it difficult to compare the results. However, the Ugandan CHWs’ were found to be challenged with the RR classification step of the WHO case management algorithm, and the authors recommend that tools that can enable CHWs to make a correct classification of the RR would minimise misdiagnosis.8

Overall, FCHVs had significantly lower adherence to the WHO case management steps than the device manufacturer IFU, indicating that the overall low adherence was not related to the operation of the device itself, but to FCHVs’ inability to correctly position the child and ensuring that the child was calm. While several FCHVs expressed concerns in the qualitative interviews about their inability to operate ChARM, they also stated getting more comfortable over time. FCHV age and illiteracy was seen by many as constraints to correct device use; yet the attributes of the device that supports classification through a red or green light was seen as supportive, especially for those who cannot read or count. While we tested the association between literacy and adherence scores, we did not find any significant difference between those who can and cannot read Nepali, and even though the device is only available with English instructions and Arabic numbers on the display, it uses important signs to help users select appropriate age group, to indicate when an error has occurred or to display the results (red/green light). The low adherence of Nepalese FCHVs to the WHO case management guidelines was primarily a result of a failure to ensure that the child was calm before and during the assessment. A previous study has shown that there is a small risk that ChARM attachment creates fluctuations in the child's RR that lead to misclassification 25; hence, it is worth further exploring if ChARM device attachment effects children's behaviour and causes them to cry during the assessment. Other contextual factors, such as the low level of literacy, numeracy and training among FCHVs, as well as the remote location of the study where children were not used to strangers and easily daunted by their different appearance and behaviour, likely explained these findings. A study of the performance of Ugandan CHWs found that receiving feedback from health facilities and having drugs available increased overall CHW performance.26 FCHVs in Nepal were neither well supported by the health facility staff nor were they provided with medicines to treat pneumonia, which could partially explain their poor performance of WHO case management guidelines.

FCHVs saw relatively few children during routine practice: just over five children in 76 days on average. This might be explained by the two festivals that occurred during the study, when FCHVs often travelled to visit relatives, and because of seasonal migrations during cold months. FCHVs also mentioned the low patient load and expressed concerns that they did not get to practice using the device as much as they would have liked. The government sees FCHVs as an important bridge between families and communities to health facilities.13 Yet, the recent policy change which disallows FCHVs from providing treatment for pneumonia has made it difficult for FCHVs to convey the diagnostic results of ChARM to the caregivers, as a ‘pneumonia diagnosis’ resulted in caregivers demanding antibiotics. This policy change, along with programmatic fragmentations, seems to also be causing FCHV confusion on recording and reporting tools, a declining sense of motivation, has been claimed to pose a major risk to children living in hard‐to‐reach and disadvantaged communities.11 It is therefore crucial that factors such as CHW roles and mandate are considered before considering introducing a new technological innovation, such as ChARM, into the health system. Implementation research is recommended to understand how the innovation itself, communication channels, time and the social system in which it is meant to operate affects its uptake and spread.27 Evaluations are also needed on the performance and cost‐effectiveness of ChARM and other respiratory rate devices to inform policy decisions in countries with a high burden of childhood pneumonia.

A limitation of this study is the lack of a ‘gold standard’ review of the FCHVs management of the child, as done in similar studies.8, 10 The silent observations of the FCHV by the research team could have caused some ‘Hawthorn effect’. Some systematic bias could be associated with the convenience sampling of the FCHVs, given that FCHVs residing in areas that were inaccessible in the winter were excluded, and the results may therefore not be generalisable to the FCHV population in the most remote areas. Courtesy bias is possible in the qualitative data, where some interviewees may have responded in ways they felt were appropriate rather than reflecting their own views. The caregiver interviews also suffered from a distinct lack of probing, many leading questions and few open questions asked, hence the qualitative findings need to be interpreted with caution.

CONCLUSION

The findings from this study support the rationale for further studies on performance, cost‐effectiveness and implementation of this and other respiratory rate devices to inform policy decisions in countries with a high burden of childhood pneumonia, in particular in the hands of low‐literate CHWs.

CONFLICT OF INTEREST

We declare there is no conflict of interest as these are discrete pieces of work.

Table A1

Steps of the child consultation that health extension workers using ChARM were observed completing

	Consultation step	Definition	Source of step
1	Correct child position	Back fully supported, either in the arms of the caregiver (younger child) or sat on the caregiver's lap with their back against the caregiver's front (older child) or lying on their back on a flat surface (older child).	Device manufacturer instructions for use
2	Correct device position	Device on the belly line in line with the nipple.	Device manufacturer instructions for use
3	Correct belt attachment	ChARM touching skin/clothing and belt not tangled.	Device manufacturer instructions for use
4	Correct age group	Child age group selected by FCHV on ChARM matches screening checklist.	WHO case management guidelines
5	Correct child behaviour immediately before ChARM attempt	Calm: not actively crying or moving.	WHO case management guidelines
6	Correct child eating/breastfeeding status during successful ChARM attempt	No eating/breastfeeding.	WHO case management guidelines
7	Correct child behaviour during successful ChARM attempt	Calm: not actively crying or moving.	WHO case management guidelines
8	FCHV classified the child's breathing status correctly using ChARM during ‘successful’ attempt	According to iCCM guidelines, based on child age group recorded during screening, and RR displayed by ChARM during successful attempt.	WHO case management guidelines
1‐8	Correct assessment and classification (steps 1‐8)a	FCHV correctly completed all steps 1‐8​	Device manufacturer instructions for use and WHO case management guidelines​
9	Correct referral using ChARM	According to iCCM guidelines, based on child age group recorded during screening, and RR displayed by ChARM during successful attempt. N.B FCHV will not be marked as 'incorrect' if caregiver refused or other valid reason for no referral recorded	WHO case management guidelines

Primary outcome.