Quality of sick child care delivered by Health Surveillance Assistants in Malawi.

OBJECTIVE: To assess the quality of care provided by Health Surveillance Assistants (HSAs)-a cadre of community-based health workers-as part of a national scale-up of community case management of childhood illness (CCM) in Malawi.
METHODS: Trained research teams visited a random sample of HSAs (n = 131) trained in CCM and provided with initial essential drug stocks in six districts, and observed the provision of sick child care. Trained clinicians conducted 'gold-standard' reassessments of the child. Members of the survey team also interviewed caregivers and HSAs and inspected drug stocks and patient registers.
FINDINGS: HSAs provided correct treatment with antimalarials to 79% of the 241 children presenting with uncomplicated fever, with oral rehydration salts to 69% of the 93 children presenting with uncomplicated diarrhoea and with antibiotics to 52% of 58 children presenting with suspected pneumonia (cough with fast breathing). About one in five children (18%) presented with danger signs. HSAs correctly assessed 37% of children for four danger signs by conducting a physical exam, and correctly referred 55% of children with danger signs.
CONCLUSION: Malawi's CCM programme is a promising strategy for increasing coverage of sick child treatment, although there is much room for improvement, especially in the correct assessment and treatment of suspected pneumonia and the identification and referral of sick children with danger signs. However, HSAs provided sick child care at levels of quality similar to those provided in first-level health facilities in Malawi, and quality should improve if the Ministry of Health and partners act on the results of this assessment.

We used rigorous measurement techniques to assess delivery of child health care by trained Health Surveillance Assistants under routine conditions.

Overall, 62% of children with confirmed fever, cough with fast breathing and/or diarrhoea were treated correctly, a finding similar to previous studies in Malawi and sub-Saharan Africa.

Inadequate drug stocks contributed to inappropriate treatment of children presenting with fever and diarrhoea; inaccurate counting of respiratory rates accounted for many errors in treatment of children with cough and fast breathing. Other weaknesses included failures to assess, recognize and refer for danger signs and to assess and treat pneumonia correctly.

Ensuring continuous drug supplies and reinforcement of physical exam skills through training and more frequent, clinically-focused supervision could improve the quality of treatment provided to sick children in the community.

Introduction

Child mortality is declining globally, but under-five mortality rates remain unacceptably high in some regions and the majority of the world’s poorest countries are not on track to meet Millennium Development Goal (MDG) 4 (Bhutta ). Simple and effective treatments are available to address several of the main causes of under-five deaths, including malaria, diarrhoea and pneumonia (Jones ; Koram ), but coverage for these interventions remains low in many countries (Bhutta ). Community case management (CCM) by community-based health workers is a delivery strategy that is increasingly promoted by the global child health community (Haines ; UNICEF 2010; de Sousa ). CCM involves training and supporting community-based health workers to assess, classify and treat common childhood illnesses, using an algorithm adapted from the Integrated Management of Childhood Illness (IMCI) that focuses on a holistic assessment of the child’s presenting signs and symptoms rather than a single disease category.

CCM programmes will only be effective in reducing mortality if they deliver care at scale while meeting quality standards. Evidence is urgently needed to demonstrate that CCM services can be provided with high quality (Bhutta ), and that adequate health system supports including regular supervision and uninterrupted drug supplies can be successfully delivered at the community level (Haines ). The challenge of providing high quality care for sick children is not unique to community-based delivery strategies; even children treated by trained clinicians in health facilities in low-income countries often receive health care of inadequate quality (Hanson ; Amaral ; Arifeen ; Pariyo ; Rowe ).

Previous examinations of the quality of care delivered through CCM have evaluated pilot programmes in limited geographic areas (Kelly ; Rowe ), or have assessed the skills of CCM providers, rather than quality of services received by sick children, through tests of knowledge (Zeitz ; Ashwell and Freeman 1995), reviews of case registers (Mehnaz ; Rowe ; Rowe ) or through role plays and case scenarios (Zeitz ; Kallander ). Previous observation-based studies of sick child care have evaluated care for a single disease or have observed CCM providers at health facilities rather than their normal practice settings (Zeitz ; Ashwell and Freeman 1995; Kelly ; Hadi 2003; Rowe ; Rowe ). This study uses rigorous, community-based observation methods with ‘gold-standard’ clinical reassessments to evaluate the quality of care for sick children in the context of a CCM programme being implemented at scale.

The study assesses the quality of care provided through CCM in a national programme in Malawi. Malawi is a landlocked east African country with a population of 13 million people, 17% of which are under the age of five. The Ministry of Health (MOH) is the primary health care provider and provides services free of charge. The Christian Health Association of Malawi, a private faith-based organization, is also a major provider and charges fees for most services. Access to health facilities is limited; it is estimated that only 54% of the population has access to a health facility within a 5 km radius, and many of these facilities are unable to provide adequate services (Malawi Ministry of Health 2004). Data on care seeking for suspected childhood pneumonia from the 2006 Multiple Indicator Cluster Survey indicate that government clinics and outreach are the major source of care (40% of suspected cases), followed by drug shops (9%) and private clinics (6.7%).

Health Surveillance Assistants (HSAs) in Malawi are employed under the Environmental Health section of the MOH. Recruitment criteria include literacy and completion of at least 2 years of secondary school. HSAs undergo a 10-week basic training programme, and their responsibilities include health promotion and delivery of services for family planning, HIV, tuberculosis, malaria prevention and nutrition. HSAs collect vital statistics and maintain village household registers. Each HSA is responsible for providing services to a population of approximately 1000 individuals. In 2008, funding from the Global Fund for Malaria, HIV and Tuberculosis supported an increase in the number of HSAs employed by the MOH, from about 6000 to about 11 000.

Malawi’s MOH began scaling up CCM nationally in 2008 by providing additional training for HSAs with support from the World Health Organization (WHO), United Nations Children's Fund (UNICEF), the Bill & Melinda Gates Foundation, and Canadian International Development Agency (CIDA) under the Catalytic Initiative to Save One Million Lives (Malawi Ministry of Health 2007; Canadian International Development Agency 2009). The initial focus of this ‘Rapid Scale-Up’ was in 10 of the country’s 28 districts. District teams identified HSAs living in ‘hard-to-reach’ areas (loosely defined as >7 km from a health facility) to be the first to receive CCM training, following the national strategy for CCM scale-up. HSAs are expected to hold CCM clinics 1–2 days per week, as well as to be available on demand to treat sick children needing care urgently.

The CCM training curriculum is of 6 days’ duration and uses methods similar to those developed for the IMCI, involving both classroom training and clinical practice (World Health Organization 2011), and introducing the use of a ‘Sick Child Recording From’ (SCRF) as a job aid (Government of Malawi, WHO and UNICEF 2008). The content of the WHO/UNICEF Caring for the sick child in the community package (WHO/UNICEF 2011) for children aged 2 months to 5 years was adapted by the MOH for use in Malawi’s CCM programme, and trains HSAs to assess and classify children presenting with uncomplicated cases of fever (as a proxy for malaria), cough with fast breathing (as a proxy for pneumonia), diarrhoea and eye infections, and to treat them with artemisinin combination therapies (ACTs), antibiotics (cotrimoxazole), oral rehydration salts (ORS) and zinc, and antibiotic eye ointment, respectively. The HSAs also learn to identify danger signs for more severe illness and to refer children with these signs, and any other illnesses they cannot treat, to the nearest health facility. Figure 1 shows the CCM SCRF as adapted for use in Malawi. The training strategy also recommends skill reinforcement through follow-up visits by a CCM trainer/supervisor to the HSA in their place of work within 6 weeks after training.

Figure 1

Sick Child Recording Form job aid, with Malawi CCM guidelines

In 2009, the MOH and partners recognized the need for information about the quality of care being delivered by HSAs under the CCM programme. This assessment was carried out as part of a larger independent evaluation of the Catalytic Initiative.

Methods

The assessment used a cross-sectional, observational design to assess the quality of CCM care provided by HSAs. Data were collected in October and November 2009, about 18 months after the first HSAs were trained in CCM. This period falls outside the peak season for childhood pneumonia in Malawi which runs from February to April (Gordon and Graham 2007), as well as the peak season for malaria which runs from December to May (Ewing ). There is no clear pattern of diarrhoea incidence in Malawi, due to the many different underlying pathogens (Vargas ).

Our aim was to include districts that had achieved sufficient implementation strength to support an assessment of CCM quality, and for which data on the quality of care provided in first-level health facilities were available. Six of the 10 Rapid Scale-Up districts met the following inclusion criteria: (1) participated in the 2009 IMCI Health Facility Survey; (2) District Health Management Team (DHMT) records indicated that at least 10% of HSAs had participated in CCM training; and (3) DHMT records indicated that at least 50% of CCM-trained HSAs had received initial drug stocks of antibiotics, antimalarials, antipyretics and ORS. All six of the districts meeting these criteria were included in the sample. No HSAs had received zinc through the national programme at the time of the assessment; therefore, zinc was removed from the assessment protocol.

The sampling frame within each district included all HSAs (n = 267) that DHMT records indicated had been trained in CCM and had received initial stocks of CCM drugs after training. We randomly selected 22 HSAs from each of the six districts, for a total sample size of 132. We expected to observe an average of three sick children to visit for each sampled HSA during a full day of service provision, based on reports of early supervision visits. This sample size, with an anticipated 396 sick child observations, was estimated to allow +/− 6% precision in our point estimate of the proportion of sick children treated and/or referred correctly for all illness classifications estimated at 60%, assuming 95% confidence and a design effect of 1.5 to account for correlation of observations among children seen by the same HSA. Eight HSAs were replaced during the assessment using random selection and the original sampling frame because they had not received initial drug stocks (3), were not providing CCM services due to resignation or absence (4), or refused to participate (1).

Sampled HSAs were asked to hold routine CCM clinics every day over the period of 1 week during which the research teams made unannounced visits to HSAs in the community settings where they provide sick child care. No additional efforts were made to mobilize sick child visits for the purpose of this assessment. Teams arrived early in the morning and observed all children aged 2 months to 5 years who presented for care for a new complaint, up to a maximum of five children for each HSA.

For this assessment we defined indicators of correct case management based on existing consensus indicators for the quality of sick child care developed for IMCI (WHO 2003) after adaptation to reflect the Malawi CCM clinical guidelines. The indicators covered assessment, classification, treatment, counselling and referral. Box 1 describes the definitions of correct assessment, CCM-treatable illnesses and danger signs. We also assessed the rational use of antibiotics by examining the proportion of children with non-severe disease, and without suspected pneumonia (cough with fast breathing), who were not provided an antibiotic. Indicators of health system support addressed whether each HSA had received a follow-up visit within 4 to 6 weeks after initial CCM training, and measures of supervision in the 3 months prior to the assessment. Indicators of drug stocks included both HSA reports of stock-outs and visual inspection of the drugs available on the day of the assessment visit. Drug assessments focused on the available first-line treatments for malaria (Artemether-Lumefantrine), pneumonia (cotrimoxazole) and diarrhoea (ORS).

Definition of correct assessment for CCM-treatable illness

Assessment for uncomplicated fever: HSA asked caregiver about the presence or history of fever or caregiver offered this complaint.

Assessment for cough with fast breathing: HSA asked caregiver about the presence of cough or caregiver offered this complaint; HSA asked about number of days of cough; HSA counted the child’s respiratory rate using a timing device (e.g. timer or watch).

Assessment for diarrhoea: HSA asked caregiver about the presence of diarrhoea or caregiver offered this complaint; the number of days with diarrhoea and the presence of blood in stool.

Definition of CCM-treatable illnesses

Uncomplicated fever: Child with fever lasting less than 7 days without any danger signs.

Uncomplicated cough with fast breathing: Child with cough for less than 21 days with fast breathing (respiratory rate of 50 breaths per minute or more for ages 2–12 months and 40 breaths per minute for ages 12 months to 5 years) without any danger signs.

Uncomplicated diarrhoea: Child with diarrhoea less than 14 days without blood in stool and no danger signs.

Definition of danger signs requiring referral

General danger signs: Child is unable to drink, feed or breast feed; Vomits everything; Has or had convulsions.

Physical danger signs: Child has chest indrawing; Palmar pallor; Red on Mid-Upper Arm Circumference (MUAC) tape; Swelling of both feet (Bipedal oedema).

Further details on the assessment methods and copies of the research instruments are available online at http://www.jhsph.edu/dept/ih/IIP/projects/catalyticinitiative.html.

Procedures

Trained surveyors observed HSAs managing sick children. After the consultation, a ‘gold-standard’ clinician (either a master CCM trainer or district IMCI co-ordinator) who was not present during the consultation re-examined the same child independently using CCM clinical guidelines. Surveyors conducted exit interviews to assess caregiver’s knowledge of the child’s condition and treatment instructions. The re-examinations and exit interviews were conducted in a location out of sight and hearing of the HSA. Surveyors also interviewed HSAs using a structured questionnaire on their background, experience, workload, health systems supports, drug supplies in the previous 3 months and knowledge of case management through responses to a set of case scenarios. Surveyors also inspected drug stocks and reviewed the HSAs’ patient registers to estimate utilization.

Six teams collected data, each consisting of three surveyors (all medical professionals), with one surveyor assigned as team supervisor. The teams participated in a 7 day initial training on the assessment protocol, methodology and research ethics. Surveyors practised the methodologies of direct observation, clinical re-examination, and caregiver and HSA interviews in a classroom setting using role plays, and in clinical practice and community settings in a district not included in the assessment. All surveyors also participated in a 2 day refresher training halfway through the data collection period to review survey protocols and methods, problem-solve and ensure reliability. At the end of both trainings, surveyors achieved 90% reliability for direct observation of consultations in role-playing exercises.

Statistical analysis and interpretation

Data were double entered using CSPro software, with data processing and reconciliation occurring in Zomba, Malawi and Baltimore, USA. For key indicators we calculated proportions and 95% confidence intervals, adjusted for clustering of sick child consultations performed by the same HSA (StataCorp 2007). We performed a simplified clinical pathways analysis to identify the proportion of HSA errors at each step of the CCM algorithm (assessment, classification and treatment) for consultations where a child was given a gold-standard classification of fever, cough with fast breathing or diarrhoea, adapted from studies in Benin and Angola (Osterholt ; Rowe ). We performed all analyses in Stata 11 (StataCorp 2009).

Ethical review

This study was approved by the Institutional Review Board at the Johns Hopkins University Bloomberg School of Public Health and the Malawi Ministry of Health National Health Sciences Research Committee.

Results

Table 1 shows the characteristics of the HSAs included in the sample. Survey teams were unable to locate one HSA, and we dropped this person from the sample. HSAs were predominately male (81%); the median age was 32 with a range from 23 to 57. About 4 in 10 of the HSAs in the sample reported they had been recruited recently (since 2007). All HSAs reported receiving CCM training, with almost three-quarters of the HSAs receiving CCM training in the initial six-month training period between July and December 2008. Over 80% of the HSAs had been trained between 5 and 9 months previous to the assessment. Patient registers from the month preceding the survey were available for review for 102 HSAs (78%). The median number of children aged 2 months to 5 years seen by the sampled HSAs for any illness (including repeat visits) in the month of September 2009 across the six districts was 41, with district-level median utilization ranging between 25 and 50 sick child visits per month (data not shown).

Table 1

Characteristics of HSAs included in sample (n = 131)

	% (n)
Gender
Male	81% (106)
Female	19% (25)
Age (years)
20–25	17% (22)
26–30	24% (32)
31–35	24% (32)
36–40	18% (24)
>40	16% (21)
Recruitment as HSA
Earlier than 1996	27% (35)
1996–2006	30% (39)
2007–2009	43% (57)
Time since initial CCM traininga
<3 months	9% (12)
4–12 months	81% (106)
13–24 months	4% (5)
>24 months	6% (8)
Median sick child visits in previous month [IQR]	41b [19-73 visits]c

Notes: aBased on 123 HSAs. bBased on 102 HSAs with complete patient registers for month of September 2009. cInter-quartile range (IQR): the full range of sick child visits among the 102 HSAs varied between 0 and 148 visits.

We observed a total of 388 sick child consultations. We excluded six sick children from the analysis because of deviations from protocol, including: (1) the child was less than 2 months of age (2 children); (2) the child was not brought to the HSA for a new complaint (1 child); and (3) we could not reconcile observed inconsistencies or notes about the encounter on the data collection forms (3 children). Table 2 presents selected characteristics of the remaining 382 sick children whose consultations were observed and included in the analysis. Sixty-four per cent of observed consultations were for children aged under 2 years; only 17% of consultations were for children aged 3 to 5 years. About half of consultations were for male children (49%), and most were accompanied during their visit by a female caregiver (98%).

Table 2

Characteristics of sick children and their caregivers (n = 382)

Characteristics	% (n)
Child’s age (months)a
3–12	35% (133)
13–24	29% (110)
25–36	20% (75)
37–48	9% (33)
49–60	8% (31)
Sex of child
Female	51% (193)
Male	49% (189)
Sex of caregiver
Female	98% (374)
Male	2% (8)
Presenting complaint of sick children included in study as reported by caregiver to HSAb
Fever	55% (210)
Cough	45% (172)
Diarrhoea	24% (90)
Other problem mentionedc	23% (89)
Vomiting	13% (48)
Malaria	12% (44)
Red eye	8% (31)
Difficulty breathing/pneumonia	3% (10)

Notes: aPercentages add to 101% due to rounding. bCategories are not mutually exclusive, Caregiver can mention multiple complaints for one child. cAlso mentioned: Abdominal pain (24); Body sores/rash/scabies (18); Trouble feeding or drinking (9); Mouth/oral sores (9); Headache (5); Abscess/swelling of body parts (5); General irritation/crying (4); Bloody/wormy stools (3); Ear problems (1); Eye problems (3); Weakness (3); Problem with urine (4); Shivering/chills (3); Wound (1); Convulsions (1); Leg pain (1); Flu (1).

In each district, we enrolled between 30 and 89 children. The most common presenting complaints of the child as reported by caregivers to the HSAs were fever (55%), cough (45%) and diarrhoea (24%). About two-thirds (66%) of caregivers reported more than one complaint; the median number of complaints was two. Twenty-three per cent of caregivers reported symptoms other than those specifically addressed by the CCM guidelines, such as abdominal pain and skin rash.

Table 3 shows the case mix of sick children seeking care for a new episode of illness from the HSA on the day of the assessment visit, as defined by the gold-standard clinician. The majority of children presented with CCM-treatable illnesses. One in three children presented with either at least one CCM danger sign (18%) or other problems requiring referral (16%). The most common danger sign was fever for 7 days or more, reported in 5% of cases, followed by palmar pallor, and red eye for 4 days or more, both reported in approximately 4% of cases. Other problems prompting referral by the HSA included skin infection or rash (3.9%), abdominal pain (2.1%) and mucocutaneous lesions (2.1%) (data not shown).

Table 3

Signs and symptoms of presenting child illnesses as determined by gold-standard clinician (n = 382)

Signs and symptomsa	% (n)
CCM – treatable illnesses
Fever (<7 days)	72% (276)
Cough with fast breathing	20% (76)
Diarrhoea (<14 days and no blood in stool)	28% (105)
One or more danger sign	18% (69)
Danger signs
Fever for ≥7 days	5% (17)
Palmar pallor	4% (15)
Diarrhoea with blood in stool	3% (13)
Red eye ≥4 days	4% (14)
Chest indrawing	2% (6)
Diarrhoea for ≥14 days	1% (3)
Red on MUAC tape	1% (4)
Swelling of both feet	1% (3)
Not able to drink or feed anything	0.5% (2)
Convulsions	0.3% (1)
Vomits everything	0
Very sleepy or unconscious	0
Other signs/classifications
Cough	65% (248)
Yellow on MUAC tape	4% (16)
Behind on vaccines	10% (39)
Other problems, refer	16% (59)

Note: aCategories are not mutually exclusive, one child can have multiple gold-standard clinician classifications.

Table 4 presents the key results related to the quality of sick child care in the areas of assessment, classification and treatment, including the rational use of antibiotics and referral of the child to a health facility. For each indicator, we present the number of children for whom the task should have been performed based on the findings of the ‘gold-standard’ clinician, the proportion of children for whom this task was performed correctly by the HSA, and the 95% confidence interval for each estimate.

Table 4

Proportions of children for whom specific case management tasks were performed by HSAs in Malawi

Indicator	N (child consultations eligible)	%	[95% Confidence Intervala]
Assessment
Children checked for presence of fever, cough and diarrhoea	382	77%	[71–82%]
Children with cough assessed for presence of fast breathing through counting of respiratory rates	270	71%	[63–79%]
Children assessed for 3 general danger signs	382	56%	[50–63%]
Children assessed for 4 physical danger signs	382	37%	[30–45%]
Classification
Children whose classifications given by HSA match all the classifications given by IMCI-trained clinician/evaluatorb,c	382	44%	[38–49%]
Children whose classifications for uncomplicated illness (fever, cough with fast breathing, and diarrhoea) given by HSA match those classifications given by IMCI-trained clinician/evaluatorb	382	68%	[62–73%]
Treatment of child illness
Children with one or more CCM-treatable illnesses (uncomplicated fever; cough with fast breathing; and/or diarrhoea) who are correctly prescribed all medications required for their illnessd	280	62%	[56–68%]
Children with uncomplicated fever who are prescribed an antimalarial (ACT) correctlyd	241	79%	[73–85%]
Children with uncomplicated cough and fast breathing who are prescribed an antibiotic correctlyd	58	52%	[39–64%]
Children with uncomplicated diarrhoea who are prescribed ORS correctlyd	93	69%	[57–80%]
Children who need an antimalarial, antibiotic and/or ORS who received the correct first dose in the presence of the HSAd	280	30%	[24–37%]
Rational use of antibiotics
Children without cough and fast breathing who leave the HSA without having received an antibioticd	255	69%	[61–76%]
Referral for danger signs
Children with general and/or physical danger signs needing referral who are referred	69	55%	[42–68%]
Children with general and/or physical danger signs needing referral who receive correct pre-referral treatment and referrale	61	52%	[39–66%]
Counselling
Children prescribed one or more treatment (antimalarial, oral antibiotics and/or ORS), whose caregivers received dose, duration and frequency counselling messages about administering treatmentsd	272	61%	[54–68%]
Children prescribed one or more treatment (antimalarial, oral antibiotic and/or ORS), whose caregiver was able to describe correctly how to give the treatmentd	244	81%	[76–86%]
Children who had their vaccination status checkedd	265	75%	[68–81%]
Children with uncomplicated diarrhoea whose caregivers are advised to give extra fluids and continue feedingd	93	55%	[44–66%]

Notes: a95% Confidence intervals adjusted for sick child consultations performed by same HSA.

bAmong all children, whether positive or negative classification of illness.

cAll clinical classifications potentially requiring action, including red on MUAC and behind on vaccination status.

dAmong children without any general or physical danger signs requiring referral.

eCorrect pre-referral treatment considered either a pre-referral first dose administered during consultation or a full course of medicine provided by HSA to caregiver in addition to the first dose administered during consultation.

HSAs correctly assessed 77% of children for the presence of cough, diarrhoea and fever. Among children with cough, 71% were correctly assessed for the presence of fast breathing by checking their respiratory rate, while lower proportions of children were correctly assessed for general and physical danger signs (56% and 37%, respectively). The danger signs overlooked most often were convulsions (assessed for 66% of children), chest indrawing (assessed for 72%) and palmar pallor (72%) (data not shown). HSAs used a MUAC tape to assess nutritional status for 64% of children, and assessed bipedal oedema in 64% of children (data not shown).

The illness classifications assigned by the HSA were consistent with those of the gold-standard clinician for 44% of sick children. This proportion increased to 68% when considering only classifications of CCM-treatable illnesses (fever, cough and fast breathing, and diarrhoea).

About six in 10 children presenting with the signs and symptoms of one or more CCM-treatable illnesses received correct treatment from the HSA for their illness(es). A breakdown by type of illness shows that children with fever were more likely to receive a correct treatment (79%) than were children with signs associated with possible pneumonia (52%) or diarrhoea (69%). Thirty per cent of children needing an antimalarial, antibiotic and/or ORS received the correct first dose of treatment(s) in the presence of the HSA. Thirty-one per cent of children not classified with cough with fast breathing and therefore not needing an antibiotic as specified in the CCM clinical guidelines received cotrimoxazole from the HSA. Just over half of children presenting with danger signs received a referral from the HSA (55%).

Table 4 also includes findings related to HSA counselling of caregivers. HSAs counselled caregivers about the dose, frequency and duration of treatment for over half of children provided ORS, antibiotics or antimalarials (61%), and 81% of caregivers described correctly how to give these treatments. HSAs checked the status of vaccination for 75% of children observed for sick child care. Just over half (55%) of caregivers of children with diarrhoea were advised to give extra fluids and to continue feeding the child during the illness episode.

Table 5 shows the proportion of HSAs who reported supervision of various types in the months preceding the assessment and the presence of essential CCM drugs on the day of the visit. Few HSAs (23%) reported that they had received a follow-up visit after initial CCM training. Although a higher proportion (38%) of HSAs reported receiving a CCM-specific supervision visit in the previous 3 months, only 16% of HSAs reported a CCM supervision visit that included the observation of a sick child consultation. Forty-four per cent reported that they had discussed their CCM work with a supervisor during a visit to a health facility. About 70% of HSAs had all critical CCM drugs (antimalarials, antibiotics and ORS) except zinc in stock on the day of the assessment visit; drug supplies varied by treatment.

Table 5

Health systems supports for CCM, as reported by sampled HSAs

	% (n = 131)
Supervision (% of CHWs who reported)
Follow-up supervision within 6 weeks of CCM traininga	23%
Supervision visit specific to CCM in the previous 3 monthsa	38%
Supervision visit specific to CCM in the previous 3 months that included observation of case managementa	16%
Discussing their CCM work with a supervisor at the health facilityb	44%
Drug supply (HSAs had in stock on day of visit)
All critical CCM drugs (antimalarials, antibiotics, ORS)	69%
Antimalarials (ACTs, any formulation)	93%
Antimalarials (ACTs, 2–11 mo. formulation)	89%
Antimalarials (ACTs, 13–59 mo. formulation)	69%
Antibiotics	96%
ORS	74%
Supplies (HSAs had on day of visit)
Watch or timing device	84%
MUAC tape	81%

Notes: an = 128 due to absence of HSA in the community during recall period and/or missing/don’t know responses. bn = 129; 2 cases missing.

Figures 2a–c present an analysis of clinical steps for children presenting with uncomplicated fever, cough with fast breathing, and diarrhoea, as measured by the gold-standard clinician; these analyses indicate the common errors made by HSAs and the proportion of cases where the absence of essential drugs or a timing device may have influenced the quality of care. Overall, more than 90% of children with uncomplicated fever and diarrhoea were assessed and classified correctly; however, a smaller proportion of children received the appropriate treatment for fever (79%) or diarrhoea (69%) (antimalarials or ORS, respectively). Among children who did not receive treatment for fever, 52% were treated by an HSA who did not have antimalarials on the day of the assessment. Among children who were correctly assessed and classified with diarrhoea who were not treated with ORS, 72% were seen by an HSA who did not have ORS in stock on the day of the assessment. Among children with cough and fast breathing, 70% were correctly assessed; of the 43 children correctly assessed, 70% were correctly classified; only 52% overall were treated correctly with first-line antibiotics. Incorrect classification of cough and fast breathing was largely due to poor counting of respiratory rates by the HSA—only 30% of children with fast breathing had a respiratory rate that was measured within +/−2 breaths per minute of the measurement made by the gold-standard clinician. There were no stock-outs of cotrimoxazole on the day of the assessment where a child with suspected pneumonia was seen.

Figure 2a–c

Clinical pathways analysis Notes: aNumber of cases based on gold-standard clinician classification. bChild was correctly classified if the HSA classification matched the gold-standard clinician’s classification. cChild was treated correctly if he/she was given correct dose, frequency, and duration of first-line antimalarial (ACT) (184 cases; includes children that received one dose of antimalarial and were referred for a reason unrelated to fever (e.g. rash) (7 cases). dIncludes over- and under-dosing of antimalarial (6 cases) and antimalarial for child with fever under 5 months of age (2 cases). eNot treated includes: sick children with uncomplicated fever who did not receive any antimalarial; sick children with cough with fast breathing who did not receive Cotrimoxizole; sick children with diarrhoea who did not receive ORS. fStock-out of both formulations of antimalarial; of 28 fever cases correctly assessed and classified but not treated, 17/28 of HSAs had stock-outs of both antimalarial formulations. gChild was treated correctly if he/she was given correct dose, frequency and duration of Cotrimoxizole. hIncorrect treatment included under-dosing of appropriate medication (3 cases of under-dose for age; 1 case fewer days’ duration and 2 cases incomplete days mentioned). iChild was treated correctly if he/she was given ORS (regardless of amount); zinc was not generally available at the time of assessment and thus not included in correct diarrhoea treatment.

Discussion

Over 30 countries have adopted CCM as a strategy to increase access to and coverage of treatment interventions for childhood illness (Marsh ; Bhutta ; de Sousa ). This study of the quality of child health care delivered by HSAs trained in CCM used rigorous measurement techniques and ‘gold-standard’ reassessments to observe HSAs’ performance in the communities where they practise as part of a national scale-up of CCM. Based on 382 sick child consultations by a random sample of 131 CCM-trained HSAs, we found that 62% of children with confirmed fever (as a proxy for malaria), cough with fast breathing (as a proxy for suspected pneumonia) and/or diarrhoea were treated correctly. This compares favourably with results from a quality-of-care survey for health facilities conducted in the same year and using similar methods by the MOH and implementing partners in Malawi, which found that 54% of children with cough and fast breathing and 64% of children with fever received correct treatment by facility workers (Malawi Ministry of Health and the World Health Organization 2010). Still, pneumonia is the single greatest cause of death among children under 5 (Liu ), and further effort is needed urgently to improve its identification and treatment both in health facilities and at community level. New time technologies that reduce errors in the counting of respiratory rates may be useful, if they prove affordable and sustainable in these remote settings (van Heck 2012). Our finding that the majority of HSAs used antibiotics appropriately (69%) is similar to that observed among IMCI-trained health workers managing sick children in fixed facilities (Gouws ; Malawi Ministry of Health and the World Health Organization 2010), and suggests that there is not widespread misuse of antibiotics by HSAs. An analysis of the clinical errors made by HSAs indicates that substantial proportions of children presenting with fever and diarrhoea may have received inappropriate treatment due to stock-outs of antimalarials and ORS, respectively. This was not true for cough and fast breathing; despite the presence of antibiotics, clinical exam errors were made in the counting of respiratory rates, as has been identified previously in observation-based assessments of community health workers (Kallander ; Mukanga ).

This assessment has limitations that should be kept in mind when interpreting the results. First, these results are not representative of the CCM programme, which is in the process of being scaled up in all of Malawi’s 28 districts. The six districts that participated in the assessment were chosen from among the group of 10 districts receiving support for the special Rapid Scale-Up project, with specific attention to selecting only those districts that had met minimum criteria for CCM implementation. This is likely to have introduced a positive bias. Second, the HSAs may have provided better care than normal because they were being observed, a phenomenon sometimes referred to as the Hawthorne effect (Leonard and Masatu 2006), which would also have introduced a positive bias relative to the quality of care normally received by sick children in the participating districts. Given these limitations, the results should be viewed as a best case ‘snapshot’ of the CCM programme, at its early stage of development in 2009.

The use of a ‘gold-standard’ re-examination by a trained clinician is a particular strength of this study, and allows us to provide valid results on the quality of care received by children for their specific, confirmed diagnoses. Another strength of this assessment is the evaluation of HSAs in the communities in which they work. In contrast to facility-based studies, this allowed for a ‘real-world’ assessment of community health workers in their natural setting, and afforded advantages such as the inclusion of on-site inspections of drug stocks.

The purpose of the assessment was to provide a basis for continued improvement of CCM implementation in Malawi. The preliminary findings were analysed in a workshop held in Malawi with participation by the MOH at both district and national levels and by partners supporting the implementation of CCM. After revision, the results were presented at several MOH meetings and used in programme reviews, and were presented by the MOH at an international conference (Nsona 2010). We focus in the remainder of this section on the key findings and the actions they have stimulated by the MOH and its implementing partners.

One major finding is that HSAs appear to be fulfilling the expectation of bringing child health care closer to communities, filling a gap in ‘hard to reach areas’ with limited access to health facilities. The subset of HSAs with patient registers available saw a median of 41 sick children over the course of one month. This level of utilization is encouraging given that the CCM programme was still relatively new at the time of the assessment and few formal efforts had been made to communicate with communities about the new capabilities of the HSAs. MOH efforts to track utilization and to reinforce the use of patient registers by HSAs trained in CCM are expected to produce further evidence that CCM is extending access to child health care in hard-to-reach populations, and to clarify the extent to which HSAs are reaching children who otherwise would not access health services or are diverting care from first-level health facilities to the community.

A second major finding relates to patterns of illness of sick children presenting to HSAs’ village clinics in these hard-to-reach populations. Based on the ‘gold-standard’ clinical assessments conducted as a part of this study, the case mix of children brought to HSAs is reasonably similar to the pattern of illness reflected in the CCM clinical guidelines. Four in five children seeking care from an HSA had a fever, indicating possible malaria, and between one and two children in five presented with either diarrhoea or clinical signs indicating possible pneumonia. Perhaps more important, almost one in five children presented with a danger sign indicating severe disease, and over half of these children were referred after receiving correct pre-referral treatment by the HSA. Although this proportion may seem low relative to a clinical standard of 100%, in some of these community settings referral to a health facility may not be feasible. Although not strictly comparable, the referral rate in this study is similar to that found in studies of IMCI-trained health workers based in facilities in Africa (Gouws ; IMCI Multi-Country Evaluation Health Facility Survey Study Group 2004; Pariyo ; Malawi Ministry of Health and the World Health Organization 2010).

Specific performance weaknesses were noted in areas such as correct assessment and recognition of danger signs, fast breathing and physical exam skills, consistent with findings from other settings (Kelly ). Poor HSA performance on danger sign recognition is of particular concern as children presenting with danger signs are at the greatest risk. The MOH is reviewing the CCM training curriculum to determine how best to reinforce skills in the management of severely-ill children, such as more clinical practice at tertiary facilities and the use of more severe illness video and/or doll simulations in training. New technologies, such as rapid diagnostic tests for malaria, may be useful in improving the performance of HSAs.

These results throw into relief the need for health system supports to be well established as a basis for successful CCM programmes, especially early in implementation. Only three in four trained community-based health workers had patient registers on the day of the assessment, there were frequent stock-outs of essential drugs, and supervision visits were far less frequent than planned. The results of the clinical pathways analysis highlight the importance of ensuring continuous stocks of first-line treatments to ensure high quality services. The importance of ensuring adequate health system supports, including adequate care at first-level facilities, is widely recognized both in Malawi and globally. Further detail on these findings and the results of a companion study on health systems supports for CCM in Malawi are available elsewhere (Callaghan-Koru ).

The MOH and partners responded to these findings by convening a meeting in December 2010 to develop new plans and request expanded commitments from donors to address these shortfalls. Mentoring for HSAs at health facilities has been introduced in order to reinforce the clinical skills of HSAs and overcome barriers to supervision in communities. The MOH and partners have engaged Assistant Environmental Health Officers and Senior HSAs to include CCM in their on-going supervision of all of the HSAs’ activities in order to improve the frequency of CCM supervision, as well as developing a supervision manual to improve the quality of CCM supervision. Recognizing the fundamental need for a continuous drug supply, the following steps have been taken by the MOH and partners: (1) a logistics officer has been placed within the IMCI unit in charge of CCM to strengthen medicines monitoring and utilization and to train districts on the Logistics Management Information System; (2) district-specific quantification exercises are planned to ensure the needs for CCM medicines are established and included in the annual quantification exercise; and (3) a supply chain management system for CCM using front-line Short message service (SMS) by which HSAs report monthly on medicine stocks and order new supplies is undergoing testing. These positive steps, focusing on improving supervision, clinical skills and drug supply logistics, are an important and necessary start to ensuring high quality CCM in Malawi.

Conclusions

This observation-based assessment of the quality of sick child care provided by CCM-trained community health workers in Malawi indicates that the strategy is feasible for implementation at scale, and that this lowest cadre of paid health workers in Malawi was able to perform at levels similar to facility-based health workers trained in IMCI. However, the CCM strategy will not realize its potential to reduce child mortality unless clinical skills related to the assessment and treatment of pneumonia and identification and referral for danger signs improve and are sustained. Adequate health system supports for CCM are needed urgently in Malawi and elsewhere. Supervision and monitoring using observation-based methods can provide the information needed to strengthen and sustain CCM programmes.

Funding

This work was supported by the Canadian International Development Agency (CIDA) and the Bill & Melinda Gates Foundation (administered through the World Health Organization) through their support for the Catalytic Initiative to Save One Million Lives in Malawi and its evaluation. Cristina Cardemil received support for her participation through the Child Health Center Board and the Robert H. Parrott REACH Program.

Conflicts of interest

Some of the authors (H Nsona, A Mtimuni, B Daelmans, L Mgalula) are responsible for the implementation of the CCM programme in Malawi.