Low specificity of a malaria rapid diagnostic test during an integrated community case management trial.

INTRODUCTION: Parasitological confirmation before administration of antimalarial treatment has been recommended by the World Health Organization in everyone presenting with symptoms suggestive of malaria at all levels of the health system.
METHODS: The authors assessed the performance of a histidine-rich protein 2-based malaria rapid diagnostic test used by community health workers in the context of an integrated approach to diagnosis and treatment for malaria and pneumonia. A total of 525 children below 5 years of age were recruited into the study. Children with fever/history of fever within the last 24 h were tested with the rapid diagnostic test (RDT) and a blood smear was obtained for delayed reading.
RESULTS: Overall, the FirstSign™ Malaria Pf (Unimed International Inc, South San Francisco, USA) has shown a high sensitivity profile of 97.9% (95% CI 96.3-98.8), but a low specificity of 53.4% (95% CI 49.1-57.7). The specificity was significantly lower during the high transmission season at 25.4% (95% CI 20.5-31.0) compared to 63.7% (95% CI 57.6-69.4%) at the low transmission season. The negative predictive value (NPV) was 95.4% (95% CI 93.2-96.9) and positive predictive value was 71.7% (95% CI 67.7-75.4). The NPV was significantly higher during the low transmission season at 98.2% (95% CI 95.7-99.3) than compared to 80.0% (95% CI 74.7-84.4) at the high transmission season.
CONCLUSION: With such a low specificity, caution should be exercised when using these RDTs for community case management of malaria.

Introduction

Malaria remains a leading cause of morbidity and mortality among those under 5 years in sub-Saharan Africa, in spite of the recent progress in the development of cost-effective tools for targeting this disease in more vulnerable groups [1-3].

Delivery of prompt and adequate treatment at the community level remains a key strategy to reduce the burden of malaria in sub-Saharan Africa [4]. Community case management was developed initially using chloroquine (CQ) and sulphadoxine–pyrimethamine. However, in recent years, with the almost universal development of the malaria parasite resistance to these drugs [5-7], artemisinin combination therapies (ACTs) are currently the best treatment option. Several studies have shown that trained community health workers (CHWs) are able to adequately use these ACTs in treating fever/malaria episodes [8-10].

Parasitological confirmation before administration of antimalarial treatment has been recommended by the World Health Organization (WHO) in everyone presenting with symptoms suggestive of malaria at all levels of the health system. Rapid diagnostic tests (RDTs) with their ease of use offer a practical means to improve diagnosis and quality of care of febrile children at community level in malaria endemic settings [11, 12].

More and more, given the overlap in symptoms between malaria and pneumonia [13], the WHO and the United Nations Children’s Fund (UNICEF) now recommend integrated community case management (ICCM) of malaria and pneumonia in endemic areas in low- and middle-income countries [14].

The authors conducted an integrated diagnostic and treatment approach trial for malaria and pneumonia, which involved training the CHWs, to use rapid diagnosis tests (RDTs) and respiratory rate timers (RRTs) in children with fever/“hot body” and to provide adequate treatment with ACTs and antibiotics based on the results of the two tests.

The results from the main outcome of this trial have been published elsewhere [15]. The authors report here the accuracy of the RDT when used at the village level by the CHWs during this trial.

Methods

This evaluation was part of a trial, the primary results of which were published [15]. In brief, the authors conducted an open cluster randomized two-arm trial. Clusters were the villages of individual CHWs. A total of six clusters were randomly assigned to each study arm.

In the intervention arm, CHWs assessed children with acute febrile illness for malaria using RDTs, and for pneumonia by counting their respiratory rate with RRTs. Treatment was then provided on the basis of the test results. Children with a positive RDT received artemether–lumefantrine and children with a high respiratory rate received cotrimoxazole.

In the control arm, all febrile children received ACTs based on a presumptive diagnosis of malaria. No RDT was performed and no antibiotics were given. Therefore, data presented here are those collected from the intervention arm.

Study Area and Population

The study was conducted in the health district of Saponé between August 2009 and June 2010. This rural area is situated 50 km south-west of Ouagadougou, the capital city of Burkina Faso.

It is an area of Sudanese savannah with a cold and dry season from November to January (monthly average temperatures varying between 11 and 30 °C), one hot and dry season from February to May (average temperature between 21 and 40 °C) and a rainy season from June to October (average temperature between 23 and 30 °C).

The transmission of malaria is high with marked seasonality. It is very intense during the rainy season and low during the dry season. Entomological inoculation rate is as high as 500 infective bites/person/year.

On average, children of less than 5 years of age experience about zero to three malaria attacks per year, with large variability among individuals [16].

Recruitment and Treatment of Study Participants

Caregivers were instructed to take their children to the CHWs whenever they had fever (“hot body”). Children were included based on the following criteria: (1) written informed consent from parent/guardian, (2) age between 6 and 59 months, (3) willing to comply with the study procedures, (4) history of fever within the last 24 h or documented fever (axillary temperature ≥37.5 °C). Children with the following were excluded and referred to the nearest health facility clinic: (1) danger signs (unable to drink or eat, incoercible vomiting, convulsions, prostration), (2) history of allergic reaction to the study drugs, (3) history of treatment with artemisinin derivatives in the past 7 days, (4) previous participation in the study within the same transmission season.

Children with positive RDT were treated with artemether–lumefantrine. Cotrimoxazole and antipyretic were also given in case of associated pneumonia and confirmed fever (axillary temperature ≥37.5 °C).

Parasitological Assessment Tools

The Rapid Diagnosis Test

FirstSign™ Malaria Pf (Unimed International Inc, South San Francisco, USA) rapid diagnostic test which detects the P. falciparum-specific histidine-rich protein 2 (HRP-2) was used. A job aid was developed based on the manufacturer’s instructions. The tests were individually sealed, transported and stored according to the manufacturer’s instructions, in key-locked boxes provided to the CHWs and were opened just when ready to be used. The main stock of RDTs was kept in the main office of the Centre National de Recherche et de Formation sur le Paludisme (CNRFP) under controlled temperature conditions and the CHWs received weekly supply during routine supervision.

The Malaria Blood Films Preparation and Reading

Thick and thin blood films were prepared and air dried by the CHWs. Slides were collected, Giemsa stained and examined in the CNRFP parasitology laboratory using a light fitted with a 100× oil immersion lens. The number of parasites and leucocytes were counted to reach 200 leukocytes for positive slides. Slides were declared negative only after 100 high power fields had been read. The number of parasites was converted to a count/μL assuming a standard leucocyte count of 8,000/μL. The slide reading was done by two independent experienced microscopists blinded to the RDT results from the field. After reconciliation of the two readings, slides in which discrepant results were found were read by a third senior microscopist. Discrepancy of reading was defined as the following: the ratio of densities from the first two readings >1.5 or <0.67; <30 parasites counted with an absolute difference in the number of parasites >10; discordance in positive–negative or species. The final result was based on the two most concordant readings.

Selection and Training of CHWs

Following discussion with communities in each of the selected clusters, they were requested to identify the CHWs that will be trained on the study procedures based on criteria provided by the study team. Among other criteria used were the availability of the person and the level of education and integrity.

Selected CHWs received standard training on CCM used elsewhere [17, 18]. In addition, they were trained to make quality blood smears and perform malaria RDT by senior microscopists from the CNRFP. The CHWs from the intervention arm were also trained by the study clinician to perform respiratory rate counting using timers.

At the end of the training session, proficiency of the CHWs was assessed and retraining organized for those who failed. At the end of the process, 13 CHWs were selected for the field activities.

In total, it took 2 weeks for the CHWs to familiarize themselves with all the study procedures.

Data Analysis

All data were recorded in Epi-Info™ 6.0 (CDC, Atlanta, USA). Using microscopy as “gold standard”, each RDT result was categorized as true positive (TP), true negative (TN), false positive (FP) or false negative (FN). The following performance indices were calculated with their 95% confidence interval: sensitivity (TP/TP + FN), specificity (TN/TN + FP), positive predictive value (PPV) (TP/TP + FP), negative predictive value (NPV) (TN/TN + FN), false-positive rate (1 − sensitivity), false-negative rate (1 − specificity) and likelihood ratios for positive and negative tests (respectively, calculated as sensitivity/false-negative rate and false-positive rate/specificity).

Ethical Approval

Ethical approval for this study was granted by the WHO Ethics Review Committee and by the National Ethical Committee for Health Research of Burkina Faso. Assent was obtained from district, local and community leaders as well as household heads. All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2000. Written informed consent was obtained from caregivers of children who participated in the study.

Results

A total of 533 participants were screened with 525 recruited into the study. The reasons for excluding the eight subjects were the presence of danger signs in three participants, history of treatment with antimalarial drug in the past 7 days for two subjects and age greater than 5 years in three subjects. The median age was 25.8 months and 52.8% of subjects were female. A total of 284 patients (54.8%) had positive blood smears for asexual forms of P. falciparum. Other baseline characteristics are presented in Table 1.

Table 1

Baseline characteristics of the trial participants

	Overall	Malaria high transmission season	Malaria low transmission season
Number of children enrolled	525	264	261
Number (%) with measured temperature ≥37.5 °C	436 (84.2)	214 (81.1)	222 (85.1)
Mean age (months)	28.7	28.4	29.2
Number (%) of females	277 (52.8)	147 (55.7)	130 (49.8)
P.f. asexual parasitemia prevalence (by microscopy)	284 (54.1)	201 (76.1)	83 (31.8)
Geometric mean parasite density in positives	11,841	12,588.2	7,903.8

Table 2 shows the comparative performance of FirstSign Malaria Pf detection assay calculated on the basis of the microscopically confirmed results.

Table 2

Performance of the rapid diagnostic test (RDT) by malaria transmission season

	Overall	Malaria high transmission season	Malaria low transmission season
Sample size	N = 516	N = 262	N = 254
False-positive rate	46.6%	74.6%	36.3%
False-negative rate	2.1%	2.0%	2.4%
LRa-positive test	2.1	1.3	2.7
LRa-negative test	0.04	0.08	0.04

aLikelihood ratio

Overall, the FirstSign Malaria Pf has shown a sensitivity as high as 97.9% (95% CI 96.3–98.8), but a low specificity of 53.4% (95% CI 49.1–57.7). The specificity was significantly lower during the high transmission season at 25.4% (95% CI 20.5–31.0) compared to 63.7% (95% CI 57.6–69.4%) at the low transmission season (Fig. 1).

Fig. 1

Diagnostic accuracy of the RDT according to malaria transmission seasons

The NPV was 95.4% (95% CI 93.2–96.9) and PPV was 71.7% (95% CI 67.7–75.4). The NPV was significantly higher during the low transmission season at 98.2% (95% CI 95.7–99.3) than compared to the 80.0% (95% CI 74.7–84.4) at the high transmission season. During the high transmission season, the false-positivity rate was twice that observed during the low transmission (74.6% vs. 36.3%). The likelihood ratio for positive tests was two times higher during the low transmission season compared to the high transmission season (2.7 vs. 1.3). For negative test, the likelihood ratio was two times lower during the low transmission season (0.04 vs. 0.08).

From the 385 positives tests, 109 (28.3%) were false positive. A total of six tests were false negative out of the 131 negative FirstSign Malaria Pf tests. From these six subjects, one subject had a low parasite density (95 parasites/μL). The parasite count ranged from 3,347 to 185,020 parasites/μL for the five remaining subjects. All of them had coincidental acute respiratory tract infection and had received cotrimoxazole. Fever was resolved when they were seen 3 and 7 days after the onset of treatment.

Stratification by age and P. falciparum parasite density showed that the lowest sensitivity and specificity were recorded in children aged 48–59 months harboring less than 500 asexual parasites/μL [respectively, 85.7% and 43.3% (33.0–54.2%)] (Table 3).

Table 3

Diagnostic accuracy of rapid diagnostic test (RDT) by parasite density and age group (any malaria transmission season)

Age group (months)	Parasite count	RDT results			Sensitivity (%)	Specificity
		N	Positive	Negative
	<500	38	17	21	100
0–11	500–4,999	6	6	0	100
	5,000–9,999	3	3	0	100	60% (48.8–70.3)
	≥10,000	29	28	1	96.6
	Overall	76	54	22	97.6
	<500	69	31	38	100
12–23	500–4,999	17	17	0	100
	5,000–9,999	5	5	0	100	60.3% (52.4–67.7)
	≥10,000	61	61	0	100
	Overall	152	114	38	100
	<500	64	36	28	100
24–35	500–4,999	9	9	0	100
	5,000–9,999	5	5	0	100	46.7% (37.8–55.8)
	≥10,000	37	36	1	97.3
	Overall	115	86	29	98.2
	<500	47	23	24	100
36–47	500–4,999	6	6	0	100
	5,000–9,999	2	2	0	100	55.8% (45.2–65.9)
	≥10,000	29	29	0	100
	Overall	84	60	24	97.6
	<500	37	23	14	85.7
48–59	500–4,999	12	11	1	91.7
	5,000–9,999	10	9	1	90.0	43.3% (33.0–54.2)
	≥10,000	21	21	0	100
	Overall	80	64	16	94.0

In terms of proficiency, at the first step, which was also a selection test, 13 of the 15 CHWs who were trained were classified as competent to perform the RDT test. The two others classified as “in training” were retrained, but did not take part in the study. At the second step, all the CHWs were able to adequately implement the trial-required procedures.

Discussion

During this trial, the authors evaluated the performance of this HRP-2-based RDT used by trained CHWs under field conditions.

A limit of this trial is the absence of data on the quality of the RDTs in the field to document that this quality has not biased the results that was obtained. However, we do not think that the quality of RDT was altered in the field. The stability under heat conditions is the main concern for RDTs and, as mentioned in “Methods”, the RDT tests were kept under temperature-controlled conditions in the research center pharmacy store and the CHWs received weekly supply. Also, during the dry season when the temperature in the field is extremely high (up to 40 °C), the test has proved to still have a high sensitivity and specificity profile as compared to that recorded during the rainy season where risk of exposure to extreme heat is minor.

The overall sensitivity of the RDT was high when compared with light microscopy in terms of detecting individuals infected by P. falciparum. This confirms what has been reported in other studies [19-21]. RDTs can be useful and reliable tools in the management of patients with suspected malaria, especially in contexts where microscopic diagnosis is not readily available, such as in remote area health centers or in the context of community case management of malaria, in which treatment is provided by trained volunteers from the community.

The sensitivity of the RDT has remained high across malaria transmission seasons and age range except in children aged between 48 and 59 months where a reduced sensitivity below acceptable threshold for RDTs was observed when the parasite count was low (below 500). It has been shown that HRP-2 tests could fail to detect low-level parasite densities [22-25]. However, the test also failed to detect two cases of P. falciparum infection with high parasite count in the same age group. A possible reason is that age-dependent immune status can reduce HRP-2 sensitivity independently of parasite density [23]. This hypothesis is highly plausible in the context of intense and marked seasonal malaria transmission where individuals will acquire semi immune protection against malaria early in life [16]. Another possible reason is that HRP-2 test sensitivity can be affected by the variability of HRP-2, the target antigen in specific settings [26]. This might not be the case in this context since the study was conducted in the same geographical area and polymorphism of the antigen was unlikely to occur. In Mali, it was reported that spontaneous HRP-2 deletions occurred in many areas with P. falciparum transmission, and this also could explain false-negative HRP-2 test results [27].

As already reported in numerous studies using HRP-2 tests, the specificity of the FirstSign Malaria Pf was extremely low and varied across seasons in our study. Indeed, the specificity was significantly reduced by half during the high malaria transmission season as compared to the low malaria season [from 63.7% (57.6–69.4) to 25.4% (20.5–31.0)]. Although the authors could anticipate that from literature, the value was, however, lower than that expected. Persistent HRP-2 antigenemia after effective treatment is one of the possible explanations of this low specificity. Indeed, in studies conducted in Uganda and the Democratic Republic of Congo where transmission is more perennial, it was shown that HRP-2 antigen could still be in the bloodstream for a long time (more than 5 weeks) after successful treatment [28, 29]. The authors could not also exclude the fact that in this context with malaria high endemicity, a high proportion of individuals carried low parasite density not detected by microscopy despite the experience of microscopists and the quality control using double reading of each individual blood smear. Only the use of polymerase chain reaction (PCR) methods that have a sensitivity superior to microscopy to detect low parasites count would have helped to rule out this possibility [30].

These findings suggest that when HRP-2 tests are used for case management in children less than 5 years living in area of intense and seasonal transmission of malaria, there is a risk of over-diagnosis, which may adversely affect the quality of care with the possibility of missing true cases of non-malaria febrile diseases, raising serious safety concerns. Also, the rational use of antimalarial drugs, which is one of the aims of introducing the use of RDT by CHWs, may be compromised.

The likelihood ratios constitute one of the best ways to measure and express diagnosis accuracy [31]. They determine the accuracy of a positive or negative result and are independent of the prevalence of a disease conditions in populations [32].

The ratios the authors computed for positive and negative tests to malaria transmission season suggested that the diagnostic efficiency of FirstSign Malaria Pf tests was highly dependent on the malaria transmission intensity. The lower the malaria transmission, the higher is the probability that patients with positive test results will have true malaria infection and vice versa.

The high rate of false positivity highlights the need for not using a positive test result as an excuse for excluding other possible causes of fever; this requires some clinical skills that are not readily available among CHWs, who in these contexts are lay persons from the community. Therefore, the approach of integrated community case management targeting malaria and pneumonia is an alternative excellent way to minimize negative effects of the low specificity, knowing the preponderance of these two diseases and their overlap in young children [5]. In this study that has implemented this approach, cure rates for fever at day 3 and day 7 were 97.8% and 99.6%, respectively [15], probably because the antibiotic associated with the antimalarial when indicated played a significant role.

Conclusion

Malaria HRP-2 antigen-based RDT used by CHWs to orient treatment of malaria cases has achieved a high sensitivity compatible with WHO requirement. However, an extremely low specificity was observed overall and with a marked reduction during the malaria high transmission season.

Caution should be exercised when using these RDTs for community case management of malaria, mainly in areas with high malaria transmission settings. Integrated community management of fever could help to mitigate the safety threat to patients from the risk of missing non-malaria illnesses when these tests are used by non-clinicians.