Introduction of Syphilis Point-of-Care Tests, from Pilot Study to National Programme Implementation in Zambia: A Qualitative Study of Healthcare Workers' Perspectives on Testing, Training and Quality Assurance.

Syphilis affects 1.4 million pregnant women globally each year. Maternal syphilis causes congenital syphilis in over half of affected pregnancies, leading to early foetal loss, pregnancy complications, stillbirth and neonatal death. Syphilis is under-diagnosed in pregnant women. Point-of-care rapid syphilis tests (RST) allow for same-day treatment and address logistical barriers to testing encountered with standard Rapid Plasma Reagin testing. Recent literature emphasises successful introduction of new health technologies requires healthcare worker (HCW) acceptance, effective training, quality monitoring and robust health systems. Following a successful pilot, the Zambian Ministry of Health (MoH) adopted RST into policy, integrating them into prevention of mother-to-child transmission of HIV clinics in four underserved Zambian districts. We compare HCW experiences, including challenges encountered in scaling up from a highly supported NGO-led pilot to a large-scale MoH-led national programme. Questionnaires were administered through structured interviews of 16 HCWs in two pilot districts and 24 HCWs in two different rollout districts. Supplementary data were gathered via stakeholder interviews, clinic registers and supervisory visits. Using a conceptual framework adapted from health technology literature, we explored RST acceptance and usability. Quantitative data were analysed using descriptive statistics. Key themes in qualitative data were explored using template analysis. Overall, HCWs accepted RST as learnable, suitable, effective tools to improve antenatal services, which were usable in diverse clinical settings. Changes in training, supervision and quality monitoring models between pilot and rollout may have influenced rollout HCW acceptance and compromised testing quality. While quality monitoring was integrated into national policy and training, implementation was limited during rollout despite financial support and mentorship. We illustrate that new health technology pilot research can rapidly translate into policy change and scale-up. However, training, supervision and quality assurance models should be reviewed and strengthened as rollout of the Zambian RST programme continues.

Background

12 million new cases of syphilis occur each year, the majority in developing countries [1]. Probable active syphilis occurs in 1.5 million pregnancies and contributes to 305,000 neonatal deaths and stillbirths each year [2,3]. Half of pregnant women with untreated syphilis will experience adverse pregnancy outcomes, such as miscarriage, stillbirth, premature delivery, low birth weight and neonatal infection [4-8]. Congenital syphilis has been a neglected public health problem in Sub-Saharan Africa (SSA), where syphilis prevalence among pregnant women ranges from 1.4 to 17% [9-12].

Universal syphilis screening of pregnant women is recommended as part of the basic antenatal care (ANC) package promoted by the World Health Organization (WHO), since the symptoms of early syphilis too often go unnoticed and late stages of the disease may be completely asymptomatic [13]. Despite the availability of screening tests and ample evidence that antenatal testing and treatment with single-dose Benzathine Penicillin improves pregnancy outcomes and is highly cost-effective, screening rates range from 1.7 to 79.9% of women attending antenatal care in SSA [6,14-21]. A recent analysis of antenatal surveillance data estimated 66% of syphilis-associated adverse pregnancy outcomes occurred in ANC attendees who were either not appropriately tested or treated for syphilis [3].

A number of key barriers to implementation of universal antenatal syphilis screening have been identified: inconsistent supply chain, patient cost and need to return for results, health worker absence or insufficient training, low prioritisation by health policy implementers and use of Rapid Plasma Reagin (RPR) as the standard diagnostic tool, which requires laboratory capacity, cold storage and electricity [6,11]. To address these barriers, new health technologies: rapid, specific and validated point-of-care (POC) syphilis tests, have been developed and successfully implemented in a variety of clinical settings [14,21-24].

However, experience gained in human immunodeficiency virus (HIV) and malaria treatment programmes illustrates that ensuring POC test use and reliability of results, particularly on scale-up, presents its own set of challenges [25-29]. Reliability is aided by adopting high quality test kits that are easy-to-use; providing adequate training to all healthcare workers (HCW); and integrating Quality Assurance/Quality Control (QA/QC) and supervision systems into programmes from the outset [23,26,30-32]. POC test implementation involves shifting testing to non-laboratory settings and non-laboratory HCWs, often unaccustomed to performing tests or routine QA/QC. Such task-shifting has important planning implications for workload of HCWs in already burdened health systems, and for testing strategy, diagnostic algorithms, QA/QC continuity and supply chain management [25,31,33-35]. Furthermore, HCW acceptance of POC tests and trust in their accuracy are key to ensuring that tests are performed and results are acted upon [25,27,36,37].

In this paper, we explore the end-user experience of new health technology introduction. We compare Zambian HCWs’ experience of RST usability, training and quality systems during a highly-supported pilot project versus a pared down national implementation programme. Through HCW and key informant interviews, we examine how health system planning and infrastructure influenced implementation of both pilot and first phase of Zambia’s national RST programme.

Methodology

Study design

Study context

Zambia is a lower-middle income country with a population of 14.1 million, spread over a large geographical area [38]. 94% of pregnant women attend at least one antenatal visit and ANC is free-of-charge [18,39]. The health care system is arranged in six tiers: outreach services, health posts, urban and rural health centres, district hospitals, secondary referral and tertiary referral hospitals [40]. Rural facilities are remote, located several hours’ drive from district hubs, lacking or with limited electricity and limited transport to convey patient samples, results or supplies back and forth to district laboratories.

In 2010–11, a six-country pilot study, including Zambia, evaluated the feasibility of introducing RSTs into existing maternal and child health (MCH) programmes [1,13,34]. The study was the first of its kind to incorporate QA systems to monitor training and accuracy of POC test results [34]. The non-governmental organisation (NGO)-led Zambian study arm introduced RST into existing prevention of mother-to-child transmission (PMTCT) of HIV services in 15 sites in two districts: Mongu, a rural district, with a syphilis prevalence of 7%, and Lusaka, an urban district, with a prevalence of 2.5% [41]. It showed that integrating RST into PMTCT programmes increased testing and treatment for syphilis in HIV positive pregnant women without compromising HIV service delivery.

Following the pilot’s success, the Zambian Ministry of Health (MoH) rapidly adopted RST into national policy and led the rollout of a national RST programme, incorporating QA/QC into programme design [42]. The rollout, supported by the Elizabeth Glaser Pediatric AIDS Foundation (EGPAF), commenced in 2012, initially in four underserved districts with high rates of maternal mortality: Kalomo, Lundazi, Mansa and Nyimba. Key implementation changes were made between pilot and rollout: supervision was reduced from monthly to quarterly and QA/QC activities were devolved from central to district laboratory level (Table 1).

Table 1

Changes in implementation methods from NGO-led RST pilot to MoH-led national RST rollout in Zambia 2008–2012.

	PILOT PHASE	ROLLOUT PHASE
Study Period	2008–2011	2012 to present
Implementers/ Funders	EGPAF, through external funders, and Center for Infectious Disease Research in Zambia implemented the Zambian arm of a six-country pilot coordinated by WHO /TDR Sexually Transmitted Diseases Diagnostic Initiative evaluating feasibility of RST introduction into prevention-of-mother-to-child-transmission of HIV (PMTCT) services.	EGPAF/CIDRZ collaborated with Zambian MoH technical working group to incorporate RST into national policy and produce national guidelines in 2011. EGPAF supported the MoH to implement first phase of national rollout by leveraging implementation funds and sharing pilot phase experience.
Study Site	15 facilities in 2 districts: Lusaka (urban; low syphilis prevalence: 2.5%); Mongu (rural; high syphilis prevalence: 7%)	All MoH facilities in 4 underserved districts with high rates of maternal mortality: Kalomo, Lundazi, Mansa, Nyimba.
Training Model	Cascaded training: Central workshop conducted by EGPAF/CIDRZ; attendees then provided on-the-job training to facility colleagues.	Cascaded training: District-level workshops conducted by MoH/EGPAF; attendees then provided on-the-job training to facility colleagues.
Treatment algorithm	Zambian syphilis treatment guideline pre-RST adoption was three weekly doses of Benzathine Penicillin (BP). During the pilot, patients were given one documented dose of BP following a positive RST test.	Treat with one dose of BP following positive RST result; run RPR confirmation and if active infection confirmed, treat with two additional doses of BP. If RPR confirmation unavailable, continue 2nd and 3rd weekly dose of BP.
Integration into existing services	RST integrated into existing staffing patterns and patient flow; alongside other routine antenatal tests (HIV, malaria, and haemoglobin).	RST integrated into existing staffing patterns and patient flow; alongside other routine antenatal tests. Rollout included MoH, Ministry of Defense and mission-affiliated facilities.
Outcome of integration	The same HCW offered same-day testing, results and treatment.	RST was variably integrated into patient flow depending on facility-level, HCW cadre and laboratory capacity.
Quality Control	Internal QC: in-built control panel External QC:1)weekly validation of RST kits with positive and negative control samples; 2) repeat confirmatory testing at a central laboratory of 2% of samples collected during study supervision visits.	Internal QC: in-built control panel External QC: Validation of RST kits with positive and negative control samples weekly and if a new shipment, new lot number or adverse environmental conditions occurred. Rarely implemented during early rollout phase evaluated here. Knowledge on QA/QC practices was rarely transferred during cascaded training. Control samples were not included in test kits or delivered to facilities by district lab personnel.
Quality Assurance	Health workers’ accuracy was checked using proficiency panels (sample RSTs prepared with dried tube specimens of serum known to be positive or negative for syphilis) during supervisory visits	Health workers’ accuracy was intended to be checked using proficiency panels sent to the facility by the district laboratory. Not implemented during the early rollout phase evaluated here due to lack of HCW time, lack of dedicated budget, logistics and manpower for QA/QC, inexperience and/or lack of initiative of the district laboratory personnel and lack of on-site lab-training in advance of rollout.
QA/QC Logistics	Control samples prepared by research laboratory and transported by study staff to facilities	Control samples were intended to be prepared by district laboratories and transported to the facilities with results transported back to the district laboratory. Not implemented during the early rollout phase examined in this study.
Supervision	Monthly visits from EGPAF study staff incorporating QA and remedial training for new or poorly performing HCWs	Quarterly visits from MoH and EGPAF staff
Cost Implications	QA/QC activities contributed significantly to pilot costs, driven by central-level personnel supervision and transport costs	QA/QC rollout costs were reduced due to decentralisation of supervision and quality monitoring to the district level; costs were driven up by higher RST kit cost during rollout and reduced economies of scale due to reduced RST uptake.

Prior to the pilot, antenatal syphilis screening was carried out using the laboratory-based RPR test when available. MCH HCWs sent blood samples to a laboratory for batched testing, with patients usually returning for results at a later date. In some rural facilities, non-laboratory workers performed RPR tests without a systematic QA approach. During the pilot, RST was performed by MCH HCWs who started same-day treatment based on a reactive RST test. National rollout guidelines described several diagnostic algorithms, depending on RST, RPR and Treponema pallidum haemoagglutination (TPHA) test availability at each site (Fig 1). None of the sites included in this evaluation performed TPHA.

Fig 1

Phased changes in syphilis testing and treatment algorithm in Zambia 2009–2012.

Legend: 4Cs: condoms, counselling, compliance, contact tracing, offer HIV test; RPR:Rapid Plasma Reagin; non-treponemal qualitative test; quantitative testing may also be available;RST: Rapid Syphilis Test, a treponemal test;TPHA*:Treponema pallidum Haemoagglutination assay, a treponemal laboratory test;BP: Benzathine Penicillin 2.4 megaunits IM. *The testing algorithm involving TPHA applied to tertiary care centres and was unavailable at sites included in this evaluation.

Data collection

This qualitative evaluation used a HCW questionnaire (S1 Appendix) and key informant interviews to determine the feasibility and acceptability of introducing RST into PMTCT services, comparing pilot and national rollout implementation phases. The concept of “feasibility”, drawn from the health technology literature, was defined as the process of RST programme implementation leading to end-user acceptance and utilisation, discussed further below [37]. Acceptability was defined as health workers’ positive satisfaction levels and their correct and consistent use of RST [41]. A health economic analysis of pilot and rollout phases is presented in our accompanying paper, Shelley et al. [43].

The questionnaire utilised both closed questions (including five-point Likert scales) and open-ended questions eliciting qualitative responses. The following domains were included: RST advantages and disadvantages, patient experience, organisational environment and workflow, training, skill and confidence in test performance and RST acceptability. For the rollout evaluation, the pilot questionnaire was adapted using current literature on technology acceptance; domains on QA/QC and supervision were added; and a topic guide was designed, covering the same domains as the questionnaire. Further adjustment, following a pilot interview and feedback from completed MoH/EGPAF supervisory visits, strengthened the internal validity of the study.

During the pilot assessment, SAS software (version 9.1) was used to randomly select four sites, two in each study district. They included a district hospital and both low and high volume health centres. Data on service and patient volumes, staffing numbers and location, time/motion studies and cost data were collected to document changes in syphilis and HIV testing and treatment and are reported elsewhere [41]. For the rollout evaluation health facilities were selected by convenience sampling (based on MoH guidance and practical limitations such as distance, staffing and vehicle availability) to reflect a range of urban and rural clinical settings with varying laboratory capacity. Review of records and key informant interviews took place at District Health Offices (DHO).

Face-to-face interviews were conducted in English with consenting health workers, with the exception of one interview, which used a translator. In November and December 2010, two EGPAF study staff administered the questionnaire to pilot HCWs, including seven midwives, two nurses and seven lay counsellors. Twenty four rollout HCWs (including four midwives, four nurses, five lay counsellors, one clinical officer, six laboratory technicians, three environmental health technologists and one psychosocial counsellor) were interviewed in August 2012 by one of the authors (É.A.), an independent researcher who accompanied MoH/EGPAF on supervisory visits. Responses were hand-written and rollout interviews were also audio-recorded and subsequently transcribed by the interviewer. Interviewees were selected by convenience sampling, aiming to interview four HCWs at each pilot site and any HCW who had ever performed RST at each rollout site. Using the MoH guideline, any errors identified in test performance were immediately remediated post-interview and discussed with the implementing partner. Informal interviews were held with key informants during MoH supervisory visits to gain understanding of the planning, implementation and costs of the national RST programme. Key informants included EGPAF senior and pilot study staff, MoH HIV/AIDS STI Programme staff, and DHO staff from Mansa and Kalomo districts. Data were collected in the form of field notes recorded after each informal interview.

Conceptual framework and analysis

A conceptual framework was created to guide the analysis, based on a model used by Asiimwe et al. which explored the feasibility of new health technology introduction. The framework divides the concept of feasibility into two inter-related domains, acceptance and usability [37]. Technology acceptance and usability originate in the study of human-computer interaction. Usability has been further broken down into various attributes [44,45]. Here, acceptance and usability were divided into six sub-domains: learnability, willingness, suitability, satisfaction, efficacy and effectiveness, attributes which have been described in other settings (Fig 2) [45]. The framework recognises that acceptance and usability may be influenced by factors related to the end-user (both HCW and client), the diagnostic tool and the health system. Health system factors include guidelines and training, quality monitoring and evaluation, supply chain, and policy, budget and planning.

Fig 2

Conceptual framework for the evaluation of RST pilot study and national programme in Zambia, adapted from Asiimwe et al (2012).

Legend: In this context, these themes were understood to mean the following: Learnability: how easy or difficult it was for HCW to learn to perform the RST, perform it accurately and learn about quality control and quality assurance; Willingness: Willingness of the HCW to perform the RST, to take part in the cascaded training i.e. being trained by or training other colleagues; willingness to take part in supervisory and quality assurance activities; Suitability: HCWs’ belief the RST test was relevant to their work and could be successfully integrated into existing services; HCWs’ belief in the appropriateness of the current supporting components of the RST programme i.e. training, supervision and quality maintenance; Satisfaction: HCWs’ satisfaction with the test itself, its impact on workflow and satisfaction with the supporting components of the programme; Efficacy: Ability of HCWs to implement same-visit testing and treatment (STAT), to incorporate the test and to integrate quality assurance and quality control activities into their workflow; Effectiveness: How the organisational and systemic environment, including implementation of policy, guidelines, supply chain and other logistics, impacted on successful delivery of the programme. In addition, how the social context (the community, patients and their partners) influenced programme delivery.

Data from the two evaluation phases were collected, entered and analysed by separate pilot and rollout teams. Quantitative data were double-entered, verified and cleaned using MS Access (2007–10) and Microsoft Excel 2003 respectively (S1 Dataset). Qualitative data were entered into MAXqda (V10) and NVivo 10, respectively, which are computer-assisted qualitative data analysis software packages. Two authors (M.G. and É.A.) used template analysis to independently analyse qualitative data from each phase [46]. The initial coding template was based on the questionnaire and included codes for: HCW acceptance and satisfaction, patient and partner experience of RST, training, workflow and integration with existing services and quality assurance activities and supervision. For this paper, the authors compared data from each phase under these theme headings and explored and refined emerging sub-themes through iterative discussion.

Ethical considerations

The pilot study protocol and consent procedures were approved by the University of Zambia Biomedical Research Ethics Committee (UNZAREC), the University of Alabama at Birmingham’s Institutional Review Board, the World Health Organization Research Ethics Review Committee and the Zambian MoH. The rollout study protocol and consent procedures were approved by UNZAREC, the London School of Hygiene and Tropical Medicine Research Ethics Committee and the Zambian MoH. To minimise use of personal identifiers, verbal informed consent was obtained from pilot HCWs and recorded on an approved consent form by interviewers. During rollout evaluation, written informed consent was obtained from each relevant District Health Officer and from each interviewed HCW using an approved consent form; verbal informed consent to participate was obtained from key stakeholders and recorded by the interviewer.

Results

In evaluating the pilot, 16 HCW interviews were conducted at four sites, located in two study districts (Mongu and Lusaka). In the rollout evaluation, 24 HCW interviews were conducted in two facilities in Kalomo and five facilities in Mansa (Table 2). Informal interviews were conducted with four senior administrative and management staff in each DHO. Using the conceptual framework, we compare and contrast results from pilot study and rollout evaluation according to the following themes: RST acceptance and satisfaction; RST usability; training and guidelines; quality assurance activities and supervision; health system and supply chain.

Table 2

Health care worker and facility data for pilot and rollout evaluation phases.

	PILOT: March—July 2010				ROLLOUT: March—July 2012
Facility Type^	UHC5	UHC6	RHC6	RHC7	UCH7	DH1	RHC3	RHC4	RHC6	UHC4	UHC8	RHC8	RHC9
Province	Lus	Lus	Wes	Wes	Sou	Sou	Lua	Lua	Lua	Lua	Lua	Lua	Lua
District	Lus	Lus	Mon	Mon	Kal	Kal	Man	Man	Man	Man	Man	Man	Man
Total Facility ANC* workers	n/a	n/a	n/a	n/a	10	41	18	4	2	n/a	n/a	n/a	18
HCW workshop-trained	1	1	1	1	1	6	1	1	1	3	1	1	1
HCW trained on-the-job	n/a	n/a	n/a	n/a	0	5	4	2	1	7	2	3	3
% total ANC staff trained	n/a	n/a	n/a	n/a	10	27	28	75	100	10	3	4	22
1st ANC Clients	n/a	n/a	n/a	n/a	n/a	638	271	144	n/a	480	n/a	n/a	n/a
Women screened with RST	n/a	n/a	n/a	n/a	n/a	154	103	114	n/a	430	n/a	n/a	n/a
% 1st ANC Clients screened	n/a	n/a	n/a	n/a	n/a	24.1	38	79.2	n/a	89.6	n/a	n/a	n/a
% of tests reactive	n/a	n/a	n/a	n/a	n/a	7.1	3.9	3.5	n/a	56.3	n/a	n/a	n/a
Number HCW interviewed	4	4	4	4	3	4	2	2	1	5	3	3	1
Cadre of HCW interviewed	4 MW	2 MW	MW	Nu	CO	Nu	EHT	Nu	EHT	3 MW	3 LT	Nu	PC
		2 LC	2 LC	3 LC	Nu	MW	LC	LC		LT		EHT
			Nu		LC	2 LT				LC		LC
Laboratory capacity?	yes	yes	no	no	no	yes	yes	no	no	yes	yes	no	no
RST kits available on interview day?	yes	yes	yes	yes	yes	yes	no	no	no	yes	no	yes	yes
RPR ever performed at facility?	yes	yes	yes	no	yes	yes	yes	yes	no	yes	yes	no	no

*ANC (Antenatal Care) worker includes all of the cadres listed below.

HCW: healthcare worker; CO: clinical officer; LC: lay counsellor; Nu: nurse; LT: laboratory technician/microscopist; MW: midwife; PC: psychosocial counsellor; RST: rapid syphilis test; RPR: rapid plasma reagin test; UHC: urban health centre; RHC: rural health centre; DH: district hospital; Lus: Lusaka; Wes: Western; Sou: Southern; Lua: Luapula; Mon: Mongu; Kal: Kalomo; Man: Mansa.

^See accompanying paper, Shelley et al., for additional facilities where only costing evaluation was performed.

RST acceptance and satisfaction

Overall, both pilot and rollout HCWs accepted RST as a suitable addition to existing services. One rollout clinical officer stated:

“I would encourage the MoH to embrace it and to roll it out in the whole country.”

Most HCWs were strongly motivated by the opportunity to deliver a better service and offer effective, same-day testing and treatment (STAT) to patients to ensure safer pregnancies. They reported perceived increases in testing rates and treatment access for pregnant women and their partners as well as perceived reductions in loss to follow-up and adverse pregnancy outcomes. They were satisfied with RST reliability and accuracy.

HCWs in both phases reported patient satisfaction with the RST but this was notably less in the rollout: 93.8% of pilot HCWs (15/16) and 62.5% of rollout HCWs (15/24) thought patients were somewhat or very accepting of the RST. A rollout lay counsellor commented that patients: “received (it) with both their hands.” Patient benefits reported by HCWs in both phases included: STAT, reduced clinic waiting time, reduced travel time and increased case detection and treatment. Reduced patient waiting time was more evident in the pilot; 68.8% (11/16) of pilot HCWs reported a reduction compared to 39.1% (9/23) of rollout HCWs. By contrast, several (26%, 6/23) rollout HCWs reported an increase in patient waiting times following RST introduction.

POC technology allows patients to directly observe their test result. Rollout HCWs reported that this, along with patient counselling, assisted in alleviating patients’ initial distrust of RST:

“Every change is quite hard to comprehend…We have to try hard so…people really adapt to new situations,”

rollout lay counsellor.

HCWs encountered some difficulties communicating RST test results to patients, in cases where tests were positive due to previously treated infection (and, therefore, requiring only a single dose of BP) or in cases of partner sero-discordance. Counselling was referred to repeatedly by all rollout interviewees as key to client and partner acceptance of testing and treatment. Many drew on their HIV counselling training.

RST usability

Health technology usability

Both pilot and rollout HCWs found the RST highly learnable; prior experience with POC tests added to their confidence. They rated the individual steps in test performance as easy or very easy.

“The [RST] procedure is simple, good and very short,”

pilot lay counsellor.

Compared to RPR testing, it was less “cumbersome” and time-consuming to perform, and easier to interpret. The mobility and convenience of RST allowed each stage of syphilis testing and treatment to be performed in one facility department, or out in the community:

“It’s like a supermarket, we do everything here…HIV test, results…Hb, the syphilis… the same day,”

rollout midwife.

However, some aspects of RST were less user-friendly. HCWs in both phases described difficulties reporting results within the appropriate time-frame and with interpreting results. These difficulties were more pronounced during rollout: only 30% (7/23) of rollout HCWs rated reporting results within the correct time-frame as “very easy” compared to 87.5% (14/16) of pilot HCWs; half of rollout HCWs were unsure how to correctly interpret weak positive results, which should be reported as positive. Several described repeating the RST or referring to the laboratory for confirmation of weak positive results; neither practice is in accordance with guidelines.

Usability—integration within existing services

Before RSTs were introduced at study sites, HCWs reported that, despite inclusion in national ANC guidelines, antenatal syphilis testing using RPR was inconsistently performed. RPR was performed at only 3 of 4 pilot sites and 6 of 9 rollout sites during the interviews. Most HCWs agreed RSTs were successfully integrated into facility PMTCT services, (16/16 pilot and 23/24 rollout HCWs) but there was high inter-site variability in algorithm followed and effect on workflow. The pilot integration pattern was standard: a MCH HCW (midwife, nurse or lay counsellor) ran both RST and HIV tests and returned results; a midwife or nurse then initiated treatment thus efficiently and effectively providing STAT. RPR confirmation was not included in the pilot protocol.

By contrast, integration at rollout sites depended on the facility’s health system level, available HCW cadres and laboratory capacity. Fewer rollout HCWs reported consistent delivery of STAT (83%, 20/23) compared with pilot HCWs (100%, 16/16). This may in part be due to variable patterns of task shifting. At larger facilities HCWs reported splitting testing steps amongst different HCWs and different departments, which led to delays in returning results and initiating treatment. In some cases confirmatory RPR was performed by the same HCW; in others, clients were required to return for RPR testing or results. Notably, midwives in one rollout facility reported a high reactivity rate of 56.3% on initial RST introduction. To verify results, they repeated RST or referred to the laboratory, where the lab technician reported repeating RST followed by RPR on all positive ANC samples, out of keeping with guideline algorithms. It also proved initially confusing to some rollout HCWs that the treatment algorithm differed in the case of negative RPR (Fig 1).

Pilot HCWs regarded RST as a user-friendly addition to PMTCT services which resulted in overall time-savings, despite the additional time required to counsel patients on both syphilis and HIV:

“…results are out in a short time…thus, cuts on time we spend on syphilis testing giving us more time to do other routine activities in MCH,”

pilot lay counsellor.

In the rollout, the effect on workload depended on both the role the HCW had in syphilis testing prior to RST introduction and the algorithm employed (Fig 3). Those previously involved in RPR testing reported overall time-savings irrespective of whether they then went on to perform RPR for some clients; those not previously involved felt RST introduction increased their workload and staffing levels were insufficient to manage:

Fig 3

Changes in Syphilis Testing before and after RST introduction and perceived effect on workload for different HCW cadres in Zambia.

HCW: Healthcare Worker; RST: Rapid Syphilis Test; RPR: Rapid Plasma Reagin; MCH: Maternal and Child Health; BP: Benzathine Penicillin 2.4 mega units IM. *Rollout HCWs used their own initiative in integrating RSTs into facility workflow. RPR confirmation was often done immediately where RPR was available and performed by the same HCW performing RST.

“I have a lot of clients …I have to see them… counsel them and do the RST to the woman and the man. So, it becomes difficult at times when I’m alone. That is in terms of staffing,”

rollout midwife.

As a result, they deferred syphilis testing to a later ANC visit or delayed reading test results.

Training and guidelines

Training experiences varied between pilot and rollout, with pilot HCWs more likely to have received training from EGPAF staff. 25% (4/16) of pilot interviewees reported attending an EGPAF training workshop; the remainder received on-the-job training from colleagues or EGPAF study staff; half received remedial booster training from EGPAF during monthly supervisory visits. 29% (7/24) of rollout interviewees attended the MoH/EGPAF-led training workshop. The remainder (17/24) received on-the-job training from colleagues.

According to MoH staff, the rollout training workshop emphasised on-the-job training but did not include specific “train-the-trainer” techniques. A training manual was provided to workshop attendees but few reported using it to train colleagues. Following training, only three (3/23) rollout HCWs used SOPs to aid accurate testing; all (16/16) pilot HCWs used them.

Most rollout HCWs found both workshop and on-the-job training acceptable. Several HCWs believed on-the-job training was too short and contained inaccurate or incomplete “second-hand information”. Some HCWs described colleagues’ distrust of them as trainers, their dissatisfaction at missing “official” training and their resultant unwillingness to perform the test:

“...they think, since you’re the one who went to the workshop you will be the one to do (the RST),”

rollout counsellor.

Quality assurance activities and supervision

Acceptance and usability of QA/QC activities were investigated as part of the rollout evaluation only. 100% of rollout respondents accepted QA/QC activities and recognised the potential benefits:

“I think it will help us know if the test kits are working properly and you would be confident that the result you would give would be the correct result,”

rollout midwife.

However, few rollout HCWs accurately recalled the QA/QC programme components described in RST training manuals. A small number reported performing internal QC to check test kit quality (e.g. ensuring the test control window was positive before reporting results). With the exception of one PMTCT counsellor who displayed self-efficacy in using RPR control samples to check kits, no HCW reported performing systematic external QC or completing proficiency panels.

Guidelines stated that district laboratories should prepare and provide positive and negative control samples for external QC and dried-tube specimens for proficiency panel testing. Laboratory facilities reported performing their own weekly QC; one district laboratory sent proficiency panels to local facilities but no results were returned. Reported barriers to the provision of district-led QA/QC were lack of trained staff, dedicated time, transport and reporting infrastructure.

Supervision differed from pilot to rollout. EGPAF made monthly support visits during the pilot and supported the MoH on quarterly supervisory visits during rollout. Rollout HCWs expressed willingness to undergo supervisory visits, describing them as an opportunity to learn and to correct mistakes. Most believed supervision was a suitable and effective tool, which boosted morale and encouraged HCWs to work more diligently and more accurately. Some dissatisfaction with supervision was expressed; a minority felt demoralized, intimidated or uncomfortable when observed performing tests.

Health system and supply chain

Supply of penicillin was consistent during the pilot but was less reliable during rollout: 12.5% (3/24) of rollout interviewees said stock-outs occurred once a month. RST kit supply was also less reliable during rollout; half of pilot sites (2/4) reported a stock-out on one occasion whereas almost a third of rollout sites (7/24) reported a complete RST stock-out during the month preceding interviews. EGPAF key informants reported that RST kits were not supplied through the usual national supply chain but rather through the DHO Pharmacy. This was to avoid incorrectly delivering to districts not yet involved in rollout but it led to confusion and ruptures in RST supply.

Discussion

This two-phased evaluation provides a detailed account of new POC test implementation, examining the transition from an NGO-led pilot project to the first phase of a national RST programme in Zambia. Novel, inexpensive and easy-to-use POC tests can address inequality in access to diagnostics in lower and middle income countries such as Zambia. However, implementation can be complex and requires a usable test, acceptance by HCWs and patients, good quality training, ongoing HCW support, quality monitoring systems and robust supply chains, which are more difficult to ensure for large-scale rollout with budget constraints [25,33,37,47]. Jafari et al described a lack of quality reporting of implementation research outcomes, other than accuracy, for syphilis POC tests, which our study helps to address [48]. This early programme evaluation takes place at multiple health system levels in four Zambian districts and may inform the Zambian MoH as they plan further phases of RST programme rollout.

Using HCW surveys and key stakeholder interviews, we examined the success of RST implementation from the perspective of “feasibility”, a concept borrowed from the health technology literature, which was further broken down into two components: “acceptance” and “usability” [37,45]. This study was not designed to report impact of RST introduction into PMTCT services in Zambia, which is reported elsewhere [41].

Overall, the study findings were positive with both pilot and rollout HCWs reporting that RST largely met our criteria for acceptability and usability. However, feasibility, as defined here, is contingent not only on end-user and technology attributes but also on health system and programmatic factors. We identified that several aspects of the rollout programme differed from the pilot, particularly training, testing algorithm, QA/QC, supervision, and supply chain; these may have negatively influenced the end-user experience and therefore the feasibility of the RST rollout programme.

Both groups found some aspects of RST non-user-friendly but rollout HCWs demonstrated significantly less correct and consistent use of RST, likely influenced by less exposure to MoH/NGO-provided training and regular supervision. Rollout on-the-job training quality appeared highly variable; the effectiveness of the cascaded training model utilised in the rollout was clearly limited. The inconsistent implementation of rollout testing and treatment guidelines may reflect the nature of the training and scaled-down supervision received by rollout HCWs compared to pilot HCWs and may have influenced the quality of care delivered. The POC literature emphasises the importance of high quality training to ensure test accuracy and reliability [25,30,31,49]. This is increasingly critical as HCWs are now expected to perform multiple rapid tests in parallel, each with varying specifications and manufacturers’ instructions. The WHO and global health workforce alliance advocate the cascaded model of training, or “train-the-trainer” (TTT) approach, as one way to meet the global health workforce crisis [50-52]. The effectiveness of TTT is supported by a recent systematic review. However, there is dearth of knowledge on TTT programme content, the training techniques that should be incorporated, or on programme sustainability [53].

Rollout HCWs were willing to take part in cascaded training, although some expressed resentment at missing the opportunity to attend an off-site training workshop. This may be linked to the widespread use of per diem payments for health workers in some low and middle-income countries, such that they have become an expected supplement, sometimes even exceeding health sector salaries. While these payments may encourage productivity and motivation, they have been shown to create conflict between HCWs and even to demotivate those who don’t receive them [54,55]. Supervision and monitoring of HCWs’ performance has been shown to increase their confidence in test performance and accuracy of test results [56], a finding echoed in this study. During the pilot, rates of both HIV and syphilis testing and treatment improved, which was attributed in part to regular supervision [34].

Overall, rollout HCWs were less satisfied with the increased workload resulting from the addition of another POC test to MCH services, but this was dependent on their previous role in syphilis testing and the algorithm employed. Our findings support a New Zealand study which showed that POC test introduction can have important, unforeseen implications for workforce planning [57]. Most rollout HCWs inaccurately described the steps involved in RST performance, confusing them with other POC tests. The assumption that integrating multiple testing programmes serves to strengthen health systems may not always hold true [34,58]. The new dual RST and HIV POC tests, which are currently under evaluation by the WHO and partners, may eliminate some of the problems engendered by running multiple POC tests in parallel [59]. Less frequent supervision may be one reason rollout HCWs reported greater difficulty with test performance and employed a variety of integration patterns, resulting in lower rates of STAT.

Rollout HCWs perceived lower levels of patient satisfaction than their pilot colleagues, which may result from the longer patient waiting times reported by some. HCWs also described difficulties around negotiating patient expectations and understanding of test results. Similar issues have been described in the literature on malaria rapid diagnostic test (RDT) introduction, particularly challenging patient expectations of malaria treatment in the face of a negative malaria test [28,29,36,60]. Specific training in counselling patients and partners around RST results may prove useful to include in further editions of RST training manuals.

The importance of QA/QC systems has been repeatedly emphasised in recent literature on HIV and malaria POC tests. Despite this, most HIV testing programmes in developing countries operate without them [61]. Ensuring reliable POC test results is of paramount importance as inaccuracies lead to incorrect diagnosis and treatment and negatively impact patient care and cost-effectiveness. Assuming the same issues regarding POC accuracy might beset RST implementation, QA/QC were incorporated into both pilot study and national rollout [26,49].

Anticipating that QA/QC could be an area of weakness during rollout, several modules of the MoH training guide were devoted to QA, QC and monitoring procedures [42]. All rollout HCWs interviewed understood the rationale for QA and demonstrated willingness to take part. However, the first round of rollout supervisory visits revealed limited implementation of these procedures. Several factors may have contributed to this: ineffective planning and communication (in that no specific person at facility or district level was nominated to coordinate QA/QC), lack of dedicated budget and logistics, lack of local expertise in this type of activity which is often undertaken by NGOs, or lack of efficacy on the part of local laboratory staff. This suggests that HCW training on QA/QC was not learnable or effective in its current form. Further research is needed to establish the ideal mix of RST QA/QC activities which is usable, acceptable, effective and cost-effective.

Supply chain issues may have impacted rollout success. For POC tests to maximise their potential, supplies of test kits, treatment and other consumables must be consistently available [25]. Weaknesses in both MoH district pharmacy and central medical supply chains were identified, which should be addressed before commencing further phases of the rollout.

Limitations of this study include the fact that it was a single-country case study whose findings may not be generalizable to other countries. Interviews and data analysis were performed by two separate research teams using slightly different collection tools which may reduce comparability of data from pilot and rollout phases. The pilot surveys were conducted by the implementing organisation and surveys in both phases were carried out during supervisory visits, increasing the risk of social desirability bias. The majority of study findings are based on a HCW survey; data are therefore response-based and susceptible to social desirability bias and measurement error. Convenience sampling of interviewees may have introduced selection bias. Quantitative data are also subject to measurement error. In addition this study design was not able to ascertain programme impact.

The Zambian RST scale-up model incorporated many success factors defined by Yamey in a recently proposed framework for successful scale-up of global health interventions [62]. However, we suggest that there are a number of aspects which ought to be refined before further rollout of the Zambian RST programme; our findings have been shared with the MoH and their implementing partner. The ideal training model, supervision type and frequency and mix of QA/QC activities for RST programmes has not been established and further implementation studies, which also examine impact, are required.

To strengthen the cascade model of training, the MoH could adopt a comprehensive “training of trainers” model incorporating adult education techniques and certifying workshop attendees as “local trainers”. Integrating training with HIV, malaria and Haemoglobin POC test training would facilitate emphasis on differing test characteristics and reduce HCW confusion. QA programme aspects would be strengthened by a distinct laboratory QA rollout strategy identifying specific responsible local personnel, and facilitating them with dedicated financing and logistics. Collaboration with those organisations already performing monitoring and evaluation of HIV testing may prove effective. Advocacy with the RST kit manufacturer to include positive and negative control samples could facilitate QC testing and address some of the difficulties experienced in communication and transport between laboratories and the surrounding health facilities. In addition, the supply chain must be strengthened.

Conclusion

This study provides insight into the challenges of integrating new health technology into existing health systems and explores the transition from pilot study to national rollout. It illustrates programmatic experience of implementing cascaded training and QA, QC and supervision activities. Use of a conceptual framework adopted from a paper on health technology “usability” facilitated the exploration of programme implementation and may prove useful in other settings. We demonstrate that it is feasible to implement a MoH-led rapid syphilis testing programme in Zambia, following a successful pilot, but that issues around training, QA/QC and supply must be addressed in planning for future phases of this and other programmes incorporating POC tests. The Zambian MoH has already demonstrated commitment to monitoring and evaluation practices and addressing these programme aspects should serve to strengthen the health system as a whole.

Topic Guide and Healthcare Worker Questionnaire.

(PDF)

Click here for additional data file.

Rollout Quantitative Dataset.

(XLSX)

Click here for additional data file.