Challenges in data quality: the influence of data quality assessments on data availability and completeness in a voluntary medical male circumcision programme in Zimbabwe.

OBJECTIVES: To assess availability and completeness of data collected before and after a data quality audit (DQA) in voluntary medical male circumcision (VMMC) sites in Zimbabwe to determine the effect of this process on data quality.
SETTING: 4 of 10 VMMC sites in Zimbabwe that received a DQA in February, 2015 selected by convenience sampling. PARTICIPANTS: Retrospective reviews of all client intake forms (CIFs) from November, 2014 and May, 2015. A total of 1400 CIFs were included from those 2 months across four sites. PRIMARY AND SECONDARY OUTCOMES: Data availability was measured as the percentage of VMMC clients whose CIF was on file at each site. A data evaluation tool measured the completeness of 34 key CIF variables. A comparison of pre-DQA and post-DQA results was conducted using χ2 and t-tests.
RESULTS: After the DQA, high record availability of over 98% was maintained by sites 3 and 4. For sites 1 and 2, record availability increased by 8.0% (p=0.001) and 9.7% (p=0.02), respectively. After the DQA, sites 1, 2 and 3 improved significantly in data completeness across 34 key indicators, increasing by 8.6% (p<0.001), 2.7% (p=0.003) and 3.8% (p<0.001), respectively. For site 4, CIF data completeness decreased by 1.7% (p<0.01) after the DQA.
CONCLUSIONS: Our findings suggest that CIF data availability and completeness generally improved after the DQA. However, gaps in documentation of vital signs and adverse events signal areas for improvement. Additional emphasis on data completeness would help support high-quality programme implementation and availability of reliable data for decision-making. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

We assess data quality in terms of completeness and availability of routine programme data in a voluntary medical male circumcision (VMMC) programme, improving the evidence base for programme improvement.

We find that data quality audits (DQAs), in general, can help increase data completeness on key factors that ensure patient safety; however, gaps identified in data completeness could negatively impact patient care.

We demonstrate the value of routine DQAs to help increase the quality of programme data, adding to the global literature emphasising monitoring and evaluation.

The study was restricted to four convenience sites with available data due to limited time and financial constraints, reducing generalisability.

No qualitative study was implemented in parallel, reducing understanding of the barriers to data quality only at included sites.

Introduction

Accurate, timely and reliable public health data are essential for the delivery of high-quality healthcare services and for ongoing improvement of public health programmes. Though a large amount of effort has gone into developing tools that can assess the quality of public health data,1–3 few quantitative studies report on the effects of these efforts to motivate improvement in data quality in low-resource settings.4 5 Data quality remains weak in many low-resource, public health settings.6 We posit that careful review of programme data and interactive discussion of data weaknesses with programme implementers could help motivate positive changes in provider behaviours and programme documentation.

ZAZIC, a consortium founded in 2013 as a collaboration between International Training and Education Center for Health (I-TECH), Zimbabwe Association of Church-related Hospitals (ZACH) and Zimbabwe Community Health Intervention Research Project (ZICHIRE), cooperates with the Zimbabwe's Ministry of Health and Childcare (MOHCC) to implement an integrated voluntary medical male circumcision (VMMC) programme in 21 districts. Ensuring quality service provision is a critical component of ZAZIC's VMMC programme. From March, 2013 through July, 2015, ZAZIC conducted 83 706 male circumcisions (MCs).

ZAZIC implements routine monitoring and evaluation (M&E) activities to identify and address weaknesses in data collection, data reporting and data use. Previous research to measure implementation of, and improvements in, elements of VMMC surgical efficiency in four sub-Saharan African countries demonstrated that these types of M&E efforts may spur positive policy and programme changes.7 As a part of ZAZIC M&E efforts, periodic data quality assessments (DQAs) were performed to assess data availability and completeness. Through one-day site visits, aggregate data that was reported to the MOHCC was compared by ZAZIC to source document data at VMMC sites, identifying discrepancies and weaknesses to be addressed at the site level. Findings were discussed with facility staff at the end of the site visit, and a detailed report with action items was provided to facilities a few weeks later. Any items identified through the DQA were followed-up with phone calls, and if necessary, with an additional site visit. In February, 2015, I-TECH completed DQAs at 10 sites to assess data quality. DQA report findings were quickly shared with sites and follow-up visits were completed for sites where data quality was initially unsatisfactory.

We conducted a retrospective, cross-sectional, quantitative study in a subset of four sites. Our objective was to assess if DQAs were associated with improvements in data quality, focusing on data availability and completeness. Using concepts from organisational change theory, we posited that our DQA efforts could encourage adoption, implementation and sustainability of facility-level improvements in data quality.8 As our activity was action-oriented and included diagnosis (record review), action planning (feedback provided), intervention (follow-up when needed) and evaluation (ongoing DQAs),9 we hypothesised that targeted post-DQA feedback would lead to significant improvements in both measures of data quality.

Methods

Study design

In February, 2015, ZAZIC conducted DQAs at 10 of the 36 sites participating in ZAZIC's VMMC programme. Several factors influenced a site's likelihood of being selected for the DQA including geographical location and prioritisation of sites providing VMMC to a larger number of clients. Among the 10 sites that received a DQA, four were selected for this follow-up study with preference given to sites found to have poor data completeness during the previous DQA and geographical proximity (figure 1). The focus of this follow-up study was data availability and data completeness of the client intake form (CIF) (see online supplementary file 1). The CIF is the primary tool used by VMMC service providers to document VMMC procedures. CIFs measure seven dimensions of VMMC service provision: demographics, vital signs, preprocedure indicators, eligibility, procedure details, adverse events (AEs) and postprocedure follow-up.

Figure 1

Study flow chart. CIFs, client intake forms; DQA, data quality audit; VMMC, voluntary medical male circumcision.

This cross-sectional, quantitative comparison of CIF data before and after the DQA was conducted using a ZAZIC-developed evaluation tool (described below). CIF data completeness three months before the DQA was compared with results three months after the DQA. CIFs from November, 2014 and May, 2015 were selected to represent the period before and after the DQA, respectively. Selected CIFs were manually reviewed in August, 2015.

Data completeness evaluation tool

With the input of ZAZIC clinicians, 34 CIF variables were identified as the most important indicators of procedure quality (table 1). A completeness score was developed, with one point awarded for each of the 34 CIF variables that had a value recorded.

Table 1

ZAZIC DQA evaluation tool: list of key variables and their classification

Code	Documentation of variable	Subcategory (points)	Total
1	Name	Demographics (6)	34 points if all listed variables are recorded
2	Age
3	Type of site
4	Referred from
5	Address
6	Telephone number
7	Weight	Vital signs (4)
8	Temperature
9	Blood pressure
10	Pulse
11	Allergy	Preprocedure assessment (9)
12	Medications
13	Operation history
14	HIV test
15	Other diseases history
16	Bleeding history
17	General condition
18	Symptoms
19	Genital examination
20	Informed consent	Eligibility records (2)
21	Clinician initials
22	Date	Procedure records (6)
23	Circumciser's name
24	Assistant's name
25	Procedure type
26	Anaesthesia records
27	Procedure time
28	Adverse events	Adverse events (1)
29	Blood pressure	Postprocedure records (6)
30	Pulse
31	Analgesia given record
32	General condition
33	Next visit schedule
34	Signature for discharge

DQA,data quality audit.

Data collection and analysis

Monthly return forms (MRF), a MOHCC form containing aggregated data on monthly VMMC programme outputs for each site, were used to determine the expected number of CIFs from each site in the selected months. CIFs are organised in binders sorted by sequential client ID numbers to identify missing CIFs. Monthly record availability for each site was calculated by comparing the number of CIFs located to the number reported in MRFs. CIF data was extracted from paper forms, entered using Microsoft Excel and analysed using Stata version 12.0 (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP). Site-specific improvements in CIF availability pre-DQA and post-DQA were compared using χ2 tests, with t-tests used to assess changes in mean data completeness score pre-DQA and post-DQA.

Ethics statement

The Medical Research Council of Zimbabwe, the US Centers for Disease Control and Prevention (CDC) and University of Washington's Internal Review Board provided non-research determination for this routine programme evaluation activity. All patient-level data was de-identified. MRFs and CIFs are the property of the MOHCC and are stored by implementing partners in accordance with MOHCC standards for the routine care of data.

Results

Study population

According to MRFs, a total of 1461 MCs were conducted in November, 2014 and May, 2015 at the four sites. Among the 1400 (96%) clients whose CIFs were located, the median age was 17 years (IQR: 18, 20) ranging from 10 to 47 years old.

Record availability

Before the DQA, the availability of CIFs at sites 3 and 4 approached 100% while at sites 1 and 2, 92.0% and 79.8% of CIFs were available, respectively (table 2). After the DQA, high CIF availability was maintained by sites 3 and 4. For sites 1 and 2, record availability increased by 8.0% (p=0.001) and 9.7% (p=0.02), respectively. This is important because a missing CIF can indicate either failure to complete a CIF for a client or failure to appropriately store the form.

Table 2

Comparison of pre-DQA and post-DQA client intake form availability by site

Site	Time period	Total VMMC*	Number of CIFs located	Per cent of CIFs located	Improvement (%)†
Site 1	Pre-DQA	237	218	92.0	+8.0, p=0.001
	Post-DQA	161	161	100
Site 2	Pre-DQA	104	83	79.8	+9.7, p=0.02
	Post-DQA	220	197	89.6
Site 3	Pre-DQA	221	223	100‡	0
	Post-DQA	183	183	100
Site 4	Pre-DQA	74	73	98.7	+1.3, p=0.22
	Post-DQA	262	262	100
Total	Pre-DQA	636	597
	Post-DQA	825	803

*As reported on monthly return forms.

†CIFs located surpassed the total number of procedures due to duplication.

‡Results from χ2 test.

CIFs, client intake forms; DQA, data quality audit; VMMC, voluntary medical male circumcision.

Overall changes of data completeness

Results showed that sites achieved high overall data completeness after the DQA visit (table 3). Three of the four sites showed statistical improvement of data completeness on CIFs with the mean percentage of improvement of 8.6% (p<0.001), 2.7% (p=0.003) and 3.8% (p=0.001) in sites 1, 2 and 3, respectively. For site 4, the completeness of CIFs decreased by 1.7% (p<0.001).

Table 3

Comparison of overall pre-DQA and post-DQA data completeness by site

			Change over time
Site	Time frame	Data completeness (%)	Mean difference (%)	95% CI	p Value
Site 1	Pre-DQA	88.0	+8.6	(7.7% to 9.5%)	<0.001
	Post-DQA	96.6
Site 2	Pre-DQA	95.6	+2.7	(1.0% to 4.5%)	0.003
	Post-DQA	98.3
Site 3	Pre-DQA	94.6	+3.8	(2.9% to 4.7%)	<0.001
	Post-DQA	98.3
Site 4	Pre-DQA	95.3	−1.7	(−2.6% to −0.7%)	<0.001
	Post-DQA	93.6

DQA,data quality audit.

Changes in data completeness across key variables

Sites showed interesting variations in completeness on seven domains examined in more detail (table 4). Specifically, the eligibility section generally achieved 80% completeness after the DQA; however, completeness of site 3 on this section decreased by 16.8% after DQA. Although the preprocedure assessment section was more complete than other categories at the two time points, any history of bleeding (yes/no) had the lowest completeness among the preprocedure questions. The overall completeness of vital signs records in sites 1, 2 and 3 were over 98%; however, blood pressure (BP) records of site 4 were quite poor at both time points and decreased after the DQA (49.3% in November, 2014 and 8.4% in May, 2015). For site 4, this weak component contributed greatly to a low composite score for vital signs. Overall, in post-DQA CIFs, 97.2% of all incomplete fields on postprocedure records were due to lack of postprocedure BP records. Finally, 7.7% of all CIFs had incomplete AE information. Sites 1 and 2 records were nearly complete at both time points; sites 3 and 4 improved after DQA with improvements of 27.8% and 12.8%, respectively.

Table 4

The percentage of data completeness by domains of VMMC service provision

	Site 1 (%)		Site 2 (%)		Site 3 (%)		Site 4 (%)
Domains	Pre	Post	Pre	Post	Pre	Post	Pre	Post
1 .Demographics	94.2	92.8	99.6	99.4	97.9	99.9	99.8	99.7
2. Vital signs	74.9	99.1	94.0	98.2	87.9	100	86.6	77.1
Blood pressure	92.2	99.4	94.0	98.5	72.7	100	49.3	8.4
Pulse	87.6	99.4	92.8	98.0	97.8	100	98.6	100
Temp	38.5	98.1	90.4	97.5	77.1	100	98.6	100
Weight	81.7	99.4	98.8	99.0	99.1	100	100	100
3. Preprocedure assessment	98.3	97.0	98.9	99.3	98.0	98.4	98.0	99.3
Allergy	99.5	99.4	100	99.5	96.9	98.9	98.6	99.2
Medication	99.5	98.8	98.8	99.5	99.1	98.9	100	99.6
Operation	99.1	100	98.8	99.5	98.7	98.9	100	99.6
HIV test	100	99.4	100	100	98.7	100	98.6	100
Other diseases history	98.6	95.7	100	99.5	98.2	100	100	99.6
Bleeding history	89.0	80.1	92.8	95.9	91.9	88.5	90.4	95.4
General condition	98.6	100	100	100	99.6	100	97.3	100
Symptoms	100	100	100	100	99.6	100	97.3	100
Genital examination	100	100	100	100	99.1	100	100	100
4. Eligibility records	45.6	89.1	88.6	93.9	97.1	80.3	97.3	96.2
5. Procedure records	79.1	97.4	91.0	97.0	97.8	99.9	96.8	98.3
6. Adverse events records	96.4	98.2	94.0	97.0	69.5	97.3	85.0	97.7
7. Postprocedure records	96.0	98.8	94.4	98.6	91.3	100	91.8	83.7

VMMC,voluntary medical male circumcision.

Discussion

Our findings suggest that the DQA had a positive effect on data completeness: three of four selected sites demonstrated significant improvements in data completeness after the DQA. In addition, improved AE records and eligibility records suggest that site staff responded to feedback from the DQA, increasing their attention to data quality. In addition to assessing the effects of the DQA, this study also identified remaining gaps that merit attention for continued quality service provision. These persistent gaps remaining after the DQA may be caused by programme–level factors. For example, at site 4 we observed that many individuals had no BP recorded. When we shared this observation with facility staff, we learnt that the facility lacked adolescent BP cuffs, which was a major cause of the poor data completeness. This also highlights an example of how the DQA was able to identify an area where ZAZIC could take action to improve the quality of the VMMC programme. By procuring adolescent BP cuffs following the DQA, ZAZIC was able to improve VMMC programme quality and safety.

Although the DQA improved data completeness, other potential challenges were identified. During the CIF review, we observed that among 395 CIFs from one site, 67.0% and 64.6% of clients had identical preprocedure and postprocedure BP values recorded in November, 2014 and May, 2015, respectively (data not shown). These findings raise serious concerns about the validity of this data point as it is implausible that these patients would have constant measurements before and after the VMMC procedure. BP may not have been measured accurately, or may have been measured a single time with the information recorded as both the preprocedure and postprocedure BP. This issue may be present in other clinics and requires additional follow-up.

Furthermore, the most important objective of the preprocedure assessment is to identify contraindications and prevent AEs; however, we observed that MC contraindications are poorly ascertained on CIFs. Of the contraindications for VMMC,10 some can be identified on physical examination, such as anatomical abnormality and genital ulcer disease. However, other serious contraindications such as bleeding disorders cannot be identified without taking a careful medical history. Among all of the preprocedure assessments listed in the 1400 examined CIFs, 128 were blank and no individuals were documented as having a history of bleeding disorders. As an expected 6.6 per 100 000 men suffer from haemophilia worldwide,11 theoretically, about five cases of haemophilia would be expected in our routine VMMC practice out of 83 706 MCs conducted. Although we would not be expected to encounter a haemophiliac among the 1400 study records we reviewed for this manuscript, failure to completely and properly evaluate contraindications could add more procedure risk within the programme overall. Attention to patient's history must remain a priority. DQAs provide an opportunity for ZAZIC to highlight the importance of these questions through interactive discussions with clinicians at each facility. It is hoped that these action-oriented discussions could ultimately result in improvements in patient care.

Our analysis has several limitations. First, due to limited time and human resources, the study was restricted to four convenience sites with available data. Thus, conclusions may not be generalisable to other VMMC sites. Second, we were unable to explore the barriers that gave rise to low data completeness by interviewing relevant personnel, reducing our ability to translate these findings to practice. Additionally, we are making an assumption that the information recorded on the CIFs is accurate and that improvements in data completeness will correlate with improvements in clinical care and accurate data for decision-making. Finally, we did not assess other dimensions of data quality. For example, it was not possible for us to systematically evaluate the accuracy of data recorded on the CIFs, since we had no source of ‘true’ data for comparison. However, we believe that the improvements we documented in data completeness represent increased efforts by facility staff to correctly document information on CIFs. We believe that this effort will translate to improvements in data accuracy.

Conclusion

Following a DQA, record availability was either maintained or improved in four sites selected for examination of data completeness. Moreover, CIF data completeness improved in three of four sites. We identified several gaps remaining in CIF completion and recommend evaluation of data accuracy and authenticity as part of follow-up efforts. By integrating these assessments into routine practice, CIF data quality will likely continue to improve, providing quality data to inform VMMC programme decision-making in Zimbabwe. Moreover, by consistently providing feedback to those providing and documenting VMMC service delivery, DQAs may lead to sustained improvements in data quality and, ultimately, in the quality of VMMC care.