REDCap mobile data collection: Using implementation science to explore the potential and pitfalls of a digital health tool in routine voluntary medical male circumcision outreach settings in Zimbabwe.

Background: Digital data collection tools improve data quality but are limited by connectivity. ZAZIC, a Zimbabwean consortium focused on scaling up male circumcision (MC) services, provides MC in outreach settings where both data quality and connectivity is poor. ZAZIC implemented REDCap Mobile app for data collection among roving ZAZIC MC nurses. To inform continued scale-up or discontinuation, this paper details if, how, and for whom REDCap improved data quality using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework.
Methods: Data were collected for this retrospective, cross-sectional study for nine months, from July 2019 to March 2020, before COVID-19 paused MC services. Data completeness was compared between paper- and REDCap-based tools and between two ZAZIC partners using two sample, one-tailed t-tests.
Results: REDCap reached all roving nurses who reported 26,904 MCs from 1773 submissions. REDCap effectiveness, as measured by data completeness, decreased from 89.2% in paper to 76.6% in REDCap app for Partner 1 (p < 0.001, 95% CI: -0.24, -0.12) but increased modestly from 86.2% to 90.3% in REDCap for Partner 2 (p = 0.05, 95% CI: -.007, 0.12). Adoption of REDCap was 100%; paper-based reporting concluded in October 2019. Implementation varied by partner and user. Maintenance appeared high.
Conclusion: Although initial transition from paper to REDCap showed mixed effectiveness, post-hoc analysis from service resumption found increased REDCap data completeness across partners, suggesting locally-led momentum for REDCap-based data collection. Staff training, consistent mentoring, and continued technical support appear critical for continued use of digital health tools for quality data collection in rural Zimbabwe and similar low connectivity settings.

Introduction

Zimbabwe is among the highest HIV/AIDS prevalence countries in the world, with a prevalence of 12.9%. As voluntary medical male circumcision (MC) is a safe, cost-effective method to reduce the spread of HIV,[2-4] the World Health Organization (WHO) set a target to circumcise 80% of men in 14 priority countries with high HIV burden, including Zimbabwe. Efforts by the Zimbabwe Ministry of Health and Child Care (MoHCC) to increase MC services are largely succeeding, as nearly 2 million men were circumcised between 2008 and 2020.[6,7] The ZAZIC consortium, comprised of the Zimbabwe Technical Assistance Training and Education Center for Health (Zim-TTECH), the International Training and Education Center for Health (I-TECH, Seattle WA, USA), the Zimbabwe Association of Church related Hospitals (ZACH) and the Zimbabwe Community Health Intervention Research Project (ZICHIRE), collaborates with Zimbabwe’s MoHCC in MC service delivery across the country, focusing on outreach sites and rural health center facilities in hard to reach regions. By 2021, ZAZIC performed over 500,000 MCs in 36 sites across 13 districts. ZAZIC sites are predominantly rural, with some peri-urban and a few urban settings, as described previously.[8,9]

Large-scale healthcare programs, such as ZAZIC’s MC program, require efficient, high-quality data to maintain patient safety and to accurately track progress towards targets. However, in low- and middle-income countries (LMIC) or low-resource settings, timely, high-quality data collection, delivery, aggregation, and, ultimately, data use for program improvement is a challenge, especially with paper-based systems.[10-14] ZAZIC’s quality assurance efforts address several data quality concerns. Yet, MC teams continue to note persistent challenges to data quality, including the workload burden and data delays, that may compromise program quality.[9,16]

REDCap is an open-source platform providing secure, web-based data capturing, management, and analytics.[17,18] Previous studies show that REDCap can improve data management in both research and routine healthcare contexts in LMIC settings, including those with limited connectivity.[19-21] An extension of the original REDCap, REDCap Mobile app, was launched in 2015 to allow for offline data capture; the features in the mobile application support its widespread use by over 800 global partners, including nearly 100 in sub-Saharan Africa. In Zimbabwe, where internet connectivity can be unpredictable, RED Mobile was selected for its expected performance in ZAZIC’s routine MC settings, many of which have connectivity challenges.

To provide more timely monitoring and evaluation (M&E) data for program improvement and target achievement, ZAZIC initiated a pilot program to use RED Mobile in settings where roving MC teams from larger health facilities performed MCs in surrounding schools, MC-focused camps, or rural health centers. Roving MC nurses entered daily MC program data from these outreach settings into REDCap Mobile, providing more timely access to M&E data. MC nurses also completed paper-based routine MoHCC MC forms at the facility in accordance with MoHCC policy.

In this retrospective, cross-sectional study, we applied the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to assess the impact of REDCap Mobile app data capture on the quality of data collected by roving MC teams during the first nine months of deployment, from July 2019 to March 2020 when MC service delivery was paused due to COVID-19 service delivery restrictions. The RE-AIM framework has been applied to assess and improve myriad, diverse global health programs; however, to the knowledge, RE-AIM has not been specifically applied to assess data quality improvement initiatives in routine settings. In this paper, therefore, we employ RE-AIM in a novel context to frame an evaluation of REDCap Mobile for routine data quality improvement in rural MC outreach settings. Use of RE-AIM provides a more nuanced understanding of the successes and failures of REDCap Mobile and provides insights for continued MC data quality improvement. The findings from this assessment may be applicable to other LMIC settings where routine data collection is challenged by inconsistent connectivity and poor cellular networks.

Materials and methods

REDCap mobile app development and deployment

ZAZIC paper-based, roving M&E tools were used to create the electronic REDCap Mobile M&E form, aiming to collect the same data in the same order as the paper form to ease training, data entry, and data management. I-TECH led initial REDCap Mobile form development; ZAZIC-wide testing and feedback led to early improvements to the REDCap Mobile app. Like the paper-based M&E form, REDCap Mobile data collection included a series of close-ended questions on productivity outputs, staff scheduling, adverse events, follow-ups, and MC productivity. The partially open-ended questions were also included in the app to solicit more detailed comments regarding service delivery, follow-ups, and demand creation. Site and ZAZIC MC nurses and M&E teams were trained in app use via a half-day, in-person training that included practice-based activities on how to input REDCap Mobile app offline data, troubleshooting, and data syncing when connected to internet. REDCap Mobile app was piloted in 4 sites from July-September 2019 before expanding to all ZAZIC sites in October 2019; there was no further paper-based data collection among roving teams. Roving nurses were provided follow-up mentoring by ZAZIC M&E leads during routine site-visits and WhatsApp-based support. ZAZIC M&E leads were trained on data management, report generation, and data downloading using both REDCap and the offline REDCap Mobile app. Data from the REDCap Mobile app was stored within a REDCap database and extracted quarterly by a ZAZIC program administrator for routine analysis.

Study setting

REDCap Mobile was implemented in all 36 ZAZIC sites from July 2019 to March 2020 using the REDCap Mobile app in offline mode, syncing daily or weekly when internet access was available.

Study population

The study population was comprised of ZACH and ZICHIRE roving nurses implementing MCs alongside MoHCC MC teams in outreach settings.

Data collection and analysis using the RE-AIM framework (Table 1)

Reach

Reach, or the number of those who participated in the intervention, was defined as the total number of REDCap Mobile roving reports submitted per month, per partner, and presented in descriptive statistics.

Effectiveness

Effectiveness, or the impact of REDCap Mobile on data completeness, was defined as completeness of roving team data in REDCap Mobile compared to previous paper-based data forms. Completeness, a marker for data quality, was calculated as a proportion: the number of non-missing variable fields divided by all variable fields per observation (roving event). Two sample tests of differences in proportions of data completeness, stratified by ZAZIC partner, were reported using RStudio.

Adoption

Adoption, or the absolute number and proportion of those willing to initiate REDCap Mobile app, was measured at the ZAZIC program level by determining the number and proportion of ZAZIC nurses, by partner, submitting data via REDCap Mobile from July 2019 to March 2020.

Implementation

Implementation, including both intervention fidelity and adaptation, was measured by the average number of reports nurses submitted each month throughout the observation period as well as by ZAZIC-led REDCap Mobile form adaptations to improve data collection.

Maintenance

COVID-19 restrictions resulted in a complete pause in MC service delivery in March 2020. Although maintenance would typically measure whether and how an intervention becomes institutionalized or part of routine practice, we assess ZAZIC’s intention to continue REDCap Mobile via internal partner communication, user feedback, and ZAZIC ownership of REDCap Mobile form optimization as proxy measures for maintenance.

Results

Reach

ZAZIC partners submitted 1773 reports via REDCap Mobile from July 2019 to March 2020 (Figure 1). Of these reports, 958 (54%) reports were submitted by Partner 1 and 815 (46%) reports were submitted by Partner 2. Thirty-five nurses submitted forms and recorded 26,904 male circumcisions: 19,090 from Partner 1 and 10,814 from Partner 2. The highest REDCap Mobile report yield was from November 2019 with 406 reports submitted: 247 from Partner 1 and 159 from Partner 2. Notable decreases in REDCap Mobile report submission by both partners occurred in December 2019 (festive season) and March 2020 (beginning of COVID-19 related restrictions).

Figure 1.

REDCap mobile reports submitted per month by program partner, July 2019 to March 2020.

Effectiveness

Using REDCap Mobile app, average data completeness rate among Partner 1 forms (Figure 2A) decreased significantly from 89.2% in paper forms (N = 161) to 76.6% completeness in REDCap Mobile app (N = 960), (p < 0.001, 95% CI: −0.24, −0.11). Average form completion for Partner 2 (Figure 2B) increased modestly from 86.2% in paper (N = 170) to 90.3% in REDCap Mobile (N = 813), (p = 0.05, 95% CI: -.007, 0.12).

Figure 2.

Average completion rate of paper forms and REDCap mobile forms by partner 1 and partner 2. Error bars indicate 95% confidence interval.

Adoption

Once implemented in July 2019, ZAZIC required all roving nurses to use REDCap Mobile for data collection, resulting in 100% adoption by ZAZIC program staff and discontinuation of paper-based reporting by October 2019.

Implementation

REDCap Mobile implementation was measured by the average number of reports submitted by nurses each month (Figure 3). During this period, 17 nurses from Partner 1 (Figure 3A) submitted a total of 958 reports from July 2019 to March 2020, an average of 11 REDCap Mobile reports per month. From Partner 2, 18 nurses submitted a total of 815 reports from July 2019 to March 2020, an average of 12 reports per month (Figure 3B). There was noticeable variation in report submission over time.

Figure 3.

REDCap mobile form submission among MC roving team nurses, July 2019 to March 2020.

Maintenance

Several programmatic activities suggest intention to maintain REDCap Mobile data collection beyond these initial nine months. First, in response to REDCap Mobile data quality weaknesses, ZAZIC led informal user interviews and gathered program team feedback on the app, providing support and refresher training tips via WhatsApp. Second, during the COVID-19 related pause, independent ZAZIC partner investigation into sub-optimal REDCap Mobile performance and form completion rates identified one nurse who submitted 50% of all Partner 1 REDCap Mobile using a previous REDCap Mobile app version with outdated form fields, decreasing the Partner 1 average data completion to 60.3%. In response, ZAZIC partner M&E teams completed another set of REDCap Mobile app improvements, including: 1) making all fields within the app form mandatory; 2) offsetting critical fields with attention-grabbing header fields; and 3) closely monitoring to ensure all users were submitting data using the most recent REDCap Mobile form version. These changes were rolled out by ZAZIC teams without help from I-TECH from October 2020 when service delivery resumed.

Discussion

Use of the REDCap Mobile app to improve data quality among roving nurses in a routine MC setting in Zimbabwe did not initially find improvements. However, application of the RE-AIM framework helped explain reasons for REDCap Mobile successes and failures and, therefore, contributed to additional app changes that ultimately resulted in improved data completeness over paper-based forms. These results support the utility of the RE-AIM approach and suggest that mobile apps may improve data quality in routine LMICs healthcare settings with limited internet access. Several key lessons learned merit discussion.

First, this assessment supports application of the RE-AIM framework as a practical tool for routine program evaluation, including for interventions in sub-Saharan Africa.[27-28] In this evaluation of a data quality improvement intervention, RE-AIM helped demonstrate reach, effectiveness, and adoption while identifying facilitators to REDCap Mobile implementation that, in turn, led to improvements in the REDCap Mobile app. As the teams noted subsequent increases in data quality, they affirmed their continued intention to maintain REDCap Mobile data collection. During the COVID-19 restriction-induced pause, ZAZIC reviewed data, iterated app improvements, and addressed issues in REDCap Mobile usability by further streamlining data collection fields. Once service delivery resumed, ZAZIC Partner teams led quarterly REDCap Mobile analysis and used the results to more closely supervise roving teams’ use of the REDCap Mobile app. Post-hoc analysis (excluding submissions from one Partner 1 nurse using the previous REDCap mobile form) from October 2020 to February 2021 (Figure 4), suggests that these locally-led changes resulted in additional improvement of REDCap Mobile data completeness for both partners. Assumption of local ownership and management, in accordance with the final component of the RE-AIM framework, suggests that ZAZIC partners may maintain REDCap Mobile as a data collection tool.

Figure 4.

Post-hoc analysis of average data completeness before (July 2019 to March 2020) and after additional improvements (October 2020-February 2021).

Second, several issues challenged the initial roll-out of REDCap Mobile app and may provide insights for other potential users of REDCap Mobile. First, data fields were not marked mandatory in REDCap Mobile, leading to nurses leaving blanks among fields required for quality improvement. Second, even with required fields, ZAZIC roving staff still required training and consistent mentoring to complete fields with a “no” rather than leave fields blank as was practiced routinely with paper forms. Lastly, consistent with other reviews of digital health challenges,[29-32] factors such as fluctuations in electricity, network access, tablet functionality, digital literacy and staff attrition affected REDCap Mobile report submission. These persistent challenges will likely extend beyond this roll-out period.

As MC roving services continue, several recommendations should be considered to improve REDCap Mobile replication, scale-up, and sustainability that are applicable to other settings. First, although REDCap Mobile data collection will likely improve M&E of MC outreach activities, consistent practice-based training on REDCap Mobile platform will be needed to increase user competence and confidence, including refreshers on data collection, form submission, and troubleshooting syncing issues. Second, ZAZIC Partner teams should regularly review REDCap Mobile data for quality to quickly identify opportunities for increased supervision to help raise the likelihood of sustained success. Third, ZAZIC REDCap Mobile managers need to archive previous versions of the app when updated versions are available and verify that all users adopt the most recent app version. Lastly, and perhaps most importantly, while REDCap Mobile is locally managed, external technical support is still required to identify and resolve some issues in system optimization. A more robust, virtual REDCap Mobile forum with heightened monitoring from LMIC experts is needed to both document challenges and leverage LMIC user crowdsourcing to identify viable solutions. In turn, this would help strengthen local REDCap Mobile technical capacity and oversight, promoting sustainability of open-source tools such as REDCap Mobile.

Limitations

This study has several limitations. First, the observation period for REDCap Mobile was interrupted by COVID-19, limiting the study period to nine months of significantly decreased MC productivity due to reductions in service delivery. COVID-19 also shifted HIV prevention resources away from MC to HIV treatment and care, reducing MC data quality efforts as well. Additionally, COVID-19 reduced access to critical REDCap Mobile training and mentoring, highlighting a potential limitation for digital tools. Furthermore, as the team transitioned to an electronic system, data entry for paper forms became more inconsistent, creating a potential bias in data completeness assessment for paper forms. Moreover, one individual’s submissions heavily skewed REDCap Mobile data and were omitted from post-hoc analysis. This exclusion reduces the bias from one individual but should not diminish the need for vigilance in ensuring all users update and refresh the app for optimization. Lastly, REDCap Mobile was chosen as the platform due to I-TECH experience and expertise; however, it is possible that other digital data collection tools may be better suited or more sustainable for the Zimbabwean rural context.[33-34]

Conclusion

Overall, use of the RE-AIM framework helped understand how, where, and why REDCap Mobile was beneficial to improve routine data quality in ZAZIC’s MC outreach settings. REDCap Mobile’s secure platform and ability to collect data offline also increased the tool’s utility, reducing connectivity and confidentiality concerns that may be similar in other healthcare contexts. These findings suggest that use of REDCap Mobile app for routine data collection should be maintained in the ZAZIC MC program and considered for replication in other LMIC rural or outreach settings.

Table 1.

Summary of RE-AIM dimensions aligned to REDCap mobile use for ZAZIC program M&E data.

RE-AIM Dimension	REDCap Mobile Outcome Measure	Level
Reach	Number of REDCap Mobile roving reports submitted per month, by partner	Individual
Effectiveness	Data completeness of nurse forms in REDCap Mobile compared to completeness in traditional paper-based forms	Individual
Adoption	Number and proportion of nurses submitting data using REDCap Mobile	Institutional
Implementation	Average proportion of monthly reports by nurse	Institutional
Maintenance	ZAZIC’s intention to continue REDCap Mobile use once MC procedures continued	Institutional