Usability and acceptability of a two-way texting intervention for post-operative follow-up for voluntary medical male circumcision in Zimbabwe.

BACKGROUND: Voluntary medical male circumcision (MC) is safe and effective. Nevertheless, MC programs require multiple post-operative visits. In Zimbabwe, a randomized control trial (RCT) found that post-operative two-way texting (2wT) between clients and MC providers instead of in-person reviews reduced provider workload and safeguarded patient safety. A critical component of the RCT assessed usability and acceptability of 2wT among providers and clients. These findings inform scale-up of the 2wT approach to post-operative follow-up.
METHODS: The RCT assigned 362 adult MC clients with cell phones into 2wT; these men responded to 13 automated daily texts supported by interactive texting or in-person follow-up, when needed. A subset of 100 texting clients filled a self-administered usability survey on day 14. 2wT acceptability was ascertained via 2wT response rates. Among 2wT providers, eight key informant interviews focused on 2wT acceptability and usability. Influences of wage and age on response rates and client-reported potential AEs were explored using linear and logistic regression models, respectively.
RESULTS: Clients felt confident, comfortable, satisfied, and well-supported with 2wT-based follow-up; few noted texting challenges or concerns about healing. Clients felt 2wT saved them time and money. Response rates (92%) suggested 2wT acceptability. Both clients and providers felt 2wT was highly usable. Providers noted 2wT saved them time, empowered clients to engage in their healing, and closed gaps in MC service quality. For scale, providers reinforced good post-operative counseling on AEs and texting instructions. Wage and age did not influence text response rates or potential AE texts.
CONCLUSION: Results strongly suggest that 2wT is highly usable and acceptable for providers and patients. Men with concerns solicited provider guidance and reassurance offered via text. Providers noted that men engaged proactively in their healing. 2wT between providers and patients should be expanded for MC and considered for other short-term care contexts. The trial is registered on ClinicalTrials.gov, trial NCT03119337, and was activated on April 18, 2017. https://clinicaltrials.gov/ct2/show/NCT03119337.

RCT Entities:   Population Interventions Outcomes

Introduction

Voluntary medical male circumcision (MC) is a critical HIV prevention intervention with global support for expansion across sub-Saharan Africa (SSA) [1-8]. Over 22 million MCs were performed across 14 sub-Saharan African (SSA) countries between 2008–2018 [9]. MC is cost-effective for HIV prevention [10-13] making investment in MC a priority across SSA [12, 14–17]. However, severe health system constraints threaten the quality and pace of MC scale-up [18-20], reducing the likelihood of meeting the ambitious global target of 5 million annual MCs in SSA [21]. Failure to reach MC targets means fewer infections averted [22, 23]. In addition, PEPFAR and UNAIDS identify scarcity of resources as a major risk factor threatening further control of the HIV epidemic [24]. New, cost-effective innovations that improve efficiency while maintaining or improving MC quality are crucial to attain HIV prevention milestones and safeguard scarce HIV epidemic control resources [25].

MC is safe and major complications are rare; thus, most follow-up visits are unnecessary [26, 27]. Adverse events (AEs) from gold-standard active surveillance [25, 28, 29] report 5–7% AEs [4, 30–32] while routine passive surveillance reports below the commonly-accepted 2% AE rate [33-37]. To ensure client safety, current global standards for high quality MC services nevertheless require at least one follow-up within 14 days [38, 39]. Post-operative guidelines in many countries, including those in Zimbabwe[40], strongly suggest three post-operative visits. However, with low AE rates, overstretched clinic staff likely waste invaluable resources conducting reviews for MC clients without complications. For clients, men healing well needlessly pay for transport, miss work, and wait for reviews. Innovations that safely target MC follow-up only for men with potential AEs would vastly reduce unnecessary visits, safeguarding critical resources for optimizing quality MC service delivery.

As a national MC implementing partner, the ZAZIC consortium, led by the University of Washington’s International Training and Education Center for Health (I-TECH) and local implementing partners, The University of Zimbabwe (UZ), Zimbabwe Association of Church-related Hospitals (ZACH), and Zimbabwe Community Health Intervention Research Project (ZiCHIRe), works in partnership with the Zimbabwe Ministry of Health and Child Care (MoHCC) to implement MC services in 13 districts [41]. Ensuring follow-up visit attendance, a routine MC program indicator, stretches scarce ZAZIC resources in an environment of few AEs: AE rates within both National and ZAZIC MC programs are less than 0.3% [33, 40]. Therefore, we implemented a randomized control trial (RCT) in two peri-urban clinics in Zimbabwe to test two-way texting (2wT) between patients and providers, allowing men healing without complication to opt-out of their routine post-operative visits. RCT design details are available [42]. Clinical results from the RCT, published previously, identified AEs among 1.88% of 2wT arm participants and 0.84% of control arm men, suggesting that 2wT could safely reduce MC follow-up visits, reduce provider workload, and serve as a proxy for active surveillance [27].

In this mixed-methods study, we aimed to explore 2wT usability and acceptability among 2wT patients and healthcare providers focusing on understanding how providers and patients interacted with the 2wT system and obtaining insights into the obstacles and facilitators of system use. High usability and acceptability of 2wT is crucial for this technology to successfully expand across Zimbabwe and the region. Usability has been defined as the extent to which a product can be used by specified users to complete a task effectively, efficiently and with satisfaction in a specified setting [43-45]. More recent usability definitions also include concepts such as learnability, encouragement for use, minimizing errors, accessible to a wide range of individuals, and easy to maintain [46, 47]. Although usability may be applied differently across contexts, for the purposes of this paper, we focus on the perceived quality of the user experience interacting with the 2wT system [48], considering both client and provider input. For acceptability, we focused on the usefulness of the system and user perceptions of the importance of system functions. In contrast with usability, acceptance should not be reported by users, but should be demonstrated via evidence that users employ the technology for the intended purposes [49, 50]. Therefore, as evidence of acceptability, we explored client interactions with the system and their perceptions of its usefulness as an alternative to in-person follow-up care.

Evidence of high usability and acceptability is consequential for MC programs in Zimbabwe and elsewhere that are considering 2wT interventions to support follow-up. If the hybrid approach to integrating computer-automated messaging with human-to-human 2wT proved highly usable and acceptable, this digital health intervention could be applied to improve care and reduce provider workload in other follow-up care contexts. We hypothesized that 2wT acceptability and usability would be high, aiding program scalability and replicability.

Theory of change

Ensuring 2wT usability and acceptability was critical to facilitating success from both provider and patient perspectives. Fig 1 maps how the 2wT intervention (program) aimed to influence individual factors (processes) in support of positive behavior change (individual outcomes) among male clients. 2wT targeted key individual-level constructs proven effective in previous HIV-related behavior change programs [51, 52]. First, 2wT providers enhanced post-operative counseling, encouraging self-efficacy to identify healthy healing [53] and control over the decision to return for in-person care if desired [54]. Then, via subsequent hybrid 2wT, providers connected with, and identified, men with potential AEs in need of provider triage and/or follow-up. These daily texts and swift individualized responses aimed to reinforce client motivation to respond [55]. Using theory to drive our intended effects, 2wT sought to empower men to correctly ascertain their healing progress and encouraged timely care seeking. We considered client engagement with the system as an indicator of usability and acceptability.

Fig 1

2wT theory of change.

2wT theory of program (intervention) change.

2wT technology overview

The 2wT system, the backbone of the intervention, was built using the Community Health Toolkit (CHT), an open source project which supports over 25,000 health workers across dozens of digital health activities. Apps built using the CHT work with or without internet connectivity, in any language, on basic phones, smartphones, tablets, and computers [56]. Since 2010 the CHT’s core framework was shaped by a rigorous and well documented human-centered design process [57]} and is highly configurable. The overall CHT (https://communityhealthtoolkit.org/) supports the vast majority of the critical needs outlined by the WHO Classification of Digital Interventions [58]. Specifically, the 2wT system provides a model of four core WHO-promoted interventions at the patient and provider levels: 1) automated and patient-to-provider interactive messaging; 2) SMS-based triaging of clients by nurses (e.g. for referrals to care); 3) daily client monitoring via SMS; and 4) longitudinal patient records (potential AEs, AE follow up, referral confirmation) and reporting (e.g. client response rates). The 2wT features were designed to support a streamlined workflow and high quality MC services while generating data to monitor program delivery. The usability and acceptability of systems built using the CHT cannot be taken for granted; it is important that they be tested for specified purposes, with specified users, in particular use contexts.

Methods

2wT intervention

Before full study implementation, message content was pre-tested for clarity of content and understandability in both Shona and English. Texts were reduced to 140 characters to ensure complete message content on multiple phone types and cell services. A pilot of 50 men assigned to texting intervention informed in-box modifications, SMS formatting, and message delivery optimizations (timing, frequency, language preferences). Pilot experience also identified wi-fi access via an internet dongle as the most resilient to fluctuations in electricity and cell networks. Automated messages were sent daily at 8am.

Implementation details of the prospective, un-blinded RCT study were described previously [27, 42]. In brief, ZAZIC follows all MoHCC protocols based on WHO guidelines including routine surgical MC follow-up on post-surgery days 2, 7 and 42 and adverse event (AE) management [59]. All MC care was provided free to clients from MoHCC. All men in the study (both 2wT and control arms) received all routine MC surgical care and both pre- and post-operative counseling at one of two public clinics: Seke South (peri-urban) or Norton (rural). However, in lieu of routine, in-person, post-operative visits, the 2wT clients received automated daily texts from days 1–13 and were asked to respond about their healing. Study texts are shown in Table 1. Daily responses were simple: only a single-digit response was required if the man believed he was healing well (“push ‘0’ and hit send”), and no further action was taken. If a 2wT MC client responded with suspicion of an AE (“push ‘1’ and hit send”), an MC nurse interacted with the client via SMS or phone, as requested. Clients chose the language of the automated texts at enrollment (English or Shona) but could engage the nurse in text communication using a mix of languages or common texting abbreviations. Clients were asked to return to the clinic if referred or desired. Follow-up communication at any time could be initiated by a client or the nurse. Participants in both arms were asked to return to the clinic on Day 14 for a study-specific review. No incentive was provided to study participants at enrollment. In recognition that sending texts in Zimbabwe costs approximately $0.05 [60], all study participants received a $5 phone card on Day 14 in appreciation for their participation.

Table 1

2wT scripted message content.

Text purpose	English	Type
Day 0 Enrollment Confirmation	Thank you for participating in this study.	Auto
8am: Days 1,3,4,5,6,8,9,10,11,12,13	How are you? Are you experiencing any bleeding, swelling, pus, pain, redness or wound opening? Enter 1 = Yes, 0 = No and press send	Auto
8am: Day 2	It is Day 2. Remove the bandage. Any bleeding, swelling, pus, pain, redness or wound opening? Enter 1 = Yes, 0 = No and press send	Auto
If Client SMS “1” to the above	Which symptoms? Bleeding, swelling, pus, pain, redness, wound opening, or something else?	Manual
If they say they have a suspected AE	Please return to the clinic. If you would like a healthcare provider to call you, beep us. Otherwise, text us your question here	Manual
If they stop responding to SMS	Please seek VMMC follow-up care at the clinic	Manual
8am: Day 7	It is Day 7. Are you experiencing any bleeding, selling, pus, pain, redness or wound opening? Enter 1 = Yes, 0 = No and press send	Auto
8am: Day 13, msg 1	Thank you for participating in the study. Please return to the clinic tomorrow for your short Day 14 visit and to receive your airtime	Auto
8:02am: Day 13, msg 2	How are you? Are you experiencing any bleeding, swelling, pus, pain, redness or wound opening? Enter 1 = Yes, 0 = No and press send	Auto
Day 14	Thank you for participating. Today is Day 14. Please return to clinic today for review and to receive airtime.	Auto

Quantitative methods

Of the 721 men in the RCT, 362 men were assigned to texting and included in this analysis. On day 14, a self-administered questionnaire was implemented with a subset of 100 of the 362 2wT MC clients (28%) to gauge satisfaction, acceptability, usability, and ascertain suggestions for intervention improvement. Responses of these 100 men were entered and frequencies explored in Excel for usability results. Data from all 362 are used to assess acceptability and usability as reflected in the system response rates and how the system was employed by the clients. Age and wage have plausible, a-priori relationships with usability and acceptability. Therefore, to determine whether the intervention was equally beneficial to all men, we explored the relationships between age and daily wage on both the number of text responses and potential AE texts among all 362 2wT texting arm men. The outcome variable, number of text responses, was a continuous measure of the number of daily texts to which a client responded, ranging from 0–13. The outcome variable, potential AE text, was a dichotomous variable defined as a client sending one or more potential adverse event responses to the daily text versus none. Both age and wage were categorized into quartiles. For daily wage, groups were: $0; 0-$5; $5-$15; and $16+. For age in years, categories were: 18–20; 21–24; 25–29; and 30 and above. We used linear regression for total text responses and logistic regression for potential AE text responses. Robust standard errors were used for all models. Coefficients and odds ratios (OR) were presented for linear and logistic regression models, respectively. Multivariate models included both wage and age variables. STATA 15.1.0 (StataCorps, College Station, TX) was used for statistical analysis.

Qualitative methods

We conducted brief (about 20 minutes) key informant interviews (KIIs) with eight clinicians involved in MC service delivery and 2wT-based follow-up to gauge acceptability; satisfaction; identify facilitators and barriers to program success; and ascertain suggestions for intervention improvement. Interviews were conducted at both 2wT intervention clinics: five interviews at Seke South and four at Norton. KIIs were audio recorded and transcribed. Thematic analysis [61] was employed, guided by a process of open coding based on anticipated responses suggested by the KII guide. Atlas.Ti software was used to create a spreadsheet of key themes, perceived barriers, and suggested facilitators to the program from KIIs. Usability questionnaires from the subset of 100 2wT texting men also contained several open-ended questions to ascertain perceptions of system use; those results were aggregated and grouped into themes using Excel.

Ethics

This study was approved by the Medical Research Council of Zimbabwe (MRCZ) and the University of Washington, Seattle, USA, Internal Review Board. All subjects, clients and healthcare workers, received comprehensive information regarding their voluntary participation in the study and signed a written informed consent prior to study enrollment.

Results

Quantitative

Study enrollment occurred from June 18, 2018 to February 11, 2019; follow-up ended March 13, 2019. RCT enrollment, assignment, follow-up, and analysis of clinical outcomes were previously detailed [27]. The RCT enrolled 721 men and randomized them 1:1 to intervention (2wT) and control (routine): 362 (50%) were in the 2wT arm (Table 2) and included in this analysis of usability and acceptability. More than half (57%) of the 362 men responded to their daily text when it arrived at 8am, with 86% of responses within 4 hours. Over 92% responded to at least one daily text, showing high system acceptance. Response rates varied over time: approximately 70% of men responded on Day two while half responded on Day 13. Potential AE responses peaked on Day 2 and diminished thereafter (Fig 2). The nurse had 30 phone interactions with clients: 19 “call me back” requests were responded to and 11 calls were made to the nurse, directly, with a concern. Over the course of the intervention, clients sent over 1200 free text SMS to the 2wT nurse.

Table 2

Characteristics of 2wT arm men.

Characteristic		2wT	Full study
		N = 362 (%)	N = 721
Age category (years)
0–20		92 (25.4)	207 (28.7)
21–24		90 (24.9)	178 (24.7)
25–29		75 (20.7)	133 (18.5)
30+		105 (29.0)	203 (28.2)
Site
	Seke South	300 (82.9)	594 (82.4)
	Norton	62 (17.1)	127 (17.6)
Language selected for 2wT daily message
	English	154 (42.5)	281 (39.0)
	Shona	208 (57.5)	440 (61.0)
Cell company
	Econet	321 (88.7)	636 (88.2)
	Other	41 (11.3)	85 (11.8)
Daily wage category ($USD)
	0	108 (29.8)	254 (35.5)
	>0–5	86 (23.8)	155 (21.7)
	>5–15	98 (27.1)	179 (25.0)
	>15	70 (19.3)	128 (17.9)

Fig 2

Response rates over time, by response type.

Closed-ended client survey responses suggest that most clients felt confident, comfortable, satisfied, and safe with SMS follow-up (Table 3). Few noted challenges with texting or were concerned about their wound healing without in-person review. Clients also felt that 2wT saved them time and money. As compared to alternative texting schedules, men preferred one daily text (76%), and they preferred that the text arrive in the morning (86%).

Table 3

Responses on 2wT acceptability and usability from 2wT clients.

(N = 100)	Mean Likert score
I was clear on when to SMS, call, or return for care	4.68
I felt safe with the daily SMS	4.63
I was comfortable with this approach to MC follow-up care	4.59
I understood the SMS process well	4.57
I felt prepared to remove my bandages on Day 2	4.5
I felt confident that I would get the care I needed at the clinic	4.31
I felt that I could come to the clinic when I wanted	4.14
I understood the signs of poor healing	4.05
I want fewer SMS	3.05
I wanted more SMS on signs of adverse events	2.83
I wanted help to remove my bandages on Day 2	2.73
I was worried about my wound healing	2.56
I had challenges sending SMS	2.33
I had challenges receiving SMS	1.89
I wanted more in-person follow-up	2.42
I am worried about my privacy receiving these messages	2.77
No mandatory follow-up visits saved me money	4.35
No mandatory follow-up visits saved me time	4.35
I feel satisfied with the SMS follow-up	4.61
I would recommend SMS follow-up to my friends	4.50

Both age and wage were associated with increased text response rates among 2wT men in univariate models (Table 4). On average, men from ages 21–24, 25–29, and 30+ sent an SMS on 2 more days than their younger peers, ages 18–20. Men with daily wages of $5-$15 or more than $15 sent an average of 1 or 2 more messages, respectively, over the follow-up period than those without income. In the multivariate model, age remained significantly associated with increased number of daily SMS responses. Although daily wage of $5-$15 compared to no income was associated with an increased likelihood of sending a potential AE message, this relationship was not significant when controlling for age in the multivariate model. Age was not associated with the likelihood of sending a potential AE text.

Table 4

Factors associated with texting response rate and potential AE texts.

N = 362	Total SMS responses† Coefficient (95% CI)		Potential AE text†† OR (95% CI)
	Univariate	Multivariate	Univariate	Multivariate
Age (years)
0–20	ref	ref	ref	Ref
21–24	2.24*** (1.07–3.41)	2.06*** (0.87–3.25)	1.64 (0.90–2.98)	1.63 (0.88–3.01)
25–29	1.83** (0.59–3.08)	1.49* (0.15–2.84)	1.47 (0.79–2.76)	1.39 (0.71–2.74)
30+	2.38*** (1.25–3.51)	1.91** (0.56–3.32)	1.64 (0.92–2.92)	1.66 (0.85–3.23)
Wage ($)
0	ref	ref	ref	ref
0–5	0.97 (-2.00–2.15)	0.81 (-0.36–1.98)	1.54 (0.86–2.74)	1.48 (0.82–2.68)
5–15	1.39* (0.26–2.53)	0.75 (-0.47–1.98)	1.84* (1.05–3.22)	1.56 (0.86–2.89)
15+	1.93** (0.71–3.15)	1.08 (-0.32–2.38)	1.11 (0.60–2.07)	0.89 (0.44–1.81)

*p<0.05

**p<0.01

***p<0.001.

† Results from linear regression.

†† Results from logistic regression.

Qualitative

2wT clients

For the open-ended responses among the 100 men who completed the usability survey on day 14, 45 responded (45%) to the question on formatting improvements; almost all responses suggested no change. Of the 42 men who responded about improved message content, the vast majority also suggested no change. However, addition of the day number (e.g., “this is the day 1 text”) was suggested while a few others noted more information on the terms, e.g., “puss,” should be explained. For the question on how to improve SMS follow-up, most responded that they liked the system as it is, but several noted that they would like to receive a call if they did not respond by text. Two men requested to be given airtime on Day 0. Overall, most men responded similarly to these participants who wrote, “messages reduce time and if there is a problem you are given time to communicate. It's just an efficient way of communicating.” Another client expressed that, “the follow-up process is good because you can do what you want and also can make a monetary saving by not visiting the clinic.”

Healthcare workers

Additional insight from the eight KIIs conducted after the completion of the full study suggested several themes to inform 2wT scale-up. Overall, the healthcare workers felt favorably towards 2wT, as men “don’t like coming for follow-up review visits.” Another KI revealed that, “in VMMC we have challenges with adult clients, they don’t come [back for reviews], the numbers are not as you would want them to be.” As a result, 2wT was welcomed by the HCWs, one of whom stated:

We are looking forward to positive results that will lessen the burden on the service provider and improve post-operative quality care on the circumcised client and avoid any AE because of the on-going communication and feedback coming from the client.

The KIs believed that 2wT reduced their overall workload. During the study, they “only did a few reviews since clients were reviewed via cellphones.” Scaling up 2wT with fewer post-operative reviews could lead to “reducing burnout on health care workers. If you don’t go for reviews it means you have enough time for other duties at the clinic.” Several clinicians echoed the opinion that that 2wT could “reduce the demand for staff to be going out and cut on costs for going to the field looking for clients when doing reviews."

KIs felt the 2wT intervention made the men partners in their healing process and “gave the patients responsibility to cater for their health.” It also gave the clinicians more confidence in their patients and “actually helped us realise that men are actually more capable of taking care of themselves.” Another KI noted that the intervention empowered men to be more vigilant in wound care.

“We used to doubt that clients could manage themselves but because of the study we realized that people were responsible for their own health… They were empowered. They become part and parcel of the programme.”

KIs also expressed several advantages specifically for clients. The KIs thought that 2wT would save men money “by reducing transports costs. And also they were not phoning which means they were only using texts which cost only 5 cents.” They also felt that clients saved time, as “most of our clients are self-employed so the time wasted in returning to the clinic was reduced which was positive to the clients as well as to the service provider.” When asked if they were concerned that men would ignore signs of AEs, one KI summed bluntly: “VMMC is a very sensitive area, men are worried about their organs. If they have got a problem they react, so there won’t be any problems that they would ignore and not seek help.”

Lastly, clinicians shared several suggestions for 2wT scale-up. First, clients need clear instructions and post-operative counseling on wound care, bandage removal, and how to text to be able to safely use 2wT. Furthermore, clients needed reminders that 2wT was not for after-hours or emergency care. Several KIs noted potential issues for more rural areas with less reliable cell network or consistent electricity. One KI emphasized the need to maintain active tracing for non-responder, as provided currently for those who do not return to care. Lastly, although few clients noted financial challenges, four KIs suggested providing airtime on Day 1.

Discussion

The results from mixed-method assessment of 2wT usability and acceptability strongly suggest that the hybrid automated/human-to-human SMS system is both acceptable and easy to use, complementing the clinical outcome data that showed 2wT between providers and clients helped ensure healthy wound healing and reduced unnecessary in-person visits [27]. The automated 2wT daily messages reminded clients of the signs of poor healing and encouraged wound observation for the most critical 13 days after MC, the period during which 95% of AEs occur [62]. Clients thought 2wT provided a similar level of reassurance as routine visits, but without the added time or money incurred from an in-person review. Clinicians believed 2wT empowered men to engage in their own healing, assuring them that their clients were capable of identifying potential problems and seeking care when they wished rather than on a mandated schedule. 2wT would not entirely replace clinical care or oversight, but could reduce the burden of unnecessary care on overstretched healthcare workers and reduce client costs while encouraging visits for those who desire or require care. 2wT’s usability and acceptability appears to have helped ensure success. There are several lessons learned for future 2wT within the MC context or other shorter-term follow-up situations that may benefit from this type of direct provider-to-patient SMS.

Although digital health interventions document challenges in retention and adherence [63], 2wT improved upon previous texting interventions in several substantial ways. First, the technology, itself, was designed with usability at the forefront. Although mobile health (mHealth) interventions are growing in popularity [64-70], many short message service (SMS)-based health promotion efforts blast pre-defined messages to many people simultaneously [71-73], removing patients’ ability to communicate back with healthcare workers [74]. By contrast, designing technology to support or augment end user skills and capabilities, as we did by enabling clients to ask questions and initiate care via SMS, is recognized as a central feature of human-centered approaches to system design [57]. Second, two-way texting with a mix of automated and human-to-human messaging optimizes for individualized content, efficiency for staff, and rapid follow up for the fraction of cases with complications, enabling interactive communication between clinicians and patients [75, 76]. Third, the short 13-day 2wT intervention maximizes client adherence, while common barriers such as phone theft, damage, and change of phone numbers [77] are minimized over the brief follow-up period. Fourth, the 2wT MC intervention implemented core characteristics of mHealth excellence, including client accessibility and acceptance; low technology costs, effective local adaptation, strong stakeholder collaboration, and government partnership for sustained impact [78]. Lastly, due to the high proportion of routine MC visits rendered moot when men confirm healing well via SMS, 2wT for MC realizes immediate gains in efficiency, unlike texting for patient education or behavior change.

The familiarity and flexibility of SMS (human-to-human messaging) likely played a role in usability and acceptability. All enrolled clients received their initial text while still in the clinic with MC healthcare providers, providing a personal connection and immediate reassurance that the texting system worked. Post-operative counseling, enhanced with emphasis on expected text responses and the healing process, likely increased client confidence in their ability to recognize and report potential AEs via SMS and act on both information and guidance provided via text. The automated daily text messages also appeared to comfort clients, helping them feel attended to without the need for an in-person visit. Few men requested calls or called the nurse directly, likely suggesting that the men felt that their needs were well met via SMS. For providers, the automated daily SMS significantly reduced the burden on the study nurse as the system, itself, handled the majority of outgoing messages. This enabled the nurse to prioritize time and communication with the clients who suspected an AE or who, for any other reason, initiated a text conversation or requested a call. By reducing the burden on the nurse, and removing the need for further communication with clients who responded with “No AE,” she was able to respond to many more clients than she would have been able to manage in person at the clinic.

This intervention increased male engagement in care, a recognized need in HIV programs. Despite the fact that men’s uptake of healthcare services is lower than that of women [79], interventions that increase male involvement in HIV-related care are absent in much of SSA. Men are less likely to enter the HIV care cascade [80, 81], especially in rural areas [82], threatening HIV prevention efforts. mHealth interventions, like 2wT, show progress in increasing male engagement with care[83, 84]. 2wT appeared acceptable, feasible, and empowering for male clients [85], successfully garnering active participation from men in post-MC healing. 2wT in Zimbabwe appears beneficial to most adult men involved in the RCT, with minimal changes in effectiveness due to income or age. Older men were more likely to respond to the texts, but this small effect does not seem programmatically meaningful. Although clients were not given airtime until day 14, response rates of almost 93% suggest that men were able and willing to pay SMS costs up front. Clients likely paid an average of $0.05 per SMS, and were asked to respond to a daily text for a minimum combined cost of $0.65, or far less if they bought text bundles. Many men exchanged further texts with the nurse, suggesting that men were able and willing to pay these SMS costs in lieu of an in-person visit that would likely require travel time, transportation, and absence from work.

The results of the quantitative and qualitative analysis suggest several key improvements for 2wT at scale. First, the nurse managing the clients urged future scale-up interventions to clarify that 2wT is not for emergency or after-hours care. This could easily be completed by an automated response recommending alternatives for seeking care after hours. Similarly, at scale more nurses would be needed to rotate responsibility for client management via the 2wT system in anticipation of client communication requests and timing; further user interface optimizations could support these programmatic changes. Moreover, most mobile network operators offer ‘reverse billed’ SMS numbers that are free to clients and paid for by the sponsoring organization; while costly for a small study, this would be viable at scale. These steps may be critical for 2wT at scale to reduce the workload on the nurse managers and ensure continuous client care.

It is also worth noting several limitations to the 2wT intervention. The majority of study clients liven in an urban setting, and it remains to be investigated whether 2wT interventions may be similarly usable among rural populations. In settings where phone ownership remains low, the efficiency gains of 2wT may be limited to a subset of the total client base. For safety and verification, our study protocol also excluded any client who did not have a phone with him at the time of enrollment and who did not receive a confirmation message from the 2wT system before leaving the clinic. 2wT at scale could opt for less strict inclusion criteria to enroll more clients and increase 2wT access; however, that could introduce patient care and verification concerns if hardware malfunctions, SIM card issues, poor electricity access, or poor cell networks jeopardized potential client contact. Consideration of these constraints or other challenges such as phone sharing were beyond the scope of the present 2wT study. Lastly, although we did not encounter high rates of refusal, there are privacy and stigma concerns among men opting for MC that affect follow-up rates. However, if men were allowed to access 2wT at any time during the healing period, without requiring 2wT enrollment at registration, 2wT could provide men with concerns an alternative pathway to access care, improving patient outcomes overall. Despite these limitations, and combined with the clinical results demonstrating safety [27], it is clear that 2wT warrants further exploration for replication and scale-up in other MC care contexts and settings.

Conclusions

Our usability and acceptability results show that post circumcision 2wT interaction for follow-up care is highly useable and acceptable for clients and healthcare professionals, contributing substantially to the successful achievement of MC program efficiencies. By designing a hybrid workflow, we were able to integrate the efficiency benefits of automated messaging with the familiarity and flexibility of human-to-human conversational texting. While our usability and acceptability findings are grounded in the clinical and operational particulars of MC, our results are largely attributable to the software and hybrid 2wT workflow rather than to a specific disease or condition, contributing to the growing applicability of 2wT in healthcare interventions in developing countries, including SSA [86-89]. Similar automated/human-to-human 2wT merits further study in other care contexts, such as childhood diseases, respiratory infections, or post-operative care that could similarly benefit from intensive, direct provider-to-client communication.

Based on our mixed-methods findings, we would emphasize four characteristics of health programs that may benefit from the hybrid automated/human-to-human workflow: 1) facility-based follow-up care is perceived as burdensome to patients and/or providers; 2) most clients heal, adhere, or otherwise succeed with routine care and standard counseling, minimizing risk; 3) clients with complications or questions are likely to be able to self-care or triage safely and effectively with the support of a remote clinician; and 4) the follow-up period is brief enough to sustain engagement and minimize obstacles due to lost phones, changed phone numbers etc. Programs with these characteristics may be more amenable to a new kind of task-shifting, in which digital tools enable clients and their care givers to be more informed, engaged, and capable of self-care in situations that otherwise might have required a visit to a health facility.

The open-source nature of our intervention should aid efforts to replicate this intervention or adapt it for other settings and care contexts. However, 2wT adaptations, replications, and advances are not without risks and not without costs. Further research on hybrid 2wT interventions across diseases and care contexts, with attention to the implementation challenges that may limit scale-up, could further establish the benefits and limitations of this new approach to digitally supported self-care in the community.

(DOCX)

Click here for additional data file.

(XLSX)

Click here for additional data file.

17 Feb 2020

PONE-D-19-27561

Usability and acceptability of a two-way texting intervention for post-operative follow-up for voluntary medical male circumcision in Zimbabwe

PLOS ONE

Dear  Dr Feldacker,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by 30th March 2020. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Kwasi Torpey, MD PhD MPH

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for letting us know that Figures 1 and 3 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a)    You may seek permission from the original copyright holder of Figures 1 and 3 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b)    If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Important topic. However the way the mansucript reads currently, it requuires major edits to give it focus on the one objective of this study. To assess usability and acceptability.

Overall: Important topic and research results. The manuscript needs major editing to improve clarity.

Abstract

1. Objective that are not part of this paper are included. Need to focus only on what is included in the current manuscript. Other objectives of the overall study could be mentioned in the methods section.

2. Remove numbers on the aims – and they should only be two – according to the tittle: usability and acceptability of the 2wT

3. Conclusion: Findings on “men with concerns…” is not included in the results section of the abstract

Main Manuscript

Introduction

1. Lines 94 – 109 – should be part of the methods section

2. Lines 87 – 97 – should be in the setting section of the manuscript under methods

3. Lines 123- 124 – revise to indicate objective/aim of this study.

Methods

1. Theory of change section should be moved to the introduction section

2. Line 150 - 2wT Technology overview – reads like a descript of the intervention

3. Intervention – reads more like study procedures.

4. Line 176 – Only Shona and English were used in the testing, what about other languages like Ndebele. What was done if the participant did not understand any of those two languages (Shona and English)

5. Line 213 – Quantitative methods: Open-ended questions are part of qualitative research. Likert scale questions and demographics possible quantitative. It might be better in this section to put data collection from MC participants, and data collection from healthcare worker

6. Lines 231 – 236: Data analysis and the statistical analysis could be included in this section.

Results

1. Healthcare workers perspective – too long, and can be summarized highlighting the major findings

Discussion

1. Lines 410 – 415 – edited and be moved to study procedures or intervention description

2. Line 467 “it is clear that 2wT is warrants further” delete the word is

Reviewer #2: This manuscript addresses concerns regarding the utility of adherence to post-imperative follow-up visits after VMMC. The paper would be strengthened with better organization and more focus on the quantitative outcomes. Specific comments are below:

1) Line 67 – More recent data than 2017 are available for MCs done in SSA.

2) Line 77 – I would specify that when you are only referring to post-operative AEs, not intra-operative.

3) Line 81 – I would specify the post-operative recommendations recommended by Zimbabwe’s MOHCC.

4) Lines 94-108 – While this is important information, most of it should appear in the methods and results sections, rather than the introduction.

5) Line 124 – The authors refer to “Aim 3” without any context, so this does not make sense to the reader. I would recommend dropping the references to Aims here and throughout the paper.

6) Line 138 – There might be an error with the figures and tables, as Figure 1 is the CONSORT table, not a mapping of the intervention.

7) Lines 150-171 – This information should be incorporated into the background/introduction rather than the methods.

8) Lines 174-190 – This detail about pretesting is interesting but not critical to the reader to understand the piloting and refining that took place as part of this study. I would recommend deleting this section and reducing this text to a sentence or two.

9) Line 197-198 – This description of the control arm is confusing, as it appears that the intervention arm had the same follow-up schedule (additional visit on Day 14) I would recommend a clearer description of the arms, or an additional table if possible.

10) Line 223 – Specify that the wage groups are for daily wages earned.

11) Line 230 – It’s unclear if these are facilities, sub-districts, etc.

12) Line 255 – There is a typo—it says “participate” when it should be “participant”.

13) Line 270 – Inconsistent use of the oxford comma here and throughout.

14) Line 275 – This table would be easier to interpret if it was in descending order by Likert score or grouped into relevant categories.

15) Line 281 – I don’t see how this point is relevant to the broader findings of this study.

16) Line 293 – 369 – The qualitative findings are not compelling enough to add to the overall value of this paper. I would recommend either deleting them and focusing on the quantitative findings, or developing this section further to have more robust findings and conclusions.

17) Results section - I assume that this was previously described in more detail elsewhere, but a major gap in this manuscript is that the AE results by arm are not reported in the results and the findings in this paper do not engage with the overall findings from the main study outcomes. The reader would also need to know how many AEs were missed through the self-diagnosis.

18) Line 377 – Why is Day 13 most critical?

19) Line 384 – The content of these daily messages need to be presented somewhere in this manuscript.

20) Line 388 – I think you mean “substantial”, not “significant”.

21) Line 467 – It is impossible to know if 2wT warrants further explanation without knowing how effective it was to diagnose actual post-operative AEs by having the men self-diagnose.

22) Tables and Graphs – Several of the graphs lack titles, and the bar graph showing time of day would work better as a couple of sentences, rather than a figure.

Reviewer #3: 1. The manuscript is not organized. Methods are in different places and need to be concisely captured to clearly bring out the study design. Introduction lines 99 to 108 are the only section that tend to describe the design. This section should be moved to the methods section and the results therein should be in the results not methods.

2. Related to above, is the RCT the best design for this study? There is limited analysis that shows comparison of the two study arms. The randomization of 771 mentioned is not followed through in the analysis. Analyses follows N=100 and these seem to be in the intervention arm. Where is the rest of the sample?

3. How was the sample of 771 determined?

4. The study outcomes are not clearly stated. What is the primary outcome and what are the secondary outcomes? These must be measurable in both arms. You have text responses and AEs for instance. Are text responses measured in the control arm?

5. Because study outcomes are not clear as they relate to both arms analyses herein may not be optimum for the design. It suffices to say that analyses must show study outcomes for both outcomes. You could explore difference in difference analyses for example but even that needs to be informed by the data and theory of change.

6. The theory of change is also not clear and requires recasting. Graphical presentation of this showing variables being measured in the study and direction of change with clear outcomes as mentioned above is important for clarity. As it stands it mentions even variables that are not presented in the analyses.

7. The whole section on technology is not necessary. This should be rolled into the intervention description and need not be this long.

8. You have under methods a sections on quantitative and qualitative. These two should be rolled out into design description and should bring out design aspects more than tools description.

9. Related to above, the qualitative narrative suggests that coding was informed by the study guide. Codes should come from themes in the data not the guide. In this section you also mention that analysis is of a subset of the sample, why? This is not explained.

10. The statistical analysis is not clear on the comparison needed for an RCT. Why are robust errors used in the analysis?

11. Ethics section should mention approval references from the ethics boards mentioned.

12. In the results table 1 compares intervention arm to the total study. Why? You need to compare the two arms to determine control variables.

13. Subsequent tables 2 and 3 only analyze data for the intervention arm. What was the need for the control. AEs outcome should be compared between the two arms. Where is analyses presented?

14. Interpretation of ORs appears problematic. When Cis cross 1 results are not statistically significant. This does not appear to be applied in these analyses.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Limakatso Lebina

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Manuscript Review PLOS - 16012020.docx

Click here for additional data file.

12 Apr 2020

To the editor and reviewers,

Thank you very much for the thorough review. The authors have revised the manuscript in accordance with the suggested changes and believe that the manuscript is stronger as a result. We prepared a point by point response to the reviewers below. Our responses are in italics with line and page number to indicate new text or changes.

We appreciate your consideration and support for this important contribution to the literature.

Best,

Caryl Feldacker on behalf of all authors

Reviewer 1:

Abstract

1. Objective that are not part of this paper are included. Need to focus only on what is included in the current manuscript. Other objectives of the overall study could be mentioned in the methods section. Revised

2. Remove numbers on the aims – and they should only be two – according to the tittle: usability and acceptability of the 2wT Revised

3. Conclusion: Findings on “men with concerns…” is not included in the results section of the abstract This is addressed in the results. Men interacted with the nurse via individual, tailored messages. There were over 1500 messages exchanges, signaling that the men were interacting with the nurse over their healing concerns. The phrasing of the abstract was updated for added clarity.

Main Manuscript

Introduction

1. Lines 94 – 109 – should be part of the methods section Much of this was deleted or moved as the clinical results were published.

2. Lines 87 – 97 – should be in the setting section of the manuscript under methods. This information in the introduction sets the background for the RCT. We shortened it but prefer to leave it in the introduction. We did note the study clinics within the 2wT intervention information on lines 170-171.

3. Lines 123- 124 – revise to indicate objective/aim of this study. Updated in lines 97-101.

Methods

1. Theory of change section should be moved to the introduction section. Moved and clarified.

2. Line 150 - 2wT Technology overview – reads like a descript of the intervention. This section was revised and streamlined to focus more on the technology background and its core components for this intervention. The 2wT system was the backbone of the intervention.

3. Intervention – reads more like study procedures. The study was the 2wT intervention, itself. We tried to rewrite this section in accordance with reviewer suggestions, and we noted only the core components of the intervention for the texting men.

4. Line 176 – Only Shona and English were used in the testing, what about other languages like Ndebele. What was done if the participant did not understand any of those two languages (Shona and English). In this region of Zimbabwe, Shona and English predominate. We did not encounter anyone who noted a desire for automated texts in another language. Clients could send and respond to texts using any language, something that we now noted on lines 176-178.

5. Line 213 – Quantitative methods: Open-ended questions are part of qualitative research. Likert scale questions and demographics possible quantitative. It might be better in this section to put data collection from MC participants, and data collection from healthcare worker. All information on the open ended questions from the client usability survey was moved to qualitative.

6. Lines 231 – 236: Data analysis and the statistical analysis could be included in this section. Changed and all moved to the quantitative methods section.

Results

1. Healthcare workers perspective – too long, and can be summarized highlighting the major findings. The quotes were shortened as was the section. For scale-up, it is critical to include enough information from the HCWs to inform considerations moving from research to practice.

Discussion

1. Lines 410 – 415 – edited and be moved to study procedures or intervention description. Moved

2. Line 467 “it is clear that 2wT is warrants further” delete the word is. Deleted

Reviewer #2: This manuscript addresses concerns regarding the utility of adherence to post-imperative follow-up visits after VMMC. The paper would be strengthened with better organization and more focus on the quantitative outcomes. Specific comments are below:

1) Line 67 – More recent data than 2017 are available for MCs done in SSA. Updated. Line 62

2) Line 77 – I would specify that when you are only referring to post-operative AEs, not intra-operative. As much of the literature refers to both or do not specify, I am referring to all AEs here.

3) Line 81 – I would specify the post-operative recommendations recommended by Zimbabwe’s MOHCC. Specified line 77

4) Lines 94-108 – While this is important information, most of it should appear in the methods and results sections, rather than the introduction. Largely moved to methods

5) Line 124 – The authors refer to “Aim 3” without any context, so this does not make sense to the reader. I would recommend dropping the references to Aims here and throughout the paper. Removed throughout.

6) Line 138 – There might be an error with the figures and tables, as Figure 1 is the CONSORT table, not a mapping of the intervention. All figure and table numbers revised.

7) Lines 150-171 – This information should be incorporated into the background/introduction rather than the methods. This section now comes before the methods. It is difficult to determine the best placement for this critical section as evidenced by the conflicting advice from the reviewers. We are open to placing these sections in the most logical order according to the reviewers and the Editor.

8) Lines 174-190 – This detail about pretesting is interesting but not critical to the reader to understand the piloting and refining that took place as part of this study. I would recommend deleting this section and reducing this text to a sentence or two. Reduced and placed in the 2wT intervention section of the methods. Line 156 -162.

9) Line 197-198 – This description of the control arm is confusing, as it appears that the intervention arm had the same follow-up schedule (additional visit on Day 14) I would recommend a clearer description of the arms, or an additional table if possible. This section was revised for clarity. Please see lines 163-181.

10) Line 223 – Specify that the wage groups are for daily wages earned. This is now clarified throughout.

11) Line 230 – It’s unclear if these are facilities, sub-districts, etc. Clarified in line 168, “Interviews were conducted at both 2wT intervention clinics: five interviews were conducted at Seke South and four at Norton”

12) Line 255 – There is a typo—it says “participate” when it should be “participant”. Corrected, thank you. I wish my editing were as good as yours!

13) Line 270 – Inconsistent use of the oxford comma here and throughout. We have done our best to correct this throughout.

14) Line 275 – This table would be easier to interpret if it was in descending order by Likert score or grouped into relevant categories. Reordered by Likert score. Thank you!

15) Line 281 – I don’t see how this point is relevant to the broader findings of this study. Age was positively related to increased number of SMS responses – or older men were more likely to respond more frequently. We think this is important to understands who might find the system more acceptable especially as the age pivots to focus MC on those 15-29.

16) Line 293 – 369 – The qualitative findings are not compelling enough to add to the overall value of this paper. I would recommend either deleting them and focusing on the quantitative findings, or developing this section further to have more robust findings and conclusions. We understand that different readers will have different preferences and find value in specific components of any research. We believe your point is valid; however, it was this depth of information that helped the MoHCC review the 2wT proposal for expansion favorably. We believe that hearing what the men said, in their own word, is important to understand how the men used the system. Likewise, we believe that hearing from the HCWs, themselves, about why they think 2wT succeeded and how to move from research to practice is invaluable. We did shorten this section, including making the quotes more succinct. We hope this will make them more compelling.

17) Results section - I assume that this was previously described in more detail elsewhere, but a major gap in this manuscript is that the AE results by arm are not reported in the results and the findings in this paper do not engage with the overall findings from the main study outcomes. The reader would also need to know how many AEs were missed through the self-diagnosis. The previous paper on the clinical outcomes focused only on AE ascertainment, overall client safety, and workload. It is available here: https://journals.lww.com/jaids/Fulltext/2020/01010/Reducing_Provider_Workload_While_Preserving.3.aspx/ While we have to give some information about that aspect of the clinical trial, we are trying very hard to focus only on usability and acceptability of the system, itself, in this paper. We did add one sentence to the background to make this more clear. We do not know how many AEs were missed through self diagnosis or through simple healing before identification; however, as there was only 1 previously undetected AE identified at the Day 14 visit (within the control arm), we noted within that clinical paper that we are confident that there was not a huge backlog of unidentified AEs that went undetected over the course of the study. In the introduction, we now comment, “Clinical results from the RCT, published previously, identified AEs among 1.88% of 2wT arm participants and 0.84% of control arm men, suggesting that 2wT could safely reduce MC follow-up visits, reduce provider workload, and serve as a proxy for active surveillance [27]. However, the other reviewer suggested taking out all previous findings. We hope the readers will review the clinical outcomes for more information on these results.

18) Line 377 – Why is Day 13 most critical? Our previous study that informed this aspect of the RCT found that 95% of AEs happen before day 13: https://www.ncbi.nlm.nih.gov/pubmed/30192816. That line is now augmented with the citation, “The automated 2wT daily messages reminded clients of the signs of poor healing and encouraged wound observation for the most critical 13 days after MC, the period when 95% of AEs occur” on line 331.

19) Line 384 – The content of these daily messages need to be presented somewhere in this manuscript. Excellent suggestion. Added a table 1.

20) Line 388 – I think you mean “substantial”, not “significant”. Changed.

21) Line 467 – It is impossible to know if 2wT warrants further explanation without knowing how effective it was to diagnose actual post-operative AEs by having the men self-diagnose. Changed this to reference the clinical results demonstrating safety, “Despite these limitations, and combined with the clinical results demonstrating safety [27], it is clear that 2wT warrants further exploration for replication and scale-up in other MC care contexts and settings. lines 418-420.

22) Tables and Graphs – Several of the graphs lack titles, and the bar graph showing time of day would work better as a couple of sentences, rather than a figure. Graph replaced and language changed: “More than half (57%) of the men responded to their daily text when it arrived at 8am, with 86% of responses within 4 hours.” Lines 230-232

Reviewer #3:

1. The manuscript is not organized. Methods are in different places and need to be concisely captured to clearly bring out the study design. Introduction lines 99 to 108 are the only section that tend to describe the design. This section should be moved to the methods section and the results therein should be in the results not methods. Thank you for this comment. We reorganized this manuscript and hope it will read more logically for readers now. We are trying to briefly present the methods from the overall RCT (clinical results and design details already published here: https://journals.lww.com/jaids/Fulltext/2020/01010/Reducing_Provider_Workload_While_Preserving.3.aspx#O8-3-2), not overlap, and give enough context for these critical study results on usability and acceptability. We need to describe part of the overall study to orient the reader, but the RCT methods, overall, are background to this exploration of the system response. We did reorganize and hope that it is more clear now.

2. Related to above, is the RCT the best design for this study? There is limited analysis that shows comparison of the two study arms. The randomization of 771 mentioned is not followed through in the analysis. Analyses follows N=100 and these seem to be in the intervention arm. Where is the rest of the sample? The RCT was already conducted and clinical outcomes published. The design of the RCT was published here and cited in the background of the paper: (https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-019-3470-9). Of the 712 randomized 1:1 in the RCT, 362 were in the texting arm (Table 2). We clarified in the quantitative methods on lines 187-196: “Of the 721 men in the RCT, 362 men were assigned to texting and included in this analysis. On day 14, a self-administered questionnaire was implemented with a subset of 100 of the 362 2wT MC clients (28%) to gauge satisfaction, acceptability, usability, and ascertain suggestions for intervention improvement. Responses of these 100 men were entered and frequencies explored in Excel for usability results. Data from all 362 are used to assess acceptability and usability as reflected in the system response rates and how the system was employed by the clients. Age and wage have plausible, a-priori relationships with usability and acceptability.” In Tables 2 and4, we now note n=362 while Table 3 notes n=100. We hope this makes it more clear for the reader.

3. How was the sample of 771 determined? The methods of the RCT are presented in the JAIDS publication and are not the focus of this paper. We do not want to replicate previously published content. In brief, For the clinical component (safety as measured by AE rates), we determined the sample size based on hypothesis testing for non-inferiority to examine the outcome of interest: AE rate (moderate or severe) occurring ≤ 14 days. The non-inferiority margin, based on statistical and clinical considerations, is the maximum difference between the rate of AEs ≤ 14 days in the control and 2wT arms where we would conclude that 2wT is not inferior to routine care. We set that margin to 1.6%, which would create a non-inferiority cutoff of ≤ 2% AEs in 2wT. The cutoff for the proportion of AEs to define 2wT non-inferiority was set at a conservative 2% of AEs to match the lower bound of AE rates reported in previous rigorous studies and because an AE rate of 2% is regarded as a commonly used standard of VMMC safety, including in Zimbabwe. Including a possible 10% LTFU, 361 participants per arm provides 90% power at alpha of 2.5% to detect AEs above the non-inferiority margin of 1.6%, finding 2wT non inferior if we rule out an AE rate > 2%. We found that AE rates were not different per arm but that we could not rule out non inferiority as we ascertained more AEs using 2wT. We believe, and asserted in that paper, the 2wT acted as a proxy for active surveillance.

4. The study outcomes are not clearly stated. What is the primary outcome and what are the secondary outcomes? These must be measurable in both arms. You have text responses and AEs for instance. Are text responses measured in the control arm? The control arm received no texts, so we cannot compare usability or acceptability across arms. The main clinical outcomes, that compare across arms, was published previously. We clarified our aim in the introduction, paragraph 4: “In this mixed-methods study, we aimed to explore 2wT usability and acceptability among 2wT patients and healthcare providers focusing on understanding how providers and patients interacted with the 2wT system and offering insights into the obstacles and facilitators of system use.” Earlier in the introduction, as background, we note the previous RCT results, “Therefore, we implemented a randomized control trial (RCT) in two peri-urban clinics in Zimbabwe to test two-way texting (2wT) between patients and providers, allowing men healing without complication to opt-out of their routine post-operative visits. The trial design details are available {Feldacker, 2019 #928}. Clinical results from the RCT, published previously, identified AEs among 1.88% of 2wT arm participants and 0.84% of control arm men, suggesting that 2wT could safely reduce MC follow-up visits, reduce provider workload, and serve as a proxy for active surveillance [27]”

5. Because study outcomes are not clear as they relate to both arms analyses herein may not be optimum for the design. It suffices to say that analyses must show study outcomes for both outcomes. You could explore difference in difference analyses for example but even that needs to be informed by the data and theory of change. The theory of change applies only to the intervention group. The control arm received routine VMMC care and was not exposed to the intervention nor allowed or offered texting as a form of follow-up.

6. The theory of change is also not clear and requires recasting. Graphical presentation of this showing variables being measured in the study and direction of change with clear outcomes as mentioned above is important for clarity. As it stands it mentions even variables that are not presented in the analyses. The theory of change has been rewritten for clarity and better written description of the figure, lines 120-132. We note how each one of the outcomes was measured in the overall approach. This figure shows the intervention theory of change which encompasses the clinical outcomes. Interaction with the system in the manner depicted signifies usability and acceptability of the system, overall.

7. The whole section on technology is not necessary. This should be rolled into the intervention description and need not be this long. This section was shortened and the figure removed. However, we feel like this brief introduction and overview of the technology is critical to the reader’s understanding of the usability and acceptability of the technology. Informing and potentially adapting the technology, itself, is the main focus of the usability and acceptability study.

8. You have under methods a sections on quantitative and qualitative. These two should be rolled out into design description and should bring out design aspects more than tools description. Respectfully, we disagree and believe that these methods belong in the current section of the paper. We are open to moving them if also suggested by the Editor.

9. Related to above, the qualitative narrative suggests that coding was informed by the study guide. Codes should come from themes in the data not the guide. In this section you also mention that analysis is of a subset of the sample, why? This is not explained. Coding was informed by the study guide that contained questions of interest about the 2wT intervention. As a process of open probing in response to the informants also occurred, we followed themes that stemmed from the guide-prompted questions as well as from the free response of the informants. As noted in the quantitative methods, a subset of 100/362 men were asked to complete a usability questionnaire at the Day 14 visit. The open-ended responses of those 100 respondents form the qualitative data from clients.

10. The statistical analysis is not clear on the comparison needed for an RCT. Why are robust errors used in the analysis? Some statisticians, including the one who worked on this paper, believe that logistic regression is heteroskedastic by nature, requiring robust standard errors. Others would use robust standard errors as we have data on only a few predictors. Some would disagree, but the robust standard errors make the CIs larger, and therefore more conservative. We chose to be more conservative with our approach.

11. Ethics section should mention approval references from the ethics boards mentioned. I am unclear. In the text, we note, “study was approved by the Medical Research Council of Zimbabwe (MRCZ) and the University of Washington, Seattle, USA, Internal Review Board.” These are the approvals we received.

12. In the results table 1 compares intervention arm to the total study. Why? You need to compare the two arms to determine control variables. I did not want to duplicate previously published material. The first publication on the clinical outcomes contains these analyses. This paper is only about the 2wT (intervention) men and, therefore, I only wanted to highlight their characteristics in this paper.

13. Subsequent tables 2 and 3 only analyze data for the intervention arm. What was the need for the control. AEs outcome should be compared between the two arms. Where is analyses presented? This analysis that includes the clinical outcomes that compared outcomes between the intervention and control was previously published below. However, usability and acceptability only applies to the intervention arm, so only those men are included.

Feldacker, C., et al. (2020). "Reducing Provider Workload While Preserving Patient Safety: A Randomized Control Trial Using 2-Way Texting for Postoperative Follow-up in Zimbabwe's Voluntary Medical Male Circumcision Program." JAIDS Journal of Acquired Immune Deficiency Syndromes 83(1): 16-23.

14. Interpretation of ORs appears problematic. When Cis cross 1 results are not statistically significant. This does not appear to be applied in these analyses.. The Table 4 was confusing. The Total SMS responses is a linear regression model. Those CIs do not include “0”. The Potential AE text was a logistic regression model. Therefore, columns 2 and 3 are coefficients and (as noted in the column header) while columns 4 and 5 are ORs, as noted in the column header. Those mostly included “1” and were not significant. We added labels for linear and logistics regression models for clarity.

________________________________________

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Limakatso Lebina

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Manuscript Review PLOS - 16012020_response.docx

Click here for additional data file.

1 May 2020

Usability and acceptability of a two-way texting intervention for post-operative follow-up for voluntary medical male circumcision in Zimbabwe

PONE-D-19-27561R1

Dear Dr. Feldacker,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Professor Kwasi Torpey, MD PhD MPH

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Limakatso Lebina

Reviewer #3: No

5 Jun 2020

PONE-D-19-27561R1

Usability and acceptability of a two-way texting intervention for post-operative follow-up for voluntary medical male circumcision in Zimbabwe

Dear Dr. Feldacker:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Kwasi Torpey

Academic Editor

PLOS ONE