Usability, acceptability, and self-reported impact of an innovative hepatitis C risk reduction intervention for men have sex with men: A mixed methods study.

Hepatitis C virus (HCV) elimination among men who have sex with men (MSM) is unlikely to be feasible without effective behavioural interventions. We developed a multilevel intervention to reduce HCV transmission among MSM in Amsterdam. The intervention includes a toolbox to facilitate risk reduction among MSM and support health care professionals in risk reduction counselling. To assess the use of the toolbox and its impact on behavior, we conducted a mixed-methods study. We collected data through online questionnaires (n = 49), and in-depth interviews with MSM at risk of HCV (n = 15) and health care professionals (n = 7). We found that the toolbox has been well received by MSM, increased awareness of HCV risks and has facilitated preventive behaviours and risk-reduction communication with peers. Professionals reported the toolbox to be a useful aid for discussions about HCV risk and risk reduction strategies with their clients.

Introduction

Since 2000, hepatitis C virus (HCV) outbreaks among HIV-positive men who have sex with men (MSM) have been reported globally [1]. In the Netherlands, a decline in primary HCV infection and HCV reinfection has been observed in this population after the introduction of unrestricted access to direct-acting antivirals (DAAs) in 2015 [2]. Between 2008 and 2015 the HCV incidence fluctuated between 8.7 and 13.0 per 1000 person-years and significantly declined to 6.1 per 1000 person years in 2016. Between 2017 and 2019, HCV incidence fluctuated between 4.1 and 4.9 per 1000 person-years. HCV reinfection incidence also declined from 41.4 per 1000 person-years in 2016 to 11.4 per 1000 person-years in 2019, but it remained high, illustrating the ongoing HCV transmission in HIV-positive MSM in the Netherlands [2].

HCV incidence among HIV-negative MSM is more challenging to predict as HIV-negative MSM are normally not in routine clinical care. Two systematic reviews showed a 16-to-19-fold lower HCV incidence in HIV-negative MSM compared to HIV-positive MSM including studies from 2000 to 2016 (0,4/1000 person years in HIV- negative and 6.4–7.8/1000 person years in HIV-positive MSM) [3, 4]. However, the introduction and increased uptake of pre-exposure prophylaxis (PrEP) against HIV infection, has resulted in changed patterns of sexual networks and behaviour among HIV-negative MSM [5, 6]. A recent meta-analysis, including studies from 2000 to 2019, estimated a 123-fold higher HCV incidence in HIV-negative MSM using PrEP compared to HIV-negative MSM not using PrEP and a pooled HCV incidence of 14.8 per 1000 person years in HIV-negative MSM using PrEP [7]. An additional concern is the high HCV reinfection incidence among HIV-negative MSM using PrEP. During follow-up in a PrEP demonstration project in the Netherlands an HCV re-infection incidence of 278 per 1000 person years was found [6].

HCV transmission among MSM occurs mainly through sexual contact [8]. Several studies have identified certain sexual techniques and settings as risk factors for HCV infection, including condomless anal intercourse, (unprotected) fisting, sharing of sex toys, group sex and chemsex (the use of recreational drugs immediately before and/or during sex to facilitate or enhance sexual pleasure) [8, 9]. In addition, having an ulcerative sexually transmitted infection (STI), injecting drug use, sharing of straws or other equipment for snorting drugs, sharing anal douching equipment and rectal bleeding are also associated with an increased risk of incident HCV infection [8, 9]. A study in the Netherlands found that receptive condomless anal intercourse, sharing of sex toys, group sex, anal rinsing before sex, having 10 or more sex partners in the last 6 months are strongly associated with HCV reinfection [10]. To prevent HCV (re)infections among MSM, scaling up HCV treatment and the implementation of risk reduction interventions for MSM are recommended [11]. Evidence on how to achieve HCV behavioural risk reduction in MSM is limited. Up to now, the focus of sexual risk reduction interventions has been on the use of condoms to prevent the transmission of HIV and other STIs. Yet, the prevention of HCV requires more than promoting condom use alone. Informing men at risk about the possible HCV risk factors other than anal intercourse, as well as motivating them to integrate such risk reduction strategies into their sex lives are essential. The first HCV specific sexual risk reduction intervention was developed in 2016 as part of the Swiss HCVree trial [12]: an intervention for HIV/HCV co-infected MSM provided in combination with DAA treatment and consisting of individual counselling sessions aimed at reducing sexual risk taking [13]. In 2017, we initiated the NoMoreC project, a multilevel intervention aimed at reducing HCV transmission among all MSM at risk of HCV in Amsterdam [14]. Our approach was to target MSM based on their sexual practices, rather than on their HIV or HCV-status. NoMoreC includes web-based and face-to-face components as well as an anonymous testing service. The NoMoreC website (www.nomorec.nl) provides information about hepatitis C, HCV transmission routes, risk reduction strategies, testing and treatment options, and partner notification. The face-to-face component comprises a risk reduction toolbox, training for health professionals, and providing tailored advice to sex on premises venues. NoMoreC is promoted by a voluntary community campaign team.

In this study we focus on one component of the project, the NoMoreC toolbox. The approach used to develop the toolbox was innovative, and such approach was not applied and evaluated elsewhere before. The toolbox development was based on the information motivation and behavioural skills (IMB) model [15], as IMB-based interventions have shown to be effective in preventing risky sexual behaviours among risk groups [16-18]. The NoMoreC toolbox contains practical tools, to encourage HCV risk reduction in different settings (e.g. during one-on-one sex, group sex, sex parties, sexualized drug use). Examples of these tools are condoms for safer sex, gloves and hand disinfectant for safer fisting, single use drug equipment for safer chemsex and various cleaning and disinfection products to create safer conditions for group sex.

The Toolbox is the first comprehensive HCV risk reduction tool to provide information on HCV risks in combination with a broad range of products for risk reduction and, furthermore, it has been co-created with the gay community. We conducted a mixed-methods study among MSM at risk of HCV and health care professionals to assess the use of the toolbox and its impact on behaviour. We aimed to identify reasons for obtaining the toolbox, motives and barriers to its use, measured aspects of usability and acceptability, and assessed the toolbox’s impact on HCV awareness and personal behaviour change, as self-reported by toolbox users.

Methods

Toolbox development and distribution

The development of the toolbox was guided by a co-creation process that included various sessions with a group of MSM. This group was formed specifically for the NoMoreC project and was composed of members of the Amsterdam gay community. It included men at risk of HCV, men who had been HCV-infected in the past, and men who had concerns about becoming infected. The IMB model [15] applied by us to the HCV context, suggests that individuals will initiate and maintain HCV preventive behaviours if they are well-informed regarding HCV, HCV transmission routes, HCV infection risks, and possible preventive behaviours, are motivated to prevent infection, and perceive themselves as being capable of applying the recommended preventive strategies. In our first meetings with the community group we explored the needs and views of participants regarding a practical, action-based approach to HCV risk reduction. Based on the first feedback round we set the following IMB-inspired goals: on the information and motivation levels we wished to provide detailed HCV risk information and information on practical risk reduction steps to avoid HCV transmission and increase the response efficacy (i.e. increase a person’s belief that the recommended steps will be effective in avoiding an HCV infection) and hence improve attitudes towards preventive behaviours. On the skills level, we wished to provide (illustrated) tips on how to perform such steps to increase perception of self-efficacy an provide the actual products that can be used to perform the suggested steps. We subsequently held several sessions about the actual toolbox content and supporting instructional materials. During these sessions it became clear that many community members were uncertain about how to effectively clean and disinfect their toys, hands and other surfaces, and they voiced their need for cleaning and disinfection instructions. This resulted in adding a number of cleaning and disinfection materials and instructions to the toolbox. An example of these instructions is the step-by-step explanation on how to disinfect hand and forearms: 1) wash with soap to remove all lube from your skin and rinse well; 2) dry your skin, 3) make a cup with one hand and fill it with hand disinfectant; 4) disinfect hands and forearms by rubbing hand disinfectant for at least 30 minutes. Another example is how to disinfect sex toys and anal douches which was explained in 4 steps: 1) Clean sex toys and anal douche with dish-washing liquid to remove all lube and rinse well; 2) mix 1 part bleach with 9 parts cold water; 3) submerge for 5 minutes in the bleach solution: 4) rinse thoroughly with water and wipe dry with a clean towel.

Decisions on the addition of other products to the toolbox were collectively made with the community (Fig 1). Consensus was easily reached for most of the products except for the safe drug use equipment. The suggestion to add needles and syringes to the toolbox led to an extensive discussion within the community group. Some members were in favour of including safe drug use equipment as drug use is a reality in their environment. Whilst other members believed that injection equipment would be experienced as shocking or may trigger drug use. However, because safe drug use is essential to prevent HCV among men who engage in chemsex, a compromise was reached and the safe drug-use equipment was included in the toolbox but packaged in a separate sealed off box, so that the injection equipment was not directly visible when opening the toolbox. After the contents of the toolbox was decided upon, items were ordered, and all boxes were assembled and packed by the community members. A total of 700 toolboxes were packed; 450 in 2018 and an additional 250 in 2019. The final version of the toolbox contained an information booklet and referrals to the NoMoreC website where supporting information, testimonials and instructional videos were available. The toolbox also contained condoms, fisting gloves, disinfectants, equipment for safe drug use and a booklet with practical information on HCV risk reduction (Figs 1 & 2). Table 1 shows how the IMB model constructs have been integrated in the intervention.

Fig 1

NoMoreC toolbox.

The box contains a spray bottle (to clean surfaces where sexual activity takes place), sterilium® med hand disinfectant, hydrogen peroxide cleaning wipes (to disinfect surfaces where sexual activity takes place), gloves (for safe fisting), condoms (for safe sex), disinfection instruction card for hands and forearms, disinfection instruction card for dildos, toys & anal douche, a safe drug use box and booklet with information about HCV risk reduction. Republished from www.NoMoreC.nl under a CC BY license, with permission from the NoMoreC project group, original copyright 2018.

Fig 2

Safe drug use box.

A small cardboard box containing: snorting straws, a sharps container, syringes, needles, stericup® mixing cups and alcohol wipes. A sticker with the text “Never share drug-use equipment. Pick your own colour” is stuck on the box. Republished from www.NoMoreC.nl under a CC BY license, with permission from the NoMoreC project group, original copyright 2018.

Table 1

Toolbox intervention components focussing on information, motivation and behavioural skills constructs and their aims.

Toolbox component	IMB-construct	Aim
Information booklet	Information	Increase knowledge about: hepatitis C and its transmission, personal HCV risk, practical HCV prevention steps and their efficacy (response efficacy).
	Motivation
	Behavioural skills
Referral to instruction videos on the NoMoreC website	Behavioural skills	Teach HCV prevention skills;
		Enhance self-efficacy to engage in HCV risk reduction behaviours
Referral to personal video testimonials about what men at risk do to reduce their HCV risk	Motivation	Develop positive attitude toward HCV risk reduction, improve motivation to engage in HCV prevention behaviours
Sterilium® med disinfectant for disinfection of hands and forearms	Motivation	Lower threshold to use the product
Disinfectant cleaning wipes for disinfection of surfaces where sexual activity takes place	Motivation	Lower threshold to use the product
Spray bottle for cleaning surfaces where sexual activity takes place before disinfection	Motivation	Lower threshold to use the product
Black latex gloves for safe fisting	Motivation	Lower threshold to use the product
Black nitrile gloves for safe fisting	Motivation	Lower threshold to use the product
Black condoms for safe anal sex	Motivation	Lower threshold to use the product
Illustrate instruction card: How to disinfect dildos, toys and anal douche	Behavioural skills	Teach disinfection skills;
		Enhance self-efficacy to engage in this HCV risk reduction step
Illustrated instruction card: How to disinfect ands and forearms	Behavioural skills	Teach disinfection skills;
		Enhance self-efficacy to engage in this HCV risk reduction step
Clip with suction cup to hang the instruction card for disinfecting hands and forearms on bathroom mirror	Motivation Behavioural skills	Lower threshold to use the product
Safe drug use box (Fig 2) for safe chemsex	Motivation	Lower threshold to use the product

From March 2018, the toolbox could be ordered online free of charge from the NoMoreC project website (www.nomorec.nl/en/get-yourself-a-toolbox/). They were sent to addresses in The Netherlands only. Toolboxes were also available for pick-up at 17 locations in Amsterdam: at four fetish shops, a condom shop, the city central STI clinic of the Public Health Service, six HIV outpatient treatment centres, four GP practices, and a non-governmental organisation (NGO) dedicated to improving the health and quality of life of people who use drugs (PWUD). Health care professionals were given the toolboxes to both distribute to MSM at risk and to discuss HCV prevention strategies with their clients.

Study design, participants and procedures

A mixed-methods study was conducted among toolbox recipients, using quantitative and qualitative data collection methods. Study participants of both the quantitative and the qualitative studies were included if they were at least 18 years of age, were a man who had sex with men, had received the toolbox and spoke the Dutch or English language. Firstly, online questionnaires were sent to toolbox recipients by email, 12 weeks after receipt of the toolbox in the period between March 2018 and March 2020. The questionnaire contained questions about HCV infection concerns, reasons for obtaining the toolbox, its use, context of use and impact of the toolbox on HCV risk reduction awareness (S1 Appendix). Answer options included yes/no, (3 and 5 point) Likert scales and lists with pre-defined answers (see S1 Appendix for complete questionnaire and answer options). Two members of the community group pre-tested the questionnaire and were asked to give feedback about the clarity of the questions and answer options. After the pre-test, some small revisions to the questionnaire were made. The validity of the questionnaire was not pre-tested. The questionnaire provided data that were subsequently used as the basis for the qualitative study, to further explore the themes of the questionnaires, build our understanding of the experiences of toolbox recipients and give more meaning to the quantitative findings.

The qualitative part of the study consisted of in-depth interviews with men who had received the toolbox from March 2018 onwards. Men who had ordered the toolbox online were given the option to leave their email address if they were willing to share their opinion about the toolbox in the future. Men who had received the toolbox in person from a health care professional or had picked it up at a fetish shop were asked if they could be contacted in the future, to share their experience with the toolbox. Those who agreed to be contacted left their email. In February 2020, all toolbox recipients who agreed to be contacted were sent an email outlining the research aims and request for participation in the study. 20 men reacted to the email and agreed to participate in the study. In the period March till May 2020, 15 out of 20 participants who had given their consent took part in an in-depth telephone interview, after which thematic saturation was reached. The interviews lasted 30 to 60 minutes. A topic guide (S2 Appendix) with open ended questions was used, to give participants the opportunity to describe their personal experiences with the toolbox. During the interviews, topics discussed included: experiences with hepatitis C, reasons for obtaining the toolbox, the actual use of the different toolbox products, motivations for, and barriers to the use of the toolbox products, context of use and experiences with the use of the products. We asked participants to reflect on the possible impact of the toolbox on their HCV awareness and on the participants’ sexual and drug-use behaviours. We were interested to learn if participants had made any behavioural changes in their sex lives to reduce the HCV infection risk, based on the information in the toolbox, and what these changes were. For example, did participants start cleaning and disinfect dildos and the area where sex takes place with the suggested products? Did participants who use drugs, safely administer their drugs by single use drug equipment and not share their equipment? Did men who enjoy fisting start using gloves and clean and disinfect their hands and forearms immediately after fisting? Furthermore, we asked the participants if the information provided by the toolbox assisted in making a personal HCV risk assessment. To thank the participants for their contributions they received a gift voucher of €20.

In addition, professionals from the STI clinic, HIV centres in Amsterdam and NGO for PWUD were contacted by email in February 2020, requesting their participation in our study. In March and April 2020, 7 out of 8 professionals who consented to be contacted took part in an in-depth telephone interview, after which thematic saturation was reached. The interviews lasted 10 to 30 minutes. Again a topic guide with open ended questions was used, to give professionals the opportunity to describe their experiences with the toolbox in their work setting. During the interviews, topics discussed included: their professional experience with hepatitis C, use of the toolbox in their daily practice and triggers and barriers to using the toolbox. Interviews were audio-recorded, transcribed verbatim and anonymised.

Data analysis

Simple descriptive statistics were used to analyse the quantitative data.

Interpretation of the qualitative data followed an inductive thematic analysis approach [19]. Two researchers (TP/JI) coded the transcripts independently and organized the data into open codes reflecting the major categories of information related to the study. Subsequently, codes were categorised into themes. Following the independent assessments of the transcripts, the researchers met frequently to compare and refine existing codes until they reached consensus on main themes and codes. MaxQDA Plus 2020 software (Release 20.0.8) was used for the qualitative data analysis.

Ethics

Participants gave their informed consent by clicking the consent box prior to completing the online questionnaire. Participants, who were interviewed gave their informed consent verbally, which was recorded on audiotape and documented by transcription. We received exemption from the Amsterdam University Medical Centre’s medical ethical committee for extended protocol review (Reference number: W20_110 # 20.145).

Results

Participants

Toolbox recipients

A total of 174 men were sent an email with a link to an online questionnaire between March 2018 and March 2020. The questionnaire was started by 65 participants and completed by 49, resulting in a response rate of 37% (65/174) and a completion rate of 75% (49/65). Median age of participants was 47 years (IQR 40–57), 100% were MSM living in the Netherlands (Table 2). The majority was born in the Netherlands (88%). More than a third reported they were HIV negative (n = 18, 37%), of whom 78% were using PrEP. Twelve participants (24%) reported to have had an HCV infection in the past, of whom one participant reported more than one infection. The majority (83%) of men who reported a past HCV infection were HIV positive. Fourteen participants (29%) were worried about getting HCV infected, 22 (45%) were somewhat worried and 13 (27%) were not worried.

Table 2

Characteristics and concerns of toolbox recipients who filled out the online questionnaire (March 2018 and March 2020) or took part in an in-depth telephone interview (March-May 2020).

	Online questionnaires (n = 49)	Interviews (n = 15)
Socio demographic characteristics
Age, years	47 (40–57)	48 (41–59)
Born in the Netherlands	43 (88%)	14 (93%)
Educated to college degree or highera	NA	13 (87%)
Employeda	NA	12 (80%)
HIV and HCV status (self-reported)
HIV status: positive/negative/ not disclosed	29 (59%) / 18(37%) / 2 (4%)	8 (53%) / 7 (47%) / 0 (0%)
Current PrEP use	14 (78% of HIV-negative MSM)	6 (86% of HIV-negative MSM)
Past HCV infection: Primary/ Re-infection	11 (22%)/ 1 (2%)	5 (33%)/ 0
Combined:
Past HCV (re)infection and HIV positive	10 (20%)	4 (27%)
Past HCV infection and HIV negative	2 (4%)	1 (7%)
HCV-related concerns
I worry about getting hepatitis C	14 (29%)	4 (27%)
I worry somewhat about getting hepatitis C	22 (45%)	7 (47%)
I do not worry about getting hepatitis C	13 (27%)	4 (27%)

Data are in n (%) or median (IQR). PrEP = Pre-exposure prophylaxis. NA = not applicable.

a:Only asked during the in-depth interviews.

All 15 participants who took part in an in-depth telephone interview were MSM; 14 were living in the Netherlands and one abroad. All but one were born in the Netherlands. The median age was 48 years (IQR 41–59; Table 2). Most participants had at least a college education (87%) and were employed (80%). Almost half (47%) were HIV negative. Of the HIV-negative MSM, 86% were using PrEP. Five participants (33%), of whom 4 were HIV-positive, reported an HCV infection in the past. Four participants (27%) were worried about getting HCV infected, 7 (47%) were somewhat worried and 4 (29%) were not worried.

Health care professionals

Seven professionals took part in an in-depth interview. Two professionals worked at the STI clinic and four professionals at an HIV treatment centre in Amsterdam: five as registered nurses and one as nurse practitioner. One professional worked for an NGO dedicated to improving the health and quality of life of PWUD. The median age was 56 (IQR 44–58). All participants regularly saw MSM at risk of HCV in their care settings. They offered the toolbox to their clients and referred them to the NoMoreC project website. Professionals reported to have used the toolbox to discuss HCV risk and reduction during the consultations.

We present the combined results of the quantitative and qualitative analyses under four main themes: 1) Reasons for obtaining toolbox; 2) Toolbox usability and acceptability; 3) Context of use of the toolbox; and 4) Self-reported impact of the toolbox on HCV awareness and behaviour change. For each theme we commence with the questionnaire results and subsequently offer more in-depth related perspectives from the qualitative data. Perspectives of health care professionals are integrated into theme 1 (Reasons for obtaining toolbox).

Theme 1: Reasons for obtaining toolbox

Participants

The following reasons for getting a toolbox were given by 49 respondents of the online questionnaire: 35 (71%) wanted to reduce the risk of getting infected with HCV, 20 (41%) wanted to reduce the risk of transmitting HCV to someone else, 28 (57%) were curious to know the contents of the toolbox, 2 (4%) wanted to give the toolbox to someone else, 27 (55%) wanted to use it during sex parties and 6 (12%) got the toolbox because it was recommended to them.

During the in-depth interviews interesting paths and reasons for obtaining the toolbox were revealed. Exposure to information about HCV infection risk and the toolbox was sufficient to persuade men to try and use the toolbox:

“I was looking into PrEP. I wanted to start using that and my GP advised me to investigate what that involved. So then I came across the toolbox. That was the moment that I realised: yeah, I am at serious risk (of HCV). I didn’t even know it existed. I only knew of hepatitis A and B and I got vaccinated for that.”

“So when you were informing yourself about PrEP and getting PrEP care, you learned about the NoMoreC toolbox. Is that correct?”

(interviewer)

“Yes, that’s correct.”

(participant 3)

Some participants mentioned they wanted to inform their friends and sexual partners and openly talk about their sex lives in relation to HCV risk. They felt the toolbox could assist them in discussing these topics.

“You can reduce risk by openly discussing your sex life with friends. Or when you have a sex date or go to a place where you can have sex and just talk to like-minded people about it (HCV risk reduction). That box helps me a lot with that…. For example, I have shown the toolbox to a friend I regularly have sex with. I told him what the doctor had told me about all the products in the box. We then had a nice discussion about it.”

(participant 6)

Health care professionals indicated that the toolbox supports them in discussing HCV risk and HCV risk reduction. They want to inform their clients about HCV infection and how to prevent it. One professional explained:

“I always ask first: "We pay attention to hepatitis C, shall I tell you more about that? Would you like me to show you something?" And then I show the box. I tell them that they can have the box. But I always first show what it contains and say: ’Some things apply to you, other things do not apply to you, but this is all aimed at preventing hepatitis C or reducing the chance of getting hepatitis C.”

(professional 2)

Some professionals mentioned that they showed the NoMoreC website first, including the risk reduction video’s and subsequently explained the use of the toolbox products. Many professionals used a client centred approach when discussing HCV risk, based on the client’s knowledge about HCV and sexual behaviour. A lack of HCV knowledge, high risk sexual behaviour and a past HCV-infection were motives for introducing the toolbox:

“During a consultation I find out what men know and what they don’t know and how sexually active they are. Some say, ‘No, a box like that is not for me’ and others ask for more information. When I notice that someone is really interested, I tell them they can have the toolbox. For many men it (the toolbox) is an eye-opener and it also helps me discussing behaviour and risks. It helps because you can give very practical information.”

(professional 4)

Some professionals indicated certain barriers for using the toolbox during consultations. The medical history form that is used during a consultation at the STI clinic is not inviting for introducing the toolbox and discussing settings in which it can be used as questions about group sex or visiting sex parties are not included. Several professionals also mentioned that sexual techniques such as fisting or sharing toys can be difficult to bring up during a consultation. Building a relationship with a client and seeing them more regularly, lowered this barrier and in some cases clients would bring up their sexual preferences more easily, as one nurse reported:

“Sometimes someone says: ‘I like fisting, I go to fisting parties’. Then I will ask: ‘have you heard of hepatitis C?’ Also, I have information about hepatitis C on my table, so some men will ask about that and then it’s an easy entry point to talk about it. But I don’t just ask: ‘Do you practice fisting?’ or ‘Do you share toys?’”

(professional 2)

Another professional mentioned that he did not have enough time to extensively discuss HCV risk reduction and explaining all the items in the toolbox. An HIV nurse explained that her focus is on HIV treatment and less so on HCV prevention:

“Well, I think it is a very handy box. It is just that our consultations are mainly focused on HIV treatment and if we notice that there is risk behaviour, we will of course discuss it. It just doesn’t happen very often that we go into that matter (hepatitis C risk reduction) in more depth. So, it is not in my system to introduce that box.”

(professional 7)

The use of the toolbox in the context of sexualised drug use was a problem for the professional who worked for an NGO dedicated to improving the health and quality of life of PWUD because he works with men who have recently stopped or want to stop engaging in chemsex. He was concerned that the safe drug use equipment in the toolbox would serve as a trigger for his clients who were battling addiction. Therefore, he did not hand out the toolbox to this group.

Theme 2: Toolbox usability and acceptability

Of 49 online respondents, 38 (78%) respondents used one or more toolbox products in the past 12 weeks and 41 (84%) intended to use products in the future. The use of the individual toolbox products since receipt of the toolbox is presented in Table 3.

Table 3

Use and intended future use of the individual products of the HCV prevention toolbox among 49 online respondents.

Toolbox product	Product use Number of participants who used the product, n (%)	Future product use Number of participants who intend to use the product in the future, n (%)
Disinfectant wipes	28 (57%)	41 (84%)
Sterillium® med hand disinfectant	25 (51%)	32 (65%)
Hand & lower arm disinfection instruction card	23 (47%)	17 (35%)
Condoms	17 (35%)	16 (33%)
Syringes for rectal administration of drugs	15 (31%)	19 (39%)
Sharps container	15 (31%)	17 (35%)
Latex gloves	14 (29%)	25 (51%)
Spray bottle	12 (24%)	27 (55%)
Nitril gloves	11 (22%)	15 (31%)
Syringes and needles for injecting drugs	11 (22%)	14 (29%)
Stericup® mixing cups	6 (12%)	11 (22%)

Number of participants who used (product use) and intend to use (future use) the product is given per product, and percentage of the total respondents (n = 49).

The great majority of the participants (46/49, (94%)) rated the instructions of the toolbox as clear or very clear, two participants rated them as neither clear nor unclear and one participant reported to have not read the instructions. Responses to other questions related to toolbox usability and acceptability are shown in Table 4.

Table 4

Measures of toolbox usability and acceptability.

Number of participants (%)
Usability
The instructions for the items in the toolbox were:
Very clear	21 (43%)
Clear	25 (51%)
Neither clear nor unclear	2 (4%)
I haven’t read the instructions	1 (2%)
The instructions for disinfection were:
Very clear	21 (43%)
Clear	24 (49%)
Neither clear nor unclear	1 (2%)
Unclear	1 (2%)
I haven’t read the instructions	2 (4%)
Used the toolbox to start a conversation about hepatitis C with:
Friends and sex partner(s)	18 (37%)
Friends (s) only	4 (8%)
Sex partner(s) only	5 (10%)
I have not used the toolbox to start a conversation	11 (22%)
No response	11 (22%)
Acceptability
It is good that a box with a sharps container and drug equipment is included in the toolbox
Strongly agree	25 (51%)
Agree	14 (29%)
Neither agree nor disagree	9 (18%)
Strongly disagree	1 (2%)
I would recommend the toolbox to a friend or sex partners
Definitely	34 (69%)
Probably	11 (22%)
Maybe	3 (6%)
Probably not	1 (2%)

Responses of 49 participants who completed the online questionnaire 12 weeks after having received the toolbox.

During the in-depth interviews, the use of the products were typically mentioned in relation to the sexual technique and practices of the participant. All toolbox recipients had used or tried the toolbox products; some used many and others used few products, depending on their personal HCV risk. Participants continued to use the products and some had ordered several products again. It was highlighted that some products were easier to use than others. Products that were received positively, were the disinfectant wipes, hand disinfectant and gloves. They were widely used and were highly accepted.

“The disinfectant wipes, I’m very happy with those. They are used everywhere. I now have them in my suitcase, always”

(participant 6)

“That (sterilium® med) disinfectant is really nice. It doesn’t stink. I use it especially when I am done fisting. Then I wash my arms and use it.”

(participant 1)

“Those black gloves have been a really good tip, I still use them when necessary.”

(participant 3)

Mixed reactions were given about condoms and the safe drug use box. While some were happy the toolbox contained these products, others did not use them or disapproved of their inclusion.

“I used the condoms, I always have a box of condoms.”

(participant 3)

“To be honest, I haven’t used the condoms very often.”

(participant 13)

“I have given away the condoms, because I don’t use those. I use a different brand of condoms”

(participant 8)

“It (the safe drug use box) is of added value. The only thing I haven’t used are the mixing cups. All the other things I have used.”

(participant 7)

“I don’t use (drugs) myself, but I know friends use. So it is nice to have.”

(participant 1)

“.. it contained needles and that sort of stuff. I was bit shocked by that. On the one hand, I think it is really good that it is included, but on the other hand I think: are you not promoting something?”

(participant 13)

Theme 3: Context of use of the toolbox

Of 49 online respondents, 28 (57%) used the toolbox during one-on-one sex, 22 (45%) used it during group sex with 3 to 4 men or at small sex-parties. Few respondents, 3 out of 49 (6%), reported the use of the toolbox at bigger sex parties with 5 men or more. One respondent used the toolbox in a group sex setting with both men and women and one respondent used it during tantric massages. A third of the participants (16/49 (33%)) discussed the use of the toolbox products with their sexual partner(s) before having sex, 10/49 (20%) discussed it with some of their sex partners, 12/49 (25%) did not discuss it and 11/49 (22%) participants did not answer the question.

During the in-depth interviews, all participants mentioned that they used the toolbox with sexual partner in a home-setting. The following participants described in what context they informed their sexual partners of the toolbox:

“I just use the toolbox during sex. This can be one-on-one or in a group. I always have the disinfectant to clean your hands after having played. And I tell people: ‘Here is the dishwashing liquid and here is the disinfectant. This card shows how to clean your hands’.”

(participant 5)

“When I organize a party, I just put that box in the room and say: ‘Gentlemen, all kinds of tools for a safe party’.”

(participant 6)

One participant described the use of the toolbox in the context of sexualized drug use:

“We slam (inject drugs) a lot. The fun thing is, the toolbox contains everything that you need to prevent needles and stuff being exchanged, because of those different colour syringes from the toolbox, everyone uses his own colour. So I put everything on the kitchen table with a note so everyone knows: this is my colour. Because of the toolbox we have started to work very systematically”

(participant 7)

Some participants expressed doubts in their ability to consistently apply the risk reduction strategies promoted by the toolbox, to their sex lives. It was felt that the products were easy to use in a home-setting but when having sex outside their home it was more difficult, as nobody would take the toolbox to a club or sauna. However, some reported that they compensate that by taking individual products when going out or on holidays.

“I don’t take the toolbox to a club, that’s a no brainer. But I pack some things in my suitcase when I go to a festival or when I travel, like gloves and hand disinfectant. But never the whole box, that is too much luggage.”

(participant 6)

It was also reported that the use of the products was easier if the sex partner had seen the toolbox before and was already aware of the products.

Theme 4: Self-reported impact of the toolbox on HCV awareness and behaviour change

Of 49 online respondents, 23 (47%) strongly agreed, 23 (47%) agreed and 3 (6%) did neither agree nor disagree with the statement: “The NoMoreC toolbox gives me a better understanding of what I can do to reduce the risk of hepatitis C”.

During the interviews, participants described that the toolbox had contributed to increasing their knowledge and awareness about hepatitis C and its transmission. They mentioned being able to better predict their risk of contracting hepatitis C and that the toolbox products helped them to reduce this risk. Both the toolbox products and being able to make different choices regarding the risks they took gave many participants a sense of safety as well as a feeling of being in control of their own health. The ability to protect others was also considered important. Participants reported that they had considered the preventive measures suggested in the toolbox, and implemented those that were attainable for them. Small changes in their behaviour were mentioned, such as the consistent use of disinfectant wipes or cleaning of sex toys.

“I used to clean things in a certain way, but dildos for example, you are supposed to put them in a bucket of water with bleach. I used to clean them with soap and then disinfect them. But with bleach it’s better, because you get into the pores….so I know that now, and put the dildos in a bleach solution before I use them, so they are clean.”

One participant mentioned that since he had implemented the new hygienic measures and use of disinfectants that friends felt safe to join his sex parties. Others said that since exposure to the toolbox and being better informed about HCV risks, they had made a conscious decision to stop visiting sex parties. They felt by having one-on-one sex only, they could better control their risk:

“My sexual behaviour has changed. I don’t go to big parties anymore. I prefer one-on-one, because I think that protects me more than the measures during sex.

When you say: ‘I have changed my behaviour’, have you done that, based on the information in the NoMoreC toolbox?

(interviewer)

Yes, it has contributed of course. I have also visited the website a few times to inform myself. And you know, at the dates I have had in the past 2 years [after toolbox reception], I have always asked: ‘How do you deal with hepatitis C?’.”

(participant 9)

Next to the practical use of the toolbox products, the toolbox was also felt to provide good information to assess HCV risk and aid discussions about hepatitis C and risk reduction:

“That box helps a little to get a conversation going about hepatitis C, collect information and make a risk assessment. Especially in combination with the website

Can you explain how the toolbox helps you to get a conversation going?

(interviewer)

“Well, someone will see the toolbox in my house and say: ‘Hey, did you also get that box?’ and then we will start talking about it. It sticks in people’s minds when they have had that box, seen it or spoken about it. I hear from some of my friends: ‘I have had that box and I certainly learned something from it’, we talk about the things (toolbox products) we use.”

(participant 2)

Participants recommended or had given the toolbox to a friend and had used it to educate friends and sexual partners on risk reduction strategies. It was mentioned that there was still a lot of ignorance and lack of knowledge about hepatitis C among MSM.

“The good thing about that box is that I have talked about it with several friends, who didn’t know anything about hepatitis C. And when I talked about hepatitis, they thought I was talking about A and B and said: ‘I am vaccinated for that’. So, I thought it was a good thing that the toolbox got that conversation started.”

(participant 13)

Discussion

Our study adds to the understanding of how MSM at risk of HCV and health care professionals respond to an innovative HCV risk reduction intervention such as the NoMoreC toolbox. The mixed-methods approach used, which combines quantitative and qualitative data, offers a valid and deeper understanding of participants’ experiences with the toolbox and its possible impact on HCV awareness and behaviour change. We show that the toolbox gave men a sense of safety and control by being more aware of their HCV risks and by providing easier access to the right protection tools for themselves and others. Data collected from professionals, indicated that the toolbox is a useful aid to discuss HCV risk and risk reduction with clients.

The main motivation for MSM to obtain the toolbox is to reduce the risk of getting an HCV infection. All participants used the toolbox products to some degree and indicated good usability of the toolbox and found the instructions clear. Some participants indicated that the provision of the products has lowered the threshold for them to use these products, and facilitated their risk reduction behaviour. Furthermore, men reported the intention to use some of the products in the toolbox in the future. The use of the toolbox to openly discuss one’s sex life, hepatitis C risks and related risk reduction strategies with friends and sexual partners was reported by many participants and suggests that the toolbox has contributed to encourage men to discuss sensitive HCV related topics. It has previously been shown that peers are more likely to influence behaviour, compared to mass media programs, since they are able to build trust among fellow group members, which allows for open discussions on sensitive topics [20]. We believe that the ‘viral’ quality of the toolbox can contribute to further inter-peer dissemination of HCV knowledge and awareness. In the field of HIV prevention it has also been shown that peer interaction is an effective tool with long term effect for behaviour change among groups at high-risk of HIV [21].

The personal experiences with the toolbox demonstrate that the impact is different for each user. Some users reported that the NoMoreC toolbox has mainly impacted their knowledge and awareness about HCV, risk reduction and has helped them to assess their personal HCV risk. Others explained that they have changed their sexual behaviour based on the information provided. The use of the toolbox as a conversation starter was appreciated by many men, showing the wide acceptability of the toolbox within the target group. Furthermore, the users reported to use the toolbox products during one-on-one sex, group sex and chemsex, showing that the toolbox is suitable in a variety of contexts. Even though some dismissive reactions from users were reported in relation to the safe drug use box in the toolbox, almost one third of the quantitative study participants uses items for safe drug use and almost 40% intended to use them in the future. This suggests that the provision of safe drug equipment is important to some of the men in our target group and was reported in some cases to facilitate more structural safe practices of drug use. We were not surprised to find that some users were somewhat shocked by the safe drug use box, because its inclusion in the toolbox caused the most discussion during the development process [14]. We believe that the provision of safe drug equipment is important to some men of the target group, and even though controversial it does answer to a need.

Our results show that among professionals of the STI clinic and HIV treatment centres in Amsterdam the toolbox is a well-received intervention. They reported that the toolbox is a good educational tool that assisted them in discussions about risk reduction with clients. However, for some professionals, bringing up certain sexual techniques such as fisting and sharing toys can be difficult. In addition, consultation protocols that do not offer time or space to thoroughly discuss high-risk sexual behaviour and risk knowledge hamper the possibility to identify men at risk of HCV. This was also highlighted by a Swiss study among HIV-positive MSM with a past HCV infection, which showed that if professionals rely only on condomless anal sex with non-steady partners as a criterion to identify men at risk of HCV infection, disregarding information on other high risk-behaviours, such as fisting or group sex, they miss a proportion of MSM at risk [22]. We recommend the use of a comprehensive list of potential risk behaviours to identify men whose sexual and drug use behaviours increase their risk of HCV infection. The toolbox can assist health professionals in identifying MSM at risk of HCV and prompt risk reduction conversations within consultation settings.

There are some limitations to this study. Participants of the quantitative study received the online questionnaire 12 weeks after the receipt of the toolbox, therefore we measured their experiences with the toolbox over a relatively short period. However, we did provide in our qualitative data the perspective of users who had received the toolbox up to two years prior to the interview, giving some insight into their experiences and possible behaviour change over a longer period of time. Future studies into the psychosocial, cultural, and personal factors that contribute to long term HCV risk reduction behaviour are recommended. In addition, support needs for MSM who are willing to change their behaviour and for MSM who are resistant to change are likely to be different. Therefore, studying both groups is essential for the development of future effective interventions.

We acknowledge that our study may have limited representativeness as data was collected only from participants who already ordered the toolbox and were willing to participate in the study, or from professionals who had reported use of the toolbox in the past. These participants may have been more positive about the intervention than those who did not want to obtain or use the toolbox, giving more favourable study results. Nevertheless, even among this highly motivated sample we uncovered some of the barriers to the (consistent) use of the toolbox.

Conclusion

This study has illustrated how a practical and relatively simple intervention, might contribute to facilitating HCV risk reduction. The HCV prevention toolbox was well-received by both MSM at risk and professionals. It contributed to raising awareness of HCV risk, implementing risk reduction strategies and it has given men a better understanding of the risk factors and how to reduce these. Our study showed how the toolbox was used to disseminate inter-peer HCV-related information and suggest the potential of the toolbox approach in the effort to impact the community norms. Furthermore, the toolbox can serve as an example intervention for countries that have similar sexual HCV transmission dynamics among MSM.

Questionnaire sent to toolbox recipients.

(PDF)

Click here for additional data file.

Interview guide.

(PDF)

Click here for additional data file.

29 Jul 2021

PONE-D-21-13933

A mixed methods evaluation of an innovative hepatitis C risk reduction intervention for men who have sex with men

PLOS ONE

Dear Dr. Prinsenberg,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please ensure that a thorough proof-read is conducted on the manuscript. Both reviewers have noted that the IMB model is mentioned but not discussed in depth. I agree with this and encourage the authors to elaborate on the inclusion of the model and integrate it better throughout the manuscript.

Please submit your revised manuscript by Sep 12 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see:  http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at  https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Benjamin R. Bavinton

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information. If the original language is written in non-Latin characters, for example Amharic, Chinese, or Korean, please use a file format that ensures these characters are visible.

3. Please state whether you validated the questionnaire prior to testing on study participants. Please provide details regarding the validation group within the methods section.

4. Please include a copy of the interview guide used in the study, in both the original language and English, as Supporting Information, or include a citation if it has been published previously.

5. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“This project was performed within the MC Free consortium. MC Free is funded by grants from Gilead Sciences, AbbVie, Janssen-Cilag, Merck Sharpe & Dohme, and Roche Diagnostics. The funders had no involvement in the study design, writing of the manuscript, and decision to submit the article for publication. “

We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

”TP and MP report speaker fees and grants from Gilead Sciences, Merck Sharpe & Dohme, and AbbVie paid to their institute. MvdV's institute received unrestricted research grants and consultancy fees from AbbVie, Gilead, Johnson & Johnson, Merck Sharpe & Dohme, and ViiV Health Care.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

6. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: A mixed methods evaluation of an innovative hepatitis C risk reduction intervention for men who have sex with men

This paper details the mixed-methods evaluation of a toolbox provided to MSM at risk of HCV. The methods, analysis and write-up are straightforward and the output is suitable to a range of audiences within academia, clinical settings and the voluntary sector which is great to see. I have a few comments to further strengthen this publication.

The authors state that the contents of the tool-box were collectively decided with community members. It would be really helpful to have a greater understanding of this process and the reasons which underpinned intervention design. Co-creation is an important approach and it would help the reader to understand how this worked and what the implications were.

A related point, the authors state the IMB model was used as an explanatory framework in describing how the toolbox was formed. It would be helpful to have a greater elaboration on this. Currently it is only mentioned briefly in the introduction and the discussion, and I think there needs to be greater reflection. For example, was there a theory of change or a logic model produced which used on this model to conceptualise how the toolbox could facilitate behaviour change? If so this should be described and presented, perhaps in supplementary information. I’m also not sure how well the results demonstrate the utility of IMB model in this application, more reflection on this in the discussion is required.

Finally, and this is more of a suggestion, the results from the healthcare providers seem to mirror those of the MSM who used the tool-box quite closely. I think these could be merged easily to create a more coherent narrative around the tool-box and its utility. This would also improve legibility.

I did not proof-read or look for minor grammatical errors.

Thank you for the opportunity to review this interesting work.

Reviewer #2: Thank you very much for the opportunity to review this manuscript. This manuscript aimed to assess the use of an innovative hepatitis C risk reduction intervention and the impact on behaviors among men who have sex with men. The topic is of great importance, however, the research and result interpretation part may require further adjustment. The comments are as follows:

1. Title: the authors referred this study to an evaluation of the intervention. The study seemed more like a presentation of the current implementation of the intervention rather than an outcome evaluation, as the latter may require more sophisticated research design to conduct a causal inference. It may be better to adjust the title to better fit the research objectives.

2. Introduction: the authors regarded toolbox as an innovative intervention. It may be more persuasive to add some background of this intervention that indicates how innovative it is by comparing it with prior interventions.

3. Methods: the authors reported that the toolbox had been designed according to the IMB model. “According to the model, HCV prevention information and motivation each have a direct effect on behavior as well as combined effect through behavioral skills (P4 Lines 83-85). Please further elaborate this sentence by specifying how the IMB model informed the intervention design, what was the specific direct effect of each tool and what were the combined effect of this toolbox.

4. Research design: the authors reported that the objective was to assess the use of an innovative hepatitis C risk reduction intervention and the impact on behaviors. I assumed that the authors mainly concerned the consumers in terms of intervention utilization rather than providers. If it is the case, I suggest the authors focusing on reporting findings regarding consumers attitude and behaviors. Some findings from the interviews with providers could be integrated into main themes, for instance, barriers for intervention utilization behaviors.

5. Methods: the authors conducted online survey and telephone interview, respectively. Were these two samples mutually exclusive to each other? If not, it may not be appropriate to conduct inter-group comparison.

6. Results: please move the descriptions on research methods (e.g. “the interviews lasted 30 to 60 minutes” P9 Lines 178-179) to the Methods section.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

13 Oct 2021

Response to reviewers’ comments

Title: A mixed methods evaluation of an innovative hepatitis C risk reduction intervention for men who have sex with men

This document includes the reviewers’ comments and our response to each comment. The changes we have made in the manuscript in order to address these comments are detailed in our responses

As suggested by reviewer #2 (comment 1) the title is changed to: Usability, acceptability, and self-reported impact of an innovative hepatitis C risk reduction intervention for men who have sex with men: a mix methods study

Reviewer #1:

This paper details the mixed-methods evaluation of a toolbox provided to MSM at risk of HCV. The methods, analysis and write-up are straightforward and the output is suitable to a range of audiences within academia, clinical settings and the voluntary sector which is great to see. I have a few comments to further strengthen this publication.

1. The authors state that the contents of the tool-box were collectively decided with community members. It would be really helpful to have a greater understanding of this process and the reasons which underpinned intervention design. Co-creation is an important approach and it would help the reader to understand how this worked and what the implications were.

We have added more information on the co-creation process, under ‘Toolbox development and distribution’ in the methods section. In short, it involved: 1) meetings to assess information needs, 2)discuss and generate the intervention ideas, 3) brainstorm sessions on the content of the toolbox, 4) final decision making on the toolbox content and 5)packing of the toolbox. (clean revised version: lines 95-123)

2. A related point, the authors state the IMB model was used as an explanatory framework in describing how the toolbox was formed. It would be helpful to have a greater elaboration on this. Currently it is only mentioned briefly in the introduction and the discussion, and I think there needs to be greater reflection. For example, was there a theory of change or a logic model produced which used on this model to conceptualise how the toolbox could facilitate behaviour change? If so this should be described and presented, perhaps in supplementary information. I’m also not sure how well the results demonstrate the utility of IMB model in this application, more reflection on this in the discussion is required.

The IMB-model was solely used to guide the development of the intervention, and not for evaluation purposes. We have explained the reason for using the IMB model, as the basis of the intervention in the introduction IMB- based interventions have been shown to be effective in preventing risky sexual behaviours among groups exposed to these interventions (lines 82-84).

In addition, we have elaborated on how the IMB model constructs were integrated in the intervention. We have given the aims for each construct and how the different intervention components contribute to reaching these aims. (clean version lines 104- 109) The intervention components and their focus on information, motivation and behavioural skills have been specified in Table 1.

The reviewer is correct that we did not use the IMB-model for the assessment of the effect of the intervention. We concentrated on users’ experiences with the toolbox and report on these under four main themes: 1) Reasons for obtaining toolbox, 2) Toolbox usability and acceptability, 3) Context of use of the toolbox, and 4) Self-reported impact of the toolbox on HCV awareness and behaviour change.

3. Finally, and this is more of a suggestion, the results from the healthcare providers seem to mirror those of the MSM who used the tool-box quite closely. I think these could be merged easily to create a more coherent narrative around the tool-box and its utility. This would also improve legibility.

We combined the perspectives of the toolbox recipients and health care professionals by integrating the results of the health care professionals into theme 1 (reasons for obtaining the toolbox).

I did not proof-read or look for minor grammatical errors.

The manuscript was proof-read by an English native speaker.

Thank you for the opportunity to review this interesting work.

Reviewer #2:

Thank you very much for the opportunity to review this manuscript. This manuscript aimed to assess the use of an innovative hepatitis C risk reduction intervention and the impact on behaviors among men who have sex with men. The topic is of great importance, however, the research and result interpretation part may require further adjustment. The comments are as follows:

1. Title: the authors referred this study to an evaluation of the intervention. The study seemed more like a presentation of the current implementation of the intervention rather than an outcome evaluation, as the latter may require more sophisticated research design to conduct a causal inference. It may be better to adjust the title to better fit the research objectives.

We agree that our study does not measure causal inference but rather the self-reported impact of the intervention by its users. Our study is still an evaluation, but not an impact evaluation that aims to establish causality between the intervention and outcomes. We concentrated mainly on the usability, acceptability and the self-reported impact of the intervention by its user. Therefore, we have changed the title to: Usability, acceptability, and self-reported impact of an innovative hepatitis C risk reduction intervention for men who have sex with men - a mix methods study

2. Introduction: the authors regarded toolbox as an innovative intervention. It may be more persuasive to add some background of this intervention that indicates how innovative it is by comparing it with prior interventions.

To the best of our knowledge, there is only one other published HCV risk reduction intervention for MSM that also aims to reduce high risk sexual behavior. This intervention was developed in 2016 as part of the Swiss HCVree trial. The intervention is a counselling intervention for HIV/HCV co-infected MSM, to improve self-regulation of risks associated with high risk sexual behaviours and sexualised drug use. The counseling intervention was provided in combination with DAA treatment. We have taken a different approach by targeting MSM based on their sexual practices and not their HIV/HCV-status.

Furthermore, our intervention is innovative because: 1) there is a lack of interventions that go beyond promoting condom use alone, 2) it contains practical tools for HCV risk reduction, 3) was co-created with the gay community. We have added this to our introduction.

3. Methods: the authors reported that the toolbox had been designed according to the IMB model. “According to the model, HCV prevention information and motivation each have a direct effect on behavior as well as combined effect through behavioral skills (P4 Lines 83-85). Please further elaborate this sentence by specifying how the IMB model informed the intervention design, what was the specific v and what were the combined effect of this toolbox.

We used the IMB model as a theoretical framework to guide the development of the intervention. We did not use the model to assess the effect of the intervention, we were interested in the users’ experiences with the toolbox. In addition, see the response to comment 2 of reviewer 1.

4. Research design: the authors reported that the objective was to assess the use of an innovative hepatitis C risk reduction intervention and the impact on behaviors. I assumed that the authors mainly concerned the consumers in terms of intervention utilization rather than providers. If it is the case, I suggest the authors focusing on reporting findings regarding consumers attitude and behaviors. Some findings from the interviews with providers could be integrated into main themes, for instance, barriers for intervention utilization

We integrated the perspectives of the professionals in theme 1: Reasons for obtaining the toolbox.

5. Methods: the authors conducted online survey and telephone interview, respectively. Were these two samples mutually exclusive to each other? If not, it may not be appropriate to conduct inter-group comparison.

The reviewer is correct, there could possibly be overlap. We are not able to correct for that overlap since the questionnaires were collected anonymously. Therefore, we decided it would be best to remove the comparison.

6. Results: please move the descriptions on research methods (e.g. “the interviews lasted 30 to 60 minutes” P9 Lines 178-179) to the Methods section.

Study period, number of participants and duration of interviews have been moved from the results to the methods section.

We have amended our Acknowledgements section and Funding Statement to:

Funding statement

This study was performed within the MC Free consortium. MC Free is funded by grants from Gilead Sciences, AbbVie, Janssen-Cilag, Merck Sharpe & Dohme, and Roche Diagnostics. The funders had no involvement in the study design, writing of the manuscript, and decision to submit the article for publication. TP and MP report speaker fees and grants from Gilead Sciences, Merck Sharpe & Dohme, and AbbVie paid to their institute. MvdV's institute received unrestricted research grants and consultancy fees from AbbVie, Gilead, Johnson & Johnson, Merck Sharpe & Dohme, and ViiV Health Care.

Acknowledgements

We would like to thank all participants and professionals, who shared their experiences with the toolbox with us.

Data availability

We have uploaded our data to Figshare, they can be access on:

https://doi.org/10.6084/m9.figshare.16676872.v1

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

26 Nov 2021

4 Jan 2022

We have submitted a revised version of the manuscript that addresses the new points raised by one of the reviewers.

Thank you for the opportunity to strengthen our paper.

We have uploaded the response to reviewers document with a description of how the suggestions of the reviewer were incorporated in the manuscript.

Submitted filename: Response te reviewers.docx

Click here for additional data file.

25 Jan 2022

Usability, acceptability, and self-reported impact of an innovative hepatitis C risk reduction intervention for men who have sex with men: a mixed methods study

PONE-D-21-13933R2

Dear Dr. Prinsenberg,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Benjamin R. Bavinton

Academic Editor

PLOS ONE

9 Feb 2022

PONE-D-21-13933R2

Usability, acceptability, and self-reported impact of an innovative hepatitis C risk reduction intervention for men have sex with men: a mixed methods study

Dear Dr. Prinsenberg:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Benjamin R. Bavinton

Academic Editor

PLOS ONE