Conversations that count: Lessons from evaluating a men's digital mental health response during COVID-19.

This paper details disruption and innovation in digital evaluation practice at Movember, as a result of the COVID-19 pandemic. The paper examines a men's digital health intervention (DHI) - Movember Conversations - and the product pivot that was necessary to ensure it could respond to the pandemic. The paper focuses on the implications of the pivot for the evaluation and how the evaluation was adapted to the COVID-19 exigencies. It details the redesign of the evaluation to ensure methods wrapped around the modified product and could deliver real-time, practical insights. The paper seeks to fill knowledge gaps in the DHI evaluation space and outlines four key principles that support evaluation re-design in an agile setting. These include a user-centred approach to evaluation design, proportionate data collection, mixed (and flexible) methodologies, and agile evaluation reporting. The paper concludes with key lessons and reflections from the evaluators about what worked at Movember, to support other evaluators planning digital evaluations.

Introduction

Digital health interventions (DHIs) have occupied a pivotal role in supporting health systems (World Health Organization, 2016). They provide high reach, low cost, accessible and scalable interventions that overcome time and resource constraints (Hewitt et al., 2020), transforming how health care is provided and experienced (Hambleton and Aloizos, 2019). This has been especially critical during the COVID-19 pandemic, with a proliferation of technological solutions to support the tracking, diagnosis, treatment as well as sharing of information related to the disease (Sarbadhikari & Sarbadhikari, 2020).

Of equal importance are DHIs which tackle the mental health impacts of COVID-19. In Australia, telehealth services ‘peaked at the end of April 2020 when half of Medicare Benefits Schedule (MBS) mental health services were provided remotely’ (Australian Institute of Health and Welfare, 2021a). Beyond Blue – an Australian mental health and wellbeing support organisation – received over 27% more contacts during the 4 weeks leading to January 24, 2021 compared to the same period in 2020 (Australian Institute of Health and Welfare, 2021a). Demand for such services has coincided with widespread restrictions to movement and physical isolation, disruption to schooling and office-based work, as well as rising unemployment and the loss of social interaction. Commentators have observed negative psychological effects including elevated stress, anxiety (World Health Organization, 2021), confusion, and anger, associated with COVID-19. The World Health Organisation (WHO) have warned that long-term mental health issues such as depression and substance misuse could be exacerbated by the pandemic (World Health Organization, 2021).

COVID-19 impacts on men

New research is providing deeper insights into how COVID-19 is impacting the mental health of men. The latest Ten to Men: The Australian Longitudinal Study of Male Health report showed that mental ill health remains high among Australian men with up to 25% being diagnosed with a mental health disorder in their lifetime, and anxiety and depression being the most commonly reported. The rate of male suicide is alarmingly high. Men consistently show higher rates of suicide – around 3-4 times that of the female rate (Australian Institute of Health and Welfare, 2021b) – yet they are less likely to discuss or seek help for mental health concerns (Mental Health Foundation, 2021; also see Black Dog Institute, 2020). The Ten to Men study further flagged a significant link between loneliness and experiences of depression and suicidality, a cause for concern noting increasing social isolation experienced through the pandemic (Australian Institute of Family Studies, 2020). With this in mind, the Australian Institute of Family Studies (AIFS) Director has noted that it is ‘more important than ever for Australian men to be reaching out and seeking support when they need it’ (Australian Institute of Family Studies, 2020).

Movember’s digital health interventions

Movember is the leading charity changing the face of men’s health on a global scale, focusing on mental health and suicide prevention, prostate cancer and testicular cancer (Movember, 2021). Having funded over 1,250 men’s health projects in Australia and around the world, Movember works in collaboration with local partners and academics to design and develop accessible, evidence-based programs which reach and appeal to men. Specifically during the pandemic, Movember has orientated efforts towards digital mental health interventions, identifying at-risk groups within the male population – such as fathers and adolescent boys – to tailor evidence-based solutions via open-source channels. This approach supports maximum reach, particularly given the COVID context.

Movember conversations: an evidenced-based tool

Movember Conversations is one such product in the Movember mental health suite of programs – a free online interactive tool designed to give practical guidance on how to talk with men who might be struggling (Movember, 2020a). Users can choose from a range of scenarios and work through simulated conversations to practice how to navigate a difficult conversation. The concept grew from research commissioned by Movember in 2014 that examined social connectedness among men, with past studies demonstrating this as a protective factor against depression (Arbes et al., 2014). The study found that most men in their middle years will experience dissatisfaction with their social lives at some point, with 25% having nobody outside their immediate family to rely on, and one in three men feeling dissatisfied with the quality of their relationships. While there are a range of complex factors that may contribute to poor social connection, a significant finding was that while many men desired greater openness and would like to discuss personal problems with their friends, they felt they lacked the skills to initiate open conversations and would not know how to respond if a friend were to open up to them (Arbes et al., 2014). One of the outcomes of this research was to build an online tool to guide users through the process of how to initiate and maintain confronting conversations, as well as provide support to friends or family to do the same (Arbes et al., 2014).

An evolving tool for the COVID context

On 16 March, a State of Emergency was declared in the state of Victoria with 71 confirmed cases of COVID-19 in the community (Department of Premier and Cabinet, 2020). At this time, Movember Conversations was being designed and developed, with content and program packaging being tailored towards organisations with a high number of working aged men (workplaces and universities). From March 21, 2020, social distancing rules were imposed and many non-essential services and offices began to close (Storen and Corrigan, 2020), prompting questions about how Movember Conversations could effectively reach its identified market. This became particularly evident during the introduction of Stage 3 restrictions in Victoria on March 30, 2020, as well as rising concerns about the mental health impacts of COVID-19, seeing millions of dollars being channelled towards mental health services across Australia (Storen and Corrigan, 2020).

In response, Movember identified an opportunity for Movember Conversations to address the emerging problem associated with the effects of social isolation. It seemed likely that the product could have universal appeal given the unprecedented circumstances Australians – and men around the world – were experiencing through the physical severance of social connection.

Movember made the decision to reorientate towards an open access product which could speak to men from all backgrounds. They began a process of identifying common scenarios or key issues relevant to men’s experiences of COVID-19 that would resonate across their six key markets (Australia, Canada, Ireland, New Zealand, the UK and the US). Whilst the pandemic was being experienced differently across the world, Movember’s evidence-based directive was to focus on social connectedness and shape the product to speak to universal issues, while providing practical strategies to build and maintain social ties.

Three different scenarios were developed to support men who may be

• unemployed and uncertain,

• juggling work and family and

• withdrawn and obsessing.

Movember Conversations addresses each of these scenarios within an interactive episode whereby the user (the viewer) is presented with two protagonists: the first experiencing a difficult time and the second representing the viewer. Multiple choice options are presented after the scene is set, allowing the viewer to select the response and determine the course of the conversation. The tool guides the viewer through these challenging conversations, teaching the kind of prompts and responses that help to open up a conversation and keep it flowing meaningfully (Movember, 2020b). To structure the conversations, Movember employed the ‘ALEC’ model (ask, listen, encourage and check-in), a simple four-step construct to encourage men to open up, developed by RU OK. The product was launched during Movember’s May Eight campaign which encourages Australian men to use the date to catch up with a mate and tackle social isolation. Within 5 months of operation (launch to October 2020), Movember Conversations reached 654,909 unique users across over 100 countries.

Evaluating digital health interventions

DHIs have huge potential as scalable tools to improve health outcomes. However, a sound knowledge base is required to inform development and deployment (Murray et al., 2016). Systematic DHI evaluations are currently few in number, meaning that there is an absence of best practice guidelines to structure assessment (Kowatsch et al., 2019). Evaluating DHI further presents a set of unique challenges (World Health Organization, 2016), including the rapid change of the technological landscape, and iterations as products evolve from prototype to stable and scalable interventions (Murray et al., 2016). While randomised controlled trials (RCTs) are the most commonly recognised evidence for healthcare interventions, they are few in number and are not always practical given the speed of digital product development and the need for closer to real-time insights (Guo et al., 2020).

Different types of evaluation activity are required through a product’s lifecycle – from monitoring technical functionality and stability, to usability and accessibility testing, through to assessment of impact at scale – with methodologies needing to be as adaptive and agile as the product itself.

Movember Conversations showcases these challenges and solutions. As the first DHI evaluation at Movember, the process has been one of testing and learning, and exploring what works in a digital setting. A key feature supporting this dynamic, iterative approach has been embedding evaluation early in the product’s evolution and working closely with product developers prior to release.

Evaluating Movember Conversations

An embedded approach to evaluation at Movember has facilitated evaluators’ knowledge of the product, as well as developers’ understanding of evaluation. Furthermore, it has supported agility in terms of shaping the evaluation around the revised COVID-19 (open access) edition of Movember Conversations and refining key evaluation questions, timeframes and methodological considerations in collaboration with the product team. Table 1 provides an overview of the key changes made to the evaluation arising from the COVID-19 product pivot.

Table 1.

Changes to monitoring and evaluation due to COVID-19 product pivot.

Evaluation focus	Changes to evaluation arising from the COVID-19 product pivot
1. Product reach across the intended markets and whether the product was delivered as intended	The shift to an open access product involved expanding data collection from a contained audience to a global population. Google Analytics (GA) was an effective mechanism to track and report traffic to the website.
2. Engagement among users of the product including completion of episodes and associated learning outcomes	Episodes were modified to include scenarios pertinent to COVID-19 (such as isolation and boredom); however, the intent to upskill and build people’s confidence to have conversations remained. GA was useful for tracking user engagement across episodes to understand which content was resonating with users by country.
3. Action taken by users following product engagement	The evaluation scope was widened to consider ways to collect data from a global population in a cost effective and efficient way. Surveys functioned for this purpose; however, distributing surveys was a key consideration.
4. Long-term habits adopted by users; the impact Movember Conversations has had	The scope was widened to consider how to sustain evaluation participation with a global – and potentially anonymous – population.

As outlined, the evaluation scope and methodological conditions changed as the product pivoted to a COVID-19 response and a global audience. However, four consistent principles were retained when designing and developing the evaluation, as addressed below.

User-centred approach to evaluation design

A key component of DHI development is user experience (UX) research to ensure the product is accessible and functional and understands user needs. UX research during prototyping can support product usability through examining whether users can seamlessly access and engage with a DHI, and if it meets their needs (Wilson et al., 2018). This is important before any assessment can be made about the impact of the DHI. If a user is deterred from fully engaging with a product, this can compromise or dilute the intended outcomes.

This stage of development further offers opportunities to guide evaluation design. For Movember Conversations, a user-centred approach was taken when designing how and when to collect data for the evaluation. Primacy was placed on minimising disruption to the users’ experience of Movember Conversations – for example, not requiring users to register to experience the product – but also integrating data collection into the experience to support evaluation participation. A process of user journey mapping – a common UX activity – helped to identify potential data collection points, noting that the average time spent in product was estimated at 2.5–5 minutes. As such, there was a narrow window of opportunity to embed evaluation data collection – and it needed to be proportionate to time spent in product.

Proportionate data collection mechanisms

The principle of proportionality narrowed the selection of data collection tools, as well as those which would not divert the user away from Movember Conversations. This dictated a short but targeted set of questions that could be incorporated into the experience but would clearly demarcate these from the experience. The tool adopted was Hotjar: a web-based feedback poll that pops-up in the corner of the screen during an episode but is clearly separate to Movember Conversations. This provides users with the option of participation: they can either answer the questions, ignore the pop-up, or simply click on the ‘x’ to exit the poll.

Furthermore, just four questions were incorporated into Hotjar and comprised multiple-choice answers to facilitate quick insights. Questions were also designed to be cognisant of what users could reasonably provide feedback on at that point in time, but also what would be of most use to Movember in the early release stages. It would be unreasonable to ask questions concerning impact or application of lessons during the experience. Instead, questions focus on usefulness of the experience, knowledge acquired and motivation for visiting Movember Conversations – a particularly salient question given the COVID context.

In addition, Hotjar offers the functionality to collect free text data, which can be used as a mechanism to obtain consent for further contact. Users are asked to provide a contact email if they wish to participate in the evaluation and consent to receiving a longer follow-up survey. This is an optional question meaning that participants were able to provide Hotjar responses anonymously if they did not want to provide their email and participate in further evaluation activity.

The response rate through Hotjar has been encouraging. Over the first five months, 845 responses were obtained, with 215 opt-ins for the survey, representing a 25.4% response rate. It is likely that a sufficient sample will be achieved for Hotjar responses by the end of the evaluation data collection period later in 2021 to provide reliable data (95% confidence level with a +/−5 confidence interval).

Mixed methodology to collect impact data

Whilst Hotjar was considered the most proportionate data collection mechanism for Movember Conversations from an in-product user perspective, it was limited in the insights it could provide to inform the evaluation about health outcomes. This was particularly in terms of collecting data about behavioural changes after a user engages with the product (such as application of learnings or having a conversation). As with any evaluation, a mixed method approach is preferable to fully capture all the complexities of how programs operate, generally through combining quantitative and qualitative data (Better Evaluation, 2012).

An online survey (Qualtrics) was selected for this purpose, due to the ease of distribution to an online population, and the low cost and convenience of automated data collection. Emails acquired through Hotjar can be fed into Qualtrics using an application programming interface (API): a software intermediary that allows applications to interact with each other. This enables Qualtrics to automatically send a survey to participant emails eight to 24 hours after completing Hotjar. This time delay ensured that participants have time to reflect on their learnings while the experience is still fresh in mind.

Questions were designed to capture demographic information about participants (to understand the audience of Movember Conversations), as well as their reflections about the product. Questions focus on whether confidence and understanding to have conversations has increased, as well as if there is intent to put these learnings into practice. It was also valuable to understand whether participants were likely to share Movember Conversations, as indicator of its perceived utility but also of potential reach via referrals.

Through participants providing their email in Hotjar and opting into a survey, it has been possible to link responses between data sources over time, to support longitudinal analysis about how Movember Conversations has been adopted by users. This is further supported by a follow-up survey sent to participants four to 6 weeks after completing the first survey. Follow-up questions focus on whether application of learning has occurred, as well as how well participants felt their conversations had gone using skills acquired from Movember Conversation.

The third data collection method is online interviewing to obtain deeper, personal insights into participants’ use of Movember Conversations. Recruitment is via the opt-in survey, with the option for participants to participate in an interview as opposed to the follow-up survey. Qualitative interviews are beneficial to explore how Movember Conversations has been applied in different contexts – particularly given the COVID climate – to understand ‘what works, for whom and in what circumstances’ (Pawson and Tilley, 2004, 2). A purposeful sampling strategy has been undertaken to identify participants from across Movember’s key markets to distil contextual lessons which can help in future scaling of Movember Conversations.

An agile approach to inquiry and reporting

As aforementioned, a different type of evaluation approach was required within an agile product development setting. Traditional evaluation approaches and reporting cadence was not appropriate for Movember Conversations as an evolving digital tool within the changing COVID context. As a result, different methods were employed to capture the different types of information required over time.

In the early stages of the product release, the emphasis was on monitoring product reach via Google Analytics which captured data such as traffic to the site, bounce rates and time spent in product. This supported real-time insights about the reach of Movember Conversations (including number and location of users) and engagement (such as number of completed episodes), which is vital for continuous improvement and product modification. Similarly, Hotjar was a critical data point for the early stages of release, to obtain insights about the usefulness of the product experience and relevance of episodes for the COVID context.

As the product matured, evaluation data served as the critical data point to understand the usefulness of episodes for the COVID context, and whether the product was fulfilling its objective to prepare users to have conversations with men going through a tough time. Survey and interview data was required to unpack these deeper insights and explore whether skills and knowledge to support men had increased. More frequent evaluation knowledge sharing about these emerging insights – through bite-sized reports and presentations – enabled a closer to real-time view about product performance and raised new questions along the way for the evaluation to consider.

The evaluation was therefore flexible enough to pivot and include emerging questions that could deepen insights and support product evolution. In particular, there was appetite to drill down into the motivations for why people arrived at Movember Conversations. This was important from a marketing and distribution perspective, but also in terms of shaping content or modifying the product to cater for specific audiences. This resulted in a revision to the Hotjar questions (after obtaining a sufficient same size), to focus on motivations, and monitoring responses to see whether this generated relevant insights. This has been a key feature of the evaluation, in terms of evolving with the product, as opposed to a set and forget approach to data collection.

Key lessons for evaluating a digital COVID response

Movember Conversations was the first DHI evaluation conducted at Movember and presented a unique opportunity to both test and learn what works in a digital setting, as well as how to evaluate a COVID response which by nature, was unknown and unprecedented. A set of three key lessons have been extracted, which will inform future evaluation design at Movember.

Embed evaluation early-on

A key enabler to support an agile approach to digital evaluation was integration with the product team. Working with the product team early-on during the prototyping of Movember Conversations enabled evaluation design to be considered and incorporated within the technical build. This had three benefits: the first being that the evaluation could be designed with the product in-mind and with an understanding of what was possible from a technical perspective. Secondly, it helped obtain buy-in among the product team through their involvement in the evaluation design. It worked particularly well in terms of ensuring that data collection mechanisms were considered in relation to the product itself. Thirdly, it meant that decisions about product build could be influenced by evaluators to ensure evaluation was embedded as a component of the product, supporting integration, as opposed to ‘bolting-on’ evaluation.

Establish agile evaluation that can complement agile products

The key lesson above discussed the benefit of integration to support agility. Leading from this is evaluation design that adopts agile methods that can complement agile product development. In short, an iterative approach to evaluation characterised by ‘flexibility (in methods), rapid evaluation and changing protocols as technology changes’ (Wilson et al., 2018) was a key strength as inquiry could evolve in line with the product. Underpinning this was a working relationship between evaluators, product owners and developers that ensured that the evaluation could more readily wrap around the product in terms of being receptive to product changes and therefore emerging lines of inquiry. Tweaks and adjustments were made to data collection instruments, such as Hotjar questions, to support product changes and ensure evaluation data continued to be useful and fill knowledge gaps as the product evolved.

It is important to remember that digital evaluation is not one-size fits all, and evaluation methods will need to be tailored to each product. The paper discussed proportionality of data collection and the appropriateness of Hotjar for Movember Conversations; the type and style of data collection suited this digital product. However, success of these methods will vary depending on the type and nature of the product. Use of Hotjar in other DHIs at Movember has been less successful, necessitating agility in terms of the methods used to ensure these are complementary to product’s type, style, format and target audiences. In keeping true with the first principle, it is essential to take a user-centred approach to evaluation design of DHIs.

Establish agile reporting to support real-time insights

The final lesson for future DHI evaluations is to accelerate the provision of evaluation findings where possible. Monitoring data collected via Google Analytics supported real-time understanding about the reach of Movember Conversations, as well as user engagement among markets. However, emerging evaluation insights were also valuable. This was seen for the short-term impacts of Movember Conversations, where it was possible to observe emerging insights about how users were using and applying knowledge learnt through the product. For example, the survey administered within 48 hours of using Movember Conversations captured data on whether users either had, or intended to have, a conversation with a man, which was a valuable data point to inform product refinement. These emerging insights could serve as a ‘pulse check’ or indication about progress towards the intended outcomes, while emphasising to the product team the limitations of inconclusive evaluation data at this interim point. This balance – between what is known and unknown – is a crucial aspect to the role of the evaluator in DHIs, in both supporting continuous improvement and roll-out, and ensuring that conclusions about impact are underpinned by robust data.

Conclusion

Digital health interventions have huge potential to improve health outcomes. However, there is a deficit of systematic DHI evaluations to inform the evidence base about what works and how to evaluate digital products effectively. This paper intended to help fill this gap by outlining Movember’s experience evaluating Movember Conversations, showcasing innovative practice to overcome the challenges that were faced in an agile environment. Based on this experience, it presented key principles for designing and conducting evaluation around a dynamic digital response. This included embedding evaluation into the product build phase; close monitoring of data collection to adjust and supplement methods where necessary; and an accelerated reporting cadence for sharing real-time insights. These components worked hand in hand to transition the evaluation towards a more flexible and agile way of working which complemented and supported digital product development. It is hoped that these lessons can assist other evaluators as they plan for and work within the digital landscape.