Crowdsourcing to design a marketing package to promote a WHO digital mental health intervention among Chinese young adults.

BACKGROUND: Digital mental health is an emerging field that can leverage technology and mobile apps to deliver mental health treatment to populations in areas with limited mental health services. Despite widespread availability of these apps, uptake remains low. Enhanced marketing is necessary to increase public engagement. There is growing recognition that mental health intervention beneficiaries should be engaged in all phases of treatment development, adaptation, and delivery. Crowdsourcing - consulting the public to solve problems and sharing the solutions - can foster community-informed ideas for public health, but has yet been applied to digital mental health marketing.
OBJECTIVE: With the goal of engaging potential intervention beneficiaries in digital mental health dissemination, the current project implemented a digital mental health crowdsourcing designathon for Chinese college students in Macao SAR, China and evaluated the feasibility of the contest and the products. The contest asked participants to design marketing packages for Step-by-Step, a scalable WHO digital mental health intervention focusing on depression.
METHODS: Designathon participants, recruited from a global health class, were sorted into teams with balanced areas of expertise. Two judging panels - one of experts in relevant fields and another of Chinese college students - evaluated the marketing packages and selected finalists. The designathon was held in-person over four days and involved debriefing, workshops, a contest, and an awards ceremony. A parallel mixed-methods approach was applied, including qualitative feedback from judges and participants alongside quantitative data on participant satisfaction and depressive symptoms to enrich our understanding of the event. Additionally, based on judges' feedback given to participants, the communication packages of the contest were ranked.
RESULTS: 49 participants (8 teams of 6-7 members each) were involved in the designathon. Using a cutoff score of 10 on the Patient Health Questionnaire-9 (PHQ-9), 11.4% of participants had moderate or higher depressive symptoms. All teams successfully produced complete digital mental health marketing packages. Four finalists' packages were selected quantitatively with judges' scores and the top finalist's package was described by judges' comments as simple, thoughtful, and appealing, although not informative enough. Participants were overall satisfied with the designathon, but some mentioned that time was insufficient and that organization/instruction clarity could have been improved.
CONCLUSIONS: The designathon is a novel, feasible strategy to collect crowd input for the dissemination of a mental health intervention. Compared with traditional communication strategies, this bottom-up approach included and engaged potential intervention beneficiaries to take an active role in creating digital mental health marketing communication. Future contests should allow participants more time and reconsider aspects of event organization.

Introduction

Within the field of global mental health, alternative methods for mental health treatment delivery are needed to reach a large population in need, for whom mental health services are not available (Patel et al., 2018). With the global proliferation of digital devices and smartphones (a 75% increase of smartphone ownership from 2013 to 14 to 2017–18 in low- and middle-income countries) (Pew Research Center, 2018), the field is now developing and evaluating digital approaches to deliver mental health interventions at scale (Carswell et al., 2018; Garabiles et al., 2019; Hall et al., 2018). As of 2017, mental health apps comprised about one third of approximately 325,000 mobile health apps available on the market (IMS Institute for Healthcare Informatics, 2015). However, despite the rise in digital mental health, uptake of these innovations remains variable – a small subset of digital health apps draw over 90% of downloads overall, while most apps have low public usage (IMS Institute for Healthcare Informatics, 2015; Fleming et al., 2018).

A major challenge for digital mental health implementation science is tailoring marketing to effectively reach target users (Muñoz et al., 2016; O'Connor et al., 2016). Work is needed to increase public awareness of digital health products, but there is a gap in the literature surrounding which engagement, dissemination, and marketing strategies are most effective in promoting uptake of digital health interventions to different groups of users (O'Connor et al., 2016; Claborn et al., 2018). A proposed framework suggests that a “stakeholder driven approach” could be a new path to implement technologies into practice (Nijland et al., 2011). One such idea is a crowdsourcing designathon (Tucker et al., 2017).

Crowdsourcing – asking a group to solve a problem through inventive challenges, hackathons, or other activities and then sharing the solutions with the public– is an emerging strategy that engages the public to foster innovative and community-informed ideas (Tucker et al., 2017; Tucker et al., 2019; DePasse et al., 2014). As a type of community-based participatory approaches, crowdsourcing invite the target audience users to engage in the development and implementation of an intervention. (Burdine et al., 2010). It can take the form of a contest with a purpose of solving the health problem (Brabham et al., 2014). Although it is a relatively new tool, especially in healthcare areas, it was deemed feasible and successful in several studies (Tucker et al., 2017; Tucker et al., 2019; DePasse et al., 2014; Burdine et al., 2010; Brabham et al., 2014; Zhang et al., 2015; Ong et al., 2017). DePasse and colleagues addressed the three core principles for crowdsourcing event: problem-based approach, participation from diverse disciplines, and challenging existing paradigms (DePasse et al., 2014). Centered on users/target audiences, crowdsourcing events focus on localization and cultural contextualization [14]. Zhang and colleagues presented their experience using crowdsourcing contests for sexual health intervention designs (Zhang et al., 2015). It is considered as a cost-effective “bottom-up” approach to collect crowd input, comparing with conventional “top-down” approach where only experts and professionals are involved [16]. Crowdsourcing contests also show positive results in community engagement (Ong et al., 2017).

Crowdsourcing designathon is one such contest. This is an event bringing participants together to design media campaigns to address a specific challenge or purpose; it can leverage in-person workshops and teamwork to create holistic communication packages (Tucker et al., 2017). Tucker and colleagues used a crowdsourcing designathon to develop a community level HIV testing program in China [12]. No study has applied such an approach to improve access to mental health or digital mental health interventions. The growing recognition within global mental health is that intervention beneficiaries should be engaged in all phases of treatment development, adaptation, and delivery suggests that crowdsourcing approaches could be a novel and equitable approach to implementation and scale-up (Ryan et al., 2019; Semrau et al., 2016).

Recently, the World Health Organization has engaged in the development, adaptation, and testing of scalable mental health interventions (Carswell et al., 2018; Garabiles et al., 2019). Step-by-Step is one of the guided self-help interventions promoted, [3, 4] aiming primarily at reducing depressive symptoms through behavioral activation, but it also has the potential to be transdiagnostic, addressing anxiety, PTSD, other stress reactions, and client-defined psychosocial problems. The intervention aims to fill the gap between treatment needs and available mental health services in contexts where there are few mental health providers. A Chinese version of the Step-by-Step intervention was developed, targeting college students in the Macao Special Administrative Region (SAR) of the People's Republic of China. To better advertise the intervention to the target users, we implemented a crowdsourcing designathon with college students in Macao to develop marketing packages for the intervention, who were themselves the targeted users. Different from applying traditional marketing product from a professional marketing company, the purpose of using a designathon in this effort was to engage and empower the intervention's end-users to create an audience-informed marketing strategy that bridges an evidence-based global mental health intervention with the target users to increase the intervention uptake with focus on cultural localization. This paper presents the process of the novel crowdsourcing designathon approach used to develop marketing packages for mental health intervention. To help inform the process of organizing the event, steps outlined by the WHO/TDR Practical Guide on Crowdsourcing were used as a basis for the designathon protocol (World Health Organization, 2018).

Methods

Steering committee

A community steering committee was formed including five individuals from relevant backgrounds, including public health, psychology, and anthropology with diverse degree levels. With the knowledge in global and community mental health and experience in organizing crowdsourcing contests (Tucker et al., 2017), the steering committee structured the designathon and developed the evaluation criteria. The committee was also responsible for preparing and presenting guidelines to participants and for providing feedback to participants throughout the designathon.

Participants

The target population of the Step-by-Step intervention is Chinese young adults. In the spirit that members of marketing package's target audience would best determine how to market an intervention intended for their community, we recruited participants from a general education course on global health at the University of Macau who are the target users of the Chinese version Step-by-Step. The designathon was incorporated as part of the global health course content.

Using demographic information from the class roster, eight teams of six to seven participants were formed based on fields of study, place of origin (local Macao Chinese or non-local Chinese), and year in university to enhance the diversity of perspectives and balance areas of expertise within each group. With the awareness that local Macao Chinese who speak Cantonese and non-local Mandarin speakers are the main student subsets in Macao, diversity in terms of place of origin was prioritized.

Judging panels

Two groups of judges, an expert panel and a student panel, were invited to evaluate the communication packages during a half-day presentation event. With the intention to evaluate the products from a professional public health perspective, psychological perspective, and communication perspective, we invited eight experts from communication, social work, clinical psychology, counseling psychology, and global health background from the university and local community organizations that provided mental health services to form an expert panel. Members of the student panel were recruited through student associations in the university and in Macao as well as through their social networks. The student panel consisted of members of students in diverse majors from different universities in Macao with the purpose to judge the products through the lens of the targeted population to evaluate the cultural appropriateness and acceptance.

Designathon process

The designathon was a 4-day event integrated into a general education course on global health at the University of Macau. Day 1 was an orientation in which the steering committee introduced important concepts to the participants, including a brief overview about health communication, scalable digital mental health programs, the Step-by-Step program including the process of guided self-help, and the contest rules/guidelines (Multimedia Appendix 1), as well as the rubric by which the judges would score them (Multimedia Appendix 2). These concepts and materials were shared with the class through a presentation and in a document posted on a class resources website (Multimedia Appendix 3). Participants were incentivized with prizes (tickets or coupons for entertainment events) donated by a local corporate sponsor.

Days 2 and 3 of the designathon were workshop days when each team was required to generate a full communication package to advertise the Step-by-Step program. The following components were required for each communication package: a title to name the “Step-by-Step” program, a health message participants aimed to convey with the package, a logo, a slogan, and two components to spread the ideas (a poster and a 15–30-s-long video). Participants were given freedom to choose what language(s) to incorporate in the package, in hopes that they would choose appropriate languages to address the target audience. Participants were also told to present in their first language to make the presentation or their ideas most comfortable for them.

On Day 4, representatives from each team were required to make an 8-min PowerPoint presentation to present their package to the judges followed by a 5-min question and answer session in a university auditorium. Note-takers, a moderator, and helpers responsible to keep time and calculate scores were arranged in advance. Following a lunch break, during which the results were calculated and discussed, an awards ceremony was held. All participants received certificates for participation, and winners received certificates for their awards.

Judging process

Prior to the presentations, judges were debriefed about the purpose of the Step-by-Step program, the purpose and rules of the contest. They were given a template to record their scores and qualitative feedback including strengths, weaknesses, suggestions for improvement, and questions for each package, as well as a rubric to judge each marketing package (Appendix Table 2). Package components were primarily assessed for creativity, comprehensiveness and concision, appropriate tone for addressing the target audience, and visual appeal.

Table 2

PHQ-9 scores (n = 44).

	Proportion	SE	95% CI
Minimal (0–4)	50.0%	0.08	0.3509	0.6491
Mild (5–9)	38.6%	0.07	0.2508	0.5421
Moderate (10–14)	6.8%	0.04	0.0211	0.1986
Severe (15–19)	4.6%	0.03	0.0108	0.1725

For scoring purposes, the expert panel's scores were given more weight in determining the designathon award winners to ensure that the judging process was informed by a comprehensive knowledge of mental health and health communication. The first, second, and third prizes were determined solely by the expert judges' scores, a most popular prize was determined by the students' panel, and bonus awards (Best Video, Best Poster, Best Logo, Best Slogan, and Best Title) incorporated scores from both judging panels in equal proportion. Results were calculated by score-keepers, who entered the judges' scores into a pre-prepared Excel formula during the contest. Judges were given the opportunity to discuss the score-determined results and veto them, if necessary, to determine the winners.

Measures

Demographic variables

The following demographic information was obtained from participants before the designathon using the survey tool Qualtrics: sex, age, major, grade, and place of origin.

Depressive symptoms

The Patient Health Questionnaire 9 (PHQ-9, Multimedia Appendix 5), one of the most commonly used instruments to assess depression, was used to in this study to evaluate clinical characteristics (Kroenke et al., 2001).It consists of 9 self-report items. The Chinese version of PHQ-9 has demonstrated good reliability and validity in Chinese populations (Wang et al., 2014; Yu et al., 2012). For each item, respondents indicate how often they have been bothered by a symptom on a scale from 0 (not at all) to 3 (nearly every day). A sample item is “little interest or pleasure in doing things.” The severity of depressive episode can be assessed by the nine items' summed score, which ranges from 0 to 27 with five severity groups: minimal (0–4), mild (5–9), moderate (10–14), moderately severe (15–19), and severe (20–27). A score greater than 10 indicated clinical depression. The internal consistency reliability of the scale was acceptable (Cronbach's α = 0.77).

Participant satisfaction

Participant satisfaction was assessed with a process evaluation feedback measure (Multimedia Appendix 6) developed for this study, which is based on an official evaluation for training classes and activities from Macao Social Welfare Bureau (Social Welfare Bureau, 2019). The first 9 items of the survey asked participants to rate the degree to which each item described how they felt about the designathon from 1 (strongly disagree) to 5 (strongly agree). A sample item is “duration of the designathon is appropriate.” The 10th item asked if the responded would recommend friends to participate in the designathon, and additional space was provided for written comments or feedback. The internal consistency reliability for the first 9 items of this scale was excellent (Cronbach's α = 0.91).

Data analysis

Parallel mixed methods approach was used in this study, including qualitative feedback from judges and participants alongside quantitative data on participant satisfaction and depressive symptoms. We chose this study design to provide a richer portrait of the experience of participation and some preliminary data on the quality of ideas generated.

Qualitative data from judges' textual feedback and participant satisfaction feedback were respectively collected during the judging process and after the designathon contest. Judges' feedback about the strengths and weaknesses of the marketing package were synthesized by two coders to extract the main ideas from the comments (Table 3 & Appendix Table 1). Participants' feedback was coded simultaneously and independently by two coders. All feedback was categorized into positive, negative, and neutral categories with sentence-to-sentence coding and 5 themes emerged during the process, which were used as pre-set codes for second cycle coding work. The coders re-coded participants' feedback with 5 pre-set codes in excel, including feedback in terms of time, organization, the judging process, general feedback, and ‘other’ and calculated the frequency of each code to identify the most prominent issues.

Table 3

Description of winners' package and feedback from judges during the designathon.

Group	Title	Slogan	Strengths	Weaknesses
Group Four(1st)	Step by Step慢慢來	Step by Step, everything will be ok! You are not alone, we're all here for you慢慢來，一切都會好起來的!	• Design was simple, thoughtful, and appealing with potential inclusion criteria and QR code for enrollment included in the poster. The physical invitation cards distributed to the audience was thoughtful and creative. Confidentiality and privacy were emphasized on the invitation letter, which made the product less threatening. The use of WHO logo and green color also helped.	• Information was not sufficient nor purposive enough. Health message was not clear or strong enough for audience to understand the purpose of the product. Target population was not well stated.
Group One(2nd)	Stay with You與你同在	Shines Your Life與你並肩，在你手邊	• The general design was suitable for young adults. • Logo was meaningful, interesting, and novel, and the other components were catchy. The symbol of WHO in the design made it look professional and dependable.	• Wording in the package was not appropriate. ‘Psychological doctor’ does not exist. • The English slogan was not clear. Poster was too complicated and the health message is not clear. There was no link to download the app in poster/video.
Group Seven(3rd)	M.Y. Health	步步走出自我 漸漸擁抱新生Be Confident Step by Step, Embrace New Life Step by Step (translated by authors)	• Design was generally good and easy to understand. • Package was systematic and thick. It was good to put marketing survey in presentation. Background was clear. • Positive thinking was included in the package. Design was good for normalizing mental health issue.	• Logo could be modified. It was not consistent with the title name and it was not relevant with health. Video was too short to state clear message and no message guidance was included to download the app.
Group Five(the most popular)	搭把手Give a Hand (translated by authors)	敢嘗試 齊面對 跨難題 無問題Dare to Try, Face the Challenges, Conquer the Problems, Find Solutions (translated by authors)	• Package was well-designed and attractive, including positive thinking element. • The idea of 4H was informative, which was easy to catch and understand. • The idea of “peer company” was helpful to reduce stigma. It was attractive for mild mental health problem. The role of E-helper was included in the design. The name seems very natural local. Hotline provided.	• Title was not easy to pronounce in Cantonese. • Although the design was meaningful, it was too complex and not clear in action taking guidance. Wording was not appropriate. The key message of “seek a doctor as soon as possible” seems to be designed for people with severe problems.

Table 1

Demographic characteristics of the participants (n = 49).

Demographic characteristics		Frequency (n)	Percentage(%)
Sex
	Men	14	28.6
	Women	35	71.4
Age
	23	3	6.1
	22	7	14.3
	21	13	26.5
	20	17	34.7
	19	9	18.4
Students' status
	Local	41	83.7
	Non-local	8	16.3
School year
	1	12	24.5
	2	31	63.3
	3	5	10.2
	4	1	2.0
Major
	Faculty of Arts and Humanities (FAH)	12	24.5
	Faculty of Social Sciences (FSS)	5	10.2
	Faculty of Science and Technology (FST)	8	16.3
	Faculty of Business Administration (FBA)	14	28.6
	Faculty of Education (FED)	10	20.4

For the quantitative data, StataMP 14 was used to generate descriptive statistics for participant demographics, self-reported depression, quantitative satisfaction feedback on the designathon, and to estimate intraclass correlation coefficient of the mean ratings for each panel.

Results

Table 1 provides an overview of all participants' demographic information (Table 1). Of the 49 participants involved in the event, 14 (28.6%) were men, and 35 (71.4%) were women. The age of participants ranged from 19 to 23 and the mean age was 20.55 (SD = 1.14). Forty-one (83.7%) participants identified themselves as local Macao students, and eight (16.3%) participants identified themselves as non-local students (seven mainland Chinese and one Malaysian Chinese). Forty-three (87.8%) participants were in year one or year two of their undergraduate program, and six (12.2%) participants were in year three or four. Fields of study were diverse, such as computer science, communication, Portuguese study, education, and political science. No participants with a psychology background were involved in the class, which was ideal for the contest since psychology students might not best represent students in general.

Table 2 demonstrates the results of PHQ-9 scale (Table 2). Among those who participated, 44 completed the pre-designathon survey. The mean score of PHQ-9 was 5.73 (SD = 4.03), ranging from 0 to 17. Using the cut-off score of 10, 11.4% of participants had moderate or higher depressive symptoms.

Final product

All eight teams developed a comprehensive package during the 4-day designathon. Group four, Group one, and Group seven were ranked as the top three communication packages by the multidisciplinary expert judging panel. And the finalist's package with highest scores from student judging group is Group five, chosen as the most popular package. The correlation among mean ratings for each panel of judges is 0.84 (0.19–0.98). Table 3 provides the description of the winners' package and feedback from judges (Table 3). The description and the feedback for the other four groups were available in Multimedia Appendix 4.

Group four won the first prize. They designed the title and slogan in both Chinese and English. They kept the English name of the intervention, Step by Step, and they additionally developed a Chinese name with similar meanings. Their slogan in English was Step by Step, everything will be ok! Their logo (Fig. 1-a) was two footprints set slightly above/below each other, representing moving forward step by step. The poster (Fig. 2-a) was green and white with leaves as decoration. A QR code to download the app was included. The video depicted a teenager who initially appeared sad, but smiles and seems to feel better after using the app, indicating that the app helps young adults overcome his depressive symptoms. Group four distributed printed invitation letters to the audience and the judges during their presentation; the letters were designed consistently with the poster and logo in terms of elements and color.

Fig. 1

Logos designed by group four, one, seven, five (from left to right).

Fig. 2

Posters designed by group four, one, seven, five (from left to right).

Group one won the second prize. They also designed the title and slogan in both Chinese and English. Their title was Stay with You in both English and Chinese, suggesting that users can use the app whenever they need it. Their logo (Fig. 1-b) included several elements as visual metaphors: a heart suggesting care for people with mental problems, gears intended to represent Step-by-Step and social networks, and an electrocardiogram suggesting fluctuating emotion. Their slogan was Shines Your Life in both English and Chinese. The poster (Fig. 2-b) was blue with keywords like professional, convenient, and eHelpers (digital program guides) to demonstrate the features of the intervention. The video emphasized the theme of Stay with You by showing the importance of guidance and app-based mental health support.

The title for group seven was M.Y. Health, which can be read as make sure your health. M in the name indicates Macao while Y indicates Youth. Their logo (Fig. 1-c) was a bee flying in a bubble with a green background. The bubble was intended to symbolize an emotional trap, and the color green was used to make people feel at peace. Their slogan was Be Confident Step by Step, Embrace New Life Step by Step (translated by authors). Their poster (Fig. 2-c) was in pink and displayed the title, slogan, and QR codes for different social media channels (Instagram, WeChat, Facebook), as well as images of sunshine, hands, and a ladder. The key message of the video was that it's okay to be not okay and that mental health problems can be solved with help and support.

Additionally, group five was chosen as the most popular package by the student judging panel (Table 3). Their title was Give a Hand (translated by authors), implying that people can get help from others when they are troubled. Their slogan was Dare to Try, Face the Challenges, Conquer the Problems, Find Solutions (translated by authors), suggesting that people can conquer their difficulties by working together with eHelpers. Their logo (Fig. 1-d) included several elements: a grey door suggesting negative emotion, hands at the bottom suggesting support and encouragement, two people who open the door for the users (representing eHelpers), and an earth image inside the open door suggesting new life. The poster (Fig. 2-d) was yellow and displayed four letter Hs, representing the words “helper,” “happy,” “health,” and “hope.” The video was an animation with a character explaining the four Hs; a QR code was presented at the end.

The following were the winners of the bonus prizes: group eight won best title; group seven won best slogan; group one won best logo; group four won best poster; and group three won best video. Group eight's title was Hug Hug (translated by authors), which is a pun in Chinese. In written form, it means to raise mental health literacy with guidance. In spoken Cantonese, it means a hug. Group three's video began by mentioning common problems for students such as an overwhelming school workload. The narrator then recommended viewers to take a deep breath to relax before the video displayed an image of the app.

Participant satisfaction feedback

Results of satisfaction feedback from participants are listed in Table 4 (Table 4). Forty respondents (88.9%) reported that they would recommend other students to join the designathon. In terms of the qualitative data we got from the last question in the questionnaire, 14 participants were overall satisfied with the designathon while 18 participants mentioned suggestions for improvement. Thirteen participants mentioned that there was limited time to fully develop their marketing package. Four participants said that the designathon's organization could have improved by providing more information and modifying the package requirement. Three participants were not satisfied with the judging process.

Table 4

Participants' satisfaction feedback (n = 45).

Items	Mean	SD	Min	Max
1. Content of the designathon match the learning objectives.	3.93	0.69	2	5
2. Information and technique support are provided.	3.93	0.69	2	5
3. Content of the designathon is easy to understand and follow.	3.78	0.85	2	5
4. Duration of the designathon is appropriate.	3.58	0.92	1	5
5. Time distribution of theory and practice is appropriate.	3.78	0.79	2	5
6. Process of the contest is appropriate	3.93	0.62	3	5
7. Diverse grouping is appropriate	3.78	0.77	2	5
8. Device and equipment are well functioning during the designathon	3.64	0.91	1	5
9. Overall, the designathon is great.	3.82	0.61	2	5

Note. N = 45.

Discussion

This paper aimed to introduce crowdsourcing designathon as a feasible strategy to engage mental health intervention beneficiaries in the process of intervention dissemination. In this study, college students, the target population of the digital intervention, provided ideas and methods for mental health treatment dissemination by creating health communication packages that were assessed carefully by two judging panels (student judges and experts judges). Forty-nine participants designed eight health communication packages that aimed to increase participant uptake of a digital mental health intervention developed by the WHO. The packages created by the finalists will be modified and used as health promotion and recruitment materials for a digital mental health intervention effectiveness trial.

Crowdsourcing allows members of an intervention target audience to directly partake in creating health communication materials. This contrasts with traditional top-down methods of expert or designer-led promotional campaigns. Crowdsourcing as a strategy has additionally been used to obtain information to expand knowledge on the public's mental strengths and symptoms, as well as on public perceptions about mental health topics (Palmer et al., 2018; Krieke et al., 2016). To our knowledge, it has not been used to increase uptake of mental health interventions and this study is the first health communication designathon event for mental health promotion globally. Community engagement in digital mental health applications is a critical first-step towards service utilization, and therefore, it is essential to develop marketing strategies to increase reach and promote engagement. The current study presented an approach that utilizing ideas and design coming from the target audience to make the dissemination more culturally appropriate and more acceptable by the audience.

Satisfaction evaluation was used to evaluate the logistical quality of the designathon. A large proportion of participants provided positive feedback about the event. Although the mean for the question of “Duration of the Designathon is appropriate” is overall high (3.58), thirteen participants commented that they needed more time to prepare the communication packages. Insufficient time was the most cited concern, followed by event organization, and information clarification for participants and judges. Future health communication designathons should consider these participant experiences in event planning to improve the quality of the process and products. Designathon participants should be allowed additional time to brainstorm ideas ahead of the actual desiganthon period. Since the participants relied on their own resourcefulness to identify development resources (websites to create media), future iterations could also provide a standard set of resources and websites that could aid their design efforts.

Overall, the marketing products and participants' feedback supported that a designathon is a useful approach to engage students in the creation of marketing materials and achieve academic learning objectives related to global mental health. Although 11.4% of participants had moderate or higher depressive symptoms, all groups finished their design in time with creative ideas. We believe that it is a promising method to engage stakeholders to contribute their ideas and efforts to mental health services in the community. Incorporating a second judging panel comprised of the target audience, rather than solely relying on expert judges, was an innovation that could be followed in subsequent crowdsourcing events. Results of ICC inter-rater agreement showed an ICC of 0.84 indicating good to excellent concordance between the two panels of judges, using the designed rubric (Koo and Li, 2016; Cicchetti, 1994). Meanwhile, two panels came up with different finalists with highest scores (the groups winning the first prize and the most popular package are different), suggesting that students' choices could be different with experts. With the participation of the students judging panel, we were able to identify both teams.

While this approach is an equitable and potentially useful method, there were several limitations in the design. One is that effectiveness of the developed marketing packages has not yet been empirically evaluated, which will be conducted in the future. Secondly, efficacy and cost-efficiency evaluation of the crowdsourcing materials is needed to provide evidence to justify the appropriateness of this approach. Previous studies have established the cost-effectiveness of a crowdsourcing program for HIV, suggesting that crowdsourcing could be an effective approach with a wider reach and at a lower cost (Tang et al., 2019; Tang et al., 2016). Thirdly, the participants might not best represent the target audience of the intervention since the students were recruited from the same course and only 11.4% of participants had moderate or higher depressive symptoms. However, the current study of designing the marketing package focused on cultural and social factors that might influence university students' acceptance instead of clinical characteristics. Also, we were concerned about the feasibility of only including participants with depressive symptoms since depression might affect their cognitive, behavioral, and social functioning. We found that the collaboration of the mixed sample did not adversely affect their creation of the products. Future studies could consider balancing participants' clinical characteristics when it is applicable. Participants in the study were involved in a course and received incentives as prizes to put in their best effort in the design of the programs. While it is typical to include incentives within designathons, it would be useful for future studies to include participants who were not engaged in a course. We also acknowledged that experts from marketing could be helpful for the judging process, which was missing in the current study. Nevertheless, the steering committee was comprised of experts in the use of crowdsourcing methods to develop health programs. Last but not least, judges' feedback on the communication packages was collected using paper and pencil methods, preventing the judges from writing in detail due to limited time and obscuring the legibility of comments. Due to legibility issues, follow-up by the research team to clarify the feedback was necessary, which was time-consuming. To solve these problems, future events could apply digital text input with survey on devices (e.g., mobile phone, iPad, or laptop).

Conclusion

A crowdsourcing designathon is a feasible method to collect ideas to design communication plans to promote a digital global mental health product tailored for young adults. Moreover, it engages community participants to learn more about mental health topics and provides a communication platform for different community stakeholders. Although it is not the authors' primary goal, we did recognize it as a novel teaching approach to engage students in global and community health practice in class settings. This research provides evidence of the feasibility and acceptability of the crowdsourcing designathon to promote a scalable digital mental health program. Future studies can utilize this method to engage diverse stakeholders including members of the broader community, or people with lived experience of mental health concerns and who are also potential intervention beneficiaries. Further research is needed to examine the effectiveness and cost of the crowdsourced materials to enhance digital mental health uptake. As a feasible strategy to market digital mental health interventions, crowdsourcing might also be a possible solution to address the adherence and retention concerns in digital interventions. Future studies are needed to see if this approach can be used to improve long-term engagement.

The following is the supplementary data related to this article.

Multimedia Appendix 1

Orientation presentation.

Multimedia Appendix 2

Criteria/rubric for the designathon.

Multimedia Appendix 3

Health Communication Designathon: Guidelines for students.

Multimedia Appendix 4

Non-winning communication packages developed during the designathon.

Multimedia Appendix 5

PHQ-9.

Multimedia Appendix 6

Process evaluation: satisfaction survey.

Abbreviations

Macao Special Administrative Region of the People's Republic of China

Declaration of competing interest

The authors declare no conflicts of interest.