Joint international consensus statement on crowdsourcing challenge contests in health and medicine: results of a modified Delphi process.

OBJECTIVES: To develop a consensus statement to provide advice on designing, implementing and evaluating crowdsourcing challenge contests in public health and medical contexts.
DESIGN: Modified Delphi using three rounds of survey questionnaires and one consensus workshop.
SETTING: Uganda for face-to-face consensus activities, global for online survey questionnaires. PARTICIPANTS: A multidisciplinary expert panel was convened at a consensus-development conference in Uganda and included 21 researchers with experience leading challenge contests, five public health sector workers, and nine Ugandan end users. An online survey was sent to 140 corresponding authors of previously published articles that had used crowdsourcing methods.
RESULTS: A subgroup of expert panel members developed the initial statement and survey. We received responses from 120 (85.7%) survey participants, which were presented at an in-person workshop of all 21 panel members. Panelists discussed each of the sections, revised the statement, and participated in a second round of the survey questionnaire. Based on this second survey round, we held detailed discussions of each subsection with workshop participants and further revised the consensus statement. We then conducted the third round of the questionnaire among the 21 expert panelists and used the results to finalize the statement. This iterative process resulted in 23 final statement items, all with greater than 80% consensus. Statement items are organised into the seven stages of a challenge contest, including the following: considering the appropriateness, organising a community steering committee, promoting the contest, assessing contributions, recognising contributors, sharing ideas and evaluating the contest (COPARSE).
CONCLUSIONS: There is high agreement among crowdsourcing experts and stakeholders on the design and implementation of crowdsourcing challenge contests. The COPARSE consensus statement can be used to organise crowdsourcing challenge contests, improve the rigour and reproducibility of crowdsourcing research and enable large-scale collaboration. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Recruitment for the first round of survey questionnaires included 140 lead authors of crowdsourcing manuscripts.

Two additional rounds of surveys were completed by a multidisciplinary expert panel to obtain from a broad range of stakeholders with expertise in leading crowdsourcing challenge contests.

A combination of in-person and digital methods were used for discussion and voting on consensus items.

The study is limited by an absence of in-depth interviews to develop the initial statement items and potential recency bias.

Introduction

COVID-19 continues to test governments and healthcare providers around the world. In response, crowdsourced projects have created new forms of personal protective equipment,1 developed participatory citizen-science apps for contact tracing,2 and organised mutual aid organisations.3–5 Crowdsourcing is the process of having a group, including experts and non-experts, solve a problem and then share solutions with the public.6 Crowdsourcing has been used to inform WHO policy,7 develop machine learning algorithms,8 and identify innovative health services.9

Crowdsourcing is increasingly being used to find innovative, stakeholder-engaged solutions to challenging medical and public health problems.10–12 The effectiveness of crowdsourced health solutions has been demonstrated in randomised clinical trials,10 and social science research has also demonstrated the power of crowdsourcing to increase the engagement of stakeholders in health problem-solving, resulting in solutions that are more effective at addressing local contexts and community concerns.13 14 For example, a crowdsourcing approach has been previously been used to engage communities disproportionately impacted by the HIV epidemic to help develop creative, culturally-appropriate messaging on HIV cure research.15 Additionally, crowdsourcing approaches have been found to be highly effective for engaging marginalised lay populations in HIV cure research,16 and for engaging youth in developing effective youth-friendly HIV self-testing promotion strategies.17

An array of crowdsourcing approaches have been used by health and scientific research organisations, including the National Academies of Sciences, Engineering and Medicine,18 the National Institutes of Health Research Office of Behavioral and Social Science Research,19 and The Lancet Healthy Cities Commission.20 While there are many ways to implement crowdsourcing approaches, one common approach is in the form of challenge contests (also called open calls, innovation challenges, prize inducement contests).6 Challenge contests typically involve a call for community solutions in response to a specified problem; contributions are evaluated to identify exceptional ideas, with finalists being awarded prizes and their ideas disseminated for implementation.12 The goals of challenge contests can vary; for instance, a systematic review of challenge contests found that process-oriented contests focused on mass engagement, while outcome-oriented contests focused on producing high-quality outputs.21

Despite the growing interest in and value of crowdsourcing challenge contests, there are few resources available to inform the design, implementation and evaluation of crowdsourcing contests related to health and medicine. The Special Programme for Research and Training in Tropical Disease (TDR) ‘Practical Guide on Crowdsourcing in Health and Health Research’12 is the single most comprehensive guidance. However, this guide did not include a systematic review of the evidence in makings its recommendations, nor did it include a consensus statement. Robust consensus guidance may help to mitigate potential risks of inconsistent application of crowdsourcing methodology, as well as help to establish greater trust in the use of challenge contests as an innovative approach to health research among both health researchers and public stakeholders.22 Consensus guidance can also help to enhance the rigour and reproducibility of crowdsourcing challenge contests. Given that the heterogeneous nature of crowdsourcing stifles comparisons,6 consensus guidance may be important for encouraging health researchers to implement this approach across a wider array of challenging problems in need of stakeholder-driven solutions. In order to provide rigorous guidance to assist healthcare professionals, policymakers and citizen scientists in applying a crowdsourcing approach, a multidisciplinary group of international experts reviewed evidence on crowdsourcing in health and medicine to develop a consensus statement. The goal of this paper is to describe our consensus development process and present the final statement as a tool for informing crowdsourcing approaches to health and medical research.

Methods

Study design

We followed recommendations on guideline development from the Guideline International Network.23 Development of the final consensus statement proceeded through a modified Delphi process,24 which proceeded through four stages: (1) convening of an expert panel to initiate the consensus development process; (2) initial statement development and first round of survey questionnaires; (3) an in-person workshop with expert panel members and a second round of survey questionnaires; (4) a third round of survey questionnaires and finalisation of the consensus statement items. These stages are described in detail below.

Patient involvement

Patients were not involved in the design, conduct, reporting or dissemination plans of our research.

Expert panel recruitment

To initiate the consensus development process, we convened a panel of experts at a consensus development conference in Uganda. The conference was convened by PA (conference director), jointly with the following partner organisations: TDR Social Innovation in Health Initiative (SIHI), Makerere University, University of Malawi, University of Philippines Manila, Universidad Icesi, Centro Internacional de Entrenamiento e Investigaciones Medicas, Pan American Health Organization, Social Entrepreneurship to Spur Health and the Bertha Center for Social Innovation and Entrepreneurship at the University of Cape Town. The meeting received organisational support from Fondation Merieux. The conference director and cochair (JT) and partner organisations appointed a multidisciplinary expert panel of internationally recognised academics representing several scientific disciplines, including internal medicine, public health, health policy, social innovation, social entrepreneurship, psychology and primary care. The panel included individuals from each of the five TDR SIHI hubs. These individuals all have experience leading health-related challenge contests in local, regional and global settings. In addition, we invited five individuals from the Ugandan public sector (innovations, consumer organisation, clinical health services, education, science and technology, public health) and nine Ugandan potential users.

The members of the expert panel had voting rights for the entirety of the consensus development process. They were selected because of expertise about crowdsourcing challenge contests and relevant publications. Each of the SIHI hubs used their own criteria to decide who should join the expert panel. Criteria included leadership in organising challenge contests on health, participation in TDR SIHI activities related to open calls and open innovation, and participation in other committees related to crowdsourcing. One independent, non-voting member (EK) with previous experience in Delphi methodology administered questionnaires for the modified Delphi process.

Modified Delphi consensus development

Building on the evidence from our group’s previous systematic review of crowdsourcing in health and medicine,10 existing reviews of crowdsourcing in health,21 25–31 and previous guidelines,12 a subgroup of expert panel members developed an initial statement and set of recommendations. An online survey questionnaire was developed using the initial statement following our review of existing guidelines (online supplemental material 1). Second, we sent a link to the survey (hosted on Sojump, an online survey platform) to 140 corresponding authors of previously published articles that had used crowdsourcing methods. The survey included sections on considering, organising, promoting, assessing, recognising, sharing and evaluating crowdsourcing challenge contests. Throughout the survey, participants were presented with a series of statements pertaining to specific elements in the ideal design and implementation of each stage of a crowdsourcing challenge contest, including what processes, goals and considerations should be part of a crowdsourcing activity. The participants were asked for their level of agreement with each statement (strongly agree, agree, neutral, disagree and strongly disagree) and given the choice to amend or make comments. Note that at this stage of consensus development, the term crowdsourcing rather than the more specific format of crowdsourcing challenge contests was used throughout the questionnaire, in order to capture participants’ views on crowdsourcing as a method more broadly (thus potentially encompassing other forms of crowdsourcing, such as hackathons). Participants were assured that their responses were confidential and only used for the purposes of the consensus statement development. Electronic informed consent in lieu of written consent was obtained from all online survey participants. Third, we presented the survey results to the expert panel at a workshop. This in-person workshop, sponsored by TDR SIHI, was held in Uganda from 8 to 11 October 2019.32 Participants of the workshop included all 21 members of the expert panel. As proceedings of the workshop are a matter of public record, the names and affiliations of panel members are available from the workshop report,32 and are summarised in table 1. Participants discussed each of the sections, revised the statements, and finished the second round of the survey questionnaire. At this stage of consensus development, survey participants were asked to consider their responses in relation to the specific crowdsourcing approach of challenge contests. Written informed consent for survey participation was obtained from all 21 workshop participants. Fourth, we held detailed discussions of each subsection with workshop participants (online supplemental material 2) and revised the consensus statement accordingly. Then we conducted the third round of the questionnaire among the 21 expert panellists. Finally, we summarised the results of the final questionnaire.

Table 1

Members of the expert panel/consensus workshop32

Name	Expertise	Position	Affiliation	Country
Phyllis Awor	Maternal and child health, HIV/AIDS	SIHI Uganda lead, research fellow	Makere University	Uganda
Donny Ndazima	Respiratory infections	Strategy and partnerships manager	Makere University	Uganda
Maxencia Nabiryo	Maternal and child health	Project officer	Makere University	Uganda
Emmanuela Oppong	Biomedical engineering	TDR intern	Makere University	Uganda
Noel Juban	Clinical epidemiology, antibiotics	SIHI Philippines lead, professor	University of Philippines Manila	Philippines
Jana Deborah Mier	Municipal health	Project manager	University of Philippines Manila	Philippines
Jean Francis Barcena	Media, communications	SIHI communications officer	University of Philippines Manila	Philippines
Arturo Ongkeko	Mobile health, health information management	SIHI network facilitator	University of Philippines Manila	Philippines
Don Mathanga	Infectious disease epidemiology	SIHI Malawi lead, associate professor	University of Malawi	Malawi
Barwani Msiska	Reproductive health, adolescent health	Project manager	University of Malawi	Malawi
Ruth Mputeni	Media, communications	Communications coordinator	University of Malawi	Malawi
Diana Castro-Arroyave	Diseases of poverty, HIV, and hepatitis B	SIHI Latin and Central America lead	CIDEIM	Colombia
Maria Isabel Echavarria	Capacity building, implementation	IR training and M&E coordinator	CIDEIM	Colombia
Joseph Tucker	Crowdsourcing, infectious diseases	SESH, SIHI China lead, associate professor	SESH, LSHTM, UNC	China, UK, USA
Weiming Tang	HIV, STDs, crowdsourcing	SESH manager	SESH	China
Shufang Wei	Social media, website development	Communications director	SESH	China
Huanyu Bao	Challenge contest implementation	Implementation officer	SESH	China
Eneyi Kpokiri	Clinical pharmacy, challenge contests	Research fellow	LSHTM	UK
Tiarney Ritchwood	Family medicine, community health	Assistant professor	Duke University	USA
Katusha de Villiers	Health delivery, financial management	Manager	Bertha Center, University of Cape Town	South Africa
Uche Amazigo	Parasitology, onchocerciasis, sustainability of social innovations	Technical advisor, fellow of the Nigerian Academy of Science	Pan-African Community Initiative on Education and Health	Nigeria

LSHTM, London School of Hygiene and Tropical Medicine; SESH, Social Entrepreneurship to Spur Health; SIHI, Social Innovation in Health Initiative; STD, sexually transmitted disease; TDR, Special Programme for Research and Training in Tropical Disease; UNC, University of North Carolina.

Consensus statement definitions

A supermajority consensus rule was pre-specified. Specifically, all statements that had agreement rates of 80% or higher were included in the final consensus statement. Individual statement items were iteratively revised to maximise agreement across the three rounds of questionnaires. The degree of consensus for each statement was graded as follows: grade U was classified as unanimous (100%) agreement; grade A was 90%–99% agreement; and grade B was 80%–89% agreement.

Results

The first round of survey questionnaires (online survey) received 120 responses (response rate 85.7%). The first questionnaire included 35 items, including four items on sociodemographic characteristics. All 21 members of the expert panel participated in the second round survey questionnaire at the in-person workshop. After the workshop, the third round of survey questionnaire was completed by 19 of the 21 expert panel members (response rate 90%). In total, over the three rounds of questionnaires and in-person workshop, first round survey participants and the expert panel eliminated 12 items that were redundant or unnecessary. The iterative process resulted in 23 final consensus statement items, all with greater than 80% consensus. This final consensus statement on crowdsourcing challenge contests is presented in table 2. Note that for the sake of brevity, the general term crowdsourcing’ is used throughout the consensus statement rather than the more specific phrase ‘crowdsourcing challenge contests’. Table 2 also indicates the grade received for each statement item. Seven of the final 23 items achieved unanimous agreement; 15 achieved grade A agreement and 1 (item 6a) achieved grade B agreement.

Table 2

Considering the appropriateness, organising a community steering committee, promoting the contest, assessing contributions, recognising contributors, sharing ideas and evaluating the contest: final consensus statement on crowdsourcing challenge contests in health and health research, and the consensus grade achieved by each item

Item		Grade
Stage 1. Considering crowdsourcing in health and health research
1a	Before starting a project, the organisers should consider the benefits and risks of crowdsourcing in order to understand if this is an appropriate method.	U
1b	Crowdsourcing may be particularly useful in settings in which there are diverse networks (eg, groups, professional societies, social media movements, in-person teams) to solicit contributions.	A
1c	Crowdsourcing organisers should consider whether they are asking for something that would be feasible for an individual layperson to develop.	U
1d	Crowdsourcing organisers should ensure that they have selected an appropriate activity, based on feedback from community members and other stakeholders.	A
Stage 2. Organising crowdsourcing activities
2a	Before starting a project, the organisers should establish a steering committee to develop the call for entries, decide the format of submissions, and provide details.	A
2b	The steering committee should include people from different disciplines, including the following: (1) people who are living with disease, community leaders, civil society leaders, or other community stakeholders. (2) Key opinion leaders and network leaders who can help to distribute the contest. (3) If focused on local implementation, a member of the government or public sector. (4) If focused on research, a leader of research studies. (5) In some cases, funders as non-voting observers. (6) In some cases, private sector leaders as non-voting observers	A
2c	The steering committee should work together to promote the crowdsourcing activity, finalise the judging the process, develop a finalist recognition plan, finalise the prize structure, and develop a sharing plan.	A
Stage 3. Promoting crowdsourcing activities
3a	A crowdsourcing activity should build trust in the activity in a way that is appropriate to the local context (eg, in-person activities).	A
3b	A crowdsourcing activity should be promoted through social media platforms with an acknowledgement of the limitations of social media (ie, limitations on who will view and respond to social media calls).	A
3c	A crowdsourcing activity should be inclusive and allow contributions from diverse individuals.	U
3d	A crowdsourcing activity should be promoted with groups and networks of interest identified by the steering committee. Accommodation for participation of people with disability should be considered based on the purpose of the crowdsourcing activity.	A
3e	A crowdsourcing activity should have a clear deadline. If needed, the steering committee can extend the deadline, but this should be updated in a clear way and allow for revision for those who already submitted.	U
Stage 4. Assessing crowdsourced contributions
4a	The judges should provide feedback on contributions independent of each other.	A
4b	Criteria for selecting judges are similar to the criteria for selecting steering committee members (see above), with the additional requirement of having sufficient time to undertake judging.	A
4c	The contest organisers should first assess eligibility and then provide eligible contributions to judges for them to evaluate.	A
4d	Judges should recuse themselves from evaluating entries where there is a potential conflict of interest.	A
Stage 5. Recognising crowdsourcing activities
5a	Steering committee will make the final selection of finalists and respective prizes based on the prespecified criteria.	A
5b	Personalised announcement first: after deciding the final selection but before making a public announcement, all participants should be contacted about the decision regarding their submission.	A
5c	Crowdsourcing organisers should clearly explain how finalists were selected.	U
Stage 6. Sharing contributions from crowdsourcing activities
6a	Providing open access resources, images and templates related to the outputs from a crowdsourcing activity is important.	B
6b	When possible and after permission has been obtained from participants, seek permission from finalists to use their ideas and distribute them widely.	A
Stage 7. Evaluating crowdsourcing through research
7a	Research on crowdsourcing is important to demonstrate the value of crowdsourcing in health and health research.	U
7b	A crowdsourcing activity can be evaluated by using qualitative, quantitative, or mixed methods research.	U

The 23 final items are organised by the seven stages of implementing a crowdsourcing challenge contest, which are summarised in figure 1.12 Here, we briefly describe the seven stages. Detailed descriptions are available (online supplemental material 3).

Figure 1

The seven stages of a challenge contest, adapted from WHO/Special Programme for Research and Training in Tropical Disease guidelines.12

Consensus statement

Considering the appropriateness of challenge contests

Considering whether a challenge contest is an appropriate method for solving a problem is an important first step. Challenge contests are prize challenges where a call is issued by contest organisers, and solutions and ideas are then solicited from the public. Other forms of crowdsourcing include hackathons and online collaboration systems. Challenge contests, hackathons, and online collaboration systems differ from one another in terms of the amount of time, resources, and processes required to implement each. Researchers considering whether to use crowdsourcing should consider which method would be the most effective and feasible based on the local setting. Table 3 shows unique aspects of each of these methods.

Table 3

Challenge contests, hackathons and online collaboration systems

Method	Definition	Important differences
Contests	Open prize challenges where a call is issued to the public and then contributions are solicited and evaluated	In-person or online Medium term (months) Prizes awarded
Hackathons	Events where a diverse group of individuals are brought together to advance a common goal	Typically in-person Short-term (weeks) Prizes awarded
Online collaboration systems	Online platforms that allow individuals to exchange and share contributions and ideas	Online Permanent Prizes not typically awarded

Organising a community steering committee

If a challenge contest is deemed to be the most suitable crowdsourcing method for the local context, a steering committee should be organised. The steering committee often includes local community members, health professionals, community-based organisation leaders and private sector leaders. Importantly, efforts should be made to recruit committee members from diverse fields to provide an array of different perspectives. Including individuals with direct, personal experience with the problem, such as patients or at-risk groups, on the steering committee is essential. The steering committee plays an important leadership role throughout the contest, including deciding the structure and purpose of the contest, outlining the rules and requirements for entries, developing a call for contributions, and establishing the prize structure.

Promoting the challenge contest

Many people are unfamiliar with challenge contests and will need a clear description of the purpose, expectations and rules. Although there are many private companies that organise challenge contests, a simple website may be sufficient to communicate with potential challenge contest participants. The website should contain all information related to the challenge contest, including an overview of objectives, timeline, guidelines for contributions, criteria for judging, prizes and frequently asked questions. Infographics and short videos can help make the challenge contest more accessible to a general audience, allowing participation to move beyond expert audiences to engage the public. Challenge contests should include in-person events when possible. One study found that participants were twice as likely to learn about challenge contests through in-person events compared with social media.33

Assessing contributions

The steering committee will need to consider how contributions will be received and judged. For some topics, receiving contributions exclusively online may be the best approach. For example, an online receiving platform would be the most efficient choice for a challenge contest seeking video contributions, as videos can be readily uploaded and submitted from any location with internet access. Text-based contributions can also be easily collected using an online submission form. However, it is important to note that limiting the receiving platform to online contributions may exclude some participants. In-person events in partnership with local organisations can provide alternative ways to receive offline contributions. Once all contributions are collected, a panel of judges will evaluate them to determine finalists. The judging panel often consists of a mix of experts, laypersons and members of the contest organising committee. Judges who have a potential conflict of interest should recuse themselves from reviewing contributions.

The quality of crowdsourcing contributions can vary broadly, resulting in both low and high-quality contributions.34 It is thus recommended to conduct an initial eligibility screening of all contributions in order to remove invalid, incomplete or duplicate entries before forwarding to the judges. When there is a smaller number of contributions, panel judging can then occur after this initial screening. If a contest receives a large number of contributions (eg, more than 10 contributions per judge), the judging process can be conducted in three phases: eligibility screening, crowd judging and panel judging. In phase one, two independent judges examine contributions based on prespecified criteria. Invalid, incomplete or duplicate contributions are deleted and do not advance to the next judging round. In phase two (crowd judging), a group of laypersons evaluates the eligible contributions using an evaluation rubric. Each contribution is reviewed by three independent judges. Only those contributions that are deemed exceptional (eg, mean score of 7/10 or greater) will then proceed to the final round of panel judging. A panel of experts and non-experts individually evaluates each contribution forwarded from the previous round of crowd judging. Once the evaluations have been received, the steering committee reviews all evaluations to rank order the scores and identify the finalists. Judges should be thanked for their assistance and notified when an announcement of the finalists will be made.

Recognising contributors

Recognition in the context of a challenge contest can be difficult given that organisers bring together experts, non-experts and many other individuals who have different training and expectations. However, this is an important component for sustaining challenge contests. The first stage in recognising individuals is to establish clear expectations for all those who contribute. Among judges and steering committee members, this typically involves mentioning that their contributions are done on a voluntary basis when they are invited. The amount of time required of judges and steering committee members varies but is usually less than 4 hours total. Among contributors, the amount of time required to create a submission should be commensurate with the prize structure. In instances where there is uncertainty, asking individuals who submit entries to estimate the total time spent creating the submission may be helpful. Finalists may be recognised through public announcements using various platforms, including organisational websites, social media platforms, online public fora and in-person events. While more attention is given to finalists, it is important to acknowledge the efforts of all people who submitted entries. There are many ways to do this, including sending emails notifying contributors of the outcome or thanking them for their efforts on an open platform. Written feedback on contributions may be shared with selected contributors. Public announcements should be timely, occurring shortly after the conclusion of a contest. Terms such as ‘winner’ and ‘loser’ are often avoided to acknowledge the hard work of all contest contributors and encourage future participation.

Sharing and implementing ideas

One of the most important stages of community-engaged research projects is the process of sharing results35 beyond scientific audiences.36 The main aim of the dissemination stage is to share ideas generated through the contest and to implement selected ideas where appropriate. Challenge contest dissemination depends on the context and the audience. There are several reasons to widely share selected contest contributions. First, since crowdsourcing involves soliciting outputs from a group, sharing allows organisers to give back to the group who made the project possible. Second, crowdsourcing projects are often supported by public funds, enrol local participants, and are sanctioned by local public authorities. Despite the strong rationale for sharing, there are also many factors that may limit wide sharing. Contest participants may be appropriately concerned that sharing their contribution could pose risks, such as inadvertent disclosure of private information (eg, sexual orientation). In terms of research, scientists may be concerned with disseminating materials that interfere with blinding in randomised controlled trials (RCTs). The risks of sharing contributions need to be carefully considered and addressed during contest planning.

Evaluating challenge contests

Crowdsourcing challenge contests can be evaluated in many ways, including quantitative and qualitative approaches. Quantitative studies can examine the crowdsourcing activity itself and may include the number and quality of contributions, the number of website views and related social media metrics.37–39 Such evaluations can be used to determine the overall reach and level of participation in a challenge contest, which can further indicate the interest of stakeholders in addressing the health problem targeted by the contest. Observational studies can provide useful information about challenge contests such as the acceptability of the challenge contest for relevant stakeholders, the impact on related participant behaviours and motivations for participation. This can provide insights on the extent to which a crowdsourcing activity was successful at engaging a diverse array of stakeholders in ways that are meaningful to them. This may be particularly important for studies seeking to engage populations whose perspectives are often excluded from the production of health research knowledge—for example, individuals with low levels of education14 and youth.17 RCTs are used to assess the effectiveness of crowdsourcing interventions compared with interventions developed using other methods. For example, one study evaluated an online, peer support intervention and found that crowdsourced social interactions enhanced user engagement and decreased rates of depression compared with online expressive writing.40 While considered to be the gold standard evaluation method, RCTs are often time and resource intensive. Qualitative evaluation can synthesise themes identified in the text of contributions or provide more context on implementation.

Discussion

We developed a consensus statement, considering the appropriateness, organising a community steering committee, promoting the contest, assessing contributions, recognising contributors, sharing ideas and evaluating the contest (COPARSE), on crowdsourcing challenge contests in health and health research. COPARSE brought together data from a systematic review and meta-analysis, existing global guidelines on crowdsourcing, and structured feedback from experts and end-users using a modified Delphi methodology. The consensus statement provides a harmonised framework to enhance crowdsourcing challenge policy, implementation and research. First, the statement provides a structure for policymakers to organise open calls for suggestions about health policy. Second, both the shorter consensus statement and the longer implementation considerations provide a range of practical suggestions for implementers using challenge contests in the field. Finally, researchers may find COPARSE useful in developing studies to evaluate the process or outcomes of crowdsourcing studies.

COPARSE represents a novel approach to solidifying expert advice on the design, implementation and evaluation of challenge contests. Based on the results of our literature review, published literature on crowdsourcing is limited to methodological descriptions. Although there have been published reviews on crowdsourcing,10 11 41 the heterogeneity of recommendations has precluded consensus development. COPARSE extends the scope of TDR’s Practical Guide on Crowdsourcing in Health and Health Research12 by developing a consensus statement with extensive feedback from clinical and public health experts, as well as implementers with experience using crowdsourcing. Our consensus statement helps to refine the practice of crowdsourcing in health and medical research based on a convergence of expert and non-expert review. Establishing consensus on crowdsourcing challenge contest procedures through the input of experts with direct crowdsourcing experience may help to further legitimise and encourage the use of this approach in solving complex health and medical problems.

COPARSE expands the literature by including diverse voices from around the world, using a modified Delphi method, and focusing on challenge contests. However, there are some methodological limitations to this study that should be considered. First, we did not use in-depth interviews at the start of the process to inform survey development. However, the survey was informed by a thorough review of the existing literature. Second, the in-person workshop was also conducted over 2 days, which may introduce recency bias among workshop participants whose responses were used to develop COPARSE. Third, while the workshop was attended by a diverse range of participants with highly relevant expertise, participation was ultimately limited to 21 individuals. An iterative procedure over a longer time horizon, and with a larger group of participants, may allow for a greater diversity of opinions and recommendations for inclusion in future iterations of COPARSE, which we envision as a statement that can be revisited, updated and further refined over time and in response to innovations in the growing field of crowdsourcing for health research.

Conclusion

Challenge contests are simple, inclusive, and inexpensive ways to solicit community feedback on health and medical problems. COPARSE should not be used as a rigid guidebook, but rather as a set of core principles to inspire further challenge contests. Only through iterative implementation will the science and practice of crowdsourcing for health and medicine improve.