Health hackathons: theatre or substance? A survey assessment of outcomes from healthcare-focused hackathons in three countries.

BACKGROUND: Healthcare-focused hackathons are 48-hour platforms intended to accelerate novel medical technology. However, debate exists about how much they contribute to medical technology innovation. The Consortium for Affordable Medical Technologies (CAMTech) has developed a three-pronged model to maximise their effectiveness. To gauge the success of this model, we examined follow-up outcomes.
METHODS: Outcomes of 12 hackathons from 2012 to 2015 in India, Uganda and the USA were measured using emailed surveys. To minimise response bias, non-responding teams were coded as having made no progress.
RESULTS: 331 individuals provided information on 196 of 356 projects (55.1% response rate), with no difference in responses from teams participating in different countries (Cramer's V=0.09, p=0.17). 30.3% of projects had made progress after a mean of 12.2 months. 88 (24.7%) teams had initiated pilot testing, with 42 (11.8%) piloting with care providers and 24 (6.7%) with patients. Overall, 97 teams (8.1 per hackathon) drafted business plans, 22 (1.8 per hackathon) had filed patents on their innovations and 15 (1.3 per hackathon) had formed new companies. Teams raised US$64.08 million in funding (average US$5.34 million per hackathon; median award size of $1800). In addition, 108 teams (30.3%) reported at least one member working on additional technologies with people they met at a hackathon. Individual confidence to address medical technology challenges was significantly increased after attending (t(1282)=192.77, p 0.001).
CONCLUSION: CAMTech healthcare hackathons lead to consistent output with respect to medical technology innovation, including clinical trials, business plan development, securing investment capital/funding and new company formation.

Introduction

Low-income and middle-income countries (LMICs) bear nearly 90% of the global burden of disease1 2 and yet have severe shortages of trained healthcare providers. Innovative medical technologies are included in one of six essential building blocks of the WHO Health System Framework3 4 and can have particular impact in LMICs. Health technology developments can be transformative.5 They can offset limitations of healthcare workforces by optimising the efficiency and effectiveness of providers. Yet, medical technology innovation largely occurs in high-income countries (HICs) and an estimated 40–70% of resulting technologies fail when implemented in LMICs.6–8 Thus, there may be significant benefit in joining efforts by innovators from LMICs and HICs to ensure relevance, feasibility and acceptability of new technologies. This could help maximise relative ‘value’ (defined as health outcomes over cost).9

The term ‘hackathon’ combines ‘hack’—a solution reached through intense innovation—and ‘marathon’—an event of defined length and concentrated effort. Healthcare hackathons champion the process of ‘co-creation’, in which serendipitous meetings of people across geographies and disciplines such as healthcare, design, engineering and business enable diverse teams to develop potential solutions in a time-limited format. Hackathons are often incorrectly described as ‘crowdsourcing’ occurrences. First appearing in 2006, ‘crowdsourcing’ is a portmanteau of ‘crowd’ and ‘outsourcing’ to indicate a process of obtaining ideas or services from a large—usually online—community.10 In contrast, healthcare hackathons are 48-hour events in which a group of curated individuals from different backgrounds come together to drive innovation in healthcare.11 Hence, we suggest that hackathons instead represent ‘enriched crowdsourcing’.

The Massachusetts Institute of Technology's (MIT) Hacking Medicine, a frequent partner of the Consortium for Affordable Medical Technologies (CAMTech), first pioneered health hackathons in 2011. Software hackathons have been conducted for decades, and few existing empirical reports indicate that up to 36% of teams formed at hackathon-style events continued to work together after 3 months.12 However, these reports do not identify additional markers of progress. In addition, unlike medical technologies, software development has fewer constraints and participants are often themselves the intended end users. In addition, there is active debate in the press and academic literature about the utility of healthcare hackathons. On the one hand, it has been argued that healthcare hackathons recruit and empower a new mix of innovators to work collectively towards novel solutions not possible within traditional academic or commercial environments.13 However, some suggest that although the hype of these events generates enthusiasm, healthcare hackathons fall short of developing lasting innovations because healthcare challenges are too complex to be addressed in a 48-hour event.14 15 Others state that hackathons actually may deter relevant innovation by creating an ‘artificial vacuum’ in which solutions are proposed.13 Ultimately, the question is whether hackathons are merely theatre—a show without tangible outcomes—or do they represent events with substantive results?

The goal of the CAMTech is to accelerate sustainable and effective medical technology innovation and enhance entrepreneurial capacity to improve health in resource-constrained settings globally. To realise the full potential of healthcare hackathons, CAMTech has developed a model in which (1) preceding priming activities focus on subsequent work, (2) 48-hour hackathons catalyse innovations and (3) a suite of posthackathon offerings stimulates the progress of innovations towards commercialisation. We evaluated outputs of events in India, Uganda and the USA to determine the impact of this hackathon model. Primary outcomes included prototypes pilot tested, funds raised, business plans created, patents filed and companies formed.

Methods

Hackathons

From October 2012 to October 2015, CAMTech hosted 12 hackathons. Five were held in India, 3 in Uganda and 4 in the USA (3 of which were organised in collaboration with MIT's Hacking Medicine). Hackathons were advertised through academic, industry and non-governmental organisation email lists, flyers, as well as website and social media postings. All were focused on health in LMICs and other low-resource settings. Several hackathons had specific themes including reproductive, maternal, newborn and child health (RMNCH); Ebola care; diabetes; and trauma/road safety. Each hackathon was preceded by activities to maximise participant readiness including identifying healthcare challenges by engaging hackathon participants with multisector stakeholders online and at clinical summits; visiting venues coping with healthcare challenges; and engaging mentors from clinical, business, technology and design disciplines to participate (table 1).

Table 1

CAMTech healthcare hackathon model

Component	Timeline
Prehackathon
1. Advertise hackathon and solicit mentors with domain expertise Event website, social media, flyers, emails and word of mouth	1–3 months in advance
2. Curate both applicants and mentors Patients (when possible) and clinicians Engineers and designers Business personnel	1 month in advance
3. Some problems (ie, pain points) posted on an event website Clinical/public health challenges solicited from experts and partner organisations	1 month in advance
4. Clinical site visits Participants tour partner healthcare facilities and hold discussions with providers	1–2 days in advance
5. Clinical summits Clinical, public health and policy experts discuss challenges	1 day in advance
6. Social mixer Participants socialise to initiate cross-disciplinary relationships	Evening before a hackathon
Hackathon	48 hours
Day 1 Introductory speeches highlighting challenges and incentive prizes; introduction to ‘hacking’ One-minute problem verbalisations (pitching pain points) Team formation and registration Hacking and mentoring	AMPM
Day 2 Hacking and mentoring Practise concept pitches Three-minute team concept pitches ‘After the hack’ opportunities and suggestions Hackathon awards	AMPM
Posthackathon
1. Innovation mentorship Tailored individual project mentorship Co-creation laboratory space for prototyping and project maturation Connection with targeted organisations/individuals having stage-specific skills needed by teams Networking, mentorship and dissemination of opportunities and information through resources such as the online CAMTech Innovation Platform	Anytime
Monetary awards to the most promising innovations Grant writing mentorship and workshops Dissemination/notification of additional funding sources	30–90 days posthackathon and beyond
3. Entrepreneurial expertise Facilitation of teams through existing Business Accelerators* Entrepreneurial group learning through 2-day Entrepreneur Bootcamps (these consist of in-person, intensive workshops with guest experts designed to enhance participant entrepreneurial skill in the medical technology space) Facilitation and funding through the CAMTech Business Accelerator Program	Variable

*Business Accelerator: a business programme that includes mentorship, educational components and networking that aims to grow a business rapidly. The CAMTech Accelerator Program includes the resources available on the CAMTech Innovation Platform in addition to more in-depth team coaching at frequent intervals.

CAMTech, Consortium for Affordable Medical Technologies.

Individuals apply to participate with a statement of their experience and motivations for attending. These applicants are curated by organisers to permit a pool of participants with a broad mix of skills who describe compelling motivations. The 48-hour hackathons focus on framing health challenges, verbalising specific problems (pitching pain points), team formation, ideation and prototyping through hacking, and the competitive presentation of resulting concepts to judges. In the months following hackathons, CAMTech offered support to teams by facilitating clinical, technological and entrepreneurial mentorship; providing laboratory workspace and connections to production resources; enabling seed grants for promising projects; and building capacity for external fundraising (including grant writing) and engagement of investors—all with the aim of accelerating promising innovations towards commercialisation. Although prehackathon activities were in place, several posthackathon activities have become formally operationalised and/or have come ‘online’ recently. For example, the online CAMTech Innovation Platform came online on 12 June 2015.

To determine the impact of the CAMTech healthcare hackathon model, we conducted two surveys. The first was conducted in September to October 2014 to measure outcomes of the first six hackathons. The second survey was conducted in December 2015 to January 2016 to measure outcomes from all 12 hackathons.

Recruitment and participants

At least one team member's email address was collected at each hackathon during team registration. A survey request consisting of an introductory email explaining the purpose and providing a link to the online survey was sent to registered email addresses for each team and to additional team members for which an email address was recorded. Two reminder emails were sent after each survey.

To be eligible, respondents had to be at least 18 years old. Since all hackathons occurred in Uganda, India or the USA, where English is the primary language, surveys were administered in English. An incentive for survey completion was offered as entry into a raffle for either a US$100 Amazon gift card or prepaid mobile phone minutes. Each survey had separate drawings for residents of India, Uganda and HICs. If respondents opted to enter these raffles, they were directed to a separate site to enter their personal information in order to maintain confidentiality.

The survey

The 15–20 min survey included three sections: (1) evaluation of hackathon experiences; (2) information on specific hackathon projects and (3) demographic information. Responses were confidential and reported anonymously by project code. Depending on response-driven skip patterns, 41–53 responses were possible. A respondent could provide data for more than one hackathon project if s/he was on more than one team (at the same or different hackathons). However, responses measuring a personal hackathon experience were collected only once from the initial survey response using a survey function that skipped to project-specific questions if the participant reported having responded to a survey for another project. Survey questions relating to primary outcomes are presented in online supplementary table S1.

Survey Questions pertaining to primary outcomes.

Data analysis

The duration of follow-up was calculated using an elapsed-time calculator from the date of each hackathon to the time-stamped response from the last team member to reply. Project-specific codes prevented double counting of project-level data. Answers for project-specific codes were counted as positive if progress afterwards, business plans, trials, patents filed, team members obtaining a job/position, or companies formed were reported by at least one respondent. If discrepant answers were received from the two surveys, answers from the later survey were used. If, within the same survey, discrepant quantitative answers were received from multiple respondents for the same project, the averages of responses were used. Such items included number of people per team, total monetary investments/awards and agreement with the statement: ‘I would have made similar progress on my project WITHOUT the hackathon’ (with answers scaled 1 ‘wouldn't have made any’ to 5 ‘would have progressed just as far’). Currencies were converted to US$ on 10 February 2016 using a standard currency converter (http://www.oanda.com/currency.converter).

To correct for possible response bias, teams for which no response was received were considered to have made no progress after the hackathon. Data were analysed via SPSS V.23 (Chicago, Illinois, USA). Statistical tests included Cramer's V test, one-way analysis of variance and paired samples t-test. Statistical significance was set at p<0.05.

This study protocol was reviewed by the Partners Healthcare Institutional Review Board and determined to be ‘exempt’.

Results

Of the 363 teams participating in the 12 CAMTech health hackathons, 356 (98.1%) had at least one functioning email address (ie, for which no ‘undeliverable’ message was received), and 196 project teams (55.1%) provided at least one survey response. From 1367 email addresses to which the survey was sent, 331 responses (24.2%) were received. There was no statistically significant difference in response rates of teams from hackathons held in India, Uganda or the USA (Cramer's V=0.09; p=0.17). Of the 331 responses, 48 were from individuals who worked on more than one CAMTech hackathon project. Hence, there were 283 responses to individual experience queries.

The average project team size was 4.8 people (median 5), and over 60% of teams had three or more professional disciplines represented. Respondents were predominantly young (18–25 years of age) and slightly over three-quarters were male (table 2).

Table 2

CAMTech hackathon project teams

Host country	Projects surveyed	Responses (%)
India (N=5*)	192	99 (51.6)
Uganda (N=3)	96	53 (55.2)
USA (N=4)	68	44 (64.7)
Total (N=12)	356	196 (55.1)
Hackathon projects	Mean	Range
Projects surveyed/hackathon	29.7	11–63
Project team size (people)	4.8	1–12
Disciplines/project**	Per cent
1	20.0
2	18.5
≥3	61.0
Respondents	N	Per cent
Gender
Male	256	77
Female	75	23
Age
18–25	207	62.5
26–33	75	22.7
34–40	22	6.6
41–55	19	5.7
55–65	6	1.8
≥65	2	0.6

*N=number of hackathons.

**Disciplines identified in survey: (1) medicine; (2) engineering; (3) business; (4) design; (5) life sciences and (6) other.

CAMTech, Consortium for Affordable Medical Technologies.

Public health and healthcare challenges identified in prehackathon activities and addressed by hackathon teams were diverse and ranged from infectious to non-communicable/chronic diseases to RMNCH (figure 1).

Figure 1

Health topics addressed.

Teams conceived of predominantly preventive, diagnostic or therapeutic medical devices. In total, 61.2% of projects included hardware in their innovations, with 39.8% having both hardware and software. A significant minority of projects (19.9%) included ‘process’ innovations or non-technical solutions. A similar number of projects, almost 18.9%, included software only.

Over 30% of teams reported continued work over a mean of 12.2 months after hackathons, and another 10% planned to engage in future work on their projects (table 3). One-quarter (N=88, 24.7%) of projects surveyed had begun piloting including in preclinical settings (N=52, 14.6%), on healthcare workers (N=42, 11.8%) and/or involving patients (N=24; 6.7%). Notably, 22 patents were filed for projects emerging from the 12 hackathons, and 15 new companies were formed.

Table 3

Posthackathon team-level responses

	N	Per cent	Mean/hackathon
Continued work beyond a hackathon
Yes	108	30.3	9.0
Not yet but plan to	36	10.1	3.0
Total	144	40.4	12.0
Prototype progress after a hackathon
Teams reporting any progress	126	35.4	10.5
Teams reporting major progress	70	19.7	5.8
Projects initiating pilot work
Any	88	24.7	7.3
Preclinical studies	52	14.6	4.3
Clinical studies with providers or healthcare workers only	42	11.8	3.5
Clinical studies with patients	24	6.7	2.0
Teams initiating a business plan	97	27.2	8.1
Patents filed	22	6.2	1.8
Companies formed	15	4.2	1.25
Financial support raised	Number of awards	Number of teams obtaining any award
Awards	58	31
	Total raised (millions of US$)	Mean per hackathon (millions of US$)	Median award size (US$)
Amount in US$	64.082	5.34	1800
Perceived progress without a hackathon (1=would have made no progress; 5=would have made as much if not more progress)	Scale average	SD
All teams	1.92	0.90
Teams raising funds	1.87	0.79
Teams forming companies	2.01	0.90

Teams surveyed: 356; mean follow-up period: 12.2 months (range 1.9–40.0).

One team that founded a company raised $62.5 million for its project in 40 months following a hackathon. A member from this team reported, “The hackathon catalyzed our work…I don't think the company would have come together had we not participated in the hackathon” and also that “we now have more than 200 team members, offices and pharmacies around the country”. In total, $64.08 million was raised in funding through investment capital, awards or grants with a mean of $5.34 million per hackathon and a median of $1800 per team that raised funds. The distribution of award sizes is shown in figure 2. Even removing the largest company as an outlier, 30 teams raised funding for their continued work for a mean of $131 803 raised per hackathon.

Figure 2

Distribution of funds raised.

When asked about agreement with the statement, ‘I would have made similar progress on my project WITHOUT the hackathon?’ with 1=‘I wouldn't have made any progress’ to 5=‘I would have progressed just as far if not further’, the average response of team members was 1.92. The average response of teams that had received monetary awards was 1.87 and was 2.01 among teams that formed new companies with no significant differences between these three values (p=0.65).

In addition to markers of team progress, several indicators of a strengthened innovation ecosystem emerged. Of responding teams, 45 (3.75 per hackathon) reported that at least one member had obtained a job or position due to connections made at the hackathon. One hundred and eight teams (30.3%) reported at least one member working on additional technologies with people they had met at a hackathon. Finally, responses from participants showed significantly increased confidence to address medical technology challenges in healthcare after attending a hackathon (t (282)=13.88, p<0.001; table 4).

Table 4

Hackathon attitudes and impact (N=283)

	N	Per cent	Mean N/Hackathon
Would attend again
Yes	247	87.3
No	5	1.8
Unsure	31	11.0
	Mean score (SD)
Would recommend to a friend (1=not at all likely to 5=extremely likely)	4.7 (0.62)
	Mean score before (SD)	Mean score after (SD)	p Value
Confidence to address medical technology challenges (1=not at all confident to 5=very confident)	3.25 (1.21)	4.25 (0.84)	<0.001
	Mean score (SD)
Attending hackathon increased confidence to begin a new venture (1=not at all confident to 5=very confident)	4.22 (0.94)

Discussion

Despite debate on the impact and utility of healthcare hackathons, we are not aware of previously published quantitative evaluations of their outcomes. We found that 30% of teams continued work after hackathons and yielded 1.8 new patents, 1.25 new companies and $5.34 million of follow-on funding per hackathon with a mean follow-up of 12 months. Remarkably, 25% of all projects initiated at the hackathons had begun preclinical or clinical pilot testing. We believe these are key steps towards accelerating technical innovation for LMICs. In settings with limited resources or where financial expenditures are under increasing scrutiny, this model of innovation offers favourable returns.

Medical technology development requires a specific focus on users who may be distinct from innovators and on unique considerations such as evidence of clinical effectiveness, regulatory approval and frequent ‘third-party payors’, for example, government health systems or insurances that pay on behalf of a beneficiary. A network or ‘ecosystem’ that includes skill sets necessary to commercialise high-value health products will be important to support impactful innovations. We documented several indicators of a strengthened health-focused innovation ecosystem emerging from the hackathons. Individual participants reported significantly increased confidence in their ability to address medical technology challenges in the future. Furthermore, many of these individuals would not have encountered each other without the hackathon structure. Notably, over 30% of teams reported having at least one team member who subsequently worked on other projects with people they had met at a hackathon. For example, one member on a team from the first hackathon responded that, as a mechanical engineer, he would never have met a paediatrician from sub-Saharan Africa and ultimately formed a new company together. This strengthened ecosystem helps create a structure that is better capable of solving healthcare challenges beyond the scope of hackathons.

Our study has several strengths. We had a robust (55.1%) response rate for an email-solicited internet-based survey. This response rate itself suggests a high level of continuing engagement of hackathon participants. In addition, we dealt with potential response bias very conservatively by including non-responding teams as ‘non-progressors’.

Our study also has several weaknesses. With self-reported surveys, it is possible that respondents over-reported markers of progress due to social desirability bias. However, the anonymous survey should minimise this bias. Another weakness is that the time frame of follow-up may have been short considering typical timelines of medical device development. The range of follow-up from our surveys was 1.9–40.0 months, whereas typical medical device development can range up to 10 years depending on clinical and regulatory requirements.16 In addition, projects involving drug, vaccine or entirely novel diagnostics were not pursued during these hackathons as they are considered less amenable to this format and have even longer development timelines. Finally, while it is impossible to determine how many innovations would have arisen and progressed without this hackathon model, participants reported a low likelihood that they would have proceeded without the hackathon.

Nevertheless, markers of progress that we identified are typical for traditional product development pathways—that is, fundraising, prototyping, patenting, trialling, company formation and licensing. In fact, with significant progress seen in <40 months—on average within 12 months—even greater progress towards commercialisation would be anticipated with longer periods of follow-up. For example, two projects that had made progress after the first hackathon have progressed to forming companies and raising significant capital (US$62.5 million and US$730 000), and one of these has already commercialised. Two other projects have been licensed to commercialisation partners. The CAMTech health hackathon model thus appears to have utility in compressing the innovation cycle.

Sastry and Penn13 argue that anticipating outcomes from a hackathon itself is futile. It does indeed seem that structures need to be in place to facilitate progress of good ideas beyond the typical 48 hours of hackathons. CAMTech increasingly has focused its consortium towards both priming before and nurturing nascent innovations after hackathons. The preceding activities are designed to maximise the ability of participants to focus their hackathon projects to address real-world challenges; these are followed by support to stimulate progress after hackathons. For example, CAMTech developed an innovation award programme for early stage ideas, an online community through the CAMTech Innovation Platform (http://camtechmgh.org), entrepreneur development opportunities through in-person and virtual accelerators such as the CAMTech Accelerator Program (CAP), and physical workspaces such as at the CAMTech Co-Creation Laboratory in Uganda. In response to participants' stated needs, CAMTech initiated ‘Entrepreneur Bootcamps’ to increase innovators' entrepreneurial skills and has offered grant writing workshops in India and Uganda. It may be argued that these supportive initiatives helped create the remarkable progress of the healthcare hackathons on which we report. In response to the increased numbers and stated needs of innovators, the postevent suite of offerings has been expanded over the follow-up period. It is likely that these will continue to expand over time. To aid with dissemination of this model, CAMTech has developed a standard operating procedure guide and a lessons-learnt ‘living’ document to better enable organisations to develop their own healthcare hackathons.

These ‘enriched crowdsourcing’ events help to ensure an optimal mix of experience directed towards focused challenges. Whether similar results can be seen through replication remains to be seen. However, these 12 hackathons indicate a remarkable source of med-tech innovations.

This study highlights several areas for future research to determine the impact of healthcare hackathons. Investigation into specific characteristics predictive of the greatest progress is needed. In addition, investigation into which postevent offerings are principal contributors to success would be helpful as the model matures. Further research is also warranted to determine the number and, ultimately, the value of commercialised projects over a longer time frame.

Conclusion

Healthcare is in need of value-based solutions to improve outcomes and curtail costs, particularly in LMICs. There has been debate whether healthcare hackathons represent merely an enthusiastic hype of creative participants or substantive sources of healthcare innovations. Data presented indicate that, within three different global locations, a healthcare hackathon model including preceding priming activities and targeted postevent support is a reliable source of solutions to healthcare challenges. To a great extent, these events help develop a healthcare solution ecosystem primed to solve as yet unaddressed challenges.