Benefits, Challenges and Potential Strategies of Open Source Health Economic Models.

Introduction

A common criticism of the pharmaceutical industry is that it fails to achieve transparency [1, 2]. To allay concerns, the industry needs to collaborate openly with stakeholders to make medicines widely available and affordable—this will mean companies need to address some of their practices [1, 3, 4]. We suggest that an important practice to address is the presentation of health economic models.

Formal health economic models are commonly used to support decision makers when deciding whether funding particular pharmaceuticals will improve the health of a population [5, 6]. The value of a formal model lies not only in its results but also in the revelation of the assumptions about both the data and the logical relationships it embodies [6]. Presented properly, the model provides clarity on key issues.

The value of adopting the correct modelling methodology has been recognised by national and international bodies who have issued guidance for developing models; for example, the reports generated from the recent US Panel on Cost-Effectiveness in Health and Medicine [7] and the collaboration between the International Society for Pharmacoeconomics and Outcomes Research and the Society of Medical-Decision Making (ISPOR-SMDM) [8, 9]. Similarly, the European Network for Health Technology Assessment (EUnetHTA) is also working towards international level collaboration, providing a core health technology assessment (HTA) model framework and a guide to best practices [10]. Multiple guidelines can produce some inconsistencies [5, 10–14] but they have helped produce some clarity in dealing with uncertainty and validating and reporting models transparently [9]. Despite these welcome developments, individual models developed in the same therapeutic area may be significantly different from one another. Occasionally, efforts have been made to standardise them [15]. However, standardisation is not the norm and, in our experience, a model code is rarely shared.

It is not just pharmaceutical companies that have been challenged regarding the level of information shared. For example, the UK National Institute of Health and Care Excellence and the US Institute for Clinical and Economic Review are involved in debates around the extent their health economic models should be made publicly available [16, 17]. To facilitate this transparency, there is suggestion in the literature that open source modelling is desirable [8, 9]. The definition of open source can vary by discipline [18-21]. An open source health economic model is defined here as one that is available to those who wish to access it. This means that the model, its underlying code and a written report describing its aim, methods, structure and results would all be accessible [8]. This could be in a fully public space (e.g. freely downloadable on the Internet) or only readily available to those who request it with conditions attached to access.

An open source culture allows existing models to be updated to answer new research questions and decision problems and creates a transparent public arena for model validation, education and collaboration across research, industry and healthcare communities [4, 8, 9]. There are few examples of published fully open source models; Sullivan et al. is a recent example in pain therapy where the code was made available in both Excel and R [22].

The objective of this letter is to highlight the paucity of examples of open source models in the literature and to report on a short piece of research undertaken to gain insight into the appetite of researchers to provide and use open source models.

Exploratory Expert Survey

An exploratory double-blind survey of health economic stakeholders was undertaken between 25 April, 2016 and 6 May, 2016 to assess whether there is any desire for open source health economic models. To gain access to a wide variety of people with experience and an active interest in health economics, the LinkedIn platform was used to access the relevant groups: ISPOR; HTA in Europe (maintained by EUnetHTA); and Institute for Medical Technology Assessment.

Eight out of the ten survey questions were quantitative, while two were qualitative, with the opportunity for the respondents to populate open text boxes. Within the qualitative responses, a framework analysis was carried out to identify themes based on the respondent’s answers.

Despite the short time the survey was open, 46 participants responded, and 35 completed the whole survey. Respondents worked predominantly in the pharmaceutical industry (35%), academia (33%) and government (20%), and the remaining worked in other fields such as independent research, non-government organisations, community pharmacy and hospitals. Among the respondents, 33/41 who answered (81%) had experience with one or more health economic models in their careers. Among the participants, 18/41 (44%) had both requested information and received a request for information about a model that was not publicly available. There were a number of reasons that the participants needed the information; Fig. 1 shows the most frequent (24/41; 59%) was learning about the technical aspects of the model for use in a different disease or decision problem. Across all stakeholders and countries, making health economic models available in an open format was considered beneficial; 34/35 (97%) of participants said that it would be occasionally or very beneficial to have access to the code when reviewing existing models.

Fig. 1

Reasons for needing access to health economic models (41 responded, who could choose up to six answers that applied to them)

One of the survey questions focused on if respondents ever experienced challenges in accessing health economic models. Among the respondents, 26/41 (63%) had experienced challenges in accessing full details of a health economic model. Challenges included ownership and confidentiality issues, such as a lack of willingness to share and not knowing who to contact or being unable to contact someone to request information on a health economic model. Two qualitative questions were asked: what was their perceived value of improving access to health economic models and what were the best strategies to enable sharing. Figure 2 highlights the most common key themes mentioned by respondents relating to the perceived benefits.

Fig. 2

Frequently mentioned benefits of open source health economic models

The best strategies to encourage the sharing of health economic models as suggested by the respondents included:

Collaboration: different stakeholders working together to develop open platforms or libraries to encourage sharing and to maximise the social value of their research; examples were journals, HTA agencies or consultants actively involved in publishing and sharing their health economic models and promoting a sense of sharing and value of open source.

Transparency: such as authors providing supplementary materials and highlighting relevant sources where information was gathered.

Confidentiality: finding ways to share models while keeping the necessary proprietary aspects confidential.

Changes to regulatory processes: potentially integrated open access processes across organisations and formal request processes by health authorities.

Consistent use of language: making models easier to understand across stakeholders and countries.

Discussion

The limited results of this exploratory analysis supported viewpoints found within the available guidelines and limited literature. Across all stakeholders and countries in the survey, making health economic models available in an open format is expected to be beneficial within industry, academia, government, non-government organisations or hospitals. This is consistent with arguments made by Fazio et al. that transparency with models are desired by various stakeholders [23]. The ISPOR-SMDM guidelines (specifically the ISPOR guidelines on model transparency and validation [8]) and the EUnetHTA joint action projects support the views of respondents on collaboration, transparency, confidentiality, processes and consistency, while the literature explores how best to improve this [4, 8–10]. It is also consistent with a proposal by Afzali et al. that there should be “reference models” that are built by an expert group for a particular therapy area and then made available to pharmaceutical companies for drug submissions [24]. However, we caution that any reference model approach would need to allow stakeholders the ability to debate, challenge and change the model as appropriate.

There were inconsistent qualitative responses from some participants who wanted tighter confidentiality yet more open access. This could be partially owing to the often high-profile and emotional nature of healthcare allocation decisions. This means policy makers and other stakeholders may not want to share all information in fear of it complicating the debate or hindering current or future decisions, as well as an organisation’s specific needs for not wanting to be fully transparent. The ISPOR-SMDM guidelines address some of these issues. They outline that there is potentially a conflict between the scientific desirability of making all methodological and technical details of a model available to peer reviewers and to other researchers and the need to protect intellectual property generated by substantial investments in the development of a model [8]. Rejecting the latter would significantly reduce the incentive to create models, and intellectual property rights cannot be ignored. One possible solution is to make a full technical documentation available within whatever agreements right holders feel are necessary to grant them adequate protection [8]. This should allow for detailed review of any model by other researchers, provided they accept the confidentiality restrictions [8]. We suggest that this is often the most practical solution in the short term. Despite challenges, there are already examples of open source models that are fully available in the public space [22].

Conclusion

The study reported here was small and lacks generalisability. However, it did demonstrate that among the respondents there is an appetite for greater use of open source models and the topic is deserving of further exploration. Future investigation could be particularly beneficial if it also focuses on the obstacles to transparency and how they may be overcome.