Model Registration: A Call to Action.

Evidence regarding the effectiveness of health technologies can be distorted via numerous mechanisms, including publication bias, methodological errors, over-interpretation of findings and fraud. In recent years, popular science writers have brought these issues to the fore, resulting in popular movements to improve the credibility of science. One particularly effective campaign has been that of the AllTrials movement, which focuses on the pre-registration and reporting of all clinical trials [1]. Many of the concerns that led to this campaign can be observed in the context of decision modelling.

In 2010, researchers called for the creation of a collaborative organisation to oversee a registry of decision models [2]. The concept was not realised. In recent years, the landscape of academic publishing and collaboration has changed dramatically. In this editorial, we restate the call for a model registry and recommend actionable steps for its introduction.

The Problem

Compared with clinical trial analysis, the potential for distortion in model-based economic evaluation—intentional or otherwise—is considerable [3]. There are at least four reasons for this. First, the analyst is not constrained to analysing those parameters elicited within a single primary research study; there are more avenues by which to influence the results via parameter selection, particularly in relation to costs [4]. Second, it behoves the analyst to identify (and perhaps even define) the perspective for the analysis; the decision about which parameters are relevant can be subjective [5]. Third, models often adopt a lifetime horizon that requires extrapolation from studies with short follow-up periods, meaning that sources of bias may be amplified to constitute a major influence on results [6]. Moreover, the selected length of the time horizon can have a major impact. Finally, models invariably require assumptions due to the complex dynamics of the real world that must be condensed, and any of these might influence the results.

The primary purpose of clinical trial registries, at least in their conception, was to address publication bias [7]. There is reason to believe that this problem may be relevant and prevalent in the context of decision models [8, 9]. Many submissions to health technology assessment (HTA) agencies are never formally published [10]. There is no easy way to identify previously conducted modelling studies or those currently underway. Evidence suggests that there is significant bias in the conduct and reporting of cost-effectiveness studies [11]. Even where studies are published, reporting standards can be poor and models often constitute a black box [12].

Model transparency is an issue that has been given much consideration, perhaps most notably by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and Society for Medical Decision Making (SMDM) Modeling Good Research Practices Task Force [13], yet there are few signs of improvement in practice. The traditional infrastructure of scholarly publishing does not provide an adequate means by which to ensure transparency. Peer reviewers usually do not have sight of the files underpinning a decision model and may be unable to identify sources of bias. Some journals, including PharmacoEconomics—Open, encourage models to be included as supplementary material. The majority of journals do not, and there are none that mandate it. Thus, most editors, peer reviewers and readers can do little more than assume good faith.

As some publishers adopt more progressive arrangements for the availability of data and analysis files, there remains inconsistency in the way journals oversee the publication of supplementary materials [14]. Online appendices are not indexed and usually reside behind paywalls. There is little scope for versioning that would allow for updates or revisions.

Reports of model-based studies often lack details of any attempts at either internal or external validation (e.g. [15-17]). Even where model developers do report on validation, it is vital that decision makers and other model users are themselves able to assess model validity [18]. Restricted access to underlying data and analysis files means that this is rarely possible. A lack of transparency in modelling—in terms of study availability and reporting standards—may contribute to a lack of validation. Transparency and validation are co-dependent and together allow users to assess the credibility of a model.

There may always be intentionally misleading models, with which regulators will have to deal. However, most models that suffer from validity issues will do so in spite of researchers’ best intentions. At present, researchers have few incentives to share full details of their models and any validation attempts. Modellers working for manufacturers and consultancies may have strong incentives to maintain secrecy around their models in order to protect intellectual property, pricing information or other business interests.

The Solution

We propose the creation of a registry and linked database of model-based economic evaluations. Despite previous calls to establish a model registry [2, 19], there are no signs that any organisation is moving to do so. This may in part be due to the lack of a clear proposition for the initiation of a registry. We see the following steps as a means of constructing a sustainable registry from the bottom up, with minimal resources.

Formation of a Task Force

A small group of volunteers will probably be needed to drive the creation of the registry and follow through with the subsequent steps outlined here. Ideally, modellers from both academia and industry would be involved. The group should start by establishing a set of policies. Minimum requirements for inclusion in the registry would need to be determined. These could correspond to the non-technical documentation items described by the ISPOR-SMDM Task Force [13]. Standards for designation and classification of models and a system of record identification and numbering must also be determined.

Creation of a Website

Each entry into the registry should have a corresponding webpage including information about the model, its versioning, its creators and any associated publications. The task force should develop a template that satisfies the standards and policies previously determined. It is vital that the registry supports linked versioning to provide transparency around the development of a model. The website should be built on the tenets of openness and collaboration. To this end, the website could be created using wiki software or elicit contributions via webforms. The content of the website itself should be provided under a Creative Commons Licence. Crucially, the website should facilitate discussion to encourage peer review.

Proactive Retrospective Registration

Once a website has been created, the priority for the registry will be to gather as many entries as possible. This will prompt a phase of learning and adjustment as a variety of model types and sources are identified. It may be necessary to revise policies accordingly. In the first instance, models should be identified by the task force through literature review. Every version of every unique decision model should be entered into the registry. Ideally, modellers would enter their own models, but other volunteers will be needed. Ultimately, the registry should be complemented by a database of supporting files. These files should include technical and non-technical documentation, citations, manuals and code. However, the development of a database will be costly. Therefore, in the initial phase, the registry should be supported by a decentralised bibliographic database. Existing services such as Figshare [20], Zenodo [21] and Open Science Framework [22] can be used to collect currently available modelling files. Springer Nature recently partnered with Figshare to provide bespoke data repositories for journals [23]. The registry should link to relevant files published online as supporting material by journals, in data repositories or on researchers’ own websites.

Stakeholder Engagement

The principal challenge for the success of the registry will be in encouraging researchers to prospectively register models and provide supporting files and information for those already published. It is, therefore, vital that the task force engages with researchers across sectors. As is the case for clinical trial registration, researchers could be incentivised to register their models by journal editors making registration a prerequisite to manuscript consideration. Registration could become a funding requirement for those conducting publicly funded research, or mandated by HTA agencies for appraisal submissions. The task force should strive to make arrangements with, and gain public support from, publishers and funders. In preparation for the second phase development outlined below, the task force should engage with relevant scholarly societies and other stakeholder organisations.

Second Phase Development

There will be costs associated with maintaining a registry and database [2], and it will be necessary to establish the financial capacity to pay for required support services. The second phase of development must focus on sustainability. There are a variety of possibilities for the long-term development of a registry. As is the case for clinical trial registration, a competitive market of registry providers could develop. Alternatively, a single organisation may seek to adopt the project or the task force may wish to seek financial support and establish a new organisation to ensure the sustainability of the registry. Creating the registry collaboratively and in the public domain, as described above, will ensure that all of these pathways remain open. The priority will be to secure the means of sustainable preservation of the registry. Each entry in the registry (i.e. each version of each model) should be associated with a digital object identifier (DOI). The creation of a centralised database may also be worthwhile. The database could be developed in order to satisfy funders’ and journals’ existing data sharing policies, such as that of PharmacoEconomics—Open, which encourages data and analysis files to be deposited in a public repository and made available to all researchers.

The Benefits

By mandating registration of models, research funders and technology assessment agencies could bring an end to publication bias. Even if models are not subsequently described in academic journals, their being recorded in the registry would provide a fuller understanding of the evidence base. The registry would also facilitate feedback and discussion, forming the basis for pre- and post-publication peer review. It could also stimulate collaborative validation efforts.

A model registry would help guarantee intellectual property. Plagiarism or use of models without appropriate attribution could easily be recognised and acted upon. A registry could become the basis for academic competition by creating incentives for researchers to provide more information about their models in order to increase their citability. The registry would facilitate citation of models, rather than journal articles based on models, allowing credit to be given more appropriately. Financial interests need not be undermined by inclusion in the registry. Indeed, licensing information could be included within the registry and used as a basis for attracting consultancy or other forms of income.

The registry could be consulted before work was commissioned, and could inform funding applications. This would help prevent duplication of efforts and waste of research resources. The registry itself could become the subject of research, as have other registry and database projects such as the Tufts Medical Center CEA Registry [24]. Evidence gaps could be recognised and the genealogy of models and broad methodological trends identified. In this way, the registry would also contribute to methodological development, as model structures move beyond traditionally recognised taxonomies [25, 26]. This could be valuable to disciplines outside of health economics.

Closing Remarks

There are reasons to be hopeful. A growing number of registries and databases have been established in our field, such as DIRUM [27], CEA Registry [24], ScHARR HUD [28] and a database of mapping studies [29]. Recent years have seen interjournal agreement [30] and co-publication of guidelines [31, 32]. Some researchers have gone to great lengths to test the validity of models, signalling appetite for a more concerted effort. The Mt Hood Challenge is a prime example of collaborative validation [33]. Moreover, preliminary research has signalled a desire for open-source models [34].

An open and transparent registry would enable fuller assessment of the validity of models, which may be used to inform policy decisions with direct implications for people’s health and well-being. A registry could also help prevent waste of research resources, which is itself an ethical concern. We call on publishers, research funders, HTA agencies and most importantly researchers themselves to move to establish a model registry.