Bridging the gap between evidence and policy for infectious diseases: How models can aid public health decision-making.

The dominant approach to decision-making in public health policy for infectious diseases relies heavily on expert opinion, which often applies empirical evidence to policy questions in a manner that is neither systematic nor transparent. Although systematic reviews are frequently commissioned to inform specific components of policy (such as efficacy), the same process is rarely applied to the full decision-making process. Mathematical models provide a mechanism through which empirical evidence can be methodically and transparently integrated to address such questions. However, such models are often considered difficult to interpret. In addition, models provide estimates that need to be iteratively re-evaluated as new data or considerations arise. Using the case study of a novel diagnostic for tuberculosis, a framework for improved collaboration between public health decision-makers and mathematical modellers that could lead to more transparent and evidence-driven policy decisions for infectious diseases in the future is proposed. The framework proposes that policymakers should establish long-term collaborations with modellers to address key questions, and that modellers should strive to provide clear explanations of the uncertainty of model structure and outputs. Doing so will improve the applicability of models and clarify their limitations when used to inform real-world public health policy decisions.

1. Introduction

Public health policy decisions must balance a range of scientific, budgetary, social, and political considerations. Ideally, each of these elements should be considered in a transparent fashion before reaching a decision or implementing a specific policy. While socio-political considerations will always be somewhat subjective, scientific evidence can – in theory – be used to evaluate the potential epidemiological or economic impact of alternative decisions. For example, in the setting of a high-profile outbreak, the probability of making political gains or alleviating public fears is not objectively quantifiable (despite their importance to the decision-making process), but scientific outcomes, such as potential trajectories of the outbreak under different policy decisions, can be estimated quantitatively with appropriate tools using the best available data as inputs, such as the known incubation period.

In the realm of infectious diseases, the tools for integrating and translating scientific data into policy-relevant outcomes are often classified in the domain of ‘mathematical models’,[1,2] which are defined here as quantitative frameworks for the analysis of dependent happenings (events where the number affected at one time depends on the number already affected[3]). For example, systems of diagnosis and treatment are represented in mathematical terms such as the rate of movement from an infectious to a treated state. These models have the ability to translate existing scientific evidence into projected outcomes at the population level for both endemic diseases like tuberculosis (TB) and epidemic situations such as the Ebola virus disease (EVD) outbreak in West Africa in 2014–2015,[4,5] in a way that is transparent and verifiable or refutable by external observers. These estimates can also help with clinical decision-making at the individual level, to improve patient outcomes.

Unfortunately for most public health decisions regarding the control of infectious diseases, such models are seldom constructed – and when they are, they often have limited impact upon the decision-making process. This is likely due to several factors, including perceptions that models are too complex to understand or too dependent on assumptions, coupled with a history of insufficient communication between public health practitioners with specific policy questions and modellers with the quantitative tools to address them.

Here, the potential role of mathematical modelling in decision-making for health policy in the realm of infectious diseases is explored, and key reasons why mathematical models have historically not fulfilled this potential are evaluated. To do this, the current status of modelling in public health decision-making is first outlined and a case study modelling question described. Details of how to construct a relevant model and how to link it to policy are then given, and some of the potential limitations and challenges of using modelling described. Finally, a framework by which improved collaborations between public health stakeholders and modellers may broadly benefit public health is proposed.

2. Current role and potential opportunities for modelling in public health decision-making

The use of structured frameworks for applying evidence to public health decision-making is well established.[6] For example, the World Health Organization (WHO) advocates the use of the GRADE process,[7] which is a framework that connects a public health question to an evidence-based analysis and recommendation.[8] The United States Preventive Services Task Force (USPSTF) similarly uses decision-making algorithms to assess the level and quality of evidence to support the introduction of specific interventions.[9] However, these frameworks for using scientific evidence to support policy decisions often lack quantitative assessments of how different decisions will impact health at a population level.

This is especially true in the realm of infectious diseases, where dynamics of transmission may cause great disparity between the individual-level benefit or harm of an intervention (for example, side effects of a vaccine for a rare disease such as polio that may outweigh an individual’s risk of contracting the disease) and its population-level impact (for example, maintaining elimination of polio through herd immunity). As a result, in settings where population-level benefits are unproven, interventions with strong scientific evidence for individual effectiveness may be recommended over those with a potentially dramatic impact for populations. This decision-making process, if uninformed by insight at the population or system level (as provided by models), may perversely result in outcomes that are good for certain people, but bad for the population as a whole.

Models can address this knowledge gap by estimating the effects of interventions when the collection of population-level empirical evidence (e.g., from cluster-randomized trials) is infeasible, unethical, or untimely. For example, mathematical models suggested that universal voluntary HIV testing and immediate antiretroviral therapy (ART) might dramatically reduce future HIV transmission,[10-12] even though the individual-level effectiveness of ART at higher CD4+ T-cell counts is small,[13] and reduced transmission at the population level is difficult to prove empirically. By projecting population-level effects of potential interventions, the models informed not only key policy decisions but also the design of future clinical trials.[14]

Despite the potential impact that model outputs can have on public policy decisions, the use of models by public health decision-makers has traditionally been limited.[2] Many public health and policy decisions must be reached rapidly, in too short a time for new models to be developed, parameterized, and calibrated. Modellers must therefore achieve a balance between anticipating future policy questions (in which case models may ultimately not speak to the specific policy question at hand) and responding to existing questions (in which case models may be constructed too late to inform policy decisions). In addition, as mentioned above, complex models that are poorly presented are unlikely to be used by time-pressured policymakers. Furthermore, it remains unclear in most settings how to weigh evidence from models against other epidemiological and clinical data. As described below, all models must make certain assumptions and manage attendant uncertainty. These aspects of models are often not well-understood by public health stakeholders, and as a result, model outputs may be seen as difficult to interpret and untrustworthy. A framework by which modellers and decision-makers can work together to more appropriately incorporate evidence from infectious disease models into public health decisions, without over- or underemphasizing the importance of those models, is proposed here.

3. Modelling infectious diseases for policy: the example of a rapid TB diagnostic

To demonstrate the utility and process by which mathematical models can inform infectious disease policy, the case study of a new molecular diagnostic test for TB is used: the Xpert MTB/RIF test (Xpert).[15] Xpert provides a comparatively rapid, point-of-treatment diagnosis in under two hours, if placed in settings where individuals present for initial TB diagnosis and/or follow-up evaluation. Xpert is also substantially more sensitive than the most widely used diagnostic test for TB worldwide (sputum smear microscopy). However, at over 10 times the cost of sputum smear microscopy (which costs less than $2 fully-loaded per test, compared to about $20 for Xpert), scale-up of Xpert has the potential to dramatically increase the cost of TB control in high-burden settings.

The key policy-related questions around the use of Xpert are the following: Do the clear individual-level benefits of improved diagnosis translate into population-level effects on transmission, and if so, would scale-up of Xpert have sufficient impact to justify the added cost (i.e., would Xpert be cost-effective)? These questions can be, and have been, addressed effectively using mathematical modelling.

In the case of Xpert, an initial modelling study projected the impact on TB-associated morbidity and mortality in six countries of southern Africa.[16-18] This study adopted a regional approach, which allowed the authors to use a single model framework (due to similar epidemics across the six countries) and existing data (which are reported on the national level). A global model would likely have required more model complexity, whereas a sub-national model might have been limited by available data or generalizability. The authors aimed primarily to publish their results in the scientific literature, although the model has subsequently been used in country-level discussions and extended to other regions. The model predicted a relatively large potential population level impact of Xpert on TB transmission and mortality, based on the assumption that increased rapid diagnosis would increase treatment rates. In the absence of pre-existing data, this model had to make a number of reasonable simplifying assumptions, including the proportion of individuals with TB who would ultimately be diagnosed by the existing algorithm in the absence of Xpert and the speed at which that diagnosis might happen. These results – which reflected the best available data and assumptions at the time – were used to support policy recommendations to scale up Xpert in the region and worldwide.

Subsequent clinical trials revealed that Xpert did not identify many more patients than were already being started on treatment, due to unexpectedly high levels of empiric treatment practices in TB-endemic settings.[19,20] Using this new information, the model was then revised to account for empiric treatment practices, and new estimates predicted a much smaller impact of Xpert.[19] Several of these models were extended into cost-effectiveness analyses of Xpert scale-up.[18,21]

This case study illustrates the ability of models to iteratively incorporate updated data, leading to better estimates and highlighting existing weaknesses in both model assumptions and available data over time. A pertinent translation of individual-level effect to population impact is also displayed. Unfortunately, the case study also demonstrates the challenges in linking model results to policy. Major obstacles exist to the implementation of Xpert, especially in trying to use Xpert as a true point-of-care test.[22] Nevertheless, despite its impact on population health being initially relatively uncertain, Xpert has received strong support from policymaking bodies such as the WHO,[23] based primarily on systematic reviews of sensitivity and specificity. This recommendation places pressure on many high-burden countries to scale Xpert up,[24] and at tremendous expense. Even in light of emerging data and updated model projections that suggest Xpert may not improve population-level outcomes,[19] recommendations to implement Xpert have become increasingly strong, partially due to political momentum and the known individual-level benefits of Xpert.

The policy–modelling disconnect in recommendations for Xpert contrasts, for example, with that for systematic screening for TB in high-risk populations. Systematic screening for TB has been shown in multiple mathematical models to have a potentially dramatic impact at the population level,[25-28] but the benefits of systematic screening at the individual level are difficult to prove.[29] The recommendation for systematic screening is therefore much less enthusiastic.[30] As a result of this discrepancy between individual-level evidence (which is easier to collect directly but arguably less important to public health) and population-level evidence (which is difficult to collect directly and thus often requires models but is critical to public health decision-making), many countries are pressured to implement Xpert rather than systematic screening. This pressure exists despite the evidence from models that systematic screening might have much greater impact on reducing TB at the population level – and potentially at a more favourable cost–effectiveness ratio.[26]

In summary, as shown by the case study of Xpert, modelling interventions is often the only way to evaluate the comparative effectiveness (and cost-effectiveness) of interventions at the population level in the short term. In doing so, models may not only help to prioritize those interventions that might have greatest impact at the population level, but may also identify the data elements needed to better inform estimates of such impact for different public health policies.[21] This process – which ideally occurs iteratively as new data emerge – can lead to better alignment between research efforts and policy priorities. However, major obstacles exist to the implementation of this process, and current practice continues to prioritize infectious disease interventions with more benefit for individuals than for populations.

4. Building useful models

Developing a useful model includes identifying, in sequence (1) a useful question and its epidemiological context, (2) a framework through which that question could be addressed, (3) the parameters required to address the specified question using that framework, and (4) the empirical evidence available to inform meaningful values of those parameters (for examples and further details see publications by Vynnycky and White[31] and Keeling and Rohani[32]). Once a question, framework, parameters, and empirical evidence have been identified, the model can then be used to inform decision-making by projecting the potential outcomes associated with different policy decisions. For example, in the Xpert case study, the question of interest was how big would the impact of this new diagnostic test be? The epidemiological setting was six countries with high TB incidence.[18] The framework utilized was a transmission model that used both natural history and TB control parameters, such as treatment success, informed by country-level TB programme data.

For models to be useful to decision-makers, they must be both relevant and methodologically sound. In general, useful models are built to answer a key question that should guide the structure and complexity of the model (rather than the model determining the question). One such question might be to evaluate the expected epidemiological and economic impact of different strategies for scaling up Xpert for TB diagnosis (e.g., centralized or in individual clinics), or the required bed capacity during the recent EVD outbreak.[33] Defining a central question also helps to inform the structure of the model, which should incorporate relevant scientific data (e.g., transmission rates, existing levels of infection control and treatment). The epidemiological setting is also important; for example, a model of implementing Xpert should include not only the sensitivity and specificity of the test but also the diagnostic processes, underlying disease prevalence, and clinical algorithms in the chosen setting. A model of TB diagnosis in the USA would need to account for immigration, for instance, whereas a model of TB in Sub-Saharan Africa would require a more detailed description of ART scale-up.[18] In some cases, the same model structure can be modified to explore ‘first-pass’ results across a range of settings and interventions;[34] in other cases (or when more precisely calibrated results are needed), separate models will be required for each setting.

In general, models allow for an exploration of the system and give a holistic picture of the realm of possible outcomes. As such, uncertainty and sensitivity analyses around the main components of the model are critical. Uncertainty analysis translates uncertainty in model ‘inputs’, such as the proportion of patients accessing different diagnostic services, into uncertainty in model ‘outputs’, such as the impact of a new diagnostic on mortality. Sensitivity analyses aim to attribute portions of this uncertainty to specific parameters. In a one-way sensitivity analysis, for example, a key parameter (such as the TB transmission rate) might be set sequentially to its highest or lowest plausible values, and the model results assessed at each of those points. In the case of Xpert, population-level impact has been shown to be very sensitive to existing levels of empirical therapy for TB.[16,17] Broader consideration of model uncertainty would explore the impact of a range of plausible empirical treatment levels as well as other model parameters (for example, transmission rates) or indeed do a more comprehensive sampling over all parameters.[35]

Ultimately, the estimates of any model can only be as accurate as their supporting data, but models can also describe that uncertainty to decision-makers, allowing them to make the most appropriate decisions given existing, imperfect evidence. As such, modelling results with wide confidence intervals that reflect this uncertainty are often valuable to policymakers, as they demonstrate the current state of knowledge.[36] The alternative, where false confidence in predictions is gained via modelling based on strong unsupported assumptions, must be avoided despite the temptation of the ‘clarity’ of results that such assumptions can provide.

5. Linking model results to public health policy around infectious diseases

Models can inform infectious disease public health policy in at least three ways (see Figure 1). Firstly, models can systematically use epidemiological data to better understand the larger systems in which policy decisions must be made. For example, mathematical models of Xpert scale-up in Africa have suggested that baseline diagnostic patterns affect the incremental benefit of a novel, more sensitive test,[17] thereby suggesting that policymakers should target Xpert roll-out to areas with the weakest existing diagnostic systems.

Figure 1

Roles and challenges of infectious disease models for public health decision-making.

Secondly, as described above in the case of universal HIV testing and treatment, models can apply a transparent framework to compare the potential population-level impact of interventions in situations where collecting empirical evidence might be logistically, monetarily, or ethically infeasible.[37,38] Even when broader empirical studies are feasible, interim policies must nonetheless be set; models can help these policies make maximum use of existing evidence before definitive results are known.

For example, modelling estimates for the recent EVD outbreak in West Africa published in September 2014 predicted that without interventions, Liberia and Sierra Leone would have approximately 550 000 reported Ebola cases by January 2015.[47] The predictions over a shorter timeframe were closer to what actually occurred; for estimates of case numbers by September 2014, the model overestimated the number of cases by only 8.8% in Liberia, and underestimated the number of cases by 7.6% in Sierra Leone. The later projections to January 2015 were substantially less accurate. These results, however, demonstrated at the time what most needed to be done to control the outbreak. Thus this is an example of where modelling results may appear ‘wrong’ but can still be useful. Furthermore, the results made under the assumption of no intervention highlight the impact that outbreak control interventions had on the magnitude of the outbreak – and these could later be compared to an alternative model structure that incorporates those interventions to further evaluate the impact of those measures. By highlighting how serious things could be if nothing was done, models emphasized the need for control interventions and the aspects of those interventions that might be most important from an epidemic control perspective.

Thirdly, modelling can point to data gaps that, if filled, could better assist decision-making and control of infectious diseases in the future. For example, TB models might find that the comparative impact of Xpert scale-up strategies depends strongly on the amount of ongoing TB transmission in a community, which in many places may not be known.[6] These results could motivate further data-gathering activities (e.g., molecular epidemiological characterization of acommunity,[39,40] or synthesis of existing programmatic data on TB incidence[41]) that could help to improve decision-making related to Xpert scale-up in the future. For the spread of rubella, modelling has already led to the collection of new missing data.[42]

In each of these cases, mathematical models provide public health decision-makers with key pieces of knowledge that can inform evidence-based decision-making. Furthermore, unlike expert opinion (which often holds sway purely on the basis of reputation or existing dogma), models accomplish this task in a way that is quantitative and open to questioning (or modification) by others in the field. When a model’s structure, methods, assumptions, and parameters are laid out in a reproducible manner with direct communication and guidance to those with less methodological expertise, their results should be transparent and accessible rather than being perceived as a ‘black box’ that is susceptible to manipulation and is too difficult to understand.

6. The challenges of using models and approaches to addressing them

These potential benefits of infectious disease models can be offset by excess complexity.[37] While in theory, models are fully transparent, many models are made so complex that few outside the modelling community can fully understand their mechanics. In addition, while models should be tailored to answer a specific policy question of interest, they are often presented in such a fashion that does not speak readily to key policy decisions.[37] Modellers must strive to produce transparent outputs that can directly inform the policymaking process, even when this requires some simplification to be made.

An important way to improve transparency is to publish the raw data as well as any modelling code.[43] Whether provided in the appendix of the publication or in online format, this would provide readers with access to the model and the ability to closely review its methods, thus making a more informed determination as to whether the assumptions and data used were sufficient. Open data and code would also allow for the model to be improved and developed iteratively by others in the modelling community to answer other policy-relevant questions.

In addition, care must be taken when broadly applying a model specifically structured to answer a certain question. For example, a modelling evaluation of Xpert scale-up across southern Africa[18] is unlikely to provide useful guidance on where to place Xpert machines in the USA,[44] or whether to supplant other funds to pay for Xpert testing.[45,46]

However, models can be, and often are, reconstructed as new evidence or considerations come to light. The development begins with models that provide initial insight; such initial models are gradually replaced by more complete models that incorporate updated information. This process of iteratively evaluating modelling output provides a framework in which to place new evidence and improves our understanding of both the problem and the utility of the modelling tool being applied. While the need for such iteration can be seen as a problem with the initial model or input data, it is more appropriately seen as reflecting the natural course of scientific inquiry, in which better data and better tools to utilize those data are continually being developed.

Presenting the uncertainty around modelling results, as described above, presents a further challenge as it often reflects limited available data on key parameters. In such cases, modellers should honestly portray this uncertainty rather than providing results that appear more reliable, and policymakers should make decisions based on this uncertainty rather than requesting results that are more precise. Not including such uncertainty may be justified in specific cases[47] – such as when the infectious disease situation being modelled (e.g., when considering just a ‘no intervention’ scenario) is unrealistic and specific policy decisions are not based on this quantitative value – but is generally not recommended.

There are also significant ethical considerations that must be taken into account when using the results of mathematical models to design public health interventions. These include considering the values and preferences of the target population as well as the available resources.[48] Determinants of health behaviour must be viewed in a social context, including the cultural beliefs, historical patterns, and choices available to the local population. If applied in a vacuum, modeling results may not be relevant to the target population and the estimated impact of a chosen intervention will likely not be realized.

7. Recommendations for a future framework

How can we improve the utility of modelling for infectious disease relevant to public health decision-making in the future? One important component of any such strategy should involve ongoing collaboration and interaction between infectious disease modellers and public health stakeholders. Such communication and collaboration enables decision-makers to appropriately understand the complexity of model structure and uncertainty in modelling results, and modellers to inform additional refinements or data-gathering efforts to reduce that complexity and uncertainty. In addition, both modellers and public health policymakers must view the results of models within the social context of the target population to consider the ethical impact of applying modelling results to specific settings. Fostering such collaboration will improve the availability of models to public health practitioners, as well as the quality of model structure and relevance of results. It will also improve the likelihood that the perspectives and needs of all critical stakeholders are included.

To promote such public health stakeholder collaboration and optimize the role mathematical modelling can have in public health decision-making in the field of infectious diseases, an iterative process for model development is recommended (Figure 2). The key stages in this process are: (1) Policymakers engage modellers early in the decision-making process and inform them of the key public health questions that are being considered in a given population and/or clinical setting. (2) Modellers use this information to construct models that are most likely to effectively address these questions. (3) Modellers strive to reduce, or fully justify, the complexity of their models and explain the context and uncertainty of their outputs to decision-makers and others who could openly interrogate their methods. (4) Decision-makers seek to incorporate model results into their decision-making process (including decisions for more data gathering) and inform modellers of where model structures, outputs, and uncertainty could be refined for future decision-making.

Figure 2

Proposed framework for interaction between infectious disease modellers and public health practice. Stakeholders in public health practice should identify the key relevant public health questions for modellers to address and the existing data that can be used to inform model structure and parameters. Once shared and communicated with modellers, further discussion should pertain to the relevant model structures best suited to address the question and the required inputs. Subsequently, through ongoing communication, evidence gaps can be identified and modelling outputs can be reviewed and understood in the context of model uncertainty and generalizability. Once the model is refined and finalized, the outputs can be used to support data-informed decision-making. In this way, long-term collaboration between public health practitioners and mathematical modellers can ensure that models have optimal impact on evidence-based public health decision-making.

To succeed in this endeavour, it is important that modellers and decision-makers build mutual trust over time; these steps are difficult to complete in a ‘one-off’ fashion for each new question that arises. Platforms for such interaction, such as conferences and workshops, should be developed and promoted to stimulate discussion and interaction around the important questions and how to address them, including how to share key data. As effective communication is only possible when modellers and stakeholders speak the same ‘language’, it is critical that these communities work together to establish long-term collaborative relationships.

As an example of such longer-term collaboration between infectious disease modellers and infectious disease policymakers, the TB public health community has established platforms where engaged stakeholders can interact. For example, the TB Modelling and Analysis Consortium (TB MAC) brings together quantitative researchers, policymakers, TB programmes, and donors to identify TB control questions that require modelling input.[49] Discussion and interaction between TB control stakeholders is also supported by the TB Modelling group at Johns Hopkins,[50] which holds frequent international conference calls to examine the latest research and areas of interest. Such multidisciplinary collaboration is also being seen in other infectious disease fields, where modelling studies have been used to assist public health interventions and policy-making decisions around HIV, influenza, and EVD.[12,51-53] While such relationships take time to build, are difficult to incentivize from both the academic and public health perspectives, and may not yield immediate results, increased communication between modellers and public health policymakers is arguably the only viable path towards bringing the wealth of existing epidemiological evidence to bear in making public health decisions for infectious diseases.

8. Conclusions

Infectious disease modelling can provide important and useful data to inform public health policy by improving our knowledge of epidemic disease spread, comparing the impact of potential public health interventions and understanding gaps in existing data used to inform public health decision-making. For models to be most useful, challenges in applying modelling results to public health practice, including the complexity of model structure and uncertainty of model outputs and their relevance to important policy questions, must be understood and considered. While modelling will not provide all the answers for public policy, it can provide useful quantitative evidence when large clinical studies are not possible or are still underway. A framework that can be used to improve the process of applying modelling to public health decision-making for infectious diseases is proposed. Collaboration between public health stakeholders and modellers is essential to heighten the transparency and public health relevance of models, to optimize the use of epidemiological data for decision-making, and to develop policies that incorporate scientific evidence to improve the control of infectious diseases worldwide.