Challenges on the interaction of models and policy for pandemic control.

The COVID-19 pandemic has seen infectious disease modelling at the forefront of government decision-making. Models have been widely used throughout the pandemic to estimate pathogen spread and explore the potential impact of different intervention strategies. Infectious disease modellers and policymakers have worked effectively together, but there are many avenues for progress on this interface. In this paper, we identify and discuss seven broad challenges on the interaction of models and policy for pandemic control. We then conclude with suggestions and recommendations for the future.

Introduction

The COVID-19 pandemic has seen a hugely expanded involvement of epidemic modelling in political decision-making. This is partly the next step in an increasing awareness of the importance of modelling that has built up through a sequence of recent epidemics, including SARS, pandemic flu, and Ebola; but even more because the COVID-19 pandemic has made an unusually wide range of challenging demands on decision-making (Dobson, 2021), from first responses through interventions to mitigating the effects of virus evolution. In parallel, it has placed unusual demands on public trust and communications. Attempts to suppress the impact of the pandemic have been helped by much increased public understanding of disease dynamics (with modelling parameters such as the time-dependent reproduction number R (Thompson et al., 2020), previously the private property of scientific specialists, becoming familiar in news headlines as R) but hindered by public dissent and inequalities.

During the pandemic, politicians have frequently claimed to be ‘following the science’. Yet the science does not give a single answer and government responses, both in policies and actions, have varied widely and with variable success (The Independent Panel for Pandemic Preparedness & Response, 2021). Here we shall examine the interface of policy with modelling, exploring the difficulties involved and identifying ways in which both parties can help to tackle future pandemics effectively. Pandemics in general pose major difficulties at the interface of policy and modelling. They require surveillance, planning, and preparation on a time-scale longer than normal political horizons, coupled with a readiness to act very quickly. Outbreaks do not respect political boundaries, so combatting them is most successful when there is a high degree of international cooperation. However, political, social, and environmental differences may mean that the most effective forms of action vary considerably between countries, making it difficult to define best practice. Early intervention can save large numbers of lives, but the earlier the response the less certain the situation; and since the best outcome is quick suppression, politicians risk being criticised for major expenditure and disruption when ‘nothing happened’. That these difficulties are severe is evident if we look back to the pandemics and near-pandemics of recent years. After the SARS epidemic of 2003, advice from the scientific community was clear: it was not a question of whether there would be a pandemic in the future, but only of when (May et al., 2004). Plans were drawn up by many countries and assessments made (NTI & JHU, 2019; WHO, 2011), but COVID-19 still caught most governments by surprise with some of those judged best-prepared faring among the worst. Will the severity of the COVID-19 pandemic (both in health and economic terms) make governments around the world prepare better for the next pandemic, for example, by setting up the best structures for surveillance, decision making, and logistical support? And will governments maintain preparedness over the long term? This paper builds on the work of Metcalf et al. (Metcalf et al., 2015). Many of the core challenges identified in 2015 are still relevant – challenges in communicating the limits of modelling, integrating modellers into policymaking spaces, and incorporating other branches of science. We extend these discussions, placing a particular emphasis on pandemic modelling and policymaking, where further challenges arise from the necessary time constraints of outbreak response. Here, the challenges are grouped into seven broad themes (Fig. 1 ).

Fig. 1

Infographic depicting seven broad challenges of pandemic modelling for policy, around the central equation of epidemic spread. Optimal decision-making and outbreak control aim to reduce the time-dependent reproduction number below 1. This figure was inspired by the era-defining illustrations of (Hanahan & Weinberg, 2000) in the field of Cancer Biology.

We start the discussion by considering the problems of long-term planning (Section 1). Best practice for planning and prevention may conflict with a country’s political culture, for example in emphasizing the public or private sector, or requiring devolution of responsibility from the centre to local government and communities. For similar reasons, countries may find it difficult to collaborate fully, although international cooperation is essential to defeating pandemics (Section 2). Next, decisions on policy need to weigh up the different effects of a pandemic on society, especially on broad public health and on the economy. Health and economic outcomes are often considered in isolation (Sections 3 and 4) whereas an integrated approach may facilitate better political decisions. Other important aspects needing integration include communication (Section 5) among scientists and with the public; the latter is especially important because trust is needed if non-pharmaceutical interventions (NPIs) and vaccination programmes are to succeed. Communication between modellers and policymakers can be difficult because of their different perspectives on the world. A key policy challenge is the need to communicate uncertainty in exchanges between modellers and politicians – not only the uncertainty within models, but also the uncertainty of modelling itself (Swallow et al., 2021). At all stages, and especially at the beginning of a pandemic, a willingness to act under great uncertainty may be essential. This cuts across the norm of advice to government, where a single best prediction is preferred. And this conundrum is exacerbated by the fact that the best outcome in tackling a pandemic is that it does not happen (Godlee, 2010) - it is always difficult for a government to justify significant disruption and large expenditure for a null outcome. One complication that is not easy to communicate is the need for a range of models to help answer the many problems associated with pandemics. Because of its complexity, a single all-embracing model is about as unachievable, and as useless, as the map in children’s book Sylvie and Bruno at a scale of 1 mile to 1 mile: “We now use the country itself, as its own map, and I assure you it does nearly as well” (Carroll, 1890). In Section 6, we discuss some of the main technical challenges in developing models that address key policy questions. In the last section (Section 7), we discuss some of the problems in capacity and attitude required to build a modelling and policy community able to address challenges of future pandemics.

Long-term thinking & preparedness

For long-term control of pandemic spread, broad challenges for policymakers are: to reduce opportunities for new infections to arise and take hold in human populations; to detect new infections as they start to spread by maintaining global surveillance systems; to have established processes in place that determine when and what actions should be taken when a new disease emerges; to have preparations in place to enable prompt and effective actions to control outbreaks, both locally and globally. The part that modelling can play in each of these areas is described briefly below.

Emergence of new infections

Opportunities for new infections to emerge in animal species and to cross the animal-human barrier are dependent on many factors, including the dynamics of zoonotic viruses in reservoir species and the extent of close human contacts with wildlife which, in turn, will be highly dependent on the impact of changing patterns of land-use and of trade in wildlife, and on the effects of climate change. These opportunities could be limited by reduction of both human stress on the environment and close contacts with wildlife, that are themselves a result of wider plans to conserve biodiversity. Modelling the evolution of pathogens in wildlife and of human-wildlife interaction plays an important part in understanding these effects. However, other papers in this volume (Roberts et al., 2021) address the many substantial challenges in these areas including those for policymakers, and they will not be discussed further here.

Global surveillance systems

Surveillance and study of viruses with zoonotic potential, both in wildlife and in humans who have direct contact with wildlife, is the first stage in pandemic preparedness. Systems that record and react to alerts of patients showing unusual symptoms are needed for many purposes, but for pandemic surveillance, it is especially important to note patients in groups that may have had infectious contacts with wildlife. International cooperation on such surveillance is essential.

When to take action

Building on appropriate national/international surveillance or other indicators, modelling can help to determine when it is necessary for action to be taken in the face of a pandemic (or an outbreak with pandemic potential), and what action is necessary to bring transmission under control. This will vary by pathogen. Agreement needs to be reached in advance on what is meant by control (adaptable for different pathogens). Model development can then occur and the criteria to be used to trigger action can be determined as a part of pandemic preparedness. There may well not be time to do this after an outbreak is detected. Effective action depends on decision making under many forms of uncertainty; expert probabilistic judgement is an essential component (Swallow et al., 2021). Policymakers may have to follow the precautionary principle in making decisions based on models of a reasonable worst-case scenario, whilst trying to avoid over-reaction. At the beginning of a pandemic, models should aim for as rapid as possible assessment of risk, by estimating basic parameters (e.g., basic reproduction number, growth rate, infection and case fatality ratios) and their uncertainties, and determining likely modes of spread to inform effective NPIs. Different thresholds for action may be appropriate for different regions, but the longer that action is delayed, the more disruptive and costly it may be to bring a pandemic under control. Again, international collaboration is vital, and cooperation needs to be established in advance.

Advance preparations

Policymakers face challenges in setting up the best structures for surveillance, decision making, and logistical support. In emergency situations with limited information, support will be needed to ensure that technical analyses feed through into policy decisions following established principles. Modelling can contribute to ensuring that preparedness is both flexible and cost-effective with regard to provision of PPE; facilities, equipment, and pharmaceuticals for treatment of patients; and capacity for fast development, regulation and production of novel treatments and vaccines. Planning for a global pandemic rather than a national emergency is key. Long-term investment in people is needed to build/maintain modelling capacity and broad interdisciplinary networks (Section 7). Education on epidemiology/modelling across the undergraduate curriculum in medicine, veterinary and biological sciences would help to improve understanding and collaboration across the disciplines working to control and eradicate pathogens. To devise effective NPIs, it is essential to establish good links between epidemiologists, behavioural scientists, and economists, and also with policymakers in government. It is key to get good scientific advice into governments in a flexible and timely manner. In the UK, Chief Scientific Advisers play an important part here. General training on the use of models for those civil servants and politicians likely to be involved could also improve understanding and enable better communication of results and uncertainties. Meeting the needs outlined above requires significant investment worldwide, that may raise questions regarding international collaboration or competition. Action needs to be maintained over a time horizon that is longer than typical durations in office of politicians and governments. Because of the extreme heterogeneity in size and timing of disease outbreaks, for much of the time effective action will look like over-reaction or waste. And the more successful these measures are, the more this will appear to be the case and the expenditure unnecessary. However, it is important to ensure that policymakers understand that not only are further pandemics certain to occur, but also that they may well take a different form from preceding ones. Being prepared for the unknown is essential.

International cooperation

International cooperation is essential

The COVID-19 pandemic has highlighted the need to consider not only local spread of the pathogen but the potential for repeated introductions through international travel. A number of epidemiological features of SARS-CoV-2 hinder efforts to prevent cross-border spread (Byrne et al., 2020). Visual and temperature checks miss many infected individuals due to a delay between infection and symptom onset, and variation or a lack of clinical symptoms. Prevention of transmission in transit and on arrival is complicated by the potential for transmission via infected surfaces and in respiratory particles. Testing on departure does not eliminate transmission of infection in transit and in any case rapid tests are likely to miss a significant proportion of newly infected cases where the viral load is low. Effective control requires cooperation between countries to minimise the risk of cross border transmission. To do this, all countries need to have the tools available to suppress local epidemics. An essential part of this toolkit is access to epidemiological modelling expertise for regional pathogen control and surveillance. However, neither a situation in which a small number of countries dominate the modelling field nor one in which all countries work independently is optimal. Dominance of a small number of groups means that modelling effort is likely to be unevenly distributed and local variations in human behaviour, demographics, and environment, which may have a large impact on disease dynamics and control efforts, are missed. Equally, working as independent countries leads to a duplication of effort and less overall progress confounded by a disparity between country expertise and availability of resources, and so a happy medium must be found.

More equal access to expertise

Some efforts are already being made to address the variability in country level expertise with a number of intensive and master’s level courses. However, funding is still needed to support epidemiological training that is based in low- and middle-income countries, enabling members of the local community to take part. There is also a need for grants which enable experts from different countries to collaborate and facilitate cross-border application of modelling skills. The reality in rapid response work is that researchers rely on existing relationships. Development of these relationships is therefore an essential component to effective outbreak response.

Sharing generalised, transferable models

Models that address general questions, such as estimating case incidence, epidemic growth rates, or the time-dependent reproduction number (, or colloquially ), can be applied and compared across borders provided there are consistent approaches to data recording. Although is an intuitively simple measure that has been conveyed to the public, the mathematics underlying its calculation are not clear cut with several mathematical and statistical approaches published (Cori et al., 2013; Thompson et al., 2019). Expertise in coding, epidemiological and statistical methodology is required to implement and interpret these methods (Vegvari et al., 2020). International collaboration to support the development and maintenance of a freely accessible suite of models, together with training and documentation, would empower all countries to make and interpret their own calculations of important epidemiological parameters.

Creating international opportunities for modelling communities

The international community of infectious disease modellers has grown rapidly across government, pharmaceutical, and academic settings. To facilitate communication between modellers, there needs to be an inclusive mechanism through which infectious disease modellers can rapidly work together to share information and discuss modelling issues, without fear of plagiarism or misinterpretation. One example is provided by the Isaac Newton Institute for Mathematical Sciences, Cambridge, UK. The institute responded to the current crisis by establishing a virtual network in May 2020, based around a program of talks, workshops, and collaborative projects (Isaac Newton Institute for Mathematical Sciences, 2021). The use of a virtual programme, run by an independent body, enabled infectious disease experts from across the world rapidly to share methods, understand the situation in each country from an epidemiological perspective, and help those with spare capacity to target their efforts. A second example is the MIDAS Network who similarly created a broad community of Covid-19 modellers to discuss and address epidemiological needs in the US (MIDAS, 2021). Both are excellent examples of fruitful international modelling communities.

Integration of modellers into the decision-making process

Ensuring models are used to inform policy

The mechanisms by which scientific advice is integrated into government policy differs from country to country. For example, in the UK scientific advice is coordinated by SAGE, the Scientific Advisory Group for Emergencies, who collate evidence from the modelling subgroup SPI-M (Scientific Pandemic Influenza Group on Modelling) and other expert subgroups. SAGE is responsible for ‘ensuring that timely and coordinated scientific advice is made available to decision makers to support UK cross-government decisions in the Cabinet Office Briefing Room’ (COBR). The advice provided by SAGE and SPI-M does not represent official government policy (SAGE, 2021) and the relationship between Government and scientific advisers is set out in terms of reference (Civil Contingencies Secretariat, 2012). Without effective integration of model projections into policy, models stay as academic artefacts. It is therefore critical that policymakers can consider and understand model outputs during the decision-making process. Modellers can act as translators here, communicating the key findings and nuances of the scientific research. Furthermore, in countries such as New Zealand, there has been a distinction between modelling for policy decisions and modelling for operational decision making, the second being more streamlined only presenting results for agreed ‘working scenarios’ (Hendy et al., 2021). This allowed clearer communication of key model findings for the operational needs of the government. This split has not been observed in the UK but is a topic for further work.

Maintaining policy relevance

Advisers advise and ministers decide. Yet advice from modellers needs to be informed by policy, and policy needs to be informed by modelling advice. As such, it is critical that a symbiotic relationship is set up based on reciprocal trust. This trust is required for producing actionable timely advice in what is ultimately a political process; modellers cannot feasibly model every possible outcome. In addition, there is a distinction between the trust between modellers and policymakers and the trustworthiness of the models themselves. In order to provide advice that meets the policy objectives of governments, governments should clearly articulate these policy objectives.

Avenues for effective interpretation

Research needs to be communicated effectively so that it can be interpreted correctly. As such, modellers should build communication of scientific ideas into their preparation, while policymakers should include understanding of science into theirs. In the UK, the Government Office for Science occupies this brokerage position, with the intention that policymakers are given the best advice. SAGE, as a part of the Government Office for Science, convenes this advice and offers this to ministers, with ministers asking specific questions to be answered by modellers (HM Government Civil Contingencies Secretariat, 2012). Other modelling groups, for example those established by learned societies, such as the Royal Society, may also provide expertise and a brokerage role to connect experts from outside existing modelling expertise.

Organisational structure of modellers within and outside government

The risk is that the organisational structure of modelling advice becomes too bureaucratic, setting up organisational structures to ensure that decisions are appropriately recorded. A challenge exists when subgroups (for example modelling and behavioural experts) have interdependencies. Modelling needs to be developed in strong collaboration with policymakers rather than seen as a service to them, so that new insights from policymakers are included in models, and new insights from the modelling process are included in policy. One approach would be to embed more modellers in policy units within Government rather than being remote from them. However, the independence of the scientific advice is critical regardless of funding institution, and steps should be taken to ensure that this independence is protected by allowing all modellers to publish their findings without clearance from Government. An alternative structure that keeps this independence is for strong links to be developed between government and researchers outside government. Regardless of the structure used, ‘translators’ may be required to interpret policy questions for modellers and to interpret model outputs for policymakers. Further research is needed to ensure that gaps in scientific policy advice are filled, and research is also needed on the best organisational structure to produce the best advice for policymakers.

Understanding competing factors

Any political or research decision requires an assessment of ‘competing factors’, streams of available tools or advice that must be weighed against each other before one is chosen. Understanding the competing factors that occur in decision making (both for modellers and for policymakers) is important for ensuring science-based policy.

Considerations for modellers

Modellers may, for example, be restricted by the availability and quality of data, time taken for data cleaning, and time available to review or seek a second opinion. Model runtime can also be a limiting factor with more complex, spatially-explicit models. From a modelling perspective, a balance must be sought between a mathematically rigorous exploration of the parameter space and the available research time. Policymakers can support modellers by clearly defining the scenarios of interest.

Considerations for policymakers

What currently restricts the way modelling work is used by policymakers? First, economic impacts will underpin every major decision. Space, cost, and staff are essential for any outbreak decision. Increasing space or staff increases the cost (e.g., creating more space for ICU patients requires more ventilator beds to be purchased, at a likely premium price). Secondly, cultural and social context, language, and communication are all important. For example, what is socially accepted with regards to face masks in the communities of interest? What interventions will the general population adhere to? How are new recommendations introduced? Economic considerations are a clear motivator in decision making (see Dangerfield et al., 2021 for a full discussion), whereas the behavioural and other social science considerations can be more difficult to quantify (Van Bavel et al., 2020; Kretzschmar et al., 2021).

Combining scientific expertise

In the UK at least, separate scientific advisory groups exist for each of epidemiological modelling, economics, and behavioural science (Government Office for Science, 2021; HM Treasury, 2021). Broadening the modelling to include social and economic factors could bring in new perspectives (Dangerfield et al., 2021; Kretzschmar et al., 2021; Marion et al., 2021). Recently a number of scientists have begun working on cross-over models, incorporating different scientific disciplines (Rowthorn & Maciejowski, 2020). Deeper collaboration between epidemiologists, economists, and behavioural scientists could be funded from the outset to encourage combined modelling and an appreciation of the contributing factors for pandemic response. However, it should be acknowledged that governmental structures may demand scientific advice be kept separate or be organised by the appropriate department, depending on the relevant remit and expertise.

Communication (to policymakers, between scientists, and to the public)

Communication on the science-policy interface

Communication between professions is a long-standing challenge and is exacerbated by the necessary speed of pandemic decision making (Koplan et al., 1999; Metcalf et al., 2015; Sandman, 1991). It is not clear whether the challenge of communication is a matter for research, or simply for implementing known good practice, or both. Training for modellers from those with direct experience in communicating technical results to policymakers could be useful. Funders could address this by creating a time requirement for science-policy communication training. In addition, creating opportunities for short-term secondments at all levels could be useful for generating understanding of effective communication.

Identifying the policy question

One challenge is communicating effectively with policymakers to convert questions of interest into something tractable and analysable, in a form that modelling might be able to address. Holding a dialogue at the start of any project is also essential for ensuring both understand the nature of the policymaking decisions and environment. An academic and a policymaker may have different initial ideas on what is the right research question. They may also have a different philosophical stance on what is rational. Both must work together to form a research question that is both useful and answerable. Building in non-traditional academic rewards, such as for effective communication and documentation, could motivate progress on this challenge (Kucharski et al., 2020). There are currently few academic incentives for making models more accessible and beneficial to the end user. The way that universities and funders appraise research groups and individual researchers should reconsider this.

Communicating uncertainty

Understanding how to communicate uncertainty (including outputs from multiple models and undertaking multi-model comparison) in an unbiased way is important. Mathematical modelling is based on predicting possible future realities and as such it is reliant on the parameters and assumptions it uses. The same model solved for different parameters (e.g., numbers of contacts within different settings or efficacy of an intervention) or initial conditions (e.g., number of people infected at the onset) results in different outcomes. Stochastic models add further variability from both demographic and environmental sources. Statistical estimation typically generates a confidence or credible interval that quantifies uncertainty in knowledge of parameters and even model structure. This uncertainty should be clearly explained when results from modelling are presented, ensuring that both the modeller and the policymaker understand the relationship between model uncertainty and its physical relevance. Communication of those uncertainties on the scale of the decision process (particularly in terms of meaningful quantities for practitioners) is an important challenge for modellers and end-users. A further challenge is that of remaining unbiased when communicating model uncertainty. The modeller must choose which particular points of uncertainty to emphasise, in the knowledge that only a high-level summary of results will be available to the final decisionmaker. In this sense, the modeller has an important but perhaps hidden role of in making value judgements when deciding what uncertainties to emphasise. Further to this, the modeller must also highlight any inequalities in the model that may arise from unconscious bias of the modeller (such as a focus on minimising burden for the masses over disadvantaged minority populations who may be at far greater risk). Promoting diversity in modelling teams and recruiting health experts from minority populations, as has been done in for example New Zealand (Ministry of Health NZ, 2021), will help to reduce this bias. For deeper discussions on the role of values in science advice, we direct the reader to Douglas (Douglas, 2009).

Model transparency

Thought should also be given to the way model results are presented to policymakers. How can the insights modelling has given be explained in a straightforward way? It is essential to present the assumptions of the model in a transparent manner, being mindful of what is practically, and not just statistically, significant. A “Research in Context” box or lay summary aimed specifically at policymakers could be included in all pre-prints and papers. This would state how, why, and when the model can and cannot be used, allowing all stakeholders to consider the validity of the assumptions and whether any driving factors have been accidentally or intentionally omitted.

Accessible and unambiguous presentation

A further question to consider is what graphics will best illustrate findings. Improving visualisation during an outbreak is difficult due to the additional time taken and the need to prepare a set of results that will be seen by many different audiences (Chen et al., 2021). An interactive dashboard or other tool could be helpful for demonstrating uncertainty to different audiences. Is the uncertainty large enough to make a difference to outcomes? Will long-term predictions with large confidence intervals be misinterpreted? Selective publishing may be necessary – this could mean deciding what features of a graph should be omitted or included to ensure it is interpreted usefully by non-scientists. Such questions are worth considering when results are presented to policymakers or other non-scientific audiences.

Dealing with an evolving system

With ever-changing data, modelling suggestions and predictions will also change. Model findings should be updated as parameters and assumptions are updated, and allowances made for temporally changing responses to the evolving pandemic. This dynamic updating of models requires continuous feedback between modellers, policymakers, and other researchers.

Communication within the modelling community

What should be done when models disagree? Techniques such as ensemble modelling and inverse modelling can offer some insight but there is unlikely to be one clear-cut solution (see Section 6 and Swallow et al., 2021). Dissenting opinions reduce the risk of groupthink, a psychological phenomenon where the desire for harmony or conformity in a group of people can result in an irrational or dysfunctional decision-making outcome (Jarvis, 1971; MacDougall & Baum, 1997). The challenge here is to allow dissenting opinions while still communicating a consensus view (if there is one). When considering work within the modelling community, we should also acknowledge the significant modelling research that informs policy but is not directly presented to policymakers. An example of this would be research that calculates parameters such as the generation time (Hart et al., 2021), a necessary input for models calculating (which in turn inform policy). These parameters are crucial inputs for policy-facing models and broad communication within the modelling community will ensure best estimates continue to be used. One main avenue for communication in the modelling community is through academic peer review. Traditional blind peer review is often unfeasible on the timescale of outbreak response and presents an ongoing challenge for the assessment and validation of timely model findings. One solution is through the use of ‘rapid review’ or ‘live review’ – in addition to many journals accelerating their peer review process at the beginning of the Covid-19 outbreak (Horbach, 2020), some implemented open-source platforms for dedicated rapid review. An example of this is the Outbreak Science Rapid PREreview platform for preprints relating to emerging outbreaks, partly funded by the Wellcome Trust (Johansson & Saderi, 2020). Reviewers respond to short yes-or-no questions on the high-level importance and quality of each study and all reviews are open. Platforms such as this could be a useful intermediary for scientific assessment and validation in future outbreaks.

Communication across scientific disciplines

How should unified response across scientific fields be ensured? Direct communication between modellers and field scientists is vital to ensure that data collection results in data that can inform models. For this, capacity needs to be developed for rapid collection of useful data, and for data cleaning, linkage, and anonymisation. Regular meeting of minds between scientific fields can also enable knowledge sharing and creation of a common language. When classifications are used and changed (for example, differing testing strategies), this information must be clearly communicated within the data set (Shadbolt et al., 2021; Swallow et al., 2021). Good data can improve the predictive value of models, enabling identification of routes of transmission and enabling identification of optimal control policies

Communication on the public interface

Public-facing professionals need to stress that policy is not entirely based on models. Modelling is only part of the evidence. Economics, public response, and politics (among other things) also affect precise implementation of the policy. It should be recognised that while modelling provides useful insights, models will not give “correct” or “true” depictions. Those who are presenting scientific results to the public need a sufficient understanding of the original model such that the uncertainty can be communicated clearly. This is likely to require joint education on policy and on modelling (Brownson et al., 2017). Furthermore, one should be mindful of not only what is said to the public but how it will be received. This challenge applies to all public-facing professionals, not just policymakers. Getting across the correct message is difficult (Bubela et al., 2009). The way that advice and interventions are communicated will feed back into the response. For example, how do you communicate a result of the form “there is a 40-50% chance of a 30-40% increase in mortality”? This is not a straightforward concept (Horby et al., 2021). When scientists have a national audience, a science budget for communication and PR support could help ensure this valuable audience is maintained and used effectively.

Technical model challenges for policy

How do we make decisions under different modelling frameworks?

Consulting a range of models may result in conflicting inferences or predictions, and it is important to consider how those differences impact policy decisions. Different modelling techniques may be preferred by individual modelling groups and can be seen as a strength rather than a weakness (The Royal Society, 2020). Model averaging and modelling ensembles can be used in these instances where there is no specific reason to favour a single model. However, how to weight and combine model inferences when resolutions and decision landscapes are potentially very different still presents methodological challenges. Providing syntheses of evidence across models accounts for variability but how to interpret that variability requires expertise across disciplines. For example, an individual-based model may be an appropriate tool to enable understanding of early transmission dynamics, but compartmental models, network models, and contact matrix approaches may be preferred as the epidemic grows (Robertson, 2019).

Complexity and realism versus speed

There is an inevitable trade-off between minimal introduction of bias from existing models developed on other diseases (e.g., potential influenza bias in early COVID-19 UK modelling), and the speed benefits of using existing models. Often decisions need to be made on a timescale that is not conducive to developing models completely from scratch. Particularly at early stages in a pandemic, when knowledge of the new disease is limited, reliance on knowledge of potentially similar diseases is required. This is somewhat analogous to updating of Bayesian priors with data – models and policy are updated as knowledge and data improve. Ensuring trust in models can be difficult at this stage when reliance is weighted towards using tangential knowledge and outputs are changing rapidly. The availability of new research on a short timescale brings its own unique challenges to modelling. For many streams of science, scientists can only use existing research when a crisis hits. You cannot set up and run a clinical trial in a matter of hours, but you can update and run a model. This allows new insights mid-outbreak but there is unfortunately little time for the many iterations and review processes, both internally and externally, that a model would usually undergo. Policy should inform modelling pipelines that allow these iterations to take place where feasible. Modelling ensembles can also guard against problems of single models introducing biases if not correctly verified.

Dealing with uncertainty

Models developed in a scientifically and statistically principled manner, allowing for justified and tractable/reproducible choices, should aim to offer unbiased estimates of processes. However, there is always a risk that an unknown important process is not being accounted for. There will always be uncertainties, even with the best data available, and communication of those uncertainties on the scale of the decision process (particularly in terms of meaningful quantities for practitioners) is an important challenge for modellers and end-users. Suppose that a policy decision is to be made to achieve some outcome, for example that the number of ICU beds occupied by COVID-19 patients does not exceed some pre-specified limit and suppose that a mathematical model depending on a set of inputs is available to predict this. The current approach is to set up and run a number of scenarios. These are not intended in any sense to be what will happen in the future but rather ‘possible’ futures that are indicative of what might happen. Usually, a small number of such scenarios is used to help span the outcomes of possible decisions. An alternative approach is to optimise the final outcome over the possible decisions. In many ways, this is better than using scenarios as it gives a suggested set of policies that are in some sense optimal, but it is computationally much more expensive. Setting up paths for modellers to access high performance computing resources in quick timescales is an important part of the process. Also, by choosing an optimality criterion, it can seem to tie the policymakers' hands. Another approach is to set up the decision support problem as an inverse problem, by finding those decisions (usually sets of model parameters) that lead to ‘good’ outcomes (for example, not exceeding hospital capacity). This idea is closely related to history matching (Andrianakis et al., 2015) as a way of calibrating models (estimating parameters). Traditionally, producing inverse models is a complex and difficult process but the approach is made feasible by using a model emulator: a fast surrogate model that approximates the full model and gives a measure of its own accuracy (thus allowing the additional uncertainty of the surrogate model to be accounted for in calculations). The fast surrogate can be used in inverse mode relatively easily, simply by running it forward everywhere and diagnosing the inverse model from the results.

Understanding discrepancies and uncertainties across models

One of the challenges is communicating the differences between models, but a precursor to this is understanding why the differences exist. Is it a mechanistic difference between the models (mathematical/statistical) or are the models answering different questions or using subtly different data? Most of all, are the differences between models important with respect to their use? As mentioned above, ensemble modelling allows some of these discrepancies to directly be accounted for. There has been good work in the area of weather/climate modelling and prediction on the combination of different models (Gneiting & Raftery, 2005). These methods have only recently been used with epidemic models, so far with limited success (Bowman et al., 2020).

Involvement and recognition of all contributors

Glory for outcomes is often taken by end users, not those in data collection. Models cannot replace good data collection, and funding should continue to be allocated for primary data collection. A challenge is to create meaningful collaborations between modellers, data collectors/ domain experts, and independent scientists. Co-authorship of outputs recognises the contribution of all parties, but this is more challenging with public/ open data sources.

Implementing change

Achievable change

The necessary precursor to implementing change is to agree what change is needed. This paper has discussed many aspects of what ‘good practice’ at the modelling-policy interface might look like. The discussion now needs to be overlaid by practical considerations, such as the form in which advice and communication will be useful in policy formation, and current skills and practices in government. There is also a need to plan for multiple timelines, as while continuous improvement might be desirable, the best situation in several years’ time might not be achieved only by successive incremental annual cycle projects. Implementing change requires both the capability and the will. It is very likely that there will be a stated will for improved practice in pandemic response, and this needs to be followed through by action if or when comparatively normal times return. One feature of such normal times is that development of decision support processes might be seen as bringing benefits for a ‘next time’ that may never come, rather than at the time money is spent. This section discusses challenges in implementation, in terms of both developing technical modelling capabilities and the will to achieve what is necessary. From a policy perspective, there is a difference between wishing to see improved processes and practice, and recognition of the scale of change that this might require. An example is the UK’s HM Treasury Aqua Book which provides an excellent statement of high-level principles across the use of analysis for decision support, oversight of analysis studies, and technical requirements in modelling (HM Treasury, 2015). Some of the content, such as that on governance, maps well to current skills and practices – but it is an entirely different matter, for example, to implement on larger scale models the recommendations for treatment of uncertainty. If these more challenging ambitions are to be realised, major changes may be required to decision support procedures, technical methods considered, and the related skills base, along with a large-scale, multi-year innovation program across government. Research and development into how to translate advanced analytical methodologies into widespread practice may also be needed.

Community organisation

Through the COVID-19 pandemic, the capacity for analysis has likely increased considerably and there have been substantial advances in available applied methodology. However, this experience has been gained during an emergency, and preparing a community to be ready to provide advice in future emergency situations with highly uncertain circumstances is a separate task. As part of a community reorganisation, structures including funding should be designed to facilitate entry of new ideas, people, and disciplines, where this will have value. There is often an advantage of incumbency both in accessing funding and in influencing policy processes – while we certainly do not wish to downplay the vital role of the mainstream applied disciplines, it is necessary to assimilate new thinking where this will be beneficial. The development of community capabilities should also consider the incentives on people in different employment situations. In particular, depending on the norms of their discipline, university academics may not have direct incentives to coordinate with the rest of the community in being ready to advise in emergencies. In that sense, national laboratory-type research institutes could be more natural structures for this kind of advisory work. This matter also links to discussion in earlier sections as to whether it is better to have an ecosystem of different models, or a smaller number of models specifically developed for policy support. There are certainly dangers inherent in picking winners among analytical approaches, but whether or not a government has directly commissioned the models it uses, it is clearly beneficial for governments and community leadership explicitly to have considered what kind of community structures will provide good practical decision support.

Advancing communication

Lastly, depending on the specific question at hand, it may be that new research is required, or it may be that it is necessary to specialise existing knowledge to the particular case. As discussed earlier in this paper, there are areas of public communication where implementing known good practice could be very effective. In internal communication between policymakers and analysts, there is likely to be a need for innovation on how to feed properly caveated evidence into decision making (e.g., a full uncertainty treatment associated with a modelling study, or the synthesis of multiple modelling studies).

Conclusions

Research into pandemic preparedness has been carried out in the past, but the COVID-19 pandemic has allowed us to review this assessment and consider how modellers and policymakers can be best prepared for future emergencies.

Preparedness

Preparedness is key, but a decision to invest may not give quick returns. In long periods without a pathogen outbreak, investment in infrastructure and readily mobilised resources may appear to be wasted, but there is commonality with other possible national emergencies. As discussed previously, due to the heterogeneity in pathogen outbreaks, effective preparedness will often look like over-reaction. There will be pressures to cut funding but this may lead to continuing lack of preparedness. While the timing of further pandemics is unknown, they are certain to occur. Common infrastructures will allow rapid response, but these should be sufficiently flexible to allow moulding to the specific pathogen. It is essential to be open-minded to the form that a new pandemic might take.

Collaboration

A key theme in this paper is that policymakers and modellers require efficient methods of collaboration. Policymakers should aim to understand the research that is communicated to them, and equally, modellers should aim to understand the requirements of policymakers, both in the speed of advice and the communication of that advice to the wider public. As discussed in the main body of the paper, this could be achieved by embedding more modellers into the structure of government. However, a far more practical solution and one that need not require significant government restructuring, is the development of strong collaborations between policymakers and non-governmental researchers. For example, the UK Government maintains a regularly-updated list of Areas of Research Interest to improve alignment of academic research with policy development and decision-making (Government Office for Science & Cabinet Office, 2021). These projects form an ideal starting point for meaningful collaboration.

Communication

Finally, even the best modelling is redundant if it is not communicated well to policymakers. Modellers need to act as translators – converting scientific research into non-specialist language and clearly presenting key findings to policymakers. In the UK, Chief Scientific Advisers currently perform this role (HM Government, 2021); after quickly sourcing expertise, much of their role is in translation. Modellers, and the broader academic community, need to supplement this work, ensuring that the key themes of their research are communicated correctly. In summary, collaboration and communication between modellers and policymakers are fundamental for pandemic preparedness and effective future outbreak control.

Authors’ contributions

All authors took part in discussions and wrote sections of the manuscript. L.H. coordinated discussions throughout and compiled the final version of the manuscript. All authors edited the manuscript and approved the final version for publication.

Declaration of Competing Interest

V.I. and D.M. declare a conflict of interest as guest editors of the Virtual Special Issue. Both have refrained from participating in the review process of this paper.