Guidance on good practice in conducting scientific assessments in animal health using modelling.

The EFSA asked the Panel on Animal Health and Welfare to develop a guidance document on good practice in conducting scientific assessments in animal health using modelling. In previous opinions, the AHAW Panel has responded to two-thirds of animal health-related mandates using some kind of modelling. These models range from simple to complex, employing a combination of scientific, economic, socio-economic or other types of data. Hence, there is strong interest in the development of a guidance document to integrate modelling efforts into the routine process of EFSA working groups. In this document, an 'operating procedure' (OP) for the use of modelling within an AH working group is presented. The OP provides a detailed flowchart enabling modelling to be transparently and consistently integrated in the assessment. The OP is structured into phases. These phases combine the relevant standard operating procedures and working instructions of EFSA with the modelling process. Each phase includes roles and actions to be taken, expected output and the sequence of agreements that need to be made between all partners in the scientific assessment. In conclusion, it is expected that adherence to the OP will improve transparency of models in EFSA outputs, and it is recommended to adopt it as a standard procedure when responding to AHAW mandates.

Summary

The European Food Safety Authority (EFSA) asked the Panel on Animal Health and Welfare to develop a guidance document on good practice in conducting scientific assessments in animal health using modelling.

In the past, EFSA has used a range of models to inform opinions on important problems in animal health (AH), including qualitative models, pathway‐based decision tree models, epidemiological transmission or disease spread models, diagnostic test simulations on virtual hosts or host groupings. These models have ranged from simple to complex, employing a combination of scientific, economic, socio‐economic or other types of data.

Modelling was used during the preparation of approximately two‐thirds of past AH mandates (22/31). The use of modelling in this work is logical, providing a structured representation of our knowledge about the ‘real world’ underpinning each animal health problem. There is strong interest in the development of a guidance document to integrate modelling efforts into the routine work of EFSA working groups.

In this document, we present an ‘operating procedure’ (OP) for the use of modelling within an AH working group, to support animal health decisions or to inform scientific risk or benefit assessments. A detailed procedure is presented, providing the chair and other members of a WG with a flowchart that lists the actions to be taken and the decisions to be made, to enable modelling to be transparently and consistently integrated, and objectives achieved.

The OP includes four main phases, which include the initial receipt of the mandate, the development of a strategic work plan, the implementation of the work plan and the reporting for final adoption of the resulting opinion. Each phase includes actions to be taken, the contributing actors, the expected output and the relevant approval stages (i.e. sequence of agreements that need to be made between all partners in the steps towards a final scientific assessment). The OP highlights the points when agreement must be achieved within the working group on key methodological issues. Therefore, within the OP relevant decision points are scheduled.

The guidance document recommends the development of a dynamic glossary containing standard terminology for the use of mathematical and statistical models in risk and benefit assessments, to be maintained and continuously peer reviewed by EFSA experts. This will support consistent use of terminology by AH opinions, at least to be consistent within the Panel and preferably across all EFSA outputs.

In conclusion, it is expected that adherence to the OP will improve transparency and acceptability of models in EFSA outputs. The Panel recommends its adoption as a standard procedure when responding in an AHAW mandate.

Background and Terms of Reference as provided by the requestor

During the process of assessing a potential risk for animal health, the use of models can be a prerequisite where mental simulation is not able to represent multiple causal links within a system. Models can help experts from different fields to interact and use all the available scientific evidence to answer with best confidence the risk management question. Models present a reflection of our understanding of the ‘real world’ and allow explaining or predicting effects.

Different types of models exist, and different approaches can be used to categorise models, one being to group them into qualitative and quantitative models, and their roles in a scientific assessment process will vary according to the circumstances.

During scientific assessments, however, the danger may exist that good practice is overlooked whilst applying models under time pressure. Potential hazards to using modelling in risk assessments may be poor communication among subject experts and modelling experts, which may lead to poor understanding of the objectives for the modelling or its role in the scientific output.

Considering the above, the aim of this mandate is to produce guidelines for the use of mathematical and statistical models to be provided to chairs and members of AHAW Working Groups (WG) as a support while conducting scientific assessments in animal health during their specific tasks and mandates. The goal is that models should be well‐integrated into the scientific work and reflect the intention of the mandate, input of parameters into models and integration of model outcomes into the scientific output.

TERMS OF REFERENCE AS PROVIDED BY EFSA

A glossary should be created, and definitions should be provided in order to establish a standard terminology for the use of mathematical and statistical models.

Description of the main types of models, including usefulness and limitations, is needed.

A set of criteria or questions to guide a WG through the process should be developed, starting by the decision of using a model or not, choice of model and the possible objectives and roles of a model and recommended processes for model verification and validation.

Procedural guidelines for the proper integration of modelling into WG standard operational procedures should be developed. This may include effective communication among WG members and risk managers, common understanding of objectives and the role of the model in the scientific output, as well as guidelines on the role of the subject and modelling experts within the WG.

During the review of the guidance document by the AHAW Panel in November 2019, it was agreed to revise the guidance to take account of recent relevant developments in this area.

The following changes have been made:

In the main document:

Update of Section 2.3 regarding data limitations and the link to EFSA’s guidance documents on uncertainty;

Addition of Section 2.3.4 on model transparency;

Update of Section 2.4 on the procedural guidelines for appropriate integration of modelling into Working group standard operational procedures, including an update of Figure A.1.

In the Appendices:

Update of the standard terminology for the use of mathematical and statistical models.

The other sections of the document have not been modified.

Assessment

Introduction

Models in general are a reflection of our understanding of the ‘real world’. The type of model used in a specific situation is determined by the purpose of the task, and by the availability and type of data, and often also by the available expertise and resources.

A model can be developed for one or several reasons including simplification of complexity, synthesis, optimisation, analysis, explanation, assessment, prediction and simulation of complex systems. Considering these broad possible uses, models can assist:

where multiple causal links within a system cannot be adequately represented by mental simulation (Lempert et al., 2003),

when structuring available information or hypotheses about potential causal processes,

in appropriately integrating available scientific evidence, and

in explaining, predicting, forecasting and nowcasting effects.

Hence, models (i.e. ‘modelling’) are likely to be at the core of the scientific assessment.

To assess the role of modelling in past EFSA AH opinions, a procurement was launched to systematically review the application of quantitative modelling in these documents. It was found that 21 of the 31 used some kind of model and 12 included a quantitative assessment applying in total 21 different modelling approaches (Singer, 2010).

Decisions on the development of new or use of existing models for a specific mandate have to be an essential element of the planning of the scientific assessment process. Experience with previous EFSA mandates has demonstrated some of the challenges faced when models are used as part of the scientific assessment. As a consequence of communication problems within the working group (WG) and beyond, some models have not been optimally integrated into the resulting scientific output of the Panel.

The main causes of communication problems between subject and modelling experts were (a) the unclear reasons for selecting different model types to deal with apparently similar risk questions, (b) a lack of understanding of the model development process by those not directly involved, (c) insufficient communication among all those involved in scientific assessment process (risk modeller, WG, Panel members and requesting party), (d) a lack of documentation about the model structure, (e) the time constraints and (f) lack of transparency in relation to decisions taken with respect to the modelling process. Consequently, discussions about the objective and purpose of the modelling, and of the appropriateness of selected tools and resulting outcomes have often only been held during the final phase of scientific assessment process, when adjustments to the modelling process are no longer possible.

The overall aim of this mandate therefore is to produce guidelines to chairs and members of AHAW WGs for the use of models in support of scientific assessments in animal health. The objective is to establish a structured process that results in models which are:

more transparent,

accepted by the majority of stakeholders,

directly related to the scientific work,

produced in a timely fashion,

consistent with the Terms of Reference laid out in the Mandate.

and account for uncertainties about the model structure (EFSA Scientific Committee, 2018a).

A large amount of literature is available on the topic of modelling (e.g. Scott and Smith, 1994; Mollison, 1995; Grenfell and Dobson, 1995; Hudson et al., 2002; Grimm and Railsback, 2005). It would be beyond the scope of this guidance document to cover all the technical aspects of any modelling that could be useful for addressing future mandates. Therefore, it was decided to include only a rather general description of main model types including their usefulness and limitations, as an Appendix A (thereby specifically addressing ToR item 2).

The core of this guidance document addresses items 3 and 4 of the ToR. It is comprised of a flowchart laying out the process of (a) deciding whether a model is needed early during acceptance and clarification of a given mandate, (b) assuring that all partners agree on the objective and scope before starting the modelling process and (c) establishing sufficient communication and feedback loops within the WG and between WG, Panel, Secretariat and Commission during the scientific assessment. This flowchart is annotated with explanations and accompanied by a short general glossary of technical terms used in mathematical and statistical modelling (ToR item 1) to establish a common level of understanding of the relevant terminology.

Standard terminology for the use of mathematical and statistical models

For the guidance document, the definition of a ‘model’ includes a wide range of approaches from conceptual models (Dresner, 2008; Thulke and Grimm, 2010; i.e. verbal description or graphical representation of possible structures and relationships), to complex mathematical implementations (translations) of such conceptual models (i.e. computational models). The conceptual modelling is based on concepts and knowledge arranged to represent the processes and interaction in a disease‐host‐management problem explored without using technical tools, e.g. by mental simulation according to Lempert et al. (2003). The computational modelling refers to construction of technical tools (e.g. from semantic logical axioms, scoring systems, semi‐quantitative flow diagrams, mathematical equations, or numerically simulated dynamic systems). Appendix A identifies characteristics of computational models that may enhance communication of a particular model in use. Moreover, the set of standard terms referring to the use of mathematical and statistical models was collated from previous AHAW documents.

While identifying standard terminology, a recurrent problem of the absence of universally agreed definitions in risk or benefit assessment terminology was faced. With the exception of some instances e.g. the OIE handbook on Import risk analysis (OIE, 2004) and some of the EFSA opinions, glossaries containing definitions in risk assessment for animal health are rare. One reason might be that the terms used in scientific assessments can vary between different disciplines such as ‘risk’ in animal health and quantitative microbiology. This might explain why glossaries are rarely included in textbooks on quantitative scientific assessment. Further in qualitative risk and benefit assessment, the interpretation of the verbal grading of levels of occurrence and consequences may vary among different assessments.

Procedure followed to select the preliminary list of terms and definitions from previous AHAW opinions

In order to reply to the first ToR, it was decided to search for the relevant terms and respective definitions used in previous AHAW scientific opinions and reports dealing with animal health issues. This review was carried out in 2010 (Singer, 2010). A total of 31 opinions have been analysed for terms related to risk assessment and modelling. The following terms were searched at the beginning: risk, assess, model, statistic, probability, prevalence and epidemiology. An Excel table was developed with the terms found, the definition provided or the text related and the location in the document. Some terms were found in the opinions' glossary, some were retrieved from the text. A reference number was given to each term/definition. Experts were asked to evaluate these terms and the definitions provided and to classify them using the following criteria:

GGeneric in context of modelling.

SSpecific in context of modelling.

NRNot relevant in context of modelling.

DTo be discussed important concept for risk communication (to be included, e.g. a recommendation).

It was decided to include in the standard terminology only those terms that were classified as ‘Generic’ (G). Additional terms with definitions were included according to the expert's knowledge, e.g. if used in the current guidance. Terms and definition were provided with a source reference if the definition was cited without modifications from that reference. All the other definitions were elaborated and agreed by the experts (see Appendix B – Standard terminology for the use of mathematical and statistical models). Because that list is not exhaustive, a wiki‐based approach was recommended for an EFSA glossary on standard terminology for the use of mathematical and statistical models.

Proposal for a harmonised terminology using a Glossary

Although there are some generally applicable terms in modelling, their precise interpretation may vary slightly according to the discipline or context of a specific scientific assessment. One option has been to define terms for each particular EFSA scientific report/opinion. This, however, leads to an unstructured assemblage of relevant terms and concurrent definitions of the same term.

Such a glossary would facilitate the process of comparing outputs from different assessments, e.g. by identifying the differences between different model types. The already existing definitions in the EFSA AHAW opinions as well as the standard terminology list of this guideline could serve as a basis.

The recommended glossary should be preferably dynamic and kept in a central internal repository of EFSA containing standard terminology for the use of mathematical and statistical models in risk and benefit assessments. The access to the glossary should be limited to EFSA and its WGs. The formalities of updating the terminology should be specified.

The internally available compendium of agreed terms is recommended to be used as follows: (a) a self‐sustaining glossary is still required for each scientific output, but the general glossary should be considered as first source when definitions are provided, (b) if an existing definition is found to need improvement, the WG can recommend an update.

As a consequence, definitions used in EFSA communications would be consistent at least within the Panel and preferably across all EFSA outputs.

Model characterisation, model selection and model transparency

Model characterisation

Models can be classified according to different criteria. These criteria indicate whether a certain feature is present in the model. Examples are quantitative vs. qualitative; analytic vs. simulation; complex vs. abstract; dynamic vs. static, strategic vs. tactical (Holling, 1966; May, 1973); top‐down vs. bottom‐up (Grimm and Railsback, 2005); associative vs. process (King and Soskoline, 1988); empirical vs. explanatory (Thrusfield, 2005); non‐spatial vs. spatially implicit vs. spatially explicit; frequentist vs. Bayesian. The purpose of model characterisation and categorisation is to provide a common, shareable, configurable framework of information to describe a model of any type. These approaches of categorising modelling approaches are often seen as competing or mutually exclusive.

According to the scientific review done through the procurement (Singer, 2010), the previous EFSA AH mandates show that:

Mandates, some with relatively similar terms of reference, differ in the type of modelling techniques that were used,

Stochastic decision trees, constructed from conceptual pathways, were the most‐frequently used modelling approach,

Heterogeneous methods were applied to answer questions on specification of diagnostic test characteristics, and

The spread or the transmission of diseases was seldom asked to be modelled.

Since the development of this guidance document in 2010, disease spread and transmission models have been used in several of EFSA scientific assessments such as those on Rift Valley Fever, African swine fever and Lumpy Skin Disease (Nielsen et al., 2020; EFSA AHAW Panel, 2020, 2021a).

The exhaustive model characterisation will assist in communicating the particular modelling approach adopted. For a selected model, the classification of the modelling technique (Appendix A, Section A.1), the methods of analysing the model (Appendix A, Section A.2) and the related uncertainties should be specified in the model report to enhance transparency in communication.

This guidance does not specifically consider artificial intelligence techniques, including machine learning or deep learning (see Appendix A – Model characterisation). Concerning models based on machine learning techniques, implementation and validation phases may follow different approaches than those described in the present guidance. For this reason, the application of the guidance for this type of models must be evaluated on a case‐by‐case basis.

Model selection

The specific objective or purpose of a model‐based scientific assessment along with the data available will guide whether a particular modelling approach will be suitable (Starfield et al., 1990; Roughgarden et al., 1996; Philips et al., 2004; Grimm and Railsback, 2005; Garner and Hamilton, 2011). A thorough and structured assessment of the available data and data that would be needed (and the uncertainties associated with it) should be performed in the initial steps of the scientific assessment before a final decision on the model selected is adopted (see Section 2.3.4).

The use of formalistic decision flows as a guide during model selection (part of ToR item 3) is not recommended. It is recognised that such a decision tree would (a) never completely reflect the scope of existing models, (b) not be agreeable to all modelling experts, and – most importantly – (c) reduce the flexibility of future WGs during the selection of model(s) that would be fit for purpose in achieving the specific objective. Often, there are several approaches that might be suitable when addressing a particular mandate (Philips et al., 2004).

Three general objectives have been described in the literature (Figure 1). The purpose of modelling either focuses on more comprehensive description of collected data or aims at a systemic understanding, or the prediction/nowcasting/forecasting of future events (Hall and DeAngelis, 1985).

Figure 1

Identified purposes of modelling with typical examples

Note: Overlapping ellipses symbolise potentially non‐distinct purposes of practical models.

The purpose of models made for description is to clearly and systematically extract information from available data (including empirical data sets). It is important to gain an understanding of data availability, and of their inherent content and uncertainty. Models applied for descriptive purposes can provide important clues for explaining relationships. The purpose might relate naturally to the concept of ‘data‐driven’ modelling. The outcome of descriptive models may be hypotheses, which can then be challenged if new data become available.

The second purpose of models aims at improving our understanding of a system. The modelling can result in relative comparisons and show variability in outcomes given different assumptions. Subsequently, it allows to understand the impact of some parameters, assumptions/scenarios. Accuracy is often a less important issue when modelling relative or worst‐case scenarios to aid decision‐making. The complexity of a system and its internal relationships can generate emergent findings, and then lead to improved system understanding. This purpose can be related to the concept of ‘knowledge‐driven’ modelling.

The last group of models deals with prediction/forecasting of future outcomes. Models intended for predictive purposes often attempt to mimic nature in great detail, leading to so‐called ‘naive realism’ (Grimm and Railsback, 2005). High accuracy, high predictive value and minimal uncertainty are desired, but on the other hand unforeseen changes can have an impact on the precision of the forecast. This model category can be associated with both ‘data‐ and knowledge‐driven’ modelling.

The three categories reflect differences in the expected outcome of a modelling procedure. However, as the overlapping ellipses in Figure 1 indicate, in practice objectives of modelling reflect smooth transitions between the identified purposes. In addition to combined model purposes ‘description and understanding’ and ‘understanding and prediction’ shown in Figure 1, it is also possible that a model addresses both the purpose of ‘description and prediction’, without the purpose of understanding.

Modelling process

The usefulness of a model can be enhanced by transparent and comprehensive communication of the model structure, the assumptions and outputs, including related uncertainty. When the outputs and conclusions are based on the result of a model or statistical analysis, uncertainties not quantified within the model or analysis, as well as uncertainties about the assumptions of the model should be considered. This should be carried out as part of the process of characterising the overall uncertainty (EFSA Scientific Committee, 2018b) (see subsections below). Recent hierarchical schemes of model integration in decision‐making outline the sequence of steps in a modelling process (e.g. Schmolke et al., 2010). The process can be divided into three groups of activities that need to be completed in order to produce a transparent and comprehensive model:

Model design & model formulation.

Model implementation & model evaluation.

Model application & output communication.

Correspondingly, the associated decisions and outcomes have to be documented along with the procedural steps to allow retrospective justification, the communication to external participants as well as guidance for methodological reporting.

Model design and formulation

This activity comprises of two elements: (a) the formulation of objectives to be addressed by the modelling and (b) the description of the model design.

The formulated objective, e.g. an identified risk question, is needed to focus the discussion amongst participants, and is part of the first step in the process of the scientific assessment during the protocol development phase (EFSA, 2020a). At this stage, evidence needs for the methodological approach proposed (i.e. the model) should be also defined, thus setting the minimum set of data that will be required to conduct the scientific assessment. Availability of data/information about the system to be modelled will then be assessed, e.g. through systematic literature reviews, as described in the EFSA guidance document on the application of systematic review methodology to food and feed safety assessments to support decision‐making (EFSA, 2008). Uncertainties associated with the assessment (both linked to the data inputs and to the assessment methodology) should be also identified in a systematic manner (EFSA Scientific Committee, 2018a,b) and, if needed, the approach initially considered should be revised. Agreement needs to be reached amongst participants about what represents acceptable model inputs and model outputs. A key question will be how the model output eventually will inform the conclusions expressed in the scientific opinion. Participants involved in the modelling process need to be made aware that the formulation of the objectives and data review can be one of the most time‐consuming steps in the whole modelling process.

The description of the model design refers to the joint specification of the conceptual model (i.e. verbal and graphical description). The conceptual model provides the basis for the model development process (Pascual et al., 2003; Philips et al., 2004). Based on this, the model type and its overall complexity can be justified considering available resources and timelines. Here, the participants in the modelling process identify all important simplifying assumptions. The relationship between the model design and the model objectives needs to be documented. Written documentation should be provided at this stage because it will be essential for transparency and for communication with reviewers, Panel members or requesting party.

Model implementation and model evaluation

This activity comprises of two elements: (a) model implementation, parameterisation and calibration; and (b) model analysis and evaluation.

Model implementation refers to the physical realisation of the technical model. This is a technical task to be completed by a modelling expert. Modelling experts will use a variety of methods to translate a particular conceptual model into a formal mathematical representation, which ranges from functional equations to one‐by‐one sequences of conditional rules implemented in computer software. A core element of the model implementation is the source code (in the form of a collection of files) that represents the ‘virtual’ model together with any specification choices made at this stage. To support transparency (see Section 2.3.4) and comprehensiveness, all model parameters, including their units, data source (e.g. own experiments, literature, expert estimations), and their variability/uncertainty should be documented. The technical component of model implementations includes qualitative and quantitative calibration of the model. All participants in the process need to be made aware that they can review any documentation associated with this step.

Model evaluation refers to a systematic technical analysis of an implemented model and is the subsidiary part of the model development process. During model evaluation, it is decided whether the model performs better applying different methods. This analysis does not target the resulting outcome of the modelling exercise but the detailed knowledge about the model and its properties. Accordingly, the analysis for a model evaluation can involve (Schmolke et al., 2010):

Verification: to test whether the model is working according to its specification;

Uncertainty analysis: to investigate the effects of lack of knowledge and other potential weak sources contributing to the model (e.g. the ‘uncertainty’ associated with model parameter values, or model structure), essential for the assessment of the reliability of the model‐based findings;

Validation: to compare model output or output of submodels with independent field data to strengthen confidence in usefulness of model for the specified problem;

Peer review: Review of the model (including questions, conceptual model and model evaluation) by a third party (not included in the modelling exercise) to increase confidence in model.

The documented outcome of the model evaluation activity provides the basis to reach acceptance of the implemented model by all participants

Model application and communication

Once a model is accepted by the participants in the modelling process, it will be applied to produce outputs relevant to the objective of the modelling exercise. This activity comprises of two elements: (a) model application; and (b) communication of model output.

The application of the model generates outputs that lead to findings and recommendations. The modelling expert has to develop an appropriate pathway to analyse the model. Choosing a method of analysis still allows for some flexibility (although narrowed down by the existing model) to account for available technical resources and the agreed time horizon. Together with generating model outputs, the analysis might include:

Sensitivity analysis: to measure the effect of changes in input values or assumptions on a model's output; when conducted in combination with uncertainty analysis, it allows a model user to be more informed about the confidence that can be placed in model‐based findings (Pascual et al., 2003);

Robustness analysis: to consider the impact on model output (e.g. estimates, scenario ranking, relative risk level) if changing certain structural aspects of the model;

Threshold analysis: to specify the range of uncertain assumptions or parameters to which critical findings like breakpoints, thresholds and other quantitative or qualitative statements do not alter;

Model Uncertainty analysis: Uncertainty about model structure can be quantified statistically, e.g. by model averaging, while other types of uncertainty about model structure must be assessed by expert judgement and taken into account when characterising the overall uncertainty. Judgements about model uncertainties should be expressed not as a probability that the model is correct, but as probability distributions or probability bounds for the difference between the model output and the real quantity it is intended to represent.

The communication of the output refers to the transparent and comprehensive explanation of model output based on the conceptual model and already demonstrated model properties. The communication enables the check whether initial questions could be answered. The communication facilitates the identification of findings and scientific conclusions as outcome of the modelling.

Model transparency

Model transparency means the comprehensive documentation and communication of all data, information, assumptions, methods, results, discussion and conclusions from the model used in the risk assessment and uncertainties. The transparency is essential because data are often uncertain or incomplete and, without full documentation, the distinction between facts and the modeller’s value judgements may blur (OIE, 2004, chapter 2.1).

The models used in assessments should be accompanied with documentation that

provides qualitative description of the model to readers who want to understand in general how the model works;

provides sufficient technical information to readers who want to evaluate the mathematical and programming details of the model, and possibly replicate it (Eddy et al., 2012).

Grimm et al. (2006) presented an ‘ODD (Overview – Design concepts – Details) protocol’ to structure the information of simulation models (individual‐based models or agent‐based models). First, it provides an ‘Overview’ on the purpose and main processes of the model. Second, in the ‘Design Concepts’ block, the general concepts underlying the model design are depicted and third, in the ‘Details’, all of the necessary information is given that would allow for a reimplementation of the model. The three blocks (ODD) are subdivided in seven elements: for the block ‘overview’, these are (i) purpose, (ii) state variables and scales and (iii) process overview and scheduling; for the block ‘Design concepts’, this element is (iv) design concepts and for block ‘Details’, the elements are (v) initialisation (vi) input and (vii) submodels (parts or modules).

Common practices for transparency focus on making model source code open access, documenting key equations and parameter values and providing data (‘FAIR data’). This ensures in the first place the reproducibility of results (DeCarolis et al., 2012, 2017). Bistline et al. (2020) mention that these first steps improve transparency for modellers and can help building confidence but are insufficient for achieving what might be called ‘deep transparency’, which requires making structural assumptions explicit, creating opportunities for interdisciplinary engagement and explicitly communicating value‐laden assumptions to stakeholders. To achieve this, they mainly recommend three major best practices:

Additional sensitivity analysis and model diagnostics including uncertainty. This analysis identifies the impact of model parameters and assumptions on the model outputs;

Better model documentation (beyond equations) to enable open communication across audiences and disciplines. This includes peer review of methods applied allowing periodic updates and eventually broader engagement (e.g. crowdsourcing and collaborative websites, e.g. Github), citations of sources and data posted to trusted repositories with guidance for use to non‐modellers and possible limitations, model code publicly available and extensively commented for other users and external verification of the code;

Greater discussion across modelling groups and other disciplines (e.g. by model inter‐comparison projects). The latter may include ethical implications and periodic reassessments with interdisciplinary teams, guaranteeing broader engagement.

Making models as transparent as possible should help to communicate insights to a broader range of stakeholders by appropriately explaining the strengths, limitations and assumptions to avoid misinterpreting results.

Procedural guidelines for appropriate integration of modelling into WG standard operating procedures

This chapter describes the recommended operating procedure for a WG in relation to the three stages of the modelling process. It reflects the activities that are particular for situations where the assessment requires the application of a modelling.

The roles of subject and modelling experts within the WG

The roles of WG experts in the context of the modelling process need to be clearly defined. WG members may have one or both of two roles with regard to their responsibilities, particularly concerning the type of contributions to the draft scientific output:

Subject experts provide the scientific foundation for the problem to be addressed and identify the relevant scientific literature and data sources. If technical modelling is advised, subject experts provide scientific guidance and information for the development of the technical model. Based on their subject knowledge, they support the modelling experts by putting the relevant information into a logical framework appropriate for addressing the mandate. This includes an identification of assumptions, limitations and uncertainties, such as expert opinion as expressed in a semi‐quantitative or quantitative manner. They should validate the output of models and derived findings in the light of their knowledge and experience. Subject experts will be responsible for ensuring state‐of‐art science with regard to the subject matter.

Modelling experts select an appropriate modelling approach, which requires an understanding of the subject issues (with the support from the subject experts). Modelling experts are responsible for communicating their requirements to the WG (data, expert opinion, working time). This must be done in a timely and transparent manner to allow model verification and validation before application of the model in the assessment process. Modelling experts will be responsible for ensuring state‐of‐the‐art with respect to the modelling techniques. Table 1 specifies responsibilities for each of these two roles.

Table 1

Responsibilities of the subject and modelling expert(s)

Responsibilities of the subject expert(s)	Responsibilities of the modelling expert(s)
Review the mandate and determine the required relevant scientific literature and data sources	Review the mandate and determine relevant needs for the type of model and its requirements
Contribute to laying out the conceptual model within the framework of the objectives of the ToRs and translating the ToRs into specific risk questions	Contribute to laying out the conceptual model within the framework of the objectives of the ToRs
Determine the scientific soundness of the proposed modelling approach(es) including assumptions and biological aspects	Determine the relevant modelling options, considering timeframe, data and available resources; propose a structured model plan with required data
Assess the reliability of the data sources and the quality of data	Assess the feasibility of the intended modelling
Assess the reliability of the practical model implementation	Provide proper implementation of the model tool
Assure transparency of the modelling process including reviewing the validity of assumptions, limitations and potential uncertainty together with the model experts	Assure transparency of the modelling process including documentation of the model that identifies to the subject experts the underlying assumptions, limitations and potential uncertainty of the expected model output
Provide scientific guidance and information for the justification of the model within the framework of the objectives of the TOR	Provide sufficient evaluation of the model to demonstrate consistency with scientific guidance and information (e.g. correctness and validity) within the framework of the objectives of the TOR
Assess the scientific validity of the model output	Apply the model to answer the objectives
Review and assess critically the practical relevance of the model outputs aiming to derive sound findings as outcome from the modelling	Present, explain and justify the model output, aiming for transparent communication of technical details and respond to questions from the subject experts
Draft model findings and conclusions for the scientific output	Draft final model description and model output for the model report
Being the advocate in promoting to the Panel members and others the scientific findings generated by the modelling process	Confirm the conclusions and recommendation of the model report derived from the model output

Operating Procedure

The recommended operating procedure for guiding a WG´s response to a mandate whose ToRs are being addressed using modelling has been formulated based on relevant EFSA standard operating procedures (SOPs) and working instructions (WINs) and the experiences of members of WGs of the AHAW Panel. In the following, the operating procedure is documented along the sequential stages of a WG’s work together with the required actions and expected outcomes. The stages are presented in Table 2. The operating procedure has been partitioned into four main phases to mirror EFSA’s SOP on generic mandates. Each of the phases comprises a series of steps. Moreover, within each step of the procedural sequence, the readers’ attention is called to the particular needs for appropriate integration of the modelling process (see Section 2.3.3). Agreements and final decisions are highlighted and involved participants and required communication between these is emphasised. Producing a decision tree to select a particular modelling approach, however, is not intended by the operating procedure (see Section 2.3).

Table 2

Risk Assessment activities and actors including specific tasks related to modelling

Generic Risk Assessment macrophases	Generic Risk Assessment steps	Actors‐Tasks	Outcome
Macrophase 1 Mandate Intake	1 Negotiation of the mandate	No action for the Panel Taken care of by EFSA staff	Not applicable
	2 Receipt of a mandate	EFSA assigns mandate to the AHAW Panel EFSA staff forwards mandate to the Panel	Panel informed of the new mandate for the AHAW Panel
	3 Chartering and acceptance	European Commission presents mandate to the Panel, and Panel and European Commission discuss/clarify mandate (including identification of background, objectives and questions) and the need for modelling at a plenary meeting of the Panel EFSA staff, WG chair and European Commission discuss the mandate at the kick‐off meeting with the European Commission	Mandate preliminarily clarified (TORs/goal/target/aim/problem/question understood) and accepted by the Panel with WG chair and other Panel members (including modelling advice) designated Mandate clarified (defined and accomplishable goals, purpose, question, expected answers and timelines agreed including deadlines) Potential strategic approaches described (including draft roadmap, potential models and their expected contribution, required and available resources, sources of information/data)
	4 Workforce Mix Definition	EFSA staff and Panel discuss whether the RA can be fully or partially outsourced to Art.36 Organization or other Tenderers/Grant beneficiaries EFSA staff and WG chair propose experts for WG EFSA staff invites selected experts	Decision on outsourcing of RA AHAW Panel comments on WG composition (by written procedure) WG established
	5 Definition of outsourced tasks	EFSA staff assures that appropriate contracts and/or grant agreements are put in place	Outsourcing contracts/agreements
Macrophase 2 Activities preliminary to Risk Assessment	6 Protocol development	EC/WG chair present the clarified mandate and its purpose to the WG EFSA staff and WG discuss strategic approaches to respond to the mandate (determination of specific risk assessment questions and expected answers in relation to timelines) EFSA staff and WG decide if a quantitative assessment is needed or not (if not, follow the procedure except for points related to modelling) EFSA staff and WG present, justify and discuss the proposed modelling approach with Panel EFSA staff and WG discuss and further develop modelling approach according the Panel's comments	Draft protocol with approach to respond to the TORs, including the modelling approach, draft work plan, task distribution and action plan proposed. Proposed protocol with modelling approach commented by Panel
	7 Protocol check (Tollgate 1)	EFSA staff and WG inform the Panel and European Commission about the draft protocol Panel and European Commission check if the protocol sufficiently describes the methodologies, the scientific (mathematical/statistical/computer) models and the required expertise, data and scientific clarity and completeness needed to reply to all relevant questions of the mandate in relation to the scientific value agreed with the requestor	Agreed protocol, including the modelling approach to be followed Tollgate passing recorded in case management tool
	8 Protocol approval	If a public consultation on the draft protocol was agreed during the mandate negotiation, the draft protocol is updated based on the comments received during the public consultation	Agreed protocol Tollgate passing recorded in case management tool
	9 Meetings	No action for Panel Taken care of by EFSA staff	not applicable
Macrophase 3 Risk Assessment	10 Preparation of first draft output	EFSA staff and WG collect data and expert opinion for the model, implement the model, discuss and revise the model report, inform the Panel on progress of the model report EFSA staff, WG demonstrate the model and its suitability (valid, representative, fit for purpose) EFSA staff, WG apply the model and communicate the model output	Applicable model and documentation and agreed draft model report Eventual feedback from the Panel on the model report and modelling follow up Agreement on the application of the presented model in contributing to the response to the mandate Model output as basis for findings Discussion of uncertainties as basis for transparency
	11 Draft output integration (Tollgate 2)	EFSA staff, WG, panel representatives and European Commission discuss the draft output, including model‐based findings	Interpretation of findings (limitations, assumptions and uncertainties) agreed Version of the draft output presented to the Panel and needs for further improvements agreed
	12 Draft output finalisation (Tollgate 3)	EFSA staff and WG revise the draft output based on the feedback received from Panel and European Commission EFSA staff and WG decide if draft output is ready for adoption	Revised draft output for possible adoption
	13/14/15 Endorsement/ adoption of scientific output	EFSA staff and WG chair present the model report and model outcome/derived findings to Panel and European Commission Panel adopts the scientific opinion based on the accepted model report/assessment and model	Adopted scientific opinion and accepted model report on modelling
Macrophase 4 Output publication & dissemination	16 Editorial checks and corrections	No action for Panel Taken care of by EFSA staff	Not applicable
	17/18/19 Publication of scientific output	No action for Panel Taken care of by EFSA staff	Not applicable
	20 Correction of published scientific output	No action for Panel Taken care of by EFSA staff	Not applicable

The four macrophases of the recommended operating procedure are:

Output publication and dissemination.

The procedure is the same for self‐tasks of the Panel and Commission mandates, although the involvement of the Commission in the former is informative and consultative in the latter.

Phase 1: Mandate intake

Within this first phase, the animal health‐related background for addressing a specific mandate is presented, discussed and clarified within the AHAW Panel. If applicable, possible relationships with existing legislation used by risk managers are considered.

If EFSA accepts the mandate, an appropriate chairperson is subsequently selected for the WG. Additionally, the Panel assures modelling advice as needed to assist the discussions in the consecutive steps, e.g. with the requesting party, tailored to clarify whether the achievement of certain objectives may be enhanced by a modelling study, and what expectations are realistic with regard to technical limitations or required resources.

Panel experts together with the AHAW secretariat start drafting potential strategic approaches. Each term of reference (ToR) should be explicitly addressed. The strategic approach should identify possible questions where a model should be used, specify required expertise, data and resources as well as timelines. The possibility to partially or fully outsource the risk assessment should also be discussed.

The working group members (experts) are invited based on the requirements identified in the draft assessment protocol. They review the mandate and the proposed strategy and define the roles and responsibilities of the respective WG members. If unclear or impractical details in the mandate are identified, clarification should be sought from the requestor. The WG develops a strategic work plan from the draft assessment protocol. It is important that the strategic work plan is concise, clear, target‐oriented and free from technical jargon which may compromise its readability.

If the strategic work plan incorporates the use of modelling, then the WG has to formulate the objectives of the modelling and develop a conceptual representation of the model (see Section 2.3.3.1) that allows the modelling expert to propose a modelling approach (Pascual et al., 2003). The conceptual model represents the structural relationships between all the knowledge the WG identifies relevant to answer those questions addressed by application of modelling. The conceptual model should be developed jointly by subject and model experts. The objectives and the respective conceptual model should be documented in the assessment protocol and the scientific assessment, e.g. using flowcharts.

At the end of phase 1, the mandate has been clarified and all partners should be in agreement on the interpretation of the mandate, the terms of reference (ToR), the scientific approach including, if applicable, the preferred modelling approach. The potential strategic approaches (defined and accomplishable goals, purpose, questions, expected answers and timelines, including deadlines, agreed with the requestor, draft roadmap, potential models and their expected contribution, required and available resources, sources of information/data), including the Panel’s decision whether or not a technical model will be used, have been described. This forms the basis for the development of the assessment protocol in phase 2. The WG with the agreed expertise composition has been established.

Phase 2: Activities preliminary to Risk Assessment

The requestor or the WG chair present the clarified mandate and its purpose to the WG and discuss the proposed strategic approaches to respond to the mandate. The WG discusses if a quantitative assessment is needed or not and drafts the assessment protocol that proposes a strategy and has reached consensus among the WG members. If a model in the technical sense is considered to address any of the ToRs, then the expected contribution(s) of this model to answer the ToRs should be documented. The resources in terms of time, data, methods and expertise, which are deemed necessary to implement the model, including potential outsourcing, should be specified. More details on how to develop the assessment protocol are described in EFSA’s framework for protocol development for EFSA’s scientific assessments (EFSA, 2020).

EFSA staff and the WG chair present, justify and discuss the draft assessment protocol and the proposed modelling approach with the Panel. The Panel comments on the proposed assessment protocol, including the proposed modelling approach and the WG further develops modelling approach according to the Panel's comments. If a public consultation on the draft protocol was agreed during the mandate negotiation, the draft protocol is updated based on the comments received during the public consultation.

The updated draft assessment protocol is reviewed by the Panel and European Commission to check if it sufficiently describes the methodologies, the scientific (conceptual/mathematical/statistical/computer) models and the required expertise, data and scientific clarity and completeness needed to reply to all relevant questions of the mandate in relation to the scientific value agreed with the requestor.

The deliverables for Phase 2 are an agreed assessment protocol that describes the approach to respond to the TORs, including the modelling approach, draft work plan, task distribution and action plan, the expected outcomes, potential limitations and resources (time, external support, etc.) required to complete the task. The responsibilities of individual experts have been accepted and all partners are committed to follow the approved assessment protocol and the strategic work plan, and further modifications are not expected.

Phase 3: Risk Assessment

In this phase, the assessment protocol including modelling is implemented by the WG, and results are generated. There is an ongoing communication between WG and the Panel members on the progress made, e.g. presentation and peer review of first results in the context of plenary Panel meetings. The review of the modelling may trigger iterative loops of data gathering and quality assessment. Draft model reports including the output of the models are made available to the Panel to identify gaps in understanding of the structure, output and interpretation of the model, and to make the process as transparent as possible. If possible, selected Panel members should act as scientific reviewers. The milestone of this phase is the generation of a model report prepared to be presented to the Panel.

If a technical model belongs to the work plan, the following steps related to the implementation, the evaluation and the application of the model (see Sections 2.3.3.2 and 2.3.3.3) should be taken:

Implementation of the model: This step refers to the technical realisation of the model that translates the conceptual model into mathematics or computer programming (see Section 2.3.3.2). To address a specific objective, usually several modelling techniques are applicable. The technical task of model implementation is likely to become the responsibility of modelling experts. Selection of the appropriate technical approach may depend on the particular data structure, accessible expertise, available timeframe, resources and desired precision of model output. The outcome of this task is a functioning technical tool. Model documentation should be produced in parallel with the model implementation, and a flowchart, formal or standardised model description (e.g. ODD protocol in 2.3.4) should accompany the model. The level of model documentation should enable subject experts to conjecture on the model outcome.

Demonstration of model suitability: This step refers to the model evaluation (see Section 2.3.3.2). The modelling expert has to evaluate the correctness of the model realisation (verification) and the adequate representation of the underlying conceptual model (validation) and the uncertainty in model behaviour arising from: (a) scenarios, (b) modelling technique and (c) data or parameters.

The outcome of this step is a comprehensive demonstration of the usefulness and trustworthiness of the model. The documentation of the evaluation effort should identify critical assumptions in model structure, uncertain parameters and secondary model outcomes that were used to validate the model. For transparency, the source for model inputs and assumptions should be clearly stated. During the demonstration, the WG and the Panel need to decide whether the model is fit for purpose, that the model represents the relevant scientific knowledge adequately, and what kind of model(s) has/have been used by relevant sources to address similar problems, what are similarities and differences compared with other models. The technical documentation of the model implementation and its evaluation steps should be provided by the modelling expert and allow for independent reproduction.

The evaluation of the model with successive communication and approval by the Panel should be performed before application of the model to the mandate’s question. The objective of this step is a second approval, which means that all partners are committed to the approved model and further modifications are not expected. The model is now ready for application to contribute to the WG’s response to the mandate.

Model output: As the final main task for the modelling, the tool is applied to the study question. The final model output is presented and explained to the WG. The WG will use the resulting model output to derive findings and recommendations. Again, the individual steps of the model analysis should be documented to allow repeatability of the investigations. The model output must be accompanied by estimates of the associated uncertainty and sensitivity of model outcomes in relation to uncertain model inputs or particular assumptions. WG, Panel and other partners might check whether the analysis provides a substantial knowledge gain concerning the system; whether the model output is consistent with expectations, and if not, whether deviations could be understood by the model assumptions.

The outcome of this phase will be a set of justified findings agreed between subject and modelling experts. The interpretation of the model‐derived findings is a matter of discussion and agreement between the WG, the Panel and other stakeholders, e.g. the requestor. Given agreement, the draft model report can be finalised for possible acceptance.

The model report should contain adequate documentation of the whole modelling, e.g. as an Annex. Standardised model documentation schemes recommend following the sequence of activities during the modelling process (see Section 2.3.3) when documenting for support in decision‐making:

Model objectives, assessment questions, definition of outputs;

Conceptual model including data presentation: crude data or references to data sources;

Standardised model description, including the theoretical and empirical basis e.g. computer code of the model, including design concept;

Assumptions regarding model inputs, ranges, distributions, other;

Discussion and comparison of alternative model formulations and justification for choices made about model structure;

Model verification, validation;

Scenarios presentation: e.g. in the context of risk assessment, one could include the temporal and spatial aspects of the exposure scenarios, the specific hazards addressed, exposed populations and exposure pathways;

Applied model analysis with uncertainty and, where needed/feasible, sensitivity analysis.

At a plenary meeting, EFSA staff and the WG chair present the draft output, including the model‐based findings and discuss it with Panel and European Commission representatives. This meeting is the final platform for constructive communication of the interpretation of findings between WG and Panel. The findings are often presented as conclusions. As the Panel has already approved the assessment protocol and work plan, including the model approach and the suitability of the implemented technical model based on the suitability demonstration and justified documentation, at this stage no discussion of the assessment protocol and the work plan or the way how answers to the mandate’s questions were achieved are expected. Insufficient transparency of the assessment protocol and working plan implementation at this stage is an indicator that the regular updates during Panel meetings need to be improved. The interpretation of findings (limitations, assumptions and uncertainties) and any needs for further improvements are agreed.

EFSA staff and the WG finalise the draft output based on the feedback received from Panel and European Commission and decide if the draft output is ready for adoption. The revised draft output is presented to the Panel for possible adoption. The model report is accepted by the AHAW Panel and subsequently the scientific opinion is adopted; the latter being the outcome for this phase and the completion of the task. If the model report is not accepted, the model report and model should be returned to the WG for further improvement. The revised model report and model need to be presented again at a later Panel plenary meeting.

Phase 4: Output publication and dissemination

In this phase, no specific model‐related tasks need to be carried out.

Conclusions

Models have the potential to play an important role when addressing mandates, given their role in representing multiple causal links within a system, structuring available information, examining hypotheses about potential causal processes, integrating appropriately available scientific evidence and explaining or predicting effects, e.g. to verify or quantify the effectiveness of control or prevention measures.

It is not appropriate to predetermine a decision flow for model type selection that can be applied to all scientific questions as this would never completely reflect the scope of existing models, not be agreeable to all modelling experts, and reduce the flexibility of future WGs during the selection of model(s) that would be fit for purpose in achieving the specific objective.

Agreement among participants about the objectives and conceptual model(s) underpinning the mandate will strengthen decisions as to whether a technical model tool will be needed during the scientific assessment.

There is a need for early and ongoing involvement and communication among all participants in the assessment process. A detailed draft assessment protocol and strategic working plan are integral to transparent communication.

There is a need for adequately harmonised terminology across scientific assessments.

Guidelines produced for modelling in relation to AH may be helpful to WGs considering animal welfare topics.

The operating procedure presented in this guidance document was designed so that it can provide guidance to others (outside AHAW) on the development, evaluation and application of models.

Recommendation

When addressing mandate questions, it is important that a decision is taken early regarding the use of modelling.

For each ToR, the use of modelling should be considered based on a conceptual representation.

The conceptual model for each ToR should be presented in the protocol to allow judgement, justification and peer review of any decision made with regard to modelling.

All participants in a mandate (WG, Panel and requesting party) should agree on the objectives and be actively and regularly involved in the development of the conceptual model, the strategic working plan and the derivation of findings (regardless of whether modelling is used or not).

Terminology should be harmonised to enhance transparency and clarity in assessments where modelling is used.

A glossary of modelling terms that is used for all scientific assessments using modelling should be established. A specific glossary section is required for each model report.

Modelling definitions should be consistent, both for communication within the AHAW Panel and more broadly. The adoption of these definitions should be explored with international organisations such as OIE and FAO.

For those mandates where modelling is used, all partners should contribute to the transparent assessment process, as proposed in the operating procedure.

All stages in the modelling process, apart from the technical implementation of model tools, should be dealt with by all participating expert(s). The technical implementation of model tools is taken care of solely by the modelling experts.

All partners should evaluate the output of models and derived findings in the light of their knowledge and experience.

Standard protocols for reporting should be developed and used when describing models, model outputs and related findings. These should reflect EFSA’s framework for protocol development.

Standard reporting protocols should be developed for the most commonly used modelling approaches in AH mandates to enhance preparedness, transparency and coherence across assessments.

The code of the model used should be published in open access platforms, e.g. Zenodo.

The proposed operating procedure should be considered in relation to other disciplines within EFSA.

Abbreviations

animal health

EFSA Panel on Animal Health and Welfare

Overview – Design concepts – Details protocol

Standard Operating Procedure

Terms of Reference

Working Group

Working Instruction

Appendix A – Model characterisation

A.1. Modelling techniques

Modelling techniques have defined characteristics. Examples are inclusion of randomness, mathematical formulation, representation of functional relations, distributional laws and inclusion of heterogeneities (entities, temporal, spatial). These basic features do not provide a unique hierarchy but help to identify the technical aspects of a model used (Hurd and Kaneene, 1993). None of these features alone can guide model selection in exclusively one way. Different features can be linked according to the problem the model should address. In Figure A.1, connecting lines show available examples that may combine the respective features within a particular model.

Figure A.1

Representation of various combinations of model characteristics

Sample paths (lines) represent existing models from epidemiological or ecological studies. Horizontally, different forms of the same model feature are shown, overlapping represents possible mixtures or hybrid models.

In the following text the model features are presented. It is obvious that all of them could be combined according to purpose, expertise, timeframe and resources available. The list is not comprehensive.

A.1.1. Variability expressed as randomness – Deterministic to Stochastic

Whether a model is classified as deterministic or stochastic depends on whether randomness is included in some way. Deterministic modelling is based on the assumptions that an average of some quantity (parameter describing a distribution or dynamic), is relevant and that uncertainty about the ‘average’ can be neglected. Most existing political legislations, at the end, reflect deterministic thoughts. Deterministic models require that model inputs are single‐valued, e.g. mean, median or percentiles of known distributions. In some applications, default values are used, which are either derived from data using standard methods or set by expert Panels. It has also been noted that the use of upper percentiles for a large number of independent model input values ends up in a scenario which has virtually zero probability to occur in the real world (Greiner et al., 2007). Based on deterministic models, different scenarios or concepts can be compared and a worst‐case scenario can be presented. Using ‘pessimistic’ values in deterministic assessments may intentionally or implicitly introduce an element of precaution.

If variability of any input quantity is expected to matter, the model will include some parameters or processes that are modelled in a stochastic way, e.g. by randomly drawing values from a probability distribution, leading to a stochastic (probabilistic) model. Accounting for variability and/or uncertainty may increase confidence in the model outcome. However, this gain of stochastic modelling is offset by less unique outcome that requires interpretation of the joint contribution of variability and uncertainty on the endpoint of the model. For example: the arithmetic mean of a data set with or without deviance descriptors (e.g. confidence limits); and correspondingly the epidemic SIR model with or without randomised infections of spatially distinct individuals that might soften the theoretically sharp threshold value for mass vaccinations (e.g. Anderson et al., 1981 vs. Eisinger and Thulke, 2008). In both examples the first provide the reader with a clear and decisive endpoint, but only the second allows judging the strength of the end‐point message and a plausible range of outcomes. The purpose of the modelling is the only way to decide which of both is relevant for the assessment.

A.1.2. Mathematical formulation – Analytic to Rule‐based

Mathematical formulation is the favourite feature for characterising models, but often inadequate as a good indicator for the right model type for a given problem.

Models can be formulated based on equations (e.g. SIR dynamic model with differential equations, or the normal distribution as probability model) or arbitrarily grainy (fuzzy) as sequence of logical rules (e.g. SIR dynamic model with explicit spatial movement of vehicles, daily decisions of farmers and infection depending on numbers of hosts and farm type; or using the empirical distribution as probability model). Examples from epidemiological literature can illustrate selection between the two characteristics (e.g. EFSA, 2009). Categorising mathematical formulation may be confused with the description of functional relationships that are ‘continuous versus discrete’ (see next point) or the topic of solving a model by an ‘analytic (closed) solution’ or ‘simulation‐based solution’ (see sub‐section Methods of model analysis).

Analytic approaches have the advantage of describing the model in a concise way (in mathematical language e.g. a set of equations). But this simplicity presumes mathematical soundness which means that assumptions have to be agreed and accepted prior to model building. In other words, the dynamics of analytically formulated models are imposed top‐down, for instance at the host population level. In the other approach, based on detailed and explicit rules, no assumptions on structural characteristics, functional shapes or characteristics of dynamics are required. In the extreme, everything will be progressively developed in the model. The dynamics of such rule‐based models are implemented bottom‐up, i.e. from the entities‟ behaviour. However, the communication and documentation of such detailed models require a considerable effort, although standard protocols start to emerge (Grimm et al., 2006). This usually requires an element of translation between the technical jargon used on either side.

Most analytic formulations have well understood properties or dynamic behaviour. Rule based model formulations have usually rather unknown (i.e. emergent) properties or dynamic behaviour. Here, specific methods for verification and validation need to be deployed (e.g. Wiegand et al., 2003; Grimm et al., 2005; Kramer‐Schadt et al., 2007).

Since more than one model type may be appropriate and different model types require different skills, experience has shown that the choice of the model depends strongly on the particular expertise of the modeller Often modellers impose their favourite modelling technique (Schmolke et al., 2010). It may be beneficial to elaborate those identities between different model realisations if the reconcilability and interpretation of given scientific literature is enhanced. Indeed, techniques do exist to align models from different mathematical formulations to each other, e.g. adding more differential equation to the system, or substituting sets of rules by an aggregated description and thus allowing a certain comparison or transition between analytical and rule based models (Levin and Durrett, 1996; Bolker and Pacala, 1997, 1999; Wilson, 1998; Fahse et al., 1998; Grünbaum, 1998; Bolker et al., 2000; Dieckmann and Law, 2000; Sato and Iwasa, 2000; Picard and Franc, 2001; Law et al., 2003).

A.1.3. Representation of functional relations and/or distributional laws ‐ Continuous to Discrete

The aspect is used to identify how model entities (Grimm et al., 2006), like random variates or their relation to each other are represented. Model entities can be represented continuous or discrete, for example time scale, spatial structure, host population, transmission factors and many others. This model aspect is illustrated by probability distributions of random variables in a model (e.g. birth) with the examples of the normal and the binomial model – the first being continuous and the second discrete. Models also vary according to the use of continuous and discrete functions used to represent outcomes of real‐world processes (e.g. seasonal birth distribution). The choice of distribution in models often follows the nature of the input data which is being modelled (e.g. count data vs. measurement data) and the corresponding sample statistics.

A.1.4. Inclusion of heterogeneous entities in dynamic models – population, time, space

Stratification or factor levels in models are used to describe different states of a model entity, for instance all hosts are represented as one population number, several herds or numerous individuals (e.g. population dynamics model or individual‐based model). If in epidemiological models, susceptible sub‐population or intervention measures change with time the model can incorporate heterogeneous time (e.g. continuous or discrete changes). Studies that take into account spatial patterns as having an impact on for example disease spreading will represent heterogeneous space (e.g. geographic location of herds, habitat maps, patchy or regular grids with units representing changes in space or connectivity among farms by vehicles). The inclusion of these heterogeneous entities characterises a model and all combinations of heterogeneity in spatial, temporal and population units might be used in epidemiological modelling. However, a decision whether or not explicit heterogeneity will matter for the problem at hand has to be made with respect to the particular objective of the modelling.

Heterogeneity might be assessed in the context of the process generating the data. But if reasons for heterogeneity in data are known, this should be reflected by a stratified analysis or incorporation of factor levels.

A.2. Methods of model analysis

The way a model’s output is calculated can be classed at least by two main types (Hurd and Kaneene, 1993): the analytical solution versus simulation‐based solution (Fine, 1982). Related concepts feature either or both of these principal approaches. Stepwise analysis may for example use different techniques or even combine their application, e.g. if a model is analysed with simulations but stochastically formulated then techniques of statistics are applied to explain model output.

A.2.1. Analytical analysis

Analytical analysis depends on mathematical manipulation to explore the relationship between different (dynamic) variables. Ideally, a solution is sought to describe the state of these variables at equilibrium (Hurd and Kaneene, 1993). Most classic epidemiological models employ this approach, which requires sound mathematical skills (Bailey, 1982). The great advantages are the ease of presenting closed‐form solutions (e.g. comprised in some functional expressions) and the rigor of evaluation. Closed expressions can lead to the identification of functional relationships among model parameters, which may generate new insights. Examples include the relationship between prevalence and incidence or the calculation of the basic reproduction number R0 from the parameters of a SIR model.

A.2.2. Numerical analysis

The numerical analysis concept incorporates algorithmic procedures combined with an analytic solution (today often computer‐based routines). Examples include the Newton‐Raphson (NR) or expectation‐maximisation (EM) algorithms for calibrating model parameters in generalised linear models through estimation (GLM). The importance of these algorithms in statistics arises from their use in maximum‐likelihood (ML) estimation, when the likelihood function and its derivative cannot be written in closed form. The outcome is not necessarily of a closed form – that is a set of equations ‐ and may be presented for instance as a graphic solution. The choice of numerical techniques depends on the complexity of the problem. NR and EM algorithms, for example, depend among others on starting values. The latter is especially an issue if the target function is a function in several model parameters and local maxima (or minima) exist. So‐called life‐science algorithms such as simulated annealing or genetic algorithms usually overcome the problem of multidimensional, multimodal target functions at the cost of long run‐time. All these algorithms are examples of optimisation algorithms and therefore require an objective or target function. For statistical estimation, this is a function in the parameters given the data. In other applications (e.g. optimisation of surveillance sampling), the objective function may reflect utilities such as testing costs, risks and benefits of testing.

A.2.3. Statistical analysis

By statistical analysis, the formulated model is fitted (parameters, factors or distributions) to optimise representation of the data. In that sense parameter estimation in statistics might be seen as model calibration using data. The accuracy of model fitting (e.g. in terms of maximum R2 or AIC value) might be seek with patterns observed on different data (e.g. temporal, spatial), different scales (herd and country or different hierarchical levels (population and individual). Ideally, different patterns are simultaneously considered as quality of model accuracy (Grimm et al., 2005).

A.2.3.1. Frequentist vs. Bayesian analysis

According to the classical, ‘frequentist’ approach in statistical inference, all evidence about a parameter is derived from the data used in the estimation. Common estimation methods in this framework include maximum‐likelihood (ML), least‐squares, method‐of‐moments, minimum chi‐square to mention a few. Bayesian methods, on the other hand, have in common with ML technique, that all information about a parameter is extracted from the data in form of the so‐called likelihood. The latter expresses the probability of the data given a set of model parameter(s).

A particular feature of Bayesian analysis is that the unknown parameter is interpreted as a random quantity and that all existing (or non‐existing) knowledge about this quantity can be expressed in terms of a probability density. Therefore, in the Bayesian framework, prior knowledge (the priors) exists about the parameter(s) of interest. Choosing ‘non‐informative’ or ‘flat’ priors allows expressing that virtually no prior information exists. For example, the non‐informative prior distribution of a prevalence parameter p may just state that the minimum is 0 and the maximum is 1 and that every value between and including these limits has the same chance to be correct. A statistical distribution reflecting this flat prior is the beta(1,1), which is also equivalent with a continuous uniform distribution U (0,1). In the Bayesian framework, prior distributions may be subjective or objective (empirical). An example for the first is to choose a beta prior based on expert opinion (e.g. ‘The most plausible value for p is 0.1 and I am 95% sure that p is less than 0.2’). An example for an empirical prior distribution based on count data such as ‘k out of n were positive’ is a beta (k + 1, n – k + 1). It is important to ensure that the prior is in fact independent of the data.

Bayesian estimation is a process of updating the prior using the likelihood. This process may be simple or computationally complex. In the example given above, the beta prior is said to be conjugate to the binomial likelihood of the data. Therefore, and due to the additive properties of the beta distribution, beta (k + 1, n – k + 1) can also be interpreted as posterior of the likelihood L (k, n) and the beta (1, 1) prior. In other cases, the update requires iterative algorithms as those implemented in Markov‐chain Monte Carlo (MCMC) methods (see Section A.2.6).

The advantage of frequentist methods is that the potentially controversy about valid prior information is avoided. ML estimators have favourable statistical properties, especially so for large data sets. The advantage of Bayesian methods is that they are applicable even for sparse data, that prior information can be combined with new study data and complex models or estimation of parameters with unknown sampling distributions can be more easily implemented. An additional advantage is the availability of the posterior (non‐parametric) distribution of parameter estimates for further analysis.

A.2.4. Simulation‐based analysis

Simulation‐based analysis refers to the procedural solving of models by iterative evaluation of scenarios. Particular difference to numerical approaches is that the solution must not even have a form of a function e.g. when studying several management scenarios for the eradication of a disease. The technique might be used to derive a proxy solution for models that were formulated analytically (see Section A.1.2) if temporal and/or spatial dimension are made discrete by intervals or spatial segments. The technique may also be helpful for stochastic models. Then multiple simulations (repetitions) of the same model (constant parameters) accumulate to a frequency distribution as solution (this is not equivalent to Monte‐Carlo simulations).

A.2.5. Analysis using graph theory

The method of analysis explores model properties using descriptors from graph theory (degree, betweenness and centrality). Recently, this analysis is applied to solve network models that reflect e.g. transport structures in animal industry or large‐scale networks of transport and trade. The analysis enables the identification of specific structures like central nodes that are high risk with regard to disease spread. This will be relevant to perform exposure assessments in compartmentalised trading systems or to assess consequences after incursion of a disease.

A.2.6. Monte‐Carlo analysis

Monte‐Carlo analysis focuses on the explicit representation of uncertainties within the model outcome and relies on random sampling from distributions describing the ranges of these uncertainties. The technique is often applied to analyse models based on simulations. Usually, the solution is derived from multiple analyses of the model with randomised sets of parameter values reflecting the associated uncertainties. Hence with Monte‐Carlo techniques, the analysis outcome is again a distribution. The resulting probability distributions are typically non‐parametric hence they cannot be described in analytically closed form. Rather the output is described using statistical techniques, e.g. based on moments and percentiles of the resulting distribution.

Monte‐Carlo techniques allow functions of random variables being evaluated without need for analytical convolution of probability density functions. Therefore, in probabilistic risk assessments, where highly non‐linear functions of random variables need to be considered as outcome functions such techniques are useful.

In Life‐sciences, the Monte‐Carlo approach is used for models constructed with great detail (e.g. agent‐based models). Latin‐Hypercube sampling (LHS) performed to sample parameter combinations for which the model is solved can be used in combination with a Monte‐Carlo approach. More technical, LHS enhances convergence to the specified sampling distributions by drawing values stratified for intervals with sampling weights proportional to the area over the respective intervals, e.g. simple random sampling might become cumbersome if lognormal distributions are involved – due to their heavy tail LHS will quicker represent extreme realisations and thus converge faster to the true distribution. Often the Monte‐Carlo concept is inconsistently called for if model outcome is compared for different sets of scenarios (as already discussed under Section 2.4). This, however, is rather a simulation‐based analysis as long as for each scenario a constant set of parameters applies.

Appendix B – Standard terminology for the use of mathematical and statistical models

Table B.1 Terms and definitions for ToR 1 on standard terminology

Accuracy	Generally, in the modelling context accuracy describes the degree of agreement between the observation and the model outcome, i.e. how close the model outcome is to the observed value (e.g. R2). In specific statistical models, accuracy of the model output in comparison to the observed data is often expressed as a summary value, and the objective in model fitting is to optimise this value. The smaller/larger it is, the higher accurate the model is depending on the particular summary value (e.g. Chi2, R2, AIC).
Assumption	Assumptions, or working hypotheses, are important components of many models. They can be defined as propositions taken for granted on which models may be based, and under which these models will give valid results. The validity of and therefore results from those models partly depend on the plausibility of such propositions. Assumptions often are found to be the most plausible, reliable, or suitable conditions (but often without formal proof).
Bayes’ Theorem	A theorem developed by Thomas Bayes that is the backbone for Bayesian Inference and thus a Bayesian framework. Bayes' Theorem is a simple mathematical formula used for calculating conditional probabilities. The Theorem relates the ‘direct’ probability of a hypothesis conditional on a given body of data, P E (H), to the ‘inverse’ probability of the data conditional on the hypothesis, P H (E): With that the most general formulation of Bayes' Theorem is provided by: P E (H) = [P(H)/P(E)] P H (E).
Bayesian framework	Within a Bayesian framework, Bayes' Theorem is used as a method to combine new evidence or observations (data) with prior (to data collection) probability of a certain condition or event into a new (posterior) probability for that condition/ event. This is to be contrasted to the frequentist framework in which the new probability of a condition or event is exclusively derived from the data (and the used model with inherent assumptions).
Classification of risks	The division of risk into classes according to specific criteria of both their probability to occur and their consequence. The classification will depend on the hazard, the risk assessment process as well as the risk management and communication needs.
Closed solving	Method of analysing a model resulting in a ‘closed solution’. A closed‐form solution solves a given model in terms of functions and mathematical operations.
Compartmental model	These models describe how individual units like animals or herds move between defined compartments (states) of a system on the basis of transition probabilities. One basic assumption is that all entities in a compartment are assumed in an identical status (homogeneous) with regard to the described dynamics.
Compound distribution	A secondary probability distribution specified by a first probability distribution in which one or more parameters that define this primary distribution are not fixed values but follow yet another (second) probability distribution. Compound distributions are sometimes used in stochastic models to describe specific probabilities. (Oxford Dictionary of statistical terms).
Conceptual model	Descriptive representation of a system based on current knowledge as well as on assumptions about its components, their inter‐relationships, and system boundaries. Conceptual models often are depicted by visual methods (diagrams) that exhibit assumed causal relationships. They form the basis for further modelling approaches.
Confidence Interval	A range of estimates, with a lower bound and an upper bound, statistically derived from a sample designed to include (capture) an unknown (true) population parameter with a certain level of confidence.
Continuous variable	Quantitative or metric variable measured on a continuous scale. It may take on any value within a given interval, and the meaning of unity does not change along the interval. The interval (valid data range) could be finite or infinite.
Covariate	Explanatory variables likely to affect the outcome variable of a model or the relationship between this outcome variable and other explanatory variables of primary interest of primary interest
Data‐driven model	Quantitative models where the relationships between the factors are directly determined/estimated from observed data. A simple example of a data‐driven model is a linear regression model. Coefficients of the regression equation are identified (‘trained’) based on the existing data.
Decision tree model	The model translation of a decision tree or risk pathway diagram. Usually applied as unidirectional evaluation of a sequence of alternative (stochastic) events that contribute to the final outcome of the tree (end‐point calculation).
Deterministic model	A model (or system) in which no random process is involved in the derivation of future states of the model. Deterministic models thus produce identical outputs (results) for a given unchanged set of input values (starting conditions). (Wikipedia)
Discrete variable	Quantitative or metric variable that takes on selected values (typically equally spaced) within an interval; the interval could be finite or infinite.
Dose‐response model	A dose‐response model describes the likelihood of a specified response resulting from exposure to a specified pathogen or hazard in a specified population, as a function of the dose. The result of such a model described the change in response with changing levels of dose (exposure).
Estimate	Expert knowledge: subjective indication of the value of a parameter based on the information available to the expert including his own ‘field experience’. Statistics: Calculation of the value of an unknown parameter based on observed data from a sample of individual units using statistical functions and assumptions.
Evidence	Includes specific information that is used to demonstrate the truth of an assertion or allow the estimation of a parameter. Scientific evidence is generated through population studies or observations or through experiments and is used to support or reject a hypothesis. Anecdotal evidence is derived from unsystematic individual (case) reports and is weaker than scientific evidence in supporting or rejecting hypotheses.
Expert opinion	Information on a specific question or the value of a parameter that was provided by one or more experts based on their personal experience, opinion and (often) assumptions. Expert opinion is important in areas where data is needed but not readily available through other sources.
Explanatory variable	Variable which seeks to predict or explain the outcome variable (also known as independent variables although they may not be independent of one another)
Exploratory data analysis (EDA)	Statistical techniques (mostly graphical) not based on prior assumptions on data structure describing the distribution of values within variables, and subsequently exploring relevant relationships between factors or differences between population groups of interest. EDA is frequently used to identify potential research questions.
Exposure assessment	The quantitative and qualitative evaluation of the likelihood of hazards occurring in a given population as a result of exposure.
Generic model	Generalised format of existing models not yet adapted to a specific hazard (e.g. individual pathogen, disease, population, or combination of all). They incorporate standardised relation types, together with the entities or objects that may be related.
Hazard characterisation	The qualitative and/or quantitative evaluation of the nature of the adverse effects associated with the hazard.
Hazard identification	The identification of any factor, from birth to end of life, capable of causing adverse effects on a studied subject / population.
Import risk assessment	Formal risk assessment to evaluate the probability of importing a specific hazard into a defined (animal) population or (geographic) region (to be checked with other risk assessment glossaries).
Individual‐based model	Model with individuals as basic entity. Individuals differ in their status and exchange information between them or with the environment (e.g. host animals, farms, and free rooming herds). In such models, the history of individually identified units (animals, people) is modelled and thus can be followed.
Input parameter (model)	A factor/ component in a model which is provided with a value/ specification at the beginning of the calculation process (output parameter).
Intermediate parameter value	Intermediate output of a stepwise (iterative) model analysis that is necessary for the next analysis step but is not a model result.
Knowledge‐driven model	Models where the system relationships, key parameters and their values are predominantly based on a synthesis of existing knowledge including published and unpublished data sources as well as expert opinion, but not from sample‐derived estimation (see ‘data driven models’).
Likelihood	Probability. In statistics often used in the context of estimation, e.g. the ‘maximum likelihood estimator’ as being the estimator of a certain value or model component which gives the highest probability (likelihood) to the observed data given the applied model.
Linear regression model	A regression model assuming a linear functional relationship between outcome and explanatory variables, i.e. assuming that there is a linear (straight line) relationship between those.
Logistic regression model	A regression model assuming a linear functional relationship between the logit (log odds) of an event probability (ln(p/(1‐p)) as outcome variable and the explanatory variables.
Mathematical model	Models that are formulated (can be written down) by mathematical language.
Meta‐analysis	A statistical analysis that combines the results of several independent studies that have addressed the same research question. As combination may increase statistical power of the estimation, results may be a more accurate reflection of the unknown property than those derived from a single study under one set of conditions.
Metapopulation model	The term metapopulation originates from ecology. A meta‐population consists of a group of spatially separated populations of the same species that interact at some level. A metapopulation model links multiple sub‐populations to represent spatial structures. Linkage of these population might be determined either explicit (e.g. landscape map) or implicit (e.g. intensity value of exchange).
Model	A (simplifying) representation of the essentials (parameters, relations, processes, or mechanisms) of an existing system (or a system to be constructed) which incorporates existing knowledge and/or assumptions about the relationship between all system components in an explicit form that can be investigated by systematic or manipulative experiments.
Model input	Any part of a model which is specified (e.g. by a value/ distribution/ functional relation / mechanistic rule) before model analysis (‐* model output).
Model output	General: All output that is generated by the analysis of a model (e.g. qualitative or quantitative values/distributions/proportions).
Model prediction	A process where models, based on specific input, are used to forecast (predict) results for yet unobserved (unobservable, new or future) situations.
Modelling approach	The methods used to construct, validate and analyse the model, including estimation techniques for the model analysis.
Monte Carlo simulation	Iterative technique applies in modelling (with Markov chain Monte Carlo or MCMC sampling as a common example) to estimate the range of possible output (i.e. a distribution) that involves repeatedly drawing random numbers from input (parameter) probability distributions. The technique usually is applied in stochastic models in which the exact parameterisation cannot be taken for granted (substantial uncertainty in input values).
Multivariable model	A model in which several explanatory (predictor/ risk factor) variables are assessed simultaneously for their relationship to a single outcome variable (univariate model), thereby allowing control for confounding relationships between the explanatory variables.
Multivariate model	A model in which one (univariable) or several (multivariable) explanatory (predictor/ risk factor) variables are assessed simultaneously for their relationship to two or more outcome variables; this relationship is often expressed in the form of matrices.
Outcome variable	The variable of primary importance in investigations since the major objective is usually to study the effects of treatment and/or other explanatory variables on this variable and to provide suitable models for the relationship between it and the explanatory variables
Output value	Qualitative or quantitative value of designated output parameters at the end of the model analysis.
Output parameter (model)	Factor / component in a model for which the final value is derived or estimated during the calculation process (as a function of the model structure and the model input). Consistent use only possible if the output structure is pre‐specified and has itself „parameters‟ to estimate/ evaluate (see ‘output value’).
Parameter	Numerical characteristic of a model element, system or function. Parameters can take a range of values from qualitative classes via single values to probability distributions, depending on their role in a model (‐, input, intermediate or output parameter).
Point estimate	The single‐valued result of the application of a point estimator to the data. In statistical models, this is often provided by the maximum likelihood estimation (MLE) of the (unknown) true population parameter. Point estimation usually is accompanied by its confidence interval, i.e. the calculation of an interval estimate from the same data.
Population dynamics model	A model that represents dynamic processes of a system on the level of population changes, i.e. proportions of populations or sub‐populations that change their ‘state’. From these models, population averages can be derived, but no individuals fate can be ‘simulated’ (see ‘individual based model’).
Prediction Interval	An interval estimate in which future observations will fall, with a certain probability (e.g. 95%), given what has already been observed. Prediction intervals are often used in regression analysis.
Predictive model	see ‘Model prediction’
Probability distribution	A model of occurrence of possible values (probabilities) of a random variable. There are theoretic probability distributions with defined shape (e.g. normal, exponential, binomial) and empirical distributions reflecting raw data on occurrence that have no defined shape.
p‐value	The probability that a sample characteristic or model output (e.g. difference between mean of two groups) might have been observed by chance, given that the null hypothesis (of no difference) is true in the population from which the sample was drawn. The p‐value can range from 0 to 1. By specifying a threshold level of significance (often 0.05), sample characteristics (difference between means) are judged statistically significant (‘not plausible by chance’) if the p‐value is smaller than the threshold.
Qualitative Risk Assessment	An assessment that generates an estimate of categorical nature or based on an ordinal scoring system. The outcome of such an assessment is a classification of output into descriptive categories.
Quantitative Risk Assessment	An assessment that generates an estimate of a numerical nature directly tied to a measurement or calculation. Depending on the type of model tool used, an indication of the associated uncertainties ‐ expressed either as extreme values, ‐, confidence intervals or ‐, prediction intervals are needed.
Regression model	A mathematical model that describes the relationship between an outcome variable (y) and one or more explanatory (predictor/risk factor) variables (x1, x2, x3...) using a specific functional form of the relation (e.g. ‐, linear, ‐, logistic, exponential).
Relative risk	The comparison of risk estimates from two samples or risk scenarios by dividing the two risks, i.e. expressing on risk as a relative value to the other (often denoted as baseline) risk value. Possible value range is 0 to infinity, with a relative risk of 1 indicating that the two compared risks were identical.
Risk	Epidemiology: Likelihood (probability) of a certain event (outcome) to occur in a cohort, where the event usually is considered ‘negative’. Risk assessment: A function of a probability of an adverse health effect and the health effect and the (negative) consequence, severity of that effect, consequential to a hazard. General: subjective summary for a hazard, its probability of occurrence and the severity of that effect, consequential to a hazard.
Risk Analysis	A formal process consisting of three components: risk assessment, risk management and risk communication.
Risk Assessment	A systematic approach to assess the effect of an exposure to a hazard/stressor. The approachformally includes hazard identification, characterisation and consequences assessment. These steps are implemented in the risk assessment model, and respective procedural guidelines are available.
Risk characterisation	Element of a ‐, Risk Assessment, determining/ describing the effect of a hazard qualitatively or quantitatively, including attendant uncertainties about the occurrence and severity of known or potential adverse effects on a given population.
Risk factor	A factor that influences the likelihood for the disease or health event to occur. Risk factors are often identified through epidemiological studies and related risk factor analyses (such as uni‐ and multivariable regression models).
Risk mapping	Diagrammed technique to prioritise risks according to frequency (alternatively likelihood) and severity (alternatively significance). For each risk, the severity is plotted on one axis and the frequency is plotted on the other axis. Geographical representation of spatial variation in risk.
Risk pathway	A (conceptual) representation that illustrates the sequential events of risks considered to be leading to the risk outcome. The risk pathway will serve as guidance for data collection, logical deductions, and any quantification required in the subsequent risk assessment e.g. using ‐, decision tree models.
Rule‐based model	Model that is constructed from simple and generic rules that reflect expert knowledge as close as possible. The method is purposeful if limitations due to structural assumptions have to be avoided and usually results in more complex models.
Scenario	A certain combination of input parameter values that is used in a specific model run. When there is uncertainty about the value of a specific input parameter, a range is considered (selected), and (randomly) chosen representatives are tested in separate model runs (‘scenarios’). Alternatively, different hypotheses about the modelled system (control options) might lead to specific parameterisation of the model, each reflecting a scenario.
Scenario analysis	Assessment of the model output depending on specified scenarios. Often the analysis includes at least a worst‐case scenario, i.e. with values selected for important input parameters that are assumed to (all) be at the maximum likely negative (adverse in the risk context) value.
SEIR model	Compartmental model that incorporating four possible ‘states’ (compartments) in which subjects can be found: S=susceptible, E=latently infected but not (yet) infectious; I= infectious; and R= recovered/immune.
Semi‐quantitative or qualitative risk scale	Within ‐, Risk Assessment, probabilities of an event are assessed and described textually on a scale from negligible, indicating that the probability of an event or the estimated risk cannot be differentiated from zero (and in practical terms can be ignored) to extremely high.
Sensitivity Analysis	A method to qualify the output of a model by measuring the variation in model outputs resulting from changes in inputs. Through this, the ‘sensitivity’ of a model to the respective changes can be assessed, and work can be focused onto those input parameters that have substantial impact on the model output. Testing changes in model output caused by changing certain structural aspects of the model usually may be referred to as Robustness Analysis.
Simulation model	A model that is evaluated via explicit (e.g. step‐by‐step) simulation of the implemented structural processes and their interactions. Simulation as method of model analysis/model solution allows arbitrary complexity of the model. Alternatively: a mathematical representation of the essential characteristics of a real‐world system or situation, which can be used to predict future behaviour under a variety of different conditions. The process of developing a simulation model involves defining the situation or system to be analysed, identifying the associated variables, and describing the relationships between them as accurately as possible. Alternatively: Simulation is a computerised (iterative) approach to derive solutions (often in form of outcome distributions) for models that either do not have a closed mathematical solution or in which uncertainty in the input parameter values needs to be accounted for.
SIR model	Compartmental model that incorporating three possible ‘states’ (compartments) in which subjects can be found: S = susceptible, I = infectious; and R = recovered/immune.
Spatial model	Model that explicitly or implicitly incorporates the effect of spatial heterogeneity, i.e. spatial differences in either population density, outcome‐related (risk) factors or both.
Statistical analysis	Any method applied to explore, describe or model the information contained in a given set of data, in most instances samples derived from larger populations, to make inferences from that sample to specific (source) population parameters.
Statistical significance	The a‐priori fixed (threshold) level of maximum error probability (alpha, type 1 error) that one accepts when concluding – based on the results of a statistical test – that the alternative hypothesis (inequality) is correct. Depending on the nature of topic maximum error probability of alpha = 0.05, 0.01, or even less are used in statistical hypothesis testing, i.e. ‐, p‐value.
Stochastic model	A model in which randomness is involved in the derivation of future states of the model. Stochastic models thus produce distributions as output even for a given starting condition. Randomness might be incorporated via stochastic parameterisation, i.e. accounting for variability and uncertainty of event occurrence.
Systematic Literature review	Conducting a literature review using predefined criteria for searching/selection of the relevant literature with scientific tools to assess the findings from the published studies in a transparent and reproducible way.
Transmission model	Specific models in which pathways describing the transmission of (infectious) diseases/agents in populations are constructed, and values for the transmission probabilities along that pathway either entered (to simulate disease spread) or estimated based on observed population data.
Uncertainty (statistical)	Lack of knowledge in the exact value of a population parameter. Statistic methods derive estimates for that parameter as well as the associated uncertainty using fundamental concepts and theories of sampling, probability and randomness. In models uncertainty can be incorporated by probability distributions with information coming from either data or expert opinion.
Uncertainty analysis	Uncertainty analysis is defined as the process of identifying and characterising uncertainty about questions of interest and/or quantities of interest in a scientific assessment.
Univariable model	A model in which a single explanatory (predictor/ risk factor) variable is assessed for its relationship to one or more outcome variables.
Univariate model	A model in which one or more explanatory (predictor/ risk factor) variables are assessed for their relationship to a single outcome variable.
Validation	The concept of checking the validity of the model formulation with regard to its intended purpose; ideally done with independently observed patterns. Checking correctness is intended task of model ‐, verification.
Variability (biological)	True (inherent) biological, measurement or system‐based variation in the possible values (value range) for a given parameter. In models, that variability, similarly to what is done with uncertainty, can be incorporated as probability distributions with information coming from either data (and classic statistics) or expert opinion
Verification	The concept of checking the correctness of the model implementation; ideally done by measuring back any input pattern, code review, implausible scenarios (e.g. assuming no effect of a proven treatment). Checking appropriateness for purpose and consistency with conceptual thinking is the intended task of model ‐, validation.
Worst‐case scenario	A situation where everything that can go wrong, does go wrong. Used in risk assessment to consider the worst predictable outcome by using extreme (risk increasing) model inputs.