Rapid assessment of the risk of SARS-CoV-2 importation: case study and lessons learned.

During the early stages of an emerging disease outbreak, governments are required to make critical decisions on how to respond, despite limited data being available to inform these decisions. Analytical risk assessment is a valuable approach to guide decision-making on travel restrictions and border measures during the early phase of an outbreak. Here we describe a rapid risk assessment framework that was developed in February 2020 to support time-critical decisions on the risk of SARS-CoV-2 importation into Australia. We briefly describe the context in which our framework was developed, the framework itself, and provide an example of the type of decision support provided to the Australian government. We then report a critical evaluation of the modelling choices made in February 2020, assessing the impact of our assumptions on estimated rates of importation, and provide a summary of "lessons learned". The framework presented and evaluated here provides a flexible approach to rapid assessment of importation risk, of relevance to current and future pandemic scenarios.

Introduction

When a novel pathogen with pandemic potential emerges, governments need to make critical decisions about how to respond, despite limited data being available to inform these decisions. For countries in which the pathogen is not yet present, preventing or delaying importation and local transmission can buy valuable time for health authorities to establish response measures.

Key decisions that must be made in the context of disease importation include: the level of screening that should be applied to incoming travellers; quarantine arrangements for incoming travellers, including self-isolation advice and managed quarantine; epidemiological case definitions, which can include recent travel in a list of affected countries, and can determine how potentially scarce testing resources are used; and finally, the possibility of restricting travel. Each of these may be applied in a targeted fashion based on the country in which travel originated.

For a previously uncharacterized pathogen, these decisions are hampered by at least two key sources of uncertainty. The first is the natural history of a novel pathogen, which may be poorly understood, including factors such as the duration of latent and infectious periods, and the relationship between infectiousness and symptoms. The second is the effectiveness and capacity of health and political systems in different countries, in particular with respect to their ability to detect imported and locally acquired infections, and to report identified cases in an accurate and timely manner.

Given these uncertainties, analytical risk assessment is a valuable approach to guide decision-making on travel restrictions and border measures during the early phase of an outbreak. While multiple studies have described mathematical and statistical tools developed to estimate importation risk (Chinazzi et al., 2020, Lai et al., 2020, Gilbert et al., 2020), critical evaluation of such tools is less common. However, such evaluation is beneficial as it can review and assess the design decisions that were made at the time, and provide useful guidance for future practitioners (Kerkhove and Ferguson, 2012, Chowell et al., 2017).

In this paper we describe a rapid risk assessment framework that was developed in February 2020 to support time-critical decisions on the risk of SARS-CoV-2 importation into Australia. In Section 2 we briefly describe the context in which our framework was developed, the framework itself, and provide an example of the type of decision support provided to the Australian government, here framed in terms of estimated rates of importation into Australia.1

Due to limitations of both development time and data availability, our framework necessarily made a number of simplifying assumptions about pathogen characteristics, global epidemiology, and health system capacity. In Section 3, we revisit these assumptions to evaluate their impact on our estimates of importation rate. This study thus makes two contributions: first, it presents a flexible approach to rapid risk assessment, specifically designed with communication in mind; second, it provides an honest appraisal of the strengths and limitations of the proposed approach, based on our subsequent critical evaluation.

Case study — arrival of SARS-CoV-2 in Australia

Context

On 29 December 2019, Chinese authorities reported a cluster of cases of atypical pneumonia in the city of Wuhan, Hubei Province, later identified to be caused by a novel coronavirus, SARS-CoV-2 (World Health Organization, 2020a). The disease caused by this virus is now known as Coronavirus disease 2019 or COVID-19 (World Health Organization, 2020b). By 31 January 2020, 9720 cases of COVID-19 had been reported in mainland China. A further 120 cases had been confirmed outside of China, including 95 cases in countries of the South East Asia and Western Pacific regions (World Health Organization, 2020c). Australia had detected and managed 9 imported cases, all with recent travel history from, or a direct epidemiological link to, Wuhan (Australian Government Department of Health, 2020a, World Health Organization, 2020c).

On 1 February 2020, Australian authorities placed restrictions on all travel to Australia from mainland China, to reduce the risk of importation of the virus. Only Australian citizens and residents (and their dependants) were permitted to travel from China to Australia (Australian Government Department of Health, 2020b). At that time, models estimated that 75,815 individuals (95% CrI 37,304–130,330) had been infected in Greater Wuhan up to 25 January (with WHO reporting 1320 confirmed cases for the same time period) and projected that the epidemic could peak in Wuhan as early as late February (Wu et al., 2020). Before the restrictions, Australia was expecting to receive approximately 200,000 air passengers from mainland China during February 2020 (Australian Bureau of Statistics, 2019a). Travel numbers fell dramatically following the imposed travel restrictions.

With air-travel from China restricted and strict quarantine measures in place for those who did return, a remaining concern for Australia was the epidemic status of other countries in the Asia-Pacific region with large, and at that stage unrestricted, travel volumes to Australia. We therefore developed a risk analysis framework to estimate importation risk to Australia from countries other than China.

Rapid risk assessment framework

We introduce a modular framework for assessing the risk of SARS-CoV-2 being imported from a source country (here China) to a country of interest (here Australia) via other intermediary countries in the region. The framework was developed very rapidly to provide a rational basis for decision-making on border measures and case definitions in Australia at a time when global transmission of SARS-CoV-2 was not yet established. In developing this framework, we focused on the risk of (potentially undetected) spread to countries in the South East Asia and Western Pacific regions because they are highly connected to both China and Australia, relative to the rest of the world. Our analysis takes into account the COVID-19 epidemiological situation and mobility restrictions imposed as of mid-February but is adaptable to other contexts for future outbreak response.

The framework includes a series of analyses based on epidemiological evidence at the time, patterns of air travel, and model components described by De Salazar and colleagues (De Salazar et al., 2020) and developed by the authors. Our framework considered a single point of origin for all exported infections, as China was the only country reporting uncontained transmission at the time of analysis. Each step of the analysis is outlined in Fig. 1 and described in more detail below.

Fig. 1

Key quantity estimated at each step of the full analysis of SARS-CoV-2 importation risk to Australia (country of interest) via countries in the South East Asia and Western Pacific regions (intermediary countries), given travel restrictions from mainland China (source country) (top). A brief indication of the relevant data and/or method used at each step (bottom).

Step 1: Importation risk from China to intermediary countries

SARS-CoV-2 first emerged in China (World Health Organization, 2020a) and hence the risk of importation for countries in the South East Asia and Pacific regions, in the early stages of the outbreak, was primarily dependent on travel from China. The expected numbers of imported cases in each intermediary country was estimated using an approach proposed by De Salazar and colleagues based on air travel volume estimates from China since COVID-19 emergence (De Salazar et al., 2020). Their model estimates the expected number of imported cases in countries by regressing the number of imported cases reported by each country against their relative incoming travel volumes from China (under unrestricted travel). They assume that the expected case count would be linearly proportional to air travel volume. Bootstrap sampling was used to estimate 95% prediction intervals. We fitted the model to reported cumulative case counts for each country extracted from WHO situation reports 2, 9, 16, 23, and 30 (i.e., one per week from 22 January to 19 February 2020).

Step 2: Number of potentially undetected cases in intermediary countries

The number of potentially undetected introductions in each intermediary country was based on the discrepancy between expected (Step 1) and reported cases (noting that cases due to local transmission were excluded from these counts). Under the assumption that all reported cases were effectively isolated (24 h after symptom onset) with reduced risk to onward transmission, the difference between the expected and reported numbers of cases per country provided a crude estimate of the number of unreported cases. We assumed unreported cases were undetected and therefore more likely to contribute to local transmission and potentially a large outbreak.

Step 3: Probability of an outbreak in intermediary countries

A branching process model was used to generate stochastic projections of the initial stages of an outbreak for each intermediary country. This model incorporates country-specific rates of SARS-CoV-2 importation (as estimated in Step 1) and country-specific detection probabilities based on the ratio of reported cases to expected cases (with a maximum detection probability of 1). We assumed an of 2.68 (within the range estimated for SARS-CoV-2 in Wuhan in early January Wu et al., 2020), no individual-level variation in transmission and independence of all undetected introductions (Chinazzi et al., 2020). The probability of local transmission was defined as the proportion of simulations with no locally transmitted cases five weeks after simulation commenced (i.e., 26 February).

The model assumes individuals can be either Exposed (), Infectious (), Recovered (), or isolated (). The exposed and infectious classes are split into two compartments (giving them an Phase-type distribution) and symptom onset is taken to correspond to the transition between exposed and infectious states (see Fig. 2).

Fig. 2

The Susceptible, Exposed, Infected, Recovered model with isolation and external importation of cases. The average duration of time spent in each state is indicated over the circles. The class represents isolated cases that can no longer transmit.

Imported cases are assumed to be in the first exposed class. At the time on symptom onset there is a probability, , that the individual is detected (either from direct contact tracing efforts or enhanced case finding) and hence with probability , the case is missed. This probability depends on the current public health system workload and hence time. Once the case is detected there is a 24-h period before they are completely isolated during which it is possible for them to infect others.

The probability of detection is related to the workload, , by where is the baseline probability of detection and is the workload capacity. So while the workload is less than the capacity, the probability of detection remains high, but decreases once the capacity is exceeded.

For each country, the import rate is calculated as the weighted mean of daily expected cases, with weights set according to the time intervals between data points. The initial detection probability () of each country was calculated as the mean of the ratio of cumulative reported cases to the cumulative expected cases starting from 29 January. The maximum value of was 1.0. The public health system workload capacity , used for updating the detection probability, was conservatively set to 10 detected infections per day.

Step 4: Estimated size of an uncontained outbreak in intermediary countries

The stochastic transmission model described in Step 3 was also used to estimate the likely number of locally transmitted cases in each intermediary country, conditional on local transmission occurring. This model assumed no public health intervention (beyond case isolation), and that both importation rate and detection probability were constant over time. The transmission model was run from 22 January, with epidemic curves (separated by imports and local transmission) projected forward by one week beyond the last data collection date (19 February).

Step 5: Importation risk from intermediary countries to a country of interest

The likelihood that a passenger arriving in Australia from a country in the South East Asia or Western Pacific region would be infected by SARS-CoV-2 was then estimated based on the likely source prevalence in a particular country and the travel volume from that country to Australia. Samples of the estimated epidemic state of each intermediary country (from Step 4) were used to set a rate of importation of exposed individuals, where the rate of importation is the product of the number of exposed individuals in the country and the rate of travel from the country, divided by the country population size. We assumed that prevalence in each country was growing exponentially with a doubling time of seven days; hence imports from each country grew exponentially. Accordingly, we modelled the importation times into Australia as an inhomogeneous Poisson process. The doubling time of seven days was consistent with early estimates (Wu et al., 2020, Li et al.). We then computed the expected number of imported infections per day into Australia, and here we report the date that this number exceeded a seven-day moving average of five per day.

Travel volumes from countries within the South East Asia and Western Pacific regions to Australia were extracted from the Australian Bureau of Statistics (Australian Bureau of Statistics, 2019a, Australian Bureau of Statistics, 2019b).

Risk assessment outputs

We presented the outputs from our framework as a ranked table of aggregate risk of SARS-CoV-2 infections being imported into the country of interest (Australia), based on sustained transmission occurring in the source country (China) by originating country of travel (Fig. 3). The table includes key quantities estimated at Steps 1 to 5 of the analysis outlined in the methods and depicted in Fig. 1. The potential number of undetected imported cases (as of 19 February 2020) in each country of the South East Asia and Western Pacific regions (intermediary countries) is displayed in Fig. 4.

Fig. 3

Summary table of indicators for assessing importation risk of SARS-CoV-2 to Australia from locations in the South East Asia and Western Pacific regions. Analysis as of 19 February 2020. At this time, the only country reporting uncontained/sustained transmission was China and restrictions to travellers from mainland China to Australia were imposed. Relative risks are indicated by colour shading, with red representing the highest risk and green the lowest. Note that for efficiency, this rapid analysis used the limited case data provided in World Health Organization situation reports. We did not make use of more detailed local data available from individual country health agencies, and therefore these analyses may not have reflected the epidemiological situation in each location at the time. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 4

Estimated number of undetected imported cases and 95% prediction intervals for each location in the South East Asia and Western Pacific regions as of 19 February 2020, based on the method described in De Salazar and colleagues (De Salazar et al., 2020). The De Salazar model evaluates detection capacity against an average case-detection capacity across countries. It is therefore possible for the estimated number of undetected imports in settings with strong surveillance systems to be zero with high confidence (as in Hong Kong and Singapore). This limitation was communicated to decision-makers at the time.

Projected epidemic curves for selected countries are shown in Fig. 5, with reported imported and local cases also shown for context. All four countries shown have high levels of expected imported cases (Fig. 3). Thailand and Indonesia have fewer reported cases than expected, and hence a higher estimated number of undetected cases (Fig. 4), leading to projection of considerable undetected local transmission. In contrast, Malaysia and Singapore both reported a number of cases equal to or greater than expected, and hence have a lower estimated number of undetected cases, leading to projection of more modest levels of local transmission. Note that the projected levels of local transmission in Singapore are lower than the actual reported local cases. These known local cases were not further incorporated into our analysis, but rather signalled to decision makers that the likelihood of further undetected transmission was relatively low. Furthermore, for efficiency, our rapid analysis used the limited case data provided in World Health Organization situation reports. We did not make use of more detailed local data available from individual country health agencies and therefore our analyses may not have accurately reflected the epidemiological situation in each location at the time.

Fig. 5

Imported cases (left panels) and epidemic curves (right panels) from 22 January using WHO data up to 24 February, and projecting forward to 4 March, for selected countries. Lines and shaded regions in each panel show median and 95% quantiles for the estimated cumulative number of cases (imported or locally transmitted). Black points show cumulative imported cases (left panels) and cumulative local cases (right panels) for each country, as reported by the World Health Organization (WHO). In the right hand panels, reported local cases (black dots) are shown for additional context, they are not incorporated into our analysis. The epidemic curves represent an estimate of the likely size of the local epidemic and not the estimated number of detected cases. Our estimates of outbreak size are informed by the number of undetected infections as estimated by the De Salazar model, which evaluates detection capacity against an average case-detection capacity across countries (De Salazar et al., 2020). It is therefore possible for projected levels of local transmission in settings with strong surveillance systems to be lower than the actual reported local cases (as in Singapore). This limitation was communicated to decision-makers at the time.

Critical evaluation

The outputs from our risk assessment framework contributed to an evidence base for decisions on COVID-19 border measures and case definitions in Australia up until late February. These outputs were used alongside other evidence streams, including dedicated reporting on the epidemiological and health system statuses of other countries. During the month of February 2020, with travel restricted from mainland China, extensive testing informed by an evolving case definition, and case targeted interventions (case isolation and contact tracing) in place, Australia detected and managed only 17 cases of COVID-19.

By early March, it was apparent that widespread transmission of SARS-CoV-2 was occurring in many countries outside of China, and regions beyond Asia and the Western Pacific. As more data became available on observed outbreaks, our estimates of potential undetected outbreaks in intermediary countries became less relevant to the decision-making process. In early March, Australia imposed travel restrictions on three countries with large uncontained outbreaks: Iran (as of 1 March), South Korea (as of 5 March) and Italy (as of 11 March). As transmission continued to escalate globally, Australia experienced a so-called “first wave” epidemic in March and April, largely comprising of cases in returned travellers. With prevalence increasing in Australia and an increasing threat of importation from all global regions, Australia closed its borders to all non-citizens and non-residents on 19 March.

The combined strategy of early, proactive management of the risk of importation, case targeted interventions, and physical distancing effectively contained Australia’s first epidemic wave of COVID-19. By May 1, 2020, Australia had reported 6808 confirmed cases of COVID-19, including 98 deaths. A peak daily incidence of 456 cases was reported on 29 March. The epidemic was driven by importations, with more than two thirds of cases acquired overseas. A detailed description of the early phase of Australia’s COVID-19 epidemic and the public health response is provided in Price et al. (2020).

By November 2020, local case incidence was at zero or very low in all of Australia’s eight state and territories. At that time, we conducted additional analyses to critically evaluate the impact of model assumptions on the evidence provided in February 2020.

Evaluation approach

To evaluate our original framework, we assessed the impact of a range of alternate modelling choices on a selected risk assessment outcome: the expected date of exceeding a seven-day moving average of five imported infections per day into Australia from mainland China via intermediary countries.

Estimating undetected cases

At the time of our rapid risk assessment, we used a method developed by De Salazar and colleagues (De Salazar et al., 2020) to estimate the number of potentially undetected cases in each intermediary country at Step 2 of the framework. This method evaluates detection capacity against an average case-detection capacity across countries. A consequence of this approach is that it is possible for the estimated number of undetected imports in settings with strong surveillance systems, such as Hong Kong and Singapore, to be zero with high confidence. Around the same time, Bhatia and colleagues (Bhatia et al., 2021) proposed an alternate approach to estimating the likelihood of undetected cases of COVID-19 in countries outside of mainland China. Bhatia and colleagues showed Singapore to be an outlier in terms of surveillance performance, with relatively many detected imported cases of COVID-19 compared to their incoming traveller volume. They therefore constructed a statistical model to estimate the expected number of imported cases in a given country, relative to Singapore (or a group of countries with best detection effectiveness). Because this approach estimates imported cases relative to a location with “optimal” detection it systematically increases the number of potentially undetected imported cases in each intermediary country compared to the method developed by De Salazar and colleagues, leading to higher estimated numbers of imports into Australia at earlier time points. We therefore evaluated the impact of the Bhatia and colleagues’ approach on our risk assessment outcome.

Uncertainties in disease model parameters

Early in the emergence of a novel pathogen, many uncertainties exist around key pathogen characteristics and thus the most appropriate structure of a disease model. This was the case for SARS-CoV-2 in early February 2020. Two key uncertainties were the timing of symptom onset relative to infectiousness and the individual-level variation in transmission. The branching process model used in Steps 3 and 4 of our February risk assessment assumed that the onset of symptoms coincides with the onset of infectiousness, i.e., no pre-symptomatic transmission. We now know that a considerable proportion of SARS-CoV-2 transmission occurs prior to symptom onset (He et al., 2020). Furthermore, in the absence of information on the degree of individual-level variation in the transmission of SARS-CoV-2, we assumed this to be uniform. Studies have since suggested that there is high individual-level variation (i.e., over-dispersion) in the number of secondary infections (Endo, 2020).

In our evaluation exercise, we explored variants of the branching process model, based on our updated knowledge of SARS-CoV-2, that were anticipated to impact epidemic timescales and thus potentially impact our risk assessment outcome. Specifically, in the rapid risk assessment conducted in February 2020, we had used a single value of 2.68. Here we explored the impact of a lower () and higher () value of . We also varied the timing of infectiousness and case isolation relative to symptom onset. These variant models allowed us to explore the impact of different durations of the effective infectious period, i.e., the time from onset of infectiousness to isolation of detected cases, on our risk assessment outcome. Table 1 shows the mean duration in days of latent, pre-symptomatic infectious, infectious and effective infectious periods of the branching process model used at Steps 3 and 4 of the risk assessment framework for the original model and the three model variants. For the rapid risk assessment, we assumed no individual-level variation in offspring distribution. Here we incorporated over-dispersion in offspring distribution by sampling ’s for infected individuals from a gamma distribution with a mean of 2.68 and shape parameter of 0.1.

Table 1

Model variants explored in the evaluation exercise.

Model	Latent period	Pre-symptomatic	Infectious period	Effective infectious
		infectious period		period
February 2020 model (baseline)	5.2	0	7.68	1
- no pre-symptomatic transmission
- isolation 24 h after symptom onset
Model variant 1	5.2	0	7.68	0
- no pre-symptomatic transmission
- isolation at symptom onset
Model variant 2	3.2	2	9.68	2
- pre-symptomatic transmission
- isolation 24 h after symptom onset
Model variant 3	3.2	2	9.68	3
- pre-symptomatic transmission
- isolation at symptom onset

Our branching process model also incorporates a parameter for public health system workload capacity. For the rapid risk assessment conducted in February, this parameter was set to a conservative 10 detected infections per day. Here we explored the impact of increasing workload capacity to 20, 50, and 100 detected infections per day on the rate of importation into Australia.

Outputs of evaluation exercise

The expected dates of exceeding a seven-day moving average of five imported infections per day into Australia from mainland China via intermediary countries for each evaluation scenario are displayed in Table 2.

Table 2

Estimated timing of exceeding a seven-day moving average of five imported infections per day into Australia from mainland China via intermediary countries as of 19 February 2020 (expressed as lower and upper bounds of 95% credible intervals).

Sensitivity analysis	Date range (95% CIs)
February 2020 model (baseline)	(March 27, April 27)
Bhatia et al at Step 2	(March 22, April 12)
R_0= 2	(April 5, April 30)
R_0= 3	(March 23, April 25)
Effective infectious period = 0	(March 28, April 30)
Effective infectious period = 2	(March 20, April 22)
Effective infectious period = 3	(March 18, April 19)
Over-dispersion in offspring distribution	(March 16, April 30)
Workload capacity = 20	(March 28, April 28)
Workload capacity = 50	(March 28, April 28)
Workload capacity = 100	(March 28, April 28)

The baseline model used in our rapid risk assessment in February 2020 estimated that the threshold would be exceeded between March 27 and April 27 (lower and upper 95% credible intervals). Across the parameter values and model variants explored in our evaluation exercise, the threshold was exceeded within a date range from March 16 to April 30. The directions of shift in the expected date intervals for exceeding the threshold are intuitive. For lower and higher values of , the threshold was more likely to be exceeded later and earlier respectively. Model variants with a shorter or longer effective infectious period of detected cases were more likely to exceed the threshold later or earlier respectively. Incorporating over-dispersion in offspring distribution increased uncertainty in the estimated timing of exceeding the threshold (i.e., wider 95% credible intervals) with the lower 95% credible interval corresponding to the threshold being exceeded 11 days earlier than for the baseline model. Increasing the public health system workload capacity from 10 to 20 detected infections shifted the expected timing of exceeding the threshold marginally later. Increasing it further from 20 to 50 and 100, did not change the expected timing, indicating that the maximum workload reached in each intermediary country was not substantially greater than 10 detected infections per day.

The potential number of undetected imported cases (as of 19 February 2020) in countries in the South East Asia and Western Pacific regions based on the model described in Bhatia and colleagues (Bhatia et al., 2021) are displayed in Fig. 6. As anticipated, the absolute numbers of potentially undetected cases are systematically higher compared to the estimates used in our February 2020 risk assessment (Fig. 4). This increase is most marked in two intermediary locations, Taiwan and Malaysia. Consequently, the estimated date of exceeding the threshold of five imported infections per day into Australia is seven days earlier compared to when the De Salazar approach is used at Step 2 (Table 2).

Fig. 6

Estimated number of undetected cases and 95% confidence intervals for each country in the South East Asia and Western Pacific regions as of 19 February 2020, using the method described by De Salazar and colleagues (De Salazar et al., 2020) (in red) and by Bhatia and colleagues (Bhatia et al., 2021) (in blue). Note that in the Bhatia model implementation, Singapore is the reference country and is therefore assumed to have perfect detection. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Discussion

We developed a framework to assess the importation risk of SARS-CoV-2 into Australia during the early phase of the epidemic, from late January to mid-February 2020. The dominant importation risk to Australia at the time of analysis was directly from China, as the only country reporting uncontained transmission. With travel restrictions from mainland China to Australia imposed from 1 February 2020, our framework was designed to consider the importation risk from China (source country) into Australia (country of interest) via potential intermediary countries. We focused on countries in the South East Asia and Western Pacific regions as potential intermediary countries since they are highly connected to both China and Australia, relative to the rest of the world.

The primary output of our framework was a risk table (Fig. 3) that was updated and provided to the Australian government each week. The risk tables had two main purposes. First, to provide evidence of likely exposure over the preceding week to inform the epidemiological case definition for use in near patient decision making for practitioners caring for returned travellers. Second, by projecting the likely future epidemic course in intermediary countries, to inform travel advisories and the likely utility of border restrictions.

Our risk assessment as of 19 February 2020 estimated that the number of infections imported into Australia from the intermediary countries considered in this exercise would exceed a weekly average of five cases per day between 27 March and 27 April, 2020. In reality, this threshold was reached on 5 March 2020, largely driven by importations from countries not included in our initial risk analysis. When considering only importations from countries included in our analysis, the threshold was reached later in March, but still earlier than our originally estimated range.

Our initial analysis focused on identifying undetected epidemics rather than modelling observed epidemics (other than in the source country). Thus from late February, the risk appraisal shifted focus to observed case data from many countries, leading to the imposition of travel restrictions on specific countries and culminating in a blanket travel ban on 19 March 2020.

In November 2020, we revisited our original analysis to evaluate the impact of model assumptions on the evidence provided in February 2020 (Fig. 2). We considered the impact of assumptions about transmissibility (), the effective infectious duration, health workforce and screening capacity, and dispersion of secondary cases on the expected date of exceeding five imported infections per day into Australia from mainland China via intermediary countries. We observed that the direction of change aligned with expectations, with increased transmissibility, longer effective infectious duration, decreased health workforce and screening capacity, and over-dispersion of secondary cases bringing forward the expected time. The magnitude of changes was found to be relatively modest, with the greatest shifts occurring when we varied the model used to estimate the number of undetected imports in each intermediary country, and the over-dispersion of secondary cases, which brought the lower limits of the 95% CIs forward to 22 March and 16 March respectively.

In the remainder of this discussion, we summarize strengths and limitations of our framework, and lessons learned for future rapid-response efforts (Table 3). While these lessons are supported by our experience with the framework described here, we believe they are more generally applicable to policy-engaged modelling.

Table 3

Strengths, limitations, and lessons learned.

Transparency	Separately reporting the individual factors contributing to importation risk, while increasing the size and complexity of the risk tables, enabled decision-makers to incorporate their own expertise in interpreting estimates.
Simplicity	The simplicity of each individual component of the framework, while requiring abstraction of a more complex reality, was critical for its rapid development and deployment. The simplicity of the overall framework also made it relatively straightforward to explain to decision-makers how risk estimates had been arrived at. Eventually, this simplicity was also a limitation, as the framework was not equipped to incorporate emerging empirical data on local transmission in intermediary countries. The constrained focus on the Asia-Pacific region also proved to be a limitation when it emerged that transmission, and hence sources of importation to Australia, were more widespread than initially thought.
Flexibility	Decoupling the calculations involved in each component of the framework enabled specific sub-models to be substituted without extensive modification of other components. As exemplified in the critical evaluation reported here, this decoupling also allowed ready comparison of the impact of alternative sub-models. While this modular approach allowed incorporation of existing models—for example, for estimating undetected cases (De Salazar et al., 2020)—that did also entail accepting limitations of those models, such as specific assumptions around baseline case-detection capacity.
Uncertainty	The use of stochastic models in several components of the framework (estimating undetected cases and local epidemic growth) enabled quantification of uncertainty around risk estimates. Conveying this uncertainty to decision-makers was critical as there was considerable uncertainty about key epidemiological parameters for COVID-19 during this period. A key omission of our initial analysis was the over-dispersion of secondary cases, which proved to be both an important characteristic of COVID-19 transmission, and a factor to which our risk estimates were particularly sensitive.
Data	One of the most time-consuming aspects of developing and applying the analytical framework involved locating and processing relevant data sources on mobility, policy response, epidemiology, and so on. As has been recognized elsewhere (Kraemer et al., 2021), streamlining the consolidation, standardizing and sharing of epidemiological data will play a vital role on supporting rapid-response analysis in the event of future pandemics.
Timeliness	Decision-making in the early stages of a pandemic is time-critical. Using this framework to produce initial risk estimates within days of request provided a foundation for subsequent engagement. These initial estimates were then extended and refined with the addition of emerging data and methods. At the other end of the life-cycle, transparency around the limitations of the framework enabled a transition to risk forecasts based on emerging epidemiological data both internationally and locally.

A key strength of our approach is its transparency with respect to both individual factors that contribute to overall risk and the relationship between model projections and the reported epidemiology. This enables decision-makers to incorporate their own expertise when interpreting the estimates. Even when limited or highly uncertain data are available to inform absolute estimates of risk associated with plausible importation routes, comparisons of relative risk using our approach are still possible and valuable. In addition, each analysis component has modest data requirements and low computation cost, making rapid preliminary assessment across a range of countries feasible. As the outbreak progressed, mismatches between model projections and observed epidemiology could be readily observed and incorporated into decision making.

A strength of our modular approach is that it enables individual components to be adapted as new data and models become available, or as information needs change in response to the evolving situation. The breakdown of our workflow (Fig. 1) provides clear guidance on how to adjust the method. Importantly, the framework itself still applies, but components of the analysis would need to be adapted.

A further strength of our approach is its use of a stochastic model, including control efforts, for early epidemic response in intermediate countries. Here, without access to detailed additional information on intermediate country’s capacity to respond, we used a simple model of country response capacity. As outbreaks progress, differences in how effectively countries are able to respond, both in terms of their health system capacity, and their ability to implement population-based measures such as social distancing may introduce systematic effects that are not captured by the framework. Additional information or dedicated further research in anticipation of future global events could be incorporated into the framework (at Steps 3 and 4) to improve predictive capabilities. It is also worth noting that here we used a stringent definition of outbreak control (truly no cases) compared to other approaches in the literature, which may define control as a substantially smaller number of cases compared to baseline (Hellewell et al., 2020).

The time pressure under which our risk assessment framework was developed required us to make several strong assumptions, limiting the generality of the framework.

As implemented, our estimates of importation rates into countries only consider air travel (Step 1). Several countries/regions have high volumes of land travel, for example mainland China and Hong Kong. By not accounting for these, it is likely that our framework would underestimate importation risk to these countries/regions. Furthermore, the estimated importation rates are relative to global reports of imported cases of COVID-19. Given the systematic under-detection of COVID-19 across all countries at the time of analysis (Niehus et al., 2020), we would expect further underestimation of undetected cases and importation rates. Our approach did not attempt estimate rates of import prior to 22 January when case counts for countries started being reported by the WHO.

Our analysis framework is specific to a scenario in which the dominant source of infection can be mitigated by border measures. Once established outbreaks occur in countries other than the primary source, assessing importation risk becomes more complex. Further to this, we primarily focused on identifying undetected epidemics — over the course of time emerging observed data from many countries became more influential in the risk appraisal. Thus, as it stands, our analysis is only applicable in the early stages of an outbreak. By late February 2020, substantial transmission had been reported in a number of additional countries, in particular Iran, Italy and South Korea (World Health Organization, 2020d). In order for our tool’s risk assessment to reflect the true risk of importation to the country of interest, components of the analysis, specifically Steps 1 and 2, would need to be adapted to account for high levels of known local transmission in countries other than the initial source country.

When developing the framework in February 2020, we faced a trade-off between the need for a general approach that could be rapidly applied to a wide range of countries and a more complex approach that could make use of richer data in a subset of those countries. Incorporating methods that can better account for heterogeneous data quality would enable the use of more detailed individual country data (where available) and more accurate estimates of the evolving epidemiological situation in each location. Ultimately, our analysis made a number of simplifying assumptions about global epidemiology and health system capacity, due to limitations in both time and data, constraining its relevance to a specific phase of the early COVID-19 pandemic.

The framework we have developed provides a risk assessment from solely from an epidemiological perspective. It does not consider the potential social, political and economic implications of future border measures and mobility restrictions which are both substantial (Errett et al., 2020) and will exert an influence on epidemiology as people change their behaviour (Funk et al., 2015). This framework is therefore only one key element to be considered by decision-makers contemplating possible border policies and mobility restrictions, which are ultimately a political determination.

In conclusion, by developing a modular framework that describes not only the underlying mathematical models of transmission and control, but how each component integrates with the next to generate an overall assessment of importation risk of an emerging disease, we have provided a decision-making tool that is flexible to the analysis requirements at different phases of an outbreak. The framework provides an evidence base for time-critical decisions on border measures and case definitions, and it has been successfully used during the early phase of the COVID-19 response in the Australian context, when limited cases had been reported outside of mainland China.

Rapid-response modelling analyses will continue to be an essential input to emergency outbreak decision-making. Using existing frameworks enables researchers to rapidly pull together the available data in order to provide timely advice that reflects uncertainties, and is communicated in a transparent and effective manner. Additionally, retrospective, documented evaluation of these frameworks is critical to their improvement and utility for future outbreak response.

CRediT authorship contribution statement

Freya M. Shearer: Conceptualization, Methodology, Investigation, Writing. James Walker: Methodology, Investigation. Nefel Tellioglu: Methodology, Investigation. James M. McCaw: Writing. Jodie McVernon: Conceptualization, Project administration. Andrew Black: Conceptualization, Methodology. Nicholas Geard: Conceptualization, Methodology, Investigation, Writing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.