Forecasting new diseases in low-data settings using transfer learning.

Recent infectious disease outbreaks, such as the COVID-19 pandemic and the Zika epidemic in Brazil, have demonstrated both the importance and difficulty of accurately forecasting novel infectious diseases. When new diseases first emerge, we have little knowledge of the transmission process, the level and duration of immunity to reinfection, or other parameters required to build realistic epidemiological models. Time series forecasts and machine learning, while less reliant on assumptions about the disease, require large amounts of data that are also not available in early stages of an outbreak. In this study, we examine how knowledge of related diseases can help make predictions of new diseases in data-scarce environments using transfer learning. We implement both an empirical and a synthetic approach. Using data from Brazil, we compare how well different machine learning models transfer knowledge between two different dataset pairs: case counts of (i) dengue and Zika, and (ii) influenza and COVID-19. In the synthetic analysis, we generate data with an SIR model using different transmission and recovery rates, and then compare the effectiveness of different transfer learning methods. We find that transfer learning offers the potential to improve predictions, even beyond a model based on data from the target disease, though the appropriate source disease must be chosen carefully. While imperfect, these models offer an additional input for decision makers for pandemic response.

Introduction

Epidemic models can be divided into two broad categories: data-driven models aim to fit an epidemic curve to past data in order to make predictions about the future; mechanistic models simulate scenarios based on different underlying assumptions, such as varying contact rates or vaccine effectiveness. Both model types aid in the public health response: forecasts serve as an early warning system of an outbreak in the near future, while mechanistic models help us better understand the causes of spread and potential remedial interventions to prevent further infections [1], [2]. Many different data-driven and mechanistic models were proposed during the early stages of the COVID-19 pandemic and informed decision-making with varying levels of success [1], [3], [4]. This range of predictive performance underscores both the difficulty and importance of epidemic forecasting, especially early in an outbreak. Yet the COVID-19 pandemic also led to unprecedented levels of data-sharing and collaboration across disciplines, so that several novel approaches to epidemic forecasting continue to be explored, including models that incorporate machine learning and real-time big data data streams [5], [6]. In addition to the COVID-19 pandemic, recent infectious disease outbreaks include Zika virus in Brazil in 2015, Ebola virus in West Africa in 2014–16, Middle East respiratory syndrome (MERS) in 2012, and coronavirus associated with severe acute respiratory syndrome (SARS-CoV) in 2003. This trajectory suggests that further improvements to epidemic forecasting will be important for global public health. Exploring the value of new methodologies can help broaden the modeler's toolkit to prepare for the next outbreak [7], [8]. In this study, we consider the role of transfer learning for pandemic response.

Transfer learning refers to a collection of techniques that apply knowledge from one prediction problem to solve another, often using machine learning and with many recent applications in domains such as computer vision and natural language processing [9], [10]. Transfer learning leverages a model trained to execute a particular task in a particular domain, in order to perform a different task or extrapolate to a different domain. This allows the model to learn the new task with less data than would normally be required, and is therefore well-suited to data-scarce prediction problems. The underlying idea is that skills developed in one task, for example the features that are relevant to recognize human faces in images, may be useful in other situations, such as classification of emotions from facial expressions. Similarly, there may be shared features in the patterns of observed cases among similar diseases.

The value of transfer learning for the study of infectious diseases is relatively under-explored. The majority of existing studies on diseases remain in the domain of computer vision and leverage pre-trained neural networks to make diagnoses from medical images, such as retinal diseases [11], dental diseases [12], or COVID-19 [13], [14]. Coelho and colleagues (2020) [15] explore the potential of transfer learning for disease forecasts. They train a Long Short-Term Memory (LSTM) neural network on dengue fever time series and make forecasts directly for two other mosquito-borne diseases, Zika and Chikungunya, in two Brazilian cities. Even without any data on the two target diseases, their model achieves high prediction accuracy four weeks ahead. Gautam (2021) [16] uses COVID-19 data from Italy and the USA to build an LSTM transfer model that predicts COVID-19 cases in countries that experienced a later pandemic onset.

These studies provide empirical evidence that transfer learning may be a valuable tool for epidemic forecasting in low-data situations, though research is still limited. In this study, we aim to contribute to this empirical literature not only by comparing different types of knowledge transfer and forecasting algorithms, but also by considering two different pairs of endemic and novel diseases observed in Brazilian cities, specifically (i) dengue and Zika, and (ii) influenza and COVID-19. With an additional analysis on simulated time series, we hope to provide theoretical guidance on the selection of appropriate disease pairs, by better understanding how different characteristics of the source and target diseases affect the viability of transfer learning.

Zika and COVID-19 are two recent examples of novel emerging diseases. Brazil experienced a Zika epidemic in 2015–16 and the WHO declared a public health emergency of global concern in February 2016 [17]. Zika is caused by an arbovirus spread primarily by mosquitoes, though other transmission methods, including congenital and sexual have also been observed. Zika belongs to the family of viral hemorrhagic fevers and symptoms of infection share some commonalities with other mosquito-borne arboviruses, such as yellow fever, dengue fever, or chikungunya. Illness tends to be asymptomatic or mild but can lead to complications, including microcephaly and other brain defects in the case of congenital transmission [18], [19].

Given the similarity of the pathogen and primary transmission route, dengue fever is an appropriate choice of source disease for Zika forecasting. Not only does the shared mosquito vector result in similar seasonal patterns of annual outbreaks, but consistent, geographically and temporally granular data on dengue cases is available publicly via the open data initiative of the Brazilian government [20].

COVID-19 is an acute respiratory infection caused by the novel coronavirus SARS-CoV-2, which was first detected in Wuhan, China, in 2019. It is transmitted directly between humans via airborne respiratory droplets and particles. Symptoms range from mild to severe and may affect the respiratory tract and central nervous system. Several variants of the virus have emerged, which differ in their severity, transmissibility, and level of immune evasion [21], [22], [23].

Influenza is also a contagious respiratory disease that is spread primarily via respiratory droplets. Infection with the influenza virus also follows patterns of human contact and seasonality. There are two types of influenza (A and B) and new strains of each type emerge regularly. Given the similarity in transmission routes and to a lesser extent in clinical manifestations, influenza is chosen as the source disease for knowledge transfer to model COVID-19 [24], [25].

For each of these disease pairs, we collect time series data from Brazilian cities. Data on the target disease from half the cities is retained for testing. To ensure comparability, the test set is the same for all models. Using this empirical data, as well as the simulated time series, we implement the following transfer models to make predictions.

•  Random forest: First, we implement a random forest model which was recently found to capture well the time series characteristics of dengue in Brazil [26]. We use this model to make predictions for Zika without re-training. We also train a random forest model on influenza data to make predictions for COVID-19. This is a direct transfer method, where models are trained only on data from the source disease.

•  Random forest with TrAdaBoost: We then incorporate data from the target disease (i.e. Zika and COVID-19) using the TrAdaBoost algorithm together with the random forest model. This is an instance-based transfer learning method, which selects relevant examples from the source disease to improve predictions on the target disease.

•  Neural network: The second machine learning algorithm we deploy is a feed-forward neural network, which is first trained on data of the endemic disease (dengue/influenza) and applied directly to forecast the new disease.

•  Neural network with re-training and fine-tuning: We then retrain only the last layer of the neural network using data from the new disease and make predictions on the test set. Finally, we fine-tune all the layers' parameters using a small learning rate and low number of epochs. These models are examples of parameter-based transfer methods, since they leverage the weights generated by the source disease model to accelerate and improve learning in the target disease model.

•  Aspirational baseline: We compare these transfer methods to a model trained only on the target disease (Zika/COVID-19) without any data on the source disease. Specifically, we use half the cities in the target dataset for training and the other half for testing. This gives a benchmark of the performance in a large-data scenario, which would occur after a longer period of disease surveillance.

The remainder of this paper is organized as follows. The models are described in more technical detail in Section 2. Section 3 shows the results of the synthetic and empirical predictions. Finally, Section 4 discusses practical implications of the analyses.

Materials and methods

Data

For the empirical analysis, we use official weekly case reports at the municipal level of four diseases from the open data platform of the Brazilian government [27]. Dengue and Zika data are collected from the Notifiable Diseases Information System (NDIS) for the years spanning 2014–2020 and 2016–2020, respectively. NDIS also reports cases of Severe Acute Respiratory Syndrome (SARS), reliably across municipalities from 2013 onwards. SARS in this dataset refers to individuals presenting with symptoms such as fever or difficulty breathing, which may be caused by different pathogens. For this study, we restrict the data to the SARS cases with laboratory-confirmed influenza (strains A and B). Daily COVID-19 case reports are collected from March 2020 to September 2021 [28]. They are aggregated to weekly case counts to match the reporting frequency of the other datasets.

The synthetic time series are generated using a stochastic model with compartments for susceptible, infectious, recovered, and dead population groups (SIRD) given by Eq. (1).where: β is the effective contact rate (loosely referred to as the transmission rate), γ is the recovery rate, ζ is the waning immunity rate, and μ is the disease-specific death rate. Infection, recovery, waning immunity, and death from disease are treated as stochastic events. At each time step, the number of occurrences of each event are sampled from a binomial distribution with probabilities given by the SIRD model parameters (Eq. (1)) as e −, e −, e −, and e −, respectively.

To simulate different disease settings, we vary the model parameters. Our endemic disease has parameters β = 0.191, γ = 0.05, ζ = 0.008, μ = 0.0294, while the target diseases (for transfer) have transmission and recovery rates β ∈ {0.25,0.3,0.35} and γ ∈ {0.01,0.1,0.15}, respectively. This results in nine target diseases, each with 100 observations over 1000 time steps. The parameter ranges were chosen from estimates in the literature of an empirical COVID-19 model [29]. For each pair of parameters, we generate two datasets of 100 time series of length T = 1000 - one dataset for training and another for testing. Each time step in the simulation may be interpreted as one week. In the context of this simulated data, new disease refers to any scenario that may result in variation of epidemiological parameters, including new variants or changes in the contact rates within a population.

Data preparation

In the empirical analysis, we split the cities randomly into train and test sets. We also generate separate synthetic training and test sets of equal size.

The time series are split into short sections to train the machine learning models (see Fig. 1 ). Each section consists of the last nine observations (input features), x , x , …, x , and the future number of cases (target feature), y , where g indicates the time gap between input and target. The target feature is between two and nine time steps ahead of the input features, allowing us to compare prediction horizons up to nine weeks ahead. To ensure a fair comparison of performance across the horizons, the first eight time periods are removed from all output datasets.

Fig. 1

Methodology.

The time series for the source (A) and target disease (B) are prepared for forecasting up to 9 time steps ahead using 9 time steps as input features. The data is used to train four kinds of models: the base model trains and tests on the target disease. Direct transfer entails training on the source disease and predicting the target disease. The parameter-based transfer model trains on the source disease and uses the target disease for parameter update in the last layer of a neural network as well as for prediction. This method may also include an additional fine-tuning step where all layers are updated. Instance-based transfer selects features from the source disease to complement the target disease data. The model is trained on the combined data and tested on the target disease.

For all transfer models, we compare different levels of data availability, in order to simulate the predictive performance at different time points in the outbreak. We compare three different cutoff dates of the empirical data, four weeks apart, and periods of 25, 30, 35, and 100 steps of the 1000-step long time series.

Methods

In both the synthetic and empirical analyses, we compare three different transfer learning approaches, eight different forecast horizons, and four different levels of data availability using the methodology presented in Fig. 1. This section describes each of these models in more detail.

No transfer

We first train a machine learning model on the endemic disease and then use it directly to forecast the target disease, without any adjustment of the model weights. Two algorithms were chosen for direct transfer: (i) a random forest algorithm with 50 trees and (ii) a fully-connected feed-forward neural network with three hidden layers and (64,32,32) neurons in the respective layers.

Random forests (RF) [30] is an ensemble method that aggregates predictions from multiple decision trees. A decsion tree determines the best splitting values of the input features in order to separate the observations according to the output value. The splitting criterion is the mean squared error (MSE). Random forests grows such trees iteratively, introducing variation by using only a subset of the observations for each tree and a subset of the features at each branch in the tree.

Feed-forward neural networks (NN) ([31]) are constructed from several layers of neurons (also known as processing units or hidden units), each of which combines and transforms input features and passes them on to the next layer of neurons. Transformations entail weighting and linearly combining the vector of input features together with a bias term, and then applying a non-linear function (activation). The weights and biases are optimized to minimize the prediction errors according to a loss function, in this case the MSE, using gradient descent.

Transfer

We implement one instance-based and one parameter-based transfer method, which are selected to complement the strengths of the two machine learning models. Using the TrAdaBoost algorithm [32], the instance-based method selects observations from the source data (endemic disease) which will help improve predictions on the target data (new disease). We implement TrAdaBoost with the RF algorithm for 10 boosting iterations. At each iteration, the TrAdaBoost algorithm adjusts the importance assigned to each observation in the source dataset, so as to reduce the differences in distributions in the source and target domains. Examples of the source disease time series that are dissimilar to the target disease time series therefore receive a smaller weight and have less influence on model training [10].

Using a neural network architecture, the parameter-based approach instead works as a warm start to training, by maintaining the parameters of the source disease model and updating only the weights in the last layer of the neural network with data from the target disease [33]. We freeze the parameters in the first layers and update the output layer for 500 epochs with early stopping and exponential learning rate decay. Finally, after updating the weights of the last layer of the neural network, we unfreeze the weights of the remaining layers and update all weights using a very low learning rate (α = 0.00001) and short training time (10 epochs).

Baseline

In order to better judge performance of the transfer models, we implement a baseline model that is both trained and tested on the target disease. We use the same machine learning algorithms and architectures as in the direct transfer case. To train the baseline models, we use the data from half the cities in the target dataset over the full time period. We test the baseline models on the same dataset as the transfer models, consisting of the other half of cities. Given the overlapping time period used for training and testing as well as the relatively long time series, these baseline models are not representative of a model that would be available at the early stage of an outbreak. Rather, we include this model as a benchmark for the performance the given ML algorithm could achieve with longer surveillance.

Results

Synthetic analysis

Fig. 2 shows a ranking of the models according to prediction errors on the nine synthetic datasets at the lowest level of data availability. The models are compared using a percentage measure of the mean absolute error (MAE). The raw MAE varies between diseases because the parameter combinations produce different outbreak sizes. We correct for this variation by dividing the errors by the total number of cases in each city.

Fig. 2

Ranking of models by their mean absolute errors for predictions of synthetic data at lowest data availability (cutoff 1).

The results show that transfer learning has the potential to improve predictions even over the baseline models,. This is especially the case for diseases whose parameters are close to the source disease. For example, the top left panel in Fig. 2 shows the target disease with the most similar parameters, an effective contact rate of β = 0.25 (c.f. source effective contact rate β = 0.191) and a recovery rate of γ = 0.01 (c.f. source recovery rate γ = 0.05). The three neural network models, NN fine-tuned, NN transfer, and NN no transfer, outperform both baselines for all prediction horizons. Conversely, when the source and target diseases differ more widely in the epidemiological parameters, the baseline models are harder to beat. For example, the bottom right panel in Fig. 2 shows that highest predictive performance is achieved by the RF baseline, followed by the NN baseline. The next best result is achieved by models trained only on the source data (RF no transfer and NN - no transfer). This confirms the intuitive notion that the similarity of diseases matters to the potential value of transfer learning for disease forecasting. When the similarity cannot be known, an ensemble approach of both the direct transfer (endemic disease) and weight- or instance-based methods may be most promising.

For low recovery rates (γ = 0.01), regardless of transmission rates, all three neural network models consistently beat both baselines for prediction horizons of 3 weeks and above. As the recovery rates increase, the NN models are more often replaced by the RF models (RF no transfer and TrAdaBoost) (Fig. 2 and Supplemental Fig. 1).

In lieu of known disease parameters (e.g. transmission and recovery rates), we may leverage data-based similarity measures to assess ex ante the benefit of transfer learning in general, and the different algorithms, in particular. A simple, yet effective measure is Pearson's correlation. Fig. 3 shows the median pairwise correlation between the time series of the source disease and each of the simulated diseases, where the x- and y-axis show the different γ- and β-values used to generate the data, respectively. The top left square represents the most similar disease in terms of the epidemiological parameters, which is also most strongly correlated with the source data. The bottom right square represents the least similar disease and has a correlation coefficient close to zero.

Fig. 3

Median pairwise correlation coefficient of source time series with simulated datasets of varying effective contact (β) and recovery (γ) rates.

Empirical analysis

Fig. 4 compares the performance of the different model types for the two disease pairs at the lowest level of data availability. In the case of Zika, RF models perform better than the NN models, though neither outperforms the baselines (see Fig. 4 and Supplemental Fig. 2). The performance of the TrAdaBoost algorithm depends very little on the quantity of data available on Zika (see Fig. 5a). Similarly, the RF model trained only on dengue performs surprisingly well, with nearly the same median error rate up until a prediction horizon of five weeks. However, at increasing prediction horizons, the dengue and TrAdaBoost models increasingly deviate from the baseline models trained only on Zika (Fig. 4, Fig. 5a).

Fig. 4

Ranking of models by their mean absolute errors for empirical predictions at lowest data availability (cutoff 1).

Fig. 5

Percent mean absolute errors for different prediction horizons and model types within the best-performing algorithm of each disease.

Panel (a) compares the different random forest models implemented for Zika prediction: TrAdaBoost with three different cutoff dates, random forests trained on dengue data, and the baseline random forest model trained on Zika data. Panel (b) compares the different neural network models implemented for COVID-19 prediction: the left plot shows models with parameter transfer in the last layer of the neural network at different cutoff levels, while the right plot shows models with transfer including an extra fine-tuning step updating parameters in all layers. Both plots also show the baseline neural network model trained on COVID-19 data and the flu-trained neural network without transfer.

In the case of COVID-19, the direct NN model has the lowest error (Fig. 4), outperforming both baselines for most prediction horizons. The finetuned NN model also performs well and has lower errors than the baselines for predictions up to 4 weeks ahead. This suggests that transfer models can not only approximate the performance of a model of the target disease, but even improve predictions due to the relatively larger overall training dataset of the source disease. Increasing data availability further improves the performance of the NN models relative to the baseline (Fig. 5b). For early prediction horizons, all cutoff levels produce lower errors than the NN baseline (week 2 for the NN transfer model and weeks 2–4 for the NN fine-tuned model). As in the Zika case above, the direct transfer model performs relatively well and has lower errors than the baseline for most prediction horizons.

As in the synthetic analysis, the empirical results also suggest that the best algorithm varies by disease. RF performs better for Zika forecasting, while NN models achieve lower errors for COVID-19 predictions (Fig. 4, Fig. 5). We observe less variation between cities: for Zika, an RF algorithm is chosen for nearly all cities and all prediction horizons as the best model, while NN models are chosen for COVID-19 (Fig. 6 ). We also note a subtle shift from the direct models toward the transfer models as more data becomes available. For example, for eight-week ahead predictions, moving from cutoff level one to three, transfer and fine-tuned NN models are increasingly favored over the direct transfer NN (Fig. 6).

Fig. 6

Distribution of models chosen as best 8-week-ahead predictors of (a) Zika and (b) COVID-19 at varying cutoff levels.

Discussion

In this study, we compared transfer learning methods and machine learning algorithms to forecast new diseases at varying levels of low data availability. Analyses with both synthetic and empirical data suggest that transfer learning has the potential to approximate or even outperform comparator models trained on larger datasets of the target disease. Our results confirm the intuitive idea that the value of transfer learning relative to the baselines is greater when more data of the target disease is available and when the two diseases have more similar epidemiological parameters. We also find that the best combination of transfer method and machine learning algorithm differs by disease, but that there is less variation across cities. RF models performed better for Zika forecasts, while NN models were more successful in predicting COVID-19. In the synthetic analysis, NN transfer models predicted the most similar target disease with greater accuracy, while TrAdaBoost and the direct RF model faired better for less similar target diseases.

The strong performance of the direct transfer models is noteworthy, given that they produce good predictions without incorporating any knowledge on the new disease in the training process. This finding is consistent with existing work [15]. Training only on the source data enables earlier deployment and potentially lower computational cost. However, the risks of such extrapolation must be carefully considered when predictions are to be used for decision-making. Additionally, this study focused on diseases that are known to be similar. The limits of direct transfer without target data must be explored more broadly for dissimilar diseases and wider study contexts.

A challenge in assessing the value of transfer learning for epidemic forecasts is the difficulty of defining a baseline model. We chose a baseline trained on longer time series, though it still involves extrapolation from one half of cities to the other. An alternative approach would be to train a model on part of the time series for all cities and then test it on the remaining time series. However, this would restrict the test set to certain seasons within a year, such as the winter in Brazil, which is the low season for Zika. This would not give a general estimate of model performance and especially not in the context of the early stages of an epidemic. In this study, we aim to reduce the extrapolation bias by randomizing the city selection across all of Brazil.

Several sources contribute to the overall prediction error of the transfer models in this study, including measurement bias of evolving surveillance systems, inherent stochasticity of disease spread, and the variation between source and target diseases as well as their respective data collection methods. Thoughtful implementation of transfer learning helps limit some of these biases. For example, comparator diseases may be carefully selected using either knowledge of the disease or data-based methods such as the Akaike information criterion (AIC) or Pearsons correlation [9]. Any forecasting tool should take into account the changing nature of diagnostic capacities, public health policies, and human behavior. A flexible methodology can help quickly update the choice of algorithms and data sources as both the outbreak and surveillance system evolve. Similarly, given variability in the best-performing algorithm and transfer method, an ensemble approach may be most promising in early stages of an outbreak.

Disease forecasts provide projections of the likely trajectory of a disease based on data from the past, which can serve as an early warning system for outbreaks and help policymakers efficiently allocate resources. Since historic data is not available for new diseases, we explore the value of transfer learning. However, even when historic data is available, long-term forecasts are challenging for any predictive model, because data of the past may have been generated in settings that will not be replicated in the future, especially when public health interventions are applied to control the outbreak [1]. In that case, forecasts may be supplemented by results from mechanistic models, which enable simulations of public health interventions, such as prioritizing different vaccination strategies [34]. These models provide useful insights into the mechanisms or possible causes of underlying disease dynamics, such as the impact of racial disparities on the herd immunity threshold [35]. These insights in turn may help inform forecasting models, for example by helping identify appropriate predictor variables. Results from transfer learning models should therefore be seen as only one of many information sources for public health decision-making.

Data availability statement

All data is from public sources. The python code used for the analysis is available at .

CRediT authorship contribution statement

The author contributions were as follows: K.R., F.A.R., and C.C. conceived of the idea. K.R. collected data, performed the computations, and drafted the manuscript. F.A.R. and C.C. supervised the analysis and reviewed the manuscript.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.