Fundamental limitations on efficiently forecasting certain epidemic measures in network models.

The ongoing COVID-19 pandemic underscores the importance of developing reliable forecasts that would allow decision makers to devise appropriate response strategies. Despite much recent research on the topic, epidemic forecasting remains poorly understood. Researchers have attributed the difficulty of forecasting contagion dynamics to a multitude of factors, including complex behavioral responses, uncertainty in data, the stochastic nature of the underlying process, and the high sensitivity of the disease parameters to changes in the environment. We offer a rigorous explanation of the difficulty of short-term forecasting on networked populations using ideas from computational complexity. Specifically, we show that several forecasting problems (e.g., the probability that at least a given number of people will get infected at a given time and the probability that the number of infections will reach a peak at a given time) are computationally intractable. For instance, efficient solvability of such problems would imply that the number of satisfying assignments of an arbitrary Boolean formula in conjunctive normal form can be computed efficiently, violating a widely believed hypothesis in computational complexity. This intractability result holds even under the ideal situation, where all the disease parameters are known and are assumed to be insensitive to changes in the environment. From a computational complexity viewpoint, our results, which show that contagion dynamics become unpredictable for both macroscopic and individual properties, bring out some fundamental difficulties of predicting disease parameters. On the positive side, we develop efficient algorithms or approximation algorithms for restricted versions of forecasting problems.

Prediction is very difficult, especially if it’s about the future.

Niels Bohr (1885–1962)

Background and Motivation

As demonstrated by the COVID-19 pandemic, large unexpected disease outbreaks can lead to devastating economic consequences. Thus, there is increasing interest in the development of systems that can provide early warnings regarding epidemics. This is borne out by the large number of epidemic-forecasting challenges issued by various agencies; examples include COVID-19 forecasts from the Centers for Disease Control and Prevention (CDC) (1), the “CHIKV Challenge” (for chikungunya virus) by the Defense Advanced Research Projects Agency (2), and the Intelligence Advanced Research Projects Activity (IARPA) flu challenge (3). There has also been a burst of activity on attempting to forecast different kinds of phenomena, such as sizes of epidemic outbreaks (e.g., refs. 4–6), cascades in social media, and civil unrest (e.g., refs. 7–9). In the context of evolutionary biology, pioneering work by Jacob (10) has clearly pointed out that one cannot reliably predict the properties of a complex system at one level based on what is known about the system at simpler levels. The difficulty of forecasting geophysical phenomena such as earthquakes has also been noted in the literature (e.g., ref. 11). A special issue of Science on “Prediction and its Limits” (12) contains many articles that point out the difficulty of accurately forecasting the behaviors of complex social systems (e.g., refs. 13–16). Despite a lot of work, epidemic forecasting remains poorly understood. A case in point is Google Flu Trends (GFT) (17). In its initial years, GFT produced very good forecasts of flu incidence rates, just based on search query results. However, the forecast accuracy decreased over time [e.g., overestimates of the A/H3N2 epidemic (18)]. Since the spread of flu-like diseases is a stochastic process, which depends on many time-varying factors (19, 20), one cannot expect accurate long-term predictions from a model such as GFT that relies on just one form of data. Several recent articles and news stories have highlighted the difficulties in constructing good models and producing reliable forecasts for influenza (e.g., ref. 21) and the COVID-19 pandemic (e.g., refs. 22 and 23). A comprehensive list of models that are currently used for forecasts related to COVID-19 can be found in ref. 24.

Many researchers have observed the need for carrying out a systematic study of the various issues associated with forecasting contagion dynamics in networked systems. Some of these observations are summarized below.

Drake (19, 20) observes that while systems may provide good forecasts of some epidemic measures (e.g., timing), there are fundamental limits on the effectiveness of such systems in forecasting other measures (e.g., final epidemic size). Several important reasons (such as the nature of stochastic disease-propagation models and high sensitivity of disease parameters to changes in the environment) are articulated in refs. 19 and 20 to explain such limits.

Cheng et al. (25) address the question of predicting whether a cascade will continue to grow in a social network. In particular, they focus on identifying the features of a cascade that can help in predicting the future course of the cascade. They remark that “a robust way to formulate the problem of cascade prediction remains an open problem.” In other related work, Althouse et al. (26) observe that superspreader events have played a big role in the spread of COVID-19 and that prediction of such events is important in controlling disease outbreaks.

Martin et al. (7) examine the limits of predictability in complex social systems; they observe that even a small degree of uncertainty can limit predictability. Further, they raise the question of whether a phenomenon that one is trying to predict is itself inherently unpredictable or the available data and techniques are inadequate to develop a reliable prediction. Hofman et al. (14) mention that “theoretical limits to the predictive accuracy of complex systems must be better characterized.”

Lazer et al. (18) use GFT as an example to caution against the use of social media and search information as a substitute for traditional data collection and analysis methods to predict epidemic measures. They highlight the need for systematically “studying the evolution of the socio-technical systems that are embedded in our society.”

Shaman et al. (27) use a differential equation-based model along with new data analysis techniques to predict the timing of the peak number of infections for influenza outbreaks. They demonstrate the usefulness of their framework in predicting the peak timing for the 2012–2013 season. They suggest that their methods can be made more robust when additional data are available.

Biggerstaff et al. (28) discuss results from the CDC’s “Predict the 2013–2014 Influenza Season Challenge.” They conclude that “forecasting has become technically feasible, but further efforts are needed to improve the accuracy so that policy makers can reliably use those predictions.”

May (29) points out that care must be exercised in drawing conclusions about the dynamics of ecological systems when there is uncertainty in the structure and parameters of the underlying network model. As mentioned earlier, a similar observation was made by Jacob (10) in the context of evolutionary biology.

Beckage et al. (30) point to another challenge for forecasting, which they refer to as “computational irreducibility”—this is the property of systems where the dynamics cannot be predicted without observing the evolution of the system. However, this can be difficult if determining the properties associated with the evolution of a system is itself computationally demanding—this is referred to as “computational intractability” (31) and has been a cornerstone of modern theoretical computer science. Indeed, many researchers [e.g., Buss et al. (32), Moore (33), and Wolfram (34)] have observed that computational intractability results for discrete dynamical systems provide an indication of the unpredictability (or “chaotic” behavior) of such systems. The theory of computational intractability has helped to place computational problems into a number of classes, such as P, NP, #P, etc. (see, e.g., refs. 31 and 35 for an introduction to this topic). Informally, problems that are NP-hard or #P-hard are unlikely to have efficient algorithms, i.e., algorithms that run in time that is a polynomial function of the input size. Computationally intractable problems arise in various fields (e.g., Mathematics, Physics, Biology, Social Science, Computer Science, and Operations Research). In the context of forecasting, some computational intractability results that arise in testing weather forecasts are presented in ref. 36; however, the model used in their work is different from network-based epidemic models.

In this paper, we show that many fundamental problems related to short-term predictions of epidemic properties in network models are computationally intractable, even when all the model parameters are known and assumed to be insensitive to changes in the environment. Using the observations in refs. 32–34, these results are indicative of the unpredictability of epidemic dynamics in networked systems. Thus, our results bring out a fundamental difficulty of predicting disease parameters. Scarpino and Petri (6) use a form of entropy as a model-independent measure of predictability of epidemic dynamics. They suggest that heterogeneity of social networks is a likely barrier to effective predictability. This observation is consistent with our results, which indicate that network structure plays an important role in determining the computational complexity of short-term forecasting problems.

Computational intractability results, such as NP-hardness and #P-hardness, capture worst-case scenarios; they indicate that there are problem instances for which efficient algorithms are unlikely to exist, unless some widely believed hypotheses in computational complexity turn out to be false (35). In other words, these results point out some fundamental limitations on efficiently computing certain epidemic parameters. For that reason, our complexity results should not be interpreted as implying that algorithms that work well in practice do not exist. Problem instances arising in practice are generally different from worst-case instances. In Section 4, we mention several approaches that are used in practice to obtain good solutions to some computationally intractable problems in networked epidemiology. Also, our computational intractability results do not point to weaknesses of epidemic models, nor do they raise concerns regarding the usefulness of models. Indeed, forecasts generated from commonly used models and practical solution techniques, with clearly stated assumptions and limitations, are invaluable to both policy makers and the general public.

Many computational models for epidemics have been studied in the literature (see e.g., refs. 37–39). Some recent models also take into account complex behaviors of contagions (such as evolutionary adaptions and spread of multiple strains) and strategies for vaccine dosages (40–43). Our results hold under the well-known Susceptible–Infected–Recovered (or SIR) model initially proposed in ref. 38, as well as under a broad class of related models, such as Susceptible –Infected (SI), Susceptible–Infected–Susceptible (SIS), and probabilistic threshold (PT) (complex contagion) models. We note that our results are for networked populations, and not for homogeneous mixing of populations, where the SIR and similar models were originally studied.

Networked Epidemic Model and Problems Studied

SIR Epidemic Model for Networks: A Brief Description.

To present the definitions of the forecasting problems considered in our work, we now provide a brief discussion on the SIR epidemic model for networks. For a more detailed discussion, the reader is referred to .

In the simplest SIR model on a network with unit infectious duration, the contagion is assumed to spread on an undirected network G(V, E), where V and E represent the set of nodes and edges, respectively. At any time* instant, each node is in one of the states from the domain = , . The values , and represent susceptible, infected (or infectious), and recovered states, respectively. For any node , the neighborhood of v, denoted by N, contains each node u such that the edge {u, v} is in E. Each edge is associated with a transmission probability p with the following interpretation. At time t, suppose node v is in state , and X(v, t) denotes the subset of N such that each node in X(v, t) is in state . Then, at time t, each node tries to infect node v with probability independently of the other nodes in X(v, t). If any of these attempts is successful, then the state of v changes to at time t + 1.

A configuration of an SIR system at time t is an n-vector , where is the state of node v at time t, . A single transition of an SIR system from one configuration to another is obtained by updating the state of each node v synchronously in the following manner.

If the state of v at time t is , then the state of v at time t + 1 is also . (Thus, once a node reaches the state , it remains in that state forever.)

If the state of v at time t is , then the state of v at time t + 1 is . Thus, each node remains in state for exactly one time unit.

If the state of v at time t is , then the state of v at time t + 1 is determined by the following stochastic process. As mentioned earlier, let denote the set of neighbors of v whose state is at time t, and let be defined as follows. (The following expression for is a simple consequence of the assumption that each node in X(v, t) tries to infect v independently.)

The state of v at time t + 1 is with probability and with probability .

Initially (i.e., at ), one or more nodes are in state , and the other nodes are in state . Starting from the given initial configuration , the system goes through a sequence of configurations over time, as indicated in the following example.

The graph of an SIR system consisting of seven nodes is shown in Fig. 1. The nodes are labeled v0 through v6, and the transmission probability of each edge is also shown. Suppose at , node v0 is in state , and all other nodes are in state . Starting from this initial configuration, one possible sequence of configurations that the system may go through is shown in the table in the figure.

Fig. 1.

The graph of an SIR system and one possible sequence of its configurations, leading to a fixed point at time t = 4. [Each configuration is a seven-tuple , where is the state of node v, .]

As shown in that table, at t = 1, nodes v1 and v2 get infected, v0 changes to state , and other nodes remain in state . Further, at t = 2, nodes v4 and v6 get infected, v1 and v2 change to state , and nodes v3 and v5 remain in state . Given the configuration at time t = 1, the probability that v4 gets infected at time t = 2 can be computed as follows. The infected neighbors of v4 are v1 and v2, and the transmission probabilities of the edges and are 3/4 and 1/2, respectively. Thus, the probability that v4 gets infected at time t = 2 is given by = 7/8. In a similar manner, the probability that node v6 gets infected at t = 2 is = 1/2.

We note that at t = 3, node v6 (which was infected at ) causes v3 to get infected. At t = 4, the system reaches a configuration with node v5 in state and all other nodes in state . Since this configuration does not have any node in state , no further state changes can occur; that is, the system has reached a fixed point.

Problem Definitions.

We introduce a number of short-term forecasting problems under the SIR model. (Similar problems can also be posed under other epidemic models mentioned in Section 1.) An example of a problem, denoted by pr-num-inf-at , under this model is the following: Given a social network, the set of initially infected nodes, a subset S of nodes, an integer , and a time value t, find the probability that at least q nodes of S get infected at time t. Another problem, denoted by pr-peak-inf-at , asks for the probability that the number of new infections reaches a peak at time t. Several such problems can be formulated. Table 1 gives concise descriptions of the main problems studied in our work.

Table 1

Concise descriptions of the forecasting problems considered in the paper

Problem name	Description
pr-num-inf-at  (t,q,S)	Compute the probability that the number of new infections within a given subset S of nodes at time t is at least q.
pr-num-inf-by  (t,q,S)	Compute the probability that the total number of infections within a given subset S of nodes by time t is at least q.
pr-inf-at  (t,S)	Compute the vulnerability of the nodes in subset S at time t (i.e., the probability that all the nodes in the subset S get infected at time t).
pr-inf-by  (t,S)	Compute the vulnerability of the nodes in S by time t (i.e., the probability that all the nodes in S get infected by time t).
pr-peak-inf-at  (t)	Compute the probability that the number of new infections in the network reaches a peak at time t.

Problems pr-inf-at and pr-inf-by are special versions of pr-num-inf-at and pr-num-inf-by , respectively, with . When S = V (the set of all nodes in the network), we denote the first four problems by pr-num-inf-at , pr-num-inf-by , pr-inf-at , and pr-inf-by , respectively. Results for these variants are stated in Table 3.

Table 3

Extensions of intractability results to more realistic networks

Problem	Result(s)
pr-num-inf-at  (2,q,V)	#P-hard even when both the diameter and the average clustering coefficient are 1(part 1 of Theorem 5.3).
pr-num-inf-by  (2,q,V)	#P-hard even when both the diameter and the average clustering coefficient are 1(part 2 of Theorem 5.3).
pr-num-inf-at  (t,q,V)	1. #P-hard for any t≥3, even when all edge probability values are equal to 0.5(Theorem 5.2).
	2. #P-hard for any t≥2 even for power-law graphs (Theorem 5.7).
pr-num-inf-by  (t,q,V)	#P-hard for any t≥3 even when all edge probability values are equal to 0.5(Theorem 5.2).

The theorems mentioned in the above table appear in the .

Summary of Results

To our knowledge, only a few references (44–46) have addressed the computational complexity issues related to forecasting under the SIR or other epidemic models. However, none of these references addresses short-term forecasting problems. Our contributions can be summarized as follows.

For general graphs, we establish computational intractability results for the first four problems in Table 1, namely

pr-num-inf-at ,

pr-num-inf-by ,

pr-inf-at , and

pr-inf-by ,

even when the time horizon t is as small as two. We also prove a result that provides an indication of the difficulty of obtaining approximate solutions to any of these problems for any . In addition, we demonstrate the tightness of our intractability results by showing that the four problems are efficiently solvable for t = 1. Further, we show that the pr-peak-inf-at problem is computationally intractable, even for t = 1. We also present a randomized approximation scheme for the problem pr-inf-by for any fixed t and any set S of fixed size. Table 2 provides formal statements of our results for general graphs.

Table 2

Results for forecasting problems over general graphs

Problem(s)	Result(s)
pr-num-inf-at  (t,q,S), pr-num-inf-by  (t,q,S), pr-inf-at  (t,S), and pr-inf-by  (t,S)	1. #P-hard for any t≥2 (part 1 of Theorem 4.1).2. For any t≥2, there is an ϵ>0 such that unless P = NP, the quantity log (2^n p^*) cannot be efficiently approximated to within the factor n^ϵ, where p^* is the solution value and n is the maximum number of nodes that can get infected at t = 1 (part 2 of Theorem 4.1).
pr-peak-inf-at  (t)	#P-hard for any t≥1 (part 3 of Theorem 4.1).
pr-num-inf-at  (t,q,S), pr-num-inf-by  (t,q,S), pr-inf-at  (t,S), and pr-inf-by  (t,S)	Efficiently solvable for t = 1 (Corollary 6.5).
Compute the expected number of new infections at time t	Efficiently solvable for t = 1 and t = 2 (Corollary 6.10).
pr-inf-by  (t,S)	Randomized approximation scheme for any fixed t and any set S of fixed size (Theorem 6.18).

The theorems and corollaries mentioned in the above table appear in .

We extend the above intractability results to more realistic social networks (e.g., networks with low diameter and high clustering coefficient or power-law networks) and to networks in which all the edges have the same transmission probability. These results are shown in Table 3.

We also present extensions of our results to prove the intractability of forecasting various epidemic measures (e.g., peak value, take-off value, or take-off time) introduced in ref. 47. These results are summarized in .

Further, we extend our intractability results to three other epidemic models, namely, SI, SIS, and PT. These results are summarized in

In ongoing work, we show that many of the forecasting problems under the SIR model can be solved efficiently when the treewidth of the underlying graph is bounded.

Implications of the Results

We briefly discuss the implications of our results with respect to the short-term forecasting problems considered in our work.

Unpredictability of Epidemic Dynamics Sets in at a Very Early Time.

Our results show that from a computational complexity point of view, epidemic dynamics become unpredictable, even when the time horizon is as small as two units. This is in contrast to previous complexity results on the unpredictability of dynamical systems that needed an exponential number of time steps for deterministic systems (e.g., PSPACE-hardness of reachability problems for dynamical systems shown in ref. 48) and a polynomial number of time steps for probabilistic systems (e.g., PSPACE-hardness of simulating quadratic dynamical systems shown in ref. 49).

Unpredictability Holds for both Macroscopic and Individual Properties.

Our computational intractability results hold for predicting macroscopic properties (e.g., finding the probability that there will be at least q new infections at any time ), as well as individual properties (e.g., finding the probability that a specific node is infected at time under the PT model) over a short time horizon.

Unpredictability Holds Even When Prior Knowledge Regarding the System Behavior Is Available.

Most of our computational intractability results hold for any time value . An examination of the proofs of these results shows that from a worst-case standpoint, the unpredictability results do not change, even if the behavior of the system is known for most previous time steps.

The Unpredictability Results Are Pervasive.

Our unpredictability results hold for a variety of problems and contagion propagation models. The problem variants include predicting the number of new infections at a certain time, the total number of infections up to a certain time, the time and size of the peak number of new infections, etc. The contagion models studied include SIR, SI, SIS, and PT. Furthermore, the complexity results hold for several classes of networks, such as power-law networks and small-world networks (e.g., complete graphs whose diameter is one).

One Random Parameter Is Adequate for Unpredictability.

Our complexity results rely on exactly one source of randomness, namely, the transmission probability. The results hold even when the remaining model parameters (e.g., the network structure and infectious period) are deterministic and do not vary over time.

Provable Predictability Results for Certain Special Cases.

For certain special forecasting problems, our results provide provably good approximations when the time horizon is fixed. Such special cases include computing the expected number of infections, the probability of a node being infected by a certain time, and the probability that the total number of infections exceeds a specified count by a certain time. (These results are presented in .)

For purposes of exposition, we presented our results in the context of epidemics. However, our results are applicable to other contagions in networked populations. Examples of such contagions include opinion/information propagation in social networks and malware propagation in computer networks. A good discussion of models for such contagions can be found in ref. 50.

Methods

We briefly summarize the methods used to establish the results mentioned in Section 3. The #P-hardness results and the difficulty of approximating the solution value shown in Table 2 are obtained through reductions from the problem of counting the number of satisfying assignments to a given monotone 2-satisfiability formula. The complexity of this problem was established in ref. 51, and the difficulty of obtaining provable approximations for the problem was established in ref. 52. For t = 1, we prove the efficient solvability of the first four problems in Table 2 by developing a dynamic programming algorithm for a more general problem that encompasses all four problems. The randomized approximation scheme for pr-inf-by (when t and are fixed) is based on a reduction to the problem of counting the number of satisfying assignments to a Boolean formula in disjunctive normal form. In establishing some of the results for realistic (e.g., small world) networks (Table 3), we use Fortuin–Kasteleyn–Ginibre inequalities from probability theory (53). The complexity results for various epidemic measures (presented in ) and for other epidemic models (presented in ) use appropriate modifications of the constructions used to prove the results mentioned in Table 2.

Discussion: Coping with Computational Intractability

We briefly discuss how some known approaches to cope with computational intractability are used in practice to solve problems in networked epidemiology.

One approach is to develop provably good approximations that work across a range of inputs by exploiting special properties of problem instances. Examples of this approach for the forecasting problems considered in our work are presented in . Recent examples of references that present approximation algorithms for optimization problems arising in epidemic dynamics include ref. 54 (which studies the problem of distributing vaccines under a budget constraint as a stochastic optimization problem) and ref. 55 (which considers the problem of developing strategies to decrease the spectral radius of the contact network as a way of reducing the spread of an epidemic). An approximation-based approach has also been used in the literature to reconstruct the history of an evolving network from a small collection of its snapshots over time. For example, an efficient algorithm that constructs an approximate history that is strongly correlated with the true evolution history of a network is presented in ref. 56.

A second approach is to consider structural restrictions on contact networks that can lead to efficient forecasting algorithms. As an example for this approach, our ongoing work has led to efficient algorithms for forecasting problems when the social contact networks have fixed treewidth. Such an approach has been used in the literature for a number of other computational problems in epidemiology. For example, Pinto et al. (57) consider the problem of locating the sources of infection in networks. The source-detection problem provides a way of formalizing the reverse contact-tracing approach used for epidemics (58, 59). While the forecasting problems considered in our work are based on forward trajectories with respect to time, the source-detection problem involves examining backward trajectories. As noted in ref. 60, the source-detection problem is difficult in general. Pinto et al. (57) present an efficient algorithm when the underlying network is a tree. Karrer and Newman (61) use a message-passing model for the computation of certain epidemic probabilities in networks. They observe that their approach gives exact values for tree networks and good upper bounds for general networks. Shrestha et al. (62) extend use of the message-passing approach to models such as SIS, where a node may return to the same state many times during the course of an epidemic; their results closely approximate those obtained by more computation-intensive Monte Carlo methods. Milling et al. (63) consider the problem of identifying the causative network—that is, the network on which an infection is spreading—given two candidate networks and a sample of infected nodes. For some special classes of graphs (namely d-dimensional grids and Erdös–Renyi random graphs), they provide sufficient conditions under the SI model for identifying the causative network with high probability. Altarelli et al. (64) consider the computationally hard problem of finding the marginals of certain probability distributions for an SIR epidemic over a network and develop an approach based on belief propagation (65). They show that their approach produces exact solutions for tree networks.

A third approach is to adapt heuristic techniques that work well in practice for general optimization problems. A number of such methods (e.g., genetic programming, simulated annealing, randomized search, and tabu search) have been developed for many optimization problems (66). Sophisticated forms of such approaches have been used by researchers for problems in computational epidemiology. For example, Shah and Zaman (67) use a maximum-likelihood approach in conjunction with network centrality measures to develop a highly effective algorithm to find the source of infections for a virus propagating through a network. Horn and Friedrich (68) employ a combination of random walks and the maximum-likelihood approach to develop an algorithm for locating the source of infections for foodborne diseases. Other researchers have used approaches based on message passing and belief propagation to obtain practical solutions to the problem of identifying the source of infection in general graphs (64, 69).

Finally, one can also try to reduce the time for computing forecasts by reducing the size of the contact network. One possible approach for this is to consider a higher-level aggregation of the underlying network (e.g., consider a network where each node represents a county instead of an individual). Moon et al. (70) use this approach for scaling up epidemic simulations. Finding the right level of aggregation that is appropriate for various disease parameters remains an important direction for future research.

Materials and Methods

Our results on the hardness of forecasting problems are obtained by using reductions from problems that are known to be computationally intractable. Our algorithms and approximation algorithms for restricted versions rely on standard techniques in algorithms for combinatorial problems. Our experimental results were generated by using simulation software written in Python.