On Ambiguity Reduction and the Role of Decision Analysis during the Pandemic.

The COVID-19 pandemic has created a multitude of decision problems for a variety of fields. Questions from the seriousness and breadth of the problem to the effectiveness of proposed mitigation measures have been raised. We assert that the decision sciences have a crucial role to play here, as the questions requiring answers involve complex decision making under both uncertainty and ambiguity. The collection, processing, and analysis of data is critical in providing a useful response-especially as information of fundamental importance to such decision making (base rates and transmission rates) is lacking. We propose that scarce testing resources should be diverted away from confirmatory analysis of symptomatic people, as laboratory diagnosis appears to have little decision value in treatment choice over clinical diagnosis in patients presenting with symptoms. In contrast, the exploratory use of testing resources to reduce ambiguity in estimates of the base rate of infection appears to have significant value and great practical import for public policy purposes. As these stances may be at odds with triage practices among medical practitioners, they highlight the important role the decision analyst can play in responding to the challenges of the COVID-19 pandemic.

INTRODUCTION

Fields such as medicine, economics, law, and many others must confront complex questions in responding to the global public health crisis that is the COVID‐19 pandemic. The decision sciences also have a central role to play. In fact, their collective role is (or should be) an overarching one. Each other discipline must confront decision problems that require dealing with uncertainty, making complex tradeoffs under imperfect information, and mediating between conflicting objectives—all skills seated firmly in the core of the decision analyst's repertoire.

Nearly every pandemic‐related decision problem facing society takes the form of contemplating some action in the presence of uncertainty about the prevalence and transmissibility of the coronavirus. Consider a person making a 500‐mile journey between two major cities. Ordinarily, this journey would be made by plane. But during the pandemic, travel in such enclosed proximity to other potentially infected people may have an unacceptably high risk. Travel by plane or car: what decision should be made?

This highly stylized problem could, of course, be reformulated to represent nearly any pandemic‐related problem facing an individual (Should I go grocery shopping?), a community (Should we reopen schools?), a business (Should we engage physically with customers?), or a nation (What mitigation measures should be proposed or required?). It is the process of making these questions explicit that reveals critical gaps in our knowledge.

We can formalize this problem as a decision tree (Fig. 1), which in turn reveals that parameters such as the probability of an accident during travel, as well as the probability of coming in contact with an infected person on the flight, are essential for reaching a reasoned choice. Given the tree, the decision analyst can then seek to estimate the required parameters. Some uncertainty quantifications are well supported with data, such as the risks associated with automobile and airline travel. Much less understood are the risks associated with infection and case resolution.

Fig 1

An example decision tree reflecting choice of travel mode and incurred risks related to coronavirus and accidents.

The value of the decision analyst becomes readily apparent. The decision analyst contributes a formal elucidation of the decision problems present in some situation in order to reveal: (i) the information available, (ii) the determination of the decision‐relevant information, (iii) the quantification of the relevant information and its uncertainty, (iv) the value of improving the information available, and (v) the mechanics of any tradeoffs involved. But this raises a question: when does the decision analyst get a seat at the table? When do decisions benefit from a decision analytic approach? We propose that the answer must be at the very beginning. This early participation is essential, and not only for individual decisionmakers. Decisions that depend on information about prevalence and transmissibility appear in many fields, multiplying the value of this information when considered across disciplines, and also for use with future decision problems.

The COVID‐19 pandemic is an ongoing, evolving challenge. Our purpose in this essay is fourfold. First, in Section 2, we highlight how little is known about the inputs required for good pandemic‐related decision making. Second, in Section 3, while many of the uncertainties involved will be resolved over time, we illustrate the challenges imposed by ambiguity in this environment during its early stages, and how more information can be obtained to reduce that ambiguity. Third, in Section 4, we highlight social and political issues that emerge from the presence of such additional information, which has decision‐theoretic value, but that may lead to adverse consequences deserving of public debate. Finally, we conclude in Section 5 by addressing the impact of these issues beyond the present pandemic and the lessons that can be learned in dealing with future instances of what we will call policy‐critical mass ignorance.

THE ABSENCE OF INFORMATION AND ITS CONSEQUENCES

What We Need

Amid the COVID‐19 pandemic, even the most routine decisions can have serious health implications. Unfortunately, among the most pressing challenges that we face is a lack of information. Imagine any sort of decision that one makes. The potential benefits of that decision must be balanced against the potential costs of that decision. If I go to the store, how likely is it that I will contract the virus? The probability of contracting the virus depends on two quantities. First, how likely is it that a person selected at random from a group of shoppers (in that store, at that time) has the virus? This is the base rate. Second, how likely is it that the virus is transmitted given that one comes in contact with an infected person? This is the transmission rate. The problem is that neither of these probabilities is currently known (with even a modest degree of confidence).

To date, much of the testing that has taken place has been among people who show symptoms. However, we know that it is possible for someone to have the virus and be contagious but be asymptomatic (Li et al., 2020). We also know that community spread is possible (Bendavid et al., 2020). But we do not know how prevalent it is. The headlines we see are—not surprisingly—based on the facts we have, but the picture they paint can be misleadingly incomplete.

Consider this April 2, 2020, news headline: “U.S. nears 1‐in‐1,000 infected” (Wallbank, 2020). As of April 1, 2020, the United States had approximately 213,000 confirmed cases (Center for Systems Science and Engineering, 2020). Did that mean that, in a country of 330 million people, the odds of someone being infected were approximately 213,000 in 330 million or 1 in 1,550? Of course not. Those figures reflect only the reported cases. The far more consequential (and decision‐relevant) figure must include the number of “hidden” cases. For every case that we do know about, there are more cases that we do not (Silverman, Hupert, & Washburne, 2020). Unaccounted “hidden” cases are estimated to exceed reported cases by as much as 20 times (Chow, Chang, Gerkin, & Vattikuti, 2020). These cases may be asymptomatic, and therefore untested, or symptomatic, but simply not yet diagnosed. Confirmed case data absent base rate information does not provide an accurate picture of risk. Estimates of the base rate vary widely by location and evolve over time. The estimates in Table I, which cover a period of several months early in the pandemic, range from 0.27% to 50%. As a result, there is considerable ambiguity (Ellsberg, 1961). There is “uncertainty about uncertainty” (Einhorn & Hogarth, 1986) with regard to the base‐rate estimates. In addition, the presence of a significant reference‐class problem only exacerbates this ambiguity. The methods used to derive the estimates vary. The construction of the samples used in the nonmodel methods varies.

Table I

Estimates of Coronavirus Base Rates

		Sample		Method
Area	Estimate (%)	Random	Nonrandom	Diagnostic	Antibody	Model
United Kingdoma	0.27	X		X
Austriab	0.33	X		X
Seville, Spainc	2.3	X			X
United Statesd	3.58					X
Francee	4.4					X
Miami, Floridaf	6	X			X
Madrid, Spaing	11.3	X			X
Gangelt, Germanyh	15	X			X
Diamond Princessi	17		X	X
United Kingdomj	29					X
Massachusettsk	32		X		X
United Kingdoml	>50					X

U.K. Office for National Statistics (2020, May 14).

Ogris (2020).

Instituto de Salud Carlos III (2020, May 13).

Bommer and Vollmer (2020).

Salje et al. (2020).

Miami‐Dade County (2020, April 24).

Instituto de Salud Carlos III (2020, May 13).

Streeck, Hartmann, Exner, and Schmid (2020, April 9).

Russell et al. (2020).

Stedman et al. (2020).

Saltzman (2020, April 17).

Lourenço et al. (2020, March 26).

A similar analysis can be done for estimates of the transmission rate. The question “what is the transmission rate?,” however, is not a well‐formed question. How is a “contact” defined? Physical touch? General proximity? Social distancing with a poorly fitted mask? How long a duration must such a contact involve? There are serious ethical concerns surrounding our ability to make empirical statements about transmission rate, in the sense that deliberately infecting test subjects with a deadly disease in a controlled environment is, in practice, impossible. And yet, how can reasonable decisions be made absent this information?

Notwithstanding these deep and hugely consequential challenges, estimates of transmission rates exist because they must. Our inability to say virtually anything useful absent such a figure creates an insatiable demand for such an estimate—regardless, perhaps, of its origins, as Funtowicz and Ravetz (1987) note:

In spite of these manifest inadequacies in the available information, the policy‐maker must frequently make some sort of decision without delay. The temptation for her/his advisors is to provide her/him with a single number, perhaps even embellished with precise confidence limits of the classic statistical form. When such numbers are brought into the public arena, debates may combine the ferocity of sectarian politics with the hyper‐sophistication of scholastic disputations. The scientific inputs then have the paradoxical property of promising objectivity and certainty by their form, but producing only greater contention by their substance. (p. 62)

And so, transmission‐rate estimates used in prepandemic modeling of the risks of transmitting a respiratory disease on an airplane flight (Hertzberg, Weiss, Elon, Si, & Norris, 2018) and evaluating the effectiveness of social distancing measures for a novel strain of influenza (Kelso, Milne, & Kelly, 2009) exist. These figures, then, take on a life of their own, separate from the methodological details of their origins.

For example, Kelso et al. (2009) base their transmission‐rate estimate on a figure derived from Milne, Kelso, Kelly, Huband, and McVernon (2008), which includes arbitrary parameters for variables such as “infectivity of the viral strain.” In contrast, Hertzberg et al. (2018) estimate the transmission rate on an airplane flight as less than 3% for passengers and crew, except for those within one row of an infected person, for whom the rate rises to more than 80%. Those empirical estimates from 10 flights, however, are based on using four times the estimate from a single paper (Moser et al., 1979) that itself had one data point: a single flight from an airplane with an inoperative ventilation system. The inferential value of these transmission‐rate estimates, therefore, is subject to question, but the estimates themselves are carried forward into pandemic policy without further consideration.

How valuable is this information? The information has value if knowing the base rate or transmission rate causes one to change decisions. Would you go to the store in search of hand sanitizer if you knew that 1% (or 20%) of the people in the store were infected? What if only 1% were infected, but the probability of transmission from mere proximity was 90%? Scale that problem to a national level, and it is easy to see that the value of this information is potentially enormous.

What We Have Instead

Instead of base rates and transmission rates, or even probability distributions of each, we have data dominated by observed outcomes: number of confirmed cases and deaths. While these data points are informative, they are not directly useful to decisionmakers absent further context. The risk in using outcomes alone, both absent the context of relative magnitudes (e.g., deaths per unit of population) and absent probabilistic uncertainty, is that extreme outcomes tend to exert an outsized influence on decision making (Lichtenstein, Slovic, Fischhoff, Layman, & Combs, 1978).

In lieu of the likelihood of infection, we have the consequences of infection. Policy positions that led to issuing stay‐at‐home orders and shutting down businesses emerged from forecasts that millions of people could die in the United States (e.g., Ferguson et al., 2020), but that did not address the probability of such a scenario occurring or the degree to which that outcome was likely to be avoided by mitigation measures. The potential death of millions, coupled with large estimated values of statistical life, produced estimates of avoided U.S. mortality losses alone of as much as $7.9 trillion (Greenstone & Nigam, 2020) that were in turn used to justify incurring the costs of aggressive social‐distancing measures such as business‐closure orders.

In lieu of transmission rates, we have measures of epidemicity, such as .1 This measure, which is the number of new infections caused, on average, by a single contagious person, is used in epidemiology models to forecast the spread of infection and the severity of consequences. And yet, the estimates of have themselves varied widely. Values greater than 1.0 imply exponential growth in the number of cases, but these estimates for COVID‐19 range from 1.4 to 2.5 (World Health Organization, 2020) to 2.24 to 3.58 (Zhao, Ran et al., 2020).2 This range of estimates is associated with a range of values for the base probability of transmission of more than a factor of three in some models (e.g., Milne et al., 2008).

These large estimates of potential consequences are then used to justify application of precautionary principle‐type logic that minimizes the role of likelihood in the decision‐making process. But without probabilistic estimates, virtually any suggested mitigation strategy can be justified. The inevitable consequence is akin to “security theater”—the practice of pursuing mitigation strategies that make people feel safer, but that may or may not actually make people safer.

ACQUIRING MORE INFORMATION AND THE DECISION ANALYST'S ROLE

The medical community faces an enormous and unenviable challenge as the problems posed by the pandemic may quickly outstrip its capacity to respond. Triage would seem to require that scarce testing resources be limited only to those displaying symptoms. On the other hand, how much additional information is obtained by testing someone who already displays symptoms? These problems are compounded by evidence that the existing testing has high rates of false negatives (Ai et al., 2020; Kucirka, Lauer, Laeyendecker, Boon, & Lessler, 2020).

We propose that there is considerable value in broad testing of random samples of the population. That is, testing randomly selected people, whether or not they display symptoms. Correct negative results are extremely informative, because they would reveal exactly how prevalent the virus is. We cannot make effective decisions in real time about social isolation and community spread without understanding the base rate of infections “in the wild.” How does risk vary across different subpopulations of the country, such as age groups, income levels, race, urban and rural populations, and state boundaries? Useful short‐, medium‐, and long‐range planning all hinge on accurate estimates of these base rates. The same high error rates that limit the diagnostic value of confirmatory testing for an individual become less relevant when the unit of analysis moves to the population. In particular, the degree to which any single test may be wrong becomes less important when the objective is to make public policy‐related decisions for a population (given a reasonable level of knowledge about the testing error rates).

Any attempt to solve a decision tree, such a Fig. 1, quickly runs into the absence of base‐rate and transmission‐rate data. In their absence, as we have noted, people have tended to draw inferences from other data points (such as the number of confirmed cases or deaths). But the “proper” figures have not been quantified. As this coronavirus is novel, we do not have meaningful prior information on which to base beliefs (such as the likely consequences of the annual flu season). Neither, however, is this a problem of “deep uncertainty,” in which there is no agreement on the relationship between actions and consequences (Lempert & Collins, 2007; Lempert, Groves, Popper, & Bankes, 2006). Instead, we refer to this type of problem as one of policy‐critical mass ignorance. That is, it is a problem of profound importance to the public, but about which virtually nothing (about the fundamental decision problem parameters) is currently known. What becomes critical, then, is the role that information acquisition plays in the decision analysis process.

If we are unable to quantify the key parameters because they are laden with ambiguity, we can either replace the probability distributions over the probabilities with their mean (Howard, 1988) or get more information. As tests are available to determine the presence of the virus (or its antibodies) in an individual, the information acquisition path is viable. The question is: is it useful?

The decision analyst's policy should allocate scarce testing resources to where the value of the test's information is highest. Although some may argue that the high rate of false negatives may lead to an explosion in “positive” cases, overwhelming health care resources (Briggs, 2020), the reality seems much different given the data available on the progression of COVID‐19 disease (Wu & McGoogan, 2020). The choice, then, is between testing as confirmatory (i.e., used to confirm that symptoms displayed indicate the presence of the virus) and testing as exploratory (i.e., used to improve estimates of the base rate of infection). Obviously doing “both” is an option, but we are assuming that testing resources are limited, and the costs are identical, and therefore seek to direct a marginal test to where the reduction in uncertainty is more valuable.3

One argument is that testing symptomatic people is not especially informative because the resulting diagnosis is unlikely to alter treatment decisions. Some doctors have suggested that patients should assume they have the virus if they exhibit symptoms even if their test results are negative (Krumholz, 2020). In addition, no reliable model exists to predict outcome or hospital admission at this time (Burrow, Treadwell, & Roberts, 2020). Indeed, the vast majority of cases (81%) resolve with only mild symptoms (Wu & McGoogan, 2020), and in such cases, the indicated treatment is focused on relieving the symptoms and monitoring for complications (COVID‐19 Treatment Guidelines Panel, 2020). Even the treatment for more serious cases is based primarily on supportive management of the complications. If confirmation of a diagnosis via test does not alter the course of treatment, and further, the consequences of the proposed treatments are benign for patients that are true negatives, then confirmatory testing has little value. The decision rule is simple: if the patient has symptoms, assume the patient is positive and treat accordingly.

We can formalize this thinking. Consider the decision tree in Fig. 2 and assume there are two populations: symptomatic and asymptomatic people. Members of both groups may have the virus. Suppose, however, that the mean base rate among symptomatic people is high (say, 0.90), but the mean base rate among asymptomatic people is low (say, 0.20). In both cases, however, there is ambiguity. The high base rate could be either 0.85 or 0.95 and the low base rate either 0.05 or 0.35, each with equal likelihood. For outcomes, we assume that being negative is preferred to being positive, and that if you are positive you prefer to undergo treatment, while if you are negative, you prefer not to undergo treatment. It will be sufficient for this simple example to consider a ranking of outcomes such that Outcome I (Don't Treat, Negative) is preferred to Outcome IV (Don't Treat, Positive) and Outcomes II and III are in the middle.

Fig 2

An example decision tree evaluating whether to treat a patient or not given ambiguity over the base rate of infection.

Should more information be acquired, and where should information acquisition activity be directed? Is it more valuable to have information about individual cases or about general base rates? In this setting, an individual in either population would always value information about their own test results more highly than information about the base rate. However, reducing the ambiguity in the base rate (i.e., acquiring perfect information about the base rate) has value to the asymptomatic population.

More importantly, perhaps, is that from a societal perspective, there is clearly decision‐making value in acquiring base rate information about asymptomatic patients. There may be no reason to test symptomatic patients (ethical issues aside), as confirmation of a clinical diagnosis does not appear to alter treatment decisions, but randomized testing of asymptomatic patients is likely to produce very useful information for public health measures, mitigation measure effectiveness, and community spread unless the base rate among even asymptomatic people is very high.

This sort of thinking may run counter to triage practices among medical practitioners but would be apparent from a decision analysis perspective. Obviously, this simple problem can be scaled up to an arbitrary level of complexity and be modified to deal with specificity and sensitivity parameters.4 The problem can also be reformulated should an effective antiviral treatment become available, in which case testing symptomatic people may take on more value. Our objective here, however, is not a full study of this problem, but rather to indicate in simple terms that base‐rate information has value, both from a societal perspective, but also from the perspective of the asymptomatic public. This benefit can be seen, for example, in contrasting the Spring 2020 and Fall 2020 semesters at American universities. Upon the initial outbreak of the virus in the Spring 2020 semester, many universities completely shut down. By the following Fall semester, with the benefit of additional information from random testing, most universities were able to open safely to students on campus to some extent. With imperfect testing, consideration of health‐care resource exhaustion (from false positives) and unchecked spread (from false negatives), as highlighted by Briggs (2020), are important. But the existing treatment guidelines suggest such a risk is minimal. The decision analyst's contribution, therefore, brings a meaningful and informative perspective to this analysis at the very earliest stages of such a problem.

THE PRESENCE OF INFORMATION AND ITS CONSEQUENCES

Once acquired, information must be interpreted and used properly in order to be effective. Should protocols be designed to detect where differences occur (whether by state, city, or zip code)? While the theoretical benefits of the information acquired from greater testing can be determined, the collection and use of that information also involves potential negative consequences, which must be understood, communicated, and balanced against the benefits. As the country's testing abilities grow and our information level increases, a whole different set of difficult questions emerges: what do we do with the new information? Amid concerns about a second (and third) wave of infection, these questions take on a heightened relevance.

Disclosure informs, but it also divides. Large data sets are being collected by governments (Toh, 2020). Dr. Anthony Fauci, the director of the National Institute of Allergy and Infectious Diseases, suggested that the government would consider issuing “certificates of immunity” to those with certain antibodies (Forgey, 2020), notwithstanding evidence that such immunity may be short‐lived (Edridge et al., 2020; Seow et al., 2020). Apple and Google are cooperating on software to track the contacts of infected people (Fried, 2020). It has been suggested that minority communities are disproportionately at risk from the coronavirus (Mays & Newman, 2020) and the federal government is coming under pressure from some in Congress to track infections by race (Williams, 2020). How will the information from more widespread testing be used? Does more frequent testing create a “scarlet letter” of sorts that stigmatizes such communities?

There is what economists would call an “unraveling” principle (Grossman & Hart, 1980) at work here in that disclosure benefits those with favorable positive test results. But differences in infection rates could be caused by economic factors (wealthier families can purchase better protection, have better access to health care, and can avoid high‐risk alternatives like public transportation). Must lower‐income community members bear the burden of limited access to medical, commercial, and public services? Should “COVID‐19 profiling” or “redlining” be illegal even if it is useful in limiting community spread?

Information may have value, but its revelation may be costly. Could businesses limit services or schools limit access only to people with a “certificate of immunity”? What if infection rates by zip code were available? Could that information be used to limit access to (or restrict access from) certain areas? Contact tracers in New York City—essential for tracking and limiting community spread—were instructed not to ask about attendance at protests (Smith, 2020). In attempting to protect the privacy rights of individuals, the government was not (in that instance) collecting information (i.e., attendance at protests) that had decision‐making value. The inherent difficulty in determining an appropriate balance of the need for decision‐relevant information with respect for legitimate concerns surrounding privacy and free expression is made even more difficult when accurate population base rate information is unavailable. Private businesses are making similar decisions. Airbnb, for example, has prohibited hosts from referencing COVID‐19 status in posts or suggesting that properties (or hosts) are virus free (Airbnb, 2020). There is an important balancing of costs and benefits that deserves both rigorous analysis and public discussion.

To be clear, we believe fervently in the benefits of more testing and more information. It is often believed that information cannot have a negative value. Here, however, we have already seen in practice examples of base rate information that has a positive value (in the sense that it aids in decision making), but also has the potential for negative consequences. In the rush to collect more information—which is desperately needed—there deserves to be a public conversation about the issues involved with access, disclosure, privacy, and use.

POLICY‐CRITICAL MASS IGNORANCE AND THE ROLE OF THE DECISION ANALYST

The ubiquity of everyday decision problems that depend on estimates of the epidemic base rates and transmission rates suggests that the value of that information is considerable. Problems characterized by policy‐critical mass ignorance highlight not just the uncertainties and ambiguities present in a decision problem, but also the value of collecting and analyzing information in a problem environment where virtually nothing about the problem parameters is initially known. The role of the decision analyst is especially important in these types of problems ab initio because how the information is acquired and interpreted critically shapes its analysis. Further, because of the novelty of these types of problems, the benefit is doubled as the information acquired is of both practical (in the current problem) and inferential (in future similar problems) use. For example, detecting and understanding the source of a second wave.

Our analysis suggests that the use of scarce testing resources on symptomatic patients, in this instance, may not be their most valuable role. Patients presenting clinically with symptoms may be best served by proceeding directly to treatment, with testing resources deployed instead to reduce base‐rate ambiguity within the asymptomatic population. Ambiguity reduction has the potential to produce significant societal gains, and the decision sciences are best positioned to assist in realizing them.

COVID‐19—and future viruses like it—are public health crises and solid, objective data are necessary to confront them. Reducing the ambiguity surrounding base rates and transmission rates is of considerable value to creating public health policy when testing is properly directed. But selective or misinterpreted data can also be a virus. Released unintentionally or maliciously across social media, this “information virus” could cause societal harm to linger long after the physical harm is resolved. Careful application of the decision analyst's tools is essential to traverse this uncharted territory.

CONFLICT OF INTEREST STATEMENT

The authors declare that they have no conflict of interest.

FUNDING STATEMENT

This research did not receive any specific grant from funding agencies in the public, commercial, or not‐for‐profit sectors.