The road from evidence to policies and the erosion of the standards of democratic scrutiny in the COVID-19 pandemic.

The COVID-19 pandemic poses extraordinary public health challenges. In order to respond to such challenges, most democracies have relied on so-called 'evidence-based' policies, which supposedly devolve to science the burden of their justification. However, the biomedical sciences can only provide a theory-laden evidential basis, while reliable statistical data for policy support is often scarce. Therefore, scientific evidence alone cannot legitimise COVID-19 public health policies, which are ultimately based on political decisions. Given this inevitable input on policy-making, the risk of arbitrariness is ubiquitous and democratic scrutiny becomes essential to counter it. During the COVID-19 pandemic, the standards of scientific and democratic scrutiny have been, as a matter of fact, substantially lowered. This erosion potentially damages democracy.

Introduction

When the COVID-19 pandemic burst, decision-takers endorsed a precautionary approach in order to justify lockdown policies, the effects of which were largely unknown to democratic societies. As subsequent waves of infection took hold, governments introduced more targeted policies in an effort to reduce the societal impact of lockdowns. By focusing on a set of epidemiological and healthcare indicators (such as fatality rate, infection rate, reproduction number, positivity rate, ICU hospitalisations etc.), the evidential basis for policy-making has been gradually refined. Nevertheless, the definitions of such indicators are not theory-free and reliable data statistically linking such indicators to the policies’ outputs is generally meagre. Trivially but notably, this means that there is no such thing as objective, theory-free scientific knowledge from which policies automatically stem. Indeed, policies are fundamentally based on political decisions that should be, as such, thoroughly scrutinised. It is for this reason that the current erosion of the standards of democratic scrutiny is potentially damaging.

The evidence used in policy-making is theory-laden at every level of analysis

The concepts of fatality and infection rate have been central in public health policy (Verity, 2020). Far from being determined by theory-free biomedical knowledge, these concepts are operationally defined, a methodological procedure requiring theoretical choices. It is not surprising, therefore, that methodologies for estimating COVID-19′s fatality rates remain unstandardized. In July 2020, for instance, Public Health England1 decided to reduce from 60 to 28 days the timespan from detected Sars-CoV-2 infection to death in order to consider COVID-19 as the cause of the latter. This definitional change led to a substantial decrease (over 5000 people) in COVID-19-associated death count.2 For other diseases, reliable fatality rate estimates are generally available only with substantial time lags. In striking contrast, COVID-19 deaths are communicated daily. As a result, fatality rate estimates have varied tremendously during the course of the pandemic: considered to be above 10% in northern Italy and the Madrid region at the beginning of the pandemic, fatality rate is currently estimated between 0.27 and 0.57% (Ioannidis, 2020). Thus, governmental policies that rely on fatality rate estimates are likewise dependent on specific operational definitions of the concept of death-by-COVID-19 (such as ‘death within n days from detection of infection’) and the estimation methodology employed, which is contingent on the chosen time frame and requires independent statistical justification.

Unsurprisingly, theory-ladeness also affects the concept of infection rate. A COVID-19 case is defined by reference to a positive PCR (Polymerase Chain Reaction) test result rather than by observable clinical symptoms. Such a molecular definition, which curiously implies that a COVID-19 asymptomatic person is a case, hinges on the intricacies of PCR technology, which multiplies fragments of the viral RNA through several amplifying cycles. The number of amplification cycles selected can influence the test result, with a positive correlation between the number of cycles and the probability of false positives. Furthermore, after 24 amplification cycles, samples do not exhibit viral growth (Bullard et al. 2020). Indeed, no study we know of correlates PCR positivity and infectivity (Jefferson et al. 2020), which is only detectable through viral culture. Again, governments’ policy reliance on infection rate estimates depends on specific operational definitions of the concept of COVID-19-case, such as a positive PCR test, as well as on the testing methodology employed, which is dependent on the number of PCR cycles considered relevant.

Policy-making is often impaired by the lack of reliable statistical data

Moreover, the statistical grounding for policies is frequently flimsy. Face masks, for example, are believed to reduce infections based on intuitive mechanistic considerations; however, no randomised clinical trials were available at the beginning of the pandemic, while some recent studies do not show a statistically significant reduction in transmission (Frieden and Shama 2020; Olilla et al. 2021). The same lack of clear statistical correlation has characterised all major aspects of policy-making, including the relationship between lockdowns and reduction of excess mortality (Candel et al. 2021). As the authors of a comprehensive systematic review concerning the putative causal role of schools in driving contagion put it: “Our results are consistent with school closures being ineffective to very effective” (Walsh et al. 2021, p. 2). This ironic understatement epitomises the current state of affairs, with local and national governments adopting an idiosyncratic patchwork of policies. Solid statistical data on infection fatality rate; the usefulness of face masks and lockdowns; and the role of schools in transmission dynamics will most probably only emerge in the coming years. This partial limitation of Big Data is not surprising: the influenza fatality rate, for instance, can only be calculated with reliability in terms of excess deaths vis-à-vis a substantial temporal period of reference, usually the last 5 years (Rosano et al. 2019). The general implication is that, both in data gathering and interpretation, as well as in the chain leading from data to policies, there is inevitably space for arbitrariness in policy-making (e.g., prioritising specific epidemiological indicators instead of others for obscure reasons). In order to avert the risk of arbitrariness, governments must be held accountable and rigorous policy review mechanisms should be in place. Throughout the COVID-19 pandemic, however, the standards of scrutiny have been substantially lowered.

Scrutiny should be enhanced rather than curtailed

It seems to us crucial that governments uphold two fundamental responsibilities with respect to their citizenry: first of all, to make the evidential basis underlying their policies transparent; and secondly, to clarify the extra-scientific rationale of the same. During the pandemic, we observe that governments have, to varying degrees, fared badly on both counts: parliamentary control has regularly been bypassed by resorting to decree legislation, hampering the critical role of opposition; the evidential basis of policies has generally not been made publicly available, impeding the democratisation of the policy advice process; the ethical framing of policies has generally remained concealed, crippling any serious debate concerning the sustainability of their targets; the goal of vaccination programmes is often cryptic, incentivising mis-prioritisation and flagrant abuse. Furthermore, this erosion of the standards of democratic scrutiny has occurred within a context already characterised by the temporary suspension of the scientific peer-review process (London & Kimmelman, 2020). We maintain that the erosion of the standards of scrutiny: widens the gap between citizens and decision-makers; reduces understanding of and confidence in policies’ effectiveness; increases disenfranchisement; in brief, it potentially undermines the foundations of democracy. We do not deny that exceptional circumstances require exceptional policies; we rather suggest that transparency should be directly proportional to their exceptionality. This is particularly the case in circumstances that endow governments with emergency powers and entrust them with policy decisions that affect citizenry so dramatically.