Interpreting SARS-CoV-2 seroprevalence, deaths, and fatality rate - Making a case for standardized reporting to improve communication.

The SARS-CoV-2 virus has spread across the world, testing each nation's ability to understand the state of the pandemic in their country and control it. As we looked into the epidemiological data to uncover the impact of the COVID-19 pandemic, we discovered that critical metadata is missing which is meant to give context to epidemiological parameters. In this review, we identify key metadata for the COVID-19 fatality rate after a thorough analysis of mathematical models, serology-informed studies and determinants of causes of death for the COVID-19 pandemic. In doing so, we find reasons to establish a set of standard-based guidelines to record and report the data from epidemiological studies. Additionally, we discuss why standardizing nomenclature is be a necessary component of these guidelines to improve communication and reproducibility. The goal of establishing these guidelines is to facilitate the interpretation of COVID-19 epidemiological findings and data by the general public, health officials, policymakers and fellow researchers. Our suggestions may not address all aspects of this issue; rather, they are meant to be the foundation for which experts can establish and encourage future guidelines throughout the appropriate communities.

Introduction

The virus causing the Coronavirus Disease 2019 (COVID-19) pandemic, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was first identified in December 2019 and has now infected people worldwide. We observe significant differences in the risk of dying from COVID-19 when comparing the numbers of cases and deaths reported in different cities, states and countries. For example, at the end of May 2020, the proportion of total deaths in the age group of 80 years was 73.4% in the UK, 43.8% in Ireland, and 8.3% in Mexico [1]. Does this mean the virus is more deadly for elders in one place than another?

The reported headline figures are affected by several factors that vary from one specific location to another. These factors include the number of people tested for the virus, the access to healthcare, local social distancing guidelines, how a COVID-19 death is defined and the proportion of the population who are especially vulnerable to the virus, among others. Understanding the differences in COVID-19 fatality rates between regions and countries requires careful interpretation of how seroprevalence, the percent of the population positive for infection, is estimated and how healthcare providers record and report the numbers of cases and deaths.

The case fatality rate (CFR), given by the ratio of deaths divided by the number of documented infections, was 12.2% in Italy, 4.9% in Spain, 3.0% in Brazil, and 3.0% in the US as of September 17, 2020 [2]. Just two months later on November 12, 2020, these numbers converged. The CFR for each was 4.9%, 2.8%, 2.9%, and 2.3%, respectively. How do we determine which numbers accurately assess the danger and risk this virus poses? To have a reasonable estimate of the actual risk of death, we need to understand the difference between the CFR and the infection fatality rate (IFR). As opposed to the CFR, the IFR is given by the ratio of deaths divided by the total number of actual infections by SARS-CoV-2 in the population. In an ideal world, every individual would be tested, and the CFR and IFR would converge to the same number. Without this information, we lean on estimates for the total number of infections.

The CFR has the benefit of being calculated with raw data and can be useful to determine how well hospitals treat COVID-19 cases. Therefore, what might explain the decline in CFR mentioned above are the large strides we have made in the treatment of the disease and protection of those who are vulnerable. Yet, because the CFR does not consider the portion of asymptomatic and mild undocumented infections, it still underestimates the total number of people infected and overestimates the disease’s actual mortality in broader applications. Despite this shortcoming, CFR has been the most commonly used value when referring to the COVID-19 pandemic’s mortality risk.

The IFR, in contrast, estimates the disease mortality by considering the total number of infected individuals. As such, it is essential to have trustworthy estimates of the IFR, so policymakers on local, state, and federal levels can make informed decisions. The challenge is that it is very difficult to determine the total number of infected individuals unless systematic sampling and sophisticated statistical inferences are carried out.

The question now is how exactly do we make reliable IFR estimates? Studies that attempt to tackle this problem use fundamental assumptions to carry out fatality rate estimations. These assumptions must be identified and documented to adjust for their effects on the field’s output measurements. Thus, when different results emerge for studies aimed at measuring the same phenomena, the reasons they differ can be identified. When done properly, diligent reporting of a study’s assumptions, methods and results can lead to a better understanding of the origins of the study’s findings and how they might be applied to other circumstances. Accordingly, for a pandemic as widespread as COVID-19, reporting the critical assumptions, variables and contextual details – metadata – is critical. Every value, from IFR to hospitalization rate, describing the state of the pandemic has several different estimates, each showing COVID-19 in a different light. Not being able to properly interrelate and apply this data can cost lives.

Our analysis dissects epidemiological studies related to COVID-19 and brings to light the key determinants of how the SARS-CoV-2 virus spreads and causes deaths. Based on this analysis, we identify factors that contextualize the fatality rate estimates. These factors are critical to fully understanding the origin of the rate’s qualitative value. The idea of reporting the metadata for estimates can be extended to other epidemiological parameters. We believe the implementation of core metrology principles in epidemiology can help explain the discrepancies between reported pandemic values and improve how policymakers, the media and the general public use the data measured by healthcare providers and epidemiologists. First, we will delve into the uncertainties of the pandemic’s spread as we understand it today.

The problem lies in documenting and estimating infection

While the number of deaths is relatively concrete, the large number of infections with mild or no symptoms can leave many infections hidden from data banks. Thus, estimations of the seroprevalence are both important and complex. Yet, when attempting to decipher the number of infections in a population, the impact of more minute details is often overlooked. It is important to first understand the characteristics of a population before deliberating on the many studies that estimate the virus’s total spread. Then, one can more objectively assess how the metadata of a study has affected the results. Since the start of the pandemic, serology-informed studies have been used to estimate seroprevalence. With information from serology studies, virus transmission, and deaths, models can estimate or forecast the seroprevalence and pandemic dynamics. Serology-informed studies and models have differences and similarities in how they must be analyzed to maximize their applicability and accuracy within a given context. We will begin with mathematical models.

How are models utilized to estimate the infection fatality rates?

Parameters are critical for the analysis of models intended to estimate and forecast the dynamics of any pandemic. The parameter types and values are intrinsically linked to the contextual and environmental details of a lab or field study. It can be difficult to ensure the accuracy of parameters; nevertheless, the source of the parameter derivation can and should be reported. Implementing this reasoning is critical not only for those creating the model but also for those looking to apply the model. It is not lost on us that models and their assumptions come with their inherent variability and complexity. It is our goal to help diminish uncertainty and expand the number of well-informed predictions.

A study by Ioannidis, et al. [3] reviewed the state of COVID-19 models as of late August 2020 and identified factors that lead to poor forecasting. Table 3 in the paper by Ioannidis, et al. [3] lists potential reasons for the failure of COVID-19 forecasting and most are rooted in the quality of the parameters. Included in the table are examples of poor data input on key features of the pandemic (such as inflated mortality and transmission rates), incorrect assumptions regarding the demographic heterogeneity of populations, lack of incorporation of epidemiological features (such as age structure and comorbidities), poor use of past evidence on the effects of current interventions (using observational data of questionable quality and applicability to the current pandemic circumstance) and examining only one or a few dimensions of the problem (no consideration of other potential conflicting factors). Models must be informed on the key determinants of the pandemic and provide transparency on the parameter derivations to ensure accurate interpretation by readers.

Awareness of the various determinants playing into the pandemic is a critical first step for those forecasting and estimating values. Values can be cherry-picked from studies without reporting the metadata that supports these parameter values, and long-term consequences can arise from hasty and uninformed decision-making by readers of those published findings. In epidemiology and public health, forecasting models are often published without supporting metadata for their parameters. As a précis, models [4], [5], [6], [7], [8] used earlier in the pandemic to make forecasts lacked critical metadata. We must encourage accountability and rigor in models to minimize the potential for negative health impacts on populations.

How are serology studies utilized to estimate the infection fatality rates?

While models give insight into the pandemic’s general dynamics, accurately estimating seroprevalence is essential for the implementation of forecasting models. The challenge is that seroprevalence estimates are difficult to obtain as they have many variables that come into play. The goal of serology studies is to provide a better look at the pandemic’s spread by sampling directly from the population. Ideally, those conducting this type of study collect blood samples from a large set of people, representative of the broader population, and test for antibodies against the disease pathogen.

One example is a serology-based study in Spain [9], which tested 35,883 households in the non-institutionalized population (i.e., excluding people in hospitals, prisons, convents, nursing homes and other collective residencies) for antibodies against the SARS-CoV-2 spike protein. This study was conducted in late May 2020, when Spain had strict social distancing guidelines and had its steepest rise in cases before the recent resurgence seen throughout the world. The study reported an estimated seroprevalence of 5.0% for the entire Spanish population of 46.9 million. With a death count of 26,920 as of May 11, 2020, the IFR was 1.15% [10]. This describes a broad overview of how a serology study will estimate the IFR of a population.

How do we determine the accuracy of IFR estimates?

Meta-analyses of seroprevalence studies exhibit a range of values for the IFR. Over time, the portion of the population that is infected can increase and decrease, as well as spread into new groups that vary in vulnerability. This results in a dynamic mortality rate. Many studies have been conducted in different countries, sometimes even multiple in one region. If the goal is to determine what causes variability in the IFR estimates, whether it is the study’s methods, location, time or population demographics, then we must be aware of these details for each study. Here we will focus on two meta-analyses.

Meyerowitz-Katz & Merone [11] collected and sorted through articles using data from February to April 2020 and arrived at 13 total estimates (8 modeled estimates and 5 observational estimates). The overall IFR estimate was 0.75% (95% CI: 0.49%–1.01%), and there was no detectable pattern among the locations of the studies, the dates, or types of study. Additionally, they reported a high heterogeneity (I2 exceeding 99%) within the data, suggesting the point estimates for IFR used may not be reliable. Meyerowitz-Katz & Merone [11] mention the lack of age-stratified data and the variability of methods across the studies as possible reasons for skewing the data either higher or lower.

Ioannidis [10] also performed a meta-analysis of 36 seroprevalence studies from across the globe published from early April to early July 2020. There are just two articles [12], [13] used in both the Meyerowitz-Katz & Merone [11] and Ioannidis [10] studies. The Ioannidis [10] meta-analysis extracted the location, recruitment and sampling strategy, dates, sample size, types of antibody (IgG, IgM, IgA), estimated crude seroprevalence and adjusted seroprevalence. Additionally, the paper extracts the reasons for adjusting the seroprevalence to vent for the factors that cause uncertainty. Only studies with a sample that approximates the general population and with a size of at least 500 were included. The paper also corrected the IFR estimates based on the number of antibodies tested for by dividing each estimated IFR by 1.1 for every antibody they did not test for. The seroprevalence estimates varied widely, ranging from 0.222% in Rio Grande do Sul, Brazil [14] to 47% in Brooklyn, New York [15]. IFR estimates converged to a tighter range of 0.02% in Kobe, Japan [16] to 1.63% in Louisiana, USA [17], excluding the four 0.00% IFR estimates where deaths were insignificant or zero. The median IFR estimate across the 32 locations was 0.27%. With the large range of seroprevalence and IFR estimates, readers and experimentalists require a way to filter for inaccuracies and robustness. The next section of this article will assist by investigating the factors we believe to be causing the uncertainty.

Determining the cause for inconsistency in fatality rates… Is it geography or serology?

The answer is yes to both. There are innate differences in the population concerning the spreading event, population health and demographics that lead to the wide range of seroprevalence estimates and slightly smaller, yet significant, range of IFR estimates. If one serology study takes samples from healthy blood donors, one from the general population and another from outpatients at the local hospital, how can we be sure these geographical differences are the sole cause for the difference in the mortality rate of COVID-19. Ioannidis’ meta-analysis does well to include each study’s methods of recruitment, sample population demographics, test performance and how the virus spread in the study location. With this information, we can properly assess the significance of each estimate and form a more complete picture of the risk involved with the COVID-19 pandemic.

For example, 7 [18], [19], [20], [21], [22], [23], [24] of the 36 studies in the Ioannidis [10] meta-analysis use blood samples exclusively taken from blood donors. Blood donors are often required to be healthy and can exclude those who have had any signs of illness in the past two weeks. This sample is biased towards healthier individuals, who are not representative of the general population, resulting in an underestimation of seroprevalence and an overestimation of the location’s IFR. Two-hundred blood donors in Oise, France [25] gave a seroprevalence estimate of 3% while students, siblings, parents, teachers and staff in the same area recorded a seroprevalence of 25.9%. Thus, the sampling methodology can distort IFR estimates. Additionally, 5 [12], [17], [26], [27], [28] of the 36 studies focused on locations with a death count much higher than other locations within their respective countries. Locations with these discrepant numbers of deaths will lead to an overestimation of IFR. Most studies recognize the faults in their sample population and perform corrections to the data to account for the defect; however, these corrections are conjectures. There is no exact measure of the extent these factors have changed the results. More importantly, the factor(s) accounted for vary between studies. This inconsistency makes conducting a meta-analysis difficult, and it is at the root of the problem of putting IFR estimations into action. It also poses challenges in determining the best parameters to introduce in mathematical models in order to make epidemiological forecasts or investigate outcomes for different interventions.

What factors limit the accuracy of IFR estimations?

The following sections explore some of the important determinants and how they affect the IFR estimation. For estimations to be interpreted in the correct context and accurately generalized, the environment and method of analysis of the study must be discussed. Few studies, if any, acknowledge and account for the many elements that shape their results. This is understandable, as there are many factors and they come from different angles. The goal of the following sections is to elucidate the degree to which these factors can affect the IFR. By doing so, we hope to make it clear why they are important to consider and report.

How does the IFR vary based on age, comorbidities, and demographics?

It is well known that SARS-CoV-2 has a steep gradient in risk of death when it comes to age, demographics and comorbidities. More specifically, the mortality risk increases for the elderly (age >65), those with underlying conditions and those of lower socioeconomic status. By now, the vulnerability of the above groups in this pandemic is common knowledge; however, less well-known is the degree to which each affects COVID-19 numbers. Below we will discuss each topic and their numbers to help clarify their respective effects on the COVID-19 pandemic.

Evidence of how the IFR can change depending on age is also found in the serology-informed article by Ioannidis [10]. Among the study’s lowest IFR estimates, at 0.08%, was Iran [27], which despite a seroprevalence of 33%, maintained a low IFR due to its very young population, with only slightly more than 1% above the age of 80. IFR estimates for the <70 age group were lower than 0.1% in all but seven locations (Belgium, Wuhan, Italy, Spain, Connecticut, Louisiana, New York), where all seven were hotbed cities of the virus at the time. There was a median of 0.05% across all locations for the <70 age group, significantly lower than the overall median of 0.27%. Additionally, a serology study in Geneva [26] estimated the overall IFR to be 0.64%, yet for ages <50 years was <0.01%, for ages 50–64 years was 0.14%, and for ages >65 years was 5.6%. As of December 2020, the Centers for Disease Control and Prevention (CDC) Pandemic Planning Scenarios [29] sources the Hauser et al. study [30] as its best IFR estimates per age group: 0–19 years = 0.003%, 20–49 years = 0.02%, 50–69 years = 0.5%, 70+ years = 5.4%. The observed trends suggest that the mortality risk increases exponentially with age. The study by Ioannidis [1] looks at the COVID-19 deaths within eight European countries and the US confirms this exponential increase in death rate for both males and females. This increase with age can also be seen in Figure 1 of the article by Guilmoto [31]. When the virus finds ways to attack the older population, the number of deaths in elder populations will far surpass the younger and the resulting overall, age-unadjusted IFR will begin to lose quality.

The significant age-related difference in mortality risk is reinforced by the sizeable portion of COVID-19 deaths coming from long-term care facilities (or nursing homes) relative to the total portion of infections the facilities contribute. In association with the International Long-Term Care Policy Network, Comas-Herrera et al. [32] gathered evidence on long-term care facilities as they relate to COVID-19 from 26 countries where official sources made the data available. After considering the many different methods each country has taken in defining a COVID-19 death, a COVID-19 long-term care facility death and long-term care facilities themselves, the study estimates that 46% of all COVID-19 deaths have come from long-term care facilities residents based on data from 21 countries. In the US, long-term care facilities contributed to 41% of the total COVID-19 deaths as of late September 2020. This trend is relatively common in the study across countries with more than 5000 total deaths that range from 39% of deaths in Germany to 80% in Canada, and anomalies to this trend are found only in countries with less than 1000 total deaths. The disproportionate contribution of long-term care facilities to COVID-19 deaths is reasonable considering the fragility of these residents’ lives. The average length of stay in nursing homes is 2 years and people who die in nursing homes die in a median of 5 months [33]. This suggests that the deaths of people in nursing homes largely affects the COVID-19 fatality.

In the meta-analysis by Ioannidis [10], three studies taking place in New York [15], [34], [35] show high overall IFR values of 0.4% [15], 0.68% [34], and 0.65% [35]. A possible explanation for this high mortality risk would be the decision by the New York governor to allow excess COVID-19 patients to find care in nursing homes. It is not unexpected that people in nursing homes, who are there typically due to poor health conditions, would be more susceptible to this virus. Given their major contribution to the total amount of COVID-19 deaths in most countries, their under-representation throughout the majority of serology-informed studies is cause for concern. Age-specific calculations of the IFR can minimize the effect of neglecting nursing homes, but if seroprevalence in this institutionalized population is higher than in the general population, it could lead to overestimation of the IFR. Already mentioned in this paper was a large study that does not include this group of individuals, the Spain study [9]. This study addresses many of the factors we discuss in this review, yet its inability to include institutionalized individuals may be affecting the accuracy of their results more than they initially perceive.

Underlying health conditions, such as cardiovascular disease, hypertension, diabetes, chronic obstructive pulmonary disease, severe asthma, kidney failure, severe liver disease, immunodeficiency and malignancy [1] have been linked to an increased fatality risk when infected with COVID-19. These comorbidities create another at-risk group, in addition to the elderly, that must be treated with caution. The comorbidity factor contributes significantly to the interpretation of deaths that occur in the <65 age group. The age-stratified analysis on COVID-19 mortality risk by Ioannidis [1] in the early pandemic on 11 European countries, Canada, Mexico, India and 13 US states showed a small fraction of total deaths attributable to non-elderly people with no underlying conditions. A range of 4.5% to 11.2%, in European countries and Canada, and 8.3% to 22.7%, in US locations, was identified as the percent of total COVID-19 deaths in people below the age of 65. In Mexico and India, however, non-elderly individuals constitute the majority of the population. A noteworthy result regarding the impact of underlying diseases is that the study showed the proportion of total COVID-19 deaths linked to non-elderly people without underlying conditions ranged from just 0.65% to 3.6%, where data was available (France, Italy, Netherlands, Sweden, Georgia, and New York City). Additionally, these numbers were calculated while considering only cardiovascular disease, hypertension, diabetes and pulmonary disease as comorbidities. While these diseases contribute to the bulk of the <65 years old comorbidity population, studies still leave out other underlying diseases linked to COVID-19 with unknown contributions to this comorbidity population. Many countries and states vary in their definitions of underlying conditions as it pertains to COVID-19. Partitioning COVID-19 data according to the major comorbidities could prove beneficial to the analysis of the reported data, given the significant number of deaths that group contributes to the deaths in the <65 age group.

SARS-CoV-2 also disproportionately affects people by socioeconomic status, most notably in urban areas. Yet, the following paragraph will instead evaluate the mortality rate through race and ethnicity rather than socioeconomic status. The reason for this is not to suggest there is a natural vulnerability to the virus based on race. Rather, the reason is because it is well-established in the US that minority groups are disproportionately represented in lower socioeconomic statuses [36]. Additionally, socioeconomic status is not reported as often as race/ethnicity in most studies. Therefore, the association between socioeconomic status and race can be useful when considered appropriately.

The antibody survey conducted by the New York [37] government in late April provided seroprevalence estimates of 8.9–9.1% in White populations, 22.5–32.0% in Latino/Hispanic populations, 16.9%–22% in Black populations, and 11.7–14.6% in Asian populations. The APM research lab has independently compiled up-to-date data [38] regarding COVID-19 deaths by race across the US and has identified that Black people, representing 12.4% of the population, have suffered 19.9% of reported COVID-19 deaths. Additionally, compared to the White population, the latest U.S. age-adjusted COVID-19 mortality rate for the Black populations are 3.0 times as high, the Indigenous people are 3.2 times, the Latino populations are 3.0 times, and the Pacific Islanders are 2.3 times. If there is a direct association between these demographic groups and socioeconomic status in a population, as is the case in major cities in the US, then studies can use this demographic measure to assess how these factors affect the mortality rate. Social factors are affecting disadvantaged groups and low-income countries which contributes to anomalies and inaccurate interpretation of the data.  With higher rates of underlying conditions, less access to healthcare and more frontline jobs, among other factors, people of lower socioeconomic status are another group to carefully consider in the context of this pandemic. More awareness of this issue can help with public health measures like an increased availability of antibody and RT-PCR testing, increased awareness of disease symptoms, and more strict guidelines on personal protective equipment to help control the spread and mortality of this disease within these groups.

Is the sample population representative of the general population?

First and foremost are the uncertainties with the sample population. While some serosurveys are deliberately unrepresentative of the larger population, such as those using blood donors as samples, others that aim for mixed, random sampling within a population can still have variability. When recruiting individuals, certain subpopulations where COVID-19 is particularly widespread, such as among nursing homes, disadvantaged communities, people experiencing homelessness and people in prisons may be under-represented in the studies. The serosurveys do not exclude these groups, rather their method of recruitment inherently makes it difficult for these groups to participate. For example, many studies were household-based, recruiting from outpatient clinics, or contacting participants via Facebook [10]. Institutionalized populations will have a more difficult time accessing these studies as well as disadvantaged communities who do not have regular access to healthcare or technology. Recruiting fewer people from these subgroups may underestimate seroprevalence and overestimate IFR.

How do information delays affect the timestamp of COVID-19 data?

Many delays occur over the course of SARS-CoV-2 exposure and infection. Awareness of each delay can ensure that each documented case, seroconversion, hospitalization and death is properly associated with the date it represents. Overall, studies must account for the delay between exposure and symptoms (incubation period), symptom onset and documented infection, exposure and seroconversion (formation of antibodies), symptom onset and death, and finally, death and reporting. The delay between infection and seroconversion is roughly 1 to 3 weeks  [39]. The incubation period has a median time of 4–5 days, where 97.5% of people with COVID-19 who show symptoms will do so before 11.5 days after infection [40]. The time between symptom onset and documented infection is roughly 5 days [41]. The delay between symptom onset to death usually falls in the range of 13 to 19 days [42]. The delay between death and reporting is roughly 1 to 8 weeks, where roughly <25% of deaths are reported within the first few weeks and generally 75% are reported by 8 weeks [43]. In summary, it takes roughly 1.5 to 2 weeks for a rise in infections to reflect in documented cases, one to three weeks for a population’s antibodies to represent the seroprevalence, and one month or more for reported deaths to reflect the mortality of past cases. These delays can cause incorrect associations between values of seroprevalence, cases and deaths if not appropriately considered.

What is the accuracy of SARS-CoV-2 testing?

There are two types of tests used to test the presence of antibodies in individuals, the lateral flow immunoassay (LFIA) device and the enzyme-linked immunoassay (ELISA). Each test has a different way of analyzing a serum sample for IgG and IgM antibodies against a certain part of a virus, in this case, the SARS-CoV-2 spike protein. As mentioned above, seroconversion, or creating anti-spike protein antibodies, can take one to three weeks. Therefore, a serology-based study will calculate the seroprevalence of the population roughly two weeks before the study date.

LFIA devices are used as point-of-care tests, providing results in a matter of 10 min while the ELISA can take hours and requires lab equipment, yet there is a trade-off in quality as the ELISA is typically more sensitive and specific. The sensitivity of a test refers to the likeliness of giving a true positive result. The specificity of a test refers to the likeliness of giving a true negative. The sensitivity of the LFIA and ELISA devices is assessed by calculating the percent positive results against known positive samples confirmed by the RT-PCR test, the gold-standard. 100% in this case is considered perfectly sensitive. The specificity is usually tested against pre-SARS-CoV-2 outbreak samples where 100% negativity of tested samples means perfectly specific.

To assess the quality of each test, a report from the National COVID Scientific Advisory Panel [44] tested for SARS-CoV-2 IgM and IgG antibodies using ELISA and 9 different LFIA devices. The ELISA detected IgG in 34/40 PCR-positive samples, a sensitivity of 85% (95%CI 70%–94%), where all 6 false negatives were from samples taken within at least nine days from symptom onset. It detected IgG in 0/50 pre-pandemic controls, a specificity of 100%, and in 31 of 31 positive samples taken greater than 10 days after symptom onset, a sensitivity of 100%. IgM sensitivity was lower at 70%, and all IgG false negatives were also IgM false negatives. This confirms that the accuracy of ELISA tests improves when detecting IgG antibodies in samples taken greater than 10 days after symptom onset. The ELISA OD ratio can often be refined according to the study’s preferences to prioritize either sensitivity or specificity. LFIA devices, on the other hand, ranged from 55%–70% in sensitivity and 95%–100% in specificity. Higher sensitivities for LFIA devices are reported by manufacturers, but the seroepidemiological study in Spain [9] performed their own validation of the LFIA device they used. They reported an IgG sensitivity of 82.1%, an IgM sensitivity of 69.6% and specificities of 100% and 99.0%, respectively. Therefore, exact sensitivities of LFIA devices are variable, but IgG antibodies seem to be more reliable than IgM.

The lower sensitivity of LFIA devices may result in unreliable and insufficient screening of SARS-CoV-2 infection. The National COVID Scientific Advisory Panel study [44] considers the best-case scenario for an LFIA test to be 70% sensitivity and 98% specificity. Even if the sensitivity of the device were to improve without compromising the specificity, after 1000 tests there would be roughly 19 false positive documented infections. In a population of 5% seroprevalence, this would mean 35% of the tests are wrong. As the seroprevalence increases to 20%, 10% of results would be wrong, and at 50% seroprevalence, 3% would be wrong. This is concerning given the range of seroprevalence estimates in the meta-analysis study by Ioannidis [10], where only 9 of 36 studies recorded a seroprevalence 10% and only 4 were 15%. Despite this apparent flaw in the LFIA devices, the point-of-care and ELISA tests used in the Spain study [9] still recorded similar seroprevalence estimates, 5.0% (95% CI 4.7–5.4) and 4.6% (4.3–5.0), respectively. This suggests that for large serology-informed studies, such as the one in Spain, the LFIA test could be useful as it makes for greater uptake, lower cost and easier implementation.

There is more to call into question regarding testing individuals for SARS-CoV-2 infection using the reverse transcriptase–polymerase chain reaction (RT-PCR) swab test. This test uses swabs to take a sample of the subject’s upper respiratory tract and, if SARS-CoV-2 RNA is present, will use the RT-PCR technique to replicate the RNA to detectable levels. While it would be rare to see a false-positive RT-PCR test excluding instances of cross-contamination, false-negatives can occur due to poor quality or timing of the test. A study on the temporal dynamics of viral shedding and transmissibility of COVID-19 [45] showed that viral loads in the upper respiratory tract peak at and soon after symptom onset, then decline quickly within 7 days until they reach the detection limit at around 21 days. Infectiousness, however, may decline significantly after 8–10 days of symptoms, as live virus could no longer be cultured in a study by Wölfel, et al. [46]. Therefore, there may be a significant amount of time where individuals test positive for RT-PCR tests despite no longer being infectious to others.

What is known about the IgG and IgM antibody kinetics in Humans?

There may also be significant differences in post-infection antibody kinetics between asymptomatic, mild and severe infections. In a clinical and immunological assessment of 37 asymptomatic and 37 symptomatic SARS-CoV-2 infections [47], the study found significant differences in IgM detection, where 62.2% asymptomatic were positive and 78.4% of symptomatic individuals were positive. Additionally, whereas 81.1% and 83.8% of asymptomatic and symptomatic individuals, respectively, tested positive for IgG 3–4 weeks after exposure, only 40.0% of asymptomatic and only 12.9% of symptomatic individuals became seronegative for IgG in the early convalescent phase, 8 weeks after being discharged from the hospital. Therefore, to maximize the accuracy of serosurveys, conclusions made from the data must be associated with the period of time they most accurately represent.

How are COVID-19 deaths defined?

Across nations and states, the answer to the question “What is a COVID-19 death and what isn’t?” is serious and important, but also inconsistent. In general, there are three methods of defining and quantifying COVID-19 deaths. First is the method of recording a death as due to COVID-19 only for those who test positive, either before or after death. This method could be uniformly implemented if every person could get tested, however, there are many countries and states that are unable to do so. This results in deaths from exacerbation of chronic conditions due to COVID-19 and deaths not counted due to lack of testing. Therefore, this method can miss people with atypical symptoms and deaths not linked to the pandemic, such as limited access to health care services due to overcrowded hospitals. It could also incorrectly count those dying from unrelated causes, such as a car crash, after testing positive. Second is the method of counting deaths of people who test positive and those who are not tested but suspected of having COVID-19. Several countries, such as Belgium, Canada, England, France, Ireland, Scotland, and some regions of Spain, have used this approach [32]. With this method comes a risk of incorrectly associated deaths to COVID-19, but it may help in providing timely data as to the scale of the pandemic’s mortality without requiring COVID-19 tests for every hospitalized individual. Unsurprisingly, these countries report higher proportions of COVID-19 deaths [32]. The third method of quantifying COVID-19 deaths is by measuring excess deaths. This method is best for quantifying the number of deaths both directly and indirectly associated with COVID-19, capturing the full effect the pandemic has had on the public’s health. This method works by comparing the total amount of deaths that are over the expected number of deaths based on the past five years. This method will be reliable, but not for months or possibly years due to the time it takes to officially process death certificates. There may also be variability in excess deaths caused by confounding factors, such as a bad flu season, less driving accidents or decreased utilization of healthcare during the pandemic. It is important to acknowledge the different ways that COVID-19 deaths are recorded to recognize the possible underestimation, by the first method, and overestimation, by the second method, of IFR values. The third method will inherently underestimate the recent mortality rate, as deaths are being processed and documented. Over time, the overall impact of the pandemic on deaths can be evaluated, yet one will have to consider the many confounding factors in play to estimate the mortality rate of the disease itself.

What is the state of hospital COVID-19 cases, deaths and patient data?

COVID-19-Associated Hospitalization Surveillance Network (COVID-NET) [48] is a population-based surveillance system run by the CDC that collects data on laboratory-confirmed COVID-19-associated hospitalizations among children and adults through a network of over 250 acute-care hospitals in 14 states, covering 10% of the entire US population. This surveillance system acquires information about each case’s age group, sex, ethnicity and underlying health conditions. Cases are identified in COVID-NET if they test positive for SARS-CoV-2 and are hospitalized within 14 days of the positive test, and the data is collected using a standardized method of reporting by trained surveillance officers. Therefore, this database has the potential to present patients in a complete context. It is a prime example of how hospital data can be used to inform the public on the risk of COVID-19 in their area, providing data that gives both specific and generalized data points. For example, it can give the weekly hospitalization rate by age and can also give the proportion of cases resulting in death or release by race/ethnicity. COVID-NET also shows that 89.3% of all hospitalizations are in patients with some underlying health condition, the most common being hypertension 58.9% [48]. While promising, there are limitations to the application of this data. First, the network was able to perform a detailed analysis of comorbidity and ethnicity only for hospitalizations in March due to the large amount of time needed to process this data. There were 1,482 hospitalizations in their system for that month, and just 180 (12.1%) contained data regarding comorbidities. The only cases reported on the COVID-NET surveillance system website are from cases where the healthcare provider specifically called for laboratory testing for SARS-CoV-2, leading to an under-ascertainment of COVID-19 cases as each provider practices differently. Moreover, all results are provisional as each chart must be reviewed once the patients have a discharge disposition. The inefficient transfer of information is limiting this website’s ability to present a more holistic and true evaluation of the COVID-19 pandemic throughout the country. The difficulty of communicating critical data, like ethnicity/race and underlying conditions, is closely linked to the main issue addressed in this paper, providing context around COVID-19 cases.

A system like COVID-NET needs to be established much more widely throughout the US. It is critical that the flow of information from hospitals to organizations, like the CDC and Human Health & Services, is streamlined for policymakers and the public to be aware of the situation in their local area. Despite this issue, the COVID-NET interactive website [49] continues to publish current, weekly hospitalization data stratified by age which can still be very helpful for those looking to make decisions based on hospitalization data.

Using metrology principles for reporting epidemiological parameters

In this review we looked at the variety of factors affecting COVID-19 fatality rate estimates. To improve our understanding, modeling and decision-making regarding the COVID-19 pandemic or any other pandemic, epidemiological studies require standardization for reporting data. It is essential to develop a definition of minimum information (metadata) needed to correctly describe fatality rates, but also all other critical epidemiological parameters.

There are many factors associated specifically with the COVID-19 fatality rate and generally to seroepidemiological studies that must be considered for a proper contextual understanding of published data. While these factors and limitations are well-known throughout the epidemiological field, there is a habit of not including them in published work. By including this metadata, epidemiologists will better understand the provenance of parameters, and how the results of one study in a specific setting can be generalized and applied more broadly to other situations. It will allow public health officials to make more substantiated and knowledgeable decisions. At the same time, it will improve communication between epidemiologists investigating diseases and possibly reveal novel insights about previously unexplainable differences between models and studies.

This manuscript aims at sparking a conversation that considers how to create standardized guidelines for reporting epidemiological parameters in the literature. We believe this can be accomplished by applying metrology principles, which help experimentalists thoroughly dissect each aspect of their study to find where uncertainties can lie. In turn, this dissection not only leads to increased awareness of these factors and limitations but can help people understand why they are so critical to include.

The COVID-19 pandemic management showcases the urgent need for standardization

This standardization dilemma has also manifested in the disparate handling of COVID-19 across the US. There are inconsistent recommendations for social-distancing and business and institution closings across the US. While some situations require more or less action than others, disparate messaging can make it extremely difficult to coordinate a unified response when one is needed. A study conducted by the organization Resolve to Save Lives [50] demonstrates that this issue extends to their reporting of COVID-19 data. The study reviews all 50 US states’ COVID-19 data dashboards to assess their consistency and robustness. Uniform indicators across all 50 states’ data regarding COVID-19 spread, mortality and response is critical not only to ensure accountability and risk of this pandemic but also to ensure the data can be utilized accurately and to its fullest extent. The review discovered a lack of consistency that is startling across all domains of critical pandemic-related data, except for deaths. Syndromic surveillance, or the reporting of COVID-like illness and influenza-like illness, in patients who present themselves to healthcare facilities was reported in only 37% of states for COVID-like illness and 18% for influenza-like illness. The immediate reporting of these numbers’ new daily counts is critical for predicting potential upcoming virus spread. The type of COVID-19 case indicators, such as new confirmed, probable, and per-capita rates, are not clearly defined in 40% of states, apart from all the states displaying either new or cumulative cases. Only 64% of states report data for nursing homes, correctional facilities, homeless shelters and other facility-specific data. The number of tests performed is reported in >90% of states, but only 75% report PCR test positivity and 5% report the average time from symptom onset to PCR test result, which is important to be no more than two days as this is the period of peak infectivity. Slightly more than 80% of states report COVID-19-specific hospitalizations but vary between reporting cumulative or daily new, less than 50% report intensive care unit bed admissions. Also, they present numbers in counts, rather than per-capita, which does not allow for comparison of the data with other locations. Only 15% of states report occupational healthcare worker infections. Finally, only 8 states report data on the source of exposure for cases, which reflects on the region’s ability to control COVID-19 via awareness of where outbreaks occur.

Aside from the type of data reported, there are significant variations in the display of data, performance targets and what data is considered important. For example, while 92% of states report COVID-19 cases, some states report the case date as the date of specimen collection, some the date of illness onset, and some the date reported. Some states include data for both nursing home staff and residents, while others report only for residents. Among the >90% of states reporting testing, they vary in reporting either cumulative or weekly numbers and the type of test being reported. Some report PCR positivity for the day, while others require users to calculate it themselves. While all but three states include data on demographics, they vary greatly in the type of information reported (cases, deaths, hospitalizations) and the type of stratification (age, sex, race/ethnicity, or a combination).

Granted, establishing websites to inform the public and policymakers is unprecedented, but there are major flaws in the way it was carried out. The state-to-state dissimilarities considerably hinder the ability to compare the situation in one state with another. It can result in the misuse and misunderstanding of the data. It can be the cause of inconsistent public health safety guidelines. It can cost the lives of people who are affected by the absence of demographic data and blindness to the risk of the disease in their area.

Epidemiology and public health can learn an important lesson from other fields in the biomedical sciences

The rigor and reproducibility crisis in the biomedical sciences has moved scientists across different fields to establish and develop guidelines for reporting data and methods with rigor and robustness. In enzymology, there are often key measurements, reagents, temporal data, and other critical information left out leading to irreproducible studies and unreliable results. The lack of consensus within the community results in inconsistent reporting of data throughout studies. Experiments are conducted in different environments and in a variety of ways without consideration of the weight each variation carries. This has led to discrepancies in the reporting of physical constants, leading to irreproducible scientific findings. In an effort to gain control, the Standards for Reporting Enzymology Data guidelines [51] have been created to inform enzymologists on what data is critical to report for their experiments. These guidelines ensure that the identity of the enzyme, preparation, storage conditions, assay conditions, enzyme activity, methodology and any other critical information is clearly stated in order to standardize studies in their field.

Groups of experts in other fields of biology have come together in an attempt to resolve this growing issue. To encourage the reporting of critical information, these groups established guidelines such as the Minimum Information About Microarray Experiment (MIAME) and Minimum Information about a Biomedical or Biological Investigation (MIBBI). According to the metrologists in Plant et al. [52], establishing consensus requirements such as these is the first step to bringing back validity and reproducibility to published results in their respective scientific fields. Plant et al. discuss three core aspects that are vital to identifying confounding variables and assessing uncertainty within a study. First, characterizing the experimental system, such as specifying instrumentation, characteristics of the subjects and computational tools, will make results robust. Second, immutable reference materials and reference data, like including calibration of instruments and type of software, will make results reproducible and comparable between laboratories. An example is the OD level used in ELISA tests or the type of specimen used in the validation of COVID-19 test quality. Third, valid interpretation of data given the known truth and limitations of the experiment.

What are the next steps for epidemiology and public health?

There has been a promising development in the standardization of reporting figures, context, and terminology on the CDC website [53]. On this page, the CDC outlines how diagnostic and screening testing sites must be accredited, how they report their data and to which organizations (regional, state, and federal public health departments), what data elements should be reported (age, race, sex, test ordered, date, etc.) and the standard terminology that should be used. However, there are many other areas where more work is required.

For example, in the case of fatality rates, we suggest the reporting of seven categories of metadata (see, Table 1) in studies estimating the seroprevalence and/or the IFR of a population. Included are topics concerning both seroepidemiology and modeling, which have the potential to cause significant uncertainties and variations in data, as we have discussed. Again, these suggestions should be considered as a starting point for experts in the field to ensure a complete picture of how each COVID-19 epidemiological study is painted. While this table can be used by epidemiologists in their studies, the Resolve to Save Lives study [50] similarly includes a table of 15 essential COVID-19 indicators that should be reported by each county, state, and country and example data dashboards that can be used more generally by serologists, policy makers and government officials. We also recommend looking at Table 1 of another study by Plant et al. [54] that provides general guidelines to kick start the conversation of identifying any other uncertainties within serology studies that have yet to be identified. This table was created by summarizing the sources of uncertainty as described by the Guide to Expression of Uncertainty in Measurement.

Table 1

Details each category of contextual and experimental details to be included in a study that estimates seroprevalence and/or infection fatality rate.

Causes of Uncertainty and Variation	Metadata	Why is this important?	How it shows up in a study…
Demographics	Sex	Males have a slightly higher mortality risk.	The proportion of men vs women is not representative of the population.
	Age	Mortality risk increases exponentially with age.	The ages of samples are not representative of the population.
	Socioeconomic status	Groups of lower socioeconomic status have a higher seroprevalence and risk of mortality.	The proportion of minorities is not representative of population.
Underlying Conditions	Cardiovascular disease, hypertension, diabetes, chronic obstructive pulmonary disease, severe asthma, kidney failure, severe liver disease, immunodeficiency, and malignancy	Certain underlying conditions will result in a higher risk of mortality.	A high proportion of individuals with underlying conditions in the population where the death count is taken from.
Sample Subpopulation	People in long term care homes, homeless shelters, in prison, occupation	People in institutions and certain occupations risk higher exposure/spread of the virus leading to higher seroprevalence. People in long term care homes have a higher risk of mortality and can disproportionately contribute to the number of deaths within a population.	Institutionalized individuals and healthcare workers are not represented in the sample population. A low proportion of individuals in a long-term care home in population.
Information Delays	Documented infection to death & death to reporting	Delays in the transfer of information need to be considered when deciding which date to use for the death count.	An incorrect date is chosen for the death count.
SARS-CoV-2 Test	Type: LFIA, ELISA, RT-PCR	Different types of tests have different sensitivities, specificities, and timing.	A lower sensitivity & specificity resulting in an inappropriate number of false positives/negatives.
	Specifics: Antibodies tested, specificity and sensitivity according to validation tests	IgG, IgM, and IgA antibodies have different accuracies at different points of time.	One study tests only IgM, another IgM and IgG, and another IgM, IgG, and IgA.
Antibody Kinetics	Delay from infection to seroconversion and from seroconversion to seronegative	Delays between infection and developing antibodies and then the subsequent loss of those antibodies can affect the seroprevalence.	A larger time between infection and testing.
Population’s Methods of Quantifying a COVID-19 Death	Lab-tested only	Can miss deaths from causes not associated with COVID-19 or who were asymptomatic	Concerns with the death count the study uses for the target population.
	Tested + Suspected	Can overestimate death count by counting patients with COVID-like symptoms without test confirmation
	Excess deaths	Inaccuracies in reporting and the delay between death and reporting can affect recent death counts.

The next critical step is setting in motion a discussion within the epidemiological field to standardize the measuring and reporting of data. To do this effectively, we suggest an international committee of epidemiological experts to come together and establish minimum reporting guidelines in epidemiology and public health. This group could be coordinated by both the CDC and the National Institute of Standards and Technology (NIST), which are in the position of guiding the initiative effectively. The previously mentioned page on the CDC website does well to consider our concerns as they relate to lab-reported data, however, these guidelines could also be extended to all serology studies.

It will be very beneficial to establish an international committee analogous to or within The Bureau International des Poids et Mesures. This is an international body that aggregates all state members of NIST and other countries around the world to help more broadly establish what needs to be standardized within certain fields of science and what fundamental definitions of quantities people should adopt. Without an international committee and encouragement by higher institutions, it will be difficult, if not impossible, to establish global guidelines to be prepared for the next pandemic. In addition to the epidemiology field, the expectation to standardize methods of reporting COVID-19 related data will hopefully be implemented in all government health agencies across the United States, as this is the most direct way to improve the quality of data presented to the public and policymakers. While much of this data may not be immediately available in all states, instituting a set of indicators to be reported, such as those in the Resolve to Save Lives study [50], will begin this critical process. The benefits of investing resources into properly gathering this data will certainly outweigh the costs to our economy, social lives, and public health.

While we have used the current pandemic as our case for standardizing the methods of data collection and reporting, we hope that the concepts presented in this paper will become well established in the epidemiological communities. This issue is easily overlooked and is more prevalent than one might think. Fixing the problem for the current and future pandemics begins with increasing the awareness of how one’s research works with the research of others in the same field. Establishing this perspective will serve to reveal the many connections between the extensive amount of research published on any singular topic and improve our ability to utilize each and every finding. Our goal is to add this perspective on experiment design and data reporting to the arsenal of the epidemiological scientist. We believe that doing so will help further develop an already robust field and enhance the real-time impact of epidemiological research on public health.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.