Modeling the Probability of COVID-19 Based on Symptom Screening and Prevalence of Influenza and Influenza-Like Illnesses.

BACKGROUND: The importance of various patient-reported signs and symptoms to the diagnosis of coronavirus disease 2019 (COVID-19) changes during, and outside, of the flu season. None of the current published studies, which focus on diagnosis of COVID-19, have taken this seasonality into account.
OBJECTIVE: To develop predictive algorithm, which estimates the probability of having COVID-19 based on symptoms, and which incorporates the seasonality and prevalence of influenza and influenza-like illness data.
METHODS: Differential diagnosis of COVID-19 and influenza relies on demographic characteristics (age, race, and gender), and respiratory (eg, fever, cough, and runny nose), gastrointestinal (eg, diarrhea, nausea, and loss of appetite), and neurological (eg, anosmia and headache) signs and symptoms. The analysis was based on the symptoms reported by COVID-19 patients, 774 patients in China and 273 patients in the United States. The analysis also included 2885 influenza and 884 influenza-like illnesses in US patients. Accuracy of the predictions was calculated using the average area under the receiver operating characteristic (AROC) curves.
RESULTS: The likelihood ratio for symptoms, such as cough, depended on the flu season-sometimes indicating COVID-19 and other times indicating the reverse. In 30-fold cross-validated data, the symptoms accurately predicted COVID-19 (AROC of 0.79), showing that symptoms can be used to screen patients in the community and prior to testing.
CONCLUSION: Community-based health care providers should follow different signs and symptoms for diagnosing COVID-19 during, and outside of, influenza season.

In community-based settings, early diagnosis of coronavirus disease 2019 (COVID-19) is important so that decisions about clinical care, laboratory tests, hospital triage, and self-quarantine can be made accurately. In such settings, one has to rely on individuals' self-reported signs and symptoms, as laboratory tests, imaging studies, and other diagnostic tools are not immediately available. Wynants and colleagues1 have reviewed the literature on diagnosis of COVID-19 and identified 47 articles. None of these studies can be used in community diagnosis because 34 relied on imaging, and 13 included laboratory tests, that are not available in the community. The current study exclusively relies on patient-reported signs and symptoms. Thus, this article highlights how clinicians can screen patients in the community prior to triage and prior to testing.

Influenza and COVID-19 have overlapping respiratory symptoms and it may be easy to mistake one for the other. While in 2020 there were few cases of flu, this may not be the case in the future. The COVID-19 pandemic is an evolving situation. With partial vaccination, it is possible that people may return to work/school and social gatherings, in which case influenza is likely to return. We are likely to be in a situation where both COVID-19 and influenza co-occur. The current article provides 2 algorithms, one for screening for COVID-19 during, and the other outside, the flu season. Thus, this study allows one to better understand how values of different signs and symptoms change based on the prevalence of other respiratory diseases. If 2020 flu was absent, the algorithm for outside of the flu season should have been used. If the flu season returns, which we expect, the in-season algorithm should be used.

None of the studies of symptoms of COVID-19 have taken seasonality into account.2–12 Published studies compare symptoms of COVID-19 to a nondescript control group composed of patients for whom COVID-19 diagnosis has been ruled out. In different seasons, the control group could reflect different diseases. In this study, during the flu season, COVID-19 is compared to a mix of influenza and influenza-like illnesses. Outside of the flu season, COVID-19 is compared to only influenza-like illnesses. The value of various symptoms changes when the composition of diseases in the control group changes.

METHODS

Outcome

The outcome of interest is COVID-19 diagnosis as indicated by the laboratory test or by diagnostic imaging.

Data sources

The data were compiled from several different sources:

Symptoms of COVID-19 patients were extracted from China's Infectious Disease Information System, and made publicly available by the Chinese Center for Disease Control and Prevention.13 The data included 72 314 COVID-19 patients between December 2019 and February 11, 2020. These were cases from Wuhan, Hubei Province, China, which included confirmed cases (61.8%), suspected cases (22.4%), clinically diagnosed cases (14.6%, Hubei province only), and asymptomatic cases (1.2%).14,15 Among these COVID-19 patients, we included in the analysis 774 patients, who had reported at least 1 respiratory or general symptom. Symptoms such as anosmia, ageusia, or diminished sense of smell, were not reported in the 774 Chinese patients, as these symptoms were not known till later in the worldwide pandemic.

The second sample comprised 273 US COVID-19 patients, who reported at least 1 symptom, including neurological symptoms, such as altered or diminished sense of taste or smell.13 These patients were identified from a sample of 635 cases that presented with COVID-19-positive clinical phenotypes derived from the Mayo Clinic Electronic Health Records databases of 30 494 patients subjected to COVID-19 polymerase chain reaction diagnostic testing.16

Data on symptoms of influenza and influenza-like illness patients were collected from the National Institute of Allergy and Infectious Diseases' Influenza Research Database in the United States, for the years from 2000 to 2019.16 These data included 2885 influenza patients and 884 patients with influenza-like upper respiratory infections. These data were collected prior to onset of COVID-19.

Since our data came from different timeframes and different samples, the data were weighted so that the distribution of age of patients corresponded to the distribution of age of COVID-19 patients in the United States.17 Similarly, using Influenza View and ILINet, the influenza and influenza-like illnesses were weighted to represent age distribution of these diseases in the United States.18

Power of study

We tested the power of the study to detect 10% difference in probability of COVID-19. In this study, we have multiple models, each of which predicts COVID-19 and influenza using different set of assumptions. Determining the minimum sample size for some of these modeling efforts has not been worked out. Investigators have suggested to use the sample needed for logistic regression model to set the minimum for other types of multivariate modeling efforts.19 We followed Hsieh et al20 simplified procedures for determining the sample size needed for the logistic regression model. For α = 0.5, β = 0.2 (power = 80%), 10% difference in probability of COVID-19, and fever predicted from other symptoms with R2 = 0.3, the minimum required sample size was 183 subjects. The data available to us exceeded the minimum required sample and, thus, the analysis has sufficient power to detect 10% difference in probability of COVID-19.

Description of symptoms

An examination of the data from the Chinese patients showed that symptoms had not been recorded for most patients, and when COVID-19 symptoms had been recorded, there were many inconsistencies. Symptoms were misspelled (eg, runny nose reported as running nose). Multiple terms were used to describe the same symptom (eg, shortness of breath and dyspnea). Patients were reported to have “aches” as opposed to “muscle aches.” There were numerous misspelled symptoms, which were corrected manually. When no symptoms were reported, entire cases were omitted. When some symptoms were reported, the second author read the case report and classified the symptoms, removing inconsistencies. A JSON file that captures all symptom transformations is available in the online supplement (available at: http://links.lww.com/QMH/A70). This file can help other investigators to replicate our effort.

Methods of scoring odds of COVID-19

The posterior odds of COVID-19 was calculated using the Bayes independence model, also called naïve Bayes:

In this equation, C indicates COVID-19, N indicates not COVID-19 (which could be a mix of flu and flu-like illnesses during the flu season or flu-like illnesses outside of the flu season), Xi indicates one of the n signs and symptoms, that could be used to differentiate these 2 diseases; p(C) indicates prior probability of COVID-19 in the population; similarly p(N) indicates probability of upper respiratory infections (influenza or other infections); p(Xi|C) indicates the probability of observing the sign/symptom Xi among COVID-19 patients; and p(Xi|N) indicates the probability of observing the same sign/symptom among patients with influenza. The ratio of p(Xi|C) and p(Xi|N) is called the “likelihood ratio” and shows how informative a symptom is in predicting the presence of COVID-19. A likelihood ratio above 1 increases the odds of COVID-19, while a likelihood ratio below 1 does the reverse. The value of p(Xi|NI) and p(Xi|C) is initially obtained from our data samples. We report 2 sets of likelihood ratios, one for when the flu is widespread, during the flu season, and another, when the flu occurs only sporadically, outside of the flu season. The likelihood ratios used for predicting COVID-19 depend on how widespread the flu is in a given season at a given geographic location. The information about seasonal flu activity is available from the Centers for Disease Control and Prevention (CDC) Web site.

Methods of measuring accuracy

The performance of the naïve Bayes model was examined using area under the receiver operating curves (AROC). Since there are 3 different predictions (1 for each of the 3 diseases), we report the sample-weighted average AROC across the 3 predictions. The performance of the naïve Bayes model was compared to LASSO logistic regression, ridge logistic regression, random forest, Ada boost, gradient boosting, and stochastic gradient descent. These alternative models were examined to see whether naïve Bayes assumption of independence leads to significant loss of accuracy.

Human subjects research

All data used in this study were deidentified secondary data. The George Mason University IRB approved the study's exemption from requiring patient consents.

RESULTS

Table 1 shows the prevalence of various signs and symptoms in (1) COVID-19, (2) influenza, and (3) other respiratory infections that are neither influenza nor COVID-19. The 2 columns for likelihood ratios show the impact of each symptom in predicting COVID-19, one during, and the other, outside of the flu season. For example, male patients were 1.49 times more likely to have COVID-19 outside of the flu season. In contrast, during the flu season, female patients had an elevated risk of COVID-19. Patients in their 60s were 1.97 times more likely to have COVID-19 during the flu season.

Table 1.

Likelihood Ratios Associated With Symptoms in, and Outside, Flu Seasona

Signs and Symptoms		COVID-19 Number/Total (Percent)	Flu Number/Total (Percent)	Flu-Like Illness Number/Total (Percent)	Likelihood Ratio for COVID-19 in Widespread Flu Season	Likelihood Ratio for COVID-19 in Sporadic Flu Season
Ageb	0-29	129/709 (18%)	1480/2885 (51%)	241/884 (27%)	0.400	0.669
	30-39	96/709 (14%)	278/2885 (10%)	146/884 (17%)	1.208	0.821
	40-49	114/709 (16%)	278/2885 (10%)	146/884 (17%)	1.431	0.973
	50-59	132/709 (19%)	278/2885 (10%)	138/884 (16%)	1.688	1.194
	60-69	107/709 (15%)	201/2885 (7%)	89/884 (10%)	1.968	1.497
	70-79	59/709 (8%)	124/2885 (4%)	41/884 (5%)	1.907	1.792
	80c	71/709 (10%)	248/2885 (9%)	82/884 (9%)	1.141	1.072
Raceb	White	169 737/326 828 (52%)	(78%)		0.669
	Black	95 850/326 828 (29%)	(13%)		2.189
	Asian	16 167/326 828 (5%)	(8%)		0.651
	Other	3135/326 828 (1%)	(1%)		0.719
Gender	Female	316/704 (45%)	328/1719 (19%)	556/884 (63%)	1.322	0.714
	Male	388/704 (55%)	1391/1719 (81%)	328/884 (37%)	0.835	1.485
Chest pain	No	693/709 (98%)	347/486 (71%)	884/884 (100%)	1.088	0.977
	Yes	16/709 (2%)	139/486 (29%)	0/884 (0%)	0.222	17.000
Chills	No	685/709 (97%)	484/1229 (39%)	414/635 (65%)	2.005	1.482
	Yes	24/709 (3%)	745/1229 (61%)	221/635 (35%)	0.065	0.097
Conjunctivitis	No	704/709 (99%)	734/932 (79%)	53/190 (28%)	1.416	3.560
	Yes	5/709 (1%)	198/932 (21%)	137/190 (72%)	0.024	0.010
Cough	No	393/709 (55%)	288/1548 (19%)	369/695 (53%)	1.892	1.044
	Yes	316/709 (45%)	1260/1548 (81%)	326/695 (47%)	0.630	0.950
Diarrhea	No	564/588 (96%)	12 885/13 658 (94%)		0.983
	Yes	23/588 (4%)	772/13 658 (6%)		1.415
Fatigue	No	615/709 (87%)	594/1343 (44%)	445/581 (77%)	1.606	1.133
	Yes	94/709 (13%)	749/1343 (56%)	136/581 (23%)	0.288	0.566
Fever	No	174/709 (25%)	469/1583 (30%)	517/776 (67%)	0.587	0.368
	Yes	535/709 (75%)	1114/1583 (70%)	259/776 (33%)	1.296	2.261
Headache	No	676/709 (95%)	739/1416 (52%)	390/560 (70%)	1.669	1.369
	Yes	33/709 (5%)	677/1416 (48%)	170/560 (30%)	0.109	0.153
Nausea	No	695/709 (98%)	817/1159 (70%)	501/596 (84%)	1.305	1.166
	Yes	14/709 (2%)	342/1159 (30%)	95/596 (16%)	0.079	0.124
Runny nose	No	684/709 (96%)	583/1424 (41%)	465/732 (64%)	1.985	1.519
	Yes	25/709 (4%)	841/1424 (59%)	267/732 (36%)	0.069	0.097
Shortness of breath	No	666/709 (94%)	840/1160 (72%)	501/589 (85%)	1.225	1.104
	Yes	43/709 (6%)	320/1160 (28%)	88/589 (15%)	0.260	0.406
Vomiting	No	702/709 (99%)	960/1192 (81%)	534/596 (90%)	1.185	1.105
	Yes	7/709 (1%)	232/1192 (19%)	62/596 (10%)	0.060	0.095
Wheezing	No	708/709 (100%)	457/619 (74%)	459/546 (84%)	1.270	1.188
	Yes	1/709 (0%)	162/619 (26%)	87/546 (16%)	0.007	0.009
Diminished smell/taste	No	263/272 (97%)	14 678/14 695 (100%)		0.968
	Yes	9/272 (3%)	17/14 695 (0.12%)		28.602
Excessive sweating	No	241/272 (89%)	13 870/14 695 (94%)		0.939
	Yes	31/272 (11%)	825/14 695 (6%)		2.030
Change in appetite	No	254/272 (93%)	14 210/14 695 (97%)		0.966
	Yes	18/272 (7%)	485/14 695 (3%)		2.005

aTo score the odds ratio of COVID-19, multiply likelihood ratios associated with the features of the individual.

bBased on CDC data.

cMissing values were set to absence of symptom.

Table 1 also shows that cough and fever were the most common features of COVID-19. They occurred, respectively, in 45% and 75% of patients diagnosed with COVID-19. Differential diagnosis is also based on the relative frequency of occurrence of these 2 symptoms in influenza and influenza-like illnesses. A symptom that is common to all 3 diseases would be a poor diagnostic indicator. For example, fever is common in COVID-19 but fever also occurred among 70% of patients with influenza and 33% of patients with influenza-like illnesses. Table 1 shows that fever increased the odds of COVID-19 by a factor of 1.30 during the widespread flu season and a factor of 2.26 during sporadic flu occurrence, or outside the flu season. Thus, while fever is a predictor of COVID-19 during the flu season, it is an even stronger predictor outside of the flu season. This is not the case for cough. Table 1 shows that cough is common among COVID-19 but not as common as in influenza patients. Surprisingly, during the widespread flu season, cough reduced the odds of COVID-19 (likelihood ratio of 0.63). A patient that presents with cough during the flu season is more likely to have influenza than COVID-19. Outside of the flu season, when flu only occurs sporadically, the situation changes. Then, presence of cough is nearly uninformative, with a likelihood ratio close to 1.

Accuracy of predictions

The cross-validated accuracy of the independent Bayes formula, as well as other modeling efforts, is provided in Table 2. Table 2 reports weighted average AROC for the 3 different infections: COVID-19; influenza; and influenza-like illnesses. A perfect predictor will have an average AROC of 1.0 while a random predictor will have an average AROC of 0.5. The average AROC for the independent Bayes model during the flu season was 0.795. The AROC of other statistical models ranged from 0.78 to 0.90. These estimated AROCs are within range of predicted accuracy of models reported in the literature.1 After adjustment for multiple comparisons, no pairs of models were significantly different from each other. Therefore, we decided to proceed with independent Bayes, as this approach is easier to understand for clinicians.

Table 2.

30-Fold Cross-Validated Accuracy of Predictive Models

Performance Measure	Naïve Bayes	LASSO Logistic Regression	Ridge Logistic Regression	Random Forest	Ada Boost	Gradient Boost	Stochastic Gradient Descent
Micro average AROC(95% confidence interval)	0.80 (0.76-0.83)	0.84 (0.83-0.85)	0.85 (0.84-0.87)	0.88 (0.85-0.9)	0.89 (0.87-0.9)	0.89 (0.87-0.91)	0.76 (0.73-0.79)
Brier calibration score	0.19	0.12	0.11	0.11	0.07	0.06	0.12

Abbreviation: AROC, area under receiver operating characteristic curve.

The CDC and the World Health Organization (WHO) have been describing the common symptoms of COVID-19.21 This has created the mistaken perception that these features can be used to diagnose COVID-19. Unfortunately, these features may be more common in another illness, making reliance on these features problematic. It is instructive to see the accuracy of simple rules based on common features reported by the CDC. We calculated the accuracy of diagnosing COVID-19 by assuming that all patients with cough and fever had COVID-19. It had a sensitivity of 0.48, a specificity of 0.69, and an AROC of 0.49. The rule “diagnose all patients with cough, fever, and shortness of breath as COVID-19” had a sensitivity of 0.004, a specificity of 0.97, and an AROC of 0.39. Both of these rules were less accurate than randomly assigning patients to COVID-19, therefore should not be followed. The AROCs associated with these rules showed the perils of relying on simplified rules when diagnosing COVID-19.

Clinical considerations

It is increasingly clear that COVID-19 affects many body systems.22 Clinicians may wish to simplify the task of diagnosing COVID-19 by first examining what types of symptoms are present: respiratory (chest pain, conjunctivitis, cough, fever, runny nose, shortness of breath, vomiting, wheezing); gastrointestinal (diarrhea, nausea, vomiting, change in appetite); neurological (headache, diminished sense of smell/taste); or other (fatigue, excessive sweating).

Example of application of scoring system in Table 1

The calculation of the probability of COVID-19 starts with examining the status of flu epidemic, reported on the CDC Web site. During the widespread flu season one should use the likelihood ratios in Table 1 associated with widespread flu. Outside the flu season, when influenza activity is only sporadic, one should use the likelihood ratios associated with sporadic flu. If flu is assumed to be in transition period, then one should use a weighted set of the 2 likelihood ratios, depending on the extent of flu activity in a given region. Information on spread of flu is available on the CDC Web site.

The following example demonstrates how likelihood ratios in Table 1 could be used. To calculate the odds ratio for COVID-19, one would multiply the likelihood ratios (LR) associated with the individual's features to estimate the odds ratio of COVID-19, and then use the derived odds ratio to calculate the probability of COVID-19. Suppose during the flu season a Black (LR = 2.189), male (LR = 1.322) patient presents with no respiratory or no gastrointestinal symptoms. This Black, male patient reports a loss of sense of smell/taste (LR = 28.6) and headache (LR = 0.109). Since only neurological symptoms are mentioned, then only those symptoms are used and respiratory and gastrointestinal symptoms that are absent are omitted. Then, the odds of COVID-19 increase by 2.189 × 1.322 × 28.6 × 0.109 = 9.02 folds. The probability of COVID-19 can then be estimated by dividing the odds ratio by the sum of 1 and that odds ratio (ie, 9.02/(1 + 9.02) = 0.90), which is a relatively high probability of COVID-19.

Clinicians cannot be expected to perform these calculations while providing care. To assist, they would need an open-source, web-based probability calculator. The patient can describe their symptoms; the calculator can assess the probability of COVID-19; and the patient can report the result to the clinician during the clinic visit.

DISCUSSION

When clinicians, especially those who are community-based, have to diagnose COVID-19, they are more likely to rely on signs and symptoms than on laboratory findings, given limited resources including limited availability of, and delay in, reporting of current laboratory tests. As this article was prepared, the United States was experiencing a shortage of diagnostic tests and a delay in reporting of laboratory tests, often in excess of 1 week to provide the results. Furthermore, there is a limited availability of COVID-19 diagnostic tests globally. Differential screening and diagnosis are urgently needed, particularly as the epidemic emerges in lower-income settings that traditionally have limited access to diagnostics tests.

During the H1N1 pandemic, reporting of cases shifted early in the epidemic from laboratory-confirmed to symptom-based diagnosis. In the case of COVID-19, this may also happen. This article provides a novel method for screening and diagnosing COVID-19 based on patients' reported signs and symptoms and, thus, may assist clinicians make better triage decisions, especially when tests are not available.

This study shows that reliance on fever and cough when diagnosing COVID-19 has accuracy level similar to random guessing; an AROC of 0.49 for cough and fever, and an AROC of 0.39 for cough, fever, and chest pain, combined. Clearly, these simple rules, while focused on common features of COVID-19, are problematic. In this article, we provide a statistical model that has an average AROC of 0.79. In comparison, laboratory parameters such as neutrophil, C-reactive protein, lactate dehydrogenase, aspartate aminotransferase, alanine aminotransferase, urea levels in serum white blood cells, and serum albumin levels accurately predict COVID-19 with AROCs ranging from 0.835 to 0.839.23 Laboratory tests are more accurate than symptom screening. In the absence of laboratory test results, review of signs and symptoms using our algorithm provides reasonable fallback methodology.

Community-based health care providers can use the differential diagnosis tool developed here to diagnose probable COVID-19. A community diagnosis can be used to make appropriate triage decisions, for example, to decide which health care facility to transfer or refer a probable COVID-19 patient to, which entrance of the facility, and what personal protective equipment health care workers should use during the patient's visit. Differential screening and diagnosis can also be used to inform clinical management of the patient while test results are pending or unavailable. Finally, the algorithm can inform who should be prioritized for testing during times of test shortages. In a climate in which there is shortage of reliable diagnostic tests or inconsistency in test result turnaround times, patients whose symptoms suggest high probability of having COVID-19 can be presumed to have COVID-19. Those patients whose symptoms suggest low probability of COVID-19 or individuals with inconsistent clinical presentations can be prioritized for testing.

There are several limitations to this analysis. First, our analysis relies on respiratory symptoms. Many COVID-19 patients do not have respiratory symptoms, in which case differential diagnosis is moot. Our study does not speak to the accuracy of nonrespiratory symptoms in accurately diagnosing COVID-19 patients. It also does not address the accuracy of assessing risks faced by asymptomatic patients.

Second, in our analysis, the difference between COVID-19 and non-COVID-19 infections is confounded by racial differences, and differences in health care patterns between the United States and China. At times, we are comparing symptoms of Chinese COVID-19 patients to symptoms of US patients with influenza or other infections. As more data on presentation of COVID-19 in the United States become available, the current analysis should be repeated to see whether there are any changes in the findings.

Finally, the symptoms of COVID-19 change as the disease progresses. Symptoms during the first week, second week, on hospital admission, and posthospital admission may be different.24 For example, fever may not be a common symptom diagnosed in the community-based settings but is more prevalent in hospitalized patients; estimates of fever among COVID-19 patients grows from as low as 40% on hospital admission to as high as 95% during hospitalization.25 Some studies show that in 20% of test-positive COVID-19 patients, fever was absent during the first 2 weeks but emerged later in the infectious period.26 The temporal variations in the occurrence of symptoms during the course of the disease make diagnoses in community settings more difficult. Our study has not differentiated between the first and second weeks of onset of symptoms in the community. Future research needs to clarify the timing of the symptoms. In the meantime, until more data become available, the proposed differential diagnosis method can help clinicians in the community-based settings respond to the currently unfolding public health emergency.