Utilization of an Electronic Health Record Integrated Risk Score to Predict Hospitalization Among COVID-19 Patients.

OBJECTIVE: To evaluate the performance of an Electronic Health Record (EHR) integrated risk score for COVID-19 positive outpatients to predict 30-day risk of hospitalization. PATIENTS AND METHODS: A retrospective observational study of 67 470 patients with COVID-19 confirmed by polymerase chain reaction (PCR) test between March 12, 2020 and February 8, 2021. Risk scores were calculated based on data in the chart at the time of the incident infection.
RESULTS: The Mayo Clinic COVID-19 risk score consisted of 13 components included age, sex, chronic lung disease, congenital heart disease, congestive heart failure, coronary artery disease, diabetes mellitus, end stage liver disease, end stage renal disease, hypertension, immune compromised, nursing home resident, and pregnant. Univariate analysis showed all components, except pregnancy, have significant (P < .001) association with admission. The Mayo Clinic COVID-19 risk score showed a Receiver Operating Characteristic Area Under Curve (AUC) of 0.837 for the prediction of admission for this large cohort of COVID-19 positive patients.
CONCLUSION: The Mayo Clinic COVID-19 risk score is a simple score that is easily integrated into the EHR with excellent predictive performance for severe COVID-19. It can be leveraged to stratify risk for severe COVID-19 at initial contact, when considering therapeutics or in the allocation of vaccine supply.

Introduction

The COVID-19 pandemic continues to challenge clinical practice. In 2020, COVID-19 was the third leading cause of death in the United States. In general, most cases of COVID-19 are mild, but up to 14% are severe and up to 5% are critical. With up to 20% of patients needing hospitalization, algorithms have been proposed to direct the care efficiently and effectively for patients newly diagnosed with COVID-19, with mild cases managed at home and severe cases managed in hospital. Those managed at home who are at high risk of developing severe COVID-19 disease have been offered remote patient monitoring to assess vital signs and symptoms to allow early detection of severe disease in need of hospital care.[4-8] Given limited resources, it is important to accurately identify those patients at highest risk for severe disease who warrant remote monitoring while at home. An easily accessible risk score in the electronic health record (EHR) calculated using available discrete data in the chart to estimate the likelihood of hospital admission may assist clinicians to allocate remote patient monitoring resources appropriately to provide optimal care for patients. As such, we describe the performance of an EHR integrated risk score to predict the 30-day risk of hospitalization among a large cohort of COVID-19 positive patients.

Methods

This retrospective cross-sectional research study of a practice-improvement initiative was reviewed and deemed exempt from human subjects’ research by the Mayo Clinic Institutional Review Board (IRB) and approved by the Institutional COVID-19 Taskforce.

Setting and Patients

Mayo Clinic, a large multistate U.S. health system with locations in Arizona, Florida, Wisconsin, and Minnesota, sees over 1.5 million annual unique patient visits. The health system uses the same single instance EHR (Epic© Systems Corporation [ ]) allowing all clinical sites to be connected across the enterprise health system.

The present study was a retrospective observational study of 67 470 COVID-19 positive registry patients from all Mayo Clinic sites. This registry included patients based on positive COVID-19 diagnosis made at a Mayo Clinic site or imported from the health information exchange to confirm the diagnosis. The positive COVID-19 diagnosis was confirmed by PCR positive for SARS-CoV-2 between March 12, 2020 and February 8, 2021. Patients who did not give permission via the Minnesota Research Authorization were excluded from this cohort.

Mayo Clinic COVID-19 Risk Score (MCC19-RS) Development

The MCC19-RS was an adaptation of the scoring system developed by Dr. David Daniel from Confluence Health. In developing the original scoring system, Dr. Daniel used available data with regards to risk factors for severe disease with COVID-19 via articles posted on the Centers for Disease Control and Prevention website for Morbidity and Mortality Weekly Reports on COVID-19 (https://www.cdc.gov/coronavirus/2019-ncov/publications.html). Since the score was not developed using a derivation cohort, we adjusted the original score by eliminating weighting and gave each component equal weight in the score except for age. This was done because of uncertainty surrounding the contribution of each risk factor (age aside) in combined risk. The score was initially seen as a way to rapidly display pertinent comorbidities and provide information on known risk factors to bedside providers. Subsequently, the score was validated externally by Halalau et al in a small cohort of patients presenting to one of their 8 Emergency Rooms. The present study builds on this initial validation by including a larger cohort of patients with known COVID-19 diagnosis regardless of presenting location.

MCC19-RS Calculation

MCC19-RS was implemented within the EHR system in May 2020. MCC19-RS utilized an individualized score-based point system under 13 broad domains including demographics and clinical conditions with points assigned to each domain. The demographic criteria evaluated included age and sex. One point was given for male sex and 1to 3 points for age (ie, 0 points for less than 60 years old; 1 point for 60-69 years old; 2 points for 70-79 years old; and 3 points for 80 years or older). Registries were used to assess the presence of a clinical condition and were accorded 1 point including hypertension; congestive heart failure; congenital heart disease; coronary artery disease; chronic lung disease or asthma; diabetes mellitus; immune compromised (Human Immunodeficiency Virus [HIV] diagnosis; or currently receiving chemotherapy or immunosuppressive drugs); nursing home resident; chronic dialysis; chronic liver disease; and current pregnancy. An individual patient’s score equals the sum of the assigned points (0-15 points). For the current study we used the demographics and clinical conditions at the time of COVID-19 diagnosis to calculate the score.

Comparator Score System

The Charlson Comorbidity Index predicts the ten-year mortality for a patient who may have a range of comorbid conditions by calculating a score for each patient as an accepted measure of comorbid burden. We calculated a Charlson Comorbidity Index Score for each patient based on EHR documentation including appropriate codes indicating comorbid conditions, as has been published elsewhere. We used this as a comparator for the MCC19-RS to evaluate if the performance was better than an accepted standard for assessing comorbidity burden and risk.

Data Analysis

Demographics and clinical characteristics of patients were stratified based upon hospitalization status (Table 1). The significance of association between each of the variables and hospitalization status was determined by either using two-sample t-test for continuous variables such as age or by using chi-squared test for all the categorical variables. Fisher’s exact test was used to evaluate the statistical significance of association in cases where the number of classes in categorical variables was less than three. Calibration curves were constructed via logistic regression models of the risk score against the dependent variable of hospitalization risk. Area under the curve confidence intervals were determined with a bias corrected bootstrap. Statistical analysis was done with R Studio Statistical Software.

Table 1.

Demographics, Clinical Characteristics, and Univariate Analysis of COVID-19 Cohort Grouped by Hospital Admission Within 30 Days.

	Admitted (N = 4626)	Not admitted (N = 62 844)	P value
Sex—Male	2591 (56.0%)	30 754 (48.9%)	<.001
Race			<.001
American Indian/Pacific Islander	124 (2.7%)	181 (0.3%)
Asian	183 (4.0%)	1335 (2.2%)
Black/African American	248 (5.4%)	2505 (4.1%)
Unknown	251 (5.4%)	6585 (10.8%)
White	3818 (82.6%)	50 316 (82.6%)
Patient Ethnicity			<.001
Hispanic/Latino	458 (9.9%)	5162 (8.5%)
Not Hispanic or Latino	4076 (88.1%)	51 173 (84.0%)
Unknown	91 (2.0%)	4613 (7.6%)
Age			<.001
Count	4626	62 844
Median	66.000	38.000
Q1, Q3	52.000, 77.000	23.000, 54.000
Charlson Comorbidity Index			<.001
Count	4626	62 844
Median	1.000	0.000
Q1, Q3	0.000, 5.000	0.000, 0.000
MCC19-RS			<.001
Count	4626	62 844
Median	4.000	1.000
Q1, Q3	2.000, 5.000	0.000, 1.000
MCC19-RS- Age Component			<.001
Count	4626	62 843
Median	1.000	0.000
Q1, Q3	0.000, 2.000	0.000, 0.000
MCC19-RS-Lung Disease Component	1620 (35.0%)	4760 (7.6%)	<.001
MCC19-RS-Congenital Heart Disease Component	40 (0.9%)	305 (0.5%)	<.001
MCC19-RS-Coronary Artery Disease Component	947 (20.5%)	2213 (3.5%)	<.001
MCC19-RS-Congestive Heart Failure Component	681 (14.7%)	841 (1.3%)	<.001
MCC19-RS-Diabetes Component	1344 (29.1%)	3672 (5.8%)	<.001
MCC19-RS-ESLD component	274 (5.9%)	1226 (2.0%)	<.001
MCC19-RS-ESRD component	338 (7.3%)	289 (0.5%)	<.001
MCC19-RS-Hypertension Component	2586 (55.9%)	8831 (14.1%)	<.001
MCC19-RS-Immune Compromise Component	569 (12.3%)	1726 (2.7%)	<.001
MCC19-RS-Nursing Home Residence Component	79 (1.7%)	105 (0.2%)	<.001
MCC19-RS-Pregnant Component	11 (0.2%)	484 (0.8%)	<.001
MCC19-RS-Gender Component	2591 (56.0%)	30 752 (49.0%)	<.001
Died within 30 days	335 (7.2%)	70 (0.1%)	<.001

Results

This study is based on data from 67 470 COVID-19 patients. Only index infection episodes were used. Of these, 4626 (6.9%) patients were subsequently admitted within 30 days of testing positive for COVID-19 and 62 844 (93.1%) were treated entirely as outpatients; in addition, 405 (0.6%) expired within 30 days of diagnosis. Table 1 shows the individual factors used in the MCC19-RS and the tendency for them to be associated more proportionally in the admitted cohort. Of the 67 470 patients analyzed, 53 625 (79.5%) were under the age of 60 with only 1685 (3.1%) requiring admission. However, of those 60 and older (13 845; 20.5%) 2941 (21.4%) required admission, with about 1/3 in each successive decade. Admission rates for patients aged 60 to 69, 70 to 79, and 80+ were respectively 12.8%, 25.6%, and 43.2%. COVID patients who were admitted tended to be male (56%) and on average 66 years of age (52-77, lower and upper quartile respectively). Regards race, American Indian, Pacific Islander, and Asian were represented more proportionally in the admitted cohort versus the not-admitted cohort. All clinical condition components were at least 4 times more proportionally represented in the admitted than the non-admitted cohorts, except for congenital heart disease and end stage liver disease. Pregnancy was also represented more proportionally with the non-admitted cohort. Mortality was significantly more common in the admitted cohort (7.2%) compared to the non-admitted cohort (0.1%).

Although the MCC19-RS has a maximum possible score of 15, the highest observed score was 15, with a median score of 1 (interquartile range 0-2). Charlson scores had a range of 0 to 23, with a median score of 0.

MCC19-RS Performance

Performance metrics of the MCC19-RS are outlined in Figure 1. The ROC curve of the MCC19-RS having AUC of 0.837 (95% Confidence interval [CI] 0.830-0.843) in comparison to the Charlson Comorbidity Index score AUC of 0.740 (95% CI 0.733-0.748). The calibration curve, Figure 2, demonstrates that the MCC19-RS predicts well in tose with the lowest predicted risk (< 5%).

Figure 1.

Sensitivity and specificity of the COVID-19 risk score against the Charlson Comorbidity Index.

Figure 2.

Calibration curve of the actual admission rate compared to predicted risk of admission for the COVID-19 risk score.

In a sensitivity analysis comparing variations across regions, the AUC in the Midwest was 0.838, (95% CI 0.829-0.846), Arizona was 0.787 (95% CI 0.774-0.800), and Florida was 0.845 (95% CI 0.829-0.861).

Discussion

Using a large cohort of patients positive for COVID-19 in any setting, we have further validated the strength of the MCC19-RS. This risk score demonstrated excellent discrimination power with an AUC of 0.837 and reasonable calibration. The performance of the score was consistent across regions, suggesting good reproducibility. We propose using a color coding of the score with green associated with scores of 0 to 2, yellow with scores of 3 to 5, and red with scores of 6 and above. The linear score, as noted, holds good discriminatory value, and should be maintained in any display. Table 2 shows that patients with scores in the 3 to 5 range have an average risk of admission in the 17% to 35% range, while scores of 2 and under have a 10% average risk of admission or lower and scores of 6 or greater have an average risk of admission of 50%.

Table 2.

Covid-19 Risk Score by Hospital Admission.

	Admitted (N = 4626)	Not admitted (N = 62 844)	Sensitivity	Specificity
Mayo Clinic COVID-19 risk score
0	289 (1.3%)	21 747 (98.7%)
1	619 (2.3%)	26 537 (97.7%)	93.8%	34.5%
2	628 (8.4%)	6885 (91.6%)	80.4%	76.9%
3	713 (17.4%)	3382 (82.6%)	66.8%	87.8%
4	738 (26.5%)	2044 (73.5%)	51.4%	93.2%
5	657 (35.4%)	1200 (64.6%)	35.4%	96.4%
6+	982 (48.4%)	1048 (51.6%)	21.2%	98.3%

Overall, this study identified age, sex, and comorbidities to be strong predictors of 30-day hospital admission among COVID-19 patients in agreement with other published reports.[12-14] Several scores have been proposed to assess the risk in COVID-19 positive patients progressing to more severe disease in the outpatient and inpatient settings, but they are limited by bias. Of the available outpatient-based scores assessing risk to progress to hospitalization (Table 3),[16-20] all require symptoms or objective signs at the time of diagnosis except for SARS2 and MCC19-RS. The primary purpose of developing the MCC19-RS was to allow the clinical practice to utilize real-time EHR data to classify patients into the appropriate risk category and to assess risk for need of admission based on the risk score.

Table 3.

Covid-19 Outpatient Risk Scores: Currently Available Scores for COVID-19 Risk, Comparing Outcomes of Interest, Diagnostic Performance, Derivation and Validation Techniques, and Required Variables.

Score		SARS2	OUTCoV	CD65-M	SODA	SOARS
Author		Dashti	Jacquerioz	Vila-Corcoles	Lopez-Pais	Chua
Setting		Outpatient	Outpatient	Outpatient >50 yo	Outpatient	Outpatient
Outcome		Admitted hospital	Admitted hospital	ICU/30 day-Mortality	Adverse event*	Mortality
Derivation	Cohort	10 496	965	282		821
	# with outcome	3401	80	64		258
	AUC	0.75	0.81	0.828		0.82
Validation	Cohort	1851			965	290/14 231
	# with outcome	204			124	94/4319
	AUC	0.77			0.858	0.80/0.74
Demographics	Sex	X			X
	Age	X	X	X	X	X
	Race	X
	Zip (economic)	X
	Smoke status	X
Comorbidities	Diabetes				X
	Hypertension		X
	Lung disease		X
	Stroke					X
Symptoms	Fever		X
	Dyspnea		X	X
	Confusion			X
	Myalgia			X
Objective	O2Sat				X
	Respirations					X
	BMI					X

Death, ICU admission, invasive mechanical ventilation, bleeding > BARC3, acute renal injury, respiratory insufficiency, myocardial infarction, acute heart failure, pulmonary emboli or stroke.

The strength of the MCC19-RS lies in the lack of a need to assess symptoms or objective signs thereby enabling the clinical practice to independently stratify risk either at the individual positive patient level or for assessing larger cohorts of patients. Accurate risk-predicting tools for various decision points are thought to be necessary to assist managing patients in the current COVID-19 pandemic to allocate limited resources. An EHR data integrated scoring system may assist the clinician who has to decide which patients are at high-risk for severe COVID-19 and that may benefit from remote patient monitoring. Mayo Clinic has also used MCC19-RS to assist in the identification of a higher risk cohort of patients to offer COVID-19 immunization when available.

Regarding the build within our EHR, we did create cut-points for mild (0-2 points), moderate (3-5 points), and high risk (6-15 points) to allow for color coding of the risk score when displayed in chart view or list views within the system. The score was added to inpatient and outpatient lists, as well as care management lists. We included it in multiple visualizations within the chart including “story board,” “snapshot,” and “sidebar” views. Figure 3 shows the score with coloring in a patient list view with “hover to discover” regarding the components for that patient. As noted, the score value is critical, but the color visualizations can help to quickly establish a general risk for the patient. In general, a clinical scoring system should not override overall clinical judgment as a clinician’s overall clinical assessment of the individual patient does take into consideration multiple inputs to approximate the risk for admission. The MCC19-RS may act as an important input to assist the clinician in assessing the risk of admission of the COVID-19 positive patient.

Figure 3.

Visualization of MCC19-RS in the Electronic Medical Record list view.

As with all retrospective studies, there are opportunities for improvement. For example, it is likely that some patients from the outpatient setting who were not admitted to the hospital may have had co-morbid conditions that were not discretely documented in the EHR. Since most of the components for MCC19-RS were chronic conditions, the majority of the conditions would have been accounted for discretely. Additionally, pregnancy appeared to not suggest admission, but the number of patients who were pregnant were few and did not appear to cause the score to perform poorly. Finally, the study cohort is not representative of the general population of patients in the United States, both in terms of factors such as race and ethnicity, and because the patients specifically sought treatment as a multi-campus health system.

Conclusion

The MCC19-RS is a functional score easily integrated into the EHR with excellent predictive performance for severe COVID-19 that can easily inform clinical practice and cohort management. Calculating the score does not require current symptoms, objective signs, or lab values so it can be easily used to stratify risks for COVID-19 severity at initial contact with the health systems, while considering therapeutics or within the context of distributing of COVID-19 vaccine.