Development of a healthcare system COVID Hotspotting Score in California: an observational study with prospective validation.

OBJECTIVE: To examine the value of health systems data as indicators of emerging COVID-19 activity.
DESIGN: Observational study of health system indicators for the COVID Hotspotting Score (CHOTS) with prospective validation. SETTING AND PARTICIPANTS: An integrated healthcare delivery system in Northern California including 21 hospitals and 4.5 million members. MAIN OUTCOME MEASURES: The CHOTS incorporated 10 variables including four major (cough/cold calls, emails, new positive COVID-19 tests, COVID-19 hospital census) and six minor (COVID-19 calls, respiratory infection and COVID-19 routine and urgent visits, and respiratory viral testing) indicators assessed with change point detection and slope metrics. We quantified cross-correlations lagged by 7-42 days between CHOTS and standardised COVID-19 hospital census using observational data from 1 April to 31 May 2020 and two waves of prospective data through 21 March 2021.
RESULTS: Through 30 September 2020, peak cross-correlation between CHOTS and COVID-19 hospital census occurred with a 28-day lag at 0.78; at 42 days, the correlation was 0.69. Lagged correlation between medical centre CHOTS and their COVID-19 census was highest at 42 days for one facility (0.63), at 35 days for nine facilities (0.52-0.73), at 28 days for eight facilities (0.28-0.74) and at 14 days for two facilities (0.73-0.78). The strongest correlation for individual indicators was 0.94 (COVID-19 census) and 0.90 (new positive COVID-19 tests) lagged 1-14 days and 0.83 for COVID-19 calls and urgent clinic visits lagged 14-28 days. Cross-correlation was similar (0.73) with a 35-day lag using prospective validation from 1 October 2020 to 21 March 2021.
CONCLUSIONS: Passively collected health system indicators were strongly correlated with forthcoming COVID-19 hospital census up to 6 weeks before three successive COVID-19 waves. These tools could inform communities, health systems and public health officials to identify, prepare for and mitigate emerging COVID-19 activity. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study included routinely available health systems data and previously described algorithms to confirm their value for COVID-19 hotspotting across a large and diverse population of 4.5 million patients cared for at 20 medical centres and >250 medical offices.

A composite score, implemented in June 2020, using these indicators showed strong correlation with COVID-19 census across two prospective COVID-19 waves with peak correlationwith a lead time of 4–6 weeks at regional and hospital levels.

Although model generalisability will likely vary across settings and geographies, a reduced form of the model, excluding regional call centre and email data, showed similar performance.

Continuously changing routine healthcare practices and patterns attributable to COVID-19 will require ongoing score re-examination and the potential use of other statistical approaches for refinement.

The COVID Hotspotting Score performance may change with large-scale vaccination programmes and the development of effective new treatments.

Introduction

COVID-19 is an unprecedented, dynamic and persistent threat to global health.1 Since initial reports implicated the SARS-CoV-2 virus in late 2019,2 COVID-19 disease has affected 126 million people worldwide resulting in nearly 3 million deaths.3 4 Efforts to contain SARS-CoV-2 spread have seen mixed success; numerous locales that withstood an initial wave of COVID-19 disease activity are now witnessing viral resurgence.5 These examples confirm epidemiological and simulation models that predicted ongoing waves of COVID-19 disease over the next year.6–11

Given the expectation of ongoing surges of COVID-19 disease, focus has turned towards data-driven approaches to identify the earliest signals of re-emergent viral activity.12–20 With enough lead time, communities with COVID-19 clusters or hospitals with anticipated growth (‘hotspots’) could intervene to mitigate and prepare for resurgence.1 21 22 Mitigation strategies augment existing contact tracing programmes with local proactive testing, syndromic surveillance and/or social distancing policy reinforcement to suppress spread.23 24 Preparation strategies can help secure adequate hospital space, healthcare personnel and key supplies like ventilators, medications or personal protective equipment.21 25 26 Without adequate lead time, COVID-19 surges can result in critical shortages in healthcare capacity and lagging public health policies that risk further destabilising vulnerable economic activity and community well-being.26–32

Numerous efforts are underway to evaluate promising approaches to identify and predict COVID-19 hotspots using aggregated social media, viral testing patterns, mobility, biometric and symptoms data.6 12–20 In this study, we investigated the development of a composite index to identify emerging hospital COVID-19-related activity using passively collected daily electronic health record (EHR) data from a large, regional integrated healthcare system. We further quantified the potential lead time that such data—aggregated as the COVID Hotspotting Score (CHOTS)—might offer to health systems and communities by using observational data and two periods of prospective validation data across three COVID-19 waves in Northern California.

Methods

The target population was all members within the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery system which serves 4.5 million members across 21 hospitals and >250 medical offices, supported by a regional Appointment and Advice Call Center available around the clock and a single comprehensive EHR systemwide.

Candidate health systems indicators

Because our health system had experienced only a single surge of significant COVID-19 hospital activity by 8 May 2020,33 34 we evaluated candidate leading indicators retrospectively from a convenience sample of health plan data drawn from 1 January 2015 through 8 May 2020 to identify temporal patterns present in prior seasonal influenza as well as for COVID-19. We denoted the COVID-19 period as 1 March 2020 forward. Based on existing literature, clinical judgement and expert opinion, we considered diverse health system indicators across KPNC related to healthcare utilisation, diagnosis codes, antimicrobial medication use, viral testing, patient communication with providers and respiratory or COVID-19-specific utilisation. For each indicator, we generated their daily count at the regional level as well as within each of 20 medical centres (local-level). We excluded one medical centre because their typical practice was to transfer their COVID-19 patients to a larger, jointly run neighbouring hospital. There were no missing or imputed values since all the indicators are counts with values >0. Online supplemental appendix table 1 describes baseline demographic data in each of six subregions in KPNC.

For healthcare utilisation, we identified potential leading indicators including all outpatient ambulatory visits with a ‘respiratory infection’ diagnosis based on Healthcare Cost and Utilization Project Clinical Classification Software single-level groupers for pneumonia (122), influenza (123), acute bronchitis (125) and other respiratory infections (126).35 Respiratory infection ambulatory visits were grouped as in-person, telephone or video visit and denoted as routine or urgent. We evaluated utilisation indicators for emergency department (ED) influenza-like illness (ILI) visits based on primary symptoms of cough, dyspnoea and/or fever. We assessed COVID-19-specific utilisation based on urgent and routine clinic visits as well as hospital, intensive care unit and mechanical ventilator encounters containing COVID-19 diagnoses. For antimicrobial treatment, we evaluated outpatient antibiotic and oseltamivir prescriptions. For viral testing, we quantified tests ordered and positive results for SARS-CoV-2/COVID-19 tests, 14-panel respiratory viral panel PCR testing (RVP 14), and a combined rapid test for influenza A/B and respiratory syncytial viruses. For patient communication, we evaluated daily counts of regional call centre data focusing on patient calls which activated regional ‘cough and cold’ or ‘COVID-19’ protocols.36 37 We also identified patient-initiated ILI email communications based on subject headers containing terms similar to sore throat, shortness of breath, fever, cough, chest discomfort, chills, influenza and/or COVID-19 (removing those related to vaccination).38 39

Temporal patterns and aggregation

For each indicator with data available prior to 1 March 2020, we visually assessed changes in standardised counts and slopes from prior seasonal influenza surges and from the first wave of COVID-19 disease to evaluate their temporal association with increases in hospital census and health system utilisation. We identified historical periods of high regional influenza-related utilisation based on the top fifth percentile of daily summed values of influenza tests, oseltamivir prescriptions and outpatient antibiotic use. Because COVID-19 emergence caused significant changes in healthcare utilisation patterns and practices (eg, transitions from in-person to virtual visits; reduced availability of swabs for routine non-COVID-19 viral testing; decreased non-COVID-19 hospital census), we aggregated several individual data elements. For example, we grouped in-person, telephone and video respiratory infection visits together because remote visits largely supplanted on-site visits after COVID-19 onset. We also aggregated data for children (age <18 years) and adults (age ≥18 years) within each indicator.

Identifying temporal changes in leading indicators

To identify statistically significant changes over time in each indicator we used two methods: change point detection and moving average 7-day slopes. We used the change point analysis (CPA) algorithm to identify changes from a fixed initial baseline mean and from the last change point mean.40 CPA algorithms can detect changes in the mean values of time-series data and to identify periods marked by a significant change in the mean. They have been used previously to assess for changes in seasonal influenza data. Because we focused on identifying emerging hotspots of increasing COVID-19 activity, we down-weighted statistically significant changes in the new mean compared with the last change point mean after 14 days. We also calculated the slope of the 3-day moving average over the prior 7 days to identify statistically significant positive slope deflections.

To account for historical patterns among indicators with pre-COVID-19 data available, we estimated expected values based on seasonality and day of week via ordinary least squares using data from 1 January 2015 through 31 December 2019. Starting from 1 January 2020, we calculated the residuals as the difference between observed and expected values and assessed for significant changes in CPA (vs baseline and last value) and slope using the residual, rather than actual, values. Thus, at the regional and medical centre levels, each indicator underwent three significance tests on each day. We scored each combination of these three significance tests and summed them together to generate a daily CHOTS, focusing on simple points-based calculation that could be instantiated by June 2020 to prepare for the next wave of COVID-19 activity.

Because of the urgent need to establish a hotspotting tool in our health system to prepare for forthcoming COVID-19 waves, we used visual inspection and association analysis of potential indicators with prior seasonal influenza patterns as well as clinical judgement and heuristics to identify the final leading indicators and relevant score components. Final score rules are described later in the text and in online supplemental appendix tables 2 and 3; significance testing code is available in online supplemental file 1. Because we designed the CHOTS to focus on detecting emerging activity rather than on attempting to predict absolute hospital census, our health system also implemented and used more traditional infectious disease epidemiology and fitted curve models to predict shorter-term absolute hospital census estimates.6 34 41

Prospective evaluation

Because COVID-19 activity is often measured by future COVID-19 hospital census increases, we prospectively evaluated the performance of the CHOTS developed by June 2020 in two successive waves: using prospective data through 30 September 2020 (wave 2) and in a temporally independent prospective sample from 1 October 2020 through 21 March 2021 (wave 3). Within each time period, we calculated the cross-correlation between daily CHOTSs and standardised COVID-19-specific hospital census at regional and medical centre levels when the CHOTS was lagged by 7–42 days (ie, when the CHOTSs were examined against standardised census values from 7 to 42 days later). For comparison, we also examined the cross-correlation of each of the individual indicators with COVID-19 hospital census to evaluate their performance. Finally, we also generated ‘reduced’ CHOTS that included all indicators except for call centre data and secure email messages, since those might not be routinely available in all health system settings. We calculated the correlation confidence intervals via the Fisher transformation.42

Patient and public involvement

This study did not include patient or public involvement.

All analyses were conducted with SAS V.14.5 or R V.3.6.2.

Results

We examined potential COVID-19 indicators within 23 summary groups (table 1) that included a total of 35 086 325 data elements in our initial validation period through 30 September 2020. The highest count totals were seen for indicators available retrospectively from January 2015 through September 2020 including respiratory infection clinic visits (n=9 459 882) and ED encounters (n=7 423 168). Among COVID-19-relevant predictors available from 1 March 2020 forward, the most common included COVID-19 tests ordered (n=936 360) and COVID-19-related call centre calls (n=531 765).

Table 1

Health system indicators for COVID-19 activity evaluated across the Kaiser Permanente Northern California integrated healthcare delivery system sorted by total counts, including those incorporated within the COVID Hotspotting Score (grey highlights)

Predictor group	Description	Count	First date available	Score weight
Respiratory clinic visits	Clinic visits with a respiratory infection diagnosis HCUPCCS code	9 459 882	1/1/2015	Minor
ED encounters	All ED encounters	7 423 168	1/1/2015
Non-COVID-19 hospital census	Number of non-COVID-19 patients admitted to the hospital each midnight	4 133 163	1/1/2015
Call centre calls for cough and cold	Regional call centre calls which activated the ‘cough/cold’ script	4 114 425	1/1/2017	Major
Respiratory urgent visits	Urgent care visits with a respiratory infection diagnosis	3 902 026	1/1/2015	Minor
Hospitalisations	KPNC admits with inpatient or observation status, non-labour and delivery	1 603 401	1/1/2015
Electronic secure messages	Patient-initiated emails to KPNC clinicians with ILI-potential subject headers	1 141 132	1/1/2015	Major
ED respiratory encounters	ED encounters with respiratory or fever-related primary symptom	1 019 963	1/1/2015
Total COVID-19 tests ordered	Total COVID-19 tests ordered	936 360	1/3/2020
Call centre calls for COVID-19	Regional call centre calls which activated the ‘COVID-19’ script	531 765	1/3/2020	Minor
ED respiratory hospital admissions	Hospital admissions among ED respiratory encounters	230 099	1/1/2015
Influenza A, B, RSV tests	Rapid influenza, A, B, RSV tests	120 069	1/1/2015
ED COVID-19 tests	COVID-19 tests ordered in the ED	97 948	1/3/2020
COVID-19 clinic visits	Clinic visits with a COVID-19 diagnosis	99 954	1/4/2020	Minor
COVID-19 and PUI* hospital census	COVID-19 or PUI patients in the hospital at midnight	66 488	1/3/2020
COVID-19+ tests	Total positive COVID-19 tests	52 005	1/3/2020	Major
New COVID-19+ tests per member	New positive COVID-19 tests per member	49 359	1/3/2020
COVID-19 hospital census	COVID-19 patients admitted to the hospital each midnight	41 385	1/3/2020	Major
Respiratory viral panel tests	14-item respiratory viral panel PCR testing	16 638	1/1/2015	Minor
COVID-19 urgent care visits	Urgent care visits with a COVID-19 diagnosis	16 142	1/4/2020	Minor
COVID-19 ICU	COVID-19 patients admitted to the ICU each midnight	13 020	1/3/2020
ED COVID-19+ tests	Positive COVID-19 tests resulted in the ED	8656	1/3/2020
COVID-19 vent	COVID-19 patients on invasive mechanical ventilation each midnight	9277	1/3/2020

Data are counted through 30 September 2020. Data related to COVID-19 measures were only available from 1 March 2020 forward. The final column shows which indicator group data contributed to the final COVID Hotspotting Score as well as their weighting in the score: major variables contributed double the weight of minor variables.

ED, emergency department; HCUP CCS, Healthcare Cost and Utilization Project Clinical Classification Software groups 122 (pneumonia); 123 (influenza); 125 (acute bronchitis) and 126 (other respiratory infections); ICU, intensive care unit; ILI, influenza-like illness; PUI, persons under investigation; RSV, respiratory syncytial virus.

Figure 1 displays the seasonal patterns evident among selected key indicators during prior periods of increased seasonal influenza activity through September 2020. Figure 1A exhibits the temporal relationship between standardised counts of influenza testing and outpatient oseltamivir and antibiotics with the top fifth percentile of days with the highest influenza-related utilisation (blue bars). Other key health system indicators related to healthcare utilisation, patient-initiated communication and testing also showed similar increases timed with high influenza-related activity (figure 1B). The onset of the COVID-19 period in our health system on 1 March 2020 (red line) resulted in notable changes in nearly all indicator patterns relative to prior seasonal patterns.

Figure 1

Each plot displays daily standardised counts of each indicator aggregated at the regional level. The blue bands indicate periods of higher influenza seasonal activity indicative of days where the aggregate sum of influenza A/B and respiratory syncytial virus testing, oseltamivir prescriptions and antibiotic use were above the fifth percentile for all daily values. The red line marks 1 March 2020, which was the first date of significant COVID-19 regional impact in the Kaiser Permanente Northern California healthcare system.

By examining prior temporal patterns among pre-COVID-19 indicators and assessing the correlation among COVID-19-specific indicators with hospital census through May 2020, we used visual inspection and clinical judgement to select 10 variables for calculating the CHOTS including 4 major and 6 minor variables (table 1). Major variables included: (1) cough and cold calls; (2) ILI-like email message subject headers; (3) new positive COVID-19 test rates and (4) COVID-19-specific hospital census. Minor variables contributed half the weight of major variables and included: (1) COVID-19-specific call centre calls; (2) respiratory infection routine clinic visits; (3) respiratory infection urgent visits; (4) COVID-19 clinic visits; (5) COVID-19 urgent clinic visits and (6) RVP 14 tests ordered. Online supplemental appendix figure 1 exhibits an example of CPA and slope significance tests among each of the indicators from 1 January 2020 through 30 September 2020. We assigned points to each combination of the three significance tests for each indicator and summed them together to produce daily scores for the region and for each medical centre (online supplemental appendix table 1). An example of the CHOTS calculation is provided in online supplemental appendix table 3.

Figure 2A displays the CHOTS lagged by 28 days overlaid atop the regional COVID-19 hospital census through 30 September 2020; online supplemental appendix figure 2A shows the same plots at each medical centre. At the regional level, the correlation between the CHOTS and COVID-19 hospital census peaked when CHOTS was lagged 28 days reaching a high value of 0.78 (95% CI 0.71 to 0.84). With a 42-day lag, the CHOTS and COVID-19 census cross-correlation was 0.69 (95% CI 0.59 to 0.77). Table 2 exhibits the lagged correlation between medical centre CHOTSs and their COVID-19 hospital census with the lagged cross-correlation highest at 42 days for one facility (0.63), at 35 days for nine facilities (range, 0.52–0.73), at 28 days for eight facilities (range, 0.28–0.74) and at 14 days for two facilities (range, 0.73–0.78).

Figure 2

Standardised values of the 28-day lagged COVID Hotspotting Score (CHOTS; red dotted line and right y-axis) and COVID-19 hospital census (black line and left x-axis) at the regional level from 1 April 2020 to 30 September 2020 (waves 1 and 2, A) and standardised values of the 35-day lagged CHOTS (red dotted line and right y-axis) and COVID-19 hospital census (black line and left x-axis) at the regional level from 1 October 2020 to 21 March 2021 (wave 3, B). Data are smoothed using a locally weighted smoothing LOESS model. The smoothing parameter is the proportion of data in the local neighbourhood: a value near 0 results in a curve that nearly interpolates the data whereas a value near 1 is nearly a straight line.

Table 2

Correlation between the COVID-19-specific hospital census and lagged COVID Hotspotting Score between 7 and 42 days at the regional and medical centre (facility values A through T) levels

Location	Correlation between lagged COVID Hotspotting Score and forthcoming COVID-19-specific hospital census
	7 days	14 days	21 days	28 days	35 days	42 days
KPNC region	0.35	0.55	0.70	0.79	0.78	0.71
Facility A	0.52	0.68	0.77	0.78	0.72	0.58
Facility B	0.06	0.34	0.41	0.55	0.59	0.56
Facility C	0.18	0.35	0.48	0.56	0.54	0.50
Facility D	0.48	0.61	0.71	0.77	0.72	0.62
Facility E	0.26	0.45	0.59	0.65	0.64	0.56
Facility F	0.29	0.39	0.52	0.61	0.57	0.53
Facility G	0.05	0.28	0.48	0.63	0.72	0.70
Facility H	0.29	0.37	0.50	0.55	0.51	0.44
Facility I	0.29	0.48	0.63	0.72	0.73	0.69
Facility J	0.06	0.26	0.42	0.50	0.58	0.52
Facility K	0.35	0.46	0.57	0.62	0.55	0.47
Facility L	0.17	0.23	0.37	0.53	0.53	0.42
Facility M	0.41	0.43	0.35	0.36	0.33	0.28
Facility N	0.36	0.43	0.50	0.54	0.61	0.64
Facility O	0.33	0.48	0.61	0.71	0.74	0.69
Facility P	0.15	0.33	0.48	0.59	0.59	0.56
Facility Q	0.15	0.34	0.47	0.53	0.56	0.48
Facility R	0.67	0.77	0.72	0.61	0.46	0.31
Facility S	0.25	0.38	0.49	0.51	0.52	0.51
Facility T	0.54	0.69	0.73	0.72	0.67	0.54

Pearson correlation values were calculated on a daily basis including data from 1 April 2020 through 30 September 2020. The highest correlation between the COVID Hotspotting Score and lagged COVID-19 census is indicated in red font.

Among individual indicators, the cross-correlations with COVID-19 census over the next 7–14 days were highest for current COVID-19 census (0.94; 95% CI 0.93 to 0.96) and the count of new positive COVID-19 tests (0.90; 95% CI 0.86 to 0.92—figure 3). For indicators lagged 14–28 days, the correlation was highest for COVID-19 urgent visits (0.83; 95% CI 0.78 to 0.87) and COVID-19 call centre calls (0.83; 95% CI 0.77 to 0.87). Beyond 28 days, the lagged CHOTS showed the highest correlation with COVID-19 census and remained >0.60 when lagged up to 49 days.

Figure 3

The horizontal dotted line shows a correlation coefficient ≥60%. The correlation between lagged COVID Hotspotting Score (CHOTS) and hospital census increases as lag increases, peaking at 28 days while the cross-correlation of individual indicators decreases. Indicators displayed in this figure are those that have an average positive correlation with COVID-19 hospital census. ILI, influenza-like illness.

In a temporally independent prospective validation sample (coinciding with the third wave from 1 October 2020 through 21 March 2021), the CHOTS displayed a maximum cross-correlation of 0.73 with a 35-day lag (figure 2B, table 3 and online supplemental appendix figure 2B). The highest lagged cross-correlation occurred at 42 days for one facility (0.65), at 35 days for six facilities (range, 0.59–0.68), at 28 days for nine facilities (0.52–0.75) and at 21 days for four facilities (0.55–0.77). In a ‘reduced’ CHOTS, which removed call centre and secure email messages from inclusion, the regional correlation was of similar magnitude (0.74–0.75; online supplemental appendix table 4) with a maximum lag at 28 days.

Table 3

Correlation between the COVID-19-specific hospital census and lagged COVID Hotspotting Score between 7 and 42 days at the regional and medical centre (facility values A through T) levels

Location	Correlation between lagged COVID Hotspotting Score and forthcoming COVID-19-specific hospital census
	7 days	14 days	21 days	28 days	35 days	42 days
KPNC region	0.38	0.54	0.66	0.73	0.73	0.66
Facility A	0.35	0.47	0.58	0.66	0.64	0.59
Facility B	0.38	0.54	0.63	0.66	0.63	0.56
Facility C	0.60	0.72	0.77	0.75	0.68	0.53
Facility D	0.17	0.30	0.43	0.55	0.59	0.59
Facility E	0.48	0.62	0.69	0.67	0.63	0.54
Facility F	0.07	0.27	0.43	0.54	0.62	0.65
Facility G	0.40	0.57	0.67	0.71	0.69	0.60
Facility H	0.62	0.70	0.71	0.63	0.55	0.45
Facility I	0.32	0.51	0.62	0.67	0.67	0.61
Facility J	0.35	0.45	0.58	0.67	0.68	0.61
Facility K	0.04	0.26	0.45	0.59	0.62	0.62
Facility L	0.54	0.67	0.74	0.75	0.69	0.55
Facility M	0.33	0.33	0.42	0.52	0.50	0.40
Facility N	0.40	0.53	0.55	0.52	0.39	0.39
Facility O	0.39	0.52	0.64	0.71	0.68	0.61
Facility P	0.47	0.63	0.72	0.73	0.71	0.61
Facility Q	0.44	0.60	0.72	0.74	0.71	0.62
Facility R	0.18	0.34	0.47	0.58	0.61	0.60
Facility S	0.21	0.36	0.51	0.61	0.65	0.63
Facility T	0.16	0.33	0.48	0.58	0.68	0.67

Pearson correlation values were calculated on a daily basis including data from 1 October 2020 through 21 March 2021. The highest correlation between the COVID Hotspotting Score and lagged COVID-19 census is indicated in red font.

Discussion

Many countries and locales are now facing new waves of COVID-19-related infections highlighting the need for early warning systems that can alert communities, hospitals and public health officials to prepare for an impending rise in COVID-19 impact, including hospitalisations.5 In this study, we examined passively collected health system EHR data available with a single day lag to evaluate how changes in these indicator data could identify impending increases in COVID-19 hospital census. We found that, even before COVID-19, many of these EHR data showed significant temporal variability indicative of seasonal influenza’s impact on utilisation. We then developed a score, comprising 10 daily data elements, which showed strong correlation with COVID-19-specific hospital census up to 4–6 weeks in advance of two subsequent COVID-19 waves in Northern California. Even at the subregional level, where COVID-19 infections and hospitalisations have exhibited substantial heterogeneity in timing and size, cross-correlation varied but remained strong at most individual medical centres over the same periods.

Strengths and weakness in relation to other studies

Over the course of the COVID-19 pandemic, numerous forecasting and simulation tools have been developed which combine diverse types of data (eg, COVID-19 testing rates, COVID-19-specific hospitalisation or death rates, symptom surveillance, evidence of community-level social distancing, biometrics) to identify emerging hotspots.8 12–20 24 34 43 While these tools have proven valuable at an aggregate level, they often lack the geographic and temporal specificity that reflects heterogeneous emerging patterns within local communities. The most reliable indicators, like COVID-19 hospitalisations or death rates, are also known to lag significantly behind emerging disease activity. For these reasons, such tools have had limited value for informing our local medical centres about their medium-term preparation and mitigation activities prior to COVID-19 surge.41 In addition, these tools often focus on trying to predict the precise COVID-19 hospital census, which has been shown to be highly variable across serial COVID-19 waves as well as across waves of other pandemic disease including 1918 influenza and 2009 H1N1. This variability is attributable to many factors including the social distancing behaviours of individuals and communities, the policies enacted by local and national governments, the dynamic biology of the virus itself, and other factors that have yet to be elucidated.

To address these limitations, in this study, we focused on passively collected EHR data readily available within our health system that would maximally reflect daily local patterns and were amenable to urgent development, testing and deployment. We chose to use existing algorithms for identifying temporal change in indicators rather than more advanced, and potentially more complex, machine learning approaches to facilitate development and instantiation. Because the CHOTS is designed to inform medium-term decisions, we also chose not to build a model to generate precise predictions of hospital census and instead used other curve-fitting and epidemiological models to predict absolute hospital census numbers over short intervals.6 34 41 Finally, we used a points-based scoring system that reflected our intuitive approach to generating the CHOTS, a necessity driven by the paucity of reliable historical data available in June 2020 after only a single wave of significant COVID-19 activity in Northern California.

Importantly, we found that individual COVID-19 disease indicators showed very strong correlation with subsequent hospital census when these data were lagged between 1 and 4 weeks over three successive waves of activity through March 2021. For example, the number of new positive COVID-19 tests was highly correlated with forthcoming hospital census over the following 2 weeks. Similarly, call centre volume was strongly correlated with hospital census at 3–4 weeks. ILI-like email communication between patients and providers showed lower correlation with COVID-19 census. Thus, our data suggest that single leading indicators like these can be effectively used to inform short-term hospital census predictions or to identify emerging COVID-19 activity with a lead time of 3–4 weeks. Indeed, in our health system, high-value indicators like call centre data have been used to direct our response to emerging seasonal influenza activity for many years.

However, the correlation of individual indicators with census decreased steadily beyond 2–4 weeks, while the correlation of the aggregate score continued to increase and remain strong through 6 weeks. While any lead time to prepare before COVID-19 hospitalisations increase has tremendous value, narrower windows may hamper effective mitigation and adequate preparation for hospitals and public health agencies. With adequate lead time, individual hospitals can focus on ensuring that the additional staff, supplies and space needed to care for a large number of COVID-19 patients in advance of their expected need.26–31 Public health officials can also titrate social distancing policies to target expected, rather than current or lagging, COVID-19 activity. A ‘reduced’ form CHOTS also showed robust performance suggesting that the score could have value in health systems without ready access to patient-initiated data from call centres or emails.

Implications for clinicians and health system leaders

The CHOTS has been in use in our health system since June 2020 and is updated on a daily basis in a variety of dashboards that are accessible to our health system and hospital leadership. After KPNC’s COVID-19 census began to ebb following wave 2, the alarming increase in the CHOTS before wave 3 was used to inform the reopening of daily Regional COVID-19 Command Center operations. The CHOTS tool has also been used to inform decisions about health system staffing and resource allocation as well as clinical care, based on the expected rise, stabilisation or fall of COVID-19 activity across different subregions and individual medical centres. Finally, the CHOTS tools have also informed decisions about the urgency of health system communications with members, communities and public health agencies, particularly during periods when the easing of social distancing behaviours occurred concurrently with the emergence of increasing COVID-19 hotspotting signals.

Strengths and limitations of the current study

The major strength of this study was its use of comprehensive and diverse EHR indicator data across a large and diverse integrated healthcare delivery system to demonstrate the value of these data for COVID-19 preparation. Because we experienced significant heterogeneity across medical centres with respect to COVID-19 impact, we were also able to compare its performance within individual subregional communities. By focusing on a parsimonious set of indicators and available algorithms, without developing a traditional predictive model, we were able to rapidly deploy our tool which proved extremely valuable for COVID-19 preparation and planning.44 45

There are several limitations to this study. Most importantly, our data come from a single integrated healthcare system in a single region of the USA. Thus, we were likely able to capture more comprehensive data within our EHR and data systems since our patients receive the overwhelming majority of their care within our facilities. Thus, the generalisability of our tool may vary across settings and geographies, particularly for health systems which may lack robust call centre and/or email communications systems, protocols and data. Our data also lack indicators of local COVID-19 activity that does not occur in our members but that still likely impacts hospitalisation rates. Second, routine healthcare patterns have changed dramatically owing to COVID-19 including the shift from in-person to virtual care as well as the deferral of routine healthcare including surgical procedures.46 We attempted to aggregate diverse data—–for example, consolidating in-person, telephonic and video visits among adults and children for respiratory diseases—into single indicators to minimise the impact of practice changes on score calculation. However, the patterns we identified during the study period are likely to continue to change and require ongoing re-examination and refinement.

Third, we also did not attempt to develop a predictive model designed to precisely estimate absolute hospital census, instead focusing on a hotspotting approach designed to give the earliest signals of incipient COVID-19 activity that might impact hospitalisation. We have built and used other models for absolute census prediction but have found that their accuracy is greatest over very short intervals like 1–2 weeks, limiting their longer-range use in health system preparation. Fourth, we generated and deployed the CHOTS during a time of great uncertainty following the first wave of COVID-19 activity in California. As a result of the extreme urgency to prepare our health system, we depended on clinical judgement and heuristics, in addition to prior health system influenza patterns, to develop our score. With the luxury of time, more advanced machine learning or statistical techniques may have produced different calculations. Small sample sizes in each facility may have also impacted statistical significance testing. Nonetheless, the CHOTS continued to show very strong performance through the third wave of COVID-19 in Northern California. Finally, we examined the performance of the score during a period of low regional influenza activity; these patterns may continue to change based on seasonal influenza.

Unanswered questions and future research

The implications of our findings on future research remain dynamic because of the tremendous uncertainties owing to the biology and impact of COVID-19 worldwide and differences in regional responses driven by health policies and treatments for the disease. However, tools which evaluate passively collected leading indicators beyond only positive COVID-19 case counts, hospital census or deaths, will continue to provide strong utility to inform health system decisions about preparation, mitigation and suppression of this pandemic. Even with some effective vaccination campaigns underway in 2021, there is persistent concern that the impact of new variants or incomplete and waning herd immunity will mean that ongoing COVID-19 activity will continue on a seasonal or intermittent basis. In our health system, the CHOTS will continue to inform our health system response, particularly entering the fall of this year. Additional external validation of this tool will also be important to assure its generalisability beyond our system. We will make our CPA and slope algorithm available for others who are interested in applying them to their own daily health system data to assess for informative leading indicators of local COVID-19 activity. Undoubtedly, additional improvements to this tool can likely be realised by incorporating non-health system community-level data across diverse domains indicative of disease spread including mobility, wastewater, biometrics, COVID-19 genotyping, and symptoms.

Conclusion

In conclusion, we used 10 health system indicators of potential COVID-19-related disease activity to generate an aggregate score that was strongly correlated with forthcoming hospital census at 4–6 weeks at both a regional and local level. While individual indicators showed very strong cross-correlation with impending COVID-19 hospital census over a 1–4 weeks timeframe, this hotspotting tool could potentially extend the lead time for local communities, health systems, and public health officials to prepare for and mitigate emerging COVID-19 activity.