Validation of a neighborhood-level COVID Local Risk Index in 47 large U.S. cities.

OBJECTIVES: To present the COVID Local Risk Index (CLRI), a measure of city- and neighborhood-level risk for SARS COV-2 infection and poor outcomes, and validate it using sub-city SARS COV-2 outcome data from 47 large U.S. cities.
METHODS: Cross-sectional validation analysis of CLRI against SARS COV-2 incidence, percent positivity, hospitalization, and mortality. CLRI scores were validated against ZCTA-level SARS COV-2 outcome data gathered in 2020-2021 from public databases or through data use agreements using a negative binomial model.
RESULTS: CLRI was associated with each SARS COV-2 outcome in pooled analysis. In city-level models, CLRI was positively associated with positivity in 11/14 cities for which data were available, hospitalization in 6/6 cities, mortality in 13/14 cities, and incidence in 33/47 cities.
CONCLUSIONS: CLRI is a valid tool for assessing sub-city risk of SARS COV-2 infection and illness severity. Stronger associations with positivity, hospitalization and mortality may reflect differential testing access, greater weight on components associated with poor outcomes than transmission, omitted variable bias, or other reasons. City stakeholders can use the CLRI, publicly available on the City Health Dashboard (www.cityhealthdashboard.com), to guide SARS COV-2 resource allocation.

Introduction

As the SARS COV-2 pandemic continues, local response efforts have been hampered by a lack of timely and geographically granular information. In the United States (U.S.), SARS COV-2 incidence (The New York Times, 2021), mortality (Centers for Disease Control and Prevention/National Center for Health Statistics), and vaccination (Centers For Disease Control And Prevention, 2021) data are widely available at the county level, but similar data have not been as easily accessible for smaller geographies, like cities or neighborhoods. One reason for this is that the collection, cleaning, and dissemination of SARS COV-2 surveillance data is the responsibility of local health departments, which may lack the funding and staff required to undertake such efforts, especially in small and mid-sized cities (NACCHO, 2020).

In this situation, local health departments often rely on readily available county-level data. While county-level data are essential for public health surveillance and planning, city and sub-city data are vital to guide local pandemic response efforts, particularly because more than 80% of the U.S. population lives in urban areas (U.S. Census Bureau, 2021). City populations often differ substantially from the populations of counties in which they are located, causing county-level metrics to be insufficient proxies for city-level measures (Spoer et al., 2020). This is consistent with Tobler's first law of geography, which states “everything is related to everything else, but nearer things are more related than distant things”. Spatially granular data is most effective at describing the health-related conditions in a specific place, and as such, can inform more effective responses to the ongoing SARS COV-2 epidemic.

To address the need for spatially granular data that can guide city-level SARS COV-2 response, the City Health Dashboard (Department Of Population Health Nyu Langone Health, 2021) (the Dashboard), a website that provides free access to a range of health and health determinant metrics for over 750 U.S. cities, created the COVID Local Risk Index (CLRI) in June 2020. The CLRI provides a city- and neighborhood-level metric that characterizes risk of poor SARS COV-2 outcomes (high SARS COV-2 transmission and potential for severe SARS COV-2 illness) to help guide resource allocation and interventions. Given the urgent need at the time for sub-county SARS COV-2 data tools, the Dashboard released the CLRI before sufficient small-area SARS COV-2 outcome data were publicly available to validate the index.

To validate the CLRI at the smallest possible geography, the Dashboard has partnered with Drexel's Urban Health Collaborative (UHC). The UHC compiled SARS COV-2 data from several U.S. cities, leveraging publicly available data when possible, and requesting data not otherwise publicly available directly from city health departments, including SARS COV-2 positivity, incidence, hospitalizations, and mortality counts (Bilal et al., 2022b). We selected a range of SARS COV-2 outcome metrics because risk factors, transmission, and poor outcomes tend to cluster geographically, and because the CLRI was designed to capture both transmission and severity. We chose to capture both transmission and severity as they have different policy implications. On the one hand, preventing transmission may require a broader focus on avoiding exposure to SARS-COV 2, which may be the key driver of disparities (Bilal et al., 2022a). On the other hand, addressing severity may require longer term policies that reduce chronic disease burden. While the range of metrics available for each city varied, we identified a core set of metrics to validate the CLRI for a subset of cities displayed on the Dashboard. In this paper we describe the methods used by the Dashboard team to calculate the CLRI and then use SARS COV-2 data from the UHC to validate whether the CLRI accurately captures differences in risk for SARS COV-2 transmission and severity in select U.S cities.

Methods

The Dashboard, a Robert Wood Johnson Foundation-funded data platform, provides data for 766 U.S. cities, including all U.S. cities with population 50,000 or greater and 10 smaller New Jersey cities. The Dashboard team published the CLRI in June 2020 at the city- and neighborhood-level after reviewing other SARS COV-2-related risk indices (Surgo Ventures, 2020a; Social Progress Imperative, 2020) and emerging literature on demographic factors and health conditions related to SARS COV-2 transmission and severity. The Dashboard team then re-evaluated and updated the CLRI in March 2021 based on new high-quality research published through October 2020 (Gottlieb et al., 2020; Gupta et al., 2020; Hamidi et al., 2020a, 2020b; Hirsch et al., 2020; Kim et al., 2021; Petrilli et al., 2020; Reichberg et al., 2020; Rosenberg et al., 2020; Rozenfeld et al., 2020; Van Gerwen et al., 2020; Williamson et al., 2020), in consultation with SARS COV-2 expert researchers. Information about the index calculation is available in the Dashboard's technical documentation (Gofine et al., 2021).

Index components

The CLRI provides a combined city- and neighborhood-level assessment of SARS COV-2 infection risk and illness severity (census tracts were used to proxy neighborhoods). The CLRI is comprised of three groups of metrics: (1) social vulnerability, which includes metrics serving as a proxy for SARS COV-2 infection risk, (2) SARS COV-2-related chronic health conditions, which includes metrics contributing to potential increased severity of SARS COV-2 illness, and (3) SARS COV-2-related demographics, which captures groups that may be higher risk for SARS COV-2 infection and severity of SARS COV-2 illness (Table 1 ).

Table 1

COVID local risk index components.

Group	Group Weight	Sub-Group	Component	Component Weight
Social Vulnerability	30%	Socio-Economic Status	Persons below poverty	2%
			Civilian (age 16+) unemployed	2%
			Per capita income	2%
			Persons (aged 25+) with no high school diploma	2%
		Household Composition and Disability	Persons aged 65+	2%
			Persons aged 17 and younger	2%
			Civilian non-institutionalized population with a disability	2%
			Single parent household with children under 18	2%
		Minority Status and Language	Minority (all persons except white, non-Hispanic)	2%
			Persons (age 5+) who speak English “less than well"	2%
		Housing Type and Transportation	Housing in structures with 10+ units	2%
			Mobile homes	2%
			At household level (occupied housing units), more people than rooms	2%
			Households with no vehicle available	2%
			Persons in institutionalized group quarters	2%
SARS COV-2-related Chronic Health Conditions	43%	–	Chronic Obstructive Pulmonary Disease among adults 18+	4%
			Coronary heart disease among adults aged 18+	5%
			Diagnosed diabetes among adults aged 18+	6%
			Chronic kidney disease among adults aged 18+	9%
			Obesity among adults aged 18+	18%
SARS COV-2-related Demographics	27%	–	Minority (all persons except non-Hispanic white)	12%
			Persons aged 75 to 84	11%
			Persons aged 85+	5%

Social vulnerability

The social vulnerability component group captures neighborhood social and demographic factors associated with increased risk for SARS COV-2 infection. We measured social vulnerability through the Centers for Disease Control and Prevention's (CDC) Social Vulnerability Index (SVI). The SVI is a validated, peer-reviewed index that measures a community's vulnerability to harm caused by a natural disaster, including disease outbreak (Flanagan et al., 2011, Centers For Disease Control And Prevention, 2018). SVI is correlated with SARS COV-2 positivity, incidence and mortality (Nayak et al., 2020; Bilal et al., 2021). The SVI was calculated following the procedure created by the CDC using U.S. Census American Community Survey (ACS) 2014–2018 5-year estimates (Flanagan et al., 2011, Centers For Disease Control And Prevention, 2018; Nayak et al., 2020). A list of the variables included in the CLRI, including SVI variables, is available in Table 1.

SARS COV-2-related chronic health conditions

The SARS COV-2-related chronic health conditions component group incorporates known risk factors for increased severity of SARS COV-2 illness. The pool of potential components was limited to metrics for which estimates were available from CDC's PLACES Project (2018, 1-year Modeled Estimates; methods detailed elsewhere) (Places: Local Data For Better Health, 2020, 500 Cities: Local Data for Better Health, 2018). Candidate components with equivocal evidence were excluded. The following chronic health conditions were included: obesity (Williamson et al., 2020; Kim et al., 2021; Van Gerwen et al., 2020; Ebinger et al., 2020; Petrilli et al., 2020; Rozenfeld et al., 2020), chronic kidney disease (Rozenfeld et al., 2020; Gottlieb et al., 2020; Van Gerwen et al., 2020; Petrilli et al., 2020; Kim et al., 2021), diabetes (Ebinger et al., 2020; Azar et al., 2020; Van Gerwen et al., 2020; Hirsch et al., 2020; Kim et al., 2021; Williamson et al., 2020), chronic obstructive pulmonary disease (COPD) (Van Gerwen et al., 2020, Cummings et al., 2020; Kim et al., 2021; Williamson et al., 2020), and coronary heart disease (Azar et al., 2020; Gottlieb et al., 2020; Hirsch et al., 2020; Cummings et al., 2020; Gupta et al., 2020; Williamson et al., 2020).

SARS COV-2-related demographics

The SARS COV-2-related demographics component group includes demographic factors related to both SARS COV-2 infection and severity. This group includes density of older adult and racial/ethnic minority populations (Rozenfeld et al., 2020; Azar et al., 2020; Gottlieb et al., 2020; Van Gerwen et al., 2020; Ebinger et al., 2020; Hirsch et al., 2020; Williamson et al., 2020; Petrilli et al., 2020; Kim et al., 2021; Gupta et al., 2020; Cummings et al., 2020; Gottlieb et al., 2020, 2020). Though the SVI accounts for density of non-white individuals and adults aged 65+, we added weight to these specific demographic groups because older age has consistently been among the strongest predictors of poor SARS COV-2 outcomes, and racial/ethnic minority groups have experienced a higher burden of SARS COV-2 cases and mortality due to myriad factors related to structural racism (Bassett et al., 2020; Berkowitz et al., 2020; Chen and Krieger, 2021; Millett et al., 2020; Raifman and Raifman, 2020; Bailey and Moon, 2020; Bilal et al., 2021). Our inclusion of proportion minority population is intended to proxy neighborhood-level consequences of structural racism (e.g., racial residential segregation, neighborhood disinvestment, etc.), and is not intended to suggest that minority individuals are at higher risk for SARS COV-2 transmission or negative SARS COV-2 outcomes due to biological differences between minority and majority race individuals. These demographic factors have a substantial effect on the final CLRI score, which is consistent with research to date on the importance of these factors in SARS COV-2 transmission and severity. Data are from U.S. Census ACS 2014–2018 5-year Estimates (U.S. Census Bureau, 2020).

Weighting

We weighted the social vulnerability group at 30% of the overall index score in order to distribute additional weight across the demographic and health conditions component groups, for which we found robust evidence of association with poor SARS COV-2 outcomes (see Table 1 for all component weights). SVI components were weighted equally within the social vulnerability group, consistent with CDC's SVI calculation methods (Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research Analysis and Services Program, 2020, Flanagan et al., 2011). The weighting scheme was developed based on theory drawn from scientific research, before outcome data were available. Similar data projects have found success with a priori weighting schemes (Catlin et al., 2010).

The remaining 70% of the index score weight was distributed based on a population-normalized average effect size. Specifically, we reviewed “high-quality” evidence on SARS COV-2 infection and severity (defined as research with sample sizes >100, which surveyed a population-based sample, and controlled for common confounders in regression models that estimated effect sizes for specific health conditions). We then averaged effect sizes from the included evidence for each component, and multiplied the component's average effect size by its U.S. population prevalence.

Index calculation

The Dashboard calculated the CLRI separately at the city- and census tract-levels. We followed the analytic strategy utilized by CDC's SVI (Flanagan et al., 2011, Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research Analysis and Services Program, 2020). We assigned percentiles relative to other Dashboard cities or census tracts for each component (Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research Analysis and Services Program, 2020), multiplied each component's percentile by its weight, summed the weighted percentiles of all components, and reported this sum in deciles as the CLRI (calculated using SAS v9.4) (Sas Institute Inc, 2015).

For validation purposes, and since SARS COV-2 data are rarely released at the census tract level, we aggregated SARS COV-2 outcomes to the zip code tabulation area (ZCTA) level using the U.S. Census Bureau's ZCTA to Census Tract Relationship File (U.S. Census Bureau, 2010). Tract CLRI values were weighted by the proportion of the ZCTA population that resided in both the census tract and ZCTA in question, then summed.

CLRI values were computed only for ZCTAs with ≥70% population overlap with Dashboard census tracts. CLRI values were not calculated for cities with fewer than 10 ZCTAs. We conducted a sensitivity analysis using a 90% threshold for ZCTA to census tract population overlap.

Local SARS COV-2 data

To obtain sub-city SARS COV-2 data, the Dashboard partnered with the Drexel Urban Health Collaborative (UHC). The UHC systematically accessed and collected geographically granular ZCTA-level SARS COV-2 data on count of tests conducted, count of positive tests, confirmed cases, incidence, hospitalizations, and mortality cumulatively from onset through the dates specified in Table 2 . Since most cities reported only cumulative counts at the ZCTA-level, we opted to not conduct longitudinal analyses with data on trends. These data were obtained by identifying repositories of data from U.S. cities, including the 30 cities that are members of Big Cities Health Collation for which UHC is funded to provide SARS COV-2 data (Bilal et al., 2022b). Data from public dashboards were either accessed directly or downloaded, or, in select cases, copied into a spreadsheet. In cases where data were not publicly available, UHC requested data directly from health departments, entering into data sharing agreements as needed. All data were checked for consistency and outliers. In total, we obtained data for 47 of the 766 Dashboard cities. The dates of access, sources for each city, and outcomes available are displayed in Table 2.

Table 2

Cities included in the COVID local risk index validation sample.

City	N	Outcomes	Date of access	Source
Akron, OH	14	Incidence	5/21/21	Ohio Department of Health
Baltimore, MD	19	Incidence	5/12/21	Maryland Department of Health
Baton Rouge, LA	12	Testing, Incidence	5/19/21	Louisiana Department of Health
Boston, MA	29	Testing, Incidence	4/15/21	Boston Public Health Commission
Charlotte, NC	23	Incidence, Mortality	5/04/21	Mecklenburg County Health Department
Chicago, IL	55	Testing, Incidence, Mortality	5/18/21	City of Chicago Department of Public Health
Cincinnati, OH	20	Incidence	5/21/21	Ohio Department of Health
Cleveland, OH	14	Incidence	5/18/21	Cleveland Department of Public Health
Columbus, OH	29	Incidence	5/18/21	City of Columbus Department of Public Health
Dallas, TX	45	Incidence	4/22/21	Dallas County Health and Human Services
Dayton, OH	11	Incidence	5/21/21	Ohio Department of Health
Detroit, MI	25	Incidence, Mortality	5/10/21	Detroit Health Department
Fort Wayne, IN	15	Incidence	5/17/21	Indiana Department of Health
Greensboro, NC	10	Incidence, Mortality	5/21/21	North Carolina Department of Health and Human Services
Houston, TX	90	Incidence	5/21/21	Texas Department of State Health Services
Indianapolis, IN	31	Incidence	5/17/21	Indiana Department of Health
Jacksonville, FL	29	Incidence	5/21/21	Florida Department of Health
Kansas City, MO	43	Incidence	5/15/21	City of Kansas City Missouri Health Department
Las Vegas, NV	16	Testing, Incidence, Hospitalization, Mortality	5/04/21	Southern Nevada Health District
Long Beach, CA	11	Incidence	5/11/21	Long Beach Health and Human Services Department
Madison, WI	13	Testing, Incidence, Hospitalization, Mortality	5/21/21	Wisconsin Department of Health Services
Mesa, AZ	13	Incidence	5/21/21	Arizona Department of Health Services
Miami, FL	14	Incidence	5/21/21	Florida Department of Health
Milwaukee, WI	20	Testing, Incidence, Hospitalization, Mortality	5/21/21	Wisconsin Department of Health Services
Minneapolis, MN	16	Incidence	5/21/21	Minnesota Department of Health
New Orleans, LA	17	Testing, Incidence	5/19/21	Louisiana Department of Health
New York, NY	177	Testing, Positivity, Incidence, Hospitalization, Mortality	5/18/21	New York City Department of Health and Mental Hygiene
Norfolk, VA	14	Testing, Incidence	5/21/21	Virginia Department of Health
Oakland, CA	14	Incidence	5/21/21	Alameda County Public Health
Oklahoma City, OK	42	Incidence, Mortality	5/21/21	Oklahoma Department of Health
Orlando, FL	12	Incidence	5/21/21	Florida Department of Health
Peoria, IL	10	Testing, Incidence	5/21/21	Illinois Department of Public Health
Philadelphia, PA	46	Testing, Incidence, Hospitalization, Mortality	5/17/21	City of Philadelphia Department of Public Health
Phoenix, AZ	47	Incidence	5/21/21	Arizona Department of Health Services
Raleigh, NC	14	Incidence, Mortality	5/21/21	North Carolina Department of Health and Human Services
San Diego, CA	33	Incidence	5/22/21	County of San Diego Health and Human Services Agency
San Francisco, CA	27	Incidence, Mortality	5/21/21	San Francisco Department of Public Health
San Jose, CA	29	Incidence	5/17/21	County of Santa Clara Public Health Department
Seattle, WA	25	Testing, Incidence, Hospitalization, Mortality	5/21/21	King County Department of Public Health

Table 2. Cities Included in the COVID Local Risk Index Validation Sample (continued)

We identified four ZCTA-level SARS COV-2 outcomes to validate the CLRI, as they represent measures of either risk of infection or illness severity: positivity (number of people that tested positive for SARS COV-2/number tested for SARS COV-2), incidence (confirmed SARS COV-2 cases/total population), hospitalization (SARS COV-2 hospitalizations/total population), and mortality (SARS COV-2 deaths/total population). Population denominators were obtained from 2015 to 2019 ACS 5-year estimates.

We also compared select metrics for cities in the validation sample to the same metrics for cities on the Dashboard but not in the validation sample in order to gauge similarity between the two groups. The demographic factors compared included racial/ethnic diversity, percent children living in poverty, percent of the population experiencing excessive housing cost burden, and CLRI rank. Metrics were analyzed as presented on the Dashboard and are defined in the Dashboard's technical documentation (Gofine et al., 2021).

Validation methods

This analysis assessed the CLRI's construct/convergent validity in measuring risk of SARS COV-2 infection and illness severity. We tested the hypothesis that the CLRI was positively associated with SARS COV-2 infection (positivity and incidence) or severity (hospitalizations and mortality); each outcome was tested independently. First, we graphically depicted correlations using scatter plots. Second, we fitted a negative binomial model with the count of positive tests, cases, hospitalizations and mortality as the outcome, the number of tests (for positivity) or population counts (for the other outcomes) as the offset, and the CLRI as the only predictor. We chose a negative binomial model as the four outcomes were found to be overdispersed for a Poisson model. We extracted the rate ratios (RR) and 95% confidence intervals for the CLRI coefficient. Before introducing CLRI into this model, we standardized ZCTA-level CLRI values by centering by the city ZCTA CLRI mean and scaling by city standard deviation, both for each city separately. This standardization was conducted to make RRs comparable across cities. We also compared RRs to RRs produced using the same methods for another publicly available SARS COV-2 vulnerability index, the Surgo Ventures' COVID Community Vulnerability Index (CCVI) (Surgo Ventures, 2020a). The CCVI does not provide city-level values.

We also produced an overall pooled estimate of the association between the CLRI and the CCVI and the four outcomes using a mixed-effects negative binomial model of ZCTAs nested in cities, with a fixed and random coefficient for the CLRI or the CCVI. The exponentiated fixed coefficient of this model represents associations for the median city. To test whether there was heterogeneity in the association between the CLRI (or CCVI) and the outcomes, we compared this model with a model without a random slope for the CLRI (or CCVI) using a log likelihood ratio test. We fitted this model using a Laplace approximation with the glmmTMB package in R 4.1. To test whether the number of quadrature points of the generalized mixed model influenced our results, we tested increasing the number of quadrature points using StataMP v17 (see Supplemental Fig. 4 comparing results using the Laplace approximation vs. an increased number of quadrature points).

Results

Table 3 displays a list of RRs and 95% confidence intervals per city. Scatterplots displaying the association between the CLRI and the four analyzed SARS COV-2 outcomes are available upon request. In the pooled analysis, we found that higher CLRI was associated with higher positivity (RR = 1.16, 95% CI 1.09 to 1.24 per 1-SD increase, p < 0.01), incidence (RR = 1.09, 95% CI 1.04 to 1.14, p < 0.001), hospitalization (RR = 1.43, 95% CI 1.31 to 1.56, p < 0.001), and mortality (RR = 1.22, 95% CI 1.15 to 1.30, p < 0.001). This means that, in the pooled analysis, a 1-SD higher ZCTA-level CLRI score was associated with 16% higher risk of positivity, 9% higher incidence, 43% higher risk of hospitalization, and 22% higher risk of mortality. These pooled numbers varied by city (see Appendix Table 5 for comparisons of model with and without random slopes, and Supplemental Fig. 2 for a comparison between coefficients from the mixed effects model and from the stratified model), and do not account for differences due to compositional differences between cities, differences in the course of the outbreak across cities, and differences in when data were accessed.

Table 3

City-level associations (RRs) between ZCTA-Level CLRI scores, CCVI scores and four SARS COV-2 outcomes.

City	State	Index	Positivity	Incidence	Hospitalization	Mortality
Pooled∼		CLRI	1.12 (1.03; 1.22)	1.09 (1.04; 1.14)	1.48 (1.33; 1.64)	1.26 (1.17; 1.37)
		CCVI	1.15 (1.07; 1.24)	1.13 (1.07; 1.19)	1.41 (1.24; 1.60)	1.17 (1.07; 1.29)
Mesa	Arizona	CLRI		0.94 (0.82; 1.07)
		CCVI		1.04 (0.90; 1.20)
Phoenix	Arizona	CLRI		1.15 (1.09; 1.22)***
		CCVI		1.16 (1.10; 1.22)***
Tucson	Arizona	CLRI		1.16 (1.02; 1.32)*
		CCVI		1.21 (1.07; 1.36)**
Long Beach	California	CLRI		1.38 (1.28; 1.49)***
		CCVI		1.39 (1.31; 1.48)***
Oakland	California	CLRI		1.85 (1.49; 2.28)***
		CCVI		1.80 (1.44; 2.26)***
San Diego	California	CLRI		0.81 (0.60; 1.10)
		CCVI		0.83 (0.61; 1.12)
San Francisco	California	CLRI		1.24 (1.07; 1.44)**		1.30 (1.04; 1.63)*
		CCVI		1.37 (1.21; 1.55)***		1.35 (1.08; 1.68)**
San Jose	California	CLRI		1.39 (1.23; 1.57)***
		CCVI		1.54 (1.41; 1.69)***
Jacksonville	Florida	CLRI		0.98 (0.84; 1.13)
		CCVI		0.98 (0.84; 1.15)
Miami	Florida	CLRI		1.06 (0.87; 1.28)
		CCVI		1.09 (0.89; 1.32)
Orlando	Florida	CLRI		0.96 (0.85; 1.09)
		CCVI		0.96 (0.85; 1.08)
St. Petersburg	Florida	CLRI		1.07 (0.94; 1.21)
		CCVI		1.08 (0.96; 1.23)
Tampa	Florida	CLRI		1.15 (1.00; 1.32)
		CCVI		1.10 (0.95; 1.28)
Chicago	Illinois	CLRI	1.26 (1.14; 1.39)***	1.04 (0.97; 1.10)		1.44 (1.28; 1.62)***
		CCVI	1.20 (1.08; 1.32)***	1.04 (0.98; 1.11)		1.36 (1.21; 1.52)***
Peoria	Illinois	CLRI	0.68 (0.52; 0.89)**	1.05 (0.95; 1.16)
		CCVI	0.70 (0.50; 0.97)*	1.05 (0.95; 1.16)
Fort Wayne	Indiana	CLRI		0.94 (0.90; 0.98)**
		CCVI		0.94 (0.90; 0.98)**
Indianapolis	Indiana	CLRI		0.90 (0.86; 0.94)***
		CCVI		0.93 (0.89; 0.98)**
Baton Rouge	Louisiana	CLRI	1.15 (1.01; 1.30)*	1.69 (0.84; 3.38)
		CCVI	1.06 (0.93; 1.22)	3.22 (1.28; 8.10)*
New Orleans	Louisiana	CLRI	1.11 (0.95; 1.30)	1.05 (0.76; 1.46)
		CCVI	1.07 (0.91; 1.25)	1.13 (0.85; 1.51)
Shreveport	Louisiana	CLRI	0.94 (0.76; 1.16)	1.01 (0.78; 1.31)
		CCVI	0.98 (0.79; 1.21)	1.01 (0.77; 1.33)
Baltimore	Maryland	CLRI		1.11 (0.95; 1.29)
		CCVI		1.15 (0.99; 1.33)
Boston	Massachusetts	CLRI	1.35 (1.20; 1.53)***	1.24 (1.13; 1.35)***
		CCVI	1.28 (1.13; 1.45)***	1.23 (1.13; 1.34)***
Detroit	Michigan	CLRI		1.04 (0.99; 1.10)		1.16 (1.00; 1.33)*
		CCVI		1.09 (1.05; 1.14)***		0.87 (0.76; 1.00)*
Minneapolis	Minnesota	CLRI		1.10 (0.98; 1.23)
		CCVI		1.20 (1.10; 1.30)***

Table 3. City-Level Associations (RRs) between ZCTA-Level CLRI Scores, CCVI Scores and Four SARS COV-2 Outcomes (cont)COV-2 Outcomes

∼Pooled coefficient comes from a multilevel Poisson model of ZCTAs nested in cities, with a fixed and random slope for the CLRI or the CCVI. Coefficient shown is the fixed portion of the CLRI or CCVI effect, representing the effect in the median city. ******.

Table 5

Comparison of mixed effects Poisson models with and without random slopes for the CLRI or the CCVI.

Exposure	Outcome	Log Likelihood No Random Slope	Log Likelihood Random Slope	2 x logLik difference	Degrees of Freedom	P-value
CLRI	Positivity	−138340.7	−134704.5	7272.4	2	<0.001
CLRI	Incidence	−130275.0	−109469.8	41610.4	2	<0.001
CLRI	Hospitalization	−1713.3	−1661.9	102.8	2	<0.001
CLRI	Mortality	−3920.0	−3846.9	146.2	2	<0.001
CCVI	Positivity	−153917.6	−151885.8	4063.5	2	<0.001
CCVI	Incidence	−123422.7	−103968.5	38908.4	2	<0.001
CCVI	Hospitalization	−2109.9	−2039.1	141.6	2	<0.001
CCVI	Mortality	−4321.1	−4112.3	417.6	2	<0.001

Note: Model without a random slope includes a fixed effect for the CLRI or CCVI and a random intercept for city; model with random slope further includes a random slope for the CLRI or CCVI. P-value tests the null hypothesis that there is no heterogeneity in the association between CLRI or CCVI with the outcome by city.

We found that the CLRI was a good predictor of city-level positivity, with positive associations in 11 of 14 cities with positivity data (p < 0.05 in 8/11 cities). RRs in these 11 cities ranged from 1.04 (95% CI 0.98 to 1.11, p > 0.05) in Virginia Beach (VA) to 1.35 (95% CI 1.20 to 1.53, p < 0.01) in Boston (MA). For example, in Chicago each city-specific SD increase in ZCTA CLRI was associated with a 26% higher positivity rate (RR = 1.26, 95% CI 1.14 to 1.39, p < 0.01). Results regarding incidence were mixed; in 33 cities the association between CLRI and incidence was positive, ranging from 1.01 (95% CI 0.78 to 1.31) in Shreveport (LA) to 1.85 (95% CI 1.49 to 2.28) in Oakland (CA) (p < 0.05 in 17/33 cities). We found no association in 1 city, and a negative association in 13 cities, ranging from 0.81 (95% CI 0.60 to 1.10) in San Diego (CA) to 0.98 (95% CI 0.84 to 1.13) in Jacksonville (FL) (p < 0.05 in 6/13 cities).

The CLRI performed more consistently with regards to indicators of SARS COV-2 illness severity. In all 6 cities with hospitalization data (p < 0.05 in all) and in 13 of 14 cities with mortality data (p < 0.05 in 9/14 cities) the CLRI was associated with hospitalization or mortality. Specifically, the association between CLRI and hospitalization ranged from 1.29 (95% CI 1.08 to 1.55, p < 0.01) in Madison (WI) to 2.07 (95% CI 1.69 to 2.52, p < 0.001) in Seattle (WA). In the 13 cities with a positive association between CLRI and mortality, the RR varied from 1.05 (95% CI 0.92 to 1.21, p > 0.05) in Tulsa (OK) to 1.87 (95% CI from 1.37 to 2.54, p < 0.001) in Seattle (WA). A sensitivity analysis using a more stringent threshold to aggregate from census tracts to ZCTAs produced similar results to the main analysis (Supplemental Fig. 3).

We explored degree of similarity between this validation sample (n = 47) and other cities (n = 713) on the Dashboard. On average, cities included in the validation sample had higher total population count and higher average CLRI scores than did Dashboard cities that were not included. Validation cities were also more diverse and had higher percent of children living in poverty than other Dashboard cities. There were not statistically significant differences in housing cost (Table 4). Supplemental Fig. 1 depicts a histogram of CLRI values for cities in the validation sample; a histogram for the full sample is not shown as cities were, by design, equally distributed across CLRI values.

Table 4

Summary statistics for COVID local risk index and select metrics for cities included in and excluded from validation sample.

Metric	Summary of Values for Cities Included in Validation		Summary of Values for CitiesExcluded from Validation
	City Mean (Std Dev)	IQR	City Mean (Std Dev)	IQR
Children in Poverty (%)a	27.1 (9.3)b	22.9–31.5	18.8 (10.6)	10.4–25.3
COVID Local Risk Index	6.7 (2.6)b	4.5–9.0	5.4 (2.9)	3.0–8.0
Excessive Housing Cost (% of pop) a	36.4 (5.1)	33.4–38.6	35.0 (7.6)	29.5–40.1
Racial/Ethnic Diversity a	72.6 (9.7)b	64.9–81.0	61.8 (15.1)	52.1–73.0
Total Population a	818,628 (1,253,550)b	296,348–860,936	121,469 (184,637)	62,835–121,788

Children in Poverty, Excessive Housing Cost, Racial/Ethnic Diversity and Total Population were calculated using US Census American Community Survey 2018 5-year estimates.

Two sample t-tests comparing means for these metrics were significant at the p < 0.05 level.

Finally, the CLRI and Surgo's CCVI performed similarly overall with respect to association with SARS COV-2 outcomes. Overall, the CCVI was similarly associated with positivity and slightly more strongly associated with incidence (RR of 1.13 vs. 1.09 for the CLRI); CCVI RRs for incidence were larger than CLRI RRs in 29/47 cities, RRs were identical in 7 cities, and CLRI RRs were larger in 13 cities. RR differences were typically less than 0.1. CLRI was more frequently and strongly positively associated with hospitalizations and mortality (Table 3) (A table of city-level R2 values is available upon request).

Discussion

We validated the CLRI as an accurate tool to capture small area-level SARS COV-2-related risk in 47 U.S. cities, demonstrated most strongly by our pooled model results. In the majority of included cities, we found the CLRI to be strongly associated with positivity and SARS COV-2-related hospitalizations and mortality. These results underscore the extent to which social determinants of health, demographic factors, and the population prevalence of specific health conditions affect neighborhood-level risk for SARS COV-2. Given large total populations in U.S. cities, even a small increase in ZCTA-level positivity, hospitalization, and mortality (i.e. statistically significant RRs >1.00) can have important implications for policy makers and public health practitioners.

Associations between the CLRI and SARS COV-2 incidence were weaker and more heterogeneous across cities. In 13 cities CLRI was negatively associated with SARS COV-2 incidence (p < 0.05 in 6 of 13). In 4 of these 13 cities positive associations were found with either positivity, hospitalization, or mortality. These weaker and more heterogeneous associations may be due to imperfect reporting of SARS COV-2 cases, especially at the beginning of the pandemic, differences in the course of the outbreak across cities, unequal access to testing (Rader et al., 2020), or other factors that cause incidence data to be less reliable than positivity, hospitalization, and mortality data (Wu et al., 2020). This indicates that either measurement error related to SARS COV-2 incidence is higher than measurement error related to the other SARS COV-2 outcomes analyzed here, or that the CLRI is a better predictor of SARS COV-2 illness severity than of incidence. If the latter, this may be because caused by the components included in the CLRI and how they were weighted. Old age and comorbidities are weighted heavily in the CLRI, while, for example, potential for occupational exposures and area proportion of essential workers are not included. Inclusion of these variables (for which we could not find sufficiently granular data) or a different weighting approach may have produced different results. The heterogeneity in the incidence results could also be related to omitted variable bias. The CLRI does not measure how individuals interact with their social and built environments, which is a source of potential variation in SARS COV-2 exposure. The CLRI may produce more robust incidence results were such variables included.

There was substantial variation in strength of association between the CLRI and SARS COV-2 outcomes across cities. This could be driven by a number of factors. Some of these factors are related to the course of SARS COV-2 in a given city, including when and how the disease was introduced, and the course of the outbreak; cities that were exposed early in the pandemic or had larger outbreaks will likely have more cases, and so will produce larger effect size estimates. Differences may also be caused by differential access to testing, reporting of test results, or data tampering that may artificially lower counts of positive tests, hospitalizations, or mortality. This could also be caused by differences in how city residents interact with their social and built environments, as mentioned above.

Surgo's CCVI, also developed to assess neighborhood-level SARS COV-2 risks, produced a slightly stronger association with SARS COV-2 incidence than CLRI and was positively associated with incidence in more cities. In contrast, the CLRI was more strongly associated with positivity, hospitalization, and mortality. This could be for several reasons. First, Surgo's CCVI includes variables related to risk of infection that are not included in the CLRI, for example percent of population working in high infection risk settings, and long-term care residents per 100,000 population (Surgo Ventures, 2020b). Second, CCVI includes more metrics, which may reduce the impact of health conditions included in the CCVI on the final index score (Surgo Ventures, 2020b), in turn lowering strength of association with severity outcomes. Though both indices are associated with SARS COV-2 outcomes, the CLRI may be preferable for two reasons. First, CLRI is better correlated with a range of SARS COV-2 outcomes, including severe SARS COV-2 outcomes, which put more stress on the health system and have more dire consequences than do mild cases. Second, the CCVI includes numerous county- and state-level variables, reducing neighborhood-level specificity; the CLRI exclusively incorporates neighborhood-level variables.

The CLRI can be used by policy makers in several ways. Recent research has described associations between county-level SVI and vaccination coverage (Barry et al., 2021); the CLRI can help policy makers identify at-risk places at a more geographically granular level. For example, Waco, TX has used the CLRI to interpret neighborhood-level distributions of confirmed SARS COV-2 cases (Dashboard, 2020b), while Manchester, NH has used the CLRI to identify high SARS COV-2 risk neighborhoods (Dashboard, 2020a). Going forward, testing and vaccination sites could be located in neighborhoods that have high CLRI scores, and vaccine outreach and education initiatives could focus on these neighborhoods. Furthermore, given that vaccines are now widely available in the U.S., neighborhood-level disparities in vaccine access and uptake may emerge. As disparities are identified, resources should be preferentially guided to neighborhoods in which there is slow vaccine uptake that also rank highly on the CLRI. Similarly, in the event of another surge of SARS COV-2 cases, state officials could allocate more resources to cities with higher CLRI scores. Finally, in cities with neighborhood-level SARS COV-2 surveillance data, the CLRI can provide an additional point of reference, helping policy makers to bolster the case for additional state and federal resources, or to identify neighborhoods that are performing better or worse than the CLRI would suggest based on their surveillance data.

This validation analysis could have been produced sooner if granular SARS COV-2 infection and illness severity data were more readily available. Furthermore, though the results of this analysis are encouraging, they are based on data from just over 6% of Dashboard cities because geographically granular data are not available for more cities, or because smaller cities do not have enough ZCTAs for validation. Whenever possible, granular SARS COV-2 data should swiftly be made publicly available.

Strengths and limitations

Though the CLRI is calculated at the neighborhood level, the present validation analysis was conducted at the ZCTA level. ZCTAs are larger than census tracts and may capture different populations. We took steps to reduce bias introduced when aggregating index values from tract to ZCTA by only including ZCTAs that had high population overlap with census tracts on the Dashboard (>70%). Sensitivity analysis results using a more stringent population overlap threshold (>90%) were similar to results from the main analysis. Data from cities were accessed at different times, and the course of the pandemic varies across cities, so validation results for different cities may not be easy to compare. Incomplete and differentially reported outcome data may bias the results described here. However, it is likely that areas with higher SVI and CLRI scores experience more substantial underreporting. As such, this bias may dilute the association between CLRI and SARS COV-2 outcomes. Some of the variables included in the CLRI intended to capture the consequences of structural racism – namely proportion of minority individuals. While these measures may not capture structural racism as directly as other measures, for example the Index of Dissimilarity, which measures residential segregation in larger areas (e.g. cities), or the Index of Concentration at the Extremes, which measures residential segregation in smaller areas (e.g., census tracts) (White, 1986), we used proportion of minority individuals to maintain consistency with published literature while calculating the index. At the point when we constructed the index we did not identify sufficient literature examining associations between other measures of structural racism and SARS COV-2 transmission and disease outcomes. Future CLRI updates may contain these variables. Similarly, though the CLRI, via the SVI, includes three variables related to the housing built environment (housing in structures with 10+ units, mobile homes, and household overcrowding), recent research suggests public built environments also affect SARS COV-2 severity (Wali and Frank, 2021). Variables related to the public built environment, and other recently emerged SARS COV-2 risk factors, may be added to future versions of the CLRI. Finally, cities included in the validation sample were not representative of all Dashboard cities. Validation cities were larger, more diverse, had higher percent children living in poverty, and higher average CLRI ranks than did Dashboard cities not included in the validation sample. Though this limits the generalizability of these results, it also suggests the present analysis validates the CLRI for use in cities with substantial at-risk populations, where SARS COV-2 resources may be most needed.

In terms of strengths, the CLRI exclusively incorporates census tract-level data, and has been updated as our understanding SARS COV-2 has improved. The CLRI is publicly available from the Dashboard through downloadable data and neighborhood-level maps (http://www.cityhealthdashboard.com).

Public health implications

The CLRI can help guide city- and neighborhood-level SARS COV-2 resource allocation and interventions. In the absence of needed validation data, public health researchers can build potentially valid data tools while seeking validation as soon as possible.

Declaration of competing interest

The authors have no disclosures to report.