A commercially available crime index may be a reliable alternative to actual census-tract crime in an urban area.

Health research on the effects of crime has been hampered by a lack of small-scale, reliable crime data. Our objective is to assess the accuracy of a set of commercially available crime indices for use in health research. The 2016 release of Applied Geographic Solutions' (AGS) crime indices are based on data from 2010 to 2014 and provide tract-level information on crime. We use crime rates for 1069 tracts of the Los Angeles Police Department (LAPD) jurisdiction for the same years to assess (1) Spearman Correlations of major crime categories, and (2) accuracy of AGS indices in predicting falling above/below the median and into the highest/lowest quartile of LAPD crime. We also test if adding variables from the American Community Survey (ACS) to regression analyses can help to reduce measurement bias. We found that five of ten AGS indices correlated moderately well with LAPD crime. In unadjusted regressions, robbery, homicide, aggravated assault, motor-vehicle theft and personal crime achieved c-statistics from 0.81 to 0.90. C-statistics improved up to 0.13 points after adding ACS variables. Some AGS crime indices may be reliable proxies for crime in an urban area. The AGS index for total crime, most commonly used in prior research, performed poorly.

Introduction

Neighborhood level crime has been associated with various health behaviors and outcomes including, physical activity, smoking, drug-use, body weight, and birth outcomes (Stockdale et al., 2007, Messer et al., 2006, Foster and Giles-Corti, 2008, Sandy et al., 2013, Masho, 2019, Tseng et al., 2001, Norris, 2019). Hypothesized mechanisms include increased stress, reduction in the opportunity to be physically active and to socialize in one’s own neighborhood (i.e., develop community cohesion) (Masho, 2019, Anderson et al., 2015, Uniform Crime Reporting, 2018). Studies examining the effect of crime on health, or studies that want to control for the confounding effect of neighborhood crime on health, have been hampered by the lack of geo-coded crime data at geographic scales below cities and counties. Few police departments provide reliable, geo-coded information to the public, and the Federal Bureau of Investigation’s (FBI) Uniform Crime Report (UCR) only publicly provides information for counties and cities with a population of at least 10,000 (Uniform Crime Reporting Statistics, 2018, Harries, 2006). Studies using local police data show that crime varies substantially at small geographic scales (Esri Data & Reports, 2018); cities and counties are too large to provide a meaningful spatial unit to study neighborhood crime. As a result, most studies on crime are limited to the few areas in the US in which police departments provide local crime statistics. Thus, there is a dearth of informative, large-scale studies and studies on the effects of neighborhood crime on health across diverse enviroments.

The Esri Business Analyst, a for-purchase geographic data base and analytic tool (Applied Geographic Solutions, 2016), provides a set of crime indices generated by Applied Geographic Solutions (AGS). The AGS CrimeRisk© indices use information from the UCR and small-scale data from local police departments of two cities, Baltimore and Chicago (Applied Geographic Solutions, 2018). The AGS crime indices mirror the UCR major crimes-categories and include indices for homicide, rape, robbery, assault, burglary, larceny and motor vehicle theft. Only the UCR category of arson has been omitted from the AGS crime index-set. Like the UCR, AGS also includes three broad categories for total, personal and property crime (Applied Geographic Solutions, 2018). We are provided with the AGS 2016 release of the CrimeRisk© indices. AGS generated the 2016 indices by using data for 2010–2014 (Applied Geographic Solutions, 2018). Since UCR data are only available at the city and county-level, AGS uses predictive models to estimate crime rates at smaller geographic scales, including the tract, block-group and block level. Indices are scaled to the national average of 2012. If a census unit has the same level of crime as the national average the index is 100. The models generating the AGS crime indices are proprietary, and only a succinct technical description is available online (Applied Geographic Solutions, 2018). The AGS crime index is marketed for use by banks, insurance companies and realtors (Myers et al., 2016). Academic research is not listed as a target audience, and while some health studies have used the AGS crime indices (Tang et al., 2014, Ohri-Vachaspati et al., 2017, Lowe et al., 2016, Ojikutu et al., 2018, Broyles et al., 2016, Lowe et al., 2015, LAData, 2010), none has validated the AGS measures.

Here, we compare a commercially available neighborhood crime index to actual crime rates to assess its validity for health research. For this purpose, we use actual crime rates of an urban area: information from the City of Los Angeles Police Department (LAPD). The UCR provides aggregate data for the entire LAPD-jurisdiction. The LAPD, however also provides comprehensive point data on crimes that, according to the AGS technical documentation, has not been used for the AGS indices (Applied Geographic Solutions, 2018). We assessed how well the AGS crime indices predicted actual LAPD crime rates at the census tract-level. Since the AGS indices are modeled for the entire United States and derived from prediction models using data from the American Community Survey (ACS) it is likely that the AGS model fits better for some census tracts than for others depending on how similar the local relationship of ACS predictors and crime is compared to the national average. Thus, we also assessed if adding information from the ACS to adjust for Los Angeles neighborhood characteristics may affect the accuracy of the index.

Methods

Data

LAPD crime data for 2010–2014, was accessed through the Los Angeles Open Data webpage (Morenoff and Sampson, 1997). The location of the crime was provided by x-y-coordinates of the address where the crime occurred rounded to the nearest hundred block. We confirmed with the lead of LA COMPSTAT, the LAPD statistical division, that data was, to his knowledge, complete (personal communication with the lead officer of the LAPD COMPSTAT division, 1/11/2017). LAPD crime was categorized into 10 different crime categories that reflect the AGS major crime categories; crimes not matching these categories, such as fraud, identity theft and simple assault, were not considered in this analysis. Next, crimes were geo-coded and aggregated at the tract-level. All crime locations could be uniquely attributed to a single tract. The cross-walk of LAPD crimes to AGS crime-categories is available in Appendix A1. Because the 2016 AGS index uses data from 2010 to 2014 but proposes to predict crime for a given year, we created two outcome variables: one that categorized crime for 2016 only, and a 5-year aggregate of years 2010–2014 that match the years of data used by AGS. We determined population per census tract based on the 2016 US Census Bureau data and calculated crime rates per 1000 population.

Table A

Crime category designations, LAPD crimes to ESRI crime categories.

LAPD Crime Code Description	Frequency	Percent of Total Crimes	ESRI Category
ASSAULT WITH DEADLY WEAPON ON POLICE OFFICER	370	0.09	AGGRAVATED ASSAULT
ASSAULT WITH DEADLY WEAPON, AGGRAVATED ASSAULT	20952	4.81	AGGRAVATED ASSAULT
CHILD ABUSE (PHYSICAL) – AGGRAVATED ASSAULT	235	0.05	AGGRAVATED ASSAULT
OTHER ASSAULT	843	0.19	AGGRAVATED ASSAULT
SPOUSAL (COHAB) ABUSE – AGGRAVATED ASSAULT	4662	1.07	AGGRAVATED ASSAULT
BURGLARY	29252	6.72	BURGLARY
BURGLARY FROM VEHICLE	31191	7.17	BURGLARY
BURGLARY FROM VEHICLE, ATTEMPTED	578	0.13	BURGLARY
BURGLARY, ATTEMPTED	2560	0.59	BURGLARY
CRIMINAL HOMICIDE	575	0.13	HOMICIDE
LYNCHING	9	0	HOMICIDE
MANSLAUGHTER, NEGLIGENT	2	0	HOMICIDE
RAPE, ATTEMPTED	245	0.06	RAPE
RAPE, FORCIBLE	2180	0.5	RAPE
BIKE – ATTEMPTED STOLEN	17	0	LARCENY
BIKE – STOLEN	4055	0.93	LARCENY
BOAT – STOLEN	43	0.01	LARCENY
GRAND THEFT/AUTO REPAIR	1	0	LARCENY
PETTY THEFT – AUTO REPAIR	5	0	LARCENY
PICKPOCKET	263	0.06	LARCENY
PICKPOCKET, ATTEMPT	9	0	LARCENY
PURSE SNATCHING	259	0.06	LARCENY
PURSE SNATCHING – ATTEMPT	13	0	LARCENY
SHOPLIFTING – ATTEMPT	66	0.02	LARCENY
SHOPLIFTING – PETTY THEFT ($950 & UNDER)	12525	2.88	LARCENY
SHOPLIFTING-GRAND THEFT ($950.01 & OVER)	990	0.23	LARCENY
THEFT FROM MOTOR VEHICLE – ATTEMPT	367	0.08	LARCENY
THEFT FROM MOTOR VEHICLE – GRAND ($400 AND OVER)	7055	1.62	LARCENY
THEFT FROM MOTOR VEHICLE – PETTY ($950 & UNDER)	20420	4.69	LARCENY
THEFT FROM PERSON – ATTEMPT	51	0.01	LARCENY
THEFT PLAIN – ATTEMPT	304	0.07	LARCENY
THEFT PLAIN – PETTY ($950 & UNDER)	30385	6.98	LARCENY
THEFT, COIN MACHINE – ATTEMPT	7	0	LARCENY
THEFT, COIN MACHINE – GRAND ($950.01 & OVER)	14	0	LARCENY
THEFT, COIN MACHINE – PETTY ($950 & UNDER)	84	0.02	LARCENY
THEFT, PERSON	2857	0.66	LARCENY
THEFT-GRAND ($950.01 & OVER) EXCPT, GUNS, FOWL, LIVESTK, PROD	13109	3.01	LARCENY
TILL TAP – GRAND THEFT ($950.01 & OVER)	6	0	LARCENY
TILL TAP – PETTY ($950 & UNDER)	24	0.01	LARCENY
VEHICLE – ATTEMPT STOLEN	781	0.18	MOTOR VEHICLE THEFT
VEHICLE – STOLEN	34314	7.88	MOTOR VEHICLE THEFT
ATTEMPTED ROBBERY	2356	0.54	ROBBERY
ROBBERY	16930	3.89	ROBBERY
UNKNOWN	224	0.06	EXCLUDE
ARSON	768	0.18	EXCLUDE
BATTERY – SIMPLE ASSAULT	35497	8.15	EXCLUDE
BATTERY ON A FIREFIGHTER	90	0.02	EXCLUDE
BATTERY POLICE (SIMPLE)	927	0.21	EXCLUDE
BATTERY WITH SEXUAL CONTACT	2124	0.49	EXCLUDE
BEASTIALITY, CRIME AGAINST NATURE SEXUAL ASSLT WITH ANIMALS	4	0	EXCLUDE
BIGAMY	1	0	EXCLUDE
BLOCKING DOOR INDUCTION CENTER	2	0	EXCLUDE
BOMB SCARE	193	0.04	EXCLUDE
BRANDISH WEAPON	3436	0.79	EXCLUDE
BRIBERY	3	0	EXCLUDE
BUNCO, ATTEMPT	205	0.05	EXCLUDE
BUNCO, GRAND THEFT	2031	0.47	EXCLUDE
BUNCO, PETTY THEFT	1199	0.28	EXCLUDE
CHILD ABANDONMENT	25	0.01	EXCLUDE
CHILD ABUSE (PHYSICAL) – SIMPLE ASSAULT	1704	0.39	EXCLUDE
CHILD ANNOYING (17YRS & UNDER)	897	0.21	EXCLUDE
CHILD NEGLECT (SEE 300 W.I.C.)	711	0.16	EXCLUDE
CHILD STEALING	233	0.05	EXCLUDE
CHILD, CRIME AGAINST	1113	0.26	EXCLUDE
CONSPIRACY	12	0	EXCLUDE
CONTEMPT OF COURT	1807	0.42	EXCLUDE
CONTRIBUTING	27	0.01	EXCLUDE
COUNTERFEIT	153	0.04	EXCLUDE
CREDIT CARDS, FRAUD USE ($950 & UNDER)	93	0.02	EXCLUDE
CREDIT CARDS, FRAUD USE ($950.01 & OVER)	217	0.05	EXCLUDE
CRIMINAL THREATS – NO WEAPON DISPLAYED	10505	2.41	EXCLUDE
CRUELTY TO ANIMALS	199	0.05	EXCLUDE
DEFRAUDING INNKEEPER/THEFT OF SERVICES, $400 & UNDER	470	0.11	EXCLUDE
DEFRAUDING INNKEEPER/THEFT OF SERVICES, OVER $400	45	0.01	EXCLUDE
DISCHARGE FIREARMS/SHOTS FIRED	820	0.19	EXCLUDE
DISHONEST EMPLOYEE – GRAND THEFT	62	0.01	EXCLUDE
DISHONEST EMPLOYEE – PETTY THEFT	43	0.01	EXCLUDE
DISHONEST EMPLOYEE ATTEMPTED THEFT	4	0	EXCLUDE
DISRUPT SCHOOL	7	0	EXCLUDE
DISTURBING THE PEACE	752	0.17	EXCLUDE
DOCUMENT FORGERY/STOLEN FELONY	4896	1.12	EXCLUDE
DOCUMENT WORTHLESS ($200 & UNDER)	24	0.01	EXCLUDE
DOCUMENT WORTHLESS ($200.01 & OVER)	114	0.03	EXCLUDE
DRIVING WITHOUT OWNER CONSENT (DWOC)	125	0.03	EXCLUDE
DRUGS, TO A MINOR	24	0.01	EXCLUDE
DRUNK ROLL	11	0	EXCLUDE
EMBEZZLEMENT, GRAND THEFT ($950.01 & OVER)	1562	0.36	EXCLUDE
EMBEZZLEMENT, PETTY THEFT ($950 & UNDER)	150	0.03	EXCLUDE
EXTORTION	640	0.15	EXCLUDE
FAILURE TO DISPERSE	2	0	EXCLUDE
FAILURE TO YIELD	126	0.03	EXCLUDE
FALSE IMPRISONMENT	174	0.04	EXCLUDE
FALSE POLICE REPORT	79	0.02	EXCLUDE
GRAND THEFT/INSURANCE FRAUD	9	0	EXCLUDE
ILLEGAL DUMPING	108	0.02	EXCLUDE
INCEST (SEXUAL ACTS BETWEEN BLOOD RELATIVES)	7	0	EXCLUDE
INCITING A RIOT	2	0	EXCLUDE
INDECENT EXPOSURE	752	0.17	EXCLUDE
KIDNAPPING	389	0.09	EXCLUDE
KIDNAPPING – GRAND ATTEMPT	135	0.03	EXCLUDE
LETTERS, LEWD	3473	0.8	EXCLUDE
LEWD CONDUCT	283	0.07	EXCLUDE
LYNCHING – ATTEMPTED	7	0	EXCLUDE
ORAL COPULATION	390	0.09	EXCLUDE
OTHER MISCELLANEOUS CRIME	3436	0.79	EXCLUDE
PANDERING	55	0.01	EXCLUDE
PEEPING TOM	256	0.06	EXCLUDE
PIMPING	120	0.03	EXCLUDE
PROWLER	166	0.04	EXCLUDE
RECKLESS DRIVING	71	0.02	EXCLUDE
REPLICA FIREARMS (SALE,DISPLAY,MANUFACTURE OR DISTRIBUTE)	12	0	EXCLUDE
RESISTING ARREST	665	0.15	EXCLUDE
SEX, UNLAWFUL	716	0.16	EXCLUDE
SEXUAL PENTRATION WITH A FOREIGN OBJECT	597	0.14	EXCLUDE
SHOTS FIRED AT INHABITED DWELLING	537	0.12	EXCLUDE
SHOTS FIRED AT MOVING VEHICLE, TRAIN OR AIRCRAFT	75	0.02	EXCLUDE
SODOMY/SEXUAL CONTACT B/W PENIS OF ONE PERSON TO ANUS OF OTHER	300	0.07	EXCLUDE
SPOUSAL (COHAB) ABUSE – SIMPLE ASSAULT	25077	5.76	EXCLUDE
STALKING	353	0.08	EXCLUDE
TELEPHONE PROPERTY – DAMAGE	8	0	EXCLUDE
THEFT OF IDENTITY	26587	6.11	EXCLUDE
THREATENING PHONE CALLS/LETTERS	582	0.13	EXCLUDE
THROWING OBJECT AT MOVING VEHICLE	327	0.08	EXCLUDE
TRESPASSING	4721	1.08	EXCLUDE
UNAUTHORIZED COMPUTER ACCESS	318	0.07	EXCLUDE
VANDALISM – FELONY ($400 & OVER, ALL CHURCH VANDALISMS)	24298	5.58	EXCLUDE
VANDALISM – MISDEAMEANOR ($399 OR UNDER)	17261	3.97	EXCLUDE
VIOLATION OF COURT ORDER	3393	0.78	EXCLUDE
VIOLATION OF RESTRAINING ORDER	3830	0.88	EXCLUDE
VIOLATION OF TEMPORARY RESTRAINING ORDER	443	0.1	EXCLUDE
WEAPONS POSSESSION/BOMBING	34	0.01	EXCLUDE

To assess if we could improve accuracy of the index, we used a set of ACS predictors that are commonly used in health research. These variables reflect socioeconomic neighborhood characteristics that might be correlated with crime. The ACS 5-year 2012–2016 estimates initially considered included: median household income, percent population with a high school degree, percent population with a bachelor’s degree, percent population living above 150% of the poverty threshold, percent population receiving public assistance, percent population owning a car and the percent of households headed by single mothers. We also added the percentage of non-white residents to identify highly segregated, historically disenfranchised neighborhoods (Massey, 2018, Kuhl et al., 2009, Publishing Ltd et al., 2007).

Statistical methods

To assess the success of each AGS index in predicting the respective LAPD crime rate, we first calculated spearman rank correlation coefficients of each crime index with actual LAPD crime rates. LAPD tract-level crime rates are highly skewed towards low rates with the mode category often being a rate of 0 crimes. We decided not to predict crime rates as count outcomes, because it would require the use of negative binomial or Poisson regression models which do not provide model fit measures that can be intuitively interpreted or compared across models of different indices. Instead, we conducted logistic regression and used the c-statistic, or Area Under the Receiver Operating Curve (AUC), as measure of prediction accuracy. While an imperfect measure of model accuracy, the c-statistic is the only measure that provides us with an intuitively interpretable metric that can be compared across models with different outcomes and predictors (Weinberg and Abramowitz, 2002). We explored the predictive accuracy of each AGS index when predicting (1) the highest versus the lowest quartile of actual crime rates for each crime category, and (2) whether a neighborhood fell above or below the neighborhood median of a particular LAPD crime rate. We then adjusted for the set of ACS predictors listed above to improve the predictive performance of the AGS crime index. After assessing co-linearity of neighborhood characteristics, we adjusted each model with tract-level median household income, percent population with high school degree, percent population non-white, percent population living below 150% of the poverty line and percent of female headed households with children. All analyses were performed using SAS 9.4.

Results

LAPD Crime rates for 1096 census tracts in the LAPD jurisdiction vary starkly across the seven specific crime categories with homicides and rapes being the rarest and burglaries and larcenies the most common crimes. AGS indices are standardized to the national average in 2012, and indicate that robbery, motor-vehicle thefts and homicides are more common in the LAPD jurisdiction compared to the national average of 2012 (Table 1). Fourteen tracts were deleted because they were not or little populated including recreational areas such as parks, sport facilities, stadiums and airports.

Table 1

Median and interquartile range (IQR) for crimes per 1000 population and ESRI indices in N = 1096 Los Angeles Police Department, crime reporting census tracts, aggregated across 2010–2014.

Crime	LAPD Rate Median (IQR)	ESRI Index Median (IQR)
Aggravated Assaults	1.35 (0.59, 2.53)	87.00 (48.00, 134.00)
Burglaries	6.73 (4.16, 9.78)	69.00 (53.00, 86.00)
Homicides	0 (0, 0.09)	122.00 (59.00, 226.50)
Larcenies	6.47 (4.10, 10.19)	70.50 (47.50, 101.00)
Motor Vehicle Thefts	3.14 (1.88, 4.60)	162.00 (108.50, 231.50)
Personal Crime	1.61 (0.74, 2.86)	117.00 (68.00, 175.00)
Property Crime	18.8 (12.43, 27.00)	80.00 (59.00, 104.00)
Rapes	0.17 (0.07, 0.29)	71.00 (33.00, 117.00)
Robberies	1.24 (0.49, 2.71)	180.00 (101.50, 291.50)
Total Crime	40.08 (28.39, 56.31)	87.00 (61.00, 112.00)

We focus on 5-year aggregates of LAPD crime rates per population, because spearman correlations were slightly higher for the 5-year aggregate of LAPD crime than the LAPD crime rates for 2016 (Appendix Table B). Spearman correlations between 5-year LAPD crime rates per population (referred to as “LAPD crime rates”) and the AGS indices are moderate to low. The strongest correlation is 0.6 for robberies. For rape and burglaries correlations are very low, with the latter even having an opposite sign (Table 2).

Table B

Spearman correlation for LAPD 2016 data and ESRI measure.

Crime	2016 Only
Robberies	0.52
Personal Crime	0.51
Homicides	0.28
Aggravated Assaults	0.47
Motor Vehicle Thefts	0.37
Larcenies	0.29
Total Crime	0.31
Property Crime	0.25
Rapes	0.01
Burglaries	-0.11

Table 2

Spearman correlation between corresponding ESRI crime index and 2010–2014 aggregate 5 year and 2016 only, LAPD jurisdiction, crimes per 1000 population.

Crime	Aggregate 5 Year (2010–2014)	2016 Only
Robberies	0.60	0.52
Personal Crime	0.53	0.51
Homicides	0.50	0.28
Aggravated Assaults	0.48	0.47
Motor Vehicle Thefts	0.39	0.37
Larcenies	0.29	0.29
Total Crime	0.27	0.31
Property Crime	0.23	0.25
Rapes	0.04	0.01
Burglaries	−0.04	−0.11

For the logistic regression analyses, we modelled crime indices that correlated at least moderately with LAPD crime rates (spearman correlation coefficients >0.3 (Fotheringham and Wong, 1991). Table 3 shows c-statistics for logistic regressions comparing the upper and lower quartile of LAPD crime and falling below and above the median of LAPD crime rates, for both unadjusted and adjusted models. The c-statistics of unadjusted models range from 0.77 to 0.90 for models contrasting the upper and lower quartile of crime. For models contrasting falling above and below the median of LAPD crime rates c-statistics range from 0.69 to 0.80. The c-statistics for the unadjusted models reflect the relative ordering of the spearman correlation coefficients in that robbery, personal crime and homicides lead the list followed by aggravated assault and motor-vehicle accidents. Adding neighborhood predictors (i.e., adjusted models), improved the accuracy for all crime indices, and, in particular, for aggravated assault and motor vehicle accidents, for which the c-statistic increased by up to 0.10 and 0.13, respectively for models contrasting quartiles and falling above and below the median.

Table 3

Adjusted and unadjusted AUC for ESRI crime indices and Aggregate 5-year (2010–2014) LAPD crimes per 1000 population.

Crime	Unadjusted AUC	Adjusted* AUC
Below vs Above Median
Robberies	0.8	0.83
Personal Crime	0.77	0.85
Homicides	0.75	0.77
Aggravated Assaults	0.75	0.84
Motor Vehicle Thefts	0.69	0.73
Upper vs Lower Quartile
Robberies	0.9	0.93
Personal Crime	0.85	0.93
Homicides	0.83	0.85
Aggravated Assaults	0.81	0.94
Motor Vehicle Thefts	0.77	0.87

Model adjusted for census tract median income, percent with high school degree, percent non-white race/ethnicity, percent living below 150% of the poverty line, and percent female head of household with children.

We also conducted sensitivity analysis with the 1-year LAPD crime rates for 2016 as outcome and found that overall c-statistics were slightly lower with unadjusted c-statistics varying from 0.74 to 0.86 for the models contrasting upper and lower quartiles and c-statistics ranging from 0.68 to 0.75 for models predicting falling above or below the median (Appendix Table C).

Table C

Adjusted and unadjusted AUC for ESRI crime indices and 2016 Only LAPD crimes per 1000 population.

Crime	Unadjusted AUC	Adjusted* AUC
Below vs Above Median
Robberies	0.75	0.78
Personal Crime	0.76	0.80
Homicides	0.70	0.72
Aggravated Assaults	0.74	0.80
Motor Vehicle Thefts	0.68	0.71
Upper vs Lower Quartile
Robberies	0.86	0.89
Personal Crime	0.83	0.90
Homicides	0.53	0.68
Aggravated Assaults	0.80	0.90
Motor Vehicle Thefts	0.74	0.83

Adjusted for income, education, race/ethnicity, female head of household, and poverty.

Discussion

The results from our study suggest that some of the commercially available AGS crime indices are promising alternatives for studies analyzing and controlling for the effects of crime on health. AGS indices for rape and burglaries, correlated poorly with actual crime rates; burglaries, one of the most common crime-categories, even correlated negatively with actual LAPD crime rates. AGS indices for robbery, homicides, aggravated assault, motor vehicle theft and the broad category of personal crime correlated moderately well with continuous LAPD crime rates. Five crime indices (AGS indices for robbery, homicides, aggravated assault, motor vehicle theft and the broad category of personal crime) also performed well in unadjusted logistic models predicting high and low tracts defined by the upper and lower quartile of LAPD crime. Robbery, personal crime, homicide and aggravated assault indices also performed moderately well to well when predicting whether or not a census tract would fall above or below the median tract-level crime rates. C-statistics were high for robbery and the broad personal crime category (0.90 and 0.85, respectively for unadjusted models contrasting quartiles). We also showed that adding a set of five neighborhood characteristics improved the accuracy of the AGS indices. Adding these variables to analyses may help reduce the risk of bias due to measurement error in the AGS indices, this may be particularly useful for studies who aim to control for the confounding effects of crime. It is notable that the AGS index for total crime correlated only weakly with LAPD crime rates, but has been the most commonly used in health research (Tang et al., 2014, Ohri-Vachaspati et al., 2017, Lowe et al., 2016, Ojikutu et al., 2018, Broyles et al., 2016, Lowe et al., 2015, LAData, 2010).

For the vast majority of neighborhoods in the United States, crime data at small geographic scales is not available. In contrast, commercially available AGS crime indices provide block, block-group and tract-level crime information for the entire US. Building a predictive model that estimates highly variable small-scale crime rates from a larger-scale city or county-crime rate is conceptually and methodologically challenging. The modifiable areal unit problem is a concept that has been extensively studied in geography and exemplifies why predicting crime at smaller scales from large-scale data is challenging (Openshaw, 1984). Research on the modifiable real unit has shown that averages and variance of an outcome vary starkly if aggregated at different geographic scales and according to different boundaries. This research has also demonstrated that these differences can notably affect the outcomes of statistical models (Openshaw, 1984). Since the scale and boundaries for census units such as block-groups and tracts have limited meaning for the spatial distribution of crime, it is to be expected that predicting crime rates for these spatial units is difficult when only large-scale crime data at the county level is available. The AGS technical description is vague regarding its methods and model fit. It only states that each crime was modeled separately and that models used up to 100 census variables to estimate local crime (Applied Geographic Solutions, 2018). AGS only reports model fit for the “jurisdiction level” which in our case would be the entire LAPD jurisdiction. AGS states most models explain around 85% of variance at the jurisdiction level. The jurisdiction level is however just one level below the county-level for which the actual FBI crime data is available. It therefore provides a much easier prediction task compared to tract level crime. AGS does not give any information on model performance for the tract-level.

While to our knowledge, no publicly available research has validated the AGS CrimeRisk© indices, several studies have used the AGS crime indices in their research (Tang et al., 2014, Ohri-Vachaspati et al., 2017, Lowe et al., 2016, Ojikutu et al., 2018, Broyles et al., 2016, Lowe et al., 2015, LAData, 2010). These studies find effects in the expected direction and lend evidence to concordance validity. While our study has limited generalizability, we provide first empirical evidence beyond concordance validity that can inform future studies looking for data on crime.

Our research has important limitations. Firstly, our study only uses data from one urban area. While accuracy of our highest performing models is high, prediction success of the AGS might be lower in other urban areas and it might be much different in rural areas. More validation studies comparing the AGS to actual crime data are necessary to assess if and how much predictive performance varies across cities and rural areas. We did not assess block-group level AGS indices, and while it is unlikely that categories that performed poorly at the tract level will perform well at the block-group level, we cannot say how accurate the better-performing indices are at the block-group or block-level. We tested the index as predictor of dichotomized crime. Indices with high c-statistics paired with moderate spearman correlations, may point to the AGS indices being better performing for extremely low and high crime areas than for areas with less extreme crime rates. More research needs to explore the accuracy of the AGS crime index for differentiating mid-level crime neighborhoods. Finally, most researchers are likely interested in assessing crime during a given year. The 2016 release of AGS crime indices, reports to use 5 years of UCR-data. The AGS documentation indicates that data was used as time-series. Predicting 5-year aggregates of LAPD crime data provided slightly better results than 1-year data. Analysis using LAPD crime data for 2016 only showed comparable, if slightly lower accuracy (Appendix Table C). We chose to discuss the 5-year results, to allow AGS indices to predict the same data that was used to create them.

In conclusion, our findings suggest that four out of ten AGS indices are indeed good proxies of actual crime in an urban area. The remaining six indices, including the index for total crime which has been used most commonly in prior publications, should be used with extreme care, since they correlated only slightly with actual crime in the LAPD jurisdiction. The AGS index for personal crime, might be a better option for a broad crime category, if researchers do not wish to use specific crime categories in their analysis. Furthermore, our results suggest that census variables can be added to future analyses to reduce potential bias from measurement error in the AGS indices. Crime data at small geographic scales is rare and the AGS indices provide tremendous potential for large-scale, even nation-wide studies on the effects of crime and health. While our study has limited generalizability, we provide first, publicly available, empirical evidence that four of the ten AGS crime indices may indeed be sufficient proxies of actual crime in an urban area.