Evaluating the causal impact of individual alcohol licensing decisions on local health and crime using natural experiments with synthetic controls.

BACKGROUND AND AIMS: Given the costs of alcohol to society, it is important to evaluate whether local alcohol licensing decisions can mitigate the effects of alcohol misuse. Robust natural experiment evaluations of the impact of individual licensing decisions could potentially inform and improve local decision-making. We aimed to assess whether alcohol licensing decisions could be evaluated at small spatial scale by using a causal inference framework.
DESIGN: Three natural experiments. SETTING AND PARTICIPANTS: Three English local areas of 1000-15 000 people each. INTERVENTION AND COMPARATOR: The case study interventions were (i) the closure of a nightclub following reviews; (ii) closure of a restaurant/nightclub following reviews and (iii) implementation of new local licensing guidance (LLG). Trends in outcomes were compared with synthetic counterfactuals created using Bayesian structural time-series. MEASUREMENTS: Time-series data were obtained on emergency department admissions, ambulance call-outs and alcohol-related crime at the Lower or Middle Super Output geographical aggregation level.
FINDINGS: Closure of the nightclub led to temporary 4-month reductions in antisocial behaviour (-18%; 95% credible interval - 37%, -4%), with no change in other outcomes. Closure of the restaurant/nightclub did not lead to measurable changes in outcomes. The new licensing guidance led to small reductions in drunk and disorderly behaviour (nine of a predicted 21 events averted), and the unplanned end of the LLG coincided with an increase in domestic violence of two incidents per month.
CONCLUSIONS: The impact of local alcohol policy, even at the level of individual premises, can be evaluated using a causal inference framework. Local government actions such as closure or restriction of alcohol venues and alcohol licensing may have a positive impact on health and crime in the immediate surrounding area.

Introduction

There is extensive literature that describes a relationship between location‐based interventions in the alcohol environment, such as changes in alcohol outlet density or in the hours that alcohol can be sold and alcohol‐related consequences on crime and health 1, 2.

The costs of alcohol to the United Kingdom have been estimated to be somewhere between 21 and 52 billion pounds in 2016 3. Local authorities in England have powers to influence local alcohol environment by contributing to the licensing process and control the enforcement of existing licenses and can thereby help in mitigating the impact of alcohol on society, although they cannot, for example, set levels of tax 4. Licensing legislation requires that all licensing decisions examine evidence about specific outlets or local authorities, yet most published or usable evidence exists for larger spatial aggregates which typically cannot demonstrate causal links between individual outlets and harmful outcomes 3. This gap in the availability of evidence of effectiveness around individual premises is an important limitation to local licensing decisions (e.g. revoking licenses or reductions of trading hours). Legal advice to practitioners in licensing departments with respect to cumulative impact zones and other licensing activities indicated that formal evaluations of the actual impact of changes to policy could be important to supplement case law and other sources of evidence in licensing hearings. The availability of quantitative evaluations at small temporal–spatial scales, ideally linked to single premises, would thus be beneficial.

The implementation of local licensing policies or licensing actions such as the closure of specific venues generally results from political processes. The timing of these decisions makes it difficult for research teams to influence when and where these are implemented (or in some situations it would be unethical to do so) and as such evaluations using randomized controlled trial (RCT) designs are not possible. Natural experiments (and other quasi‐experimental designs) offer the opportunity to evaluate such interventions 5. The UK Medical Research Council define natural experiments broadly as studies that divide population into exposed and unexposed groups, but where the allocation to these groups was not under the control of the research team 6. The strength of the inferences that can be made from natural experiments rely largely on the plausibility that the intervention could be considered to be allocated ‘as‐if randomized’; i.e. although not controlled by the research team they can be considered random, for example due to ‘the forces of nature’ 7, 8. In such circumstances, natural experiments can generate estimates of effects comparable in causal interpretation to those from controlled trials, while they have the added benefits of avoiding the artificial setting common to trials which can improve their external validity 8. Another benefit is that they can be conducted using routinely collected data, which (i) enables relatively quick evaluations at low cost as the relevant data are already being collected when an intervention is introduced (and hence pre‐intervention data is already available) and (ii) in addition to the evaluation of planned public health interventions also allows for the evaluation of (unplanned) withdrawal of such interventions 9.

We aimed to utilize the benefits from natural experiments to assess the possibility of evaluating the impact of local alcohol licensing interventions on health and crime at small geospatial levels up to single premises and their immediate surroundings. To do so, here we compare temporal trends to their counterfactuals which, because these are by definition unmeasurable, are estimated by using synthetic controls 10. The synthetic controls are calculated using Bayesian structural time–series, a methodology that was previously shown to be beneficial in the evaluations of alcohol policies at the Local Authority level 4. Specifically, we developed three case studies together with local practitioners to evaluate the impact of two licensing decisions and the introduction and subsequent defunding of new guideline documentation on health and crime in their immediate vicinity, with the aim to directly support local council practice.

Methods

Case studies

Based on the outcomes of a work‐shop with practice partners from Local Councils, Public Health England and academic colleagues, three cases from different regions in England were selected for inclusion in the study and developed. These included (i) the closure of a venue following a licensing committee review, (ii) the closure of two colocated venues following review and (iii) where new local licensing guidance and increased inspections were established (iiia) and subsequently defunded and restructured (iiib), providing two intervention points for analyses. Outcomes were selected prior to analyses for each of the case studies through discussions between local government officials and the research team and included time–series of reported numbers of specific crimes and health outcomes, covering a pre‐ and post‐intervention period. Data were obtained by the practice partners from existing collaborations, including multi‐agency groups, and included recorded numbers of calls for service from police records classified based on national criteria based on locations of the incidents, hospital admissions data based on the home address of the individuals and ambulance data based on the location of the incidents. Counts were aggregated to lower layer super output geographical areas (LSOA; 1000–3000 people) for the first two case studies and middle‐layer super output areas (5000–15 000 people) for case study 3 11. More details of each case study are provided in the Supporting information.

Counterfactuals

The measured time–series were compared to the counterfactual time–series, which are the potential outcomes in the case the intervention had not been introduced 12 and the difference interpreted as the effect of the intervention. Counterfactual series were constructed by synthetic controls methods 10 using the weighted average of the outcome variable from a group of local control areas (described in the Statistical methodology). Control areas were areas at the same geospatial level selected from the same Local Authority and were selected by the practice partners as comparable to the case area. We limited the selection of control areas to those from the same Local Authority in this study because (a) at such small geographical‐level local practitioners would be best placed to select comparable controls and (b) these areas would be subject to similar trends resulting from policies and priorities within the Local Authority. In addition, by limiting the selection of areas to the same Local Authority this would facilitate future local use of the methodology. A possible limitation of selection by local practitioners may be the risk of implicit or explicit bias in the selection of the controls, but the complexity of how these are combined to create synthetic counterfactuals mitigates the impact of this. Each synthetic control was based on different combinations of the selected control areas (ranging from two to six), depending on the case study.

Statistical methodology

Bayesian structural time–series have been described in detail elsewhere 13, 14, but in brief are state‐space models that can incorporate trends, seasonality and regression components. Two linked equations describe (a) how the outcomes are related to an underlying latent state, and (b) how the latent state changes over time. The errors of different state components are assumed to be normally distributed with a mean of zero and independent of each other. The regression component of the model enables the inclusion of covariates. To develop the synthetic controls, a spike‐and‐slab prior was placed on the regression coefficients, where the ‘spike’ determines the probability that a covariate (here, the time–series of the outcome in a control area) has a non‐zero coefficient based on independent Bernoulli distributions and the ‘slab’ is a weakly informative Gaussian prior with a large variance. We used Bayesian model averaging across all combinations of predictors to construct an outcome time–series from the covariate time–series 15. The posterior predictive density is a joint distribution over all data points 14, and the posterior distribution of the model parameters was estimated using a sequence of Gibbs sampling steps from a Markov chain with a stationary distribution; i.e. stochastic search variable selection 16. The posterior distribution of the time–series can then be projected forward using the regression component of the model. The difference between the measured and projected (or forecasted) time–series can be interpreted as the effect of the intervention. The 95% Bayesian credible interval (CrI) provides an estimate of the precision of average and cumulative effects over the forecasting time horizon and posterior tail area probabilities, calculated as the samples where the CrI of the cumulative effect excluded zero, are interpreted as classical P‐values 13. One minus the posterior tail area probability can also be interpreted as the probability of a causal effect 17.

Individual model specifications were required for each case study and outcome combination because of differences in the data with respect to length of the time–series, number of control sets, temporal variability and other temporal patterns (such as seasonality). Each initial model was based on a random walk plus noise, or local‐level, model and sequentially other structures—autoregressive of order 1 [AR(1)] model, local‐linear time trends and additional seasonal components 18 were assessed. Common to all analyses were the following specifications: expected explained variance was set to 90%, and priors for the mean of each outcome and its standard deviation were defined as the arithmetic mean of the outcome in the pre‐intervention period and 10% of its standard deviation, respectively, and the upper limit for the standard deviation was set to 200% of that of the pre‐intervention time–series. Model evaluation was conducted graphically based on plots, and statistically using root mean square errors for the pre‐intervention (training) time‐period (partial) autocorrelation plots and Ljung–Box tests to assess independent distribution of residuals and precision of counterfactual estimates (standard deviation of forecasts). Markov Chain Monte Carlo samples (10 000) were used and convergence evaluated using Geweke and Raftery & Lewis diagnostics.

Falsification tests in which different implementation dates (temporal falsification) and replacement of case areas by control areas (spatial falsification) were also conducted to strengthen causal inference 19.

Bayesian structural time–series models were implemented using the bsts 18 and CausalImpact 17 packages in R (version 3.2.4).

The analysis plan was not pre‐registered and the results should therefore be considered exploratory in nature.

Results

Case study 1: closure of a nightclub

This nightclub was opened in September 2011 but was put under review by local government following concerns about crime and antisocial behaviour at the premises, and eventually forced to close in September 2013 (see OSM). Time–series of reported incidents of crime (all offenses), specifically antisocial behaviour, and ambulance call‐outs and emergency admissions to hospital for alcohol (i.e. the outcome data) from 2010 to September 2014 (12 months post‐closure) were used to evaluate the intervention, but incident counts from one to four were not provided because of identifiability concerns, and these were imputed using random imputation with equal inclusion probabilities. Evaluation of 12‐month follow‐up was modelled based on a semi‐local linear trend model with two seasonal components (12 months and four seasons) and 4‐month follow‐up (see below) based on a local level with seasonal components 18. The results for all outcomes are shown in Fig. 1. There was little evidence of a longer‐term impact at 12 months on any of the selected outcomes as a result of the closure of the venue (Table 1). Graphical observation of the temporal patterns was indicative of an initial, but temporary, reduction in the incidence of reported incidents of antisocial behaviour. Post‐hoc analyses of this pulse impact (i.e. the shortest period of impact 20) indicated a 4‐month impact of −18% (95% CrI = −37%, −4%), equating to 60 averted incidents. Falsification tests indicated that this was specific to the time‐point and the area of closure of the venue (as summarized in Table 1).

Figure 1

Case study 1: evaluation of closure of a venue following a licensing committee review. Impact of post‐hoc 4‐month evaluation highlighted at top left.

Table 1

Case study 1 evaluation of the impact of closure of a venue following review on selected outcomes, and results of temporal and spatial falsification tests for observed effects.

Evaluation of the 12‐month impact of the closure of the venue
Outcome	12‐month average impact	95% CrI	Posterior tail‐area probability
Anti‐social behavioura	+8%	−63%, +49%	0.36
Crime (all offences) 1	+4%	−69%, +77%	0.45
Ambulance call‐outsb	−9%	−36%, +20%	0.22
Emergency Admission to hospital for alcoholb	−19%	−155%, +113%	0.39
Post‐hoc evaluation of 4‐month impact
Outcome	4‐month average impact	95% credible interval	Posterior tail‐area probability
Anti‐social behaviour	−18%	−37%, −4%	0.01

Monthly number of reported events;

quarterly number of events;

the time‐point of the intervention was artificially moved forwards or backwards;

each control area was artificially assigned as the intervention area. CrI = credible interval.

Case study 2: closure of a restaurant and cocktail bar

Both co‐located premises were closed in November 2016 following a licence review brought by the police on the grounds of crime and disorder related to the premises. Time–series of reported incidents of crime (all offences) and antisocial behaviour specifically were available for 2015–17. AR(1) models with linear local trend and monthly seasonal components were used to model the counterfactuals, and the results for the closure of the venues are shown in Fig. 2. Uncertainty is large as a result of the low numbers of incidents in these LSOAs, and based on point estimates and precision there was little indication of a measurable effect on the incidence of reported antisocial behaviour or crime incidents in the subsequent 12‐month period.

Figure 2

Case study 2: evaluation of the impact of closure of two venue on the monthly incidence of reported antisocial behaviour and crimes in the immediate local area.

Case study 3a and b: local licensing guidance

In 2012 the introduction of new statutory guidance facilitated the development of Local Licensing Guidance (LLG) for specific areas which was taken up for a designated area within an urban conurbation. The LLG was in place during 2013/14, during which there was active engagement, including additional inspections of venues, by Public Health locality teams. However, as a result of budget cuts and changes to resource allocation there were no more active follow‐ups after the 12‐month period.

Time–series of monthly reported incidents of drunk and disorderly behaviour, sexual offences, antisocial behaviour and domestic violence were available for the period 2008–14. Different model specifications were required to best model the temporal patterns of each outcome: drunk and disorderly behaviour was modelled on an AR(1) autoregressive model with a monthly seasonal component, sexual offences on a local level model with a monthly seasonal component and antisocial behaviour and domestic violence both on a semi‐local linear time trend and a monthly seasonal component. The results of the introduction of the LLG (3a) are shown in Table 2 and Fig. 3, and suggest weak evidence for a reduction in the incidence of reported drunk and disorderly behaviour incidents [12‐month impact of −42% (95% CrI = –109%, +23%) compared to the counterfactual]. This, however, corresponds to fewer than one averted incident per month (95% CrI = –2, +0.3) on average. Falsification tests indicated that this weak evidence of effect was specific to the time‐point as well as to the area (with the exception of one neighbouring area, where a comparable effect was observed).

Table 2

Case study 3a: evaluation of the impact of new LLG and increased inspections on selected outcomes, as well as results of temporal and spatial falsification tests for observed effects.

Monthly number of reported incidents	12‐month average impact	95% Credible Interval	Posterior tail‐area probability
Drunk and disorderly behaviour	−42%	−109%, +23%	0.10
Sexual offences	+5%	−95%, +90%	0.44
Antisocial behaviour	−12%	−95%, +36%	0.40
Domestic violence	+0.7%	−28%, +30%	0.48
Temporal falsification for drunk & disorderly behavioura	12‐month average impact	95% Credible Interval	Posterior tail‐area probability
6 months earlier	−1%	−95%, +91%	0.49
6 months later	−27%	−115%, +61%	0.27
Spatial falsification for drunk & disorderly behaviourb	12‐month average impact	95% CrI	Posterior tail‐area probability
Control area 1	−9%	−64%, +43%	0.36
Control area 2	−53%	−119%, +14%	0.06
Control area 3	−46%	−183%, +89%	0.23
Control area 4	+27%	−106%, +156%	0.33
Control area 5	+329%c	−148%, +821%	0.08
Control Area 6	−64%	−196%, +60%	0.15

The time‐point of the intervention was artificially moved forwards or backwards.

Each control area was artificially assigned as the intervention area.

This percentage is misleading because it is based on very small incidence.

CrI = credible interval; LLG = local licensing guidance.

Figure 3

Case study 3a: evaluation of the introduction of the Local Licensing Guidance and increased inspections.

Results of the evaluation of the impact of defunding and restructuring of the LLG initiative (3b) are shown in Table 3 and Fig. 4. Strong evidence of a reduction in antisocial behaviour events compared to the counterfactual was observed, which was thought to be related change of the role of the Community Safety Manager at the same time, which contributed to a loss of engagement with the local community and businesses and a reduction in the number of incidents reported. Indeed, falsification tests indicated this change was not specific to the time‐point or area in which the LLG initiative was defunded. There was also moderate evidence of an increase in reported incidents of domestic violence of +11% (−10%, +35%), which corresponds to two additional incidents per month. Falsification tests indicated that the increase happened at approximately the same time as the withdrawal of public health, and was also not observed in any of the other areas.

Table 3

Case Study 3b: evaluation of the impact of defunding and restructuring of LLG on selected outcomes, as well as results of temporal and spatial falsification tests for observed effects.

Outcome	12‐month average impact	95% CrI	Posterior tail‐area probability
Drunk and disorderly behaviour	−37%	−213, +123%	0.31
Sexual offences	−8%	−152%, +138%	0.46
Antisocial behaviour	−61%	−109%, −19%	0.01
Domestic violence	11%	−10%, +35%	0.14

3 months instead of 6 months temporal falsification was a priori selected to minimize the effects of the introduction of the LLG and additional inspections, which overlaps with the earlier temporal falsification test. This allowed for 4‐month impact periods which, for consistency, were also used for the spatial falsification tests;

the time‐point of the intervention was artificially moved forwards or backwards;

each control area was artificially assigned as the intervention area. CrI = credible interval; LLG = local licensing guidance.

Figure 4

Case study 3b: evaluation of the defunding and restructuring of the Local Licensing Guidance.

Discussion

The analyses presented in this paper highlight that it is possible to quantitatively evaluate the small‐area impact of local changes to the local alcohol environment up to the level of single premises and their immediate surroundings in a robust manner through synthetic control methodology. Our single case studies did not (and would not be expected to) result in large effects, and they are implemented in complex local systems 21. Hence, the quantitative results we report in this study should be appraised in conjunction with local information about implementation and context from local practice partners. As such, the evidence from these natural experiments contribute to the evidence of effects of local licensing policies at the Local Authority level in England 4, 22, 23 and is also in agreement with results from natural experiments looking at effects of choices in the alcohol environment more broadly 24, 25, 26.

In case study 1 there was strong evidence for an immediate, but relatively short‐term, impact of closure of the venue on the incidence of reported antisocial behaviour, but not for local crime rates, emergency admissions to hospital for alcohol or for ambulance call‐outs. For that immediate 4‐month period, the impact equated to 15 incidents (95% CrI = –3 to – 30 incidents) averted per month, or 60 in total. However, this 4‐month pulse impact was defined post hoc and there was no evidence of impact after this period. Falsification tests provided reassurance that the observed impact could be ascribed to the closure of the venue, although weak evidence of an observed effect in one of the control areas highlight the complex setting, with many additional exogenous factors affecting these local systems. Feedback from the practice partner indicated that a reduction in antisocial behaviour after closure of the venue conformed to their expectations, given that this was one of the main reasons that resulted in its closure. However, there is no clear explanation why this would have returned to the normal level after the initial pulse.

In case study 2 there was little evidence that closure of the venues had resulted in measurable effects on reported incidents of crimes or antisocial behaviour in the area immediately surrounding the venues. The influence of high variability and exogenous factors that affect the time–series (here the sudden reduction in recorded antisocial behaviour in the year prior to the closure of the venues) complicated statistical inferences. There may not have been direct effects on the immediate vicinity of the venues, because these were located in the connecting area between two night‐life focal points in the conurbation and thus any effects (if these occurred) would have been observed elsewhere.

Case study 3 described weak to moderate evidence of a small impact on the incidence of reported drunk and disorderly behaviour in the area following the introduction of LLG, corresponding to approximately one to two averted events per month compared to the counterfactual. Comparable effects were observed in some adjacent areas as well which, the local practitioners indicated, also received additional inspections by the Public Health team in that period, providing additional weight to the evidence for a small effect as a result of the LLG. The subsequent defunding and restructuring of the LLG provided moderate evidence that this had resulted in a small increase in incidents of domestic violence of approximately two additional incidents monthly compared to the counterfactual. In interpreting these effects, it is important to note that these small‐scale interventions do not occur in a vacuum and, as a consequence of the restructuring process other work being carried out at the time, including work on domestic violence, also changed, which may have contributed to the observed excess incidents of domestic violence.

These analyses illustrate the difficulties of the statistical modelling in these highly localized settings, with low numbers of events, high temporal variability and relatively small expected effect sizes (if any). The Bayesian structural time–series approach allows, in theory, for an accurate modelling of counterfactual time–series but may have benefited from longer time–series to increase statistical power, and also suffers when no events occur on many time‐points. Moreover, these natural experiments occur in a complex systems setting, but these analyses are based on traditional linear evaluations of cause and effect 21. For example, the pulse effect observed in case study 1 may be the result of an unknown feedback loop not taken into account by the method. Comparable control areas were selected from the same local authorities in all cases and had varying distances to the case areas. The main assumption of this approach is that the correlations between outcomes prior to the evaluated intervention remain the same in the post‐intervention period. Although there was little evidence of spillover effects, which have been reported in geographical studies of alcohol 27, but not in England in previous analyses 22, some evidence of spatial clustering was observed, such as in study 3 where the falsification test highlighted similar effects of the introduction of LLG in a neighbouring area. Most unobserved confounding factors, such as macro‐economic changes and specific local policies or priorities, are in theory accounted for because the synthetic controls were based on weighted combinations of comparable control areas from the same Local Authority and/or are unlikely to change significantly over these time–periods such as, for example, area‐level demographics or deprivation. However, residual confounding cannot be excluded because the researchers were not in control of the randomization process. Generally, the use of synthetic controls has advantages over other analytical methods for natural experiments 10, but regardless of the analytical method the stronger the argument is that the allocation of the intervention and the selection of control areas may mimic a random process (as‐if randomization), the less likely that the results may be significantly confounded 7.

The main strengths of this study were that these analyses demonstrate that in local policy interventions where randomized controlled trials (RCTs) are not possible, natural experiments in which as‐if randomization enables assignment on observed and unobserved factors 28 can be a valuable alternative for robust evaluation if sufficient data are available 6. Moreover, this study indicates that comparisons to counterfactuals can be performed at small area‐levels with low numbers of incidents, although with the expected effect sizes in these case studies, moderate strength of the evidence was the maximum achievable 29. The comparison to the counterfactual was an important strength of the current methodology, with advantages over other alternative methods for the evaluation of natural experiments such as pre–post designs or interrupted time–series 6, 10 (note that there are alternative statistical methods to the Bayesian structural time–series that enable comparison to controls). Specific strengths related to the use of Bayesian structural time–series include the inclusion of subjective knowledge to defining the priors, the inclusion of posterior uncertainty about the covariates in the model and their coefficients, the accommodation of serial autocorrelation and cross‐sectional and longitudinal heterogeneity and the inclusion of Bayesian model averaging to mitigate issues around selection of covariates and overfitting 14.

To our knowledge, these were the first studies in which the effect of small‐scale interventions, at the levels of individual premises in two of the case studies, were evaluated using a quantitative method that enabled robust comparison to counterfactuals. The use of this methodology to conduct small‐scale evaluation has the potential to bridge the gap between the predominantly data‐orientated approach by public health teams and the need for contextualized evidence for presentation to councillors 30, at least in principle. However, the small effect sizes and complex modelling provide moderate evidence of effects, at best. Although in this study we used this analytical framework in the context of evaluation of the impact of local alcohol (licensing) policies, it is a general framework for evaluation of natural experiments, and as such is likely to be also applicable to the evaluation of natural experiments and other quasi‐experimental in the broader public health setting.

Declaration of interests

J.B. has received via his university unrestricted funding to study smoking cessation from Pfizer, who manufacture smoking cessation medications. M.H. has received unrestricted honoraria in the last 3 years unrelated to this work from MSD, Gilead, Abbvie.

Data S1 Supporting Information.

Click here for additional data file.