Linking road casualty and clinical data to assess the effectiveness of mobile safety enforcement cameras: a before and after study.

OBJECTIVES: To use police STATS19 road casualty data and accident and emergency and in-patient information to estimate the impact of mobile safety cameras on the cost of treating individuals injured in road traffic collisions.
DESIGN: A data-matching and costing exercise to link casualty and clinical information in a 'before' and 'after' study of 56 mobile safety cameras.
SETTING: The Northumbria Police Force area of the UK covering six local authority districts. PARTICIPANTS: Slight, serious and fatal casualties involved in road traffic collisions at mobile camera sites in the case-study area between April 2001-March 2003 and April 2004-March 2006. PRIMARY AND SECONDARY OUTCOME MEASURES: Changes in the number and severity of casualties at the mobile camera sites between the 'before' and 'after' period that can be attributed to mobile safety camera activity, and any impacts these changes had on the 'cost of treatment saved' by the secondary healthcare service in the case-study area.
RESULTS: Using tariff values for accident and emergency and In-patient Health Resource Groups, the impacts of the cameras in terms of the 'cost of treatment saved' are in the range £12 500-£15 000 per annum. However, inconsistencies between databases resulted in approximately one-third of the casualties not being matched successfully in the clinical databases. The number of closed fractures requiring investigations, treatment and follow-up care reduced considerably, although this was offset by an increase in head injury contusions and open fractures that require high-cost investigations and extensive in-patient care.
CONCLUSIONS: Road safety cameras could have a significant impact in terms of 'cost of treatment saved'. However, it is argued that investigating the impacts of road safety measures in the future should be based on Fully Bayesian techniques as they can produce more reliable estimates of the effects of regression to mean and general trends in casualty statistics.

To match road casualty and clinical data to estimate the ‘cost of treatment saved’ by secondary healthcare providers from the operation of mobile safety cameras in the Northumbria Police Force Area.

On the basis of matched casualty and clinical data, it is estimated that secondary healthcare providers in the case-study area saved between £12 500 and £15 000 per annum in terms of the ‘treatment saved’ as a result of mobile safety camera deployment during the study period. Savings at the national level could be considerable across all safety camera partnerships.

Inconsistencies between the databases resulted in approximately two-thirds of the road casualties in the study being matched successfully with their clinical information. This was only achieved by supplementing the automatic matching process with resource-intensive manual matching.

Conventional statistical methods can lead to under-estimates of the effects of confounding factors, thus overvaluing the benefits of road safety measures.

The main strength is a more accurate estimate of the actual benefits of mobile speed camera deployment to secondary healthcare providers in terms of ‘cost of treatment saved’. The method matches actual casualties with the cost of their clinical treatment, and accounts more realistically for the confounding factors of general casualty trends and regression to mean effects.

Inconsistencies between casualty and clinical databases limited the number of successful matches. Thus, potential problems of bias in the estimates of cost savings cannot be ignored.

Introduction

There is increasing evidence to suggest that road safety cameras are effective at reducing road traffic casualties,1–6 with clear benefits for healthcare providers in terms of a reduced economic burden of medical treatment. Mobile safety cameras have now been deployed routinely across much of Great Britain for almost a decade. However, their continued use as a road safety measure remains contentious. This is often due to ongoing disputes over how much of any observed reduction in casualties can be attributed directly to the cameras, and how much instead to non-scheme effects such as regression to mean, changes in traffic flows at camera sites and general trends in casualty numbers (eg, due to improved in-vehicle safety devices). This raises the important question of how best to account for these non-scheme effects when trying to measure camera benefits accurately? The cameras’ contribution to improved road safety, and any subsequent impact on the healthcare sector through changes in the number and severity of casualties, is overvalued when these confounding factors are either underestimated or ignored altogether. Unfortunately, it is argued here that this is often the case. Further, simply using police data regarding casualty severity can often be inaccurate due to misclassification errors 7 8 and using average ‘cost’ values for each severity class provides a less comprehensive picture of the real impacts on the healthcare sector than using the available data on the costs of treatment for individual casualties. This paper therefore addresses the following important issues:

Matching casualty and clinical data to estimate the ‘cost of treatment saved’ to healthcare providers as a result of mobile safety camera deployment;

Whether the conventional approach for accounting for regression to mean effects produces reliable estimates of camera effectiveness;

The effect of general trends in casualty figures.

In reality, medical and ambulance costs represent only a tiny fraction of the estimated overall value of preventing a road casualty in the UK—for example, less than 1% of the £1.8 m for a fatal casualty and approximately 7% (of £200 000) and 5% (of £20 000) for serious and slight casualties, respectively. In comparison, the human cost element (representing pain, grief and suffering for the casualty and their close friends and relatives) accounts for as much as 70% of the value of preventing a serious casualty, and 55% and 50% of the value for fatal and slight casualties. It is important to consider these different cost elements, for example, when a potential fatal casualty becomes a seriously injured casualty due to a safety camera. Although medical and ambulance costs are much higher for serious casualties (approximately £13 000 compared with £1 000 for a fatal), human costs are considerably different—£1 M for a fatal compared to £140 000 for a serious casualty. Thus, relatively small increases in medical and ambulance costs must be considered in the light of much larger reductions in human costs when reducing casualty severity.

Findings are presented from recent research in the Northumbria Police Force area of the UK to estimate the ‘cost of treatment saved’ to regional healthcare providers resulting from mobile safety camera deployment. Initial research involving the authors9 and funded by the (then) Northumbria Safety Camera Partnership (NSCP) established a data collection methodology and reported on the findings from a larger sample of (67) mobile camera sites. However, this research followed the conventional approach of using Empirical Bayes statistical methods to account for regression to mean, omitted the effects of general casualty trends and had no set of control sites to help estimate the expected number of casualties in the ‘after’ period. This research has now been extended by the current authors to include general trend effects within a different analytical approach for accounting for regression to mean that uses the less widely applied Fully Bayesian framework10 11 with the aim of providing more reliable estimates of the impact of mobile safety camera enforcement. Importantly, these findings have implications for the assessment of road safety interventions in general, in terms of the appropriate treatment of confounding factors, and add further evidence to the case for preferring Full Bayes to Empirical Bayes methods.

Linking road casualty information from the police and patient data from health authorities to assist road safety research has been the focus of several previous studies worldwide,12 but has often proved problematic due to incompatibilities between datasets limiting (sometimes significantly) the number of successful casualty:patient matches that can be made for further detailed analysis. The most relevant study here involved a ‘before’ and ‘after’ investigation of the epidemiological and economic impacts of 47 (44 fixed and 3 mobile) safety cameras by linking casualty and patient data in the Strathclyde region of the UK.12 For the period 1997–2005, some 10 000 (of 19 000) road casualty records were linked successfully to approximately 30 000 hospital and death records. Using straightforward ‘before’ and ‘after’ comparisons of costs, the study estimates that the cameras contributed to savings in the region of £5 M in the study area, but acknowledges that the potential effects of confounding factors on this estimate cannot be ignored.

Methods

The NSCP (now known as the Northumbria Road Safety Initiative), established in April 2003, is responsible for operating road safety cameras on the region's road network at sites with a known history of speeding and collisions in accordance with national government guidelines. A ‘before’ and ‘after’ study was conducted to assess the impact of mobile cameras on the secondary healthcare sector in the region—the ‘before’ period being the financial years 2001/2002–2002/2003 and the ‘after’ period being 2004/2005–2005/2006 to avoid the partnership's ‘bedding in’ year (2003/2004). The analysis aims to answer (1) how many casualties did not occur in the ‘after’ period as a direct result of the mobile safety cameras and (2) what would have been the cost of treating these casualties in hospital had they occurred?

In the initial stage of the project, key data (ie, age, gender, date of collision and local authority code of collision location) were extracted from the NSCP's database for every casualty that occurred at mobile camera sites during the ‘before’ and ‘after’ periods.9 Data were also extracted from accident and emergency departments’ records at the seven hospitals in the case-study area (and those in the immediate surroundings of Carlisle, Durham and the Scottish Borders). For this, approval was granted by the local Research Ethics committee in 2005, and Research and Development Trust and Caldicott approval was obtained from each NHS hospital involved. The two lists were then matched to access medical records of casualties injured at mobile safety camera sites. Thus, casualties not admitted to hospital (ie, via accident and emergency or as an in-patient), for example those who died at the collision scene, are not included in the analysis. A two-stage data-linking process was designed. The first (automatic) stage involved seeking identical matches between police and hospital databases on three key casualty variables: age, gender and date of collision (on police records)/date of admission (on hospital records). This exercise achieved a 44% matching success rate for the ‘before’ period data and 48% for the ‘after’ period data from over 18 000 accident and emergency and over 3000 in-patient records.9 To boost the disappointing sample size resulting from this automatic stage, a second and significantly more labour-intensive (manual) stage was implemented. Having obtained relevant data protection approvals, this involved obtaining the names of unmatched casualties from the police and interrogating databases at each of the 11 hospitals in the study area and the immediate surroundings. This increased the matching success rate in the ‘before’ and ‘after’ periods to 66% and 68%, respectively.

Clinical information was gathered for each matched casualty by grouping them on the basis of their operational demands on the secondary healthcare service. Casualties were clustered into Healthcare Resource Groups (HRGs) based on similar amounts of healthcare resources consumed using the International Classification of Diseases (ICD) and the Office of Population Census and Surveys classification of surgical operations (OPCS). A tariff is allocated to each HRG using data returned annually from National Health Service Trusts reflecting a national average cost of providing healthcare to patients in each HRG (table 1). Using medical records, each casualty was then allocated to one of the eight HRGs.

Table 1

Healthcare Resource Groups (HRGs) and Associated Tariff Values (2005/2006)

A&E HRG	A&E investigation codes	A&E attendance and disposal	Tariff (£)
High-cost imaging (died or admitted)	CT scan or MRI	Admitted to hospital, transferred to another hospital or died in the department	152
High-cost imaging (referred or discharged)	CT scan or MRI	Discharged to follow-up treatment by GP, referred to A&E/fracture clinic or another out-patient clinic or other healthcare professional, or left before treatment/refused treatment/did not require treatment	92
Other high-cost investigation (died or admitted)	X-ray, cross-match, histology and/or ultrasound	Admitted to hospital, transferred to another hospital or died in the department	131
Other high-cost investigation (referred or discharged)	X-ray, cross-match, histology and/or ultrasound	Discharged to follow-up treatment by GP, referred to A&E/fracture clinic or another out-patient clinic or other healthcare professional, or left before treatment/refused treatment/did not require treatment	83
Low-cost investigation (died or admitted)	ECG, haematology, biochemistry, urine, bacteriology or other	Admitted to hospital, transferred to another hospital or died in the department	109
High-cost investigation (referred or discharged)	ECG, haematology, biochemistry, urine, bacteriology or other	Discharged to follow-up treatment by GP, referred to A&E/fracture clinic or another out-patient clinic or other healthcare professional, or left before treatment/refused treatment/did not require treatment	64
No investigation (died or admitted)	–	Admitted to hospital, transferred to another hospital or died in the department	98
No investigation (referred or discharged)	–	Discharged to follow-up treatment by GP, referred to A&E/fracture clinic or another out-patient clinic or other healthcare professional, or left before treatment/refused treatment/did not require treatment	52
Invalid for Grouping	–	–	97

Source: Ref. 9.

A&E, accidents and emergency; GP, general practitioner.

Patients who were then admitted to hospital from accident and emergency departments were allocated to one of 700 in-patient HRGs that also carry a cost of treatment tariff. Here, tariffs were clustered into £500 bands to reduce the in-patient HRGs to a more manageable number. Each matched casualty in the study was then allocated to a single accident and emergency and in-patient HRG combination. These observed frequencies from the ‘before’ data can then be used to estimate the probability of a casualty, which did not occur in the ‘after’ period, falling into a particular HRG combination and hence the cost of the treatment that had been prevented.

To account for confounding factors, the conventional approach to quantify regression to mean generally relies on Empirical Bayes techniques (eg, see ref. 1 and 9) and should use casualty data from a representative sample of control sites to predict the expected number of casualties in the ‘after’ period in the absence of any intervention. The difference between the Empirical Bayes estimate and the observed casualty frequency is then attributed to the road safety camera (plus any trend effects). A weakness here is that the standard application of the Empirical Bayes method only produces a point estimate of the mean number of casualties expected in the ‘after’ period. Frequency distributions of casualties are predominantly skewed positively, with the median value being the usually accepted descriptor of such distributions rather than the mean. In positively skewed distributions, the mean is almost always higher than the median, suggesting (misleadingly) a higher number of expected casualties in the ‘after’ period after accounting for regression to mean, hence a lower regression to mean effect and thus an overestimate of the effectiveness of the road safety measure. Instead, Fully Bayesian methods are increasingly being recommended10 11 to produce a frequency distribution (rather than a point estimate of the mean) of the expected number of casualties in the ‘after’ period that can be described by a range of statistical summaries (such as the mean, median and SD or even plausible ranges for the parameter of interest). Both Empirical and Fully Bayesian approaches are implemented through multiple linear regression models to predict the number of casualties at each mobile camera site in the ‘after’ period using explanatory variables relating to site-specific vehicle speed profiles, daily traffic flow, road type and road classification. To improve the reliability of the regression model outputs, it is crucial that both sets of sites (camera and control) are as comparable as possible in terms of the explanatory variables to control for the effects of all other factors except for the effect of a safety camera. To test the degree of comparability in the explanatory variables between the control and camera sites, a Monte Carlo permutation test13 was conducted on the site characteristics data which confirmed that our sites are comparable at the 5% significance level. As the control sites are only a relatively small sample of all possible control sites that could have been included, a degree of uncertainty clearly exists over the parameter estimates for the explanatory variables in the regression model. Unfortunately, this uncertainty is ignored in the Empirical Bayes approach as only a point estimate of each parameter is calculated. In the Fully Bayesian approach, however, a statistical distribution is produced for each parameter estimate to reflect, and carry forward into the ‘after’ period, the inherent uncertainty and variability. The Fully Bayesian approach also has the added advantages of producing much more realistic SEs as all sources of variability are accounted for, being more flexible both in terms of the models that can be tested to improve model fit (eg, the Weibull/lognormal models) and in terms of enabling general trends in casualty numbers to be included relatively easily as a trend statistic in the regression model. Like-for-like comparisons between Empirical and Fully Bayesian estimates should be made with caution, however, as important differences exist in their statistical approaches.

To account for general trends in casualties, data from the Northumbria road network not covered routinely by mobile camera enforcement suggest a consistent downward trend in total casualties since the start of the ‘after’ period (approximately a 4.7% reduction per annum). This trend is assumed also to have occurred at the treated sites. The trend statistic built into the regression model reflects the trend in the estimate of the expected number of casualties at camera locations in the ‘after’ period. To represent the uncertainty about how much the changes in casualty figures at camera sites are related to overall casualty figures at non-camera sites (ie, a drop of 4.7% per year) and casualty figures from the ‘before’ period only (ie, no annual reduction in the ‘after’ period due to trend), the statistic is allowed to vary between 1 (no trend effect) and 0.906 (2 years of casualty reduction at 4.7% per year in the ‘before’ period) with equal probability.

Results

The study identified 436 casualties at the 56 mobile camera sites in the ‘before’ period. In comparison, 287 casualties were identified at the set of 67 ‘control’ sites during the same period. In the ‘after’ period, casualties fell by 132 (−32%) to 298 at the mobile camera sites. The Empirical Bayes analysis suggests that casualties would have fallen to 321 casualties anyway without camera intervention, limiting their effect to preventing 23 casualties. This figure could be expected to fall further if trend is taken into account although this is not straightforward within the Empirical Bayes framework. A crude solution is to reduce the 23 casualties by 4.7% annually for the 2 years in the ‘after’ period based on the available casualty data for the region. This gives a new estimate of 21 casualties prevented. The results from the Fully Bayesian analysis suggest an expected mean and median number of casualties in the ‘after’ period of 327 and 313, respectively and, after taking trend into account, this reduces to 319 and 306 casualties. As noted above, the median is the most appropriate single statistic to use suggesting that the 56 mobile cameras prevented (306–298) eight casualties.

Using the tariff values for each accident and emergency/in-patient HRG combinations and the probability of a casualty falling into each combination calculated from the ‘before’ clinical data, estimates can be made of the effect of the mobile cameras on the cost of treatment saved as a result of the casualties that were prevented. From the Empirical Bayes approach (which ignores the impact of trend), the estimate of treatment saved is approximately £25 600 over the 2-year ‘after’ period. For the Fully Bayesian analysis, estimates of savings are available using either the mean or median estimate of casualties in the ‘after’ period both with and without the effect of trend. Using the mean value, estimates of the cost of treatment saved during the ‘after’ period are £30 900 (without trend) and £26 500 (with trend), and with the median value, £30 900 (without trend) and £26 700 (with trend).

To provide an insight into the possible reasons behind the changes in cost of treatment, table 2 reports changes in the frequency of the most frequent injuries occurring in the ‘before’ and ‘after’ periods in the initial study,9 noting that some casualties could of course sustain more than one type of injury. For example, the number of closed fractures has reduced considerably. These injuries require investigations, treatment and follow-up care. On the other hand, the number of soft tissue inflammations increased, although these injuries are usually referred to general practitioners. The frequency of head injury contusions and open fractures increased and these injuries require high-cost investigations and extensive in-patient care.

Table 2

Changes in the frequency of common injuries between the before and after period

Diagnosis	Frequency of occurrence
	‘Before’ period	‘After’ period	Percentage change (%)
Soft tissue inflammation	125 (26%)	137 (36%)	+10
Abrasion	67 (14%)	56 (15%)	−16
Contusion	66 (14%)	45 (12%)	−32
Laceration	42 (9%)	29 (8%)	−31
Closed fracture	58 (12%)	29 (8%)	−50
Sprain injury	47 (10%)	44 (11%)	−6
Tendon injury	46 (10%)	12 (3%)	−74
Head injury	17 (4%)	10 (3%)	−41
Head injury contusion	5 (1%)	11 (3%)	+120
Open fracture	7 (1%)	11 (3%)	+57
Total	480 (100%)	384 (100%)	−20

Source: After ref. 9.

Discussion and conclusions

This study's principal findings are that, based on the matched casualty and clinical data, the estimated ‘cost of treatment saved’ by secondary healthcare providers in the study area is between £12 500 and £15 000 per annum during the study period as a result of the deployment of mobile safety cameras. If these findings are typical, then annual savings at a national level across all the safety camera partnerships that cover the vast majority of the UK could be considerable. The study identified inconsistencies between available casualty and clinical databases that limited the number of successful matches that could be made, and also that conventional statistical methods have the potential for underestimating the effects of regression to mean, thus over-valuing the benefits of road safety interventions.

The main strength of the study, which sets it apart from previous research, lies in the development of a procedure for estimating more accurately the actual benefits of mobile safety camera deployment in terms of the ‘cost of treatment saved’, with the method accounting more realistically for the confounding factors of regression to mean and general casualty trends. Casualties are matched with the cost of their clinical treatment and the procedure has the potential to be used in the evaluation of a wide range of other road safety measures. Therefore, the results clearly have implications for the cost effectiveness of mobile safety cameras, especially if the benefits in terms of casualty reduction are not as great as currently thought due to the underestimation of regression to mean. Until 2007, camera operations were funded by the fine revenues they generated through a hypothecation scheme. Since then, however, funding has been through a road safety grant where safety cameras have to compete against other road safety initiatives for financial support. This competition has therefore focused attention very sharply on safety cameras’ value for money relative to other road casualty reduction measures, and the outcome of this competition for funds will determine whether or not local road authorities in the UK continue with the policy of traffic speed enforcement through speed cameras and at what level.

The main limitation of the study is the low rate of successful matches (44% and 48%) from the automatic linking of the casualty and clinical databases used in the initial study which are disappointing, and the problems of bias this can create. Although these rates are consistent with previous matching exercises,8 14 overall rates have clearly not improved significantly over the past 15–20 years. Possible reasons for failed matches based on similar studies are suggested elsewhere.15 Here, common issues that had to be resolved in the matching process included:

Casualties of the same age and gender from separate collisions on the same day attending different hospitals;

Incorrect casualty ages and/or dates being recorded, for example for the collision or accident and emergency admittance;

Casualties injured in late-evening collisions (before midnight) arriving at hospital the following day causing a date mismatch (between the date of collision and the date of admittance);

Casualties from the same collision attending different hospitals;

Police data recording casualties’ age and hospitals recording date of birth.

These issues meant that the automatic linking procedure still had to be supplemented by time-consuming manual methods to identify unmatched casualties and boost the sample size. Approval had to be gained from appropriate data protection officials to allow direct enquires to Northumbria Police to release further casualty information. As already mentioned, the issue of unmatched casualties introduces the potentially serious problem of bias, in this case, into the estimates of the cost savings. Clearly, it is not known as to how the unmatched casualties in the ‘before’ period are distributed between the accident and emergency and in-patient HRG combinations. If this distribution of unmatched casualties is weighted more towards the higher-cost combinations (compared to the distribution for matched casualties) then the cost savings will be underestimated as the casualties that did not occur in the ‘after’ period will be under-represented in the higher-cost combinations and vice versa.

An approach involving an integrated casualty and clinical database would overcome many of these data issues, or at least a higher degree of consistency between the two in recording key information such as dates of birth and dates of collision and hospital admittance. Indeed, single databases for injury research have been advocated for some time now.16–18 This would make larger-scale assessments of road safety schemes, for example at the national and international scale, much more feasible from a human resource perspective. The data-linking approach described here also serves as a reminder of the benefits of such an approach to evaluation compared to using simple casualty classifications of fatal, serious and slight as misclassifications can often occur7 8 and there was evidence of some possible discrepancies revealed in this study.

From the statistical analysis of the effect of confounding factors, it is clear that site selection issues are extremely important in determining the location of camera sites to generate maximum return in terms of casualty reduction. From the evidence here, regression to mean effects can significantly reduce the apparent impact of mobile safety cameras, which may cause a re-evaluation of the current belief that road safety schemes generate high value for money. Deployment sites in Great Britain are selected typically on the casualty history during a 3-year baseline period. Extending this period up to (say) 5 years or longer where data exist would help highlight either a growing casualty problem that a safety camera might help solve, or simply a short-term ‘blip’ after which annual casualty rates would return to existing levels without the necessary expense of an intervention, allowing limited resources to be deployed elsewhere. Also, it is recommended avoiding aggregate casualty figures for the baseline period as this can mask downward trends in casualty numbers or atypically ‘bad’ years.

In conclusion, this paper has presented evidence to suggest that conventional approaches to account for regression to mean effects that rely on Empirical Bayes techniques could lead to overoptimistic assessments of the value of road safety measures, suggesting that value-for-money decisions may not be optimal. This problem becomes more serious when the frequency distribution of predicted casualties at a treatment site shows clear positive skewness with an increasing difference between the predicted mean and median values. It is recommended here that a Fully Bayesian approach is adopted which, it is argued, is statistically more appropriate to handling casualty data and flexible enough to allow confounding factors to be incorporated more rigorously with the end result of more reliable estimates of the impacts of road safety measures. This study has demonstrated the value of including the effects of (generally downward) trends in casualty profiles to provide a more accurate picture of scheme-only impacts. Unfortunately, these effects are often omitted. The ‘cost of treatment saved’ suggested here may seem modest once the confounding factors have been accounted for appropriately. In context, reported casualties in the Northumbria region in 2010 represent only a very small percentage (approximately only 2.5%) of the total in the case of Great Britain. If the savings suggested here were replicated proportionally elsewhere, then the total savings in terms of treatment could run into many millions of pounds over the lifetime of safety camera partnerships. Further, these calculations do not include the additional ‘costs’ mentioned in the opening section to this paper that are borne by society in general, such as pain, grief and suffering and loss of output resulting from road traffic collisions, which would increase these estimates considerably.