A review of the validity and reliability of alcohol retail sales data for monitoring population levels of alcohol consumption: a Scottish perspective.

AIMS: To assess the validity and reliability of using alcohol retail sales data to measure and monitor population levels of alcohol consumption.
METHODS: Potential sources of bias that could lead to under- or overestimation of population alcohol consumption based on alcohol retail sales data were identified and, where possible, quantified. This enabled an assessment of the potential impact of each bias on alcohol consumption estimates in Scotland.
RESULTS: Overall, considering all the possible sources of overestimation and underestimation, and taking into account the potential for sampling variability to impact on the results, the range of uncertainty of consumption during 2010 was from an overestimate of 0.3 l to an underestimate of 2.4 l of pure alcohol per adult. This excludes the impacts of alcohol stockpiling and alcohol sold through outlets not included in the sampling frame. On balance, there is therefore far greater scope for alcohol retail sales data to be underestimating per adult alcohol consumption in Scotland than there is for overestimation.
CONCLUSION: Alcohol retail sales data offer a robust source of data for monitoring per adult alcohol consumption in Scotland. Consideration of the sources of bias and a comprehensive understanding of data collection methods are essential for using sales data to monitor trends in alcohol consumption.

INTRODUCTION

The harmful use of alcohol causes 2.5 million deaths worldwide every year and is the third largest contributor to the global disease burden (World Health Organization, 2011). Levels of alcohol-related harmful effects on health and society are strongly and consistently linked to population levels of alcohol consumption (Rabinovich ; Wagenaar ). As the aggregate consumption of a country increases, so does the prevalence of heavy drinkers (Babor ). Accurate measurement of per capita consumption is therefore crucial for the assessment of the effectiveness of alcohol-related policies.

Surveys of self-reported alcohol consumption are limited by sampling bias, response bias, measurement bias and recall bias (Stockwell ). Consequently, per capita consumption estimates derived from surveys are usually substantially lower than those based on objective data such as retail sales which, according to the World Health Organization (WHO), offer the most accurate means of estimating the amount of alcohol consumed by a population (World Health Organization, 2000). Surveys are, therefore, not considered a valid source for the estimation of per capita consumption of alcohol (Stockwell ), despite some recent recall methods achieving a high coverage of alcohol sales data (Stockwell ). However, while the strengths and limitations of survey data are well understood, the validity and reliability of using sales data to estimate alcohol consumption have not been subjected to the same degree of scrutiny. This requires consideration and, where possible, quantification of possible biases in the methodology, as well as statistical techniques to identify the precision of the estimates.

In this paper, we review these sources of bias and, using Scotland as an example, assess their potential impact on the validity and reliability of using retail sales data for the purpose of measuring and monitoring population alcohol consumption.

METHODS

Literature searches were performed in Ovid Medline, EMBASE and Google Scholar for English language papers with date up to August 2011. Keywords (alcohol, alcohol drinking, sale*, sold, sell*, consumption, buy*, bought, purchas*, commerce, measur*, estimat*, under*, over*, valid*, reliab*) were searched in multiple combinations. The WHO website (Health Topic Alcohol) was also reviewed for key publications and reference lists were reviewed for further articles. Potential sources of bias identified from these sources were mapped and reviewed by an expert group. Further information, including relevant statistics or further research articles, was then obtained and, where possible, the size of the bias was quantified. This process enabled an assessment of the extent to which the different sources of identified biases might impact on alcohol consumption estimates reported in Scotland and, where relevant, in England/Wales.

Alcohol retail sales data for Great Britain were obtained from alcohol market research specialists, The Nielsen Company and CGA Strategy (Nielsen/CGA). The methods used by Nielsen/CGA to estimate alcohol retail sales in Scotland and England/Wales have been described in detail elsewhere (Thorpe ). Briefly, Nielsen/CGA maintain a list of all on-trade outlets (i.e. places where alcohol is sold for consumption on the premises), amounting to ∼134,000 separate outlets in GB. This sampling frame is stratified into 124 geographical locations on the basis of postal addresses and 109 outlet types (e.g. social club, ‘branded’ public house, hotel etc.) and data are collected from a stratified random sample of 5600 outlets. This is supplemented with automated data collection on volume of sales in an additional 57,000 outlets. On-trade sales estimates were produced by applying data on the volume, type and price of products bought in outlets of different types to similar outlet types within the same geographical area. Off-trade sales (i.e. sales for consumption elsewhere) estimates are produced from electronic sales records from most large retailers (i.e. census data which account for approximately three quarters of all off-trade sales) and a stratified random sample of smaller retailers weighted and applied to the entire sample frame.

To enable the magnitude of over- and underestimation to be illustrated, we assessed the impact of the identified biases on the 2010 per adult consumption estimate in Scotland (11.8 l of pure alcohol) (Robinson ). We then, where necessary, calculated the magnitude of bias that would be required to change per adult consumption by an arbitrary value of 0.5 l and assessed the plausibility of biases of this size. Data sources used to illustrate the impact of bias on consumption estimates are cited throughout the ‘Results’ section. A fuller description of the methods and findings are given in an NHS Health Scotland report (Thorpe ).

RESULTS

Several sources of bias, which have the potential to affect the accuracy of estimating per adult consumption in a given year (validity) and/or the extent of measurement accuracy over time (reliability), were identified (Table 1).

Table 1.

Potential sources of bias in per adult estimation of alcohol consumption from retail sales data in Scotland

Potential sources of bias leading to overestimation of per adult consumption	Potential sources of bias leading to underestimation of per adult consumption	Potential sources of bias where the direction of effect is unclear
Underestimated population denominator due to exclusion of some groups, e.g. visitors to Scotland, non-Scottish resident students studying in Scotland and alcohol consumption by those aged <16 years	Alcohol consumed by Scottish residents when outside Scotland	Representativeness of the sample frame
Personal exports	Non-inclusion of some alcohol sales outlets	Non-response bias
Alcohol stockpiling	Personal imports	Measurement error
Wastage and spillage	Home-brewed alcohol
Alcohol used in food	Illegal sources (illegal imports, illegal manufacture, undeclared release of alcohol for sale)
	Substitute alcohols (i.e. alcohol not intended for human consumption)

Sources of bias potentially leading to overestimation of per adult alcohol consumption

Underestimation of the denominator population: visitors to Scotland

People who visit Scotland for business, to visit friends and relatives or on holiday may consume alcohol while they are in the country. This may lead to an underestimation of the number of people who are consuming alcohol in Scotland and to a consequent overestimation of per adult Scottish sales estimates (World Health Organization, 2000). This is the opposite of the population of Scotland buying and consuming alcohol while in other countries, which creates a bias in the opposite direction. The size of these biases can be estimated by considering the number of visitors to Scotland and the number of trips taken out of Scotland by Scotland's residents. These data come from the International Passenger Survey and the United Kingdom Tourist Statistics (UKTS) (Barnes and Smith, 2011; VisitEngland, VisitScotland, VisitWales and Northern Ireland Tourist Board, 2011).

In total, there were 8.37 million visitors staying one or more nights in Scotland in 2010 compared with 8.22 million trips of one or more nights made by Scottish residents to other nations—an excess of 0.15 million visits spent in Scotland by international visitors or those from other regions of the UK. The data for the number of nights spent outside Scotland by Scottish residents are incomplete. Table 2 shows the estimated effects on per adult consumption estimates for varying lengths of Scottish trips abroad. If Scottish residents spend an average of just one night on international visits, per adult consumption in 2010 is overestimated by 0.2 l. If, however, an average of nine nights is assumed, the same average duration spent by international visitors in Scotland, tourism becomes a minor source of underestimation, with per adult consumption increasing by 0.04 l. This underestimation increases to 0.2 l per adult if an average duration of 14 nights is assumed.

Table 2.

Effect of differing length of trips abroad by Scottish residents on per adult alcohol consumption estimates in 2010

	Assumed number of nights per trip
	1	7	9	14
International trips by Scottish residents (000 s)	3,572	3,572	3,572	3,572
Total number of nights spent in international destinations by Scottish residents (000s)	3,572	25,004	32,148	50,008
Net influx of nights spent in Scotland (000s)	23,877	2,445	−4,699	−22,559
Adjusted Scottish population (000s)	4,376	4,317	4,297	4,249
Adjusted per adult alcohol consumption (baseline 11.8 l per adult in 2010)	11.7	11.8	11.9	12.0

Adjustments were calculated by converting the net influx of nights into person-years and assume that all trips were made by adults. Although per adult alcohol consumption in Scotland in 2010 was 11.8 L, because of rounding it is 0.2 L higher than the per adult estimate calculated using the adjusted population based on 1 night (11.7 L).

It is therefore highly improbable that the incoming visitor population is large enough to account for a large proportion of Scotland's per adult alcohol consumption. For example, in order to reduce the 2010 per adult alcohol consumption estimates for Scotland by 0.5 l, the adult population denominator would have to increase by 189,907 people. This is equivalent to a net influx of 69.32 million nights spent in Scotland or an excess of 9.90 million visitors spending 1 week in Scotland (assuming that visitors to Scotland consume the same volume of alcohol as Scottish residents). These calculations may overestimate the potential effect of tourism if Scottish residents consume more alcohol than visitors (World Health Organization, 2011).

Underestimation of the denominator population: students

Scotland is an importer of students. Data from the academic year 2009/2010 show a total of 71,660 students domiciled in other regions of the UK or outwith the UK studied at Scottish Higher Education Institutes (HEIs) and colleges. On the other hand, 12,340 students domiciled in Scotland studied in HEIs in the rest of the UK (Scottish Government, 2011). Unfortunately, data are not available for the numbers of Scottish domiciled students studying at colleges in other regions of the UK and are not held for the numbers of Scottish domiciled students studying higher education outside the UK.

The impact of the student population may be due to biases in the denominator population or because students might consume relatively high amounts of alcohol when they are resident in the area of their academic institution (Heather ). However, even assuming that Scotland imported an excess of 59,320 students in 2009/2010 (which takes no account of Scottish domiciled students studying outside Scotland at UK colleges or international institutions) and that they were all resident in Scotland for the entirety of the academic year (three 10-week terms), this would increase the Scottish adult population denominator only by 34,223 (whole time equivalent) residents. Adjusting the denominator by 34,223 people reduces the 2010 per adult alcohol consumption estimate in Scotland from 11.86 l of pure alcohol per year to 11.76 l of pure alcohol per year. This decreases further to 11.66 l if the additional student population is assumed to consume three times (arbitrary value) the Scottish average. Therefore, the impact of including non-Scottish domiciled residents in the denominator is minimal.

The imported student population would need to rise considerably, to an excess of 329,172 non-Scottish students staying for the whole of the three academic terms, to reduce Scotland's per adult alcohol consumption by 0.5 l. This would require 14 universities the size of the University of Edinburgh (Scotland's largest university) to be built and operating at full capacity.

Underestimation of the denominator population: consumption of alcohol by individuals under 16-years-old

The Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) provides self-reported estimates of alcohol consumption frequency for 13- and 15-year-old children (Black ). In the 2010 survey, 44% of 13-year-olds reported ever having had a ‘proper’ drink of alcohol (more than just a taste). Of these, 12% reported drinking at least weekly and a further 19% at least monthly. For 15-year-olds, 77% reported ever having a drink of alcohol with 27% of these drinking at least weekly and a further 33% at least monthly. If it is assumed that 14-year-olds drink at the mid-point frequency between 13- and 15-year-olds, then the proportion who have ever had a drink of alcohol would be 61% and of these 20% drink at least weekly and 26% at least monthly. To have any significant impact on Scotland's alcohol consumption, it is likely that these children would have to drink at least weekly. Applying these assumptions to mid-year population estimates gives an additional drinking population of 22,650 to be added to the population denominator. Given the relative infrequency of drinking in this group, this has a negligible impact on per adult estimates, reducing the 2010 estimate of 11.8 l only by 0.06 l.

Underestimation of the denominator population: summary

Overall, it is not thought that consumption of alcohol by visitors, students or children affects the validity of the alcohol sales data in estimating per adult alcohol consumption in Scotland because the additional number of individuals in each of these categories makes very little, if any, difference to the estimates. If, however, any of these groups substantially changes in size, consideration should be given to whether the validity of the sales estimates of per adult consumption is likely to be affected and the consequent impact on reliability over time. As such, evaluation of the effect of these sources of bias in other countries requires similar consideration.

Stockpiling of alcohol after purchase

The effect of stockpiling of alcohol on estimates of per adult consumption is thought to be small as this mainly occurs with the most expensive products in the market where less volume of alcohol is sold in Scotland (World Health Organization, 2000; Robinson ). Therefore, it is not expected that the validity of per adult estimates of alcohol consumption in Scotland has an important bias due to stockpiling.

It is possible that an anticipated change in price and/or availability of alcohol may lead to a temporary increase in stockpiling and then a temporary fall in sales as people consume what they have stockpiled. Such changes should be apparent in monthly alcohol retail sales data (if available). Thus, the longer term reliability of per adult alcohol consumption estimates from sales data should not be compromised by temporary variations in stockpiling. A longer term shift in volumes of alcohol stockpiles after the point of sale may be less apparent.

Wastage or spillage

Reported industry estimates are that the proportion of alcohol wasted or spilled is somewhere <10% of that sold (Rehm ). However, this estimate is not substantiated and the precision of the estimate has not been ascertained. This potential overestimation of alcohol consumption could bring the annual per adult consumption estimates for 2010 down to 10.7 l, a fall of 1.2 l per adult.

Wastage and spillage of alcohol occurring in the off-trade before being sold by retailers is not captured by retail sales data and is therefore not a source of bias. Wastage of alcohol by households in the UK has been estimated at 6% of all purchases (Department for Environment, Food and Rural Affairs, 2010). A US survey found 3% wastage of alcohol left after purchase via the on-trade though the applicability of this in other regions is not known (Kerr and Long, 2010). For on-trade estimates, Nielsen/CGA data captures alcohol volumes before purchase and no comparable UK estimate of wastage and spillage is available. However, both support the estimate of <10% wastage and spillage.

There is no indication that there would be a difference in the wastage/spillage estimate between Scotland and other countries or any obvious reason why the proportion of alcohol wasted or spilled would change over time. It is possible that wastage may reduce if alcohol was less freely available or was more expensive; however, this would assume that current wastage is easily avoidable.

Alcohol used in food

Some alcohol is used as an ingredient in food. Cooking processes variably reduce the alcohol content and generate a source of overestimation, while other cooking processes have no impact. The volume of alcohol used in food, the proportions of alcohol used in different cooking processes and the reduction in the percentage Alcohol by Volume (ABV) as a result of different cooking processes are unknown. Therefore, the use of alcohol in food is included as a minor unquantifiable overestimation of the volume of pure alcohol consumed per adult in Scotland. The overall effect of this on the validity of the estimates of per adult alcohol consumption is expected to be small and it is unlikely that the reliability would be greatly affected by changes in price (although a price increase may reduce the volume of alcohol used and decrease the already small source of overestimation).

Potential sources of bias leading to underestimation of per adult alcohol consumption

Alcohol consumed by Scottish residents while outside Scotland

There are no data available that quantify the amount of alcohol consumed by Scottish residents while they are outside Scotland. As discussed earlier, there is a small net excess of visitors coming into Scotland compared with Scottish residents visiting other countries. The assumption that visitors to Scotland and Scottish residents visiting other areas drink similar quantities may create a small overestimation of consumption. However, Scottish residents have higher rates of alcohol-related harm (inferring higher levels of consumption) than those of many countries, including England/Wales (England/Wales residents account for approximately two-thirds of all visitors to Scotland, see Table 2) (Leon and McCambridge, 2006). Thus, this is likely to be an unquantifiable underestimation of per adult alcohol consumption for Scotland. However, as highlighted in Table 2, it is unlikely that the validity or reliability will be affected by tourism unless the balance of visitors to Scotland and trips taken outside Scotland by residents of Scotland changes considerably.

Non-inclusion of some sales outlets

Retail sales data may not capture all alcohol purchased by the residents of a country. The impact of such sampling limitations on the validity and reliability of resultant consumption estimates must be considered.

In the example concerning Scotland, the retail sales data used to estimate per adult consumption do not include: certain internet sales, mail order, off-trade sales on military bases, sales at whisky distilleries, sales at certain music/entertainment festivals, duty-free sales and sales direct to the consumer via ‘cash and carry’ outlets. Assessment of the validity and reliability of retail sales data as an indicator of alcohol consumption must include consideration of the amount of alcohol these sampling limitations exclude.

Alcohol sales at military bases are not expected to affect the validity or reliability of estimates of Scotland's per adult alcohol consumption derived from retail sales data unless the number of military personnel located in Scotland dramatically increases. The number of military personnel located in Scotland was 12,190 in July 2010—only 0.3% of the estimated Scottish adult population (aged ≥16 years).

Approximate numbers of attendees at Scotland's five biggest music festivals were considered to determine the effect that this might have upon per adult alcohol sales estimates. If every attendee was over the age of 16 years, then to raise Scotland's per adult alcohol consumption estimate in 2010 by 0.5 l every attendee would need to consume 4.2 l of pure alcohol per day of attendance. This equates to 423 units of alcohol or 149 pints of 5% ABV beer per person per day of festival attendance (in addition to any alcohol purchased before entry as alcohol bought in the off-trade before entering a festival for consumption at the festival will be captured in the alcohol retail sales data). Thus, this is highly unlikely to be a source of bias that would have a large impact on the validity or reliability of the sales data estimates of per adult alcohol consumption in Scotland.

The same issues apply to temporary sporting events. Large-scale events such as the Ryder Cup in 2014 will increase visitor numbers to Scotland, as well as the volume of alcohol sold to Scottish residents through outlets not included in the sampling frame. Such events need to be considered individually in order to assess their potential impact on the reliability of the estimates over time should the numbers of attendees be sufficiently large.

The retail sales data used in Scotland do not include sales by mail order and certain internet operators. Supermarket online sales are captured where they are part of online grocery shopping, as these orders are packed and scanned at a local store. The degree of underestimation of per adult alcohol consumption in Scotland through mail order and internet operators is currently unquantified and could be important. It is possible that sales through this route could be sensitive to an increase in price or a decrease in availability of alcohol in Scotland, particularly if the operator is based outside Scotland. A potential means of exploring this further would be the use of other market research data that stipulate the location of alcohol purchases, although these data are often based on self-reporting or self-scanning of purchases.

Unrecorded alcohol

Adult per capita consumption includes both recorded and unrecorded alcohol consumption (Rehm ). Unrecorded alcohol is defined by the WHO as alcohol that is not taxed and is outside the usual system of governmental control (World Health Organization, 2011). Thus, legally home-brewed alcohol, illicitly produced alcohol, substitute alcohol (alcohol not intended for human consumption), alcohol produced legally but illegally redirected back into the country instead of sale abroad, and cross-border purchase will not be included in alcohol sales data. An inevitable limitation of using retail sales data to estimate consumption is the lack of data on unrecorded alcohol. This will lead to underestimation of per adult alcohol consumption in a region, the extent and variability of which is unknown. Global level data suggest that increasing levels of unrecorded alcohol consumption are associated with increasing recorded alcohol consumption, but the percentage share of alcohol consumed from unrecorded sources decreases as the total amount of alcohol consumed increases (World Health Organization, 2011). If Scotland follows this pattern, the proportion of alcohol consumed from unrecorded sources will be low relative to the volume of alcohol consumed from recorded sources. However, there may still be a significant volume of alcohol consumed from unrecorded sources and there are some areas, such as Eastern Europe, where both total volumes of alcohol consumed and volumes of alcohol from unrecorded sources are a significant problem (World Health Organization, 2011).

Estimates of the extent of unrecorded alcohol consumption generally rely upon expert judgements and therefore may be subject to considerable error (Rehm ). Using such methods, the WHO estimates that UK unrecorded alcohol consumption is around 1.7 l per capita (aged 15+ years) (World Health Organization, 2011).

Unrecorded alcohol: cross-border purchase

The term cross-border purchase is used here to refer to alcohol that has been produced legally in a different jurisdiction and then legally imported into the country where it is consumed. This may be through cross-border travel or by internet or mail order shopping. Research in Europe shows that considerable inaccuracies in per capita estimates of alcohol consumption in some regions occur because of the cross-border purchase of alcoholic beverages. For example, estimates of the proportion of per capita consumption that was unrecorded in Sweden, following specific changes to travel allowances and consequently to unrecorded imports, ranged up to 30% (Rehm ).

In the UK, Her Majesty's Revenue and Customs (HMRC) estimates ‘tax gaps’ (unpaid duty and VAT) resulting from cross-border purchase of spirits and beer (Her Majesty's Revenue and Customs, 2011). In 2008/2009, it was estimated that cross-border trade accounted for 3% of recorded spirits volume sales, while beer is unlikely to be affected (0% upper estimate). Application of this to the estimation of 2010 per adult spirits consumption in Scotland (3.4 l per adult) would increase the estimate by 0.1 l. However, it should be noted that these estimates should be used to consider trends over the long term rather than giving precise year-on-year estimates.

The British Beer and Pub Association (BBPA) also estimates personal imports as a percentage of UK consumption. Their estimates were lower than the HMRC estimates but are based on different estimates of overall consumption and exclude estimates of large-scale smuggling (Sheen, 2011). The HMRC does not publish equivalent data for wine. However, to increase per adult wine consumption in Scotland by 0.5 l, cross-border imports of wine would have to account for 15% of recorded sales. From these estimates, the extent of such imports seems highly unlikely to be as high in Scotland as in Sweden. The Swedish example does, however, highlight the importance of considering such possibilities.

Bias due to cross-border purchases is likely to be sensitive to changes in relative price or availability between jurisdictions. Within Great Britain, the average price of alcohol sold in Scotland is very similar to England/Wales (Robinson ). Thus, at present, this is unlikely to be an important source of bias impacting on the validity of the sales data. Nonetheless, the reliability of the data requires careful monitoring if relative price or availability changes.

Illegal sources of alcoholic beverages (illegal imports, illegal manufacture, undeclared release of alcohol for sale)

Illegal sources of alcoholic beverages are a potential source of underestimation of population alcohol consumption. It is unlikely that counterfeit alcohol will be included in alcohol retail sales data (Thorpe ). Thus, it is not known how much illegal alcohol is sold in Scotland. Recent estimates from the HMRC suggest that the illicit market in spirits is in the region of 3% (0–11%) of the spirits consumed in the UK and an upper bound of 14% for beer (Her Majesty's Revenue and Customs, 2011). There are no equivalent data published for wine.

Substitute alcohols (alcohol not intended for human consumption)

The use of substitute alcohol is not formally captured by any robust data source in Scotland. This is, therefore, another unquantified potential source of underestimation of per adult alcohol consumption. There is a dearth of systematic research on the health impacts of substitute alcohol. It is increasingly recognized that the harmful effects of these substances can contribute to the harmful effects caused by alcohol consumption (Leon ). The evidence base for the harmful effects caused by substitute alcohol is, however, less developed than that of legally produced alcohols intended for human consumption (Rehm ).

Substitute alcohol is not thought to be a significant source of bias in per adult consumption estimates. A cross-sectional survey of 377 patients with ‘serious alcohol problems’ was conducted in Edinburgh in 2008/2009 and enquired about the most recent week of drinking (Black ). Only one patient reported consuming very small amounts of substitute alcohol in the form of perfume and none reported illicit purchase or consumption of illicitly produced alcohol. This may, however, be a larger source of bias in other countries (McKee ).

Home-brewed alcohol

Formal quantification of the amount of home-brewed alcohol in Scotland or the UK is not available. In Canada, consumption of home-brewed alcohol proved difficult to quantify with any degree of accuracy because of widely disparate results from independent methods (MacDonald ). In Australia, recent recall methods have enabled quantification of home-made wine and beer (Stockwell ). However, as highlighted earlier, estimates based on self-reports may not be accurate because of biases such as under-reporting and low response rate. Nonetheless, the likely small scale of home-brewed alcohol in Scotland is unlikely to challenge the validity or reliability of per adult consumption estimates derived from alcohol retail sales data.

Impact of unrecorded alcohol on the validity and reliability of alcohol sales data estimates of per adult alcohol consumption in Scotland

It is expected that unrecorded alcohol, including cross-border purchase and illegal alcohol, accounts for a large underestimation of per adult alcohol consumption in Scotland, while consumption of home-brewed alcohol and substitute alcohol has a negligible effect. The WHO estimate of 1.7 l per adult in the UK compares relatively well with the underestimation calculated from HMRC estimates of cross-border purchases and illicit consumption of beer and spirits if it is assumed that volumes of wine in these categories are similar to those of spirits (1.8 l per adult). Unrecorded alcohol use could be altered by changes in attitudes and social norms or by changes in the availability of support services. Increased price or decreased alcohol availability could also make unrecorded alcohol more attractive to some drinkers.

Sources of possible bias where the direction of effect is unclear

Representativeness of the sample frame

Except for alcohol sales through outlets not included in the sampling frame, the methods used by Nielsen/CGA ensure a representative estimate of alcohol sales through on- and off-trade premises in Scotland and England/Wales (for full details see Thorpe ).

Comparison of different sources of alcohol sales data

Comparison of estimates of alcohol sales based on retail data with other sources of data enables an indirect assessment of concurrent validity (World Health Organization, 2000). Although each data source may be liable to the same biases and other sources of under- and overestimation, similar population levels and trends derived from different sources provide some reassurance that the data are representative and measure what they purport to measure. The most appropriate way to compare estimates is using natural alcohol volumes; conversion to pure alcohol volumes requires multiplication by an estimated strength (i.e. %ABV), and the strengths used by the various data producers are likely to differ slightly because of different methodological approaches.

Figure 1 shows the total estimated volume of alcohol sold (litres) in the UK based on taxation data (Sheen, 2011) and in GB based on retail sales data. As expected, HMRC estimates are consistently higher than retail sales estimates because of the inclusion of data pertaining to alcohol sales in Northern Ireland and alcohol sold through certain sales outlets not captured by Nielsen/CGA. Nonetheless, the trend over time between sources is similar, retail sales estimates accounting for ∼90% of HMRC estimates at each time point. This consistency over time is reflected by the strong correlation (r = 0.93) and acceptable statistical agreement (for Bland–Altman plots see Thorpe ) between measures. Furthermore, by applying the volume of alcohol sold per adult in Scotland to the adult population of Northern Ireland, it can be crudely estimated that the Nielsen/CGA retail sales estimates account for ∼95% of HMRC estimates for the UK as a whole (94% if data for England/Wales are applied to the Northern Ireland population).

Fig. 1.

Estimates of alcohol sales (litres) in the UK (HMRC taxation data) and GB (Nielsen/CGA retail sales data), 1994–2010.

The similarity in sales volumes and trends is also evident at drink category level (Thorpe ). Over the time period analysed, retail sales estimates of spirits accounted for, on average, 87% of HMRC clearances (with high correlation between the annual estimates r = 0.95). Volume sales of wine based on Nielsen/CGA data accounted for the lowest average percentage of comparative HMRC estimates (mean = 82%) suggesting that this drink category was subject to the greatest underestimation, but the trend over time was very similar (r = 0.98). Estimates of beer sales at UK level are also available from the BBPA, who collect invoiced sales from their members. On average, Nielsen/CGA accounted for 91% of HMRC clearances (r = 0.96) and 92% of BBPA sales estimates (r = 0.98). Finally, estimates of the volume of cider/perry on retail sales data accounted for an average of 84% of estimates derived from taxation data. The National Association of Cider Makers (NACM) also publishes data on the volume of cider/perry sold in the UK, derived from invoiced sales. Although Nielsen/CGA estimates are for GB, they were, on average, 4% higher than those based on NACM data. However, changes over time in estimates of cider/perry sales were similar across all three data sources (Nielsen/CGA vs HMRC, r = 0.98; Nielsen/CGA vs NACM, r = 0.89).

Thus, although different methodologies are used to collect data to estimate alcohol sales, the similar estimates presented here provide reassurance that the retail sales data are valid and that their use in monitoring trends in alcohol consumption in Scotland and England/Wales is justified.

Measurement error: sampling variation

Any use of a sample statistic to estimate a population parameter is subject to some degree of sampling variation. The methods used to estimate consumption based on retail sales data involves outlet sampling (Thorpe ). It is therefore necessary to understand the degree of uncertainty around the estimate in any given year. Such information should be available from the data providers. The estimated variation around on- and off-trade alcohol sales estimates in Scotland is ∼4%, while for England/Wales it is around 1%. The impact of this variation on per adult consumption estimates is shown in Table 3.

Table 3.

95% Confidence intervals (CI) around annual estimates of pure alcohol sales in Scotland and England/Wales, by trade sector, 2010

	Scotland			England/Wales
	On-trade	Off-trade	Combined	On-trade	Off-trade	Combined
1000 l
Estimate	16,957	34,113	51,070	148,728	284,272	433,001
Lower 95% CI	16,224	32,763	48,987	146,728	281,159	427,887
Upper 95% CI	17,691	35,463	53,154	150,728	287,386	438,114
L per adult (≥16 years)
Estimate	3.93	7.91	11.85	3.31	6.33	9.64
Lower 95% CI	3.76	7.60	11.37	3.27	6.26	9.52
Upper 95% CI	4.10	8.23	12.33	3.36	6.40	9.75

Quantification of the percentage ABV of different beverages

The potential for under- or over-recording of per capita consumption due to inaccuracies in the recording of the strength of different beverages has been highlighted in a number of studies. For example, an underestimation of per adult consumption in Australia has been demonstrated (Chikritzhs ). This resulted from the increasing alcohol content of wine alongside a general increase in wine consumption. UK taxation data escape this bias. The HMRC reviewed their estimates of the average strength of table wine in 2008, using new quality-assured HMRC trade data on the country of origin of wines to estimate average strengths (Ambler, 2008). Owing to the complexity of the wine market and the number of different types sold, the retail sales data providers have applied a standard ABV, chosen specifically to represent the mean ABV of all wine sold in Great Britain based on expert market knowledge. If the ABV of wine is assumed to have increased over time in accordance with the findings from HMRC, the impact on retail sales estimates of the volume of pure alcohol sold as wine per adult in Scotland and England/Wales is small (Fig. 2).

Fig. 2.

Impact of applying a consistent ABV vs an increasing ABV on estimates of the volume of pure alcohol sold as wine per adult in Scotland and England/Wales (E&W), 1994–2010.

Thus, although changes in the ABV of different alcoholic beverages have been highlighted as a concern by various researchers, it is unlikely to be an important source of bias for these sales data. Per adult alcohol consumption estimates in other countries outside the UK using different sales data methodologies could be affected by this source of bias.

Summary of the validity and reliability of alcohol retail sales data in Scotland

Table 4 summarizes the potential sources of bias in using alcohol retail sales data as a means of estimating per adult alcohol consumption in Scotland in 2010. The largest potential sources of bias are underestimation due to unrecorded alcohol [1.7 l per adult (15+ population)] and overestimation due to wastage/spillage (estimated at <1.2 l per adult). The uncertainty in the estimate of per adult sales related to sampling variability (i.e. random error) is estimated at ±0.5 l. This uncertainty relates to the calculation of sales in a single year and is much less when considering trends over time. The lack of significant year-on-year variability and the similarity between point estimates and trends derived from retail sales data compared with other data sources emphasize the precision of the estimates.

Table 4.

Potential sources of bias and uncertainty in using retail sales data to estimate per adult alcohol consumption in Scotland and their estimated magnitude (based on per adult alcohol consumption estimate in 2010)

Sources of bias in estimation of per adult alcohol consumption	Litres per adult [range around the 2010 estimate]		Comments
	Underestimation of consumption	Overestimation of consumption
Student population	0	≤0.1	Overestimation is likely to be even smaller because there are no data on the numbers of Scottish residents studying internationally or at colleges in England/Wales
Net effect of visitors coming into Scotland and Scottish residents making trips elsewhere	0.2	0.2	There are no data on the average number of nights spent by Scottish residents on international visits. Underestimation assumes Scottish residents spend an average of 14 nights on international visits; the resultant total estimated consumption is more than that of visitors to Scotland. Overestimation assumes Scottish residents spend an average of 1 night on international visits; the resultant total estimated consumption is less than that of visitors to Scotland
Stockpiling of alcohol	Unknown	Unknown	Only likely to impact on time trends rather than differences between Scotland and England/Wales. Impact should be apparent on monthly sales data
Wastage/spillage	0	<1.2	Based on industry estimate of <10%
Sampling variation	0.5	0.5
Non-inclusion of some outlets	Unknown	0
Unrecorded alcohol	1.7	0	Based on estimate from the World Health Organization for UK population aged ≥15 years
Total of known estimates	1.7–2.4	0–2.0
Net estimate	−2.4 (underestimate) to 0.3 (overestimate)		Excludes the impact of stockpiling and alcohol sold through non-included outlets

Overall, considering all the possible sources of overestimation and underestimation, and taking into account the potential for sampling variability to impact on the results, the range of uncertainty in the per adult alcohol sales estimates in 2010 was from an overestimate of 0.3 l to an underestimate of 2.4 l. This excludes the impacts of stockpiling (which are likely to be minor) and alcohol sold through non-included outlets. On balance, there is therefore far greater scope for the retail sales data to be an underestimate of per adult alcohol consumption than an overestimate.

DISCUSSION

The use of retail sales data is advocated as the best means for monitoring population levels of alcohol consumption. Understanding the strengths and weaknesses of using such data and their potential impact on consumption estimates is crucial to ensuring that any changes over time, or between countries, are not related to limitations in the method of measurement (World Health Organization, 2000). We have identified a wide range of sources of bias that may lead to either under- or overestimation of per adult consumption, using an illustrative case study of alcohol consumption in Scotland. We conclude that alcohol retail sales data offer a robust source of data for measuring and monitoring alcohol consumption, although the method is still likely to underestimate true levels of alcohol consumption.

The validity of per capita alcohol consumption estimates derived from sales data is, however, subject to change. Temporal trends in biases such as illicit alcohol and cross-border purchases, which may arise from policy changes, have the potential to impact on trends in population consumption estimates (MacDonald ). This, in turn, could hinder robust evaluation of alcohol-related policies that rely on per capita consumption as an indicator to assess impact. Although quantifying the scale of, and trends in, sources of under- and overestimation is challenging, it is essential that data collection systems are in place to provide best estimates. Furthermore, as recommended by the WHO and demonstrated in this paper, multiple data sources should be utilized to estimate both consumption and bias (World Health Organization, 2000).

Alcohol is one of the leading risk factors for death and disease globally and is a leading risk factor for non-communicable diseases (NCDs) (World Health Organization, 2011). In a recent WHO consultation regarding a monitoring framework and targets for the prevention and control of NCDs, reducing per adult alcohol consumption was removed as a target. This was because some Member States expressed concern that this indicator was not a valid proxy for harmful alcohol consumption and that obtaining an accurate measure was difficult given the high level of consumption of duty-free/stored/home-brewed/black market alcohol (World Health Organization, 2012). This highlights the importance of considering the impact of biases and methodological limitations on the accuracy of consumption estimates derived from sales data. Without such consideration, the robustness of an indicator can be misunderstood and its crucial importance in global public health surveillance challenged. We have used Scotland for our case study but the approach used is relevant to other countries using retail sales data to monitor consumption and evaluate alcohol-related policies.

CONCLUSIONS

Alcohol retail sales data offer a robust source of data for monitoring per adult alcohol consumption in Scotland. Consideration of country-specific sources of bias and a comprehensive understanding of data collection methods are essential for using sales data to monitor trends in alcohol consumption.

Authors' contributions

R.T. and M.R. analysed the data and co-wrote the first draft of the manuscript. All the authors commented on and contributed to further drafts, and read and approved the final manuscript.

Ethics Committee approval

Ethics committee approval was not required for this study.

Funding

Purchase of the sales data was funded by the Scottish Government as part of the wider Monitoring and Evaluating Scotland's Alcohol Strategy portfolio of studies. Funding to pay the Open Access publication charges for this article was provided by NHS Health Scotland. Conflict of interest statement. All alcohol sales data are copyrighted to The Nielsen Company and/or CGA Strategy. All authors are currently, or have recently been, involved in the evaluation of the Scottish Government's alcohol strategy. The views expressed in this paper are those of the authors and do not necessarily represent those of the Scottish Government or the author's employing organizations.