The evidence underpinning sports performance products: a systematic assessment.

BACKGROUND: To assess the extent and nature of claims regarding improved sports performance made by advertisers for a broad range of sports-related products, and the quality of the evidence on which these claims are based.
METHODS: The authors analysed magazine adverts and associated websites of a broad range of sports products. The authors searched for references supporting the performance and/or recovery claims of these products. The authors critically appraised the methods in the retrieved references by assessing the level of evidence and the risk of bias. The authors also collected information on the included participants, adverse events, study limitations, the primary outcome of interest and whether the intervention had been retested.
RESULTS: The authors viewed 1035 web pages and identified 431 performance-enhancing claims for 104 different products. The authors found 146 references that underpinned these claims. More than half (52.8%) of the websites that made performance claims did not provide any references, and the authors were unable to perform critical appraisal for approximately half (72/146) of the identified references. None of the references referred to systematic reviews (level 1 evidence). Of the critically appraised studies, 84% were judged to be at high risk of bias. Randomisation was used in just over half of the studies (58.1%), allocation concealment was only clear in five (6.8%) studies; and blinding of the investigators, outcome assessors or participants was only clearly reported as used in 20 (27.0%) studies. Only three of the 74 (2.7%) studies were judged to be of high quality and at low risk of bias.
CONCLUSIONS: The current evidence is not of sufficient quality to inform the public about the benefits and harms of sports products. There is a need to improve the quality and reporting of research, a move towards using systematic review evidence to inform decisions.

Introduction

Exercise is important for improving overall health across a variety of conditions.1 The promotion of exercise is therefore an important public health priority, particularly for the ‘economically and socially disadvantaged’.2

Currently, the public are faced with a large number of adverts that make claims about enhanced performance and recovery for a wide range of products, including drinks, supplements, clothing and footwear. Regulators require that marketing communications containing health claims must be supported by documentary evidence and ‘must not mislead consumers by exaggerating the capability or performance of a product’.3 In spite of this, some adverts for sports drinks have previously been shown to mislead the public into incorrectly concluding that the drinks contained no carbohydrates or additives.4 In addition, while some supplements have been shown to potentially improve performance, many have no proven benefits and may cause serious side effects.5 The marketing of sports products has become a multibillion-dollar industry,6 and the consumption of the so-called energy drinks is increasing year on year,7 but research in this area has previously been labelled as methodologically poor.8

The current confusion as to which products are actually beneficial for sports performance is highlighted by the European Food Safety Authority decision to approve certain products, such as carbohydrate–electrolyte drinks to enhance water absorption during exercising and maintain endurance performance, while not approving a variety of other products; including L-carnitine, glutamine or typrosine, which claim to aid muscle recovery.9 We therefore aimed to assess the extent and nature of claims regarding improved sports performance made by advertisers for a broad range of sports-related products, and the quality of the evidence on which these claims are based.

Methods

In order to obtain a representative sample of adverts applicable to the general population, we searched the top 100 general magazines and the top 10 sport and fitness magazines in the UK and the USA for the month of March 2012 according to the Magazine Audit Bureau of Circulations. This selection of magazines is distributed to over 30 million customers in the UK alone. We excluded magazines specifically aimed at body building. One reviewer (RD) examined each page of included magazines to identify adverts. All adverts were then assessed by second reviewers (AW, CH, MT and RD) as either relevant to sports or not. A third round of reviews (CH and RD) assessed adverts that included specific performance-enhancing claims.

Inclusion and exclusion criteria

We included oral sports drinks, oral supplements, footwear and clothing or devices (such as wristbands). To be included, adverts had to make a claim related to sports performance (defined as improvement in strength, speed, endurance, etc) or enhanced recovery related to sports (eg, reduced muscle fatigue). We excluded adverts related to purely weight loss, skin or beauty products, sports equipment (eg, bicycles) and classified adverts. We therefore only included adverts from the actual manufacturer of products rather than suppliers.

We then analysed the websites of any products making enhanced performance or recovery claims. A data extraction template (MS Word) was used to extract data from each web page, and five reviewers (BON, CH, DSL, MT and PJG) inserted page number, url and screen shots of all web pages viewed with the associated claims. To reduce errors, we directly cut and paste any claims and searched the web pages for any references related to these claims. We compiled a database of all retrieved references and then two reviewers (AS and GJ) emailed all manufacturers with the claims and the associated references asking them (1) to confirm whether our list of claims and retrieved references was complete; (2) whether other data existed to support the claims; (3) if additional data were published, could they provide us with the relevant references and (4) if the research was unpublished, could they supply us with a copy of the report.

Data extraction

We extracted the following data (from both the magazine and the websites) of included sports products into Microsoft Excel: product category (ie, sports drinks, supplements, footwear, clothing or devices); website; number of pages viewed; number and type of enhanced performance claim(s); references cited for the claims; qualifiers related to the claim (eg, such as ‘should be used in conjunction with a healthy diet and training programme’) and whether the product was endorsed/backed by a sports person or team. One reviewer (JH) acted as custodian of the data and checked all entries for consistency.

Quality assessment

We obtained full-text copies of all cited references and assessed them using the Centre for Evidence-Based Medicine (CEBM) Levels of Evidence.10 For treatment benefits, the highest level of evidence for claim is a systematic review of randomised controlled trials or an N-of-1 trial (level 1) followed by randomised trials (level 2) and non-randomised studies (level 3). The lowest level of evidence is mechanistic reasoning, which includes expert opinion and animal studies (level 5).

We assessed whether a study was appropriate for critical appraisal (recording the reasons if it was not appropriate). Six reviewers (BON, CH, DSL, JH, MT and PJG) then recorded the presence or absence of the following elements of critical appraisal: a clear hypothesis, control group, power calculation, randomisation, allocation concealment, intention to treat, blinding (investigator and/or subjects) and sports outcome (subjective or objective) that demonstrates improved performance or recovery. Extracted data were checked independently by a second reviewer. One reviewer (CH) then assessed included studies using the Cochrane method for risk of bias, assessing studies as high, unclear or at low risk of bias, which was checked by a second reviewer (JH).11 Discrepancies were resolved by consultation with other reviewers.

We also collected information on the participants involved in the included trials (categorised as ‘regular people’ who do not exercise or compete seriously in sport; amateur athletes including ‘regular people’ who exercise seriously and sports professionals); adverse events; whether study limitations were discussed; the primary outcome of interest and whether the intervention had been retested in a subsequent trial or test group.

We summarised data by raw counts and continuous data with medians and ranges, and for dichotomous data, we presented percentage and associated 95% CIs. We analysed data using Microsoft Excel.

Results

We examined 92 magazines containing 1807 adverts, of which 615 (34%) advertised sports products (figure 1). After excluding 380 adverts, which were not product specific (ie, individual shop adverts), we included the remaining 235 advertised sports products in the analysis. From these, 54 (23%) different products made 113 enhanced performance or recovery claims. Of these, we found only three (2.7%) references for one product (ACCELERADE) to back up these claims, which were appropriate for critical appraisal, and 22 (42%) products that were endorsed by athletes. Six (12%) products made direct comparisons with other products in their advertised claims and three provided disclaimers. All the latter were US-based products, and cited the US Food and Drug Administration (FDA) in disclaimers: “these statements have not been evaluated by the FDA. This product is not intended to diagnose treat, or cure disease.”

Figure 1

Flow chart of references found for magazine and web advertisements.

We then assessed products' websites for claims (one product on reassessment was designated a dietary product) and viewed a total of 1035 web pages (web appendix 1). From these, we identified 431 (median 7, range 0–65) performance-enhancing claims for 104 different products, and a total of 146 references (range 0–46) associated with these claims (figure 1). More than half (52.8%) of the sites that made claims did not provide any references. One site (http://www.poweradegb.com/) provided approximately one third (46) of the references found, of which 24 (52%) were appropriate for critical appraisal.

We contacted 42 companies and received responses from 16, of which two were unwilling to share their research (Panache and New Balance), one provided a video of the product in use and said that this was ‘sufficient’ (Nike), one pointed to the work of one researcher but did not answer whether the company had any research on its actual product (Merrell), one responded that they would get back but did not, one declined due to staff absence and one directed us back to their website (web appendix 1). In total, we received additional referenced material from nine companies; obtaining two published,12 13 one in press14 and two unpublished studies that we included in the analysis (Effect of a electrolyte replacement beverage compared with a commercially available carbohydrate supplement on the rate of fat oxidation during moderate-intensity cycle ergometry exercise, 2010; Summary of the study on the influence from compression sleeves worn during short-time intensive effort on lactatemia). We also received four bibliographies: one of these was a comprehensive bibliography of Lucozade-associated research (web appendix 2), which arrived outside the time lock, and due to its size, we analysed separately in an associated article.14a

We were unable to perform critical appraisal for approximately half (72) of the references identified (figure 1). Of note, five references could not be identified despite extensive searching involving an information specialist, and eight were animal studies15–22 including a comparative study of different diets on rat metabolism published in 1930.19 None of the 74 studies, which were critically appraised, were systematic reviews (level 1 evidence), and approximately half of the studies were categorised as level 3 evidence (non-randomised studies). As a result, 84% of the critically appraised studies were judged to be at high risk of bias. The presence of this level of bias means that the conclusions are likely to change based on future (high-quality) research.11

Table 1 shows that in the 74 studies, the total number of participants was 2031 (median 15): two thirds (1310 65%) were men. Two studies provided a quarter of the participants (n=505).23 24 Excluding these two, the average number of participants per study was 16 (range 5–69). Nearly half (48.6%) were classified as ‘regular people’ who exercise and 39.2% as endurance/serious athletes and 10.8% professional sports people (in one study, it was unclear who the participants were). Nearly three times (423:146) as many sportspersons or teams endorsed products than evidence was made available.

Table 1

Sports adverts study quality

Study component	N=74	% (95% CI)
Number of participants	2031 (median 15)	Range 5–387
Number of men	1310	64.5 (61.9 to 67.1)
Study quality
Control group	55	74.3 (62.8 to 85.9)
Randomisation	43	58.1 (43.4 to 72.9)
Allocation concealment	5	6.8 (0 to 28.8)
Intention to treat	22	29.7 (10.8 to 48.8)
Blinding (investigators, outcome assessors or participants)	20	27.0 (7.6 to 46.5)
Surrogate sports outcome	61	82.4 (72.9 to 92.0)
Repeat of the intervention	2	2.7 (0 to 25.2)
Reporting
Clear hypothesis	66	89.2 (81.7 to 96.7)
Power calculation	4	5.4 (0 to 27.6)
Adverse events reported	6	6.8 (0 to 32.3)
Study limitations discussed	8	10.8 (0 to 32.3)
Level of evidence
1	0	0
2	32	42.1 (25.0 to 59.2)
3	33	43.4 (26.5 to 60.3)
4 or 5	9	11.8 (0 to 33.0)

Randomisation was used in just over half of the studies (43/74, 58.1%); allocation concealment was only clear in five (6.8%) studies; and blinding of the investigators, outcome assessors or participants was only clearly reported as used in 20 (27.0%) studies. The majority of studies (83%, 95% CI 73% to 92%) used a surrogate outcome (rather than a direct outcome of sports performance or recovery) and only two studies (2.7%, 95% CI 0 to 25%) repeated the intervention in the study protocol.25 26 Overall, the majority of studies reported a clear hypothesis; but only four studies reported that they used a power calculation (5%, 95% CI 0 to 28%), and very few studies (11%, 95% CI 0 to 33%) discussed limitations of their studies.

We were unable to perform meta-analysis of individual outcomes across specific products due to the heterogeneity, poor reporting and the sheer number of outcomes reported across the studies.

Three of the 74 (4.1%) studies were judged to be of high quality and at low risk of bias.27–29 In the first of these, the methods of blinding by Berven et al27 were clearly reported: “capsules had the same size and appearance and were indistinguishable from the active capsules.” In addition, the study clearly reports intention to treat: “clinical and laboratory data were analysed in all included subjects (based on ‘intention to treat’). In addition, a per-protocol analysis was performed.” In the second study, Roffe et al29 clearly reported the randomisation procedure: “randomisation was performed in blocks of 10….The randomisation code was not known to the investigators who gave out the sachets. The code remained concealed from everyone except the pharmacist who prepared the sachets….” The third was one of the few studies to report a power calculation: “A priori power analysis revealed power values of 0.14, 0.71, and 0.99 for small (0.25), moderate (0.75), and large effect sizes (1.25), respectively, for the n size used in the study. These findings indicate that the n size used in the present study was sufficient to detect significant differences among groups.”28 Of note, all three of these studies reported no significant effects of the intervention.

Discussion

There is a striking lack of evidence to support the vast majority of sports-related products that make claims related to enhanced performance or recovery, including drinks, supplements and footwear. Half of all websites for these products provided no evidence for their claims, and of those that do, half of the evidence is not suitable for critical appraisal. No systematic reviews were found, and overall, the evidence base was judged to be at high risk of bias. Half of the trials were not randomised, and only 7% reported adequate allocation concealment. We found only three trials that were reported with sufficient details to be judged high quality and free from bias.

The absence of high-quality evidence is worrying. For instance, investigations have shown that in trials that did not use allocation concealment (compared with those that did) the effect estimates were 40% larger,30 and results fluctuate widely above and below the estimates.31 In terms of blinding, it is well known that “psychological effects could arise from participants' knowing that they have received a ‘promising’ new treatment”32; in terms of assessors not being blinded this also presents substantial room for bias: “outcome assessors with inclinations for or against any of the interventions being compared may make biased assessments.”32 The placebo effect of carbohydrate drinks, which has been shown previously, makes blinding especially important.33 Competitive endurance cyclists told that they were receiving a carbohydrate sports drink, when in fact it was water, performed 2% better than whey they were told the truth. In addition, in a study that tested the effect of carbohydrate ingestion in male trained volunteers, increased time to exhaustion was significantly improved when participants and researchers knew the capsule content, but not in the double-blind condition.34

Combining these problems with the fact no systematic reviews were found means that it is virtually impossible for the public to make informed choice about the benefits and harms of advertised sports products based on the available evidence. Yet, a simple search of PubMed (http://www.ncbi.nlm.nih.gov/pubmed/) reveals a number of systematic reviews that could be used to better inform the public: a meta-analysis by Vandenbogaerde, included 88 randomised crossover studies of carbohydrate supplements with or without protein before and/or during exercise provided 155 estimates for performance effects.35 Of concern is that this study reports a funnel plot, which shows ‘asymmetrical scatter is very likely the result of a publication trend towards positive effects’.35 Systematic review may come to conclusions that are different from those of individual studies. For instance, a systematic review of the effect of exercise-induced dehydration on time–trial performance concludes that relying in thirst sensation to gauge the need for fluid replacement maximises cycling time trial performance.36

We found that very few trials (2.7%) repeated the interventions under study conditions. In intervention trials retesting the intervention allows estimation of individual responses, takes account of regression to the mean and assesses the reliability of the effect measure.37 The lack of power calculations in studies is also concerning, the sample should be large enough to be able to detect a statistically significant effect; however, the exact size of the study to detect a meaningful effect was seemingly left to chance in most studies. Moreover, many studies used a surrogate outcome of performance or recovery, and undertook studies within laboratory settings, which limits the validity of the studies as “laboratory studies assessing the impact of certain interventions on athletic performance can produce results that have no relevance to the real athletic world.”38

Some limitations of the present study are worth discussing. We attempted to identify a representative sample of products, but it is possible the products we analysed are at the worst end of the spectrum. To avoid ‘cherry picking’, we undertook a search for a broad range of products. The number of adverts and the web pages we assessed required a number of reviewers for this task. We did not give the manufacturers much time to respond to requests for information, given more time a number may have provided more references. Our assessment of whether a claim was actually performance enhancing was subjective. Yet no manufacturer responded that any of the claims were incorrectly identified for their products. We also did not investigate heterogeneity of effects or publication bias as the number of outcomes and the substantial variation in these outcomes means that it was not possible to combine or undertake such analyses.

We therefore conclude that the current evidence is not of sufficient quality to inform the public about the benefits and harms of sports products. There is a need to improve the quality of the research conducted in this area and its reporting, and a move towards using systematic review evidence across the board for decision-making.