Do adverts increase the probability of finding online cognitive behavioural therapy for depression? Cross-sectional study.

OBJECTIVE: To estimate the effect of online adverts on the probability of finding online cognitive behavioural therapy (CBT) for depression.
DESIGN: Exploratory online cross-sectional study of search experience of people in the UK with depression in 2011. (1) The authors identified the search terms over 6 months entered by users who subsequently clicked on the advert for online help for depression. (2) A panel of volunteers across the UK recorded websites presented by normal Google search for the term 'depression'. (iii) The authors examined these websites to estimate probabilities of knowledgeable and naive internet users finding online CBT and the improved probability by addition of a Google advert. PARTICIPANTS: (1) 3868 internet users entering search terms related to depression into Google. (2) Panel, recruited online, of 12 UK participants with an interest in depression. MAIN OUTCOME MEASURES: Probability of finding online CBT for depression with/without an advert.
RESULTS: The 3868 users entered 1748 different search terms but the single keyword 'depression' resulted in two-thirds of the presentations of, and over half the 'clicks' on, the advert. In total, 14 different websites were presented to our panel in the first page of Google results for 'depression'. Four of the 14 websites had links enabling access to online CBT in three clicks for knowledgeable users. Extending this approach to the 10 most frequent search terms, the authors estimated probabilities of finding online CBT as 0.29 for knowledgeable users and 0.006 for naive users, making it unlikely CBT would be found. Adding adverts that linked directly to online CBT increased the probabilities to 0.31 (knowledgeable) and 0.02 (naive).
CONCLUSIONS: In this case, online CBT was not easy to find and online adverts substantially increased the chance for naive users. Others could use this approach to explore additional impact before committing to long-term Google AdWords advertising budgets. TRIAL REGISTRATION: This exploratory case study was a substudy within a cluster randomised trial, registered on http://www.clinicaltrials.gov (reference: NCT01469689). (The trial will be reported subsequently).

Background

Less than 60% of people with diagnosable depression or anxiety seek formal help from practitioners; this represents a significant treatment gap.1 The remainder may access informal care and support, alternative therapies, private arrangements such as counselling, use of the voluntary sector and the internet and other sources of information.

Nearly one in five British internet users search for information related to mental health.2 Among younger people, this may be much higher. Reasons cited included anonymity, the amount of information available and easy access.3–5 The level of searching for information on depression online has increased. In 2000, depression was not in the 10 most frequently searched for topics, but by 2005, it was the fifth most common topic.5

Others have investigated quality of health information websites, including those for depression,6 and attempted to rate the quality of these websites.7–9 Patients searching for health information online may be unable to find what they are looking for,10 possibly due to suboptimal search strategies.11 Most people follow-up between the first three and 10 results.3 12 We have not found studies of how users seek online help for depression but most people searching for health information start with a search engine.10 Google's share of the search market is debatable13 14 but we can assume that a majority in the UK searching for health information will use Google. Search results may now be personalised according to user's location15 and previous searches.16 17 Some regard this personalisation as intrusive and this has led to the development of software to hide user characteristics.18 Search results will also change over time because of adjustment of Google's databases to the changing pattern of websites. This means that there is no standard response to a Google search; however, we can ‘average’ the experience of several users to explore ease of finding given topics at any time.

Although not everyone searching online using terms such as depression is depressed, we need to make sure that the best support is available for those who are. Online cognitive behavioural therapy (CBT) is self-help treatment recommended by the National Institute for Health and Clinical Excellence (NICE) in the UK for mild to moderate depression.19 The range of resources includes licenced (paid for) sites such as Beating the Blues20 and free access websites providing access to CBT life skills resources (eg, Living Life to the Full (LLTTF)21 and MoodGYM).22

Given the NICE recommendation and the predominance of Google, it would be reasonable to expect websites delivering access to online CBT to have relatively high profiles and be easily visible to users on a Google search. We were struck by the large geographical variation within the UK in the uptake of LLTTF. The highest rate (in Kirkwall, Scotland) was 15 times that of the lowest (Wigan, England). Such variation could not be explained by variation in prevalence of depression and was most likely explained by lack of awareness of online therapies for depression. We have been exploring ways of increasing awareness.23 One method of doing this was to use online advertising, in particular, Google AdWords.24 25

Those paying for an AdWords campaign set up one or more adverts (see figure 1 for example) and enter keywords to help determine when the advert is shown. AdWords determine if an advert is relevant to a given user's search based on (in AdWords terminology) exact matches, broad matches and phrase matches with the user's entered search term.26 AdWords displays adverts as a sponsored link, either at the top of the list of search results or in the right-hand panel, depending on the phrase entered, the price you offer per advert and bids from competing adverts, and (if requested) by estimated location of the user. Google AdWords may be embedded in searches in other websites (the Google Display Network).27

Figure 1

Google advert.

However, diverting users who would have found CCBT sites without the aid of adverts, to finding it via an advert, is a waste of resources. In this case study, we reviewed the likely ‘search experience’ using Google of people searching using the term ‘depression’ to estimate the chance of finding online CBT and the improved probability by adding an online advert.

Methods

Ethics and trial registration

This was an exploratory substudy within a pilot cluster randomised trial. The whole study was approved by the Plymouth NHS Research Ethics Committee (reference. 11/H0203/8), is registered on http://www.ClinicalTrials.gov (reference: NCT01469689) and will be reported subsequently.

Setting and population

We reviewed search terms entered by 3868 internet users in the UK who used the Google search engine for terms related to depression and who clicked our presented Google advert (figure 1) between 17 April 2011 and 31 August 2011.

We enabled 126 keywords in our AdWords campaign and allowed exact matches, broad matches and phrase matches. Those clicking on the advert were taken to a website which asked them to confirm their location (postcode area) and to complete a widely used depression rating questionnaire (PHQ928) and then offered links to four sites addressing low mood in different ways. Our main objective was to offer a link to online CBT, in particular, LLTTF and MoodGYM.29 The NHS research ethics committee that considered our proposal asked us to add a link to Samaritans.30 (Samaritans is a secular UK charity aimed at providing emotional support to anyone in emotional distress or at risk of suicide mainly through their 24/7 confidential telephone helpline.) To keep symmetry in the exit page of the website, we added NHS Choices (non-CBT information about depression).31 The order of links presented on the page was randomised within each row, the top row always being the two CBT sites and the bottom row Samaritans and NHS Choices.

Search terms used

We reviewed the presentation of our advert and the search terms entered by users (who subsequently clicked on our advert) to understand better what terms people may use and how many may search directly for online CBT.

Exploring the results of normal Google search

Given that the results of Google searches are to some degree personalised and therefore will vary by computer used, to assess which websites would be found by users, we recruited a user panel of 12 people. We contacted a sample of previous users of LLTTF and through an AdWord's advert (using the keyword depression) sought volunteers in our target areas (12 of 121 postcode areas in Britain).23 These people, in a range of locations, are likely to have had search history including previous searches for depression and so would be presented with different Google search results. These results would be representative of those obtained by people in our target areas searching for depression-related terms. Although we tried to target our study postcode areas in recruiting the panel, there was some ‘leakage’. Nine of 12 were in target areas (Darlington, Lancaster, Orkneys, London SW, Kingston and Liverpool) and three were in other areas (Twickenham, Gloucester and Dorchester). We asked our user panel to search Google using the single word ‘depression’ and email us a screen shot of the first page of results (usually 10 results). We also wanted to explore the variability of search results by different entered terms. We felt unable to ask our panel to carry out and report more than one search so two of the authors searched for the next nine most entered search terms (table 2).

Table 2

The top 10 entered search terms showing the number of resultant clicks on our advert

Search term entered	Resultant clicks on our advert
depression	886
depression help	127
help with depression	71
depression symptoms	60
self help for depression	58
samaritans	57
depression test	51
dealing with depression	36
coping with depression	33
depression self help	28

Estimating the chance of finding online CBT from websites

We assessed the websites found by people using the term ‘depression’ to see how easy it was to find online CBT. We first assumed a knowledgeable user and assessed if online CBT (in this case specifically LLTTF or MoodGYM) could be reached in three clicks by someone who knew that online CBT sites existed. For those websites where online CBT could be reached in three clicks, we calculated the probability that someone (ie, an ‘extreme’ novice) who made completely random choices would find online CBT purely by chance. To do this, we noted how many choices were involved at each level of the shortest pathway to online CBT to calculate an approximate probability of reaching online CBT.

Model to estimate probabilities of finding online CBT

We used a number of assumptions (see results) to estimate the probability of finding online CBT with and without Google adverts. Figure 2 shows a schematic of the samples used and how information from different stages was used to calculate the probabilities.

Figure 2

Schematic of methods. CBT, cognitive behavioural therapy.

Results

Search terms

By 31 August 2011, Google had presented our advert 299 369 times to various Google searches that included combinations of our keywords, resulting in 3868 ‘clicks’. Google reported the 1748 most frequently entered search terms entered by users which were responsible for 71 124 (24%) of the presentations. Examples are shown in table 1. Only four (0.2%) search terms of 1748 included either the words cognitive, behavioural or behavioural and this resulted in 72 presentations (one in 1000) of our advert.

Table 1

Examples of search terms as entered (including spelling mistakes and typing errors) by users for which our Google Advert was displayed and for which the user clicked on the advert

am i depressed	how does one get depressed
anxiety help in blackpool lancs	how to deal with depression how to get over
best cures for depression	depression and loneliness
bipolar help free	how to know if your depressed
cancer sufferer with depression how to help	how to treat depression and trauma
cipralex blog	i need help depression
coping with anxiety and depression	information on depression and anxiety
cures for stress	living with depression
deperssion simtons	manic depressive
depression about dying	mental health helplines
depression and sleep	mood stabilisers
depression clinics UK	neediness and depression
depression help free	online depression test
depression in men	paradox tablets for good mental health
depression pacing up and down	post traumatic stress disorder
depression sign	reasons for depression
depression test and help	samaritans free
depression working	self help depression
does acupuncture help depression	sign of depression
episodes of bipolar disorder	stop depression
food for depression	suicidal thoughts
free on line suport for depreshon	suicide websites
getting help for depression	take a depression test
help for depressed people	the ultimate guide to getting over depression
help friend with depression	treatments for depression nhs
help with depession	what activities can help with depression
herbal remedies for depression	what is depression disorder
how can i help someone who has depression	where do i get help for depression woman & depression

Systematic sample of every 30 of 1748 entries ordered alphabetically.

Ordering the search terms by the number of resultant clicks on our advert, we selected the 10 top terms (table 2). Although these 10 terms only resulted in 1407 (36%) of the clicks, other entered terms, including the terms in table 2 with additional words or misspelled, were likely to have resulted in similar Google search results.

AdWords matched the entries made by users with the 126 keywords using broad matches, exact matches and phrase matches. We reviewed the most popular keywords as listed by AdWords and the full search terms that were entered by users. ‘Depression’ as a single keyword in our Google AdWords campaign resulted in two-thirds of the advert presentations and over half of the clicks.

Websites offered by Google search

Given that Google search results are to some degree personalised by location and previous searches, we wanted to know what a ‘typical’ search result would be. As the single word ‘depression’ was the most frequent search, we asked our UK user panel to search with that single word. The variability of search results using the single term depression was relatively small; only 14 sites in total were presented to the 12 users (table 3). (If searches were exactly the same, there would only be 10 sites, but if searches were all completely different, there would be 120.) Six sites were seen by all 12 users. The NHS Choices appeared in the top three for all.

Table 3

Websites (with hyperlinks) identified by 12 participants searching Google using the term ‘depression’: showing the number of users for whom the website was on first page of results

Website	No. of users where website on first page of results	No. of users where website was in first three results
Sites from which a knowledgeable user could find MoodGYM or LLTTF within three clicks
NHS Choices	12	12
http://www.nhs.uk/Conditions/Depression/Pages/Introduction.aspx
Net Dr	12	3
http://www.netdoctor.co.uk/depression/index.shtml
Patient.co.uk	11	0
http://www.patient.co.uk/health/Depression.htm
RCPsych	12	0
http://www.rcpsych.ac.uk/mentalhealthinfoforall/problems/depression/depression.aspx
Sites from which a knowledgeable user could not find MoodGYM or LLTTF within three clicks
BBC Health	12	0
http://www.bbc.co.uk/health/emotional_health/mental_health/disorders_depression.shtml
Depression Alliance	12	2
http://www.depressionalliance.org/
Mind	12	3
http://www.mind.org.uk/help/diagnoses_and_conditions/depression
Wikipedia	10	10
http://en.wikipedia.org/wiki/Depression
1 on 1 Health	2	0
http://www.1on1health.com/web/info/depression/english/home/Topic-Home
Fighting Depression	2	0
http://www.fightingdepression.co.uk/
Depression UK	2	0
http://www.depressionuk.org/
Clinical Depression	1	0
http://www.clinical-depression.co.uk/
Depressed Test .com	7	0
http://www.depressedtest.com/
Mental Health Foundation	1	0
http://www.mentalhealth.org.uk/help-information/mental-health-a-z/D/depression/

Websites are presented grouped into four websites where a knowledgeable user (ie, someone who knew that online CBT existed and was searching for it) could find CCBT within three clicks and 10 websites where they could not.

CBT, cognitive behavioural therapy; LLTTF, Living Life to the Full.

Finding online CBT from Google displayed websites

One researcher (LG) explored each of the 14 websites in table 3 to see if online CBT could be found within three clicks. Four of the website had links enabling access to either MoodGYM or LLTTF in three clicks (table 3).

We then estimated the probability of finding online CBT on these four websites by chance, that is, for an ‘extreme naive’ user. Table 4 is constructed knowing pathways that will lead to online CBT and while following that path, counting the total number of choices available. For example, on the Royal College of Psychiatrists site, the ‘landing’ depression web page from Google currently has seven main choices within the body of the text (figure 3). It also has some 49 items on the side menu bar, 12 on the tool bar menu, seven to 10 additional information items, 16 links on references, three links at the bottom of the page on leaflets, an online questionnaire and eight links to the ‘small print’.

Table 4

Estimated probabilities of a naive user randomly finding online cognitive behavioural therapy (CBT) from four websites where a knowledgeable user could find CCBT in three clicks

Website	Number of choices available at each level (click)			Probability of finding online free access CBT
	Level 1	Level 2	Level 3
RCPsych	7	20		1 in 140
http://www.rcpsych.ac.uk/mentalhealthinfoforall/problems/depression/depression.aspx
NHS Choices	21	6	15	1 in 1890
http://www.nhs.uk/Conditions/Depression/Pages/Introduction.aspx
Patient.co.uk	12	36	7	1 in 3024
http://www.patient.co.uk/health/Depression.htm
Net Dr	24	8	18	1 in 3456
http://www.netdoctor.co.uk/depression/index.shtml

Websites are shown in order of ease of finding online free access CBT (website names hyperlinked).

Figure 3

Screen shot from Royal College of Psychiatrists ‘depression’ landing page, showing (added labels) side bar menu, tool bar menu and additional information.

We assumed that our naive user with depression having arrived at this page from Google is able to avoid the distractions of the peripheral links and just choose one of the seven from the body of the text (figure 4) and then find ‘Free Online CBT resources’ which is one of 20 choices on the second screen (figure 5).

Figure 4

Screen shot from Royal College of Psychiatrists ‘depression’ landing page, showing link words in the body of the text and other headings that were not linked.

Figure 5

Section from second screen for http://www.RCPsych.ac.uk that has route to CCBT.

Other search terms

Examination of the first page of 10 results for each of the next nine search terms resulted in over 60 different web addresses from 37 organisations. (With 2 researchers, 9 search terms and 10 results each, the total could be 2×9×10=180.) Five of the websites could lead to online CBT, and these were among the 10 most frequently presented (table 5). Four of the five were those displayed for a search on the single word ‘depression’ (table 3).

Table 5

The 12 most frequently presented websites (out of 37 in total) from searches by two researchers using nine different search terms, showing the number of times presented (out of 180 (=2×9×10) search results) and whether the site can be used to find online cognitive behavioural therapy (CBT) in three clicks

Websites presented by Google in first 10 results for two researchers for nine search terms	Number of times presented	Could online CBT be found from this site in three clicks?
http://www.helpguide.org	20	No
http://www.samaritans.org	15	No
http://www.nhs.uk	12	Yes
http://www.clinical-depression.co.uk	10	No
http://www.depressionalliance.org	10	No
http://www.mind.org.uk	10	No
http://www.patient.co.uk	10	Yes
http://www.rcpsych.ac.uk	8	Yes
http://www.bbc.co.uk	6	No
http://www.get.gg	6	Yes
http://www.netdoctor.co.uk	6	Yes
http://www.overcomedepression.co.uk	6	No
All websites	180

We show the main website address rather than the ‘start location’ within each site.

The Samaritans websites (including main website and local variations) were only displayed on entering the Samaritans search term. (Samaritans as a search term was picked up and displayed for our advert as a result of our website having Samaritans as one of the destination links.) In summary, table 6 shows the likely mean number of websites among the first 10 that will link to online CBT in three clicks for a knowledgeable user.

Table 6

Search terms and the mean number of Google search results, out of 10 on the first page, that would offer access to online cognitive behavioural therapy (CBT) in three clicks

Search term	Google search results in first 10 that could lead to CBT
depression	4
depression self help	4
self help for depression	4
depression symptoms	3
coping with depression	3
depression help	2
depression test	2
help with depression	2
dealing with depression	1
samaritans	0

Probability model

What, therefore, is the overall probability of finding an online CBT website by searching for ‘depression’? How does this vary between the optimistic scenario of a knowledgeable user and a more pessimistic scenario of naive user making random choices? We need to make a number of assumptions, namely that:

We can (for the time being) exclude Google Ads from the choices.

Users enter one of 10 search terms, in proportion to the number of clicks as shown in table 2.

Users will be offered 10 sites on the first page by Google search.

The number of sites from which online CBT can be found by a knowledgeable user is as shown in table 5.

Google results will comprise those shown in table 3 for the term ‘depression’ and those shown in table 5 for the next nine search terms.

Knowledgeable users have probability of 1 of finding online CBT from the five websites shown in table 5 and probability of 0 from other sites.

Naive users will have a probability of finding online CBT from those sites calculated by the number of options at each level (as demonstrated in table 4) and probability of 0 for other websites.

The probabilities of finding online CBT are summarised in table 7. A knowledgeable user has a one in three chance, but a naive user has chance of just 1/150. These are probably overestimates as the search results for other entered search terms are unlikely to present more websites leading to online CBT.

Table 7

Probabilities of finding online cognitive behavioural therapy (CBT) for knowledgeable and naive users depending on search term entered

	Probability of using this search term	Probability of finding CBT knowledgeable user	Probability of finding CBT naive user
depression	0.63	0.4	0.0083
depression help	0.09	0.2	0.0075
help with depression	0.05	0.2	0.0075
depression symptoms	0.04	0.3	0.0011
self help for depression	0.04	0.4	0.0083
samaritans	0.04	0	0.0000
depression test	0.04	0.2	0.0008
dealing with depression	0.03	0.1	0.0005
coping with depression	0.02	0.3	0.0011
depression self help	0.02	0.4	0.0083
All	1.00	0.33	0.0069

Change in probabilities through addition of Google advert

What happens to these probabilities if we take into account Google Ads? We have no data on the relative chance of someone clicking on a ‘paid for’ Google Ad rather than a Google search result. However, we can estimate from our own Google Ads campaign by using the ‘click through rate’. This refers to the number of times the advert is presented at which a user chooses the advert. Our click through rate was 1.5%. The number of adverts displayed by Google, either on the right side of the search panel or as the first few searches, in the sponsored links areas is variable. For our users searching for depression, there were typically eight adverts. If we assume that a typical search and adverts screen therefore displays 10 search results and eight adverts, the user has an overall choice of 18 items. We might estimate that the chance of clicking on any one advert is 1.5% so the chance of clicking on an advert is 8×1.5%=12% and the chance of clicking on a Google search result is 88%.

Box 1 shows the Google Ads that were seen by two or more of our users, one of which (The Royal College of Psychiatrists—RCPsych) has a one in 140 chance of reaching online CBT as shown in table 4. If we assume therefore that in addition to the 10 Google searches each user is presented with eight Ads, one of which is http://www.rcpsych.ac.uk, then the probabilities of reaching online CBT, using figures from table 7, become

Sites advertised by Google Ads appearing on search using keyword ‘depression’ for two or more of the panel

http://www.fightingdepression.co.uk (11)

http://www.onlinehelpfordepression.org.uk (6)

http://www.susanjeffers.com (5)

http://www.turn2me.org/Depression (4)

http://www.getconnected.org.uk/Depression (4)

http://www.cbtwestlondon.co.uk (4)

http://www.samaritans.org (3)

http://www.healthyplace.com (2)

http://www.rcpsych.ac.uk/info (2)

http://www.aware.ie (2)

http://www.rethink.org (2)

http://www.greatvine.com (2)

http://re-root.com/ (2)

We can now calculate the increase in probability of adding an advert that leads directly to online CBT and assume that this is one of the eight adverts showing alongside that for the RCPsych.

The probability of finding online CBT via an advert becomes

So probabilities of finding online CBT become

So adding a Google AdWords advert which leads directly to online CBT would improve the chance of finding CCBT for knowledgeable users a little (from 29% to 31%) but would improve the chances of a naive user from one in 167 to one in 66.

Discussion

Increasingly in UK and elsewhere, people are encouraged to self-refer for support and guidance in using online CBT. For example, Increasing Access to Psychological Therapies (IAPT)—a major Department of Health initiative with delivery across England—is promoting self-referral and finding positive benefits.32 This is also seen in the national telephone support line NHS Living Life. Approaches such as these are likely to access a range of people who might not otherwise seek help via their GP.33 Support and guidance are recommended for all forms of CBT resources—including online CBT.34 Although national treatment guidelines such as NICE (2009) and SIGN (2010)19 35 recommend the addition of support, many people accessing online CBT through free sites will do so without support.

This study showed that the 3868 people who clicked on our advert typically searched with the single word depression, although there were a great variety of terms entered, including misspellings. However, our study demonstrated that even a knowledgeable person may have to search quite hard to find online CBT, and it would be difficult, or extremely unlikely, for a naive user of depression websites (ie, someone not specifically searching for online CBT for depression and knowing how to find it) to find online CBT websites easily using a Google search. Although our estimates use a large number of assumptions, we consider that a nearly threefold increase in the probability of finding online CBT by the addition of a Google Advert is likely to lead to more people with depression registering and using online CBT. The real test of this will be if there is an increase in uptake demonstrated in our cluster randomised trial comparing intervention with control areas.

Based on our ‘click through rate’ (the number of times users clicked on our advert divided by the number of times it was presented) of 1.5% and that typically eight adverts are displayed on a page of search results, we estimated that users may click on an advert in 12% of cases when Google displayed search results and adverts. Although this is based on our empirical data, online adverts may not be viewed positively by internet users who are looking for credible information on the web. AdWords are sometimes used to promote bogus, misleading (eg, ‘miracle cures’) and malware-ridden sites, and many users will avoid them. Our own advert (figure 1), mentioning NHS and with a ‘.org.uk’ URL, may have had a better click through rate than other online adverts,36 so we may have overestimated the impact of online adverts.

The Department of Health was heavily criticised for spending £2.5 million on AdWords between February 2009 and January 2010.37 The Department of Health declined to give further details of how they had spent this money or its cost–effectiveness. By judicious design of the website, the NHS Choices and other NHS websites should bring high profile search results. In this study, for example, the NHS Choices depression section was the only website that regularly appeared in the top three search results. If people find a website via a normal web search, then online advertising such as AdWords may be a waste of money. However, this study shows that, if there is substantial ‘competition’ for a user's attention, advertising may be worthwhile. For example, the Royal College of Psychiatrists seems to be spending quite large sums on Google AdWords as their advert appears frequently alongside their website being found by ‘normal’ Google search. For people with depression, given that adverts will in any case appear, having an advert for RCPsych that gives the user a one in 140 chance of finding online CBT for depression improves their chances very slightly.

This exploration of how choices may be made given the websites and adverts currently being found by Google gives some indication that it would be worthwhile. The number of routes that could be taken, even on websites from which it is possible to reach online CBT, also shows the bewildering choice facing those with depression using the internet.

Another approach to assessing search experience is through observation in a laboratory3 11 setting or even ‘in the wild’, but recruiting, meeting and observing people with depression while they search the internet presents many ethical and practical difficulties. We were able to recruit 12 anonymous users who are likely to have searched for depression before, at various locations around the UK. They were able to supply us with information of their searches from their natural ‘habitat’, although we restricted their searching to just one search term. Ideally, we would ask the panel to search for more search terms but we thought this would jeopardise getting any response. The results suggested that although there was some modification of search results by users' previous search history and location, the variability was not that great. Although our sample of 12 users is small, given the lack of variation, it seems unlikely that a bigger sample would have identified other websites that would have increased the probability of finding online CBT.

Our study is also limited in that we only considered Google Search and not, for example, Google Search or Google AdWords embedded in partner sites, blogs or YouTube videos (the so-called Google Display Network). Also, not only do websites change but also Google search results change by location, user and over time. Website owners are also constantly modifying their websites and taking steps to raise their profiles for search engines such as Google. Google is also not the only way of advertising online and others have used, for example, paid location and demographic targeted adverts in Facebook on.38

The approach we have taken to assess the ease of finding CCBT could easily be replicated in future. The general idea of assessing the probabilities that expert and naive users may have in finding a particular website is applicable to anyone considering the addition of online adverts to promote online services or resources. Online adverts may compete both with other adverts as well as with the results of organic search, and health service providers considering the use of online adverts need a strategy that calculates the added (marginal) benefit and cost.

Conclusions

More work is needed into how people make choices when confronted by different Google search results and adverts. Users of Google AdWords would also be advised to consider the context in which they are advertising and not just the data reported by Google Analytics. The results of a search and the additional impact of advertising should be explored in more detail before committing to a long-term Google AdWords advertising budget promoting access to healthcare resources.