Tweeting links to Cochrane Schizophrenia Group reviews: a randomised controlled trial.

OBJECTIVE: To assess the effects of using health social media on web activity.
DESIGN: Individually randomised controlled parallel group superiority trial.
SETTING: Twitter and Weibo. PARTICIPANTS: 170 Cochrane Schizophrenia Group full reviews with an abstract and plain language summary web page.
INTERVENTIONS: Three randomly ordered slightly different 140 character or less messages, each containing a short URL to the freely accessible summary page sent on specific times on one single day. This was compared with no messaging. OUTCOME: The primary outcome was web page visits at 1 week. Secondary outcomes were other metrics of web activity at 1 week.
RESULTS: 85 reviews were randomised to each of the intervention and control arms. Google Analytics allowed 100% follow-up within 1 week of completion. Intervention and control reviews received a total of 1162 and 449 visits, respectively (IRR 2.7, 95% CI 2.2 to 3.3). Fewer intervention reviews had single page only visits (16% vs 31%, OR 0.41, 0.19 to 0.88) and users spent more time viewing intervention reviews (geometric mean 76 vs 31 s, ratio 2.5, 1.3 to 4.6). Other secondary metrics of web activity all showed strong evidence in favour of the intervention.
CONCLUSIONS: Tweeting in this limited area of healthcare increases 'product placement' of evidence with the potential for that to influence care. TRIAL REGISTRATION NUMBER: ISRCTN84658943. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

RCT Entities:   Population Interventions Outcomes

This is the first randomised controlled trial that we are aware of evaluating the impact of Tweeting health-related web links on access to the target web page and/or related web pages.

This study quantifies the effects of Tweeting evidence and generates many questions for future research.

We used free-to-use software with limited functionality—more sophisticated software may highlight more effects.

We Tweeted links to large academic reviews focusing on one small area of healthcare to a relatively small ‘following’. Different techniques of Tweeting, other areas of healthcare, and a broad set of followers could result in more impact.

Introduction

Cochrane (previously known as Cochrane Collaboration) has more than 37 000 contributors from around 130 countries. They work together to produce credible, accessible health information that is free from commercial sponsorship and other conflicts of interest. The Cochrane Schizophrenia Group (which is part of Cochrane) produces and maintains systematic reviews and meta-analyses of randomised trials that evaluate the effects of interventions for schizophrenia and related psychotic illnesses. Full text versions of this work can be accessed in some high income and most low income countries.1 Each review also has an abstract and a plain language summary (PLS) to make the information in the review more accessible to people without specialised knowledge.2 Both the abstract and PLS are freely available from the Cochrane website.3

Twitter is a popular free-to-use social media platform which allows users to send a 140-character message (a ‘Tweet’) to a group who has chosen to receive (or ‘follow’) these short messages from the sender. Currently there are 288 million monthly active users sending out 500 million Tweets a day.4 Most (77%) accounts are outside the USA and 80% of Tweets are sent from mobile devices. The use of Twitter in healthcare has increased, encompassing, for example, issues relating to public health surveillance, tracking the disease activity of the H1N1 pandemic and isolating the source of a cholera outbreak in Haiti.5–7 Health-related Twitter messages now comprise an estimated 0.0001% of all Twitter traffic.8 Most Cochrane Review Groups have recently developed this method of dissemination.9 However, an investment of effort is required to undertake this activity, and the potential benefits for review groups with limited numbers of reviews and followers are unclear.

Twitter and some other platforms are not available in China. However, 91% of China's population use social media compared to 67% of the population of the USA.10 The Cochrane Schizophrenia Group has been working with a Chinese company, Systematic Review Solutions Ltd,11 to disseminate parallel messages on Weibo, a Twitter-like system, to their followers across China. Weibo is in the top 10 social media sites used in China with over 600 million registered users, of which about 140 million use it regularly (as of March 2014).12

Given the increasing use of social media in healthcare, and the investment involved in doing this, we evaluated the impact of Tweeting précis of Cochrane Schizophrenia Group systematic reviews.

Methods

Study design

This was a prospective two-arm, parallel, open randomised controlled trial with a 1:1 allocation ratio. The protocol with full details of the study design has been published elsewhere.13

Participants

Participants included in this study were published full text Cochrane Schizophrenia Group systematic reviews in The Cochrane Library with PLS which are available for free at http://www.summaries.cochrane.org. We excluded published protocols for Cochrane Schizophrenia Group reviews, any Cochrane Schizophrenia Group review not relevant to schizophrenia, unpublished and withdrawn Cochrane Schizophrenia Group reviews.

This study was conducted entirely on the internet. The participant PLSs were all available on the Cochrane Library website, the intervention was delivered via two social media platforms (Twitter and Weibo) and outcome data were collected using Google Analytics.

This study did not involve any living participants and used information that is available in the public domain. Participants are systematic reviews rather than people and routine web activity data were extracted and recorded through Google Analytics. No ethical approval was required.14 15

Randomisation

All participating Cochrane Schizophrenia Group systematic reviews were identified prior to randomisation. We defined mean baseline access activity as high (≥19 unique views per week, n=14), medium (4.3–18.99 unique views per week, n=72) or low (<4.3 views unique per week, n=84) based on the 23-week period 21 September 2013 to 28 February 2014 (defined in the ‘Statistical analysis’ section). Reviews were given a unique code, which along with the baseline activity stratum, was supplied to one of the authors (AAM) who performed the randomisation. Stratifying by baseline access activity, and using computer-generated random number sequences, reviews were first allocated to intervention and control arms, then into pairs of reviews that would have the same reference period for outcome data collection, then to day of the week and week number (1–29) that Tweeting would take place for reviews in the intervention arm. Days for Tweeting were Tuesday, Wednesday or Friday as these are considered to have the heaviest traffic.16 Finally, the sequence of the three Tweets for each review (the Tweet package) in the intervention arm was also randomised. Tweets had been prepared for all reviews. Once allocation codes were supplied to AYMB, independent of other investigators, she scheduled Tweets for the intervention group within Hootsuite.17

Procedures

Reviews in the intervention group were Tweeted in English, by the Cochrane Schizophrenia Group, three times on the same day at 10:30, 13:00 and 15:00 GMT as guided by the SocialBro web tool,18 and 8 h later in China, in Mandarin, via Weibo. There is some evidence that multiple postings, 3–4 times a day, of the same or similar Tweet can be useful for an international following.16 Each of the three Tweets had different accompanying text:

The review title as it appears in summaries.cochrane.org, and a shortened URL to the summaries web page

A pertinent extract from the results or discussion sections of the abstract, and a shortened URL to the summaries web page

An intriguing question or pithy statement directly related to the evidence presented in the abstract, and a shortened URL to the summaries web page (see table 1).

Table 1

Example of the three tweets relating to the same review

Tweet 1	#Clozapine combined with different #antipsychotic #drugs for#treatment resistant #schizophrenia http://ow.ly/yaKAU
Tweet 2	How effective is #clozapine in addition to another#antipsychotics at treating ‘hard to treat’ #schizophrenia? http://ow.ly/yaKAU
Tweet 3	Not clear if combining #clozapine with other #antipsychotics is effective for #treatment resistant #schizophrenia http://ow.ly/yaKAU

We did not compare the specific content of the different types of accompanying text, which were formulated in order to appeal to various followers of the Cochrane Schizophrenia Group Twitter page and searchers. We compared the combined impact of the package of Tweets versus no Tweets. To assist the logistics of sending out Tweets, we used Hootsuite, a social media management system. This free package allows formulation and scheduling of Tweets and is now available in China and integrates with Weibo (http://blog.hootsuite.com/chinese-localization-weibo/).

Reviews in the control group were not Tweeted by the Cochrane Schizophrenia Group.

Outcomes

The primary outcome was the total number of visits to the relevant Cochrane summary page in the 7 days following the intervention (including the day of Tweeting) as reported on Google Analytics. The secondary outcomes are listed in box 1.

1. Unique page views

2. Entrances

3. 100% of visits were single page

4. Direct click visits

5. Twitter referrals

6. Any outbound click events

7. Time on page in seconds

For reviews in the control arm, the 7-day follow-up period was the same as for the intervention review with which it had been randomly paired. For the primary outcome, repeated views of the same page during a single user session are counted in the total; unique page views are a secondary outcome. The standard free account in Google Analytics produces various reports in real-time and provides data about whom, when and where someone has visited a site, as well as how they ‘arrived’ at that site.19 20 Table 2 outlines the glossary of Google Analytics terms.

Table 2

Glossary of Google Analytics terms

Google term		Explanation	In lay terms
Direct clicks Traffic that does not originate from search-engine results or a referring link in a domain is identified as ‘direct’23	Visits	Number of times people viewed the site	Direct hits on the specific Cochrane summaries (not overall Cochrane site hits)—without being directed via our Tweets. Here people have intentionally gone to the website to look at the summaries without our encouragement.
All clicks The overview of all clicks onto the website of interest23	Page views	Number of pages viewed. Repeated views of a single page are counted.	Once on the target (summaries) page a person can click around the site and back and forward to the page. So if they hit the target page, click to another within the site and then return to the target page, that equals 3. This is the same as ‘Direct-click visits’ but includes other ways people have got to the Cochrane summaries page, for example, directed via Twitter or URLs they have clicked on from another website, etc.
	Unique page views	Number of visits during which the specified page was viewed at least once. A unique page view is counted for each page URL + page title combination.	The number of individual (non-duplicate) visits to the target (Cochrane summaries) page. This is similar to Page views but will also look at the IP address—counting it as one view from a computer even if the user accesses the page multiple times in the same session (30 min).
	Average time on page	Average amount of time visitors spend viewing a specified page or set of pages	This is the time spent on one web page—in this case the target (summaries) page. A website may have multiple web pages.
	Entrances	Number of times visitors entered your site through a specified page or set of pages	This is the number of times someone entered the summaries page through the Twitter page.
	Bounce rate	% of single-page visits (i.e., visits in which the person left the site from the entrance page without interacting with the page).	This may be an indication of getting what they need off the target (summary) page—or, alternatively—not getting anything they need and having no will to proceed. We converted this to binary for analysis: 1=all visits were single page, 0=other
Twitter referrals Clicks originating from a third-party website where a web link links to the page of interest24 25	Page views	Number of pages viewed. Repeated views of a single page are counted.	The subset of ‘All clicks target page views’—only ones from Twitter. This may include Cochrane Schizophrenia Group tweets, retweets or other Twitter's user tweets whereby the page URL has been tweeted. We were also using Wiebo.
Events—an action tracked on the website—for example, exit to Cochrane Library	Total events	Total events is the number of times events occurred.	This is the total number of times an external link is clicked

This includes all traffic to the Cochrane summary page and traffic directly from Twitter. The average half-life of a Tweet (with a web link), defined as ‘

the amount of time at which this link will receive half of the clicks it will ever receive after it's reached its peak’, has been estimated as 2.8 h.21 However, to capture any possible cascade effect of Tweeting, we extended the monitoring period to 7 days.22 Secondary outcomes provide other measures of incoming activity and exiting behaviour. In addition, we recorded country distribution of users clicking on the target sites.

One week after the final Tweets were sent, four different data reports per review were downloaded from Google Analytics for each Cochrane summary page, for the relevant 7-day period. These 680 files were uploaded into an MS Access database and then merged for analysis.

Statistical analysis

The sample size for this study is fixed by the number of published Cochrane reviews under the control of the Cochrane Schizophrenia Group (n=170). Therefore, we estimated the magnitude of the detectable between-group difference in the primary outcome. Google Analytics data for all 170 reviews during the period 21 September 2013 to 28 February 2014 suggested that log-transformed mean number of views per week is normally distributed, with a mean of 1.5 and SD of 0.9. With 5% two-sided α and a sample size of 85 per arm, between-group differences in the range 0.43–0.5 SDs, equivalent to a ratio of geometric means of 1.47 to 1.57, are detectable with 80–90% power.

The primary outcome is a count of the number of times that each review is accessed during its 7-day follow-up period. Due to over-dispersion in this, and in other outcomes that are counts, we used multivariable negative binomial regression modelling to estimate incidence rate ratios and 95% CIs, adjusted for baseline activity stratum and day of the week on which the 7-day data collection period commenced. We also investigated the use of mixed effects models to take account of pairing of intervention and control reviews. This made no material difference and so results from the simpler models are presented. Outcomes with excess zero counts were analysed using zero-inflated negative binomial regression models. Binary variables were analysed using logistic regression. Continuous variables were analysed using linear regression, with log-transformation of outcomes and presentation as the ratio of geometric means if appropriate.

We conducted further secondary analyses of the primary outcome. By including appropriate interaction terms in the primary regression model, we investigated in subgroup analyses whether any effect of the intervention differed according to baseline activity, or by day of the week that Tweets were sent. The latter was not pre-specified before analysis of the data and therefore should be regarded as exploratory. Finally, we estimated the effects of the intervention separately for visits to the review summaries that originated from China and from outside China.

All between-group comparisons analysed reviews as randomised. As the intervention was delivered as intended, and there are no missing outcome data, sensitivity analyses were unnecessary.

At the time of the trial, there were 849 followers on Twitter and 5995 followers on Weibo. Following randomisation of 170 reviews, intervention group Tweets were posted between 1 July 2014 and 13 January 2015 between which participants (systematic reviews) were recruited as outlined in the trial profile (figure 1). Reviews allocated to groups were evenly distributed within each activity stratum (high, medium, low) and this was reflected in the broad categorisation of review content and there were no statistically significant differences between the two groups (table 3).

Figure 1

Trial profile.

Table 3

Description of review by randomised group

	Group
Category	Non-Tweet	Tweet
Participants
Schizophrenia+additional issues	20 (24%)	13 (15%)
Schizophrenia	65 (76%)	72 (85%)
Intervention
Drug	59 (69%)	54 (64%)
Non-drug	26 (31%)	31 (36%)

Results are presented in table 4.

Table 4

Results

	Control (n=85)	Intervention (n=85)	Crude estimate of effect	Adjusted* estimate of effect	95% CI	p Value
Primary outcome
All page views
Total	449	1162
Incidence rate	5.28	13.67	2.59†	2.71	2.20 to 3.33	<0.001
Secondary outcome
Unique page views
Total	403	1058
Incidence rate	4.74	12.45	2.63†	2.74	2.24 to 3.35	<0.001
Entrances
Total	260	821
Incidence rate	3.06	9.66	3.16†	3.36	2.68 to 4.22	<0.001
100% of visits were single page
No	53	71
Yes	24	14	0.44‡	0.41	0.19 to 0.88	0.023
Direct click visits
Total	26	168
Incidence rate	0.31	1.98	6.46†	6.74	4.11 to 11.08	<0.001
Twitter referrals
Total	0	297
Incidence rate	0	3.49	−	−	−	−
Any outbound click events
No	60	39
Yes	25	46	2.83‡	3.20	1.63 to 6.31	0.001
Time on page in seconds
Arithmetic mean (SD)	147 (232)	164 (210)
Median (min, max)	51 (0, 1283)	109 (0, 1610)
Geometric mean	31	76	2.42§	2.47	1.32 to 4.61	0.005

*Adjusted for baseline activity and day of week.

†Incidence rate ratio.

‡OR.

§Ratio of geometric means.

For the primary outcome, there were 1162 and 449 visits in total to the 85 review summaries in the intervention and control arms, respectively, and strong statistical evidence of a nearly threefold increase in the intervention arm. There was further evidence of an effect of the intervention on all of the secondary outcomes.

There was no evidence that the intervention was differentially effective at increasing visits to reviews classified according to baseline activity as low (IRR 3.09, 95% CI 2.30 to 4.17), medium (IRR 2.65, 1.94 to 3.61) or high (IRR 1.85, 0.88 to 3.89) (overall p value for interaction=0.31). Nor was there any evidence of a differential effect according to day of the week that Tweets were sent (p value for interaction=0.68).

The data did not suggest that the intervention was any more or less effective in China compared with the rest of the world. The incidence rate ratio for the primary outcome for visits that originated from China was 2.27 (95% CI 0.35 to 14.77), and 2.15 (1.73 to 2.68) for from outside China.

Discussion

Much investment is put into the dissemination of healthcare evidence. Many choices are available to individuals, institutions and industry, and it is difficult to estimate how much of this investment is misplaced. We were able to show that a simple approach on a commonly used set of systems (i) was possible to evaluate and (ii) seemed to have some effect.

This is the first randomised controlled trial that we are aware of evaluating the impact of Tweeting health-related web links on access to the target web page and/or related web pages. This study quantifies the effects of Tweeting evidence and generates many questions for future research. We used free-to-use software with limited functionality—more sophisticated software may highlight more effects. We Tweeted links to large academic reviews focusing on one small area of healthcare to a relatively small ‘following’. Different techniques of Tweeting, other areas of healthcare, and a broad set of followers could result in a greater impact.

There are many more questions on the effects of types of social media targeting and how best to undertake this in order to encourage the implementation of best evidence. Much more sophisticated outcome analytics are available beyond those we used which would add to understanding of how best to target for substantive encouragement of behaviour for best impact.

All hit rates, whatever way they were measured, were largely modest. The degree of effect was consistently substantial but is relative to the modest standard activity. Without our social media activity many of the Cochrane reviews have little or no activity within the target week. The social media activity changes this and there are some indications that the intervention also changed how a person interacts with the site. The user would stay longer, ‘bounce off’ less frequently and exit to another site more frequently. We did not specially investigate if people tended to exit to the full Cochrane Review as access to the full text varies across countries. This study underlines the importance of this summaries web interface for the Cochrane Collaboration. The activities, however, increased by the social media intervention, would seem to indicate improved interest.

Conclusions

Use of social media in an area of healthcare that is not particularly high profile, for the dissemination of evidence that is highly specialised through a small following, nevertheless, seemed to have a genuine effect by which indicators of interest were increased. There were more visits to the target site and the time spent on the page was longer. We cannot tell if that interest translated into better care more in line with best evidence, but ‘product placement’ of good evidence is a significant step in the right direction.

Much effort is expended in disseminating health-related messages across social media with unclear effect. The Cochrane Schizophrenia Group, based in the University of Nottingham, has randomised their reviews to be also disseminated by social media compared with no additional ‘Tweeting’/(‘Weibo-ing’). There is now evidence from a trial that low-grade social media dissemination is powerful for the ‘product placement’ of health information.