Interpretation of health news items reported with or without spin: protocol for a prospective meta-analysis of 16 randomised controlled trials.

INTRODUCTION: We aim to compare the interpretation of health news items reported with or without spin. 'Spin' is defined as a misrepresentation of study results, regardless of motive (intentionally or unintentionally) that overemphasises the beneficial effects of the intervention and overstates safety compared with that shown by the results. METHODS AND ANALYSIS: We have planned a series of 16 randomised controlled trials (RCTs) to perform a prospective meta-analysis. We will select a sample of health news items reporting the results of four types of study designs, evaluating the effect of pharmacological treatment and containing the highest amount of spin in the headline and text. News items reporting four types of studies will be included: (1) preclinical studies; (2) phase I/II (non-randomised) trials; (3) RCTs and (4) observational studies. We will rewrite the selected news items and remove the spin. The original news and rewritten news will be appraised by four types of populations: (1) French-speaking patients; (2) French-speaking general public; (3) English-speaking patients and (4) English-speaking general public. Each RCT will explore the interpretation of news items reporting one of the four study designs by each type of population and will include a sample size of 300 participants. The primary outcome will be participants' interpretation of the benefit of treatment after reading the news items: (What do you think is the probability that treatment X would be beneficial to patients? (scale, 0 (very unlikely) to 10 (very likely)).This study will evaluate the impact of spin on the interpretation of health news reporting results of studies by patients and the general public. ETHICS AND DISSEMINATION: This study has obtained ethics approval from the Institutional Review Board of the Institut national de la santé et de la recherche médicale (INSERM) (registration no: IRB00003888). The description of all the steps and the results of this prospective meta-analysis will be available online and will be disseminated as a published article. On the completion of this study, the results will be sent to all participants. PROSPERO REGISTRATION NUMBER: CRD42017058941. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This will be the first prospective meta-analysis of randomised controlled trials for interpretation of health news items reporting the results of studies with or without spin.

It will address the impact of spin on the interpretation of health news by patients and the general public.

The involvement of patients and the public may help to improve the reporting of medical research in health news.

News stories are only one way that the public hears news about health.

Logistically, the recruitment of large number of participants at the same time may be a challenge, but to manage this, participants will be recruited separately for each trial.

Introduction

Health news is an important way to communicate updates about medical research to the public. News items reporting the results of medical research attract a large audience1 However, the quality of reporting in health news is uneven. The merits of a wide range of treatments and tests are overplayed, and harms are underplayed.2 Several studies have shown the presence of spin (ie, distorted presentation of study results) in health news.3–10 Distorted facts can be misleading and can affect the behaviour of physicians, healthcare providers and patients.1 11 12 However, little research has assessed whether spin can affect readers’ interpretation.13 Some studies have explored whether laypeople are able to recognise the tentativeness of research findings reported in media.14 15 Kimmerle et al found that negative framing and accentuation of the limited reliability of provisional research findings in a newspaper report made people more aware of the tentativeness of these findings.14 In another work, the authors assessed the impact of some personality factors (ie, scientific literacy, epistemology beliefs and academic self-efficacy) and previous users’ comments on an online website on laypeople’s understanding of the tentativeness of medical research findings. Laypeople’s understanding of the tentativeness of research findings was influenced by their personality factors and also by other users’ comments contributed to the forum.15

To our knowledge, no meta-analysis has assessed whether news items reported with spin can influence readers’ interpretations.

Our hypothesis is that spin can influence the reader’s interpretation of health news items. We aim to compare the interpretation of health news items reported with or without spin. We will focus on news items reporting studies evaluating the effect of a pharmacological treatment, containing the largest amount of spin in the headline and text and receiving high levels of public attention online.

Methods

Theoretical framework

Previous works have shown a high prevalence of spin in scientific articles16–19 and in the mass media.8–10 20 However, a question remains: Are readers influenced by spin or are they able to disentangle the appropriate interpretation from the news? In this study, we will consider only news items reporting studies evaluating pharmacological treatments where readers may overestimate the beneficial effect of the treatment if the news is reported with spin and change their behaviour accordingly. We will consider different types of readers: patients and the main public. To increase generalisability, we will also consider two different populations: located in the USA and in France.

Definition of ‘spin’

In the context of this study, we define ‘spin’ as a misrepresentation of study results, regardless of motive (intentionally or unintentionally) that overemphasises the beneficial effects of the intervention and overstates safety compared with that shown by the results.16

The definition of spin we used has been used for exploring spin in the scientific literature.8 13 16 19 21 22 This definition does not take into account the notion of intent because it is impossible to distinguish between the two (ie, intentional and unintentional spin) and the consequences for readers could be the same.

Study design

We have planned a series of 16 randomised controlled trials (RCTs) to perform a prospective meta-analysis (MA) and a comparison of the interpretation of health news items reported with or without spin. Each RCT will explore the interpretation of news items reporting one of four study designs: (1) preclinical studies; (2) phase I/II trials (non-randomised); (3) RCTs and (4) observational studies. The news items reporting each study design will be assessed by four different targeted populations: (1) French-speaking patients; (2) French-speaking general public; (3) English-speaking patients and (4) English-speaking general public. Each RCT will be a parallel group with two arms. In each RCT, participants will be randomly assigned to appraise health news items reported with or without spin (see figure 1).

Figure 1

Series of 16 RCTs that will be included in the prospective meta-analysis. Each RCT will explore the interpretation of news items reporting four study designs: (1) preclinical studies; (2) phase I/II trials (non-randomised); (3) RCTs and (4) observational studies. Each RCT will target four types of populations: (1) French-speaking patients; (2) French-speaking general public; (3) English-speaking patients and (4) English-speaking general public. RCT, randomised controlled trials.

The planning, implementation, analysis and writing of this protocol will follow the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT)23 and Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P)24 guidelines.

News items with and without spin

Selection of news items with spin

News items reporting studies evaluating a pharmacological treatment that received a great deal of public attention online and contained a large amount of spin in the headline and text will be selected from a sample of news items retrieved from Altmetric Explorer.

Search strategy

We will search for articles on ‘PubMed’ using the following search strategy: field ((Randomized controlled trial(Publication Type) OR Observational study[Publication Type]) OR Meta-analysis[Publication Type]) OR Randomized[Title/Abstract]) OR controlled[Title/Abstract]) OR trial[Title/Abstract]) OR cross-sectional[Title/Abstract]) OR case-control[Title/Abstract]) OR cohort[Title/Abstract]) OR Meta-analysis[Title/Abstract]) OR systematic review[Title/Abstract]) AND (has abstract [text] AND (“2014/01/01”[PDAT]: “2014/06/30”[PDAT])). The publication period will be restricted to the first 6 months of 2014 to minimise the risk of recall bias among study participants.

To retrieve relevant news coverage of these articles, we will apply the ‘PubMed search details’ on ‘Altmetric Explorer’. The Web application Altmetric Explorer provides access to all sources where the published study is mentioned online in the mass media and sorts the items according to the Altmetric score.25 The Altmetric score is one way to quantify the public attention an article received in online news outlets, blogs and social media (https://www.altmetric.com/) (a high Altmetric score=high public attention).

Screening process

Screening will be performed in two steps: first, one researcher will systematically screen the retrieved Altmetric Explorer citations, which will be sorted from the highest to the lowest Altmetric score (ie, highest to lowest amount of public attention) and will identify studies evaluating the effect of a pharmacological treatment, regardless of study design and study population (including human and animal/laboratory). For each study fulfilling eligibility criteria, the researcher will retrieve (1) the published article and (2) all related online news items available at Altmetric Explorer.

Second, the researcher will identify the news item with spin in the headline and text by using a standard scheme of spin.10 19 When several news items have spin in the headline, the researcher will select the news item with the most spin in the text. We will include news items reported by general or medical news outlets or lay press whose target consumers are the general population.

As a quality procedure, a second researcher will confirm the eligibility of all included studies and screen 10% of the excluded studies.

The screening process will be performed sequentially, the studies being sorted from the highest to the lowest Altmetric score (ie, highest to lowest public attention). We will include the first 40 studies fulfilling the eligibility criteria and relevant 40 news items containing the most spin in the headline and text: 10 reporting preclinical studies, 10 reporting phase I/II non-randomised trials, 10 news items reporting RCTs and 10 reporting observational studies.

Identification and description of spin

We will identify the spin in the headlines and text of selected news items and will classify them according to following three categories of spin—misleading reporting, misleading interpretation and misleading extrapolation—that were previously developed.10

Misleading reporting is defined as incomplete or inadequate reporting of any important information in the context of the research that could be misleading for the reader. This category includes: (1) misleading reporting of study design; (2) not reporting study population (if an animal study); (3) selective reporting of outcomes favouring the beneficial effect of the treatment (eg, statistically significant results for efficacy outcomes or statistically non-significant results for safety outcomes); (4) not reporting adverse events; (5) linguistic spin (ie, any word or expression emphasising the beneficial effect of the treatment26; (6) not reporting study limitations; (7) not reporting any caution about study design and results and (8) any other type of misleading reporting not classified under the above section.

Misleading interpretation is defined as an interpretation of the study results in news stories that is not consistent with the results reported in the scientific articles and overestimating the beneficial effect of the treatment. This category includes claiming (1) a beneficial effect of the treatment despite statistically non-significant results; (2) an equivalent effect of the treatment for statistically non-significant results in superiority RCTs; (3) that the treatment is safe for statistically non-significant results despite a lack of power; (4) safety of the treatment despite adverse events reported in the scientific articles; (5) a causal effect (ie, implies a cause-and-effect relationship between the intervention being assessed and the outcome of interest27) despite a non-randomised study design; (6) a beneficial effect of the treatment despite a small sample size and (7) a beneficial effect despite lack of a comparator as well as (8) focus on P value instead of clinical importance; (9) interpretation of relative risk as absolute risk and (10) any other type of misleading interpretation not otherwise classified.

Misleading extrapolation is defined as overgeneralisation of study results in news stories to different populations, interventions or outcomes that were not assessed in the study. This category includes extrapolating (1) animal study results to human application; (2) preliminary study results to clinical application; (3) the effect of study outcomes to other outcomes for the disease; (4) the beneficial effect of the study intervention to a different intervention (eg, broccoli, which contains sulforaphane, was claimed as beneficial by health news items, but the study evaluated the benefit of a sulforaphane compound only) and (5) from the study participants to a larger or different population as well as (6) inappropriate implications for clinical or daily use (ie, an improper recommendation or advice to use the intervention in clinical practice or daily use not supported by study results) and (7) any other types of extrapolation not otherwise classified.

All other spin that could not be classified with this scheme will be systematically recorded and secondarily classified.

Construction of news without spin

Format of the news items

Our aim is to keep the same context and format of the original news item and conceal the names of pharmacological treatments, authors and funders to avoid evaluation bias. Consequently, to rewrite the news items we will:

Keep the same context and structure;

Create hypothetical names of reported pharmacological treatments;

Conceal the names of study authors and experts by using different names selected based on the origin of the name from an online list of names including all countries of the world (http://www.studentsoftheworld.info/penpals/stats.php3?Pays) to keep the news content natural;

Keep the name of the research institute/university/hospital where the study was conducted;

Replace the name of the funding source with standardised terms for profit or non-profit funding organisations;

Delete the name of the online news outlet, date the news story was published online, name of the journalist who wrote the news with spin, name of the medical journal in which the study was published, reference to the original article and trial registration number or name (if reported).

Guidelines to remove spin in the news items

To construct health news stories without spin, we will delete the spin identified in the headline and text and will add some caution, depending on context. The guidelines used to remove the spin are described in table 1. The guidelines to add caution are in table 2.

Table 1

Guidelines to remove spin

Spin	Interventions/modifications
Spin in headline	Delete the misleading information and report the appropriate information.
Spin in text
Misleading reporting
Misleading reporting of study design	Report the appropriate study design.
Not reporting study population if an animal study	Report animal study subjects.
Selective reporting of outcomes	Report the results for all primary outcomes.
Not reporting adverse events	Report adverse events when higher in one group. (We considered reporting more frequent and serious adverse events related to treatment primarily.)
Use of linguistic spin	Delete linguistic spin.
Not reporting study limitations and caution specific to study design	Report the study limitations and cautions. The cautions with standardised text are described in table 2.
Misleading interpretation
Claiming a beneficial effect of intervention despite statistically non-significant results Claiming an equivalent beneficial effect of intervention despite statistically non-significant results in superiority RCTs	Delete this spin and use the generic wording, such as: Treatment A was not more effective on ‘primary outcome’ than the comparator B in patients with …
Claiming the treatment is safe despite statistically non-significant results in treatment and comparison groups Claiming safety despite adverse events Claiming a causal effect despite non-randomised study design Claiming a beneficial effect despite small sample size not reported Claiming a beneficial effect despite lack of comparator Focus on P value instead of magnitude of the effect (effect size)	Delete this spin; reword and provide the appropriate information when needed.
Misleading extrapolation
Animal study results to human application Preliminary study results to clinical application Study outcomes to other outcomes for the disease Study intervention to a different intervention Study participants to a larger or different population	Delete the inappropriate extrapolation.
Inappropriate implication for clinical or daily use	Delete the statement and clearly report the immediate unavailability in clinical practice.
Author’s/expert’s statement (interview)
	Delete the spin in the statement.
	Report the caution or recommendation by study authors, reported in the relevant article when available.

RCT, randomised controlled trial.

Table 2

Reporting of cautions with standardised wording

Study design	Standardised text
Animal or laboratory study	“The study was based on animals; it is impossible to know whether this treatment will work on humans or not.”
Small study	“These results are based on a small study; larger studies are needed to understand whether the treatment works across a large population.”
Uncontrolled study/Lack of comparator	“Everyone in this study took drug X. Without investigating patients who did not take that drug, it is impossible to know whether taking drug X accounted for the outcome.”
Controlled but not randomised study	“The study participants were not randomized. We do not know whether it was drug X or something else that really accounted for the effect observed.”
Important adverse event	“The benefit observed should be weighed against the adverse effects (or other downsides such as inconvenience, cost, etc).”

One researcher (RH) will identify and remove the spin in each news item selected (in the headline and text) and will rewrite the news story without spin, according to the guidelines described in tables 1 and 2. Two researchers (IB) and (AY) will check the rewritten news items. Finally, a sample of the rewritten news stories will be checked by a researcher working in the field of medical journalism (IO). Online supplementary appendix 1 provides an example of a news item reported with and without spin. Our sample of news will contain 80 news items (40 original news items (with spin) and 40 rewritten news items (without spin)).

Translation of the news items reported with and without spin

All news items will be translated into French language to be used in RCTs involving French-speaking participants. One French native speaker researcher (AY) will validate the French translation of news items. Further, a French medical journalist will also validate the French translated news items.

Population

We will compare the health news reported in English and French languages and will assess their interpretation by different types of populations to increase the generalisability of our results. Each RCT will target one of the four following study populations:

French-speaking patients,

French-speaking general public,

English-speaking patients,

English-speaking general public.

Eligibility criteria

We will enrol participants older than 18 years.

Recruitment strategy

To recruit participants, we will contact online communities of patients, patients’ associations, popular health forums and investigators of e-cohorts. We will also use the online platform (www.findparticipants.com), which enables access to thousands of interested participants to participate in research studies worldwide. We will also advertise the study in hospitals and general practitioner practices.

Each participant will provide an online informed consent at the time of enrolment.

We will send participants an invitation by email (see online supplementary appendix 2). If respondents agree to participate in the survey, an internet link included in the invitation e-mail will give them access to information regarding the study and a screening question asking them whether they are willing to participate in the study. If they answer yes, respondents will be randomly assigned to read one news item with spin or one news item without spin.

Invitation emails will be sent in waves until the planned number of participants log on and complete the assessment. A maximum of two reminders will be sent to participants.

Interventions

We will compare the interpretation of ‘health news items’ reported with spin (original news=active comparator) or without spin (rewritten news=experimental group).

Random assignment

A random assignment sequence will be computer generated by a statistician by using blocks of 10 (ie, number of news items selected × 2) for each study design type. The list will not be disclosed to investigators. Allocation concealment will be assured by the use of a computerised random-assignment system. After randomisation, participants will be asked to complete a questionnaire. Participants who log on and do not evaluate the news will be excluded, and the news item will be automatically allocated to another participant.

Blinding

Blinding of participants is not possible, but to minimise bias, participants will be blinded to the study hypothesis. All participants will be informed that they are participating in a survey about the interpretation of news reporting medical research that evaluates treatments. They will not be informed about the objectives and hypothesis of the study.

After the completion of study, each participant will be told about the study objectives, hypothesis and results.

Study outcomes

Our primary outcome will be participants’ interpretation of the benefit of the treatment measured on a scale from 0 to 10.

What do you think is the probability that treatment X would be beneficial to patients?(scale, 0 (very unlikely) to 10 (very likely))

Secondary outcomes are as follows:

What do you think is the size of the potential benefit for patients? (scale (none, small, moderate or large));

How safe do you think that treatment X would be for patients? (scale, 0 (very unsafe) to 10 (very safe));

Do you think this treatment should be offered to patients in the short term? (scale, 0 (absolutely no) to 10 (absolutely yes));

Do you think this treatment will make a difference in the existing clinical practice? (scale, 0 (absolutely no) to 10 (absolutely yes)).

These study outcomes are surrogate markers measuring the perception by readers of the treatments’ efficacy, safety, availability and use in current clinical practice.

Sample size

Each participant will read a news item with or without spin. We want to assess a mean difference of 1.0 for the primary outcome between groups on a 0–10 scale, with an SD of 2.5 (13). For each RCT, a sample of 266 assessments of news items will be needed to detect an effect size of 0.4 with a power of 90% and α risk of 5% for each RCT. Each news item will be read the same number of times (balanced design) and we will take into account clustering due to the fact that a news item will be read many times. To achieve this, we will use a sample size of 300 participants (150 in each group) in each RCT (ie, an inflation factor of about 1.1). Therefore, each news item will be assessed 15 times in each group (10 news items with or without spin for 150 participants) for each RCT.

Statistical analysis

The statistical analysis will be undertaken by a statistician who will use R V.2.15.1 (R foundation for Statistical Computing, Vienna, Austria) at the Center for Clinical Epidemiology, Paris, France. All outcomes will be quantitative and the number of participants and news items will be balanced in each group. For each RCT, the following analysis will be done: The differences between groups will be analysed by using a linear mixed model with a fixed group effect and random group effect and news items–group interaction effects. Random effects will allow us to account for the following two levels of clustering: within-group clustering as a result of the news (each news item will be assessed 15 times in each group) and between-group clustering (pairing between the news used in the two arms of the trial). Inferences will be based on the restricted maximum likelihood. This model will compare the mean difference between two arms for each trial. For primary and secondary outcomes, we will estimate the difference between means with 95% CIs. A P value of <0.05 will be considered statistically significant.

Finally, after analysing each RCT separately, a prospective meta-analysis will be done to summarise intervention effects. The mean difference with 95% CIs will be estimated by using a random-effects model based on the DerSimonian-Laird method. Forest plots will be created for visual interpretation of results. The heterogeneity will be assessed by χ2 test (P<0.05) and degree of heterogeneity by the I2 statistic (>75%) to assess statistical significance. We will also assess the variance (τ2 between trials).

Study duration

The total duration of this study will be 24 months. Expected period of inclusion of participants will also be 24 months and the duration of participation per participant/patient will be 1 hour. The anticipated start date of trials will be June 2017.

Discussion

To best of our knowledge, we present the first prospective meta-analysis of RCTs for interpretation of health news items reporting the results of studies with or without spin.

We have designed 16 RCTs which will focus on interpretation of news items reporting results of four types of study designs: (1) preclinical studies; (2) phase I/II trials (non-randomised); (3) RCTs and (4) observational studies. There will be 80 news items reporting these study designs (20 new items/study design: 10 original news items with spin +10 rewritten news items without spin). Each RCT will target one of the four types of populations: (1) French-speaking patients; (2) French-speaking general public; (3) English-speaking patients and (4) English-speaking general public. In total, 4800 participants will be involved in 16 planned RCTs (300 participants/RCT). Once the planned RCTs are completed, then the results of different RCTs will be included to perform a meta-analysis.

The concept of prospective meta-analysis allows us to compare the interpretation of health news stories reporting results of studies with or without spin by different types of populations. This new form of synthesis of evidence answers the question of whether spin can influence patients’ and the publics’ interpretation of health news.

We will document all practical issues and difficulties encountered to demonstrate that this type of synthesis of evidence is feasible. We are aware of some challenges, such as recruitment of participants. Logistically, the recruitment of large number of participants at the same time may be a challenge, but to manage this, participants will be recruited separately for each trial.

Expected results

This study will evaluate the impact of spin on patients’ and the public’s interpretation of news items reporting results of studies.