Assessing bias in osteoarthritis trials included in Cochrane reviews: protocol for a meta-epidemiological study.

INTRODUCTION: The validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews. METHODS AND ANALYSIS: Only systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration's recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ(2), estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model. ETHICS AND DISSEMINATION: Meta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required. TRIAL REGISTRATION NUMBER: PROSPERO (CRD42013006924). Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

The Cochrane Musculoskeletal Group (CMSG) is one of the 53 existing review groups in the Cochrane Collaboration, which aims to prepare, maintain and disseminate high-quality systematic reviews for musculoskeletal diseases including osteoarthritis; and help healthcare providers, patients and carers to make well-informed decisions on prevention, treatment and management of musculoskeletal conditions.1 2 (http://musculoskeletal.cochrane.org/more-about-us 29 August 2013). The Cochrane Collaboration makes a considerable contribution to reduce bias in scientific interpretation of research.3–5 Systematic reviews founded on randomised trials provide the most reliable evidence about the effects of healthcare interventions.6 Unfortunately, inadequate methodology may distort the outcomes from systematic reviews and meta­analyses7 and produce misleading results.8

Description of the problem or issue

Bias in trials can lead to underestimation or overestimation of the true intervention effect.9 Regardless of the tools used to assess risk of bias, the methods for assessing and summarising potential bias and incorporating bias assessments into meta-analyses vary greatly.3 4 Bias associated with particular characteristics of studies may be examined using meta-epidemiology, which analyses a cluster of meta-analyses where the influence of the trial characteristics—such as judgements on risk of bias on treatment effects estimates—is explored.8 By using meta-epidemiological studies, it is possible to examine the association of specific trial characteristics in a collection of meta-analyses and their included trials.10 11

Most meta-epidemiological studies focus on allocation concealment and blinding and their influence on treatment effects of dichotomous outcomes,10 12 but this study will focus on other characteristics as well: attrition bias, the issue of center status and funding. Further, this study will focus on pain—a continuous patient reported outcome (PRO). Non-drug intervention trials researching treatment in osteoarthritis are difficult to blind; the patients might know whether they are receiving the intervention or control, which could affect their (self-reported) pain scores.

Description of the methods being investigated

Much research has been conducted to understand bias in trials and how it may influence the results of systematic reviews.5 In 2001, the Cochrane Bias Methods Group was established to investigate how bias influences primary studies. Through its acknowledged work, the group developed the Cochrane risk of bias tool in 2008, based on the methodological contributions of meta-epidemiological studies. This tool has since been updated and is a mandatory component of Cochrane reviews.4 There remain additional possible contributors to bias which are not currently standard components for the Cochrane risk of Bias tool.

At the Cochrane Methods Symposium in Quebec City, Canada, October 2013, L Bero and J Sterne—among others—contributed in a debate on whether or not to include funding as a standard domain in the Cochrane risk of bias tool, or if the bias related to source of funding could be captured as part of various ‘reporting biases,’ as eligible studies tend to remain unpublished for reasons related to their ‘negative’ results.13 14

Studies have suggested that the number of study sites might have an influence on effect size.15 Single-centre trials tend to include more homogeneous populations within the (highly selected) study eligibility criteria, and they are more likely than multicentre trials to perform well in experienced teams of providers with expertise in the specific technical or other aspects of an intervention.15 However, single-centre trials risk overestimating effect sizes.15

Trial size also is known for being inversely associated with bigger effect estimates.16 This phenomenon could be a reflection of smaller trials’ being prone to reporting bias: studies with statistically significant results tend to be published more frequently than those with non-significant results.17 However, the larger the trial, the greater the probability that results are published.18 Furthermore, smaller trials might tend to be more susceptible to overall bias assessed by the Cochrane risk-of-bias tool.18

Why this review is needed

Bias domains from the Cochrane risk of bias tool (selection bias, performance bias, detection bias, attrition bias and reporting bias), and further centre status, trials size and source of funding risk positively influencing an experimental treatment if they are found inadequate or unclear.11 12 16 18–20 A meta-epidemiological study assessing binary outcomes showed that ORs from trials with unclear or inadequate concealment were on average 30% lower (ie, more beneficial to the intervention) than ORs from trials with adequate methodology (combined ratio of ORs=0.70, 95% CI 0.62 to 0.80).7 Results for binary outcomes may not be extrapolated to trials assessing continuous outcomes because such trials usually differ in medical condition, risk of bias, sample size and statistical analysis.21

In spite of this knowledge, systematic reviews and meta-analyses still provide summary data to policymakers, healthcare professionals, patients and their relatives to make decisions about the patients’ treatment and well-being. It is interesting to explore whether these systematic reviews and meta-analyses provide the true effect estimates, and if all these groups are making decisions on the basis of spurious (biased) effect estimates. The validity of systematic reviews and meta-analysis is depending on methodological quality and unbiased dissemination of trials.22 The association between methodological components and the overestimated treatment effects will be determined in this meta-epidemiological study, which will enable us to provide guidance on how to interpret OA trials in systematic reviews based on empirical evidence from Cochrane reviews.

Objectives

Our objective is to evaluate the association between estimates of treatment effects with different study characteristics including both well-recognised domains as well as novel bias-related aspects in meta-analyses of OA trials.10 11 15 21 23 24

Methods

Protocol and registration

Our protocol is registered on PROSPERO (CRD42013006924); our protocol manuscript conforms to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews and meta-analyses.25

Eligibility criteria

Only systematic reviews of randomised or controlled trials available will be used in this analysis. Only systematic reviews that compare experimental interventions with sham, placebo or no intervention control in patients with OA and which use a patient-reported pain measure as an outcome will be considered eligible. Moreover, only trials where patients in both the intervention and the control group receive the same treatment except from an add-on in the intervention group will be eligible for inclusion. Reviews will be excluded if no relevance is found when reading the title and abstract. Further exclusion will be made if the review did not report a meta-analysis on pain—preferably a meta-analysis showing overall pain, including at least two trials. Two reviewers will independently evaluate the reports for eligibility, and any disagreements will be resolved by discussion or by involvement of a third reviewer. Data will be eligible from the trials, which are included in eligible reviews and meta-analyses.

Search for meta-analyses and inclusion of trials

We will search the Cochrane Database of Systematic Reviews via the Cochrane Library for osteoarthritis reviews using a combination of MeSH terms and natural language vocabulary (see table 1). The latest update of the Cochrane review will be used. Eligible trials will be identified from the reference lists of the published Cochrane reviews.

Table 1

Search Strategy, Cochrane Library via Wiley Issue 4, 2014

Search	Terms
#1	MeSH descriptor: [Osteoarthritis] explode all trees
#2	osteoarthr*:ti,ab
#3	degenerative near/2 arthritis:ti,ab
#4	(#1 or #2 or #3)

Risk of bias in individual studies

Cochrane risk of bias tool

The risk of bias within each full-text trial will be assessed using the domains of the risk of bias tool as recommended by the Cochrane Collaboration,26 which comprise methods for sequence generation and maintaining allocation concealment, blinding and management of incomplete outcome data. Each domain will be rated as adequate, inadequate or unclear risk of bias (see table 2).

Table 2

Risk of bias assessment table

Bias domain	Bias item	Review authors judgment (adequate, inadequate or unclear)
Selection bias	Sequence generation	Sequence generation will be regarded as adequate if a random approach in the sequence generation process—referred to as a random number table, a random computer-generated number, coin tossing, drawing of lots, shuffling cards or throwing dice–is described. Date of inclusion or admission, or record number of clinic/hospital will be considered inadequate. Deficient information about the process will be considered as unclear
	Allocation concealment	Allocation concealment will be regarded as adequate if sequentially numbered, sealed, opaque envelops, numbered or coded medical containers or a centralised randomisation is applied. If it is possible to predict the assignment, allocation concealment will be regarded as inadequate. Therefore, date of birth, application of an open random allocation schedule, or any other explicitly unconcealed procedure will be regarded as inadequate. The allocation concealment will be regarded as unclear if the method of concealment is not available
Performance bias	Blinding of participants	Blinding of patients will be regarded as adequate if intervention and control treatment are described as indistinguishable, or if it is unlikely that the blinding could have been broken for both patients and data collectors. Deficient information about the blinding process will be considered as unclear
Detection bias	Blinding of key study personnel	Blinding of key study personnel will be regarded as adequate if intervention and control treatment are described as indistinguishable, or if it is unlikely that the blinding could have been broken for data collectors and key study personnel. Deficient information about the blinding process will be considered as unclear
Attrition bias	Incomplete outcome data	Data will be regarded as adequate (complete) if there are no missing outcome data or if the missing data constitute less than 10% of total number of participants at baseline and if no differences in reasons for dropout were observed between the groups. Further, outcome data will be considered adequate if missing data have been imputed using appropriate statistical methods like intention to treat (ITT). Deficient information about the blinding process will be regarded as unclear
Other bias	Centre status	A trial will be considered a multicentre trial if more than one centre is involved. In case of missing information, the trial will be classified as multicentre if it reports both several ethics committees and different affiliations of authors. If the report stated both a single ethics committee and a single author affiliation, the trial will be classified as a single centre. Information about single- and multicentre trials will be extracted from the text, statements, author's affiliations, and acknowledgement in every included trial
	Trial size	To judge whether a trial is small or large, we will use a threshold of 128 participants. If the total number of randomised participants is less than 128 patients (according to the Cochrane review), the study will be regarded as a small trial; trials with ≥128 participants will be referred to as large trials. The threshold of 64 participants in each group corresponds to a reasonable power (80%) to detect a standardised mean difference (SMD) ≥ 0.534
	Funding	Trials will be specified either as being funded by for-profit funding or non-profit funding. Non-profit funding includes money received from both non-profit organisations (eg, Internal hospital funding and governmental funding) and not funded trials. For-profit organisations will be defined as companies that might acquire financial gain or loss depending on the outcome of the trial. Further, trials with a mix of for-profit and non-profit or if the funding is not reported, will be considered as for-profit funded. Funding is defined as including provision of manpower (authorship, statistical analysis or other assistance), study materials (drug, placebo, assay kits or similar materials), or grants.27 Sources of funding will be extracted from the text, statements of sources of support, authors’ affiliations, acknowledgments and trial registration, if available

Other risk of bias items

In the course of meta-epidemiological studies, other sources, such as significant discrepancies between single versus multicentre trials,15 small versus large trials18 and source of funding27 arises as possible risk of bias domains (see table 2).

Data collection process and data items

We will use a systematic, standardised data extraction approach to gather information from all eligible OA studies. The primary outcome (pain) will be abstracted from overall pain reported in the Cochrane review’s first pain measure forest plot. Outcome will be collected at the time point closest to 12 weeks’ follow-up. Two reviewers will independently extract data from the trials.

We anticipate that pain will be measured in various ways, and thus different instruments will be applied to monitor the changes during any given trial period.28 Commonly pain is measured by either a visual analogue scale (VAS) or a numeric rating scale (NRS) and less commonly by more complex item-based instruments (eg, WOMAC29 KOOS30 and ICOAP31).

‘Systematic review-level’ data extraction

At the level of the systematic review, we will extract data on review ID, author, year of publication and accumulated trial size combined in the meta-analysis (ie, NTotal, NI and NC). Type of intervention will be categorised according to the suggested ‘sequential, pyramid approach to management of OA’ by Dieppe and Lohmander32 and the Osteoarthritis Research Society International (OARSI) recommendations33 (see table 3, OA intervention categories).

Table 3

Intervention categories

Non-pharmacological modalities of treatment (NP)	Pharmacological modalities of treatment (P)	Surgical modalities of treatment (S)
General information and advice (education, regular contact with caregiver, lifestyle alterations, etc.) Exercise and therapy (physical and occupational therapy, aerobic, muscle strength, ROM training, water exercise, etc.) Weight loss Walking and other aids (eg, canes, wheeled walkers, assistive technology such as orthoses, braces, insoles) Thermal modalities, transcutaneous electrical nerve stimulation (TENS), acupuncture, etc
	Paracetamol (acetaminophen) NSAIDs (topical or oral) Intra-articular injections with corticosteroids or hyaluronate Nutraceuticals (eg, glucosamine, chondroitin sulphate, rosehip powder) Opioids (weak: tramadol, codeine etc.; stronger: morphine, etc)
		Surgery (joint preserving; osteotomy and resurfacing and partial or total joint replacement)

NSAIDs, non-steroidal anti-inflammatory drugs; ROM, range of motion.

‘Trial-level’ data extraction

At the trial level, we will assign studies a trial ID and extract information on author name, year of publication, type of pain measure, type of intervention and type of OA condition. From the forest plot available in the included review, we will extract the following study-level covariates: mean values (mI and mC), SDs (SDI and SDC) and size of the trials.

The size of the study will be handled in two ways: (1) trial size according to the meta-analysis (ie, forest plot, nI and nC), and (2) trial size according to the original (ITT) population according to the trial report (NI and NC). Further, we will assess the following characteristics from the trials: attrition rate during the trial (aTotal, aI and aC), and trial duration (weeks). Further, we will extract information on the comparator groups applied in each trial, whether placebo/sham or waiting-list/nothing.

Data synthesis

Treatment effect sizes will be expressed as SMD by dividing the difference in mean values available from the forest plots by the SD. Negative effect sizes will indicate a beneficial effect of the experimental intervention (ie, pain reduction). We will register whether the data are expressed as values from follow-up, change from baseline or a mix, as studies have shown that there is no relevant difference between follow-up and change data SMDs in meta-analysis.35 To explore whether the choice of pain instrument (ie, VAS, NRS or item-based scores) interacts with the apparent treatment effect, we will also combine SMDs derived from each of these instrument categories, in a stratified meta-analysis. This will reveal whether there are any relevant differences in effect sizes depending on the pain-instrument chosen. To address the effects of trial size in addition to the categorisation of small and large trials we will do funnel plots based on sample sizes. If small study effects are present, funnel plots will be asymmetrical.

Statistical analysis

To empirically assess the risk of bias in the treatment effects in reviews, we will perform stratified analyses across trials according to all the different extracted trial characteristics and derive p values for interaction between trial characteristics and treatment effect. Within each systematic review, the ES for trial effects according to the different risk of bias assessments will be estimated by using a random effect meta-analysis model. The differences between the pooled estimates will be derived and combined using random effect meta-analysis fully allowing for heterogeneity, to measure the variability in bias estimates, expressed in τ2. A relevant study-level covariate is defined as one that decreases the between-study variance (τ2, estimated as Tau-squared) as a consequence of inclusion in the mixed-effects statistical model. All the statistical meta-regression analyses will be based on REstricted Maximum Likelihood (REML; ie, random-effects) models36; models will be developed and analysed using SAS software (V.9.2, by SAS Institute Inc, Cary, North Carolina, USA).

Ethics and dissemination

We believe that the findings of this meta-epidemiological study will have important implications for future research strategies and implementation of study conclusions. Meta-analyses and randomised controlled trials provide the most reliable basis for treating patients with OA. However, even though the influence of bias items is well known and described, authors tend to forget that biased trials included in systematic reviews will cause meta-analyses to be biased.37

Therefore, it is essential that those who conduct systematic reviews be familiar with the potential biases within primary studies and how such biases could influence review results and the ensuing conclusions. Still, many Cochrane review authors are reluctant or fail to incorporate the risk of bias assessment in their analysis and conclusions,38 compromising the trustworthiness of results from meta-analysis derived from these reviews. Although the Cochrane Collaboration has changed the way it assesses bias in included trials5 owing to Ken Schulz's work on bias assessment,23 the Cochrane risk of bias tool in 2008,4 AMSTAR (Assessing the methodological quality of systematic reviews checklist),39 and the grading of recommendations assessment, development and evaluation (GRADE) approach,40 meta-analyses might still overestimate the effect estimates from biased studies.

As previously described by Nüesch et al16 the presence and extent of small study effects in OA research might distort the overall evidence from meta-analyses. Thus the influence of small trials on estimated treatment effects might be considered a novel risk of bias domain that needs to be addressed in the future. However, one of the reasons why small studies may give different answers from larger studies is that the participants in small studies may be more homogeneous than those in larger studies. Thus changes with treatment may be more similar, and if they are a more severely affected group the potential for gain may be larger. This work will facilitate consensus on the need for mandatory sensitivity analyses before making conclusions of a meta-analysis if the overall result is not consistent with those of the largest trials.18

After two decades of assessing risk of bias, it is clearly time we examine whether the tool for assessing risk of bias is complete38 What items should be added or removed? Which items have the greatest impact on estimates of effect? This study will examine the evidence in CMSG reviews using patient reported outcomes from OA trials as an example and explore the true implications on effect estimates behind the possible shades of bias. As this study collects no primary data, no additional formal ethical assessment and informed consent are required.