Quality of systematic reviews: an example of studies comparing artificial disc replacement with fusion in the cervical spine.

Cervical artificial disc replacement (C-ADR) is now an alternative to anterior cervical discectomy and fusion (ACDF). Many studies have evaluated the efficacy of C-ADR compared with ACDF. This led to a series of systematic reviews and meta-analyses to evaluate the evidence of the superiority of one intervention against the other. The aim of the study presented here was to evaluate the quality of these reviews and meta-analyses. Medline via Ovid, Embase, and Cochrane Library were searched using the keywords: (total disk replacement, prosthesis, implantation, discectomy, and arthroplasty) AND (cervical vertebrae, cervical spine, and spine) AND (systematic reviews, reviews, and meta-analysis). Screening and data extraction were conducted by two reviewers independently. Two reviewers then assessed the quality of the selected reviews and meta-analysis using 11-item AMSTAR score which is a validated measurement tool to assess the methodological quality of systematic reviews. Screening of full reports of 46 relevant abstracts resulted in the selection of 15 systematic reviews and/or meta-analyses as eligible for this study. The two reviewers' inter-rater agreement level was high as indicated by kappa of >0.72. The AMSTAR score of the reviews ranged from 3 to 11. Only one study (a Cochrane review) scored 100% (AMSTAR 11). Five studies scored below (AMSTAR 5) indicating low-quality reviews. The most significant drawbacks of reviews of a score below 5 were not using an extensive search strategy, failure to use the scientific quality of the included studies appropriately in formulating a conclusion, not assessing publication bias, and not reporting the excluded studies. With a significant exception of a Cochrane review, the methodological quality of systematic reviews evaluating the evidence of C-ADR versus ACDF has to be improved.

A systematic review is an attempt to collect ‘all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question’ (Cochrane Handbook for Systematic Reviews of interventions, handbook.cochrane.org, Section 1.2.2). If the results of the individual studies are combined to produce an overall statistic, this is usually called a meta-analysis. Systematic reviews and meta-analyses inform medical and health practitioners and are highly regarded as informative and authoritative in informing decision-makers who formulate the guidelines for practice. Unfortunately, not all systematic reviews and meta-analyses follow the robust methodology of systematic reviewing that are publicised by Cochrane Collaboration and other organisations, Universities and Research Centres. It is essential to implement scientific methods in evaluating the evidence available in the clinical trials and other studies that are reviewed and this should be complemented by transparent reporting of the methods used. There is a need to alert clinicians and systematic reviewers to the indispensable quality requirement of systematic reviews and specifically in evaluating interventions in which randomisation of patients is challenging in clinical trials. Such interventions that are very difficult to randomise are surgical procedures. In this study, quality of systematic reviews of surgical intervention at the cervical spine will be assessed to provide an example of problems associated with systematic reviewing and meta-analysis of these interventions.

Cervical artificial disc replacement (C-ADR) is now an alternative to anterior cervical discectomy and fusion (ACDF). Some biomechanical studies claim that C-ADR is significantly superior than ACDF in terms of restoring sagittal profile particularly cervical lordosis as well as offering a better range of motion in the caudal segment compared to cranial segment (1). Many other studies have evaluated the efficacy of C-ADR compared to ACDF in a range of clinical and patient outcomes, including pain, quality of life, and fusion rate. This led to a series of systematic reviews and meta-analyses to evaluate the evidence of the superiority of one intervention over the other. Therefore, the aim of the study presented here was to evaluate the quality of these reviews and meta-analyses using an up-to-date assessment tool.

Methods

Medline via Ovid, Embase, and Cochrane Library were searched using the keywords: (total disk replacement, prosthesis, implantation, discectomy, and arthroplasty) AND (cervical vertebrae, cervical spine, and spine) AND (systematic reviews, reviews, and meta-analysis). The initial search was conducted on 18 August 2013 and updated on 02 February 2015. Eligibility criteria, which were applied by two reviewers independently (OAT, KA), were: the article should be a systematic review and/or meta-analysis of randomised controlled trials (RCTs) or cohort studies that compared C-ADR with ACDF at the cervical region for one or two levels and for follow-up period of at least more than 6 months and should be published in a peer-reviewed journal. No time or language limits were applied in the selection process. Two reviewers (OAT, HE) then assessed the quality of the selected reviews and meta-analyses using the 11-item AMSTAR score which is a validated measurement tool to assess the methodological quality of systematic reviews (2). The agreement between the two reviewers was tested using the fixed-margin kappa. The 11 items are summarised as follows:

The research question and inclusion criteria should be established before the conduct of the review.

At least two reviewers should extract the data independently and a consensus procedure to resolve their disagreement should be in place.

At least two electronic sources should be searched.

The authors should state that they searched for reports regardless of their publication type.

A list of included and excluded studies should be provided.

The characteristics of the included studies should be provided.

The scientific quality of the included studies should be assessed and documented.

The scientific quality of the included studies should be used appropriately in formulating conclusions.

The methods used to combine the findings of studies should be appropriate (Need to assess homogeneity).

The likelihood of publication bias should be assessed.

The conflict of interest should be declared and included.

The two assessors had to decide whether the systematic review or the meta-analysis meets the criterion (score 1) or cannot decide on an item or that the review does not meet the criterion (score 0).

Results

The initial search produced 636 citations of which 49 were duplicates. Screening of full reports of 46 relevant abstracts resulted in the selection of 15 eligible systematic reviews and/or meta-analyses. All selected articles were published in English. The two reviewers’ inter-rater agreement level was high, as indicated by kappa of >0.72 (3). In other words, the two assessors reached perfect agreement in 12 out of the 15 systematic reviews assessed. The AMSTAR score of the reviews ranged from 3 to 11 (see Table 1). Only one study (a Cochrane review) scored 100% (AMSTAR 11). Five studies scored below 5 (AMSTAR) indicating that these reviews are either of low quality or failed to report their methodology in selecting and critically appraising the studies included. The most significant drawbacks of reviews of a score below 5 were: not using an extensive search strategy, failure to use the scientific quality of the included studies appropriately in formulating a conclusion, not assessing publication bias, and not reporting the excluded studies. The AMSTAR items that were consistently missing were: 5 (86.7% of the reviews did not report excluded studies), 8 (66.7% of the reviews failed to account for the quality of articles when formulating conclusions), 10 (80% of the reviews did not attempt to measure publication bias), and 11 (80% of authors failed to disclose or report any conflict of interest).

Table 1

Quality assessment of included systematic reviews

			AMSTAR item (see text)											Total score	No. of agreements	Mean score
Ref.	Year	Assessor	1	2	3	4	5	6	7	8	9	10	11
9	2012	A	1	1	1	1	1	1	1	1	1	1	1	11	11	11
		B	1	1	1	1	1	1	1	1	1	1	1	11
18	2012	A	1	1	1	1	0	1	1	1	1	1	1	10	11	10
		B	1	1	1	1	0	1	1	1	1	1	1	10
11	2012	A	1	1	1	1	1	1	1	0	1	1	0	9	11	9
		B	1	1	1	1	1	1	1	0	1	1	0	9
17	2010	A	1	1	1	1	0	1	1	0	1	1	1	9	11	9
		B	1	1	1	1	0	1	1	0	1	1	1	9
7	2013	A	1	1	1	1	0	1	1	0	1	0	0	7	11	7
		B	1	1	1	1	0	1	1	0	1	0	0	7
16	2013	A	1	1	1	1	0	1	1	0	0	0	1	7	7	7
		B	1	1	1	0	0	1	0	0	1	1	1	7
14	2012	A	1	1	1	0	0	1	1	1	1	0	0	7	11	7
		B	1	1	1	0	0	1	1	1	1	0	0	7
6	2011	A	1	1	1	1	0	1	1	0	1	0	0	7	11	7
		B	1	1	1	1	0	1	1	0	1	0	0	7
13	2011	A	1	1	1	1	0	1	1	1	0	0	0	7	11	7
		B	1	1	1	1	0	1	1	1	0	0	0	7
4	2010	A	1	1	1	1	0	1	1	0	0	0	0	6	11	6
		B	1	1	1	1	0	1	1	0	0	0	0	6
15	2012	A	1	1	1	0	0	1	1	1	1	0	0	7	9	6
		B	1	1	0	0	0	1	1	0	1	0	0	5
5	2012	A	1	1	0	1	0	1	1	0	0	0	0	5	11	5
		B	1	1	0	1	0	1	1	0	0	0	0	5
8	2012	A	1	1	0	1	0	1	0	0	1	0	0	5	11	5
		B	1	1	0	1	0	1	0	0	1	0	0	5
10	2012	A	1	0	0	1	0	1	0	0	1	0	0	4	11	4
		B	1	0	0	1	0	1	0	0	1	0	0	4
12	2013	A	1	0	1	0	0	0	0	0	1	0	0	3	9	2
		B	1	0	0	0	0	0	0	0	0	0	0	1

Percent of overall agreement of the two assessors was: 0.799999. Fixed-marginal kappa: 0.74.

Discussion and conclusion

With the significant exception of Cochrane reviews, the methodological quality of systematic reviews evaluating the evidence of C-ADR versus ACDF has to be improved specifically in transparent reporting. It is clear that conflict of interest disclosure might have been disclosed to journal editors but failure to report this in the review will lead to lower score in AMSTAR and the readers might question the bias of authors. More significantly, reporting of excluded studies, with reasons of exclusion, in systematic reviews is an essential part of the reviewing process, and authors of systematic reviews and meta-analyses should be encouraged to report this as this assures the readers and the users of the results of the systematic reviews that the article selection process is robust and unbiased. There are, of course, some challenges in the RCTs of surgically interventions including the limitation to select a random sample of patients, the impracticality of blinding surgical interventions. Nevertheless, systematic reviews have to apply rigorous methodology to inform clinicians, patients, and guidelines decision bodies.

Recommendations

It is essential to follow the guidelines of Cochrane Collaboration, or an equivalent organisation such as the Centre for Reviews and Dissemination at York, when conducting systematic reviews or meta-analyses. There are many resources available for training and consulting whenever a research team plans to do a systematic review. It must be acknowledged that the systematic review is similar to any other scientific articles and can be of varying quality. Systematic reviewing is team work and a team of three or more is usually required with at least one statistician included in the team. The team has to have an expertise in the topic reviewed and has to carry the systematic review according to a pre-planned protocol. The work usually takes 9–12 months to complete depending on the number of articles retrieved. The Centre for Reviews and Dissemination at York summarise the steps to conduct a systematic review as follows:

Clear stated objectives.

Pre-defined eligibility criteria for studies.

A systematic search of literature with at least two databases searched.

Assessment of validity of findings (e.g. risk of bias).

Systematic presentation and synthesis of evidence.

We also recommend the use of PRISMA chick list (www.prisma-statement.org/2.1.2 - PRISMA 2009 Checklist.pdf) and AMSTAR assessment tool (www.amstar.ca/Amstar_Checklist.php) when conducting systematic reviews.