Systematic Reviews of the Literature Are Not Always Either Useful Or the Best Way To Add To Science.

Systematic reviews are becoming more popular as a way of doing research; however, not all systematic reviews are clinically useful and sometimes another type of review (scoping, topical, or critical) would be of greater value to the clinical and scientific community. The different types of review and their use are described, illustrated by examples relevant to vascular surgery.

Introduction

Systematic reviews appear to have become increasingly popular in the published literature as often they are considered important sources of clinical evidence. The reason for performing systematic reviews is to add information to what is known on a particular topic. However, all too often such reviews are not clinically useful. Some have estimated that >90% of systematic reviews are clinically useless, while others have suggested that the “gravy train” of systematic reviews constitutes research waste., The reasons for this include registered but unpublished reviews, duplicated or poor methodological reviews, or those addressing questions which are not clinically useful. Do they serve any other purpose? Perhaps they improve the citation or publication record of individual researchers or clinicians or are considered a necessary starting point to a thesis? The former is not a laudable reason, and the latter is a misplaced assumption, when perhaps another type of review, a scoping review, would better summarise the field and identify the knowledge gaps and opportunities for productive investigations. Other times they may add only incremental knowledge rather than new knowledge. In contrast, good quality reviews are valuable to clinicians and guideline committees and can enhance the impact factor of a journal. This may be one of the reasons underlying the steady increase in systematic reviews published in the European Journal of Vascular and Endovascular Surgery (Fig. 1). Scrutiny of the metrics of the 20 systematic reviews published in 2018 indicate the wide range of utility of the reviews to both journal impact factor and clinicians, but six probably could be assigned to “the gravy train” and take up journal pages at the expense of original research.

Figure 1

The increasing number of systematic reviews published in the European Journal of Vascular and Endovascular Surgery. Data show the number of systematic reviews published from 2010 to date on the vertical axis: the 2021 data show reviews recorded in Medline to end August 2021.

The pyramids of clinical evidence all place systematic reviews of randomised controlled trials (RCTs) at the apex, with reviews conducted with support of the Cochrane Collaboration on top. Such reviews are a special feature of the Cochrane Collaboration. For Cochrane reviews there is a guaranteed rigorous search and review methodology, with emphasis on uncovering and reporting potential sources of bias. Therefore, most Cochrane reviews focus on synthesising the data from adequately powered RCTs and they can provide valuable additional information. The scientific quality of any systematic review of RCTs is only as good as the quality of the included studies: small, underpowered RCTs and those of poor methodological quality can provide very misleading results. This risk of misleading information is even stronger in systematic reviews of observational studies where there is inherent patient selection, reporting bias, and overestimation of treatment effects is common: these are the reviews most likely to be of limited clinical usefulness. For reviews focusing on long term outcomes, the number of patients lost to follow up should be assessed as part of the study appraisal. Patient loss to follow up is usually higher in observational studies than in randomised controlled trials, and this can limit the value of observational studies when synthesising evidence for longer term outcomes.

Here the discussion is when it is appropriate to consider undertaking a systematic review, with a few tips for success, when a scoping review would be better than a systematic review, and when a critical or topical review of recent evidence would be more appropriate.

What is a systematic review and when is it useful?

A systematic review is used to marshal, appraise, and synthesise the evidence about a precise clinical question, for example, comparing the clinical effectiveness of two different operations for the same condition. A systematic review should follow standard rigorous methodology and reporting, and there are recommended approaches to minimise bias. The PICO (population, intervention, comparator, outcomes) for the review all require clear and careful definition to ensure limited clinical heterogeneity. Given these criteria, such reviews should not be considered as a quick and easy way of doing research.

There are several different uses of a systematic review.

To synthesise the evidence from adequately powered (large) RCTs, these are likely to be Cochrane reviews.

To synthesise the evidence from observational studies comparing the efficacy of treatments in situations where randomised trials are not possible, for example, the efficacy of e-cigarettes to promote and sustain smoking cessation.

To synthesise the evidence from RCTs and observational studies about a clearly defined important clinical question, to which the answer is not already known and there is no evidence of a similar review being in progress or recently published (by checking PROSPERO and other research registries as well as conference abstracts), for example, is carotid artery stenting still associated with lower stroke risk in asymptomatic patients given the advances in best medical therapy? Careful definition of the PICO in question and quality assessment of included studies are vital. If sufficient suitable studies are identified, meta-analysis should be provided as well as sensitivity analysis for the best quality studies, in cases where there is a wide range of study quality.

To investigate how outcomes have changed over time, to identify whether there has been improvement in outcomes and patient benefit. Meta-regression can be a useful tool. An example here is the recent updating of a 2010 systematic review evaluating the sex specific operative mortality from intact abdominal aortic aneurysm repair, given the advances in both endograft and imaging technology.,

To investigate how factors such as age, sex, ethnicity, and frailty influence clinical outcomes using evidence from both RCTs and observational studies, for example, the influence of age, sex, and contralateral occlusion on stroke and death after carotid endarterectomy or carotid stenting. Presentation of sensitivity analyses to compare information obtained from RCTs vs. observational studies may be illuminating.

To inform clinical practice guidelines about recent developments, for example, pre-emptive procedures to limit type II endoleak after aneurysm repair using evidence from observational studies and/or small RCTs. To avoid bias from small studies, it can be helpful to use a minimum threshold for the number of patients as an inclusion criterion. In the absence of sufficient evidence, this would convert to a topical review.

To obtain parameters for use in modelling studies or for estimating the sample size for a planned large RCT. Examples might include recent changes in amputation rates to inform the provision of services for amputees following the COVID 19 pandemic or current prevalence of abdominal aortic aneurysm to inform the probable effectiveness of population screening programmes.

To identify the range of reported outcomes, for example, for the development of Core Outcome sets or to identify the full range of procedure associated complications.

The timing of systematic reviews

The timing of systematic reviews is important, as the most cited and downloaded reviews address a still controversial topic for which sufficient evidence is available but which do not come too late to be useful, after clinical practice has changed. Illustration of this point comes from analysis of the citation and download rates of systematic reviews published in the European Journal of Vascular and Endovascular Surgery in 2018. The most cited and downloaded review was on how to treat type II endoleak. A review of thrombotic events after endothermal ablation of the great saphenous vein also appeared useful. These issues were clearly worries on many people's minds and were topics likely to influence clinical practice. In contrast, the role of Nordic walking in exercise programmes aroused little interest.

When is a systematic review either not needed or unhelpful?

There are several clear examples of when a systematic review is unlikely to be of clinical value.

When a recent systematic review is already available or in progress (check in PROSPERO and other research registries).

To answer questions that do not concern clinical effectiveness (PICO not applicable), for example what is the best method of measuring the size of large venous ulcers? This needs an overview of measurement methods.

To answer questions, where the answer is already known, for example, is the operative mortality for intact AAA repair in women lower after EVAR or open repair? The answer here can be derived from the interaction analyses in randomised trials and the systematic reviews of sex specific differences.

To use observational data to answer questions that can be answered only by RCTs, for example, what is the diameter threshold for repairing internal iliac aneurysms? This might be the subject of either a topical review if there are new data for the rupture of these aneurysms or a critical review of the literature.

To answer questions where there are no standard interventions or outcomes. An example is provided by the recent review of pre-habilitation interventions before elective aneurysm repair. A scoping review probably would have been more useful.

What is a scoping review and when is it needed?

A scoping review is an exploratory but systematic literature search to find out how much is known about a broad topic or to discover gaps in the evidence, to provide a narrative review without formal meta-analysis. The question(s) being addressed is broader and less specific and also may be more complex and heterogeneous than that in a systematic review. Examples include “Do prisoners have adequate access to vascular services?” or “What is the evidence for shared decision making for critical limb ischaemia?”. It might be used to identify whether a systematic review was necessary. A scoping review can identify specific unanswered questions which can be addressed either with new original research or some that can be answered by a systematic review. At the start of a thesis or other piece of research work, a scoping review often is more useful and less labour intensive than a systematic review.

What is a topical review and when is it needed?

A topical review is an up to date overview of a current hot topic. Topical reviews may present areas that are still developing rapidly and may provide an indication of the future direction of the field. Examples might include the value of high sensitivity troponin assays to guide the management of peripheral arterial disease or methods for local, rather than systemic, antithrombotic therapy.13, 14, 15 As with the previous types of reviews they need to be systematic and thorough, but unlike systematic or scoping reviews, they are guided by the literature and make more use of conference abstracts and grey literature such as scientific and charitable foundation reports, and government or industry reports. The review should report not just specific outcomes but must include the key present and future issues and/or challenges, which need to be addressed. Therefore, such reviews may be needed by government bodies and industry, as well as keeping clinicians informed about emerging technologies and processes.

What is a critical review and when is it needed?

A critical review is both an appraisal and a critique of new data on a topic, which may be either controversial or inconsistent with earlier findings and guidelines. A potential example is if a new, large series providing the diameter of ruptured internal iliac aneurysms indicates that the suggested intervention diameter criterion in clinical guidelines needs to be revised. A real example is the recent population based study from Denmark, which suggests that diabetes is not a factor that protects against the development of abdominal aortic aneurysm, although it may be protective of the development of more proximal aneurysms. The critical review then becomes a critique of the new study set in the context of a critique of the previous evidence (which did not come from population based studies).

How do the processes for the various review types differ?

The processes for these four different types of review are summarised and compared in Table 1. The varying types of review described have different purposes and methodology, but all should be thorough and systematic. There are some other specialist types of review, for example individual patient meta-analyses of randomised trials but these require full access to original data and specialist statistician input.

Table 1

Methodology of systematic, scoping, topical, and critical reviews

Stages	Systematic	Scoping	Topical	Critical
Question	Formulate the precise question	Decide on the broad topic	What is the current knowledge base?	Is the new evidence robust?
Checks before you start	PROSPERO11 and other databases for existing or similar review	Medline search for reviews on the topic	Recent flagship scientific journals for similar reviews	Recent flagship scientific journals for similar reviews
Making the question more detailed	Inclusion and exclusion criteria for relevant studies	Not usually relevant	Only after initial review of the key literature	Not applicable
Search for evidence	Use a minimum of two databases	Use a wide range of databases (to include nursing, social sciences, etc., as necessary)	By keywords in Medline, grey literature including conference and foundation reports	By keywords in Medline or scientific literature conference proceedings for unpublished support
Select and extract evidence	Use a minimum of two researchers	Use a minimum of two researchers	Guided by what you find and limit to the most pertinent reports	Guided by the new evidence
Evidence quality	Needs formal assessment. Sensitivity analysis of good quality studies	Not assessed	Validity of evidence needs discussion	Must be assessed: key part of the critique
Outputs	Usually, data synthesis with meta-analysis	Tables of evidence with narrative synthesis	Key themes and issues	Narrative viewpoint and future data required
Reporting guidelines	PRISMA4	PRISMA extension for scoping reviews17	N/A	N/A

N/A = not applicable.

So what review do you need?

The aim of this paper is to help you decide what kind of review you need to undertake and to discourage inappropriate systematic reviews, which are not likely to be clinically or scientifically useful and divert resources away from research productivity. Finding the gaps in the evidence, to which you can contribute original research, may often be more rewarding than a systematic review.

Funding

None.

Conflict of interest

None.