A brief overview of systematic reviews and meta-analyses.

Systematic reviews (SRs) are performed to acquire all evidence to address a specific clinical question and involve a reproducible and thorough search of the literature and critical appraisal of eligible studies. When combined with a meta-analysis (quantitatively pooling of results of individual studies), a rigorously conducted SR provides the best available evidence for informing clinical practice. In this article, we provide a brief overview of SRs and meta-analyses for anaesthesiologists.

INTRODUCTION

The first tenet of evidence-based medicine (EBM) is that clinical decisions should be influenced by all relevant high-quality evidences, as opposed to select studies.[1] Systematic reviews (SRs) are aimed at acquiring all evidence to address a specific research question and involve a reproducible and thorough search of the literature and critical evaluation of eligible studies. This is different than narrative reviews, which allow authors to highlight the findings from select studies. The salient features and differences between systematic and narrative reviews are described in Table 1. There are several other types of reviews such as scoping reviews, rapid reviews, systematised reviews, overviews and critical reviews, which have their own strengths and limitations.[2] For the purposes of this overview, we will focus on SRs of therapeutic interventions.

Table 1

Differences between a narrative review and a systematic review

DEFINITIONS OF SYSTEMATIC REVIEW AND META-ANALYSIS

A rigorously performed SR identifies all empirical evidence that meets pre-specified eligibility criteria to answer a specific clinical question using explicit, systematic methods to minimise bias and provides reliable findings to inform evidence-based clinical care.[3] A SR can be either qualitative, in which eligible studies are summarised, or quantitative (meta-analysis) when data from individual studies are statistically combined. Not all SRs may result in meta-analyses. Similarly, not all meta-analyses may have been preceded by a SR, though this element is essential to ensure that findings are not affected by selection bias.

PURPOSE OF SYSTEMATIC REVIEWS AND ITS POSITION IN THE HIERARCHY OF EVIDENCE

Large volumes of research are conducted and published every year, and at times with conflicting results.[4] Differences in findings lead to confusion among physicians, patients and policymakers. Resolving such situations necessitates a comprehensive and transparent synthesis of all available literature that evaluates the quality of evidence and when present, explores possible reasons for discrepancies. SRs are best suited to address such a need, especially when combined with meta-analysis. The second tenet of EBM is that there is a hierarchy of evidence, from research designs at higher risk of bias, to study designs at lower risk of bias.[5] Because of the advantages described in Table 1, well-conducted SRs of high quality randomised controlled trials (RCTs) are placed at the highest level in the hierarchy of evidence.[6]

COMPONENTS OF SYSTEMATIC REVIEW

Conduct of a SR starts with a clinical question and a protocol with clear identification of objectives, outcomes and a well-laid out plan for analysis and reporting of results. It is highly desirable to register protocols in a publicly available database, such as PROSPERO.[7] This allows for transparency and increases confidence among readers in the study results. The stages in the conduct of SRs are highlighted in the following section and a summary is provided in Table 2.

Table 2

Summary of practical steps in the conduct of a systematic review

Formulating a clinical question

A specific and clearly thought out question is the first step towards completing a SR. The question is generally framed in the population, intervention, comparator and outcomes (PICO) format and influences the scope of review.[8] A review with a narrow definition of PICO elements (e.g., effect of dalteparin on deep vein thrombosis (DVT) prophylaxis after hip replacement surgery) will apply to a much more specific patient population than a review with broader criteria (e.g., effect of low molecular weight heparin medications on DVT prophylaxis after joint replacement surgeries).

The decision on the type of study design(s) to be included, RCTs or observational studies or both, is made based on the review question. In SRs of therapeutic interventions, RCTs are desirable over observational studies. Observational study designs are considered when there are no RCTs, or the available RCTs are of poor quality or if the review question pertains to rare events. It is not uncommon for SRs to include only English-language articles, but this risks selection bias (English-language bias) and including non-English language articles is highly preferable.[9]

A detailed review protocol that identifies study objectives, outcomes, methods of study selection, data extraction, analysis and reporting should be prepared before the conduct of the review.[10] This promotes more rigorous SRs as researchers are required to carefully think through the details of their study before proceeding, and ensures that any deviation from the intended methodology is transparent to stakeholders. A research team consisting of one or more content experts, a medical librarian, a methodologist and a statistician, along with other stakeholders (e.g., patients) ensures the requisite skill set for design, conduct and reporting of the review.

Searching the literature

This process is ideally conducted by a research librarian. If unavailable, a librarian's assistance should be sought to design and complete the literature search. Common medical databases for searching include Medline, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL); however, based on the research question, additional databases may be appropriate. The search strategy for a SR is based on the research question (PICO) and may vary slightly based on database-specific indexing practices; ideally, all dataset-specific search strategies will be included as appendices when SRs are published.[10]

Literature search strategies will typically use a combination of medical subject heading (MeSH) terms and free-text keywords. For example, if we are interested in the concept of anaesthesia through the breathing of gas, the MeSH term for this concept is ‘Anaesthesia, Inhalation,’ which will capture publications that have had this subject heading assigned to them by a database indexer. In addition, we might search in the title and abstract field of citations for a specific free-text term. For example, a free-text phrase we might use is ‘an?esthe* adj3 (inhal* or insufflation).’ The question mark will capture British or American spelling variations while the asterix will capture all variations of the root word (inhaled, inhalation, inhaling, etc.). The adjacency operator (adj3) requires that the two words appear within three words of one another so that an article containing a phrase such as ‘anaesthesia delivered by inhalation’ will be captured. By combining the results of the two sets of searches using the “OR” operator, a larger set of potentially relevant records is retrieved. A balance between sensitivity (capturing all possible evidence pertaining to review question) and specificity (retrieving more relevant studies) is necessary to achieve a comprehensive and practical search. After removal of duplicates, a final list of citations (titles and abstracts) from all reviewed databases is compiled.

Selection of relevant studies

Based on the selection criteria, the articles obtained through literature search are taken through a two-stage process of screening and selection. The first stage is carried out on titles and abstracts in which a more sensitive strategy is adopted to clearly exclude unrelated articles. For the second stage, all potentially relevant studies are acquired in full-text to complete final adjudication of eligibility. Both these stages should ideally be performed independently and in duplicate by teams of reviewers to provide assurances regarding reliability. Discrepancies in selection are resolved by arbitration. Consensus exercises between reviewers at the start of each stage help promote a high rate of agreement. The kappa statistic can be calculated to inform agreement between reviewers, and is interpreted as follows: Almost perfect agreement (0.81–0.99), substantial agreement (0.61–0.80), moderate agreement (0.41–0.60), fair agreement (0.21–0.40), slight agreement (0.01–0.20) and chance agreement (0).[11] At the end of this process, a final list of articles to be included within the SR is obtained.

Data extraction

This stage involves extraction of individual study data necessary for the analysis and reporting of the review. Extractable data will typically include study design characteristics, risk of bias (RoB) items, participant demographic information and follow-up and study outcomes. Similar to screening, this is performed independently by paired reviewers to avoid errors in the acquisition of data. Clear organisation and recording of the extracted data are necessary to facilitate a review. Missing or unclear data items should be sought from the authors of studies and if unavailable, should be clearly documented. The use of web-based SR software such as DistillerSR™ (http://distillercer.com/), particularly for large reviews, helps promote faster, more organised and efficient study selection and data extraction.

Appraisal of study reporting quality

The reporting quality of a study considers the extent to which aspects of a study's design and conduct are likely to protect against systematic error (deviation from the truth). Systematic errors broadly fall into three domains of selection, measurement and confounding biases. Several tools are available for assessment of the methodological quality of included studies such as the Newcastle-Ottawa scale for non-randomised studies,[12] and the Cochrane RoB tool for RCTs.[13] Quality assessment is required for interpreting results and for grading the strength of evidence. An example of assessing and representing the RoB in individual studies and across studies is available in the cited reference.[14]

Data synthesis (meta-analysis)

It is desirable to explicitly state how data will be synthesised a priori to avoid data mining, and to reduce unsubstantiated claims during the synthesis.[15] Different study designs and outcome measures should be summarised and synthesised separately.

Meta-analysis is a statistical technique used for combining the effect on a common outcome domain (e.g., pain relief) across individual studies. By combining study results, the precision associated with a treatment effect becomes narrower.[3] Findings from meta-analyses are graphically depicted in the form a forest plot [Figure 1],[14] with individual studies and their treatment effect represented horizontally. The vertical line indicates the line of null effect, and effects either to the left or right of the line indicate the benefit of the intervention or control therapy. Individual study effect sizes are shown as a square along with the associated 95% confidence interval, and the pooled estimate is shown as a diamond.

Figure 1

Forest plot depicting the use of rescue analgesia in the early post-operative period in caudal and non-caudal regional techniques in children undergoing inguinal surgeries. Harsha Shanthanna, Balpreet Singh, and Gordon Guyatt, “A Systematic Review and Meta-Analysis of Caudal Block as Compared to Noncaudal Regional Techniques for Inguinal Surgeries in Children,” BioMed Research International, vol. 2014, Article ID 890626, 17 pages, 2014. doi:10.1155/2014/890626

For continuous outcomes that are reported using the same instrument across trials, the pooled estimate can be reported in natural units as a weighted mean difference. The more common situation is that trials eligible for meta-analysis will report outcomes tapping into the same domain (e.g., pain) using different instruments. In such cases, the effect estimates require standardisation before pooling. A common approach is to convert different instruments into standard deviation units, and report the pooled effect as a standardised mean difference (SMD) (a value of 0.2 represents a small difference, 0.5 a medium difference and 0.8 a large difference in treatment effect). However, the SMD is difficult for patients and clinicians to interpret and is vulnerable to baseline heterogeneity of trial patients: greater heterogeneity among pain scores at baseline will result in a smaller SMD versus studies that enrol patients that provide more homogeneous scores, even when the true underlying effect is the same. We have previously published guidance regarding approaches that can be used to pool different instruments in natural units, minimally important difference units, a ratio of means, or the odds or risk of achieving a patient-important level of improvement.[16] For binary outcomes, pooled treatment effects are typically expressed as an odds ratio or risk ratio.

Pooled treatment effects may demonstrate heterogeneity suggesting that an overall pooled effect may be misleading as treatment effects differ between studies. The statistical evidence of heterogeneity can be estimated using Cochrane's Q-test (heterogeneity detected if P ≤ 0.10), and the I2 statistic (a percentage measure of inconsistency) - higher I2 indicates more heterogeneity, hence higher inconsistency.[17] A rough guide for interpreting heterogeneity using the I2 statistic is as follows: 0%–40%, insignificant; 30%–60%, moderate; 50%–90%, significant and 75%–100%, considerable. If heterogeneity is high, it would not necessarily mean abandoning meta-analyses, but caution should be exercised during interpretation of results. Strategies to explore heterogeneity include subgroup analysis (exploring outcomes for predefined subgroups, based on PICO, or RoB, etc.) or meta-regression (predicting outcome according to one or more continuous explanatory variables).[18]

The strength of a SR is the avoidance of selection bias, strengths of meta-analysis include an increase in power that improves statistical precision of treatment effect estimates, and the ability to explore for subgroup effects (although between-study subgroup effects are less credible than within-study subgroup effects). The main limitation of SRs and meta-analysis are related to the quality of included studies – aggregating poor quality studies will lead to a poor quality review.

Reporting the findings

The reporting of SRs should be guided by the preferred reporting items for SR and meta-analysis (PRISMA) for RCTs[19] and Meta-analysis of observational studies in epidemiology (MOOSE) for observational data.[20] The grading of recommendations assessment, development and evaluation (GRADE) approach should be used to evaluate and report the quality of evidence for each outcome reported in a SR (http://www.gradeworkinggroup.org). Resources such as the Cochrane Database of SRs[21] provide access to high-quality SRs on treatment interventions.

SUMMARY

There are 3 tenets of EBM: (1) all relevant evidence should be considered to inform clinical decision-making, (2) there is a hierarchy of evidence that is based on the ability of different study designs to address RoB and (3) evidence alone is never enough (e.g., patient values and preferences). Rigorously conducted SRs and meta-analyses present an invaluable strategy to address the 1st two of these tenants, and are essential to the practice of EBM.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.