[Tutorial for performing systematic review and meta-analysis with interventional anesthesia studies].

BACKGROUND AND
OBJECTIVE: The systematic review of randomized clinical trials is crucial to assess the safety and effectiveness of intermediate procedures. The objective of this article is to present a tutorial for the planning and execution of systematic review and meta-analysis of randomized clinical trial studies. CONTENT: The systematic literature review is the type of research that organizes, criticizes, and integrates available evidence published in the health field. Systematization leads to less bias, however, the quality of systematic reviews may not always be perceived due to the way it is described in the articles. The information disclosed in the articles is not always free of bias. The steps for carrying out a systematic review include design, protocol registration, implementation, mathematical analysis of results, and dissemination. PRISMA statement has improved the quality of systematic review reports by providing a list of items to be described, and this article emphasizes the key steps for performing a systematic review of interventions.
CONCLUSION: The evidence generated through a systematic review can provide the clinician with greater confidence in decision making at the moment of clinical practice and optimize the benefits to his patients, serving as a tool to assist managers in making decisions regarding the implementation of new strategies in favor of the public health.

Introduction

Systematic review is a type of literature review that uses pre-defined strategies to minimize bias in the identification and analysis of data from original articles. Data from studies with potential to answer the systematic review question are often used for mathematical calculations. Narrative reviews report data from articles that praise the opinion of experts and are not evidence-based. Systematic reviews are widely known in the health field and receive on average twice as many citations in medical journals compared to narrative reviews.

Using systematic reviews as a decision-making tool in clinical practice is not new and has a worldwide initiative that offers full support for the implementation of systematic reviews, Cochrane Collaboration (http://www.cochranelibrary.com/). The methodology published by Cochrane Collaboration is considered the gold standard for data analysis of studies involving interventions. International efforts have been made to avoid failures in systematic reviews (Methodological Expectations of Cochrane Intervention Review guidelines – MERCIR), as well as in the reporting of systematic reviews publications (The PRISMA Statement).5, 6

Flaws in systematic review publications are noticeable despite efforts to improve their quality.5, 6, 7 It is important that the Brazilian Journal of Anesthesiology publishes a tutorial to allow that possible failures do not interfere with the evaluation of anesthesiologists as they consider the results of systematic reviews at the moment of a clinical decision making before a patient, as well as to enable this knowledge to be used to foster more research of good methodological quality within the specialty in Brazil.

The aim of this article is to present a tutorial for the design and execution of systematic reviews and meta-analysis with anesthesia intervention studies.

Method

A bibliographic and cross-sectional search was carried out through publications of scientific articles found in the database: PubMed (National Center for Biotechnology Information). The following descriptors were used: revision, meta-analysis, and randomized clinical trial. The mesh terms used were: “review” [Publication Type] OR “review literature as topic” [MeSH Terms] OR “review” [All Fields], “Meta-Analysis as Topic” [Mesh], and “randomized controlled trial” [Publication Type] OR “randomized controlled trials as topic” [MeSH Terms] OR “randomized controlled trial” [All Fields]. We consider the information from books that are used as mandatory references by researchers from various fields to develop systematic review projects.

Importance of systematic reviews

Systematic reviews evaluate data from the literature and summarize them critically, elucidate the results that could be opposed when assessed alone in each original article. The use of combined data increases the statistical power of analysis.

The first importance is the reduction of bias in the analysis of literature data due to the systematization and transparency required for its accomplishment. Systematization allows the identification of published articles that is able to answer a question of clinical interest. The answer, without bias, can be used by clinicians in decision making, by health managers in the implementation of public policies to solve health problems, and by patients as some journals use the systematic review data in a section specially designed for the reader.

The second importance is that systematic reviews have been used as an important tool for decision making in clinical practice by all medical specialties. The concepts of evidence-based medicine have grown in recent decades and gained popularity among physicians. The hierarchy of studies allows us to classify systematic reviews and meta-analysis as the highest level of evidence.

The third importance is the identification of failures in both knowledge and suggestions for better accomplishment in future studies. Systematic reviews make critical analyzes of the methodology used in the studies and consider mainly the biases found in the studies analyzed.

Methodological items

Review question

The relevance of a systematic review can be directly understood when analyzing the question that motivated its execution. The question must be specific to enable a response with the least possibility of bias. The current orientation is to use the framework known worldwide as Patient, Intervention, Comparison, Outcome (PICO) for study design. The type of study is not always used in the question and may be omitted. An example in anesthesia may be: how effective is sugammadex (intervention) compared to neostigmine (comparison) for reversal of neuromuscular junction blockers (outcome) in children (patient)?

The patients used in the question serve to guide the search for articles that will be evaluated, as well as to indicate to whom the results of a systematic review are intended.

The intervention should be clearly defined by the reviewer, it is advisable to describe it both in the systematic review design and in the final article. The studies have different potential to arise bias. Therefore, it is possible to create a hierarchical pyramid among them, which demonstrates that the study at the top is considered the best evidence. The nature of randomization makes randomized clinical trial the study that best evaluates the therapeutic intervention.

The comparison is the intervention, the placebo or the therapy with which to compare the intervention. The comparison group can also help identify or exclude articles for use in a systematic review.

The outcome refers to the final result after intervention. Outcomes are the variables of interest to reviewers. Outcomes may be as beneficial as time of discharge and need for analgesic therapy, as they may demonstrate malfunctions, such as mortality and complications related to anesthetic interventions.

Protocol registration

A research protocol should be developed prior to the execution of any systematic review. It is recommended that this protocol be registered in a database that is available to other researchers and health managers, demonstrates the transparency in the process of performing the systematic review. The initial version, as well as the versions with subsequent modifications, is available for analysis to those who wish to know in detail the process of systematic review or for analysis of the methodological quality of these researches.

The protocol registration allows that authors of studies that can answer the systematic review research question to contribute with their data or with the submission of the already published article, that other researchers do not initiate reviews that seek to answer the same question and avoid duplicity of publications, plagiarism, and waste of energy in analyzing articles that may have already been selected, and that the completed and published reviews are easily identified.

Registration can be done in some databases such as: The Cochrane Library (http://www.cochranelibrary.com/); PROSPERO – International prospective register of systematic reviews (https://www.crd.york.ac.uk/prospero/); and CAMARADES – Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (http://www.dcn.ed.ac.uk/camarades/default.htm) for preclinical studies.

Protocol registration in The Cochrane Library is more difficult and in some cases impossible to be completed. Registration in this database initially requires that the research question be registered in a research group of that collaboration. However, some research questions are rejected and the protocol registration is unfeasible. PROSPERO database does not require this preliminary step and does not charge for review registration, which makes the process easier. An example in anesthesia may be seen at http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42015017672.

The protocol can also be published in scientific journals as an article. Journals such as the Systematic Reviews Journal and BMJ Open submit the protocols to external reviewers and there may be modifications as suggested. Publication in journals may be associated with taxation, as well as revisions and suggestions of the submitted text. An example in anesthesia may be seen at https://bmjopen.bmj.com/content/6/5/e010885.long.

The protocol should contain the methodological strategies for carrying out the systematic review and must have at least the following information: the search strategy to identify articles of interest, eligibility criteria, data to be extracted, variables of interest, data analysis and ways to explore heterogeneities. Important items inherent in performing a systematic review may be seen at https://tinyurl.com/systematicr.

Search strategy

The search strategy may be understood as a set of words or terms that are used in a database to identify titles and abstracts of articles with the potential to answer the question that motivated the systematic review.

The words that can be used in each database are different, so it is not possible to describe the format for each base in this narrative review article. The mnemonic STARLITE method suggests some elements that are essential to provide quality to the literary search description, which must be reported in the article that discloses the systematic review. The acronym STARLITE stands for: Sampling strategy, Type of study, Approaches, Range years, Limits, Inclusion and exclusions, Terms used, and Electronic sources.

Sampling strategy refers to the ability to identify all possible original articles; selectivity refers to the identification of relevant articles and the objectivity to maintain the specific focus on a subject of interest.

The type of study refers to the type of design used. Randomized clinical trials are preferred for systematic reviews of intervention due to their lower bias potential. However, other types of studies may be used and confer greater statistical power to the results.

The approach element refers to other sources of scientific articles. The other sources give more scope to the search and quality to the sampling; some examples are: lists of references, other systematic reviews, manual search, contact with experts, and annals of academic events.

Range years refers to the start and end date of the search, as well as to the justification for this range. Search dates should be reported both in the protocol and in the article that will disclose the results of the systematic review.

Limits refer to conditions that may limit the search. The imposed limits may hide original articles if they are very specific, such as: preference for a language, year of publication, preference for a certain age group or anesthetic drug.

The inclusions and exclusions element refers to the characteristic items at a specific scope of interest, such as geographic location, specific clinical scenario or focus on some type of study.

Terms used refer to the words and the synthesis of these words for each database. Terms are different for each database. The terms of Medline database via Pubmed may be searched at http://www.ncbi.nlm.nih.gov/mesh/. The terms can be used either alone or together and form the search strategy itself, as shown in Table 1.

Table 1

Search strategy used in a systematic review that evaluated neuraxial anesthesia for orthopedic surgeries.

Data base	Search strategy
	(Therapy/Broad[filter])a
	AND
Medline (via Pubmed)	(Anesthesia, general [MeSH Terms] OR anesthesia, inhalation [MeSH Terms] OR anesthesia, intravenous [MeSH Terms] AND anesthesia, conduction [MeSH Terms] OR anesthesia, epidural [MeSH Terms] OR anesthesia, spinal [MeSH Terms])
	AND
	Orthopedics [MeSH Terms] OR Orthopedics

Therapy [Subheading] OR therapeutics’ [MeSH Terms] OR Therapy [Text Word].

The electronic sources element refers to the databases and the platforms of access to these bases. It is recommended that more than one database be used, as there is no database for all journals in the world. Medline and Embase databases, when used in the same systematic review, offer a large search range and increase the possibility of identifying articles that answer the systematic review question.

The search process and identification of original articles for inclusion in systematic reviews should be part of the result and shown in graphic form according to the PRISMA statement recommendation. A hypothetical model can be seen in Fig. 1. If there are other systematic reviews in the area of interest, their references should be evaluated for identification of relevant original articles and this information should also be reported to readers as shown in Fig. 1.

Figure 1

Flowchart of the original articles included.

Eligibility criteria

The criteria for identifying and selecting original articles are known as eligibility criteria. Such criteria are defined prior to the execution of the systematic review and differentiate it from a narrative review.

Eligibility criteria are a combination of important aspects of the review question associated with specifications concerning the type of study used to answer the review question. It is recommended that the reviewer specify the type of study, type of participant, and type of intervention.

Randomized clinical trials were historically designed to evaluate interventions with low possibility of bias. Systematic reviews should prioritize studies with less possibility of bias. Non-randomized studies are usually performed with a large number of participants which may result in greater statistical accuracy, although results may be lost in accuracy due to the biases inherent in those studies. The most recent systematic reviews make room for observational studies and perform a broad analysis of their quality. Reviewers should report the type of study they will include and the reasons that led them to make that choice.

The type of participant and intervention is a key element in creating the question that will guide the systematic review and should be reported in this section of the systematic review.

Data extraction

The data from the original articles selected should be extracted so that systematization and meta-analysis can be performed. Data collection can be a source of bias due to two conditions: first, the error in the transcription or in the collection of relevant information to answer the systematic review question; and second, by the extraction process due to the subjectivity and interpretation of the reviewer.

Biases can be avoided when two reviewers independently select and extract data with a subsequent consensus meeting to solve discrepancies.17, 18 Discrepancies can also be solved by a third reviewer. Articles initially selected for full text reading and then excluded should have the reasons explained in the review process. A flow chart should be made to facilitate the selection and inclusion process, as well as the reasons for exclusion of articles, and the final number of articles that have undergone qualitative and quantitative analysis (Fig. 1).

Variables of interest

Variables can be classified as primary and secondary. Primary variables directly answer the review question. Secondary variables aid the review response, but do not respond directly. An example in anesthesia can be given considering an analysis of ultrasound accuracy for peripheral nerve block, in which the primary variable may be the accuracy of the device and the secondary variable the frequency of failures.

Variable data may be described in the article of interest, but in an inadequate way to be used in a systematic review. It is recommended that the reviewers contact the study authors and request the data in order to be able to arrive at the answer for the review question, and when this is done it should be reported in the body of the systematic review.

Data analysis and heterogeneity

The mathematical analysis of a systematic review is known as meta-analysis. It serves to represent the statistical method used to evaluate the results of the studies that will be integrated into the systematic review. The term “Meta-analysis” was registered as a descriptor in health sciences in the 1990s and allows the identification of this type of research published in the Medline and Lilacs databases. A search was carried out at the beginning of the construction of this narrative review article to identify this descriptor in http://decs.bvs.br/ and three terms were identified: Meta-Analysis as Topic, Meta-Analysis, and Network Meta-Analysis.

It is important to emphasize that a mathematical evaluation should only be performed if there is homogeneity in the interventions evaluated, i.e., the included studies characteristics should be similar and the study variables should be conceptualized and evaluated in the same way. If the systematic review evaluates pneumonia after general anesthesia and three included studies evaluate the variable differently (e.g., by chest X-ray, by tomography, and only by clinical evaluation), the studies cannot be mathematically evaluated and in this case the review will only have a qualitative and descriptive evaluation.

A meta-analysis evaluates the combined effect of several studies for the same intervention. Meta-analysis has four main components: the intervention effect, the forest plot results, the intervention mean effect, and what is valid to combine in a meta-analysis.

The intervention effect measurement depends on the variable nature: dichotomous or continuous. The dichotomous variable, that is, the one whose results can be inserted in a double entry table (2 × 2), can be evaluated using risk ratio or relative risk (RR), odds ratio or risk differences. The continuous variable, that is, that which can be measured, can be evaluated using the difference in means or standardized difference in means.

The forest plot result represents the measurement of the effects of each study individually by straight lines and combined effects in a figure representing the meta-analysis and called diamond. Fig. 2 shows a forest plot for the readers’ understanding.

Figure 2

Forest plot of the variable ‘number of patients’ requiring rescue analgesia. M-H: Mantel-Haenszel; Randm: random effect model; CI: Confidence Interval.

In Fig. 2, the information about the articles included in the analysis, as well as their numeric data are on the right and the visual information of the forest plot is on the left. The vertical line that lies at the center of the forest plot is the statistical nullity line and indicates no statistical difference between groups. The boxes containing horizontal lines represent the effect size of each study and the line is its confidence interval. The size of the boxes represents the percentage contribution to the final result and is called weight. The diamond represents the weighted average of the effects of each study analyzed together. The homogeneity assessment is quantified by I2 and assessed using statistical tests reported at the bottom of Fig. 2. Statistical heterogeneity may be due to clinical or methodological heterogeneity, but these are not represented in the plot.. The analysis of the forest plot results may be seen in detail at https://youtu.be/0kxJr2sJ8fc.

The mean effect of an intervention may be understood as the quantitative measure of the effect size for the intervention studied. The meta-analysis and its confidence interval represent the weighted average of the combined effects considering the sample size of each study.

What should be combined in meta-analysis are the results of studies that represent clinical homogeneity, that is, they present the same research question, with similar research groups and the same intervention in samples from similar populations. If there are sources of heterogeneity it is advisable to reanalyze the data and consider the influence of these sources on the results of the meta-analysis. Results less prone to bias do not suffer interference from these sources and do not change statistically with or without these sources in the studies.

Computer applications (apps) aid analysis and make the process more agile and fast. The apps may be: R Statistical software; Stata using metan function for forest plot, metafunnel for inverted funnel graph and metareg for meta-regression; RevMan; and Comprehensive Meta-Analysis.

Risk of bias assessment

Randomized clinical trials are the main unit of analysis for systematic reviews of intervention. Inferences of these studies may be overestimated or underestimated due to failures in planning, performing, analyzing or reporting clinical trials. It is impossible to know the extent of biases that interfere in the results of clinical trials, however it is possible to verify their occurrence. Authors of randomized clinical trials should comply with the items in the Consort Statement to ensure adequate reporting of items that ensure the best quality for a randomized trial. The website http://www.consort-statement.org/ may be useful at the time of writing.

The Cochrane Collaboration has developed a tool to analyze the risk of bias and since 2005 has been using it in its systematic reviews. The tool that evaluates risk of bias covers six domains: selection, performance, detection, follow-up, reporting, and other types of bias. The domains can be better observed in Table 2. Reviewers should classify each domain as high, undetermined or low risk of bias. Indeterminate classification occurs if there is insufficient data to classify the risk of bias as high or low.

Table 2

Table domains of risk bias.

Bias domain	Bias font	Support for reviewers’ decision
Selection	Generation of randomization sequence	Detailed report of the randomization sequence to allow the comparability of analysis groups.
Selection	Blinding of allocation	Detailed reporting of the method used to blind the allocation sequence so the allocated group is not revealed before or during intervention.
Performance	Blinding of participants and staff	Detailed report of the participants and researchers blindness regarding the participant's group.
Detection	Blinding to evaluate variables	Detailed report of the blindness of the group belonging to the participant at the time of variable measurement.
Follow-up	Missing data	Report of the finalization, loss, and exclusion of data for each primary variable, as well as the reasons for losses and exclusions from the analysis.
Report	Selective report	Complete report of the variables.
Others	Any other type of bias	Other sources of bias not covered in previous domains.

The analysis of the risk of bias in the included studies increases reliability in the results of a systematic review. If the results of the primary studies were somewhat biased, then the result of the systematic review cannot be considered definitive and requires future studies with less risk of bias to arrive at the definitive answer to the question that motivated the systematic review.

Quality of systematic review

Systematic reviews are published in increasing numbers and this fact does not appear to be surprising as this type of research basically requires a computer and access to the worldwide computer network, which facilitates the preparation to perform a systematic review and manuscript.

There are criticisms regarding the presence of inconclusive results attributed to the lack of primary studies, lack of homogeneity in the included studies, and lack of meta-analysis. A systematic review does not have as its sole purpose the use of data to prepare the meta-analysis, which in turn depends on the existence and homogeneity of primary studies. The purpose of the systematic review is also to contribute to improve the methodological quality of future studies, to suggest new research questions to the academic environment, and to be a tool for health managers to plan strategies to protect the population, and these purposes are independent of the existence of the meta-analysis and the homogeneity of the original articles. The absence of meta-analysis due to the lack of primary studies and homogeneity are not related to the greater or lesser risk of bias in a systematic review. The first step for any researcher wishing to start a systematic review is to ask “whether a review for a particular topic is indeed necessary” to avoid conclusions such as “more studies are needed …”.

Failures and limitations in the study design or execution have the potential to introduce bias into the results of the systematic review. Readers should assess the existence of these failures before crediting the results. Minimizing these failures ends up allowing the extrapolation of the results of a systematic review to a population with characteristics equal to those of the evaluated participants.

The PRISMA Statement guide has been adopted to guide systematic review authors to report their results as fully as possible and thus allowing greater transparency in the reporting process of any systematic review. The guide has a list of items to be reported in the article, as well as a flowchart for the results. The Cochrane Collaboration has its own guide that can also be adopted by non-collaborating authors as a guide to reduce the flaws of a systematic review.

The process of introducing flaws in the execution of systematic reviews has been better understood in recent years, and some tools have been developed to facilitate the reader to identify these flaws.7, 14 The most widely used is The Amstar, which evaluates the methodological quality of the systematic review.

Amstar is an instrument composed of 11 question-form items that allow the answers to yes, no, I cannot answer, and not applicable. The set of responses allows the identification of failures and a critical analysis of the review process. The other tool is Robis, which can be accessed at Robis Web site (http://www.robis-tool.info/) as well as in the Appendices section at www.jclinepi.com. The tool has three phases and the first one is optional: (1) Evaluate the importance; (2) Identify flaws in the processes as a whole; and (3) Assess the risk of bias. The most recent tool is Amstar 2, which evaluates reviews using randomized clinical trials, observational studies, or both. This tool has 16 items and can be accessed in its web site (https://amstar.ca/Amstar-2.php).

Final considerations

The main steps for planning and performing a systematic review with meta-analysis have been related and explained in this review article. It is important to note that there are other types of systematic review and other ways of performing such reviews with study types other than randomized trial.

The method described in this review article will help Brazilian anesthesiologists who have never performed this type of research to begin their first systematic review. This article also has links to internet sites containing tips and tools in order to assist in the steps of a systematic review, as well as whether there is a need to review the concepts at any time or see other necessary tools that the reader can access at https://goo.gl/f8CE3q.

The major limitation of a systematic review with meta-analysis is the risk of having statistical artifact rather than true associations. It is necessary to carry out complementary analyzes to see if there is robustness in the results or if under some characteristics present in the included studies the results change and do not appear solid. The other important limitation is that the results of studies with a high risk of bias influence the results of a meta-analysis and this weakens the evidence generated through the systematic review.

Conclusion

The evidence generated through a systematic review can provide greater confidence for clinicians regarding decision-making at the time of clinical practice, improve the benefits to their patients, and serve as a tool to assist managers in making decisions about new strategies for the health of the population.

Conflicts of interest

The authors declare no conflicts of interest.