Reporting of systematic reviews and meta-analysis of observational studies.

Although randomized controlled trials (RCTs) are generally seen as providing more certainty about the evidence of an effect, many interventions in infectious diseases are examined in observational studies. These include rare cases of treatment that cannot be examined in a RCT (e.g. appropriate vs. inappropriate empirical antibiotic treatment) [1,2], complex interventions that are difficult to examine in RCTs (e.g. infection control or antibiotic stewardship) [3,4], comparative effective research and quality improvement programmes (comparisons between commonly used, approved, antibiotics, monotherapy vs. combinations or treatment durations) for which resources for RCTs are lacking [[5], [6], [7], [8], [9], [10]], or studies rapidly launched during outbreaks, such as the coronavirus disease 2019 (COVID-19) pandemic [[11], [12], [13]]. Summarizing these in systematic reviews is appealing; however, systematic reviews of observational studies suffer from the same shortcomings as primary observational studies when causal inferences need to be made. Consistent errors in observational studies may even be magnified by narrowing the confidence intervals through meta-analysis.

In this editorial note, we provide our recommendations and expectations for systematic reviews and meta-analyses of observational studies examining an exposure or intervention. Guidance on performance and reporting of such studies has been published, and we recommend following these [14,15]. We explain the specific pitfalls of such studies and what we require from authors of systematic reviews of observational studies.

As for any research, systematic reviews of observational studies should start with a protocol. We expect to see the protocol for the systematic review, preferably published in advance in PROSPERO (International Prospective Register of Systematic Reviews) or other database. This protocol should start with the review question, containing Patients, Intervention/Exposure, Comparator, Outcomes, Study design (PICOS or PECOS). The PECOS format has been proposed to distinguish between the active intervention in a RCT and the observation of an ‘exposure’ in observational studies [14]. In interventional trials, the dose and duration of the treatments are fixed by protocol, and restrictions on patient inclusion on the basis of time from disease onset are defined. Similarly, in observational studies, there should be criteria to define exposure [16], and systematic reviews can and probably should restrict inclusion criteria to observational studies examining the exposure of interest. For example, in a systematic review of hydroxychloroquine therapy for COVID-19, rather than defining for inclusion all studies examining hydroxychloroquine among patients with COVID-19, an informative review can restrict inclusion to studies where hydroxychloroquine therapy was initiated early after diagnosis, examining an appropriate dose, for a minimal duration of treatment. Too tight a definition (e.g. one specific hydroxychloroquine dose) might result in zero studies included, but too broad a definition will result in less meaningful estimates and heterogeneity. Alternatively, this can be investigated in sensitivity or subgroup analyses, and would address the time hydroxychloroquine was started relative to the diagnosis of COVID-19, acceptable dosing and the minimal duration that would be considered as treatment. The study designs included should be addressed. Inclusion of RCTs should always be considered. The types of observational studies considered and restrictions on observational study design for inclusion in the review should be specified. Possible exclusion criteria based on observational study design may include small sample sizes, before-and-after studies or lack of adjusted analysis.

An important feature of the protocol for a systematic review of observational studies is the need to tailor the risk of bias assessment tools to the review. We recommend the Cochrane tool for risk of bias of nonrandomized studies of interventions, ROBINS-I [17]. ROBINS-I is a domain-based assessment tool with signaling questions for each domain. It is comparable to the Cochrane ROB-2 tool for RCTs, which may be useful when including both RCTs and non-RCTs in the systematic review. The Newcastle-Ottawa scale is an alternative tool [18]. Both require advance planning and adaptation to the review question to enable assessment of the ‘confounding’ and ‘deviations from intended interventions’ domains (https://www.riskofbias.info/welcome/home/current-version-of-robins-i/robins-i-tool-2016). The important risk factors for the outcomes and variables presumed to affect both the presence of the exposure and the outcome need to be defined. Review authors should assess whether these predefined variables were examined when analysing the association between exposure and outcome. For example, a systematic review assessing appropriate vs. inappropriate empirical antibiotic therapy for sepsis requires the assessment of age, any sepsis severity score and the place of acquisition as potential confounders in primary studies [19]. For a study of hydroxychloroquine and COVID-19, confounding domains should probably include age, comorbidities and disease stage/severity. In addition, a list of potential cointerventions that could be different between intervention groups and that could affect outcomes needs to be specified in advance in order to compare their use between the treatment groups and to score studies on bias related to the interventions. For example, corticosteroids and availability of intensive support would be important cointerventions in the observational assessment of antiviral therapy for COVID-19, especially when comparing data among different centres. Well-conducted risk-of-bias assessment of observational studies within a systematic review is highly informative and could actually be the focus of the systematic review, as was the case in a systematic review of studies comparing monotherapy to combination therapy for carbapenem-resistant Gram-negative infections, where the main objective was to highlight the bias in existing studies to enable an evidence-based statement that these studies should not direct practice [20].

The main challenge in meta-analyses of observational studies is pooling the adjusted results. Pooling the crude unadjusted numbers is simple, but it is not more informative than the univariate analysis in the original observational studies. Primary observational studies typically use different methods for adjustment of the analysis, such as multivariable regression analyses, propensity scores that can be used in several ways or instrumental variable analyses. Results from these analyses should be used in systematic reviews of observational studies. However, a commonly encountered problem is that depending on inclusion criteria of the review, only some of the included studies may report the adjusted association for the exposure variable of interest. Included studies might not include an adjusted analysis at all; the authors might have performed an adjusted analysis without analysing the exposure of interest, or they might not include quantitative results for factors that were not retained or that were not statistically significant in the final regression model.

We encountered all these possibilities in a systematic review and meta-analysis on appropriate vs. inappropriate empirical antibiotic treatment and mortality in sepsis. Because the studies that did not find an association between inappropriate empirical antibiotic treatment and mortality in the univariate analysis did not include it in the adjusted risk model or did not quantitatively report the results of these adjusted analyses, these studies were highly skewed towards a lack of association [19]. In this case, restricting the analysis to studies reporting the adjusted odds ratio for inappropriate empirical antibiotic treatment and mortality would have strongly biased this analysis in favour of appropriate empirical antibiotic treatment. A standard solution to all these obstacles is not available. (A simple solution would be not to perform a systematic review of observational studies.) Several options and the limitations of each are presented in Table 1 . Furthermore, even for a focused question, the studies will use different variables and variable definitions for adjustment. To address this source of variability in the adjusted estimates of association, review authors may limit the meta-analysis to studies adjusting for confounders that were defined as being important in the tailored ROBINS-I. Additionally, sensitivity or subgroup analyses by optimal, nonoptimal or no adjustment can be performed [14]. Finally, adjusted results of the original studies might be reported as odds ratios, hazard ratios or risk ratios with different dispersion measures (e.g. 95% confidence intervals, standard errors). A meta-analysis should pool the same type of measure.

Table 1

Meta-analysis of adjusted associations from observational studies

Analysis method	Limitations
Include only studies that reported adjusted association	Empty review; selected set of studies most likely showing significance for exposure
Use adjusted OR when available and crude, unadjusted result otherwise	This analysis may suffer from confounding
Use uniform null effect estimate (OR = 1), with standard error depending on study size/event rate for studies not reporting adjusted estimate	Probable bias towards null association
Use other uniform effect estimate (e.g. pooled univariate OR), with standard error depending on study size/event rate for studies not reporting adjusted estimate	Depending on study question, this analysis may suffer from bias due to confounding
Request raw data from study authors or ask authors to conduct adjusted analysis and provide adjusted association	Optimal solution that will allow possible homogeneity of variables used for adjustment in different studies

OR, odds ratio.

The optimal strategy will probably be to conduct sensitivity analyses using different assumptions. None of these sensitivity analyses may be without bias, but if the bias works in different directions, this may be informative. Of six systematic review including observational studies published recently in CMI [13,[21], [22], [23], [24], [25]], only two addressed the adjusted results of primary observational studies (presenting these without a meta-analysis in a review associating vaccines with antibiotic use [22] and using multivariate odds ratios when available and univariate results otherwise in a meta-analysis of hydroxychloroquine used to treat COVID-19 [13]). A review of fluoroquinolones vs. trimethoprim/sulfamethoxazole used to treat Stenotrophomonas maltophilia infections reported a lack of adjusted results and pooled univariate results [23]. We would like to see both the crude results and the adjusted effect estimate presented. A comparison between the nonadjusted and adjusted associations can attest to the degree of confounding. Similarly, in systematic reviews including both RCTs and observational studies, their meta-analysis should be explicitly separated. If you believe that a meta-analysis of crude, unadjusted results is valid, then you must convince readers that confounding is negligible in this particular review. The complicated nature of an optimal meta-analysis of adjusted associations from observational studies limits the scope of a systematic review of observational studies typically to a single outcome of interest.

Certainty of the overall evidence presented in the systematic review can be determined using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, as for RCTs [26]. Because risk of bias will rarely, if ever, be low with observational studies, the certainty of the evidence will often be automatically downgraded to moderate, low or very low certainty of the evidence. GRADE allows upgrading the certainty of the evidence of observational studies when all confounding factors (including unobserved, residual confounding) work in the opposite direction of the observed association. This requires knowledge of the residual confounding and was impossible in the case of the systematic review on inappropriate empirical antibiotic treatment. Authors of systematic reviews of observational studies are advised to follow the GRADE-PRO handbook when performing the grading [27].

As for the Discussion and Conclusion sections, we will be strict on overoptimistic interpretation of the results of systematic reviews relying on observational studies. If the aim of a systematic review is to provide evidence about causal effects of interventions and exposures, then this should be made explicit. But at the same time, the conclusions will usually need to be cautious and include a warning about the observational nature of the presented evidence.

In summary, performing a systematic review of observational studies requires knowledge of the review field, advanced understanding of observational study design and analysis, good planning and an understanding that the evidence summarized will most probably not reach high quality. We present in Table 2 subheadings for the Methods section that we expect to see in systematic reviews of observational studies in CMI; we expect judicious analysis of data from observational studies.

Table 2

Items to be reported in Methods section of systematic reviews of observational studies

Item	Explanation
Design	Present as systematic review, with or without meta-analysis and general type of studies considered (observational, with or without RCTs).
PECOS: Patients	Describe targeted disease and patient population
PECOS: Exposure/intervention	Minimal requirements for exposure/intervention definition should be defined but not too restrictive, considering that many observational studies do not define exposure well, and considering heterogeneity between studies' definitions.
PECOS: Comparison	Define requirements for nonexposed cohort. When including case–control studies, define whether drawing exposed and nonexposed subjects from same population is a requirement.
PECOS: Outcome	Define primary outcome of review. This will be the outcome summarized appropriately if possible through adjusted analysis. Other endpoints can be listed as secondary, addressing adverse events also.
PECOS: Eligibility criteria	Study eligibility criteria should be detailed. Restrictions on inclusion of studies by design features, without adjusted analysis, exclusion of certain types of adjustment or by sample size should be defined.
Search strategy	Databases searched and search string adapted for each database should be presented (possibly as supplementary material). Study flowchart (in Results) should start transparently from results of described search strategy. Restrictions on study years, publication status or language should be avoided or justified.
Risk of bias assessment	Tool used for risk of bias assessment should be defined and its adaptation to review topic should be presented in supplementary material, including definition of relevant confounding domains and cointerventions.
Data	Types of data to be extracted should be defined, including nonadjusted and adjusted outcomes. Data on specific confounders extracted and adjustment methods used in study are special to systematic reviews of observational studies.
Data extraction	Methods of data extraction and risk of bias assessment, including whether duplicate independent extraction was performed. Data extraction and risk of bias assessment of observational studies are more complicated than in RCTs; a duplicate process with a consensus strategy is highly recommended.
Analysis	If performing a meta-analysis, univariate and adjusted analyses must be addressed, including methods of handing different adjustment methods and studies that did not report an adjusted analysis. Describe approach of pooling studies (fixed/random effects meta-analysis), heterogeneity assessment and planned subgroup and sensitivity analyses.
Grading	Preferably, quality of evidence summarized in review should be graded formally.

RCT, randomized controlled trial.

Transparency declaration

Neither author reports a conflict of interest relevant to this editorial.