Systematic reviews should be at the heart of continuing medical education.

Today, keeping up with the fast evolving evidence is more challenging than ever for practising physicians. A huge number of studies are published every day, and it is no longer possible to read all the relevant individual studies. Many physicians prefer attending continuing medical education (CME) to reading international scientific publications. Consequently, it is critical that CME is based on the best available evidence and presented in an unbiased manner free of conflicts of interest. Systematic reviews and Cochrane reviews in particular can thus provide a valuable resource of up-to-date and high-quality information on health care questions for CME providers. Of note, systematic reviews might become outdated quickly. Furthermore, some systematic reviews are fraught with limitations such as poor methodology and conduct or incomplete and misleading reporting. This article provides a brief overview of systematic reviews and Cochrane reviews, outlines how systematic reviews can be "kept alive" using today's digital opportunities and points to several common problems of systematic reviews with suggestions for solutions.

Introduction

Physicians are facing a huge, ever increasing body of medical information today: A large number of randomised controlled trials (RCTs) are published every day – a trend that can be expected to increase further [1]. The pool of electronically available scientific information is growing steadily and there is currently increasing drive towards open access publication, which allows easy and convenient online access to study results for everyone [2]. Although the availability of scientific information is generally to be welcomed [3], it also poses a major challenge to the individual physician and other health professional [4]. Keeping up to date in healthcare has always been difficult, however, today, searching and reading individual primary studies is not only insufficient but also impossible for individuals [1]. Moreover, assessing the relevancy and quality of individual studies and sources of information is often difficult for practicing physicians [4]. Since most medical research is published in English, language barriers can pose another challenge to health professionals [4,5].

A survey of family physicians in Germany to study their information needs and behaviours [4] revealed that nearly all of the 1003 surveyed physicians regard information on benefits and risks of medical treatments as (very) important. However, only about 15% frequently read international publications [4]. On the other hand, over three quarters of the surveyed physicians frequently attend continuing medical education (CME) [4]. This underlines that it is crucial for CME to be based on current best available evidence and presented in an unbiased manner free of any conflicts of interests [6]. Most recently, Marburger Bund [7] (association of salaried physicians in Germany) published a position paper calling for independence in CME. They emphasise that CME providers should include information from independent sources and, in particular, aggregated information from systematic reviews and meta-analyses [7].

Systematic Reviews – Aggregated Information from Multiple Sources

Like Marburger Bund, experts all over the world consider systematic reviews as important sources of information for clinicians [8,9] as well as for CME providers [10]. Systematic reviews aim at answering a specific research question using a pre-planned and structured approach [11]. In brief, the process of a systematic review can be summarised with the following steps [11]:

Formulating the research question and planning the organisation of the review, developing the review protocol and determining the methodology of the systematic review (including the systematic search, the data collection and extraction, the data synthesis and the reporting)

Systematic search, screening and selection of studies

Data extraction and risk of bias assessment of the identified studies

Data synthesis and assessment of the overall certainty of evidence

Interpretation and reporting of results

As a central feature and main benefit of systematic reviews, data from a set of individual studies are aggregated (i.e. synthesised) which enables drawing conclusions about the body of evidence for a specific research question. Data synthesis can be qualitative (i.e. characteristics of studies are narratively described and compared) and/or quantitative (i.e. data are statistically combined, most commonly using meta-analysis) [12]. Whether data can be synthesised quantitatively depends on several factors such as the types of included studies, the similarity of studies in terms of participants, interventions and outcome measures, as well as the nature of results [13].

The value of information from a systematic review crucially depends on the quality of the individual studies contributing to the respective review [14]. Systematic reviews are therefore required to include a critical assessment of potential biases in included studies [14], generally referred to as risk of bias assessment [15]. Common and recommended tools to assess risk of bias in individual studies are, for example, the Cochrane risk-of-bias tool (RoB 2) [16] for randomised trials or the ROBINS-I [17] for non-randomised studies of interventions. An exemplary risk of bias assessment (using RoB 2) included in a systematic review by Piechotta and colleagues [18] is shown in Figure 1. Additionally, an assessment of the overall certainty of evidence from the systematic review for each outcome investigated should be provided (i.e. simply put, authors tell the readers of their systematic review how confident they are that their calculated estimate of effect is close to the actual effect) [19]. The most common and recommended approach to assess the certainty of a body of evidence is GRADE (Grading of Recommendations Assessment, Development and Evaluation) [20]. GRADE specifies four levels of evidence [20], which are outlined in Table 1. For clinicians and CME providers as the readers of systematic reviews, the risk of bias assessment as well as the assessment of the overall certainty of evidence provide important information on how reliable the results of the respective systematic reviews are.

Figure 1.

Risk of bias assessment for the outcome “viral clearance at up to day 3” in Piechotta et al. [18]

Table 1.

GRADE levels of evidence

Level of evidence		Definition
⊕⊕⊕⊕	High certainty	We are very confident that the true effect lies close to that of the estimate of the effect
⊕⊕⊕◯	Moderate certainty	We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different
⊕⊕◯◯	Low certainty	Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect
⊕◯◯◯	Very low certainty	We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

Table adapted from Balshem et al. [20].

For adequate and complete reporting of systematic reviews, their authors should consult and refer to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [21]. For assessing the methodological quality of systematic reviews, tools such as AMSTAR (a measurement tool to assess systematic reviews) [22] or ROBIS [23] offer a valuable approach. As one might expect from the existence of such tools, the quality of systematic reviews can vary considerably. While the number of systematic reviews published every year is continuously rising, it has been argued that a considerable number of these reviews are based on poor methodology [24,25]. AMSTAR criteria such as duplicate study selection and data extraction or providing an “a priori design” (i.e. referring to a registered protocol) are frequently not met [26]. One type of systematic reviews that have repeatedly been found to overcome these criticisms to a large part are Cochrane reviews [27,28].

Cochrane Reviews – High Quality Evidence Syntheses

The non-profit organisation Cochrane (https://www.cochrane.org) is a global network promoting evidence-based healthcare and decision making by producing high-quality evidence syntheses that are free from conflicts of interest [29]. Cochrane reviews must comply with rigorous methodological standards (recorded in the Cochrane Handbook, available at https://training.cochrane.org/handbook), and a protocol must be published before they are conducted [29]. Furthermore, published Cochrane reviews are always independently peer-reviewed which means that every Cochrane review is evaluated by one or more external specialists before publication [30]. To ensure comprehensibility for non-expert readers and consumers such as people with a health condition, healthcare workers or policymakers, every Cochrane review includes a plain language summary [31] (specific search for plain language summaries at https://www.cochrane.org/evidence). These short stand-alone summaries provide an overview of the key questions and key findings of the systematic review [31]. Many plain language summaries are also available in languages other than English, such as Spanish, French or German. Each Cochrane review provides one or more “summary of findings tables” that contain the main findings of the review in a brief and concise format [19]. An example for a summary of findings table adapted from a Cochrane review by Piechotta and colleagues [18] is given in Table 2.

Table 2.

Adapted extract of a summary of findings table included in Piechotta et al. [18]

Convalescent plasma compared to placebo or standard care alone for individuals with moderate to severe COVID-19
Patient or population: individuals with moderate to severe disease Setting: Intervention: convalescent plasma Comparison: placebo or standard care alone
	Anticipated absolute effects* (95% CI)
Outcomes	Risk with placebo or standard care alone	Risk with convalescent plasma	Relative effect (95% CI)	№ of participants (studies)	Certainty of the evidence (GRADE)	Comments
All‐cause mortality at up to day 28 ‐ total	237 per 1,000	233 per 1,000(218 to 249)	RR 0.98(0.92 to 1.05)	12,646(7 RCTs)	⊕⊕⊕⊕HIGH	Convalescent plasma does not reduce all‐cause mortality at up to day 28.
Clinical improvement, assessed by liberation from respiratory support	Reporting of the clinical status or course of the disease was very heterogeneous across studies and it was not possible to pool data in a meaningful way. The reported evidence in all studies did not suggest any differences in the odds for clinical improvement or time to clinical improvement.			12,682(8 RCTs)	⊕⊕⊕⊕HIGH	Convalescent plasma has little to no impact on clinical improvement.

*The risk in the intervention group (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI).

CI: Confidence interval; RR: Risk ratio

Cochrane reviews are published in the Cochrane Database of Systematic Reviews which is a leading resource for systematic reviews in healthcare and a core component of the Cochrane Library [29]. The Cochrane Library (available at https://www.cochranelibrary.com) is a collection of databases including the Cochrane Database of Systematic Reviews, the Cochrane Central Register of Controlled Trials (CENTRAL; containing reports of RCTs and quasi-RCTs) and Cochrane Clinical Answers (providing short and clinically focused summaries of results from Cochrane reviews). Furthermore, the Cochrane Library provides access to the Cochrane Special Collections, collections of Cochrane reviews on specific health-related topics. For example, a current Special Collection brings together several Cochrane reviews on optimising health when working from home due to the COVID-19 pandemic [32].

For CME providers, Cochrane reviews as well as further resources accessible via the Cochrane Library can provide an opportunity to get high-quality information on a specific health care question (e.g. related to the effects of an intervention, the accuracy of a diagnostic test or the prognosis of individuals with a specific condition [29]) without spending a lot of time searching and acquiring many primary study reports.

Providing Current Evidence – Updating Systematic Reviews

If new studies are published on a topic covered in a systematic review, the results of this systematic review may be no longer up-to-date [33]. Relying on outdated evidence can, at worst, mean suboptimal care for patients and misleading information for researchers [34]. Cochrane therefore requires its reviews to be regularly checked to ensure that they are up-to-date and, if necessary, updated [33]. Cochrane reviews can thus be viewed as “living documents” [35]. Given the substantial time frame taken for primary studies to be included into a systematic review as well as the time and effort required to complete a thoroughly planned and conducted systematic review, keeping systematic reviews on urgent questions up-to-date is at the same time essential and challenging [36]. However, updating systematic reviews has been shown to be more efficient than conducting a new systematic review on the same research question [34]. Whether an update is needed depends on several considerations, such as whether the review addresses a current question, whether new relevant studies and/or methods are available and whether this new information is likely to affect the results and/or credibility of the present review [34]. Once decided for updating a systematic review, background, objectives, inclusion criteria and review methods have to be refreshed and search strategies have to be checked and, if necessary, adapted [34]. The updated systematic review answers a similar research question but is a carefully revised version of the original [34].

In recent years, a new type of living document has become established: Living systematic reviews (and living Cochrane reviews) go even beyond periodically updated Cochrane reviews [36].

Living Systematic Reviews

Living systematic reviews are conducted using standard systematic review methods but are continually updated as soon as new information becomes available [37]. In contrast to conventional updates of systematic reviews, living systematic reviews include an a priori definition of how often they will be reviewed for currency and, if necessary, updated [37]. They are no longer published in a static (or even paper-based) format but as dynamic online resources [36] which means they are perfectly accessible for CME. For transparency, all previous versions of a living systematic review should still be available online [38]. Living systematic reviews are time- and resource-consuming [38], therefore, not all systematic reviews need to be “living” [37]. The following criteria should be present when deciding for a living systematic review [34,37,39]:

The research question of the (planned) living systematic review is a priority for decision-making.

There is currently only low certainty of evidence from the existing sources and further information is likely to change the results of the (planned) living systematic review.

It can be expected that there will be new evidence in the near future, that is, research in the field of the (planned) living systematic review is progressing quickly.

In practice, the following fourth criterion should also be met [39]:

The research team has enough capacity and resources to maintain a living systematic review.

Given the presence of the aforementioned criteria, either a new living systematic review can be initiated or an existing systematic review can be updated and moved into “living mode” [39]. An example of a current living systematic review on an urgent research question is the living Cochrane review by Piechotta and colleagues [18] on convalescent plasma or hyperimmune immunoglobulin as a treatment for people with COVID‐19 that has first been published in May 2020, with three new published versions so far (July 2020, October 2020 and May 2021). Another recent living Cochrane review on Vitamin D supplementation for the treatment of COVID‐19 has been published in May 2021 [40]. For both research questions, it is very likely that new study results will become available in the near future that will add to the current evidence [18,40].

When, How Often, How – Further Aspects to Be Considered for Living Systematic Reviews

There are a number of issues that need to be considered and decided for a particular living systematic review. One of them is the optimal search frequency [37,38]. Cochrane suggests searches to be repeated on a monthly basis [38,39]. However, while a monthly search may be appropriate for quickly changing fields with a high number of new studies published in short periods, a lower search frequency might be considered for slower moving fields [38].

A further question is how quickly new information should be included into the review [37]. New searches will usually result in new references to screen, and, if available, inclusion of new relevant reports, data extraction, quality assessment, update of data synthesis and update of the systematic review report [37]. However, due to the large effort that a systematic review update still requires, authors may decide whether changes to the results brought about by the inclusion of new information are substantial enough to justify the effort of an update [37]. Thus, if no new evidence arises from the new search or if the impact of new information on the results is negligible, they may choose to document this information for the reader but defer the full update of the review [37]. Both questions, how frequently searches will be conducted and when and under which circumstances new evidence is incorporated into the systematic review (i.e. the review is updated), must be documented a priori in the review protocol for full transparency [39].

Another question relates to how conducting a living systematic review (but also a traditional systematic review) can be less time consuming and more automated [41]. Machine automation can, for example, assist with searching, deduplication of records, retrieval of full-text reports and to a certain extent also with eligibility assessment, data extraction and risk of bias assessment [41]. For example, there are resources that enable regular automated comprehensive searches (such as https://www.epistemonikos.org) or machine classifiers that identify RCTs, however, automation can today only partially assist with tasks such as data extraction and risk of bias assessment [41]. Further assistance in the form of human and not machine support can come from “crowds”, i.e. people who are registered with an online platform and are ready to help with tasks, such as eligibility check [41]. An example is the Cochrane Crowd, Cochrane’s citizen platform with currently over 21,000 contributors (available at https://crowd.cochrane.org), who have so far conducted over 5 million classification tasks. A further opportunity to reduce the often huge effort of data extraction may be the re-use of data that has already been extracted for a previous systematic review and have been made publicly available [42]. Platforms such as the Systematic Review Data Repository Plus (available at https://srdrplus.ahrq.gov) provide access to extracted study data from previous systematic review projects as well as the possibility to manage and share an own data extraction [42].

Finally, living systematic reviews should leave the “living mode” as soon as further maintenance becomes inappropriate [39]. To decide when to transition a living systematic review to a traditional systematic review, authors should consider the aforementioned three criteria on the appropriateness of a living systematic review (i.e. priority for decision-making, low certainty of evidence and new research expected in the near future) [39]. If one or more of these criteria are no longer met, it may be appropriate to transition out of living mode. Cochrane suggests that authors should assess the appropriateness of continuing with the living mode on an annual basis [39]. Again, the final decision must be carefully made and justified by the team of authors.

It may have become obvious, that Cochrane Reviews or living Cochrane Reviews in particular cannot and should not be conducted on all research questions. However, systematic reviews in general are fraught with a number of issues and challenges, some of which should be mentioned here with possible solutions.

Overproduction and Poor Quality of Systematic Reviews

As with RCTs [1], publication of systematic reviews has increased rapidly over the last decades [25], again making it difficult for clinicians as well as CME providers to identify the most relevant sources. Ioannidis [25] argues that many systematic reviews are not useful, i.e. they are redundant, misleading and/or conflicted. As an example, Doundoulakis and colleagues [43] examined systematic reviews on the efficacy and safety of non-vitamin K antagonist oral anticoagulants in patients with atrial fibrillation, published between 2012 and 2017. They found very high overlap between the existing systematic reviews with only 14 unique RCTs included in a total of 57 systematic reviews [43]. Multiple systematic reviews on similar topics with high overlap of included studies and meta-analyses not only constitute a waste of human resources and research funding [44], they make it difficult for health professionals to identify the most appropriate information [38] and they can often be confusing if they reach different conclusions [25]. Ioannidis [25], for example, points to a range of systematic reviews with meta-analyses addressing the effects of antidepressants that result in very different rankings of various antidepressants. One explanation for such differences can be slight differences in the definition of eligibility criteria – such as using different outcomes or focusing only on selected treatment options – which lead to diversity in the results [25]. Moreover, even the same results can be interpreted differently, especially if authors have a strong interest in a particular conclusion [25]. Independence of systematic reviews on antidepressants is not given in many cases, as there is a huge number of reviews authored by people with for-profit links (e.g. sponsorship, industry employment or other conflicts of interest) [25]. Thus, using narrow and purposively selected eligibility criteria and targeted selection of data or methods of analysis may be a common way to achieve interest-based results [25].

As mentioned before, a further common issue with systematic reviews is poor methodological quality [24,26,45]. For example, Pussegoda and colleagues [26] provided a systematic overview of reports on the quality or reporting of systematic reviews including a total of 56 reports. They found variable adherence to reporting guidelines or quality assessments; main concerns pertain, for example, to the availability of a protocol, the use of duplicate study selection and data extraction, the reporting of risk of bias assessment, additional analyses and funding sources.

Conducting Appropriate and Useful Systematic Reviews

To improve the quality of reporting, the use of reporting guidelines such as the PRISMA statement is highly recommended [21]. Today, a range of scientific journals request the reference to reporting guidelines when submitting systematic review reports [26]. This may have contributed to improvements in adherence to some reporting criteria in recent years, however, adherence to other criteria (as mentioned above) is still poor [26]. To ensure sound methodology, systematic reviews should be conducted by a research team that has both expertise in the topic area as well as methodological competence [46]. For methodological guidance, authors can, for example, consult the Cochrane Handbook (available at https://training.cochrane.org/handbook) or further Cochrane online training resources (available at https://training.cochrane.org). The PRISMA statement [21] also includes the reporting of competing interests for each review author. Ideally, authors should have no conflicts of interest, but if they do, they should – at minimum – disclose them in their report [25].

A further PRISMA item [21] refers to the registration of a review protocol in advance to commencement. Preparing and registering a systematic review protocol has a range of benefits: It enables careful planning and documentation, consistency among the review team, research integrity as well as transparency [47]. If the protocol is made publicly available, it can help to reduce duplication of systematic reviews as well as to reduce publication bias [47]. The International Prospective Register of Ongoing Systematic Reviews (PROSPERO, available at https://www.crd.york.ac.uk/prospero), funded by the National Institute for Health Research (NIHR), offers a free platform to register systematic review protocols as well as to search for ongoing systematic reviews. However, to date, only a minority of systematic reviews refer to a (published) protocol [25,26,48]. It is therefore highly desirable for planned systematic reviews with health-related outcomes to prepare a protocol and to make it publicly available [49]. Using PROSPERO as a standard register can help to reduce the unnecessary conduct of duplicate systematic reviews, given that every researcher strives to identify existing and planned systematic reviews before conducting an own systematic review [44,49].

The latter is another starting point for improvements: Carrying out a literature research in advance to conducting a systematic review enables authors to identify already existing (planned) systematic reviews, but also to formulate an appropriate research question and choose inclusion criteria with a sound rationale [11,44]. Additionally, the research question and inclusion criteria should be guided by what is important to patients, clinicians and other stakeholder [11]. Thus, what is already recommended for every Cochrane review is also highly beneficial for other systematic reviews: To involve patients, clinicians and other stakeholder when developing the research question and the systematic review protocol to ensure that the review answers a relevant question [46,50]. If the preparatory literature research identifies an already existing systematic review addressing the same research question, authors may decide to still take up this question, for example, if the existing review is out-of-date, has weak methodology or if there are relevant characteristics that warrant further investigation [44,49]. However, they should refer to the existing review and justify the planned duplication [49] and, ideally, they should update the existing systematic review instead of conducting a completely new review [34].

The Challenge of Identifying and Retrieving Relevant Documents and Data

Conducting systematic searches, screening and selecting studies and extracting data is usually a lengthy process with “great potential for waste” [51]. Over the last years, the call for full transparency along with data-sharing initiatives (which are generally to be welcomed) has led to a huge amount of information distributed over the World Wide Web [3]. As systematic reviews aim to identify all existing relevant information, numerous documents including many duplicates and reports with overlapping information often require extensive effort to screen, link related documents and extract the relevant data [3]. As a solution, Wieseler and McGauran [3] recently advocated for a central information portal linking study reports with all further information available. Such a portal would not only simplify the process of the systematic search, eligibility check and data extraction but also further support data transparency [3]. Besides, open access publishing as well as free access to databases may also be helpful in conducting the systematic search and retrieving relevant research articles. Several databases (including the Cochrane Library in several countries [52]) are not freely accessible to date, which may complicate the identification of all potentially relevant resources for a systematic review [53].

Conclusion

For CME providers, systematic reviews and especially Cochrane reviews offer a valuable possibility to answer clinical questions without spending a lot of time searching and reading the literature. For urgent questions in rapidly moving research fields that are a priority for decision making, living systematic reviews are increasingly available. Cochrane provides a range of resources that can be used for CME. However, systematic reviews also face several problems such as frequent duplication of reviews or poor reporting and methodology. A thoughtful and purposeful initiation of new systematic reviews as well as thorough reporting and conduct are highly required.