Methodological challenges for living systematic reviews conducted during the COVID-19 pandemic: A concept paper.

BACKGROUND: A living systematic review (LSR) is an emerging review type that makes use of continual updating. In the COVID-19 pandemic, we were confronted with a shifting epidemiological landscape, clinical uncertainties and evolving evidence. These unexpected challenges compelled us to amend standard LSR methodology. OBJECTIVE AND OUTLINE: Our primary objective is to discuss some challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work. Based on our experience and lessons learned from two Cochrane LSRs and challenges identified in several non-Cochrane LSRs, we highlight methodological considerations, particularly with regards to the study design, interventions and comparators, changes in outcome measure, and the search strategy. We discuss when to update, or rather when not to update the review, and the importance of transparency when reporting changes. LESSONS LEARNED AND
CONCLUSION: We learned that a LSR is a very suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience showed that the decision for updating a LSR depends not only on the evolving disease or emerging evidence, but also on the individual review question and the review teams' resources.

Rapidly emerging diseases put new challenges on living systematic reviews.

Review methods and inclusion criteria may need to be adapted for every update.

Policy relevance and important studies may influence the updating decision.

Transparent reporting of changes in methodology between review updates is key.

Transparent reporting is needed to avoid biases in the review process.

Introduction

A living systematic review (LSR) is an emerging systematic review type, which makes use of continual updating and ongoing surveillance of emerging research evidence [1]. Regular searches ensure that the systematic review includes the latest available findings and remains up to date [1]. Therefore, LSRs are most suitable for high-priority topics with substantial uncertainty, and where new evidence is published regularly [1]. In a series of four papers, the various aspects of LSRs have been discussed and elaborated on in detail [1], [2], [3], [4]. Cochrane published the first version of a Cochrane LSR series in 2017 [5], and in 2019 released guidance on the conduct and publication of Cochrane LSRs [6].

In the context of the Coronavirus disease 2019 (COVID-19) pandemic, we are confronted with a shifting epidemiological landscape, clinical uncertainties, a lack of evidence and a rapidly evolving evidence base. As methodologists conducting LSRs during the pandemic, we have recognised the need and opportunity to respond to new and unexpected challenges by amending our standard systematic review methodology.

Objectives

Our primary objective is to discuss some of the challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work.

Outline

To accomplish these objectives we draw on the experience and lessons learned from the author teams of two Cochrane LSRs [7, 8] and the methodological approaches used in other selected LSRs [9], [10], [11], [12], [13], [14]. When referring to ‘our’ experiences, we refer either to ‘review one’, investigating convalescent plasma for COVID-19 treatment [7] or ‘review two’, investigating international travel-related control measures for containing the COVID-19 pandemic [8]. The additional six LSRs were selected based on several characteristics (journal, complexity of methodology and topic, number of included studies and update strategy) to cover a broad variety in terms of LSR characteristics, see supplementary table 1. We highlight methodological considerations related to when a living review question is reasonable, particularly with regards to study designs, types of interventions and comparators, changes in outcome measures and the search strategy. We discuss when to update, or rather when not to update the review and the importance of transparency when reporting methodological changes.

Considerations regarding a living PICO – our experiences from a pandemic

To address the uncertainties related to COVID-19 research and adapt to the evolving evidence landscape, certain methodological elements needed special consideration for ensuring that LSRs are a reliable, up-to-date source of evidence that respond to the urgent health situation. Our experiences and further methodological approaches identified through other LSRs are elaborated in the following sections and summarized in Table 1 (and in more detail, in supplementary table 2).

Table 1

Summary of challenges identified from the methodological approaches used in selected LSRs.

Methodological elements with special LSR consideration and related challenges	LSRs reporting on these challenges	How the LSRs handled these challenges
Living methodology
Choice of study design: e.g. lack of RCTs/high quality studies	Juul, et al.; Schüneman, et al; Hernandez, et al; Wynants, et al; Allotey, et al; John, et al;	Inclusion of other designs, such as observational study or modelling studies
Choice of study type: e.g. inclusion of preprints	All	Inclusion of preprint
Intervention and comparators challenges	None	/
Changes in outcome measures	Juul, et al;	Post-hoc changes of the inclusion criteria for outcome measures
Search strategy	Juul, et al; Schüneman, et al; Hernandez, et al; Wynants, et al; Allotey, et al; John, et al	Included a variety of databases and search approaches (e.g. preprint server, hand search)
Handling of preprints	Juul, et al; Schüneman, et al; Hernandez, et al; Wynants, et al; Allotey, et al; John, et al	Added preprint server to their search, but no solution reported on how to track preprint updates
When to update, or rather when not to update
Updating triggers in general	Juul, et al; Schüneman, et al; Hernandez, et al; Wynants, et al; Allotey;, et al	/
Updating trigger: important studies	Schüneman, et al; Hernandez, et al; Wynants, et al;	/
Updating trigger: policy relevance	Juul, et al; Wynants, et al; John, et al	/
Information on funding	Juul, et al; Schüneman, et al; Hernandez, et al; Allotey, et al; John, et al	/
Transparent reporting of changes
Reporting updates between protocol and review	Juul, et al	Update changes mentioned in a section at the end of the text
Reporting updates between review updates if applicable	Juul, et al; Schüneman, et al; Hernandez, et al; Wynants, et al; Allotey, et al; John, et al	Update changes mentioned in the results, discussion or data supplement, trough update alerts or in a separate paragraph placed before the review introduction

Relevant design of studies? – a choice based on new conditions

Traditionally, evidence-based medicine has applied a hierarchy of evidence according to study design to achieve an adequate quality of evidence in systematic reviews and draw meaningful and valid conclusions. For standard intervention reviews, for example, randomised controlled trials (RCTs) are at the top of this hierarchy, followed by cohort studies and case-control studies in the middle of the evidence pyramid, and case series or reports at the bottom [15].

As response to clinical uncertainties in the COVID-19 pandemic and to ensure that no relevant evidence was excluded, in conducting review one we initially started with broad study design inclusion criteria. We planned to include RCTs preferentially, and to include other study designs, e.g. observational studies, only if insufficient RCT-evidence was available. We had to eventually adapt this initial plan, as we soon realised that refining inclusion criteria is an interactive process [7]. Identified studies did not report data for all our review outcomes, and in particular, some of the RCTs did not report safety data for the control group. Thus, for a better understanding of the frequency of unintended effects, we made the post-hoc decision to also include safety data from prospectively registered controlled and uncontrolled studies. Similarly, other selectively identified LSRs included observational studies at an early stage of the COVID-19 pandemic due to a paucity of RCTs [7,[9], [10], [11],13]. The lesson learned here is that authors should always rely on the best available evidence, which may be dependent on the outcome and will likely evolve and change rapidly over time. As LSR authors, we aim to synthesise and critically appraise all available evidence at a given time, but to update the review as more and potentially more trustworthy evidence becomes available. For example, we have seen that observational studies reported on positive outcomes for several interventions, e.g. convalescent plasma [7] or hydroxychloroquine [11], but were later shown to have little or no therapeutic effect against COVID-19 in higher quality studies and systematic review updates.

Generally ignored by the classical hierarchy of evidence, modelling studies have rarely been used in answering questions of intervention effectiveness, with systematic reviews focusing instead on experimental and sometimes observational evidence. This lack of consideration of modelling studies is at least partially because such studies simulate data on interventions and/or outcomes, which often require multiple sometimes questionable assumptions, rather than observing and measuring them directly. For some COVID-19 questions, modelling studies represented the sole evidence source, and it became clear early on, that decision-makers, despite the limitations of such studies, were using such studies to inform decisions. In the first version of review two [8], we included any type of modelling study due to the lack of experimental and even observational evidence.

Separate from study design, we also discussed which types of publication to consider for our reviews, e.g. peer-reviewed articles, preprints, abstracts, letters, etc. The experiences during this pandemic have shown the risks and benefits of using preprints, i.e. prompt availability versus validity and reliability (or lack thereof). Due to the prompt availability, several LSRs included preprints [7], [8], [9], [10], [11], [12], [13], [14]. However, preprints must be handled with caution as they are not peer reviewed and results might still change [[7], [8], [9],12,13]. Using preliminary preprint findings instead of data from the most updated preprint version, or the peer-reviewed journal publication could lead to different results or implications for review updates. It is challenging to identify updates of preprints, especially when the DOI remains unchanged [9], [10], [11], [12], [13], [14]. As soon as the full-text journal publications became available, some review authors prioritised these and reassessed the preprints [7,8,12,13]. To address remaining uncertainties, sensitivity analysis excluding preprints can be helpful to investigate the robustness of results [7].

Interventions and comparators

We learned that the interventions and comparators assessed by LSRs in the pandemic context evolved over time and needed adaptations. According to the Cochrane Handbook for Systemic Reviews of Interventions, it is important to consider and minimise the clinical and methodological heterogeneity between studies, to allow a valid comparison and a reliable pooled effect [16]. For review one [7], our overall main comparison remained unchanged; however we did adapt how we defined the specific intervention and comparator. Specifically, we noticed that, because there was and is no real standard care available, the best supportive care options differed widely across contexts. For instance, we observed that Chinese studies often used Traditional Chinese Medicine as part of patient care [17]. Studies that were initiated early in the pandemic often used hydroxychloroquine [18], and studies that recruited patients after July 2020 often used corticosteroids [19]. Another challenge was that for most co-interventions the evidence on safety and effectiveness remained uncertain. Hence, we tried to account for bias due to unequal distribution of co-interventions across study groups. We analysed individuals with mild and moderate to severe symptoms separately, based on existing hypotheses regarding the intervention modes of activity and our evolving understanding of COVID-19 progression to assure comparability of study participants [7].

How to deal with changes in outcome measures

At the beginning and throughout the course of the pandemic, robust and relevant outcome measures were not clear. We based our outcome selection on the COMET (Core Outcome Measures in Effectiveness Trials) initiative for COVID-19 patients [20]. As more evidence became available, we found that outcome measures needed to be refined. Thus, our outcome set was never “final” but constantly evolving. We noted for instance in review one [7], that there was broad diversity in the assessment and reporting of the clinical status or disease progression, with standard reporting measures changing over time as well. This increased the potential for heterogeneity in outcome measurement and reporting across studies. We could not find a solution to reasonably combine data and provided narrative syntheses without meta-analysis for respective outcomes. Changing outcomes were also identified in the Juul review, which added an additional post-hoc outcome for their update [9].

Developing the search strategy

We used the Cochrane guidance for LSR search methods to develop our initial search strategy. According to this guidance, there is a particular interest for LSRs to keep ongoing and emerging evidence up to date through regular searches of electronic databases, clinical trial registries and other potential sources [6]. The search strategies also need to be updated, as relevant terms, keywords or database filters may change [6].

One challenge for review two, related to maintaining searches over time [8]. The changing database landscape required constant amendments to the search strategy and literature sources. For example, the Centers for Disease Control and Prevention (CDC) COVID-19 Research Articles Downloadable Database, an early and comprehensive source of pre-print articles, was discontinued in mid-2020, but is now completely covered by the WHO COVID-19 Global literature on coronavirus disease database. A challenge for review one involved the dynamic nature of electronic databases [7], with existing databases changing and new ones becoming available. Therefore, it was not sufficient to rely only on the traditionally utilised databases such as PubMed, Embase or CENTRAL. Some reviews also explored new COVID-19 registries, such as the Cochrane COVID-19 Study Register (CCSR), [7], [8], [9], [10] a regularly updated public database for study references, particularly efficient for update searches of LSRs [21]. Also, the L*VE platform, was used in some LSRs [7,10,14]. The information specialists involved in review one used the website of “COVID-END” [22] and the EPPI Centre [23], providing guidance and listing the various COVID-19 registries, to get an overview of the numerous newly available and often overlapping registries. Another challenge identified in review one was that no suitable screening software exists to respond to the evolving inclusion criteria [7]. Because of the rapid emergence of new evidence, most LSRs decided to run a complete search each week [7,9,10,14] or month [11].

A further challenge with review one was tracking the ongoing studies [7], as the estimated study completion dates indicated in the study registries were sometimes unreliable. Therefore, it was of utmost importance to track ongoing studies through regular contact with the main investigators. For previously identified ongoing platform trials [7] or preprints [9], [10], [11], [12], [13], [14] some authors decided to perform regular manual checks for new updates.

When to update, or rather when not to update

When to update is an important issue to discuss when planning and conducting a LSR, and is highly context dependent. There is no clear standard for how frequent or at which time point updates of LSRs should be performed and published [1]. According to Cochrane, updates can be planned either when it is likely that newly identified evidence has an impact on the review conclusions or at a fixed-interval schedule when more emerging evidence is expected [6]. The panel for updating guidance for systematic reviews recommends an individualised updating approach, where the responsibility for the update decision depends on the personal resources of the authors and the editorial team [24]. The Annals of Internal Medicine provides detailed author guidance on updating and publishing paths for LSRs, which suggests committing to publish either surveillance comments, alerts of new evidence or a new article with major updates [25].

In the context of the COVID-19 pandemic, we defined additional components to consider when deciding on updating a LSR. Based on a figure from the Cochrane LSR guidance illustrating the LSR workflow [6], we reproduced a similar figure and adapted it according to our experience (blue components), to visualise the decision process for updating or publishing a LSR (see Fig 1 ). The part of the flowchart that we altered most relates to running the search. Here we added three additional considerations that can affect the decision of whether to update, publish, neither or both: the policy relevance, the importance of the study and funding. Each of these three considerations is described in more detail below.

Fig. 1

LSR decision flowchart for updating and publishing the review, adjusted for the context of rapidly emerging diseases (amendments: components in blue are the additional steps we took with the original Cochrane flowchart in grey and purple [6]).

Policy relevance

We added ‘policy relevance’ as an additional component influencing the updating decision, as policy triggers can indicate the need for an update. For example the Emergency Use Authorization and statement by the US Food and Drug Administration (FDA) that convalescent plasma may be effective in treating COVID-19 was an important consideration [26] in deciding to publish an update of review one [7], even though the conclusion of our review did not change after considering additional study data. Regarding review two [8], the development of WHO guidance on COVID-19 mitigation in the aviation sector heavily influenced the decision of when to update [27].

Important studies

We also added “waiting for important study” as an additional aspect to consider in the updating decision. As ongoing studies identified in previous versions of review one came to completion [7], we faced the decision of when to update. To make optimal use of resources, we ultimately decided to tie our updates to the completion of larger, well-planned studies, e.g. platform trials. These were most likely to produce higher-certainty evidence, and we felt that an update including such studies would be of optimal value to the end user. Through regular communication with study investigators, we were able to identify when an ‘important’ study for our PICO was going to be published. For review two [8], and potentially for other less clinical, more public health-type PICOs, diverse modelling studies and smaller observational studies can be important. The author guidance of the annals of internal medicine for instance suggests a “major update” when new evidence is substantive. Here an “important study” could be a large, well designed study in case of previous inconsistent or lower quality studies, or could be several new studies of differing size and quality [25].

Funding

Funding could influence or delay the conduct of updates and should therefore be considered already before starting a review update and running the search. If there are no resources to conduct the review update it is possible that the steps following the search cannot be conducted. This concerns not only financial resources but also to time and personnel. In one LSR the review authors indicated that they excluded grey literature for instance due to resource limitations, which could have hindered identification of important new evidence [13].

Transparency in the reporting of changes

Transparent and traceable reporting of changes to the methodology of LSRs is a challenge, and there is currently no PRISMA reporting guideline for LSRs. According to Cochrane guidance, a LSR requires the explicit reporting of certain factors, such as the screening, whether the review incorporated new evidence, and the methods changes [6]. Cochrane has established a transparent structure for reporting the differences between protocol and review in standard reviews [6]. This standard structure can be used for any review, but it may not be adequate for LSRs. Thus, we feel that there is a need for a similar structure specific for LSRs.

In the context of a living PICO and methodology, we found it highly relevant to report in a transparent way the differences between the protocol and first version of the review, as well as between the review updates. When looking at how changes from other LSRs were reported, one LSR included a section at the end of the review on “changes between protocol and review” [9]. Others indicated mainly the changes between review versions or updates, either briefly in the discussion, data supplements [9,12,14], or through update alerts published between review versions [10,11]. One review included a section “Updates from version 1” before the introduction of the review [13]. For review one [7], we decided to include an overview table of changes titled ‘Summary of PICO development from protocol stage to current review version’, which can be incorporated by other LSR authors. This table summarises the main PICO elements (e.g. participants, interventions, comparators, outcomes, study design and methodological changes) of the protocol and review version changes. For the latter, it is also important to report changes of the review´s results (e.g. the number of included studies, the certainty of the evidence) and the authors conclusion (see Table 2 ).

Table 2

Summary of PICO development from protocol stage to current review version.

	Participants (inclusion and exclusion)	Intervention	Comparator	Outcomes (primary and secondary)	Study design	Methods changes	Results	Authors conclusion
Protocol (date)
Update 1 (date)
→Changes
Update 2 (date)
→Changes
Update 3 (date)
→Changes
Update 4 (date)
→Changes

We found it important to emphasise that the choice of study design eligibility was not a selective post-hoc approach. Ideally, the methodology for each update should be adapted and (re)finalised at the beginning of each version. Studies that are excluded based on different criteria for the updated PICO could be listed chronologically in a ‘supplemental evidence set’ or incorporated in a modified PRISMA flowchart, and thereby increase transparency of the screening and study selection process [28].

Lessons learned and conclusion

Based on our experiences in the planning and conducting of LSRs in a pandemic environment and challenges identified from approaches used in other LSRs, we can conclude that a LSR is a highly suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience also demonstrated that updating the methods of a LSR, or the LSR itself, is dependent not only on the evolving disease or the emerging evidence, but also on the individual PICO and the capacity as well as resources of the review team. For a living PICO, we described the importance of transparently reporting the differences between the protocol and the review, as well as between each review update. These lessons learned could be valuable for future pandemic preparedness. An implication for further research and discussion is when to ‘retire’ and discontinue the updating of a review.

Acknowledgements

The experience report was partly developed from the framework of the CEOsys project. We would like to thank those involved at CEOsys for their support (covid-evidenz.de/).

The research was part of a project supported by the Federal Ministry of Education and Research (NaFoUniMedCovid19, funding number: 01KX2021; part of the project "CEOsys"). The contents of this document reflect only the authors' views and the German Ministry is not responsible for any use that may be made of the information it contains.

We thank the entire author team of the review ‘Convalescent plasma or hyperimmune immunoglobulin for people with COVID‐19: a living systematic review’, including Khai Li Chai, Sarah J Valk, Vanessa Piechotta, Catherine Kimber, Ina Monsef, Carolyn Doree, Erica M Wood, Abigail A Lamikanra, David J Roberts, Zoe McQuilten, Cynthia So-Osman, Lise J Estcourt and Nicole Skoetz for their contribution to the input of this guidance paper. The content of the paper was based on their experiences and lessons learned when having conducted this review.

We thank the entire author team of the review ‘International travel‐related control measures to contain the COVID‐19 pandemic: a rapid review’, including Jacob Burns, Ani Movsisyan, Jan M Stratil, Renke Lars Biallas, Michaela Coenen, Karl MF Emmert-Fees, Karin Geffert, Sabine Hoffmann, Olaf Horstick, Michael Laxy, Carmen Klinger, Suzie Kratzer, Tim Litwin, Susan Norris, Lisa M Pfadenhauer, Peter Philipsborn, Kerstin Sell, Julia Stadelmaier, Ben Verboom, Stephan Voss, Katharina Wabnitz and Eva Rehfuess for their contribution to the input of this guidance paper. The content of the paper was based on their experiences and lessons learned when having conducted this review.

We thank Toby Lasserson for his contribution to the input of this guidance paper and experience report.

We thank Maria-Inti Metzendorf, information specialist, for her contribution on the search strategy section.

CRediT authorship contribution statement

Claire Iannizzi: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. Elena Dorando: Writing – review & editing. Jacob Burns: Conceptualization, Writing – review & editing. Stephanie Weibel: Conceptualization, Writing – review & editing. Clare Dooley: Writing – review & editing. Helen Wakeford: Writing – review & editing. Lise J Estcourt: Writing – review & editing. Nicole Skoetz: Conceptualization, Writing – review & editing, Funding acquisition. Vanessa Piechotta: Conceptualization, Supervision, Writing – review & editing.