Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal.

Background: While the PRISMA flow diagram is widely used for reporting standard systematic reviews (SRs), it was not designed for capturing the results of continual searches for studies in living systematic reviews (LSRs). The objectives of this study are (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; (2) to propose an approach to reporting on that flow.
Methods: For objective 1, we identified all LSRs published up to April 2021. We abstracted information regarding their general characteristics and how they reported on search results. For objective 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs.
Results: We identified 279 living publications relating to 76 LSRs. Of the 279 publications, 11% were protocols, 23% were base versions (i.e., the first version), 50% were partial updates (i.e., does not include all typical sections of an SR), and 16% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches.
Conclusion: We identified six ways for reporting the study flow through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow. Copyright:

Introduction

During the coronavirus disease 2019 (COVID-19) pandemic, health research has proliferated exponentially . Systematic reviews are essential to synthesize the evidence and inform policy and practice. Given the pace of research publication, those reviews need to be kept up to date. Living systematic reviews (LSRs) are an emerging type of systematic review that involves the continual search of the literature and incorporation of relevant new evidence, soon after it becomes available . While many evidence synthesis groups are engaged in conducting LSRs or living network meta-analyses, others have developed living databases or living maps, including resources specific for COVID-19 literature .

An essential component of systematic reviews is to keep track of and report the number of records captured while searching the scientific literature and details of the selection process . The PRISMA statement recommends the use of the PRISMA flow diagram to depict the flow of studies through the different phases of the systematic review . While the PRISMA flow diagram is a widely used tool for reporting original systematic reviews, it was not designed to capture the results of continual searches typically used in LSRs. Hence, it’s unclear how authors of LSRs address the issue of presenting results of these continual searches.

The objectives of this study were (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; and (2) to propose an approach to documenting and reporting on the flow of studies through the different phases of a LSR, for the different updates.

Methods

For objective 1, we collected relevant data as part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs. We have described the details of that study in a previously published protocol . Briefly, we identified all living reviews published up to April 2021 available from the following electronic databases: Medline, EMBASE and the Cochrane library (see extended data of Khamis et al. for the search strategy). An eligible living review was either (1) a protocol for an LSR, (2) a base version of an LSR, (3) a full update version of an LSR, (4) a partial update version of an LSR, or (5) a combination of any of these (e.g., one living review may constitute of a protocol, a base version, and a full update version; another living review may constitute of only a Box 1 the definition of each type of living reviews.

•  LSR protocol: the protocol that describes the planned methods of the living review

•  Base version: the first version of the review that follows a living approach

•  Full update version: a subsequent version of the review that includes all the typical sections of a systematic review, including an introduction, methods, and results sections. Such a version could stand-alone in terms of content.

•  Partial update version: a subsequent version of the review that does not include all the typical sections of a systematic review, but instead refers to a previous version for complementary information. Such a version could not stand-alone in terms of content.

For the current study, we abstracted information about the following features of LSRs:

General characteristics:

Publication type, i.e., protocol, base version, full update version, partial update version.

Whether published in the Cochrane library or elsewhere.

Field (e.g., clinical, public health)

Whether COVID-19 related or not

Whether the base version of the living review conducted as a rapid review or not

Method used to report on the study flow (including the search results and the results of the selection process):

Narrative format and/or flow diagram.

Whether the results of the base and update searches are reported separately or not.

Type of flow diagram, if applicable (e.g., PRISMA).

For objective 2, we base our proposal for tailored PRISMA 2020 flow diagram approaches on the findings from objective 1, on our experience conducting Cochrane LSRs, and our methodological work on designing and reporting living evidence. Since 2017, our group has been responsible for the first series of three Cochrane LSRs, all of which address anticoagulation in patients with cancer . We conducted the base search in February 2016. Since then, we have been updating the search on a monthly basis. Through this experience, we have been able to apply and refine the guidance for conducting LSRs endorsed by the living evidence network group . Specifically, we explored solutions for the reporting of the study flow that would address different scenarios. Our goal was not to be prescriptive and narrow, but rather to cover all possible resulting flows by reviewing the LSRs we identified based on objective 1. Two authors developed a draft of the tailored approaches to presenting the study flow, and then circulated to the author team for review and suggestions for improvement.

Data handling and analysis

We used REDCap to collect and manage the data abstraction process. All data were exported from REDCap and analyzed using Stata v. 13 .

Results

Survey findings

Our search identified a total of 279 living publications relating to 76 LSRs. Table 1 shows their general characteristics. Of the 279 living publications, 11% were protocols, 23% were base versions, 50% were partial updates, and 16% were full updates. The median number of living publications per LSR was 2 (Interquartile range 1–4). Of the 76 living reviews, 22% were published in the Cochrane library, 63% were related to COVID-19, and 25% had a base version published as a rapid review. The majority were related to clinical topics (70%).

Table 1.

General characteristics of the 279 included living publications related to 76 living reviews.

	N	n (%)
Publication type	279 living publications
•  Protocol		31 (11.1)
•  Base version		64 (22.9)
•  Partial update version		138 (49.5)
•  Full update version		46 (16.5)
Living publications per LSR (Median (IQR))	76 LSRs	2 (1 – 4)
Cochrane LSR	76 LSRs	17 (22.4)
Field	76 LSRs
•  Clinical		53 (69.7)
•  Public health		20 (26.3)
•  Health system and policy		3 (4.0)
COVID-19-related	76 LSRs	48 (63.2)
Base version published as rapid review	64 base versions	16 (25.0)

Table 2 shows the results for the reporting on the study flow. Most base versions and full updates used a flow diagram to report on the search results (96% and 93% respectively), whereas only one partial update presented a flow diagram. In addition, none of the 279 living publications reported in their methods section how they plan to report on the study flow.

Table 2.

Reporting on study flow.

	N	n (%)
Inclusion of a flow diagram in a
•  Base version	64 base versions	62 (96.9)
•  Partial update version	138 partial updates	1 (0.7)
•  Full update version	46 full updates	43 (93.5)
Approach to reporting on study flow for different versions	184 update versions
Base separately; each update separately		39 (21.2)
Base separately; all updates combined		20 (10.9)
Latest update separately; all previous versions combined (including the base)		24 (13.0)
All versions combined		12 (6.5)
Latest update version only		47 (25.5)
Numbers not reported		42 (22.8)

a When a flow diagram is not reported, the authors reported on the search results in a narrative format.

Among the 184 update versions ( Figure 1):

Figure 1.

Summary of the four tailored PRISMA flow diagram approaches to present that study to presenting on the study flow for the different search updates.

21% reported the search results for the base version and for each update version separately.

11% reported on the search results for the base version separately and for all update versions combined

13% reported the search results for the latest update version separately and for all previous versions combined (including the base).

6% reported the search results for all the different versions combined.

26% reported the search results for the latest update version only.

23% did not report the search results at all (e.g., ‘new studies identified and integrated’ without specifying the number).

Proposed tailored PRISMA flow diagram approaches

Using the approach described in the methods section, we developed four approaches that allow authors to document and report the study flow for the different review update versions of an LSR.

Authors should record in detail the results of the searches to keep track of all identified records. We propose using a spreadsheet for one LSR at a time. The format we present consists of tabs for each of the respective search sources: bibliographic databases (e.g. MEDLINE, EMBASE, Cochrane databases); conference proceedings; ongoing studies as captured in clinicaltrials.gov and WHO International Clinical Trials Registry Platform (ICTRP); other tabs as needed, and a final ‘cumulative’ tab.

We show in Figure 2 a snapshot of the ‘cumulative’ tab of the spreadsheet that keeps track of all records. It shows the study flow for a hypothetical example for an LSR published first in January 2020 (i.e., base version) and updated on a monthly basis up to August 2020. Each row corresponds to a different update version. The columns present the following information for each update (columns B to E): the number of records received, deduplicated, included at title and abstract screening, and included at full-text screening (i.e., newly included reports). Additional columns (F to I) present the distribution of the newly included reports as relating to either: (1) new studies, (2) previously included studies, (3) ongoing (unpublished) studies, or (4) preprints.

Figure 2.

Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.

After manually entering the information in the first five tabs (corresponding to the different search sources) the total is automatically computed in the ‘cumulative’ tab.

The proposed spreadsheet can act as a basis for a tailored PRISMA flow diagram for LSRs. For structuring the flow diagram for LSR, one can select one out of four tailored PRISMA 2020 flow diagram approaches:

Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately) ( Figure 3).

Figure 3.

Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).

Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions) ( Figure 4).

Figure 4.

Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).

Approach 3: presenting the search results for the base version separately, and the results of all update version combined ( Figure 5).

Figure 5.

Approach 3: presenting the search results for the base version separately, and the results of all update version combined.

Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined ( Figure 6).

Figure 6.

Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.

In our Cochrane reviews, we applied the second proposal where we present the results for the different searches combined.

Discussion

Summary

This study found that authors of LSRs are not consistent in reporting on the flow of studies through the different phases of the review for the different update versions. Thus, we propose to document in detail the study flow for the different search updates and select one of four tailored PRISMA 2020 flow diagram approaches to present that study flow.

Strengths and limitations

To our knowledge, this is the first methodological survey that assesses how LSR authors report on the flow of studies through the different phases of the review for the different update versions of LSRs. In addition, the research expertise on our team covers both living approach and regular updating of traditional SR. We believe that our assessment forms a vital baseline and allows us to propose best practices for visualization options to improve consistency whilst the production of LSRs is still at a relatively early stage. Indeed, this survey is part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs , that would allow us to update our findings in the future.

Interpretation of findings

Authors tend to produce more partial updates of LSRs rather than continually updating the full systematic review. This might seem like a pragmatic approach particularly for a rapidly growing research field and when methods do not seem to change from one update to another. The heterogeneity observed in the ways LSR authors report on the study flow is likely to be explained by the lack of clear guidance on how to do so.

Implications for practice

We built our proposal on the PRISMA 2020 flow diagram and provide four approaches to tailor the needs for continual searchers used in LSR. The fourth approach is the closest to the current PRISMA 2020 flow diagram as it presents the results of the latest update version separately and the results of all previous versions (including the base) combined.

In addition, we proposed three other different approaches to provide options to LSR authors and publishing journals. Authors should choose one or the other approach based on the number of new citations, presentation preferences, and the impression of what provides the greatest transparency in reporting. Whatever approach one decides to follow, for transparency purposes, the systematic reviewers should ideally archive previous versions of the flow diagram (e.g., in an appendix). One major challenge will be to accommodate a large number of updates in the same diagram; some approaches would work better than others in that case. Also, advanced information technology solutions may allow fitting a large number of updates. A web-based prototype is available that allows readers to explore different reporting options across these four approaches: an R package ( https://github.com/nealhaddaway/livingPRISMAflow) and web-based ShinyApp ( https://estech.shinyapps.io/livingprismaflow/) were developed that allow users to enter their own data (e.g., from the spreadsheet suggested above) to produce a bespoke flow diagram according to their desired approach or to create their own interactive diagram that allows readers to toggle between different versions of the same data .

Advanced information technology can also be utilized to simplify updating and tracking the change in all LSR sections including the PRISMA diagram. It would be optimal to develop the base version in a certain platform where all SR and LSR sections are reported as units (i.e., title, authors, background, objectives, inclusion criteria, effect estimate for outcome x). With each update and for every unit, the author has the luxury to keep the same text (if no change has occurred) or edit (if change has occurred). Each unit can be updated in a differential speed based on certain criteria. The edits could be highlighted to visualize the change. For a certain section, one would easily have access to the entries in the previous versions and possibly visualize a trend across the different versions (i.e., cross-sectional view for that specific item). For example, dynamic documents can be developed using ‘R markdown’, a document preparation system, where static text can be combined with in-line code and ‘code chunks’ that produce instantly updatable documents given a modified input .

Implications for future research

This study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]. This project will pilot the proposed approaches for documenting the study flow and for structuring the living flow diagram. In addition, qualitative studies would be helpful to explore: (1) the feasibility and acceptability by LSR authors, publishers, and users towards the proposal; and (2) what the end-users would like to see in an LSR update.

Conclusions

LSR authors are not consistent in reporting the flow of studies through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates. Authors can select one of our four tailored PRISMA 2020 flow diagram approaches to present that study flow until detailed guidance will become available. Improving the reporting of study flow in LSR methodology is essential for incorporating living evidence when developing living guidance, particularly in the context of an urgent response .

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

I applaud the authors for updating their survey and for more than double the sample size. They also made some other - rather marginal -  changes and explain that their publication is only the first of a series based on an ongoing project.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

No

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Methodology, epidemiology, research integrity, open science, systematic review methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable.

I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Systematic reviews, evidence-based practice.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes . We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point. 2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

No

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Methodology, epidemiology, research integrity, open science, systematic review methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes . We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https:// www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR] ) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘ .