Characteristics of Living Systematic Review for COVID-19.

Purpose: The systematic review aims to analyze and summarize the characteristics of living systematic review (LSR) for coronavirus disease 2019 (COVID-19).
Methods: Six databases including Medline, Excerpta Medica (Embase), Cochrane Library, China National Knowledge Infrastructure (CNKI), Wanfang Database and China Science, and Technology Journal Database (VIP), were searched as the source of basic information and methodology of LSR. Descriptive analytical methods were used to analyze the included COVID-19 LSRs, and the study characteristics of COVID-19 LSRs were further assessed.
Results: Sixty-four COVID-19 LSRs were included. Eighty-nine point one percent of LSRs were published on Science Citation Index (SCI) journals, and 64.1% publication with an impact factor (IF) >5 and 17.2% with an IF >15 among SCI journals. The first unit of the published LSRs for COVID-19 came from 19 countries, with the largest contribution from the UK (17.2%, 11/64). Forty point six percent of LSRs for COVID-19 were related to therapeutics topic which was considered the most concerned perspective for LSRs for COVID-19. Seventy-six point six percent of LSRs focused on the general population, with less attention to children, pregnant women and the elderly. However, the LSR for COVID-19 was reported incomplete on "living" process, including 40.6% of studies without search frequency, 79.7% of studies without screening frequency, 20.3% of studies without update frequency, and 65.6% of studies without the timing or criteria of transitioning LSR out of living mode.
Conclusion: Although researchers in many countries have applied LSRs to COVID-19, most of the LSRs for COVID-19 were incomplete in reporting on the "living" process and less focused on special populations. This could reduce the confidence of health-care providers and policy makers in the results of COVID-19 LSR, thereby hindering the translation of evidence on COVID-19 LSR into clinical practice. It was necessary to explicitly enact preferred reporting items for systematic reviews and meta-analyses (PRISMA) to improve the reporting quality of LSR and support ongoing efforts of therapeutics research for special patients with COVID-19.

Introduction

The outbreak of coronavirus disease 2019 (COVID-19) in December 2019, has rapidly spread to 220 countries and regions, leading to the global pandemicity.1 According to the latest data from World Health Organization (WHO), over 562 million people have been diagnosed and over six million people have died from COVID-19 up tol July 22, 2022.2 To cope with the epidemic and severity of COVID-19, many countries accelerated research of diagnosis, vaccination and therapeutics, which cause a surge in the number of clinical studies on COVID-19 in a short time.3,4 From January 1, 2020 to May 6, 2020, 1694 clinical trials related to COVID-19 were authorized and can be found in five international clinical trial registries.5 However, a study from the British Medical Journal (BMJ) suggested that the large amount of studies on COVID-19 resulted in increased difficulty to respond quickly and effectively to devise an appropriate crisis response strategy to the novel coronavirus, which had also hugely impacted the mental health of health-care employees due to overload of work and exhaustion under the COVID-19 pandemic.6

As high-level evidence in evidence-based medicine, a systematic review (SR) acts as a bridge to connect research outcomes and clinical practice, providing a reference for clinical practitioners to contribute quick and accurate decisions.7–9 Although the methodology of SR is relatively mature, its updating mechanism for publication is immature.10 Once literature been published, only a few SRs were updated within two years.11 Most SRs failed to incorporate new evidence timely, resulting in their conclusions differing from the true effect of interventions.10

To solve the delayed update of publication analyzed in SRs and accurately reflect the effect of interventions on COVID-19, Elliott et al proposed the concept of a living systematic review (LSR) in 2014, and defined it as a “systematic review which was continually updated, incorporating relevant new evidence as it becomes available”.7,12 Elliott et al suggested, an acceptable and appropriate LSR required to follow three criteria: (1) the review question is a particular challenge and its solution needs to be priorities to decision-making; (2) the existing evidence is insufficient and uncertain for questions to be solved; (3) the emerging evidence is likely to impact conclusions of the LSR.7,12 However, some studies showed that LSRs might face unexpected challenges and difficulties under the COVID-19 pandemic with a changeable epidemiological landscape and substantial clinical uncertainty.13,14

Therefore, this systematic review analyzes and summarizes the characteristics of LSRs for COVID-19, which aims to identify potential problems and provide references to further improve the quality of LSRs for COVID-19.

Methods

This study was a systematic review and conducted in accordance with the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines.15

Six databases were searched systematically, including Medline, Excerpta Medica (Embase), Cochrane Library, China National Knowledge Infrastructure (CNKI), Wanfang Database and China Science and Technology Journal Database (VIP), as of May 16, 2022. The search strategy was specific for each database and included a combination of the medical subject headings and free text terms for “living systematic review” or “living system review” or “living system evaluation” or “living systematic evaluation” or “living meta-analysis”.

This manuscript included all the living systematic reviews for COVID-19 and was not limited by language. Two researchers independently screened the studies. When two researchers had opposite opinions, disputes were decided by the third researcher. EndNote™ X8 software was used for deduplication and back-to-back screening by two researchers.

The information extraction table was designed in advance, and the pretest was conducted with 10% of the included LSR. Two researchers extracted the information of the included LSR back-to-back, and the extracted results were compared by the third researcher. The information extraction table included three parts: (1) general characteristics: title, author, published year, country, institution, name and impact factor of journal, topic, population, quality assessment tool, etc; (2) methodology of LSP: type of LSR, registration information, search methods, search frequency, data synthesis methods, updated frequency, etc; (3) transitioning LSR out of living mode: time or criteria for transitioning LSR out of living mode.

Searched records were managed by EndNote™ (V.X9, Clarivate, Philadelphia, Pennsylvania, USA). Statistical analysis was done by MS Excel 2019. Descriptive analysis and tables were used to present the results. Binary variables (such as first institution’s country, study topics and study population) were displayed in frequency and percentage. Continuous variables (such as impact factor, search frequency, screening frequency, update frequency) were displayed in mean, standard deviation and range. Due to the large heterogeneity of LSR included in this study, meta-analysis was not intended to be performed in this study.

Results

Study Selection

A total of 1132 studies were initially included. After excluding duplicate studies, 1043 studies remained. After reading the title, abstract and full text, 6416–79 studies were included. A summary of the study selection process is reported in Figure 1.

Figure 1

Flow diagram of screening.

Basic Information of LSR for COVID-19

Most (89.1%) LSRs were published in SCI journal.16–21,23–44,46–51,53–57,59–69,73–79 The impact factor published in SCI journals ranged from 1.04 to 39.89, with an average of 11.72 and a standard deviation of 11.30. The impact factor of 64.1% studies was >5,16–20,23,26,27,30,32,33,36,37,39–44,46,47,49–51,54–57,60–64,67–69,73–76,79 the impact factor of 17.2% studies was >15.16,17,23,42,43,51,62–64,74,76 LSRs were published in journals with high impact factor, which indicated COVID-19 related studies have attracted much attention from readers (Table 1). In addition, the included LSRs were distributed in 19 countries and regions around the world, covering high-income countries, upper-middle-income countries, low-middle-income countries and low-income countries.

Table 1

Basic Information of COVID-19 for LSR

Category	Characteristic	Number	Percentage n=64
Impact factor of published journal	Non-SCI	7	10.9%
	IF ≤5	16	25%
	<5 IF ≤10	26	40.6%
	<10 IF ≤15	4	6.3%
	IF ≥15	11	17.2%
First institution’s country	UK	11	17.2%
	Germany	10	15.6%
	Canada	9	14.1%
	US	6	9.3%
	Chile	5	7.8%
	Brazil	3	4.7%
	China	2	3.1%
	France	2	3.1%
	Netherlands	2	3.1%
	Switzerland	2	3.1%
	Denmark	2	3.1%
	Saudi Arabia	2	3.1%
	Spain	2	1.6%
	Czech Republic	1	1.6%
	Uruguay	1	1.6%
	Tanzania	1	1.6%
	Italy	1	1.6%
	Ireland	1	1.6%
	South Africa	1	1.6%
Study topics	Prevalence	4	6.3%
	Epidemic spread	6	9.4%
	Clinical manifestation	11	17.2%
	Etiology	3	4.7%
	Prevention	6	9.4%
	Diagnosis	3	4.7%
	Treatment	26	40.6%
	Prognosis	3	4.7%
	Others	5	7.8%
Study population	General patients with COVID-19	49	76.6%
	Children with COVID-19	4	6.3%
	Pregnant with COVID-19	4	6.3%
	Elderly with COVID-19	0	0%
	Healthy people	5	7.8%
	Health-care workers	4	6.3%

The study topics covered all aspects of the health-care field, including prevalence, clinical manifestation, etiology, prevention, diagnosis, treatment, and prognosis. The topic most studied (40.6%) was the treatment of COVID-19.20,22,25,32,37,39,43,46,47,49,51,55,56,58,62–64,67–74,79 The population most studied (76.6%) was general COVID-19 patients,18–23,26,27,29,31–34,36,37,39,42–44,46,47,49–65,67–76,79 but there were fewer (6.3%) studies concerning pregnant women16,17,30,78 and children28,30,77,78 with COVID-19, even no study concerning the elderly with COVID-19. (Table 1, )

Methodology of LSR for COVID-19

In the production phase of LSR, most of the LSRs registered16–22,25,26,28–33,35,37,39–42,45–62,65–67,69–73,76–79 (81.2%) and wrote the protocol16–35,37,39–43,45–67,69–73,76–79 (92.2%) before the study started. Less than one third of the LSRs searched the Chinese database16–18,23,25,34,35,43,46,48,57,62–64,70,79 (25%), and less than half of the LSRs searched the preprint database16,20,21,23–26,28,37,39,40,43,45,46,48,51,53,57,62–64,67,68,73,76,77,79 (42.2%), and two-thirds of the LSRs searched 12 COVID-19 databases16,17,20–23,27,30,32,33,37–42,44–46,48,49,51,54–56,58–64,67,69–77,79(67.2%)(). More than half (51.6%) of LSRs used the same frequency to search different databases regularly,16–18,20,24–26,30,32,35–37,39,40,43–46,48,49,51,52,55,59,61,62,66–69,73,74,79 and the frequency of searches ranged from once a day to once every six months, with an average of once every 28 days, with a standard deviation of 42.68 (). Seventy-nine point seven percent of LSRs did not report screening frequency.16–31,33–36,38,39,42–48,50,51,53–55,57–59,61–64,66,69–72,74–79 Less than one-third (19.5%) of LSRs took into account the possible false-positive probability of repeated meta-analyses.23,42,46,48,56,63,64 Methods of data synthesis included traditional meta-analysis methods (80.5%) and nontraditional meta-analysis methods (19.5%), which respectively referred to “the statistical methods for meta-analysis in the Cochrane Guidelines for Systematic Reviews” and “use of meta-analytic methods to adjust for frequent updating, such as Bayesian, trial sequential analysis, sequential meta-analysis, the Shuster method, Law of the iterated logarithm2–4”. Only 34.4% of LSRs reported time or criteria for transitioning LSR out of living mode16,17,19,21–24,45,52,54–56,58,62–64,66,70–72,76(). Among them, the time to exit “living” mode is from one to two years, the average exit time is 1.78 years, and the standard deviation is 0.42. (Table 2)

Table 2

Methodology of LSR for COVID-19

Category	Characteristic	Number	Percentage n=64
Registration information	Registration	52	81.2%
	Protocol	59	92.2%
Type of LSR	Living network meta-analysis	5	7.8%
	Living systematic review and meta-analyses	35	54.7%
	Living systematic review only	24	37.5%
Database for literature search	Preprint database	27	42.2%
	COVID-19 Database	44	68.8%
	Chinese database	16	25.0%
Search frequency	Search different databases with different frequencies	5	7.8%
	Search all databases with the same frequency	33	51.6%
	Not mentioned	26	40.6%
Screening frequency	Mentioned	13	20.3%
	Not mentioned	51	79.7%
Methods of data synthesis	Traditional meta-analysis methods	33	80.5%
	Nontraditional meta-analysis methods	8	19.5%
Transitioning LSR out of living mode	Specify when to quit living mode	9	14.1%
	Clarify the criteria for exiting living mode	10	15.6%
	Specify when to quit living mode and clarify the criteria for exiting living mode	3	4.7%
	Not mentioned	42	65.6%
Editorial and peer review	Stable peer reviewera	5	7.8%
	Not mentioned	59	92.2%
Presentation ways to the reader	Peer-reviewed journals	64	100%
	Peer-reviewed journals and websites	12	18.8%
	Peer-reviewed journals, websites and software	3	4.7%
Timing of updates	Regular	22	34.4%
	When the criteria were met	25	39.1%
	Regular + when the criteria were metb	4	6.3%
	Not mentioned	13	20.3%
Number of updates	0	49	76.6%
	1	10	15.6%
	2	2	3.1%
	3	3	4.7%
Technological enablers	Search	7	10.9%
	Eligibility assessment	5	7.8%
	Data extraction or collection	1	1.5%
	Quality evaluation	0	0.00%
	Synthesis	0	0.00%

Notes: a“Stable peer reviewer” meant that the LSR had relatively stable reviewers, who could review in a short time after the new manuscript was submitted. b“Regular + when the criteria are met” meant that the LSR would be updated regularly, but if new evidence could change the conclusion, even if the specified update date was not met, the LSR would also be updated.

In the review, publish and update phase of LSR, only a small number (7.8%) of LSRs had relatively stable peer reviewers.23,42,43,63,64 All the LSRs presented the results to readers through journals. But there were still 23.4% of LSRs that showed the results through a more convenient and quick way (website or software).23,26,32,37,38,42,46,48,50,52,54,63,64,71,72 Thirty-four point 4 percent of LSRs were updated regularly.16,18,19,21,23,26,27,31,35,37,45,50–52,54,56,62,64,65,74,78 The update frequency of LSRs ranged from weekly to annually, with an average of every 95 days, with a standard deviation of 80.51. Seventy-six point six percent of LSRs were never updated (Table 2, and ).

In addition, less than one-fifth (18.8%) of LSRs used technology in the production process,22,23,27,37,45,56,58,64,70–72,76 and most of them (10.9%) were in the search process.22,37,45,58,70–72 (Table 2)

Discussion

This study systematically reviewed 64 LSRs for COVID-19 and analyzed the characteristics of LSR for COVID-19. Meanwhile, we found unsolved challenges in LSRs by summarizing their basic information and methodology of LSRs. Consequently, we generated some suggestions to improve efficiency and quality of LSRs for COVID-19 with regard to these challenges.

It was noteworthy that 18.8% of LSRs for COVID-19 were not registered in advance, which might be the reason for the existence of partial LSRs for COVID-19 repeated in similar topics. For example, three similar LSRs in the clinical topic of remdesivir for COVID-19 were found.20,71,74 In the meantime, three similar LSRs on the topic of cell therapy for COVID-19 also occurred.47,58,63 Potential reasons for the replication might be that some authors considered publishing the study without registration could be quicker, especially under the requirement of numerous SRs applications for registration under the COVID-19 pandemic.80 Although registration might delay publication of LSRs, to improve the quality of LSRs and reduce replicated publications, it was still recommended that LSRs should be registered prior to commencement.81–83

Brierley et al found that nearly 37.5% of the COVID-19 studies were published in the preprint database,84 but only 42.2% of LSRs for COVID-19 searched the preprint database. This might be due to concerns that preprints were not as reliable as peer-reviewed articles,85 and relying on preprints to draw conclusions could cause unstable practical expressions in clinical practice. However, Brierley et al’s study demonstrated that there was no qualitative difference after tracking COVID-19 preprints and final published versions.84 Therefore, considering the complicated and time-consuming progression of traditional peer review to be published and the urgency of COVID-19 pandemic, we suggested that LSRs for COVID-19 considered searching preprint databases, to open up the possibility of providing timely potential solutions for clinical challenges from COVID-19.

Among 38 LSRs for COVID-19, the average search frequency was 28 days, similar to the monthly search frequency suggested by Cochrane Guidance and Lansky.12,86 However, searching monthly was still difficult to persist in most studies. In studies by Cochrane Guidance12 and Millard,8 hours of monthly search could be as high as 32 hours, along with the aggravated workload. Regarding to the elevated time consumption for research of COVID-19, the establishment of COVID-19 Comprehensive Database became increasingly important in the development of LSRs.7,12,87–89 The L·OVE COVID-19 platform, which obtained information from the Epistemonikos database and used artificial intelligence and expert opinions, integrates the information and releases the heavy workload on data search and organization for the researcher.14,90 The L·OVE COVID-19 platform was characterized with real-time updating of COVID-19 studies, Pierre et al and Butcher et al demonstrated that the L·OVE COVID-19 platform had a very good sensitivity for identifying studies evaluating interventions for COVID-19.89,91 To improve work efficiency, LSR for COVID-19 should search a comprehensive database of COVID-19.

Among 20 LSRs for COVID-19, the average search frequency was 95 days, much higher than the yearly update suggested by Créquit et al,92 indicating that research evidence was produced rapidly under the COVID-19 pandemic. However, frequent updates which is premature could drain the research team’s resources and ultimately lead to the disruption of the LSR, which is a problem that would be magnified especially during the COVID-19 pandemic.14 Therefore, LSRs should have an appropriate update frequency in combination with comprehensive research focusing on this newly emerged evidence.

Subsequently, we found that only about one-third of COVID-19 LSRs reported a time or criterion for exiting living mode, and the remaining COVID-19 LSRs did not report or even consider this matter. It was impossible to stay in living mode in terms of time or funding. Therefore, clear time or criterion for exiting living mode is as important as deciding to start living mode in an LSR, especially under the COVID-19 pandemic when it was hard to predict when new evidence would emerge. Taken together, authors of the LSR should periodically review the clinical topics of interest, the speed and amount of new evidence produced, and its own resources, and then combine the above points to decide whether to transition LSR out of living mode.

Meanwhile, we found only 18.8% of LSRs for COVID-19 used machine automation to increase productivity. Thomas et al believed that human resources were scarce resources in LSR, so use of machine automation could improve the feasibility and sustainability of LSR. Automation could assist with some LSR tasks, including searching, eligibility assessment, data extraction or collection, quality evaluation, and synthesis. However, Thomas et al and Tercero-Hidalgo et al also recognized that the existing automation technology was only suitable for the early stages of the LSR production process (ie searching, eligibility assessment, etc), and the automation technology in the later stages of the LSR production process still required further research and development.87,93 Therefore, we propose LSR should be developed with automated techniques.

In addition, the poor reporting quality of the 64 LSRs for COVID-19, especially the part of the methodology of LSR, were observed. Iannizzi et al suggested that transparent and traceable reporting of changes in LSR methods became challenging under the COVID-19 pandemic.14 One of the reasons for this challenge might be currently no reporting guideline for LSRs. Although the PRISMA statement was updated in 2020, it emphasized that when used for LSRs, some additional obstacles need to be addressed.94 Four key modules assessed in traditional SRs (publication format, work processes, author team management, and statistical methods),7 could not be evaluated in PRISMA 2020 statement for LSRs. Therefore, the evaluation process following PRISMA 2020 statement only promises the reporting quality of LSR as a traditional SR, but not the reporting quality of a “living” SR. Fortunately, the deficiency of PRISMA 2020 statement was noticed by the PRISMA team. The specific protocol of PRISMA for LSR had been published, and it is claimed that PRISMA for LSR would be part of the PRISMA extension.9

In contrast, there are still several limitations in this study. First, the methodological quality and reporting quality of LSR has not been considered in this study. Although the recently updated PRISMA 202094 statement claimed that it could be used for LSR, there were still some additional problems. Moreover, the commonly used methodological quality assessment tools AMSTAR 2 and ROBIS were not suitable for LSR.95,96 Second, this study did not identify the optimal search frequency, update frequency, and appropriate technological enablers. In the future, we could focus on developing appropriate PRISMA for LSR and exploring suitable search frequency, update frequency, and more available technological enablers, which may improve the quality of LSR.

Conclusion

LSRs are high-quality, up-to-date online summaries of health research, updated as new research becomes available,7 and enabled by improved production efficiency. Transparent reporting of changes in methodology between review updates is essential. And transparent reporting is needed to avoid biases in the review process. But most of the LSRs for COVID-19 were incomplete in reporting on the “living” process. This could reduce the confidence of health-care providers and policy makers in the results of COVID-19 LSRs, thereby hindering the translation of evidence on COVID-19 LSR into clinical practice. The results of this work not only provide an evidential foundation for PRISMA for the LSR development team, but also make recommendations for further applications of LSR under COVID-19.