Evaluating the methodology of studies conducted during the global COVID-19 pandemic: A systematic review of randomized controlled trials.

BACKGROUND: The therapeutic evidence collected from well-designed studies is needed to help manage the global pandemic of the coronavirus disease 2019 (COVID-19). Evaluating the quality of therapeutic data collected during this most recent pandemic is important for improving future clinical research under similar circumstances.
OBJECTIVE: To assess the methodological quality and variability in implementation of randomized controlled trials (RCTs) for treating COVID-19, and to analyze the support that should be provided to improve data collected during an urgent pandemic situation. SEARCH STRATEGY: PubMed, Excerpta Medica Database, China National Knowledge Infrastructure, Wanfang, and Chongqing VIP, and the preprint repositories including Social Science Research Network and MedRxiv were systematically searched, up to September 30, 2020, using the keywords "coronavirus disease 2019 (COVID-19)," "2019 novel coronavirus (2019-nCoV)," "severe acute respiratory syndrome-related coronavirus-2 (SARS-CoV-2)," "novel coronavirus pneumonia (NCP)," "randomized controlled trial (RCT)" and "random." INCLUSION CRITERIA: RCTs studying the treatment of COVID-19 were eligible for inclusion. DATA EXTRACTION AND ANALYSIS: Screening of published RCTs for inclusion and data extraction were each conducted by two researchers. Analysis of general information on COVID-19 RCTs was done using descriptive statistics. Methodological quality was assessed using the risk-of-bias tools in the Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). Variability in implementation was assessed by comparing consistency between RCT reports and registration information.
RESULTS: A total of 5886 COVID-19 RCTs were identified. Eighty-one RCTs were finally included, of which, 45 had registration information. Methodological quality of the RTCs was not optimal due to deficiencies in five main domains: allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective reporting. Comparisons of consistency between published protocols and registration information showed that the 45 RCTs with registration information had common deviations in seven items: inclusion and exclusion criteria, sample size, outcomes, research sites of recruitment, interventions, and blinding.
CONCLUSION: The methodological quality of COVID-19 RCTs conducted in early to mid 2020 was consistently low and variability in implementation was common. More support for implementing high-quality methodology is needed to obtain the quality of therapeutic evidence needed to provide positive guidance for clinical care. We make an urgent appeal for accelerating the construction of a collaborative sharing platform and preparing multidisciplinary talent and professional teams to conduct excellent clinical research when faced with epidemic diseases of the future. Further, variability in RCT implementation should be clearly reported and interpreted to improve the utility of data resulting from those trials.

Introduction

The coronavirus disease 2019 (COVID-19) spread throughout many countries around the world and became a public health emergency of international concern [1]. Up to September 30, 2020, COVID-19 had caused more than 54.77 million infections and more than 1.32 million deaths [2]. Drugs that effectively treat the infection are urgently needed to save lives. However, as clinicians respond to the emerging crisis of a pandemic they must make rapid decisions based on the best available data, which is often less than they would like. These decisions are often based on authority, personal clinical experience or data from clinical trials that have not yet been through peer review [3]. Although remdesivir was approved for the treatment of COVID-19 by the U.S. Food and Drug Administration (FDA) on Oct 22, 2020 [4], [5], it was not recommended for hospitalized COVID-19 patients in the update to living World Health Organization (WHO) guideline on drugs for COVID-19 [6]. Obtaining better therapeutic evidence is urgent as a crisis unfolds, but the urgency of the pandemic makes it difficult to wait for better evidence and difficult to collect high-quality evidence-based data.

Since COVID-19 broke out, large numbers of COVID-19 clinical trials have been registered in the WHO International Clinical Trials Registry Platform (ICTRP) and ClinicalTrials.gov. More evidence is expected to be obtained from the above clinical trials in the near future. According to the statistics, there had been 6172 clinical trials registered in ICTRP and ClinicalTrials.gov up to September 30, 2020, including 3609 interventional studies and 2430 randomized controlled trials (RCTs) [7]. Further, more than 123,000 articles relevant to COVID-19 had been released in the WHO platform, including 1732 documenting studies that were controlled clinical trials [8]. Before results from such studies are used to inform clinical practice, their validity needs to be assessed.

Following the methodological principles of randomization, control and blinding, in evidence-based medicine, RCTs are supposed to produce data of the highest quality. Evidence from well-designed RCTs is needed to conclusively identify what interventions should be applied or discontinued to avoid overtreatment and potential risks in the COVID-19 pandemic. With the increasing number of registered clinical trials and the release of clinical trial results, some researchers have performed systematic reviews to analyze registered COVID-19 clinical trials [9] or review the evidence of drugs for treatment of COVID-19 [10]. But no study has systematically evaluated the methodology of RCT evidence. To provide such an overview, we conducted a systematic review of RCTs for treating COVID-19 to assess the methodological quality and variability in implementation and analyze the support that should be provided to obtain good therapeutic evidence in an urgent pandemic situation.

Methods

Data source

PubMed, Excerpta Medica Database (via Ovid), China National Knowledge Infrastructure, Wanfang, Chongqing VIP, as well as Social Science Research Network (SSRN) and MedRxiv preprint platforms were searched, up to September 30, 2020, for all RCTs intending to treat COVID-19.

Search strategy

The keywords included “coronavirus disease 2019 (COVID-19),” “2019 novel coronavirus (2019-nCoV),” “severe acute respiratory syndrome-related coronavirus-2 (SARS-CoV-2),” “novel coronavirus pneumonia (NCP),” “randomized controlled trial (RCT),” and “random.” Full searching strategies for each database and the two preprint platforms are shown in the supplementary file, Table S1.

Inclusion and exclusion criteria

Studies that meet the following criteria were included. (1) patients: patients with confirmed or suspected COVID-19; (2) intervention: any intervention related to the treatment of COVID-19; (3) outcome: any primary or secondary outcomes (e.g., vital signs, laboratory tests, imaging examination and clinical events); (4) study type: RCTs; (5) there was no limitation on controls and languages.

Exclusion criteria: (1) studies on diagnosis, prevention and epidemiology; (2) non-RCT studies, single-arm studies, case-control studies, cohort studies or other observational studies; (3) duplicated studies; (4) studies with insufficient information; (5) studies that had not been completed and only reported preliminary or interim results; (6) theoretical analysis, literature review, systematic review or trial protocols.

Study selection

All identified records were imported into EndNote x9.0 for management. Firstly, duplicate records were excluded by computer and manual search. Then, titles and abstracts of the remaining studies were screened for inclusion using the above criteria. Finally, full texts of the possibly relevant studies were read. Study selection was performed by two researchers (SST and MZZ). Disagreements were resolved by consensus through a meeting among three researchers (CZ, SST and MZZ).

Data extraction

An electronic data form was used to extract information on registration number, languages, first release time, ethical approval, number of recruitment sites, sample size, categories of participants, interventions, controls, blinding and outcomes. Two researchers (SST and MZZ) extracted data from the included studies independently. Disagreements were resolved by discussion with the third researcher (CZ). All extracted data were managed in Microsoft Office Excel (Microsoft Office 2016).

Assessing methodological quality of included studies

The methodological quality of included studies was assessed independently by two researchers (SST and MZZ) using the risk-of-bias (RoB) tools in the Cochrane Handbook for Systematic Reviews of Interventions (V.5.1.0) [11]. Evaluation included seven domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, and “other bias,” which was defined as baseline characteristics of trials not similar between different intervention groups and sources of bias that were only found in a particular clinical setting. Each domain was scored as low risk, high risk, or unclear risk by discussion among researchers (CZ, SST and MZZ).

Assessing variability in implementation

The variability in implementation was assessed by comparing consistency between RCT reports and registration information. Only studies with registration numbers could have their study methods checked against the original registration information for consistency. Consistency of seven items was compared, including inclusion and exclusion criteria, sample size, outcomes, sites of recruitment, interventions and blinding.

Data analysis

For our analysis, the RCTs that met our inclusion criteria and were published in Chinese or English language from February 1, 2020 to September 30, 2020 were grouped by the month of each study’s first release. We defined the time of first release as the date that each study was first posted on the internet. Studies with registration information were also grouped by month. For each month, the number of studies with registration information, ethical approval, site of recruitment, and sample size were documented.

“PICO” information on RCTs was analyzed, including categories of participants, interventions, controls, blinding, and outcomes. Adult patients with confirmed COVID-19 were grouped into different categories, according to the severity of their symptoms. For example, Diagnosis and Treatment Protocol for COVID-19 (Trial Version 7) edited by China National Health Commission (NHC) points out that confirmed cases are identified as mild, moderate, severe or critical case through their condition [12]. However, adults with COVID-19 can be defined as asymptomatic or pre-symptomatic, mild illness, moderate illness, severe illness or critical illness in the COVID-19 Treatment Guidelines from National Institutes of Health (NIH) [5]. Though the categorical criteria may be different among guidelines and clinical trials from different countries, the clinical status of patients included in our study was uniformly classified into four categories: mild, moderate, severe or critical. Our mild classification included NHC mild cases, asymptomatic, pre-symptomatic and mild illness, according to NIH and mild patients from other countries. Our moderate classification included NHC moderate cases, NIH moderate cases, and moderate patients from other countries. Our severe classification included NHC severe cases, NIH severe illness, and severe patients from other countries. Our critical classification included NHC critical cases, NIH critical illness, and critical patients from other countries. Interventions used in the RCTs were coded into broader categories (e.g., lopinavir/ritonavir and remdesivir were categorized as antivirals). Primary and secondary outcomes were also coded into broader categories, including vital signs, laboratory tests, imaging examination, clinical events, and psychological status. Controls included interventional control, placebo control, and blank control. Interventional control was defined as a controlled group of patients that received any treatments that might be considered effective for treating COVID-19, including mutual control such as comparisons among different antiviral drugs or same antiviral drug with different doses.

The methodology of included RCTs were analyzed using RoB graphs and RoB summary figures. A subgroup analysis of RoB assessment was conducted in 2–3 months groups, according to first release time. Comparisons of consistency are reported in frequency and percentage.

Descriptive statistics were conducted using Microsoft Office Excel (Version 2016). RoB graphs and RoB summary figures were generated using Review Manager (RevMan; Version 5.4).

Results

A total of 5886 studies were identified from Chinese and English databases, and preprint platforms (SSRN and MedRxiv). Titles and abstracts from 4022 studies were screened after removing duplicate records. Then the full text of 203 studies were read. Ultimately, eighty-one studies met our inclusion criteria. The details of the study selection process are shown in Fig. 1 .

Fig. 1

Flow diagram of searching and screening randomized controlled trials for treating coronavirus disease 2019 (COVID-19). CNKI: China National Knowledge Infrastructure; VIP: Chongqing VIP; Embase: Excerpta Medica Database; SSRN: Social Science Research Network; RCT: randomized controlled trial. The number of records from the two-preprint platform was identified through screening titles and abstracts.

Characteristics of included studies

Of the 81 studies reviewed, 30 (37.0%) were reported in Chinese and 51 (63.0%) in English. The earliest study was released in the Chinese Journal of Virology by Zhou, et al. [13], with the title of “Clinical value of diammonium glycyrrhizinate in treatment of COVID-19,” on February 28, 2020. The first English language report was posted online in the Complementary Therapies in Clinical Practice, written by Liu et al. [14], and titled “Effects of progressive muscle relaxation on anxiety and sleep quality in patients with COVID-19,” and released on February 29, 2020. Neither of these studies mentioned registration information. The number of RCT studies on treating COVID-19 rose in February 2020, reached a peak in April and decreased in May. After that, the number of studies in Chinese increased slightly from June to September. Meanwhile, the number of studies in English rose in July and reached a second peak in August. A similar trend was observed in the 45 studies with registration information, except for February when there were no studies with registration information. The number of studies included from February 1, 2020 to September 30, 2020 is shown in Fig. 2 .

Fig. 2

The number of included randomized controlled trials for treating coronavirus disease 2019 from February to September, 2020.

Sixty-five of the included studies (80.2%) were approved by ethical committees, and other 16 studies (19.8%) did not mention whether they received ethical approval or not. A large proportion of studies recruited patients from a single site 65.4% (53/81), however, 5 studies recruited participants from three sites, and 3 studies recruited participants from two sites or six sites. The largest number of recruitment sites was 121 [15]. The sample size varied from 12 to 596. Sample size between 12 and 100 accounted for the largest proportion (52/81, 64.2%), followed by the range of 101–200 (13/81, 16.0%). The distribution of sample size is shown in Fig. 3 . The general characteristics of the included RCTs of COVID-19 treatment are summarized in Supplementary file Tables S2 and S3.

Fig. 3

The number of included randomized controlled trials for treating coronavirus disease 2019 in different sample size intervals.

Characteristics of PICO

Among the included studies, 66.7% (54/81) involved confirmed COVID-19 patients with different subgroups (e.g., mild case and moderate case) which were classified in accordance with the severity of the condition, and 30.9% (25/81) included confirmed COVID-19 patients but did not use severity subgroups. Two studies paid attention to convalescent patients [16], [17]. Two studies only included suspected COVID-19 patients [18], [19]. The details are presented in Fig. 4 A and B. Interventions used in the included studies were variable and included treatments such as Chinese herbs, antiviral drugs and plasma therapy. The details are presented in Fig. 4C. The most frequent control was the interventional control, accounting for 91.4% (74/81). A small proportion of studies applied placebo control (7.4%, 6/81). The relevant data are shown in Fig. 4D. Thirty-six of the included studies (44.4%) did not report whether or not blinding was used; 8.6% (7/81) used blinding, including 5 studies with double-blind design [18], [20], [21], [22], [23] and 2 studies with single-blind design [24], [25]. Open label design was used in 46.9% (38/81) of the studies. The distribution of blinding methodologies is shown in Fig. 4E. About 56% (45/81) of studies reported primary and secondary outcomes. Detailed information is shown in Table 1 .

Fig. 4

Characteristics of patients, interventions, controls and the blinding in included randomized controlled trials (RCTs) for treating coronavirus disease 2019 (COVID-19). A: The number of included RCTs for treating COVID-19 in different patients’ categories. *COVID-19 group included suspected and confirmed COVID-19 patients and confirmed but asymptomatic COVID-19 patients were included in the mild group. Take the mild group as an example: a study was counted on the principle that it reported either the mild only or with other subgroups (the moderate, the critical and the severe). B: Distribution of categories in COVID-19 patients. One category: only report any one of the four categories (the mild, the common, the severe, and the critical); two categories: report any two of the above four categories; three categories: report any three of the above four categories; four categories: report all the above four categories. No category: studies reported patients without specific categories, or convalescent COVID-19 patients. C: Distribution of interventions in RCTs for treating COVID-19. Chinese herbs included traditional Chinese medicine, Chinese patent medicine and traditional Chinese medicine injection. Antiviral drugs referred to lopinavir/ritonavir, hydroxychloroquine, remdesivir, abidor, interferon β-1α, etc. Non-drug therapy included functional exercise, Baduanjin and oxygen therapy. Other drugs refer to hepatoprotective drugs, antioxidants, azithromycin, colchicine and glucocorticoid. D: Distribution of control types in RCTs for treating COVID19. E: Distribution of blinding in RCTs for treating COVID-19. All Arabic numerals in the above figures represented the number of studies. The percentage represented the proportion of studies under each category to the total studies.

Table 1

Outcome information on included RCTs for treating coronavirus disease 2019 (COVID-19).

Category	Outcome measure	Specifics	Number (%)
Outcomea	Vital signs	Temperature, respiration, pulse, blood pressure and blood oxygen	10/81 (12.3)
	Laboratory tests	Inflammation indexb	32/81 (39.5)
		Immunological indicators (T/B cell subsets)	6/81 (7.4)
		Nucleic acid negative rate	5/81 (6.2)
	Imaging examination	Chest computerized tomography	13/81 (16.0)
	Clinical events	Improvement of clinical symptoms (fever, cough and fatigue)	20/81 (24.7)
		Clinical efficacy	16/81 (19.8)
		Disease aggravation/mechanical ventilation time, hospitalization time, clinical status, etc.	21/81 (25.9)
	Psychological status	Psychological scale (for example: SDS and SAS)	5/81 (6.2)
Primary outcome	Vital signs	Temperature, respiration, pulse, blood pressure, and blood oxygen	2/81 (2.5)
	Laboratory tests	C-reaction protein and troponin	2/81 (2.5)
		Detection of virus nucleic acidc	15/81 (18.5)
	Clinical events	Improvement/aggravation of clinical symptoms (fever, cough and fatigue), and clinical statusd	35/81 (43.2)
		Mortality rate	6/81 (7.4)
Secondary outcome	Vital signs	Temperature, respiration, pulse, blood pressure and blood oxygen	17/81 (21.0)
	Laboratory tests	Laboratory tests	3/81 (3.7)
		Detection of virus nucleic acid	18/81 (22.2)
	Imaging examination	Incidence rate of chest imaging	7/81 (8.6)
	Clinical events	Improvement/aggravation of clinical symptoms (fever, cough and fatigue), and clinical status	41/81 (50.6)
		Clinical efficacy	7/81 (8.6)
		Aggravation from mild/moderate type to sever/critical type, mechanical ventilation	30/81 (37.0)
		Mortality	23/81 (28.4)
		Hospitalization/intensive care unit time	22/81 (27.2)
	Psychological status	Psychological scale (for example: SDS and SAS)	10/81 (12.3)

(APACHE II) scores, 10-point visual analogue scale, 6 or 7-point ordinal scales, etc. SDS: Self-Rating Depression Scale; SAS: Self-Rating Anxiety Scale. a: Outcome refers to outcomes do not point primary outcome or the secondary outcome. b: Inflammation index includes blood routine (white cell count and lymphocyte count are the key outcome), C-reactive protein, procalcitonin and d-dimer, etc. c: Detection of virus nucleic acid refers to the viral load, viral nucleic acid negative conversion rate and time. d: Clinical status is assessed by scales, such as acute physiology and chronic health evaluation Ⅱ.

Methodological assessment of included studies

All 81 studies mentioned the randomized allocation of participants. One study randomly assigned patients depending on patients’ medical record numbers [26], while most of the others (53/81, 65.4%) used a random number table or a random permuted block generated by computers. Given limited information in the reports, studies were rated as having unclear risk in random sequence generation (27/81, 33.3%), allocation concealment (30/81, 37.0%), blinding of participants and personnel (29/81, 35.8%), blinding of outcome assessment (33/81, 40.7%), incomplete outcome (27/81, 33.3%), selective reporting (36/81, 44.4%), and other biases (71/81, 87.7%). About 33% (27/81) of studies were determined to have a high risk of selective reporting, due to revised or partly reported outcomes. About 17% (14/81) of studies that had incomplete outcomes were rated as high risk, as some patients dropped out, resulting in imbalance between experimental groups and controlled groups. In subgroup analysis, the proportion of studies rated as having low risk in random sequence generation and allocation concealment increased over time. Meanwhile, the proportion of studies determined to have high risk of bias for blinding of participants and personnel, blinding of outcome assessment and selective reporting also increased. The RoB graph of all 81 studies is shown in Fig. 5 A. RoB graphs of subgroup analysis are shown in Fig. 5B–D. The RoB summary for all included RCTs of COVID-19 treatment is shown in Supplementary file Fig. S1.

Fig. 5

Risk of bias assessment of included randomized controlled trials (RCTs) for treating coronavirus disease 2019 (COVID-19). A subgroup analysis of risk of bias assessment was conducted by every 2–3 months in accordance with the first release time. A: Risk of bias assessment of included 81 RCTs for treating COVID-19. B: Risk of bias assessment of included 30 RCTs for treating COVID-19 from February to April. C: Risk of bias assessment of included 28 RCTs for treating COVID-19 from May to July. D: Risk of bias assessment of included 23 RCTs for treating COVID-19 from August to September.

Comparison of RCT reports and registration information

Among the included 81 studies, 45 (55.6%) reported registration information which included the following agencies: Chinese Clinical Trial Registry (n = 18), ClinicalTrials.gov (n = 23), Iranian Registry of Clinical Trials (n = 3) [27], [28], [29], European Union Drug Regulating Authorities Clinical Trials (EudraCT; n = 4) [15], [23], [30], [31] and Clinical Trials Registry-India (n = 1) [32]. Four of these studies were registered in both EudraCT and ClinicalTrials.gov. About 44% of studies (36/81) did not state whether they were registered or not, and only one study in Chinese was registered [33]. Further, 45 studies with registration information had inconsistencies in seven items. The top inconsistency was that published sample size did not meet the target sample size, and this accounted for about 58% (26/45) of studies. The second most common inconsistency was revision of guidelines, such as inclusion criteria and primary outcomes. The third most common one was the expansion of secondary outcomes that the study considered. The detailed information is presented in Table 2 .

Table 2

Comparison of included RCTs with registration information.

Category	Content	Number and percentage of researches in content (n [%])	Number and percentage of researches in category (n [%])
Inclusion criteriaa	Items reduced	6/45 (13.3)	27/45 (60.0)
	Items increased	5/45 (11.1)
	Item content revisedb	18/45 (40.0)
	Consistent with registration information	17/45 (37.8)
Exclusion criteriaa	Items reduced	10/45 (22.2)	22/45 (48.9)
	Items increased	5/45 (11.1)
	Item content revisedb	6/45 (13.3)
	Consistent with registration information	17/45 (37.8)
Sample size	Sample size reduced	26/45 (57.8)	37/45 (82.2)
	Sample size increased	11/45 (24.4)
	Consistent with registration information	7/45 (15.6)
Primary outcome	Items of outcome reduced	5/45 (11.1)	21/45 (46.7)
	Items of outcome increased	1/45 (2.2)
	Outcome content revisedb	15/45 (33.3)
	Time point of outcome measure revisedc	2/45 (4.4)
	Consistent with registration information	24/45 (53.3)
Secondary outcome	Items of outcome reduced	12/45 (26.7)	37/45 (82.2)
	Items of outcome increased	15/45 (33.3)
	Outcome content revisedb	10/45 (22.2)
	Time point of outcome measure revisedc	8/45 (17.8)
	Consistent with registration information	7/45 (15.6)
Research recruitment locations	Locations reduced	10/45 (22.2)	19/45 (42.2)
	locations increased	9/45 (20.0)
	Consistent with registration information	24/45 (53.3)
Intervention	Experimental groups reduced	1/45 (2.2)	10/45 (22.2)
	Experimental groups increased	1/45 (2.2)
	Intervention revisedd	9/45 (20.0)
	Consistent with registration information	35/45 (77.8)
Blinding	Blinding not implemented	3/45 (6.7)	3/45 (6.7)
	Consistent with registration information	38/45 (84.4)

In the third column, the Arabic numeral before the symbol “/” represents the number of contents in each category. The percentage represents the frequency of the compared contents. The Arabic numeral “45” is the number of studies with registration information. a: Inclusion criteria: there are six studies with insufficient information so that the consistency cannot be compared in inclusion criteria or exclusion criteria. b: Content revised means contents were changed compared to the original registration information, including two cases: one is that the content of items is revised; the other one is that one item is not reported but a new item is reported. c: Time point of outcome measure: time point of outcome measure reported is changed compared to the original registration information, including reducing, increasing or changing the time point of outcome measure in the original protocol. E.g., one study plans to measure the outcome of viral nucleic acid negative conversion rate on days 3, 7, 14; in fact, the study only reports the outcome on days 3 and 14, and increases the outcome on day 10 additionally, or reports other time except for the above 3 time points. d : Intervention revised including drugs changed, dosage and frequency changed.

Discussion

The FDA strongly recommends conducting randomized placebo-controlled, double-blind clinical trials and using a superiority design when evaluating drugs to prevent or treat COVID-19 [34], [35]. However, among the studies we reviewed, only 4 trials used a randomized placebo-controlled, double-blind design [18], [21], [22], [23], and one doubled-blind trial adopted an open label approach [28]. In this study, we assessed methodological quality and variability in implementation of RCTs studying the treatment of COVID-19. The results indicated that the methodological quality was not high, and that deficiencies in methodology have not been improved significantly over time, especially in blinding of participants and personnel, blinding of outcome assessment, and selective reporting. In addition, deviation from registered study protocols was common among the RTCs we reviewed.

In this study, we find several problems. First, many clinical researches were registered [7], but the amount of therapeutic evidence for COVID-19 from RCTs was less. A previous study reported that there were proportionally fewer well-designed RCTs published during an urgent situation. Chang et al. [36] found that among 338 COVID-19 registered trials, only 16 had adopted randomized, single-blind or double-blind designs, and only 8 studies had used randomized, double-blind, placebo-controlled design. Xie et al. [37] also found that RCT was the most common design in 51 pilot studies, but only 10 of those studies had used blinding. Second, methodology quality of the studies was insufficient. Zhu et al. [38] evaluated risk of bias in COVID-19 clinical trials registered in the ChiCTR and the ClinicalTrials.gov databases. The review indicated that the methodological quality of both the interventional trials and observational studies was low, and that reliable, high-quality clinical evidence was not provided by the trials. In this study, there was high risk of bias in allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective reporting. Third, inconsistency was common between RCT reports and registration information. In 2005, the International Committee of Medical Journal Editors indicated that investigators needed to register clinical trial before the onset of patient enrollment [39]. About 56% of the studies we reviewed reported a registration number, but all of them had inconsistencies between the registered protocol and study report; discrepancies included inclusion and exclusion criteria, sample size, outcomes, research sites of recruitment, interventions or blinding.

Faced with the public health emergency of COVID-19, which was characterized by rapid spread, a rush to respond, and poor understanding of the disease, poor support may have made it more difficult to conduct a clinical trial with excellent methodological design and implementation.

First, at the beginning of the epidemic there were no pre-existing well-designed protocols and there was no database describing relevant primary and secondary outcomes. Further, the complexity of patient conditions and the disease also made it more difficult to design clinical trials, calculate the sample size, and choose appropriate outcomes to monitor. Second, under ethical considerations, patients in control groups might receive antiviral treatment. What’s more, the enrollment of patients with different underlying comorbidities made it more difficult to achieve balanced randomization, blinding, and control [40]. Psychological factors in patients increased the complexity and reduced compliance. Research was further complicated by heightened sanitation procedures that prevented care providers from enter hospitals as easily as usual to conduct their research. Meanwhile, medical staff were unable to fully assist in conducting trials, as a higher priority was set on providing essential care [41]. Third, clinical resources were limited. Any shortage of financial, material or personnel support at each link could affect the quality of data collected in a study. For example, an inadequate supply of placebo would affect the observation of specific effects and the assessment of research bias. Finally, the progression of the COVID-19 epidemic was unpredictable. As the epidemic was gradually controlled in some countries, such as China, some clinical trials that had just-launched, or were about to be launched, might not have been able to recruit sufficient participants, forcing studies to be terminated early. In other cases, recruitment failure may have resulted in trials whose sample size was insufficient for proper statistical analysis. Yet, in an urgent situation with multiple complicating factors, some variability in methodologies may be acceptable, providing they are clearly reported and explained.

Clinical researches are hoping for more support that may come from such institutions as government departments, research institutions, pharmaceutical factories, or logistics groups. Specifically, more methodological support is needed to obtain reliable therapeutic evidence. Innovative, flexible and fast clinical study designs are needed to get past the limitations of using RCTs for rapidly evolving health-care crises. Trials with an adaptive design could be stopped early, according to the situation’s needs. The sample size and the randomization can be adjusted to improve the efficiency of clinical trials [42], [43], which is consistent with the complexity and unknowns faced by clinical research in the COVID-19 pandemic. Real-world research has a wide range, including observational research and interventional research. Compared with RCT, patients of real-world research come from medical institutions, families and communities [44]. It has fewer strict restrictions and can reflect the effect of the actual diagnosis and treatment process; its open characteristics make it more suitable for conducting clinical research on COVID-19.

It is necessary to cultivate multidisciplinary talent with systematic knowledge and to build professional teams with interdisciplinary collaboration.

Faced with limited clinical resources, it is urgent to get support by sharing resources and establishing a network and platform for collaboration. With a background of multidisciplinary collaboration, support from the internet and artificial intelligence technology will contribute to resource sharing. Some researchers have made suggestions for such collaboration. Yao et al. [45] described a concept of a national clinical trials network to enhance quality and efficiency of clinical research. Zhang et al. [46] proposed to establish a collaboration and sharing mechanism for clinical trials. In the short term, a collaborative sharing network and platform helps avoid waste of resources and promote transparency of clinical research. In the long term, the results of research can be used to make preparations to manage the epidemics of the future. Meanwhile, the proposed network can provide historical data from multiple sites, as well as centralize data from external clinics operating during the same period; these data might be used as a control for single-arm trials.

In our study, there were several limitations. First, we only searched and analyzed RCTs for treating COVID-19. There might be other high-quality evidence in clinical research with other study designs. Second, we only compared consistency in partly included RCT reports with registration information; the transparency of other studies could not be assessed. Third, with consideration of the great differences in interventions and categories of COVID-19 patients across the included studies, we only evaluated methodological quality of COVID-19 RCTs. Meta-analysis was not performed.

Conclusion

During the urgent response to the COVID-19 pandemic, the methodological quality of RCTs was consistently poor and variability in implementation was common. More support is needed for implementing the methodologies that are necessary to obtain good therapeutic evidence. We are making an urgent appeal for accelerating the construction of a collaborative sharing platform and preparing multidisciplinary talent and professional teams to conduct clinical research in future epidemics. Further, deviation in RCT implementation should be clearly reported and interpreted to improve the utility of data resulting from those trials.

Funding

This study was supported by the National Key R&D Program of China (2019YFC1710400; 2019YFC1710405) and the (No. 81803963). The funding agency had no role in study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.

Author contributions

The manuscript was conceived and designed by HCS. The data were collected and assessed by MZZ and SST. The article was drafted by CZ, MZZ and XXW.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.