How informative were early SARS-CoV-2 treatment and prevention trials? a longitudinal cohort analysis of trials registered on ClinicalTrials.gov.

BACKGROUND: Early in the SARS-CoV-2 pandemic, commentators warned that some COVID trials were inadequately conceived, designed and reported. Here, we retrospectively assess the prevalence of informative COVID trials launched in the first 6 months of the pandemic.
METHODS: Based on prespecified eligibility criteria, we created a cohort of Phase 1/2, Phase 2, Phase 2/3 and Phase 3 SARS-CoV-2 treatment and prevention efficacy trials that were initiated from 2020-01-01 to 2020-06-30 using ClinicalTrials.gov registration records. We excluded trials evaluating behavioural interventions and natural products, which are not regulated by the U.S. Food and Drug Administration (FDA). We evaluated trials on 3 criteria of informativeness: potential redundancy (comparing trial phase, type, patient-participant characteristics, treatment regimen, comparator arms and primary outcome), trials design (according to the recommendations set-out in the May 2020 FDA guidance document on SARS-CoV-2 treatment and prevention trials) and feasibility of patient-participant recruitment (based on timeliness and success of recruitment).
RESULTS: We included all 500 eligible trials in our cohort, 58% of which were Phase 2 and 84.8% were directed towards the treatment of SARS-CoV-2. Close to one third of trials met all three criteria and were deemed informative (29.9% (95% Confidence Interval 23.7-36.9)). The proportion of potentially redundant trials in our cohort was 4.1%. Over half of the trials in our cohort (56.2%) did not meet our criteria for high quality trial design. The proportion of trials with infeasible patient-participant recruitment was 22.6%.
CONCLUSIONS: Less than one third of COVID-19 trials registered on ClinicalTrials.gov during the first six months met all three criteria for informativeness. Shortcomings in trial design, recruitment feasibility and redundancy reflect longstanding weaknesses in the clinical research enterprise that were likely amplified by the exceptional circumstances of a pandemic.

Introduction

Starting in early 2020, commentators warned of COVID-19 clinical trial design deficiencies and lack of coordination of research efforts [1-4]. The large volume of small trials investigating the efficacy of repurposed medications, such as hydroxychloroquine, in the treatment of COVID-19, drew particular attention [5,6]. Such studies confounded an effective public health response by producing spurious findings, or by diverting patients and resources from well designed and executed studies.

Appropriate design, implementation and reporting is captured by the concept of trial “informativeness” [3,7]. For a trial to be informative to clinical practice, it must fulfill five conditions [3,7]. First, it must ask a clinically important question. Second, it must be designed to provide a clear answer to that question. Third, it must have both a feasible enrollment target and primary completion timeline. Fourth, it must be analyzed in a manner that supports statistically valid inference. Fifth, it must report results in a complete and timely manner [3,7].

In the following longitudinal cohort analysis of SARS-CoV-2 treatment and prevention trials registered within the first 6 months of 2020, we assess three features of an informative clinical trial—potential redundancy, design quality and feasibility of patient-participant recruitment. Multiple cross-sectional analyses and systematic reviews of SARS-CoV-2 treatment and prevention trials have been performed [2,5,6,8-11], reporting on intervention types, study characteristics and choice of outcome measure. We go beyond a description of trial characteristics and provide the first in-depth evaluation of SARS-CoV-2 trial informativeness. Knowing the prevalence of potentially uninformative trials conducted in the early stages of the pandemic can help motivate the development of more effective research policy in anticipation of future public health crises.

Methods

Sample, design and trials selection

Our cohort consisted of interventional SARS-CoV-2 treatment and prevention trials registered on ClinicalTrials.gov with a start date between 2020-01-01 and 2020-06-30. We included “Completed”, “Terminated”, “Suspended”, “Active, not recruiting”, “Enrolling by invitation” and “Recruiting” Phase 1/2, Phase 2, Phase 2/3 and Phase 3 interventional clinical trials testing an efficacy hypothesis in their primary outcome. We included trials evaluating any of the following interventions: drug, biological, surgical, radiotherapy, procedural or device. We excluded trials evaluating behavioural interventions, trials of natural products and Phase 1 trials, all of which have no legal requirement to register on ClinicalTrials.gov [12]. See S1 File for complete inclusion/exclusion criteria. Trial inclusion and exclusion criteria were independently assessed by two researchers (KK & LZ), with disagreements resolved by an arbiter (NH or MW). We did not perform a sample size calculation, as we included all trials meeting our eligibility criteria within our designated sampling timeframe.

Data curation

We downloaded clinical trial data directly as a zipped folder of XML files from the web front-end of ClinicalTrials.gov on 2020-12-01 and again on 2021-01-04 (see S2 File for ClinicalTrials.gov search criteria). This allowed us to evaluate data at the 6-month mark (from date of trial start) for all trials in our cohort (see S3 File for data directly downloaded from ClinicalTrials.gov). Additional items requiring human curation were independently assessed and coded by two researchers (KK & LZ), these included: i) treatment type (according to the World Health Organization (WHO) COVID-19 Classification of treatment types [13]); ii) illness severity (as stated by the study investigators or guided by the WHO disease severity classification [14]); iii) location of care (ambulatory, hospitalized, intensive care, unclear/not stated); iv) presence of a placebo or standard of care arm; and, v) type of primary outcome (clinical, surrogate, procedural) (see S4 File for additional double-coded data points). Disagreements were resolved by an arbiter (NH or MW) (Please see S1 Table for inter-rater agreement).

Measures

Trials were assessed based on three elements of informativeness: i) potential redundancy (as a marker of trial importance); ii) trial design quality; and iii) successful patient-participant recruitment (as a marker of feasibility). Assessment criteria for each element were designed based on face validity and easy applicability over a large trial sample.

Potential redundancy

We assessed potential redundancy by evaluating non-redundancy of the trial hypothesis. Non-redundancy was defined as: absence of a trial of the same phase, type of trial (SARS-CoV-2 prevention versus treatment), patient-participant characteristics (including location of care, disease severity and age of trial participants), regimen (including interventions used in combination in a single arm), comparator arm(s) and primary outcome (evaluating primary outcome domain and specific measurement, based on framework from [15]) launched prior to the start date of the trial of interest (as indicated in the registration record active at the 6-month mark since trial start) (S5 File). Only the trial with the later start date was labelled as potentially redundant. The assessment was independently performed by two raters (NH & KK), with disagreements resolved by an arbiter (MW of BC). We performed an additional post hoc assessment applying a broad criterion for trial similarity, which we defined as presence of a trial with an earlier start date of the same type, phase, patient-participant characteristics and treatment regimen.

Design quality

We analyzed trial design quality for those studies in our sample that were aimed at informing clinical practice–namely Phase 2/3 and Phase 3 trials. Based on the U.S. Food & Drug Association (FDA) May 2020 guidance document for SARS-CoV-2 drug and biological treatment and prevention trials [16], we considered a trial to be well-designed if it was randomized, placebo-controlled or with a standard of care comparator arm, double-blinded and included participants aged 60 years or over (as a proxy for an at-risk population). To be considered well-designed, a trial must also measure an appropriate primary outcome–a clinical primary outcome in the case of trials aimed at treating COVID-19, or the presence of laboratory-confirmed SARS-CoV-2 infection for trials testing a preventive measure.

Feasibility of patient-participant recruitment

We assessed timeliness and success of patient-participant recruitment for each trial in our cohort. A single trial was considered non-feasible if it met any of the following criteria: i) trial status was “terminated” or “suspended” and reason for stopping contained a rationale unrelated to trial efficacy, safety or the progression of science; ii) trial status was “completed” or “active, not recruiting” and final enrollment was less than 85% of the anticipated enrollment reported in the trial registration at the time of trial launch (given concerns for compromised statistical power for the primary outcome when recruitment is below the stated threshold (based on previously published methods [17]); or, iii) trial status was “recruiting” or “enrolling by invitation” and the recruitment period had been extended to at least twice as long as the anticipated length in the version of ClinicalTrials.gov registration record at the time of trial start.

Data analysis

We report the overall proportion of trials meeting all three criteria of informativeness (potential redundancy, design quality and feasibility of patient-participant recruitment) as well as the proportion meeting each of our three criteria. We performed a stratified analysis of the proportion of i) non-redundant; ii) well-designed; and iii) feasible trials by sponsor (industry versus non-industry), trial country location (USA versus non-USA), trial type (treatment versus prevention) and number of trial centers (single center versus multicenter). Ninety-five percent confidence intervals were calculated for the difference between two proportions using the prop.test package in R [18]. All tests were 2-tailed. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies (S1 Checklist) [19].

Tools and data synthesis

We performed data extraction using Numbat Systematic Review Manager v. 2.11 (RRID:SCR_019207) [20]. All analyses were performed using R version 3.6.3 [21]. We retrieved historical versions of ClinicalTrials.gov using R package ‘cthist’ (RRID:SCR_019229).

Our study was not subject to Institutional Review Board/Ethics Committee approval, as it relies on publicly accessible data and did not involve interaction with research participants. The study protocol was prospectively registered on Open Science Framework [22]. We listed the deviations from the protocol in S6 File. The code [23] and data sets [22] used in this analysis are available online.

Results

We included 500 interventional SARS-CoV-2 treatment and prevention efficacy trials (see S1 Fig for Flow Diagram). The number of trials was arrived at by chance and was not predetermined. The majority (58.0%) of trials in our cohort were Phase 2 trials; 84.6% were randomized; 84.8% were directed at the treatment of SARS-CoV-2. Study status at 6 months since trial start was “Completed” in 54 of 500 trials (10.8%) and “Recruiting” in 67.0% (Tables 1, S2 and S3). Median anticipated enrollment per trial (based on the enrollment stated in the last registration record prior to trial start) was 180 patient-participants (range 5–15000 patient-participants; interquartile range (IQR) 60–437). Median actual patient-participant enrollment at the 6-month mark, for those trials that provided actual enrollment numbers, was 129 (range 0–4891 patient-participants; IQR 32–320).

Table 1

Characteristics of trial cohort.

Category	Number of Trials (N = 500)	Percent Total (%)	Median (IQR) Anticipated Enrollmenta	Median (IQR) Actual Enrollmentb
Trial Phase
Phase 1/2 & Phase 2	290	58.0	100 (40–200)	60 (25–152)
Phase 2/3 & Phase 3	210	42.0	400 (183–1000)	241 (95–494)
Randomization
Randomized	423	84.6	200 (82–482)	142 (53–357)
Non-Randomized	30	6.0	73 (30–248)	38 (20–102)
NAc	47	9.4	37 (20–100)	27 (10–50)
Trial Statusd
Completed	54	10.8	100 (46–396)	100 (40–387)
Terminated	16	3.2	265 (150–464)	62 (7–127)
Active, Not Recruiting	71	14.2	240 (68–500)	177 (55–442)
Recruiting	335	67.0	152 (60–410)	143 (26–230)
Enrolling by Invitation	11	2.2	128 (56–400)	72 (51–152)
Suspended	13	2.6	308 (200–600)	27 (5–71)
Trial Type
Treatment Trial	424	84.8	130 (60–333)	100 (30–233)
Prevention Trial	66	13.2	672 (206–1729)	554 (75–1346)
Treatment & Prevention	10	2.0	782 (250–1500)	741 (166–1557)
Sponsorship
Industry Sponsor	112	22.4	195 (82–400)	187 (84–413)
Non-Industry Sponsor	388	77.6	177 (60–455)	100 (27–269)
Country Location
USA Trial	179	35.8	200 (60–460)	95 (24–243)
Non-USA Trial	321	64.2	165 (60–426)	121 (39–324)
Number of Centers
Single Center	198	39.6	100 (37–290)	60 (20–213)
Multicenter	302	60.4	226 (100–500)	143 (53–401)

a) Anticipated enrollment in the first registration record after trial start.

b) At the 6-month mark, for the subset of trials which provide actual enrollment information.

c) NA–Information not available in the ClinicalTrials.gov registration record.

d) Trial Status at the 6-month mark since trial start.

Less than one third (29.9%, 95% CI 23.7–36.9%) of the 194 trials eligible for assessment of all 3 criteria were deemed informative. Nineteen trials were classified as potentially redundant (4.1%), of which 10 investigated convalescent plasma and a further 4 investigated hydroxychloroquine. Sixty-three trials (13.6%) differed only by primary outcome. In our post hoc analysis, 81.9% (380 of 464 trials) were similar with respect to trial type, regimen, phase and patient-participant characteristics.

Of the subset of 210 Phase 2/3 and Phase 3 trials in our cohort, 92 (43.8%) met our criteria for trial design quality [20] (Fig 1; Table 2). The proportion of feasible trials in our cohort was 77.4% (387 of 500 trials); 113 trials were non-feasible. Of these, 12 were “Suspended” or “Terminated “for a reason unrelated to efficacy, safety or the progression of science; 20 trials were “Active, not recruiting” or completed but failed to enrol at least 85% of their target patient-participant enrollment (S2 Fig); 81 trials still “Recruiting” had exceeded at least two times the intended recruitment period (S3 Fig).

Fig 1

Flow diagram for trial design quality of Phase 2/3 and Phase 3 SARS-CoV-2 trials.

a) Refers to trial that is either placebo-controlled or has a standard of care comparator arm. b) Refers to a treatment trial with a clinical primary outcome or a prevention trial with either a clinical primary outcome or laboratory-confirmed SARS-CoV-2.

Table 2

Evaluation of design quality of trials meant to inform clinical practice.

Category	Number of Trials (N = 210)	Percent Total (%)
Randomized	200	95.2
Placebo-Controlled	179	85.2
Blindeda	100	47.6
Clinical Primary Outcomeb	203	96.7
Includes at Risk Populationc	208	99.0
Trials Meeting all 5 Criteria	92	43.8

a) Refers to trials that were at a minimum double-blinded.

b) Treatment trials required a primary clinical outcome; prevention trials required either a primary clinical outcome or laboratory-confirmed SARS-CoV-2.

c) We defined an at risk population as a trial including participants aged ≥ 60.

Discussion

Prior studies have examined the COVID-19 trial landscape, evaluating trial design quality [24,25], choice of outcome [26], and presenting descriptive statistics on COVID-19 trials characteristics [2,5,6,8-11]. This is the first study to assess the prevalence of informative COVID-19 clinical trials. In our analysis, 29.9% of early COVID-19 trials registered on ClinicalTrials.gov met our 3 criteria for informativeness. Many (56.2%) did not use rigorous design, based on assessment of randomization, control group, blinding, primary outcome, and inclusion of an at-risk population. Of these, the greatest number (110 of 210 trials, 52.4%) did not demonstrate adequate blinding. Lack of blinding among COVID-19 trials has been highlighted in several recent analyses [2,5,6,9,10] and may reflect the challenges of trial conduct in pandemic circumstances, in which significant research infrastructure and oversight is required to implement and maintain blinding. Yet, deficits in trial design were not uniform. Our stratified results (Table 3) demonstrated that trials with at least one center in the USA, in addition to trials with industry sponsorship, SARS-CoV-2 prevention trials and multicenter trials, demonstrated a greater proportion of well-designed trials than their counterparts.

Table 3

Stratified analysis of redundancy, design, trial feasibility and informativeness by sponsor, country location, trial type, number of trial centers.

Informative Condition	Yes (%)	No (%)	| Difference | (95% CI)
Non-Redundant
Industry Sponsored	99.1	94.9	4.1 (0.6–7.6)
USA Trial	95.8	96.0	0.2 (-3.8–4.2)
Treatment Trial	95.9	95.5	0.4 (-5.4–6.3)
Multicenter Trial	94.7	97.8	3.1 (-0.8–6.9)
Good Design
Industry Sponsored	73.9	35.4	38.5 (22.5–54.6)
USA Trial	72.4	32.9	39.5 (24.6–54.4)
Treatment Trial	39.2	62.9	23.7 (4.4–43.0)
Multicenter Trial	48.7	31.0	17.6 (2.1–33.2)
Feasible
Industry Sponsored	71.4	79.1	7.7 (-2.2–17.6)
USA Trial	69.8	81.6	11.8 (3.4–20.2)
Treatment Trial	78.3	71.2	7.1 (-5.4–19.6)
Multicenter Trial	73.2	83.8	10.7 (3.1–18.2)
Informativea
Industry Sponsored	52.2	23.0	29.2 (11.8–46.6)
USA Trial	40.7	25.7	15.0 (-1.2–31.3)
Treatment Trial	28.4	31.4	3.0 (-15.6–21.7)
Multicenter Trial	30.1	29.2	1.0 (-14.9–16.8)

a) Informative trials are those that meet all 3 informativeness criteria.

Despite elevated SARS-CoV-2 cases, many trials (22.6% (113 of 500 trials)) were unable to adequately and expeditiously complete patient-participant recruitment. This estimate is in keeping with other studies in which close to one third of COVID-19 trials registered on ClinicalTrials.gov or on the World Health Organization International Clinical Trials Registry Platform stopped before attaining 75% accrual [27]. In some cases failure to reach recruitment goals can be explained by decreasing case counts in the setting of rapid suppression of a COVID outbreak. For example, early stoppage of a Remdesivir multicenter randomized controlled trial after recruitment of 237 of 453 patient-participants in Wuhan, China, resulted in an underpowered trial with inconclusive results [28,29]. This has also been seen in other settings, such as in the 2014–2016 Ebola outbreak [30]. However, infeasible recruitment targets, despite high case counts, have also been documented during the COVID-19 pandemic [31]. Trial feasibility may be particularly challenging in the fractured US healthcare setting due to inter-trial competition in patient-participant recruitment, as supported by our stratified analysis in which non-USA trials were significantly more likely to be feasible than USA trials.

Lack of coordination and trial prioritization, resulting in a high level of multiplicity in investigated interventions, is a contributing factor to infeasible patient-participant recruitment. Concern about trial redundancy has been brought up frequently during the COVID-19 pandemic [1,2,4,5]. In our study, only 4.1% of trials were deemed potentially redundant, of which 4 investigated hydroxychloroquine and 10 investigated the efficacy of convalescent plasma. Our categorization of trials as potentially redundant involved matching of trial phase, type of trial (treatment versus prevention), patient-participant characteristics, regimen, comparator and primary outcome. It differs from other assessments of SARS-CoV-2 trial duplication, in which trial intervention has been the main focus of assessment [2]. While a low proportion of potentially redundant trials may be seen as an encouraging result, deeper examination reveals that sixty-three trials (13.6%) assessed for potential redundancy differed only by the choice of primary outcome, with endpoints often demonstrating small deviations from comparator trials, of questionable clinical relevance. For instance, some trials expressed the primary endpoint as a function of time e.g., time to death, whereas in others as a rate e.g., case fatality rate. Our post hoc analysis of trial similarity, which evaluated trial type, regimen, phase and patient-participant characteristics, revealed that 81.9% of trials were similar, reflecting the extent to which early clinical trials during the COVID-19 pandemic pursued comparable study designs.

Replication in research is important to clarify study results. However, lack of research coordination and harmonization of primary outcome endpoints during the COVID-19 pandemic [2,4,32,33] can thwart efforts to clarify net effects through meta-analyses. This is particularly relevant in the setting of multiple small trials of specific interventions, where the probability is elevated that at least one trial produces a positive result by chance alone [2,5]. Prospective meta-analyses (PMA), which encourage harmonization of core outcomes and draw on individual participant data, can help clarify treatment effects and reduce research waste [34]. In this way, individually underpowered studies can help address questions of significant clinical importance. Although successfully employed in other medical settings [35,36], PMAs were unfortunately not widely deployed in the early COVID-19 pandemic.

Concerns regarding research waste predated the pandemic [37-43] but intensified in the setting of this international public health crisis. Our results support arguments for devising coordinated research plans in advance of public health emergencies [44], and evaluating and prioritizing trials at institutional [45,46], state and national levels [47]. The success of multicenter national platform trials, such as RECOVERY, in the United Kingdom, in both recruiting patient-participants (over 45580 have been enrolled as of December 9 2021, https://www.recoverytrial.net) and in generating practice-changing evidence, speaks to the promise of national research prioritization [48]. Additional strategies to improve pandemic preparedness include: i) promotion of individual participant data sharing platforms to capitalize on data generated, even from small trials [49]; ii) prioritization of adaptive master protocol trials investigating promising interventions [44,49]; and, iii) increased research collaboration, in the model of the Coalition for Epidemic Preparedness Innovations (CEPI). In our stratified analysis, industry-sponsored trials were significantly more likely to meet all 3 informativeness criteria than non-industry sponsored trials (Table 3). This suggests that academic researchers require more institutional support, as well as assistance from research consortia and funding bodies to produce informative results.

Limitations

First, we limited our assessment to 3 aspects of trial informativeness–potential redundancy, design quality and feasibility of patient-participant recruitment. Other aspects of informativeness, such as integrity and reporting, were not evaluated in our study, as they cannot be assessed without access to final trial results (430 of 500 trials, 86.0% had not yet completed or terminated at the end of our 6-month follow-up period). A follow-up study evaluating data 24 months after trial launch would enable a comprehensive assessment of trial informativeness, and thus represents an area for future research. Second, we used proxy measures of informativeness, which are imperfect. For example, we adopted strict criteria for potential redundancy, resulting in only 19 trials labelled potentially redundant, many of which differed based on primary outcome alone. Our post hoc analysis resulted in over eighty percent of trials deemed similar, based on assessment of trial type, regimen, phase and patient-participant characteristics. These two results (4.1% and 81.9%) can be viewed as lower and upper bounds for the proportion of redundant trials. Missing from our assessment was an evaluation of the availability and quality (as assessed by GRADE [50]) of pre-existent evidence of intervention efficacy which may render subsequent trials redundant. We also did not assess the extent to which individual participant data were made publicly available (for example, through the Vivli platform [51]), and subsequently incorporated into meta-analyses. Our redundancy evaluation should thus be interpreted with caution and future research will be required to provide a more precise estimate. Our assessment of trial design quality, as guided by the May 2020 FDA guidance document [16], required that all trials be, at a minimum, double-blinded. We acknowledge that this may unfairly penalize the small minority of trials evaluating interventions in which double-blinding is not practicable. In addition, our assessment of the inclusion of at-risk populations was limited only to age. We did not assess whether the study included a population with other risk factors such as comorbidities. However, no trials failed our design criteria based on failure to include an at-risk population. Third, our assessment of the informativeness of COVID-19 trials depends on the accuracy of ClinicalTrials.gov registration records. Fourth, our findings may not be generalizable to all COVID-19 interventional clinical trials. For example, public health behavioural interventions are frequently labelled as “Phase NA” and would therefore not be included in our findings.

Conclusions

The SARS-CoV-2 pandemic was met with a vigorous response from clinical researchers. However, less than one third of early COVID-19 trials registered on ClinicalTrials.gov met our 3 criteria for informativeness. Shortcomings in trial design, recruitment feasibility and redundancy reflect longstanding vulnerabilities in the clinical research enterprise that were magnified by the urgency of a pandemic. Much knowledge has been gained since the first six months of the COVID-19 pandemic, both in terms of effective measures aimed at treatment and prevention of the virus, but also with respect to the conduct of informative clinical research. The task ahead will be for investigators, research institutions, sponsors and regulators alike to take stock of lessons learned and devise solutions to benefit the global research enterprise as we move forward.

STROBE statement—Checklist of items that should be included in reports of cohort studies.

(DOCX)

Click here for additional data file.

Flow diagram of trial inclusion/exclusion.

(DOCX)

Click here for additional data file.

Ratio of actual to estimated number of patients enrolled.

(DOCX)

Click here for additional data file.

Ratio of actual to estimated recruitment length.

(DOCX)

Click here for additional data file.

Inter-rater agreement.

(DOCX)

Click here for additional data file.

Additional characteristics of trial cohort.

(DOCX)

Click here for additional data file.

Range of anticipated and actual enrollment.

(DOCX)

Click here for additional data file.

Trial inclusion and exclusion criteria.

(DOCX)

Click here for additional data file.

ClinicalTrials.gov search criteria.

(DOCX)

Click here for additional data file.

Data downloaded from ClinicalTrials.gov.

(DOCX)

Click here for additional data file.

Additional data points.

(DOCX)

Click here for additional data file.

Assessment of trial redundancy.

(DOCX)

Click here for additional data file.

Protocol deviations.

(DOCX)

Click here for additional data file.

2 Nov 2021

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Dylan A Mordaunt, MB ChB, FRACP, FAIDH

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

“Marcin Waligora reports personal fees from Advisory Bioethics Council, Sanofi outside the submitted work. Other authors have declared that no competing interests exist.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Additional Editor Comments:

Thank you for your submission in a format that is considered for publication in PLoS One. The reviewers have included a number of useful recommendations for revision of the manuscript. With reference to the publication for criteria:

1. The study appears to represent the results of original meta-research. If there have been similar studies before or since, it would be worth commenting on these for completeness.

2. Results do not appear to have been published elsewhere.

3. Experiments, statistics, and other analyses require some work. These are detailed by reviewer 1, 3 and 4.

4. Conclusions are presented in an appropriate fashion and are supported by the data. The reviewers don't comment on the conclusions, however I agree with the comment on expanding on narrative synthesis in the discussion.

5. The article is presented in an intelligible fashion and is written in standard English.

6. The research meets all applicable standards for the ethics of experimentation and research integrity.

7. The article adheres to appropriate reporting guidelines and community standards for data availability. Without exhaustively detailing them here, it would be helpful to have the manuscript follow standardised reporting structures- although there may not be one specific to this study type, ones that relate to systematic reviews of observational studies or observational studies more generally, would be helpful such as PRISMA and AMSTAR-2. Specific features such as whether the protocol was prospectively registered, where it was registered, the detail of how duplicate assessment occurred etc., should be included. In that specific example, it may be worth addressing in the response if the protocol wasn't prospectively addressed, so that it could be taken into account in future meta-research.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: General comment

This seems to me a very important study worthy of publication. The paper underlines how important it is that trials are conducted by appropriately supported centres with experience of conducting trials. However, although the study compares the properties between groups within features, for example, Multicentre v Single Centre, the difference between them should be presented together with the 95% CI of that difference (see below).

Specific comments

1. Page 10, Table 1: Probably more informative if the row Phase 2c is spilt into two rows one for Phase 1 and a second for Phase 2. Also confusing as how in row Phase 3d the Phase 2 components mentioned differ from those in row Phase 2c. There is some confusion here and consequently on Page 10, line 7, where the authors state: “of 210 Phase 2/3 and Phase 3 trials”, it remains unclear as to what this group actually comprises.

2. Page 10, Table 1 think the actual range (minimum and maximum values), rather than IQR of the anticipated recruitment and actual enrolment, would be much more informative. Also, in the written text above on Page 9.

3. Page 12, Table 2 footnote c) “Age of participants �  60; the two trials not including participants �  60 years of age included healthy adults without any additional factors putting them at greater risk for severe SARS-CoV-2 disease”. It was not at all clear to me what is meant by this statement.

4. Page 12, Figure 1: Again, the confusion remains between Phase 2/3 and Phase 3.

5. Page 8, line 5 from bottom. I am not sure that statistical significance tests are required (see below). However, it is better to interpret the actual p-value rather than state “We defined p < 0.05 as statistically significant”. I suggest omit this phrase.

6. Page 14, Table 3: It would be useful to quote the statistical package used for the calculation of the exact CIs.

7. Page 14, Table 3: Too much precision clouds the message. I suggest replacing, for example, 99.11 (95.13 – 99.98) by 99.1 (95.1 – 100) although quoting these CIs is unnecessary. However, what would be useful is to quote their difference 99.11 − 95.36 = 3.75 with its 95%CI 1.02 to 6.47 and the corresponding p-value = 0.0070. I suggest a better format for Table 3 might be:

Type of Trial Yes (%) No (%) Difference 95%CI p-value

Industry Sponsor 99.1 95.4 3.8 1.0 to 6.5 0.068

USA 96.1 96.2 −0.2 −3.7 to +3.3 0.92

Therapeutic 96.2 95.5 0.8 −4.6 to +6.1 0.76

Multicentre 95.0 98.0 −3.0 −6.1 to +0.2 0.092

Technical note

When comparing differences between proportions which involve any values close to 100% (and or 0%) cause technical problems. Thus, there are several approaches to these calculations and these may give differing p-values. The Exact method is one the authors refer to in their Table 3 which seems entirely appropriate. However, my calculations above have used the statistical package Stata to obtain the p-values which differ somewhat from those of the author. As I indicate above, I am not sure it is necessary to calculate the p-values. Interpretation should focus on the magnitude of the differences and their CIs.

Reviewer #2: Interesting, well researched and timely manuscript. Clearly written. Conducive to a follow up paper in 12-18 months (6 months is a relatively short period of time), to see whether a longer snapshot e.g 12 or 24 month timeframe changes the results/conclusions.

Reviewer #3: This article described an analysis on early trials of COVID-19 and their informativeness. I would like to congratulate the authors an important piece of work describing potential shortcomings in trial design, recruitment, and potential redundancy. The protocol for this work has been prospectively registered, and there is a clear list of protocol deviations. The manuscript is well-written and easy to follow. I have a few comments for the authors to consider.

Abstract:

The abstract could be refined to be more informative as a standalone. It would be helpful to include some more specific information on how the three criteria on informativeness were defined, and how the cohort was created (eligibility criteria? Random selection of trials or own trials?).

Methods: Did the authors adhere to a reporting checklist (e.g. STROBE or PRISMA)? It would be good to include this checklist as a supplement.

Eligibility criteria: The inclusion and exclusion criteria should be listed in the manuscript, and not only provided in a supplement. The reasoning behind the choice of eligibility criteria is unclear, and should be elaborated. I wonder if some of the choices impede generalizability of results. For instance, study phase is a criterion that is usually only filled in for drug trials on trial registries, other trials often chose the option ‘not applicable’. I wonder if by restricting this analysis to certain phase trials, information on other trials was lost? In addition, restricting to Phase 1/2-3 & only trials testing for efficacy may exclude non-drug interventions such as public health messaging trials. Why were behavioral interventions, dietary supplement and Chinese medicine trials excluded?

Search string: It is unclear how trials were identified on ClinicalTrials.gov. Were filters used (e.g. COVID-19 or Phase filters?). Or did the authors include all registrations within a time frame?

Trial screening and coding of outcomes: What was the agreement between screeners? How were disagreements resolved? Were informativeness measures also assessed by two screeners (this is implied but not explicit)? How was the agreement?

Informativeness concepts: The authors refer the reader to information on ‘Informativeness articulated elsewere’ (p.6), to understand the assessed concepts. Since this is a core construct that is required to be understood to understand this paper, I would recommend introducing these concepts and what they mean in detail in the introduction.

Redundancy: I have some reservations about the assessment of this concept. Replication in research is crucial, and often trials (and particularly early trials) do not have sufficient sample size to conclusively answer a research question.

A trial is only redundant, if high certainty evidence exists that an intervention is effective or not effective (as evaluated by GRADE). This does not seem to have been assessed in this case. If certainty of evidence is low, additional replication trials are crucial to ensure early findings were not purely contextual or chance findings (and thus, they are not redundant in this case). For this reason, I would interpret this criterion very carefully.

A slightly different primary outcome does not necessarily make a trial non-redundant. In fact, as the authors point out in the discussion, it may be better if two trials collect the same outcomes so they can be combined in meta-analysis. The analysis looking at the numbers of trials labelled as redundant when disregarding the primary outcome is important, it may be worthwhile presenting this analysis more prominently.

Design quality:

Trial design was only analysed for Phase 2/3 and Phase 3 trials – but trial design is also important for earlier phase trials (albeit criteria may be different)?

‘We considered a trial to be well-designed if it was randomized, placebo-controlled (with appropriate standard of care in all arms), double-blinded and included participants aged 60 years or over (as a proxy for an at-risk population)’ What if a trial had an active control? Would that not be considered well-designed?

It would have been good to look at each trial design criterion separately in each trial (and not just the ones that satisfied previous requirements), to get an assessment of how well each design feature was fulfilled in those trials.

Feasibility of Patient-Participant Recruitment:

How would a trial that stopped early for effectiveness be assessed here? Also, from our experience of managing a registry, many registrants do not update their registration records even if they have finished recruiting, thus, a trial may have long finished recruitment and still be listed as ‘recruiting’. Do the authors have information on how many of the trials have updated their records?

Table 1: Characteristics of trial cohort. If possible, it would be great to include some additional information on the trials, such as target sample sizes and included populations.

Discussion: I would be interested in a more in-depth discussion of what needs to change on a structural level in future to improve trial informativeness, particularly in the context of health emergencies.

Reviewer #4: 1. Was the sample of exactly 500 arrived at purely by chance? If so, please make it clear that this was not a predetermined number.

2. It would be valuable to include a checklist of items according to the STROBE guidelines https://www.strobe-statement.org/checklists/and STROBE-checklist-v4-combined-PlosMedicine.pdf, with corresponding page numbers to indicate where each item is addressed,

3. I have a big problem with the definition of redundancy as the presence of another trial of the same phase, type of trial (SARS-CoV-2 prevention versus treatment), patient-participant characteristics (including location of care, disease severity and age of trial participants), regimen (including interventions used in combination in a single arm), comparator arm(s) and primary outcome (evaluating primary outcome domain and specific measurement, based on framework from ref 13. This excludes the highly desirable situation when multiple investigators who have obtained funding from a funding agency for a single smaller trial agree to undertake a prospective meta-analysis of individual participant data, as in the NeOProM Collaboration of RCTs of oxygen targeting in preterm newborns (Askie et al JAMA 2018) and (Askie et al Pediatric Obesity 2020 https://onlinelibrary.wiley.com/doi/abs/10.1111/ijpo.12618 and other next-generation syntheses of similar trials to enhance power (see Seidler et al Guide to Prospective Meta-Analysis, BMJ 2019).

4. In view of 3, it is essential in the Discussion to acknowledge that (i) even individually underpowered trials can make a valuable contribution in addressing critically important questions regarding mortality if included in individual participant data meta-analyses and(ii) inability to assess how often this was happening is a major limitation of this study.

I would recommend the manuscript be substantially revised and resubmitted.

Thank you for the opportunity to review this important work.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: David Machin

Reviewer #2: No

Reviewer #3: Yes: Anna Lene Seidler

Reviewer #4: Yes: William Odita Tarnow-Mordi

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 Dec 2021

Dear Editors and Referees:

We thank the referees and editors for their careful and constructive assessment of our manuscript. We are grateful for the opportunity to revise and resubmit our manuscript. Responses to all comments are appended below.

Editors’ Comments:

1. The study appears to represent the results of original meta-research. If there have been similar studies before or since, it would be worth commenting on these for completeness.

Yes, this manuscript does represent original meta-research. We have added the following sentence to page 14, lines 247-249, with additional citations of recently published meta-research, not included in our original submission: “Prior studies have examined the COVID-19 trial landscape, evaluating trial design quality, choice of outcome, and presenting descriptive statistics on COVID-19 trials characteristics.”

2. Results do not appear to have been published elsewhere.

Our manuscript is available on the medRxiv preprint server. Results have not been published elsewhere.

3. Experiments, statistics, and other analyses require some work. These are detailed by reviewer 1, 3 and 4.

Thank you. We have altered our analyses, as detailed below.

4. Conclusions are presented in an appropriate fashion and are supported by the data. The reviewers don't comment on the conclusions, however I agree with the comment on expanding on narrative synthesis in the discussion.

We have expanded on the discussion, addressing the following points:

-We call attention to our post hoc assessment of trial similarity (as recommended by Reviewer # 3): page 17, lines 302-305: “Our post hoc analysis of trial similarity, which evaluated trial type, regimen, phase and patient-participant characteristics, revealed that 81.9% of trials were similar, reflecting the extent to which early clinical trials during the COVID-19 pandemic pursued comparable study designs.”

-We discuss prospective meta-analyses (as recommended by Reviewer # 4): pages 17-18, Lines 312-317: “Prospective meta-analyses (PMA), which encourage harmonization of core outcomes and draw on individual participant data, can help clarify treatment effects and reduce research waste. In this way, individually underpowered studies can help address questions of significant clinical importance. Although successfully employed in other medical settings, PMAs were unfortunately not widely deployed in the early COVID-19 pandemic.

-We suggest additional strategies to improve pandemic preparedness (as recommended by Reviewer # 3): page 18, lines 326-331): “Additional strategies to improve pandemic preparedness include: i) promotion of individual participant data sharing platforms to capitalize on data generated, even from small trials;43 ii) prioritization of adaptive master protocol trials investigating promising interventions;38,43 and, iii) increased research collaboration, in the model of the Coalition for Epidemic Preparedness Innovations (CEPI).”

5. The article is presented in an intelligible fashion and is written in standard English.

Thank you.

6. The research meets all applicable standards for the ethics of experimentation and research integrity.

Thank you.

7. The article adheres to appropriate reporting guidelines and community standards for data availability. Without exhaustively detailing them here, it would be helpful to have the manuscript follow standardised reporting structures- although there may not be one specific to this study type, ones that relate to systematic reviews of observational studies or observational studies more generally, would be helpful such as PRISMA and AMSTAR-2. Specific features such as whether the protocol was prospectively registered, where it was registered, the detail of how duplicate assessment occurred etc., should be included. In that specific example, it may be worth addressing in the response if the protocol wasn't prospectively addressed, so that it could be taken into account in future meta-research.

Our study protocol was prospectively registered with the Open Science Framework, as indicated in the Abstract on Page 2, Lines 38-40: “The study protocol was prospectively registered with the Open Science Framework (https://osf.io/fp726/)” and in the Methods section on Page 9, lines 191-192.

We have added a STROBE checklist for cohort studies, as indicated on page 9, lines 178-180: “We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies.” The checklist is in the supplementary appendix (S1 Checklist).

Reviewer # 1 Comments:

This seems to me a very important study worthy of publication. The paper underlines how important it is that trials are conducted by appropriately supported centres with experience of conducting trials. However, although the study compares the properties between groups within features, for example, Multicentre v Single Centre, the difference between them should be presented together with the 95% CI of that difference (see below).

Thank you. We have made the suggested changes to our analysis, as described under question # 7 below.

1. Page 10, Table 1: Probably more informative if the row Phase 2c is spilt into two rows one for Phase 1 and a second for Phase 2. Also confusing as how in row Phase 3d the Phase 2 components mentioned differ from those in row Phase 2c. There is some confusion here and consequently on Page 10, line 7, where the authors state: “of 210 Phase 2/3 and Phase 3 trials”, it remains unclear as to what this group actually comprises.

We included the following categories of trials in our cohort: Phase 1/2, Phase 2, Phase 2/3 and Phase 3 trials. We did not include any Phase 1 trials in our cohort.

We classified trials that encompass two phases, such as so-called seamless trials: Phase 1/2 trials and Phase 2/3 trials based on the higher phase. Therefore, Phase 1/2 trials were classified with Phase 2 trials and Phase 2/3 trials were classified with Phase 3 trials.

We have updated Table 1 for clarity in the following way: Column 1, Row 2 now reads: “Phase 1/2 & Phase 2”; Column 1, Row 3 now reads: “Phase 2/3 & Phase 3.”

We did not update the following sentence: “Of the subset of 210 Phase 2/3 and Phase 3 trials…” as Phase 2/3 trials represent a specific type of trial ("seamless trial”) that encompasses both Phase 2 and Phase 3 within a single trial.

2. Page 10, Table 1 think the actual range (minimum and maximum values), rather than IQR of the anticipated recruitment and actual enrolment, would be much more informative. Also, in the written text above on Page 9.

Our goal in presenting the interquartile range was to provide an estimate of the variance. Therefore, we have kept the IQR in Table 2, but have added the range in S3 Table. We also added the range for the median anticipated enrollment per trial and actual enrollment per trial on page 10, in lines 203 and 205 respectively.

3. Page 12, Table 2 footnote c) “Age of participants �  60; the two trials not including participants �  60 years of age included healthy adults without any additional factors putting them at greater risk for severe SARS-CoV-2 disease”. It was not at all clear to me what is meant by this statement.

We agree that this sentence was confusing and have removed it from the manuscript.

4. Page 12, Figure 1: Again, the confusion remains between Phase 2/3 and Phase 3.

As described above, Phase 2/3 and Phase 3 trials both represent distinct types of trials, the former combining Phase 2 and Phase 3 within a single trial. We believe that this is accepted terminology and therefore have not changed the title of Figure 1.

5. Page 8, line 5 from bottom. I am not sure that statistical significance tests are required (see below). However, it is better to interpret the actual p-value rather than state “We defined p < 0.05 as statistically significant”. I suggest omit this phrase.

We have removed this sentence from the manuscript.

6. Page 14, Table 3: It would be useful to quote the statistical package used for the calculation of the exact CIs.

We have modified the following sentence on page 9, lines 177-178: “Ninety-five percent confidence intervals were calculated for the difference between two proportions using the prop.test package in R.”

7. Page 14, Table 3: Too much precision clouds the message. I suggest replacing, for example, 99.11 (95.13 – 99.98) by 99.1 (95.1 – 100) although quoting these CIs is unnecessary. However, what would be useful is to quote their difference 99.11 − 95.36 = 3.75 with its 95%CI 1.02 to 6.47 and the corresponding p-value = 0.0070. I suggest a better format for Table 3 might be:

Type of Trial Yes (%) No (%) Difference 95%CI p-value

Industry Sponsor 99.1 95.4 3.8 1.0 to 6.5 0.068

USA 96.1 96.2 −0.2 −3.7 to +3.3 0.92

Therapeutic 96.2 95.5 0.8 −4.6 to +6.1 0.76

Multicentre 95.0 98.0 −3.0 −6.1 to +0.2 0.092

Technical note

When comparing differences between proportions which involve any values close to 100% (and or 0%) cause technical problems. Thus, there are several approaches to these calculations and these may give differing p-values. The Exact method is one the authors refer to in their Table 3 which seems entirely appropriate. However, my calculations above have used the statistical package Stata to obtain the p-values which differ somewhat from those of the author. As I indicate above, I am not sure it is necessary to calculate the p-values. Interpretation should focus on the magnitude of the differences and their CIs.

Thank you for this helpful suggestion! We have modified our stratified analysis, as presented in Table 3. The format of the columns is as you proposed. We also now present the difference between proportions with a 95% CI for that difference. We removed p-values from the Table.

Reviewer # 2 Comments:

Interesting, well researched and timely manuscript. Clearly written. Conducive to a follow up paper in 12-18 months (6 months is a relatively short period of time), to see whether a longer snapshot e.g 12 or 24 month timeframe changes the results/conclusions.

Thank you! We agree that a follow-up study would be very interesting. We have added the following sentence in the limitations section of our manuscript to reflect this (page 19, lines 343-345): “A follow-up study evaluating data 24 months after trial launch would enable a comprehensive assessment of trial informativeness, and thus represents an area for future research.”

Reviewer # 3 Comments:

This article described an analysis on early trials of COVID-19 and their informativeness. I would like to congratulate the authors an important piece of work describing potential shortcomings in trial design, recruitment, and potential redundancy. The protocol for this work has been prospectively registered, and there is a clear list of protocol deviations. The manuscript is well-written and easy to follow. I have a few comments for the authors to consider.

1. Abstract:

The abstract could be refined to be more informative as a standalone. It would be helpful to include some more specific information on how the three criteria on informativeness were defined, and how the cohort was created (eligibility criteria? Random selection of trials or own trials?).

Thank you for this suggestion. We have updated the abstract, providing more information about our eligibility criteria: (page 2, lines 29-33): “Based on prespecified eligibility criteria, we created a cohort of Phase 1/2, Phase 2, Phase 2/3 and Phase 3 SARS-CoV-2 treatment and prevention efficacy trials that were initiated from 2020-01-01 to 2020-06-30 using ClinicalTrials.gov registration records. We excluded trials evaluating behavioural interventions and natural products, which are not regulated by the U.S. Food and Drug Administration (FDA).” We also specified that all eligible trials were included in our cohort (page 2, line 42): “We included all 500 eligible trials in our cohort…”

A more detailed description of the 3 criteria for informativeness was also added to the Methods section of the abstract (page 2, lines 33-38): “We evaluated trials on 3 criteria of informativeness: potential redundancy (comparing trial phase, type, patient-participant characteristics, treatment regimen, comparator arms and primary outcome), trials design (according to the recommendations set-out in the May 2020 FDA guidance document on SARS-CoV-2 treatment and prevention trials) and feasibility of patient-participant recruitment (based on timeliness and success of recruitment).”

2. Methods: Did the authors adhere to a reporting checklist (e.g. STROBE or PRISMA)? It would be good to include this checklist as a supplement.

We have followed the STROBE checklist for cohort studies. We have added the following sentence to the Methods section (page 9, lines 178-180): “We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies.” The STROBE checklist is now provided in the supplemental appendix (S1 Checklist).

3. Eligibility criteria: The inclusion and exclusion criteria should be listed in the manuscript, and not only provided in a supplement. The reasoning behind the choice of eligibility criteria is unclear, and should be elaborated. I wonder if some of the choices impede generalizability of results. For instance, study phase is a criterion that is usually only filled in for drug trials on trial registries, other trials often chose the option ‘not applicable’. I wonder if by restricting this analysis to certain phase trials, information on other trials was lost? In addition, restricting to Phase 1/2-3 & only trials testing for efficacy may exclude non-drug interventions such as public health messaging trials. Why were behavioral interventions, dietary supplement and Chinese medicine trials excluded?

As suggested, we expanded the description of eligibility criteria in our manuscript instead of presenting them only in supporting information (page 5, lines 89-97): “Our cohort consisted of interventional SARS-CoV-2 treatment and prevention trials registered on ClinicalTrials.gov with a start date between 2020-01-01 and 2020-06-30. We included “Completed”, “Terminated”, “Suspended”, “Active, not recruiting”, “Enrolling by invitation” and “Recruiting” Phase 1/2, Phase 2, Phase 2/3 and Phase 3 interventional clinical trials testing an efficacy hypothesis in their primary outcome. We included trials evaluating any of the following interventions: drug, biological, surgical, radiotherapy, procedural or device. We excluded trials evaluating behavioural interventions, trials of natural products and Phase 1 trials, all of which have no legal requirement to register on ClinicalTrials.gov.”

When defining the inclusion and exclusion criteria, we wished to focus primarily on clinical trials of interventions required by law to register on ClinicalTrials.gov (based on 42 CFR Part 11). Behavioural interventions, trials of natural products (which includes the majority of traditional Chinese medicine products, which are often herbal products) and Phase 1 trials have no legal requirement to register on ClinicalTrials.gov, and this informed our decision to exclude them.

We agree that our cohort therefore does not reflect the full breadth of possible COVID-19 treatment and prevention trials. We have thus added the following to the limitations section (page 20, lines 366-368): “Fourth, our findings may not be generalizable to all COVID-19 interventional clinical trials. For example, public health behavioural interventions are frequently labelled as “Phase NA” and would therefore not be included in our findings.”

4. Search string: It is unclear how trials were identified on ClinicalTrials.gov. Were filters used (e.g. COVID-19 or Phase filters?). Or did the authors include all registrations within a time frame?

We have now added our specific search criteria in supplemental S2 File:

“We downloaded clinical trial data directly as a zipped folder of XML files from the web front-end of ClinicalTrials.gov.

We used the following search criteria:

411 records identified through 12/01/2021:

Condition or disease: “Covid-19”

Study Type: “Interventional Studies”

Trial Status: “Recruiting, “Active, not recruiting,” “Completed,” “Enrolling by invitation,” “Suspended,” “Terminated”

Phase: Phase 2, Phase 3

Start Date: 01/01/2020 to 05/31/2020

110 records identified through 01/04/2021:

Condition or disease: “Covid-19”

Study Type: “Interventional Studies”

Trial Status: “Recruiting, “Active, not recruiting,” “Completed,” “Enrolling by invitation,” “Suspended,” “Terminated”

Phase: Phase 2, Phase 3

Start Date: 06/01/2020 to 06/30/2020”

5. Trial screening and coding of outcomes: What was the agreement between screeners? How were disagreements resolved? Were informativeness measures also assessed by two screeners (this is implied but not explicit)? How was the agreement?

We have added S1 Table which provides the inter-rater agreement for all items that required human curation. Coding was independently performed by two individuals and when necessary a third person, an arbiter, was involved. This is now more precisely stated on page 5, lines 98-100: “Trial inclusion and exclusion criteria were independently assessed by two researchers (KK & LZ), with disagreements resolved by an arbiter (NH or MW),” page 6, lines 109-110 and 116-117: “Additional items requiring human curation were independently assessed and coded by two researchers (KK & LZ) … Disagreements were resolved by an arbiter (NH or MW)” and page 7, lines 137-139: “The assessment was independently performed by two raters (NH & KK), with disagreements resolved by an arbiter (MW of BC).”

Informativeness criteria were assessed as follows. First, the assessment of potential redundancy combined an automated assessment of trial phase and several human curated data points (type of trial, patient-participant characteristics, regimen, comparator arms, and primary outcome). The latter elements were assessed by two individuals (KK and LW) with disagreements resolved by a third (NH or MW). The final assessment of potentially redundant trials was independently performed by two assessors (NH and KK), with disagreements resolved by a third (MW or BC). This is further described in supplemental S5 File. Second, assessment of design quality also combined an automated assessment of trial phase, randomization, blinding and age of participants, with human curated data points (presence of placebo or standard of care arm and primary outcome). The latter two elements were assessed by two individuals (KK and LW) with disagreements resolved by a third (NH or MW). Finally, assessment of the feasibility of patient-participant recruitment was automated.

6. Informativeness concepts: The authors refer the reader to information on ‘Informativeness articulated elsewhere’ (p.6), to understand the assessed concepts. Since this is a core construct that is required to be understood to understand this paper, I would recommend introducing these concepts and what they mean in detail in the introduction.

Thank you for this suggestion. We have now elaborated on the criteria for informativeness in the introduction (page 4, lines 68-73): “For a trial to be informative to clinical practice, it must fulfill five conditions. First, it must ask a clinically important question. Second, it must be designed to provide a clear answer to that question. Third, it must have both a feasible enrollment target and primary completion timeline. Fourth, it must be analyzed in a manner that supports statistically valid inference. Fifth, it must report results in a complete and timely manner.”

7. Redundancy: I have some reservations about the assessment of this concept. Replication in research is crucial, and often trials (and particularly early trials) do not have sufficient sample size to conclusively answer a research question.

A trial is only redundant, if high certainty evidence exists that an intervention is effective or not effective (as evaluated by GRADE). This does not seem to have been assessed in this case. If certainty of evidence is low, additional replication trials are crucial to ensure early findings were not purely contextual or chance findings (and thus, they are not redundant in this case). For this reason, I would interpret this criterion very carefully.

We agree that our assessment of redundancy is imperfect. We have elaborated on its shortcomings in the limitations section (page 19, lines 351-357): “Missing from our assessment was an evaluation of the availability and quality (as assessed by GRADE) of pre-existent evidence of intervention efficacy which may render subsequent trials redundant. We also did not assess the extent to which individual participant data were made publicly available (for example, through the Vivli platform), and subsequently incorporated into meta-analyses. Our redundancy evaluation should thus be interpreted with caution and future research will be required to provide a more precise estimate.”

A slightly different primary outcome does not necessarily make a trial non-redundant. In fact, as the authors point out in the discussion, it may be better if two trials collect the same outcomes so they can be combined in meta-analysis. The analysis looking at the numbers of trials labelled as redundant when disregarding the primary outcome is important, it may be worthwhile presenting this analysis more prominently.

We have added the following sentence to the discussion to highlight our assessment of trial similarity (page 17, lines 302-305): “Our post hoc analysis of trial similarity, which evaluated trial type, regimen, phase and patient-participant characteristics, revealed that 81.9% of trials were similar, reflecting the extent to which early clinical trials during the COVID-19 pandemic pursued comparable study designs.”

8. Design quality:

Trial design was only analysed for Phase 2/3 and Phase 3 trials – but trial design is also important for earlier phase trials (albeit criteria may be different)?

Yes, we agree that trial design is equally important for earlier phase trials. We chose to confine our assessment of trial design to Phase 2/3 and Phase 3 trials given that we based our assessment of design on the May 2020 FDA guidance document, which focused primarily on later phase trials.

9. ‘We considered a trial to be well-designed if it was randomized, placebo-controlled (with appropriate standard of care in all arms), double-blinded and included participants aged 60 years or over (as a proxy for an at-risk population)’ What if a trial had an active control? Would that not be considered well-designed?

A trial with an active control arm, or standard of care arm, was also accepted as well designed. The following has been updated for clarify (page 8, lines 148-150): “we considered a trial to be well-designed if it was randomized, placebo-controlled or with a standard of care comparator arm…”

10. It would have been good to look at each trial design criterion separately in each trial (and not just the ones that satisfied previous requirements), to get an assessment of how well each design feature was fulfilled in those trials.

This data is presented in Table 2 (page 12).

11. Feasibility of Patient-Participant Recruitment:

How would a trial that stopped early for effectiveness be assessed here? Also, from our experience of managing a registry, many registrants do not update their registration records even if they have finished recruiting, thus, a trial may have long finished recruitment and still be listed as ‘recruiting’. Do the authors have information on how many of the trials have updated their records?

If a trial was terminated early due to efficacy, this trial was not labelled as infeasible (page 8, lines 158-160): “A single trial was considered non-feasible if it met any of the following criteria: i) trial status was “terminated” or “suspended” and reason for stopping contained a rationale unrelated to trial efficacy, safety or the progression of science…”

Yes, we agree that our analysis was highly dependent on the accuracy of trial records on Clinicaltrials.gov, which may not reflect actual trial status. We have highlighted our reliance on the accuracy of Clinicaltrials.gov records in our limitations section (page 20, lines 364-366): “Third, our assessment of the informativeness of COVID-19 trials depends on the accuracy of ClinicalTrials.gov registration records.”

We evaluated all trials in our cohort at the 6-month mark, to provide equal follow-up time for each clinical trial. A follow-up study, evaluating trial data 24 months after trial launch would be beneficial, and represents an area of future research (page 19, lines 343-345): “A follow-up study evaluating data 24 months after trial launch would enable a comprehensive assessment of trial informativeness, and thus represents an area for future research.”

12. Table 1: Characteristics of trial cohort. If possible, it would be great to include some additional information on the trials, such as target sample sizes and included populations.

We have added S2 Table to our supplement, providing additional information about the included trials including age of participants, location of care and SARS-CoV-2 severity.

13. Discussion: I would be interested in a more in-depth discussion of what needs to change on a structural level in future to improve trial informativeness, particularly in the context of health emergencies.

Thank you for this suggestion. We have added the following to the discussion (page 18, lines 326-331): “Additional strategies to improve pandemic preparedness include: i) promotion of individual participant data sharing platforms to capitalize on data generated, even from small trials;45 ii) prioritization of adaptive master protocol trials investigating promising interventions;40,45 and, iii) increased research collaboration, in the model of the Coalition for Epidemic Preparedness Innovations (CEPI).”

Reviewer # 4 Comments:

1. Was the sample of exactly 500 arrived at purely by chance? If so, please make it clear that this was not a predetermined number.

Yes, the sample size of 500 trials was arrived at by chance. We have updated our abstract to make clear that we included all eligible trials in our cohort (page 2, line 42): “We included all 500 eligible trials in our cohort…” We have also added the following sentence in our results (page 10, lines 197-198): “The number of trials was arrived at by chance and was not predetermined.”

2. It would be valuable to include a checklist of items according to the STROBE guidelines https://www.strobe-statement.org/checklists/and STROBE-checklist-v4-combined-PlosMedicine.pdf, with corresponding page numbers to indicate where each item is addressed.

We have added a STROBE checklist for cohort studies, as indicated on page 9, lines 178-180: “We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies.” The checklist is in the supplementary appendix (S1 Checklist).

3. I have a big problem with the definition of redundancy as the presence of another trial of the same phase, type of trial (SARS-CoV-2 prevention versus treatment), patient-participant characteristics (including location of care, disease severity and age of trial participants), regimen (including interventions used in combination in a single arm), comparator arm(s) and primary outcome (evaluating primary outcome domain and specific measurement, based on framework from ref 13. This excludes the highly desirable situation when multiple investigators who have obtained funding from a funding agency for a single smaller trial agree to undertake a prospective meta-analysis of individual participant data, as in the NeOProM Collaboration of RCTs of oxygen targeting in preterm newborns (Askie et al JAMA 2018) and (Askie et al Pediatric Obesity 2020 https://onlinelibrary.wiley.com/doi/abs/10.1111/ijpo.12618 and other next-generation syntheses of similar trials to enhance power (see Seidler et al Guide to Prospective Meta-Analysis, BMJ 2019).

Thank you for this comment and suggestion. We agree that our assessment of redundancy is imperfect. We have added the following discussion of prospective meta-analyses to our discussion (pages 17-18, lines 312-317): “Prospective meta-analyses (PMA), which encourage harmonization of core outcomes and draw on individual participant data, can help clarify treatment effects and reduce research waste. In this way, individually underpowered studies can help address questions of significant clinical importance. Although successfully employed in other medical settings, PMAs were unfortunately not widely deployed in the early COVID-19 pandemic.”

4. In view of 3, it is essential in the Discussion to acknowledge that (i) even individually underpowered trials can make a valuable contribution in addressing critically important questions regarding mortality if included in individual participant data meta-analyses and(ii) inability to assess how often this was happening is a major limitation of this study.

As above, in the discussion we have highlighted how (pages 17-18, lines 314-315): “individually underpowered studies can help address questions of significant clinical importance.” We have also added the following to the limitations section (page 19, lines 353-355): “We also did not assess the extent to which individual participant data were made publicly available (for example, through the Vivli platform), and subsequently incorporated into meta-analyses.”

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

20 Dec 2021

How Informative Were Early SARS-CoV-2 Treatment and Prevention Trials? A longitudinal cohort analysis of trials registered on clinicaltrials.gov

PONE-D-21-28808R1

Dear Dr. Waligora,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Dylan A Mordaunt, MB ChB, FRACP, FAIDH

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for your resubmission. This meets the criteria for publication.

Reviewers' comments:

22 Dec 2021

PONE-D-21-28808R1

How Informative Were Early SARS-CoV-2 Treatment and Prevention Trials? A longitudinal cohort analysis of trials registered on clinicaltrials.gov

Dear Dr. Waligora:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Dylan A Mordaunt

Academic Editor

PLOS ONE