Data and Safety Monitoring of COVID-19 Vaccine Clinical Trials.

To speed the development of vaccines against SARS-CoV-2, the United States Federal Government has funded multiple phase 3 trials of candidate vaccines. A single 11-member data and safety monitoring board (DSMB) monitors all government-funded trials to ensure coordinated oversight, promote harmonized designs, and allow shared insights related to safety across trials. DSMB reviews encompass 3 domains: (1) the conduct of trials, including overall and subgroup accrual and data quality and completeness; (2) safety, including individual events of concern and comparisons by randomized group; and (3) interim analyses of efficacy when event-driven milestones are met. Challenges have included the scale and pace of the trials, the frequency of safety events related to the combined enrollment of over 100 000 participants, many of whom are older adults or have comorbid conditions that place them at independent risk of serious health events, and the politicized environment in which the trials have taken place. Published by Oxford University Press for the Infectious Diseases Society of America 2021.

In May 2020, the United States Federal Government launched Operation Warp Speed (OWS), an ambitious plan to “accelerate the development, manufacturing, and distribution of COVID-19 vaccines, therapeutics, and diagnostics” [1]. Funded by almost $10 billion in Congressional appropriations, OWS is a partnership among the Department of Health and Human Services (including the Centers for Disease Control and Prevention [CDC], the National Institutes of Health, and the Biomedical Advanced Research and Development Authority [BARDA]), the Department of Defense, and the private sector [2]. To accelerate vaccine development, OWS funded multiple large, randomized trials to assess the safety and efficacy of several candidate vaccines based on diverse technologies. OWS also agreed to purchase hundreds of millions of doses to assure timely manufacture of ample quantities of vaccine. Finally, OWS has committed to financing the rapid, equitable, and comprehensive distribution and delivery of vaccines within the United States that are shown to be safe and effective [3].

To ensure rigorous, independent, and unbiased scientific and ethical oversight of the vaccine field trials that it is funding, the National Institute of Allergy and Infectious Diseases (NIAID) empaneled a single independent data and safety monitoring board (DSMB, alternately called a data monitoring committee or DMC) in June 2020, with members invited by the Institute Director [4]. Single DSMBs have been used to oversee multiple clinical trials within networks such as the AIDS Clinical Trials Group, the human immunodeficiency virus (HIV) Prevention Trials Network, and the HIV Vaccine Trials Network [5-8]. However, their use in the OWS COVID-19 clinical trial program, as well as in the Adaptive COVID-19 Treatment Trial (ACTT), the Randomised Evaluation of COVID-19 Therapy (RECOVERY) program, and the World Health Organization’s Solidarity Trial, to oversee multiple trials of products targeting the same outcome in the setting of a global pandemic is precedent setting. Institute leadership recognized that, with independent but parallel trials aiming to test multiple vaccines against a common virus, oversight by a single DSMB would facilitate informed judgments such as those about interim analyses or about the relatedness of adverse events and would ensure that reviews of individual trials would benefit from insights gained from the complete trial portfolio. The structure of the DSMB, which is formally known as the Coronavirus Disease 2019 (COVID-19) Vaccine Data and Safety Monitoring Board, and its operating processes have not previously been described. Furthermore, although DSMBs usually operate in obscurity, the COVID-19 Vaccine DSMB was recently the subject of considerable public attention related to disagreements about results of an interim efficacy analysis [9]. We write as DSMB members and staff to inform the medical, scientific, public health, and policy communities, and the general public, about the workings of this Board.

PURPOSE AND STRUCTURE OF THE COVID-19 VACCINE DATA AND SAFETY MONITORING DSMB

The purpose of the DSMB is to ensure the safety of study participants and the rigor and integrity of the clinical trials that it monitors. It consists of 11 members from the United States, Brazil, South Africa, and the United Kingdom, including experts in infectious disease, vaccinology, immunology, biostatistics, pharmacoepidemiology, public health, and bioethics. Members, who may receive an honorarium of $200 per meeting from NIAID, are free of financial relationships with companies developing vaccines against COVID-19, including but not limited to those working in partnership with OWS. A biostatistician, who is a full-time NIAID employee, serves as Executive Secretary to the DSMB. The Executive Secretary sets agendas for meetings, drafts meeting reports, and facilitates communication with vaccine manufacturers and with a 3-person Oversight Group. Each clinical trial is managed by its own Oversight Group, which includes representation from BARDA, NIAID, and the vaccine company conducting that trial. Formally, the DSMB is advisory to the Oversight Group for each trial.

The DSMB operates under a single charter, common to all the trials that it oversees, that governs its structure, reporting relationships, and operations (Supplementary Material). The Executive Secretary drafted the charter in accordance with the NIAID’s policy on DSMBs [10]. The final version incorporates comments from members of the Accelerating COVID-19 Therapeutic Interventions and Vaccines (ACTIV) team [11] and from the DSMB members. Members agree in writing to follow the principles of the charter and to maintain confidentiality of all meeting discussions and materials. The charter can be amended for clarity as needed, subject to the approval of the DSMB members and the NIAID; to date, it has been amended 3 times, once to move the members’ names into a separate document, once to harmonize the description of the oversight group with that in the cooperative research and development agreements with company sponsors, and once to allow the DSMB chair, with company permission, to discuss safety issues with the Food and Drug Administration.

STUDY REVIEW PROCESS

Since the trials it is monitoring began, the DSMB has met by videoconference over 25 times, generally reviewing 1 trial per meeting. Scheduled meetings typically last 2–3 hours. When necessary, the DSMB holds ad hoc meetings to address emerging safety concerns and, if accrual or event milestones are met between scheduled meetings, to review interim analyses. Ad hoc meetings may be convened on short notice, including weekends, to ensure rapid reviews and to minimize delays in trial progress. A 7-member quorum is required to meet.

At initial meetings for each trial, the DSMB reviews study protocols, overall and subgroup accrual goals, choice of end points, and statistical designs including interim analysis plans related to futility, safety, and efficacy. Once trials begin, meetings focus on accrual (including of important subgroups), data quality and completeness, and safety. To date, the DSMB has reviewed 3 formal interim efficacy analyses, of trials from Moderna, Janssen, and AstraZeneca [12-14]. The DSMB is currently monitoring the Moderna, Janssen, AstraZeneca, Novavax, and Sanofi/GSK trials. The trial of the vaccine developed jointly by Pfizer and BioNTech, which is not funded by OWS, has a separate DSMB.

Prior to each meeting, members receive study reports via a secure website. Two members—1 clinician and 1 statistician—serve as primary reviewers for each trial, but all members are expected to review reports in advance of the meeting. Meetings begin in executive session, attended only by DSMB members and the Executive Secretary, during which members discuss issues noted on prereview. The DSMB then moves into open session, joined by representatives of the sponsoring company, NIAID, BARDA, and COVID-19 Prevention Trials Network (CoVPN; an NIAID-supported network to facilitate recruitment of diverse populations to COVID-19 vaccine trials) [15], and other study team members. During the open session, company representatives present aggregate data on study progress, accrual, data quality, and any anticipated changes to study conduct, along with updates from other trials they may be conducting. Safety monitors, who remain blinded to the groups to which participants are assigned and may include representatives of the sponsoring company, also describe any serious adverse events or other safety data of concern identified since the prior review.

The DSMB then moves into closed session, which includes members, the Executive Secretary, and representatives from a Statistical Support Group that has a contractual relationship with but is otherwise independent of the sponsoring company. The unblinded Statistical Support Group statisticians present demographic, data quality, safety, and efficacy data to the DSMB by randomized group. Following this presentation, the DSMB moves back into executive session, during which members agree on recommendations. Finally, the study-specific Oversight Group rejoins the meeting to receive the DSMB’s conclusions and recommendations.

Following each meeting, the Executive Secretary prepares a draft memorandum outlining the DSMB’s conclusions and recommendations. Members offer comments and edits, after which the memorandum is finalized and sent to the trial’s Oversight Group.

Figure 1 summarizes the entities with which the DSMB interacts and the flow of data and recommendations among them.

Figure 1.

Structure and process of the Coronavirus Disease 2019 (COVID-19) Vaccine Data and Safety Monitoring Board (DSMB). The single DSMB reviews multiple protocols from multiple sponsors, each with a separate protocol team and independent statistical support group. The 3-person Oversight Group for each protocol includes a representative from the National Institute of Allergy and Infectious Diseases, the Biomedical Advanced Research and Development Authority, and the corresponding sponsor.

CONTENT OF DSMB REVIEWS

Once a trial begins enrolling, reviews focus on 3 main elements: trial conduct, safety, and vaccine efficacy.

Trial Conduct

At each review, the DSMB examines metrics to ensure that the trial is proceeding as planned. The DSMB looks closely at accrual, including the numbers and proportions of participants in relevant subgroups such as those defined by age, sex, race, ethnicity, and presence of risk factors that predispose to severe COVID-19. The DSMB also reviews measures of completeness of follow-up, adherence to the allocated intervention, and data quality, such as the proportions of participants with incomplete case report forms or with unanswered queries from sponsoring companies to local sites. Recommendations related to accrual have included increased representation of participants from racial and ethnic groups that have suffered disproportionately from the pandemic and of subpopulations that, based on the epidemiologic literature, have risk factors for severe disease. For example, the DSMB requested that sponsors establish specific goals for recruitment of demographic subgroups based upon their proportions of the US population. Sponsors followed these requests, even taking steps such as pausing accrual of individuals from adequately represented groups in order to ensure that final samples included appropriate demographic diversity in response to DSMB recommendations. Recommendations related to data quality have included requests that sponsoring companies evaluate whether missing data are disproportionately attributable to particular sites, allowing focused remediation, and that they pause enrollment at sites that were not keeping up with data entry.

Safety

Participant safety is a central responsibility of the DSMB, which devotes substantial attention at each meeting to review of interim safety metrics. In addition, the DSMB regularly receives reports of individual safety events between meetings and discusses via email whether further information or actions in response are needed. When considering serious adverse events that merit individualized evaluation, the DSMB is informed of the group to which the participant was assigned in order to facilitates its ability to make judgments about causation. Given the large number of participants in the trials that the DSMB monitors, the need to review individual unblinded adverse event reports arises frequently.

Evaluation of safety, a responsibility that the DSMB shares with regulatory agencies, institutional review boards, and sponsoring companies, is the most demanding aspect of the Board’s role. The DSMB oversees multiple trials, each with tens of thousands of participants. Furthermore, by design, trials include numerous participants who are older adults or who have comorbidities that, independent of their participation in a vaccine trial, place them at elevated risk of death or serious health events. Thus, some serious adverse events, including deaths, are anticipated among study participants. When a vaccine recipient experiences such an event, the DSMB must assess the likelihood that it was related to the vaccine and, if so, whether it recommends changes to the protocol or to informed consent documents. In the most concerning instances, the DSMB must decide whether to recommend that trial accrual and administration of vaccine and placebo be paused pending further investigation and, if a study is paused, whether and when to recommend that it resume. As an example, the DSMB was involved in reviewing a case of transverse sinus thrombosis associated with thrombocytopenia that occurred in a vaccine recipient in Janssen’s clinical trial, leading to a study pause. After careful consideration, the DSMB endorsed the decision to resume accrual to the trial [16]. No further cases were reported among trial participants. However, following emergency use authorization of the Janssen vaccine by the FDA, additional cases of similar adverse events were reported among individuals receiving the vaccine outside the trial, resulting in a recommendation by the FDA and CDC to pause deployment. Ten days later, after further review and 2 meetings of the CDC’s Advisory Committee on Immunization Practices, deployment of the Janssen vaccine resumed [17].

Early in the trial program, additional safety concerns related to the possibility that administration of a COVID-19 vaccine might increase rather than decrease the incidence or severity of disease [18, 19]. Increased incidence or severity could occur due to antibody-dependent enhancement, as has been noted with dengue virus infection and, potentially, with the Dengvaxia® vaccine [20, 21]. Alternately, vaccine-associated enhanced respiratory disease, as was noted with the formalin-inactivated, alum-adjuvanted whole virion vaccine developed to prevent respiratory syncytial virus illness in the 1960s, might be seen among recipients [22, 23]. To ensure vigilance, protocols incorporate harm monitoring plans that include frequent comparisons of the incidence of protocol-defined severe COVID-19 between groups. Were the DSMB to observe a paradoxically increased incidence of severe COVID-19 among participants in a trial’s vaccine group, it would consider recommending that the trial be paused or terminated due to increased risk of harm. This situation has not arisen to date.

The DSMB’s role in overseeing a portfolio of multiple trials has facilitated its ability to perform safety monitoring across all trials. For example, when concerns first surfaced about thromboembolic events associated with AstraZeneca’s vaccine in Europe, the DSMB was able to review relevant categories of adverse events across its portfolio of trials to look for broader patterns associated with SARS-CoV-2 vaccines as a class.

Efficacy

A third component of the DSMB’s mission is to review interim analyses of efficacy outcomes. Each statistical analysis plan specifies an approach to efficacy analysis, including the numbers of events (ie, cases of symptomatic, laboratory-proven COVID-19) that would trigger a formal interim efficacy analysis and the statistical decision rules that should be applied in making recommendations about early termination of, or modifications to, ongoing trials. The DSMB reviews 2 type of efficacy analyses. The first addresses whether accumulating data suggest that it is highly unlikely that a vaccine will meet specified criteria for effectiveness (ie, futility analysis). The second addresses whether the vaccine has shown convincing evidence of efficacy by surpassing stringent, prespecified criteria. If the DSMB believes there is a compelling case for futility, or if there is overwhelming evidence of efficacy without serious countervailing concerns about safety, it can inform the study’s Oversight Group of the data and make a recommendation regarding the future conduct of the study (as an example, the DSMB recommended that the manufacturer and the sponsoring federal agencies be unblinded due to a marked reduction in COVID-19 diagnoses among vaccine compared with placebo recipients after its first interim review of efficacy data from Moderna’s clinical trial). Releasing such information to the Oversight Group implies that the independent DSMB believes the data are compelling and actionable and allows manufacturers to take actions such as submitting applications to regulatory agencies for emergency use authorization or full approval or notifying participants and the public of study findings.

CHALLENGES

The DSMB has faced numerous challenges, including the trials’ remarkable scale and pace, the need to monitor a portfolio of related trials rather than a single trial, and the politicized setting in which the trials have taken place.

The trials that the DSMB monitors are enrolling at hundreds of sites around the United States (and, in some cases, in other countries as well) and have target sample sizes of 30 000 to 40 000 each. Participants enrolled at the rate of several hundred per day and, particularly with the winter 2020–2021 peak of the pandemic, multiple participants were diagnosed with laboratory-proven symptomatic COVID-19 (the trials’ primary end point) each day. In addition, across the ongoing trials, the DSMB receives at least several reports each week related to serious adverse events. This scale and pace have placed extraordinary demands on sponsoring companies, which have worked diligently to collect, adjudicate, compile, and analyze vast amounts of data on short timelines. It has also placed demands on the DSMB and its staff to evaluate data in a timely fashion and to be available for urgent ad hoc reviews as needed.

Another challenge has been harmonization among studies. Inconsistencies among trials might have led to confusion about why different rules applied to different companies, and although OWS sought to harmonize protocols across companies, differences remained. The DSMB has recommended modifications that promote greater alignment across trials, including the specification of end points, the numbers of events at which interim efficacy analyses take place, the thresholds and corresponding numbers of events needed to assess efficacy at final analyses, and the statistical approaches and boundaries used for futility and efficacy monitoring. For example, while some protocols proposed 1 or 2 interim analyses after prespecified numbers of events, others initially proposed continuous interim efficacy monitoring starting after a small number of events. Although all interim analysis plans were statistically valid, the DSMB recommended the consistent use of the former approach across all protocols to ensure a uniform level of evidence needed to stop a trial early for efficacy and to promote public understanding of and trust in the process.

Finally, the COVID-19 vaccine trials have been perhaps the most politicized trials in history, even becoming embroiled in United States presidential election politics [24, 25]. The politicization of these trials prompted prominent figures in the scientific community to question whether vaccine approval might be rushed for political reasons and fostered public concern about whether safety would be compromised [26, 27]. Notwithstanding these controversies, the DSMB has focused throughout on its primary goals—the safety of study participants and the integrity and scientific validity of the trials that it is tasked to oversee—and has encountered no interference with its ability to fulfill its charge. The DSMB’s reporting structure to an oversight group that consists of career officials within NIAID and BARDA as well as a representative of the sponsoring company aids in ensuring the board’s independence.

CONCLUSION

Operation Warp Speed is an unprecedented effort to develop safe and effective vaccines that will help end the COVID-19 pandemic. Conducting clinical trials under these circumstances requires the utmost attention to participant safety and to data integrity so that the public and the medical community will ultimately have trust in the vaccines and the process used to develop them. Although it operates behind the scenes, by virtue of its access to unblinded interim data, its charge to recommend changes to ongoing studies based on these data, and its ability to examine emerging data across multiple parallel trials, the COVID-19 Vaccine DSMB is uniquely positioned to ensure that these goals are met. Furthermore, the single DSMB approach can serve as a model for future situations in which there is an urgent need for coordinated development of multiple therapeutic or preventive interventions to address rapidly evolving public health threats.

Supplementary Data

Supplementary materials are available at The Journal of Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.