Rapid implementation of a modular clinical trial informatics solution for COVID-19 research.

Veterans Health Administration (VHA) services are most frequently used by patients 65 years and older, an age group that is disproportionally affected by COVID-19. Here we describe a modular Clinical Trial Informatics Solution (CTIS) that was rapidly developed and deployed to support a multi-hospital embedded pragmatic clinical trial in COVID-19 patients within the VHA. Our CTIS includes tools for patient eligibility screening, informed consent tracking, treatment randomization, EHR data transformation for reporting and interfaces for patient outcome and adverse event tracking. We hope our CTIS component descriptions and practical lessons learned will serve as a useful building block for others creating their own clinical trial tools and have made application and database code publicly available.

Introduction

On January 21, 2020 the Center for Disease Control confirmed the first U.S. Coronavirus case and on March 13th a National Emergency was declared [1]. With no established treatment for the growing number of COVID-19 infections, clinical trials became an urgent path forward. However, trial start-up time can vary significantly with some trials taking as long as 8 months from mental conception to initial enrollment [2]. Such delays are not only driven by the time needed for obtaining regulatory approvals [3] but also by the availability of technology infrastructures necessary to support a new trial [2].

The Veterans Health Administration (VHA) is the largest integrated health care system in the United States [4] with services most frequently used by veterans over age 65, an age group disproportionally affected by COVID-19. In this paper we describe a modular clinical trial informatics solution (CTIS) that was rapidly built (∼1 month) to support an embedded pragmatic clinical trial (ePCT), with adaptive randomization [5]. The trial compared usual care alone to usual care plus Sarilumab in patients hospitalized with COVID-19. EPCTs are conducted during the course of usual clinical care [6,7] thereby allowing study endpoints, treatment deviations, and compliance data to be extracted from a patient's longitudinal electronic health record (EHR). The added use of adaptive randomization helps mitigate potential dangers associated with off-label drug use as occurred with hydroxyquinone [8]. A detailed rationale of the benefits associated with a pragmatic adaptive design during the pandemic, and the results of the trial, are described elsewhere [9].

At present, the VHA lacks an enterprise-level, clinical trial management system to aid the instantiation of new clinical trials. Although many options exist, most out-of-the-box software solutions cannot be readily integrated with enterprise EHR systems and often lack flexible randomization capabilities necessary to conduct an adaptive ePCT. It should be noted that within the VHA, software selection is complicated by necessary compliance with robust data security rules and integration with its mature and rigid EHR infrastructures. Indeed, formal VHA approval for the installation and use of commercial tools with protected health data can often take years. Due to the urgency for implementation, we focused on supporting essential tasks within an adaptive ePCT workflow with tools that could be developed and implemented within a few weeks. In addition to randomization functions, working with the study team, we identified other key processes during screening, consenting, and monitoring that could be optimized with software tools.

In spring 2020 we developed and implemented our own clinical trial informatics solution (CTIS) that includes a custom developed (a) eligibility screening graphical user interface (GUI), (b) consent tracking GUI, (c) a treatment randomization application, (d) outcome and adverse event reporting GUIs and, (e) auditable database tables for EHR data. In this paper, we describe our CTIS components and discuss our practical lessons learned from an expedited deployment early in the pandemic. We hope our component descriptions will serve as a useful building block for others creating their own clinical trial tools and have made code for our randomization application and database tables publicly available.

Materials and methods

VHA EHR systems and CTIS development process

The Veterans Affairs (VA) electronic health record (EHR), called the Veterans Health Information Systems and Technology Architecture (VistA), was established in 1981 and operationalized for full clinical use in 1999 [10,11]. Healthcare providers access and enter data into VistA via the Computerized Patient Record System (CPRS) user interface. A subset of VistA data is stored nightly in the VA Corporate Data Warehouse (CDW) where it resides in Microsoft (MS) SQL (MSSQL) databases for secondary use.

The CTIS described here was created to fill in a functional gap in VistA to support a multi-hospital, adaptive randomization ePCT. The trial recruited hospitalized COVID-19 patients at five VHA sites including Boston, MA; Togus, ME; West Haven, CT; Providence, RI and White River Junction, VT. Recruitment occurred via inpatient monitoring with recruitment in Boston being additionally facilitated by the Covid-19 Data Management Platform [12].

The CTIS includes applications developed in Java and Microsoft Access. It also employs custom developed graphical user interfaces (GUIs) embedded directly within the VA EHR and utilizes MSSQL databases for study data storage. Furthermore, as EHR data views are not static, custom ‘Informatics Tables’ were developed to facilitate auditing longitudinally.

During start-up, trial governance and software development teams held daily scrums to facilitate problem solving. Rapid and frequent software development cycles, collation of requirements and code versioning were managed using the VA Enterprise GitHub. Ongoing trial governance documentation and study task matrix diagrams were maintained in a Microsoft SharePoint site accessible to all study personnel.

Daily development scrums continued until applications reached maintenance mode, at which time scrums were held weekly. During each meet, the scrum leader addressed open GitHub issue tickets with the team which consisted of a data engineer, a front-end developer, a Quality Assurance engineer, and the technical Project Manager (PM). The technical PM met with stakeholders on a weekly basis, to ensure requirements were being met and issues addressed. Releases were first built, configured, and customized in a ‘development environment’ and subsequently underwent user acceptance testing (UAT) on the servers within an ‘integration environment’. Once all tests were passed, software was deployed for use by the study in the LIVE environment.

The code used to create our Informatics Tables is available to approved VA researchers via request and also available within the VA Centralized Interactive Phenomics Resource (CIPHER) library. The code used in the creation of our Randomizer application, described in the next section, is available via our public facing repository (https://github.com/bostoninformatics/Covid19-TrialRandomization).

Trial workflow

A generalized depiction of the trial workflow is shown in Fig. 1 A. First, patients hospitalized with COVID-19 were screened against trial inclusion and exclusion criteria. Eligible patients who consented to participate in the study were then randomized to treatment with standard of care (SOC) alone or SOC with Sarilumab. Patient identifiers and assigned treatment arm were tracked both in custom developed form GUIs embedded within CPRS (EHR) as well as the Randomizer application database. These data were used for real-time enrollment tracking and extraction of patient specific data from the EHR. In addition to clinician chart review and manual logging of outcomes and adverse events (AEs) in custom developed GUIs, data extracted from the EHR was used in patient monitoring, statistical analyses, and study reporting. Patient outcome data was also used to update treatment strategy as part of the overarching adaptive randomization design.

Fig. 1

Trial Workflow and Clinical Trial Informatics Solution (CTIS) Components

(A) Generalized depiction of the trial workflow with colors denoting each corresponding, custom developed, support tool. (B) Modular CTIS components numbered 1–6 were all accessible via a VHA workstation. Clinicians utilized the Computerized Patient Record System (CPRS) user interface to view and enter patient data into VistA. All VHA patient data in VistA is stored nightly in the Corporate Data Warehouse (CDW) where it resides for research use. Custom developed (B1) Eligibility Screening and (B2) Consent Tracking forms were embedded within the EHR via CPRS. The Consent Tracking form links to the (B3) Randomizer application which was used to assign trial treatment strategy. Patient EHR data was extracted nightly from the CDW into the (B4) Trial DB. Within the Trial DB custom developed ‘Informatics Tables’ were used to facilitate data consumption and auditing. (B5) Patient Outcomes and Adverse Event tracking were used to collate data entered via chart review for regulatory reporting and used by the study statistician use to periodically update the randomization schedule. These data were also validated via EHR data extracts. Data from all CTIS components were combined with relevant EHR data in the Trial DB and (B6) real-time enrollment as well as scheduled reports were generated. . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Results

The design and implementation of a modular CTIS to support our ePCT began in April 2020 and took roughly one month to complete. The CTIS includes custom developed (1) EHR embedded “Eligibility Screening” GUI, (2) EHR embedded patient “Consent Tracking” GUI, (3) a web-based trial “Randomizer” application, (4) a “Trial Database” with auditable ‘Informatics Tables’ to facilitate reporting and, (5) GUIs for patient “Outcome and Adverse Event (AE)” tracking. The CTIS also includes components for the creation of real-time and scheduled reports. A simplified depiction of the CTIS components and trial data flow is shown in Fig. 1 below.

Eligibility screening and patient consent tracking forms

All clinical trials require precise patient eligibility screening and documentation of informed consent. To assure uniform screening and consent procedures across participating sites, study-specific “Eligibility Screening” and “Consent Tracking” GUIs were custom developed and embedded into CPRS at each of the 5 participating VHA sites (Fig. 1B,1-2). Once completed and signed, each GUI form generates a clinical note within the corresponding patient's longitudinal health record.

The Eligibility Screening GUI was used to guide consenting clinicians through trial inclusion and exclusion criteria using a series of multiple-choice questions (e.g. symptom severity, respiratory rate, O2 saturation, worsening chest x-ray, Brescia risk score). Clinicians then indicated if the patient met eligibility. The Consent Tracking GUI described the purpose of the trial and prompted clinicians to follow proper consent procedures. Once signed by the clinician, the resultant clinical note functions as an EHR indicator that the patient is enrolled in the trial. Both Eligibility Screening and Consent Tracking data captured in CPRS is stored daily in the CDW and was subsequently extracted into the Trial Database (Figs. 1B and 4). The precautions and procedures used for obtaining patient consent while minimizing viral transmission through the use of electronic forms are described in detail elsewhere [13].

Trial randomization (Randomizer application)

The trial described in this paper utilized an adaptive randomization strategy. This type of strategy requires frequent modification of an existing randomization schedule based on patient outcomes (Fig. 1A). CPRS/VISTA is a mature system that cannot adequately support integration of multi-site trial randomization tools. Our solution was the “Randomizer”, a Java based web-based application that assigns treatment by reading a pre-specified randomization schedule within its back end MSSQL database tables. Furthermore, although some applications focus on the creation of randomization schedules [14] our tool was designed to facilitate routine modification of an existing schedule while also providing a GUI that blinded clinical staff (users) to the ongoing treatment strategy. A picture of the Randomizer GUI is shown in Fig. 2 . Clinicians were able to navigate to the Randomizer through the Consent Tracking GUI (Figs. 1B,2-3). Once patient information is entered into the Randomizer, a treatment assignment is returned and then logged within the Tracking GUI.

Fig. 2

Randomizer Application GUI

The Randomizer Application was used to determine and track patient treatment arm. Specifically, clinicians entered patient identifiers (solid red arrows) including name, SSN, date-of-birth, inpatient site, and the application returned a treatment assignment (dashed dark blue arrow), e.g., Standard-of-Care (SOC) or SOC with Sarilumab (Active). . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

The Randomizer also auto-generated email notifications after each randomization event. Emails included information on the date and time of the randomization as well as the total number of patients randomized for the study. These notifications not only helped keep study statisticians apprised of when an updated randomization schedule was needed but also facilitated continuity of operations during temporary server outages early in the pandemic. Specifically, knowledge of the last randomization assignment allowed study statisticians (with knowledge of the longitudinal randomization schedule) to inform study clinicians about the next treatment assignment via email or secure messaging in case the Randomizer application became temporarily inaccessible. In general, it is important to note that the Randomizer's flexible design allows it to be easily adapted for use in other trials. Specifically, the back-end database tables may be readily altered to support different types of randomization schedules. To potentially facilitate work for others, we have made our Randomizer code available via our public facing repository.

Patient Outcomes and Adverse Event tracking GUIs

Patient outcomes were tracked both via CDW EHR extracts and clinician chart review. To facilitate chart review, a custom MS Access GUI was developed (Fig. 3 ). Specifically, during the first fourteen days following patient randomization, study clinicians performed EHR chart review to identify specific patient Outcomes as well as Adverse Events (AEs). This information was manually entered into an “Outcome Report” GUI and a “AE Report” GUI. Primary Outcomes that were manually tracked in the Outcome Report (Fig. 3A) and included, intubation, discharge, and death. These standardized outcome data were also used by statisticians to update treatment strategy as part of the overarching adaptive randomization design.

Fig. 3

Outcome and Adverse Event Tracking GUIs

The CTIS includes customized GUIs to facilitate outcome and adverse event reporting via clinician chart review. (A) The Outcome Report GUI (top panel) allows selection of relevant primary/secondary study outcomes and date of occurrence as well as space for additional details as free text (red arrows). Multiple outcomes may be entered. These standardized outcome data were used by study statisticians to update treatment strategy as part of the overarching adaptive randomization design. (B) Adverse Event (AE) Report GUI (bottom panel) also includes dropdown menus and free-text boxes for detailed data entry (orange arrows). . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

The AE Report interface (Fig. 3B) was used to facilitate collection of data necessary for trial safety reporting and required manual chart review to be performed until 30 days post-discharge. To inform clinicians on when to initiate or terminate tracking of a given event or patient, AE interface logic calculates and displays relevant reporting date ranges for every patient. As with other data collection, Outcome and AE tracking were conducted remotely without patient contact. The graphical interfaces in Fig. 3 were developed iteratively in collaboration with clinical and quality assurance staff. To assure uniformity and increase efficiency of data entry, the GUIs employed drop-down menus where possible.

Trial database: patient EHR data and reporting

EPCTs rely heavily on longitudinal EHR data for patient monitoring and trial reporting. Within the VHA, EHR data is available for secondary use via the CDW. The CDW has thousands of health-related tables and in some cases only mature or recently in-process documentation is available. Indeed, new researchers less experienced with the CDW data model may spend weeks to months attempting to evaluate and join relevant tables for their research study. Furthermore, as provisioned data views are not static, the creation of auditable trial database tables that track data changes over time can also be useful.

As part of our CTIS, data including but not limited to, demographic factors, medical history, laboratory values, inpatient and outpatient procedure codes and diagnosis codes was extracted daily from the CDW. The data domains used are applicable to other VHA research efforts with similar reporting requirements. To assist others, we delineate which domains were used in our Informatics Tables (Table 1 ). The tables were updated via stored procedures that ran nightly and designed to facilitate auditing as each unique patient record maintained its initial extract transform load (ETL) timestamp. Specifically, new data were merged daily, and for records in which data had been modified a new record with a corresponding ETL timestamp was added. Code used to create the Informatics Tables is available to VA personnel by request and is also available in VA CIPHER library.

Table 1

The present study utilized VHA EHR data domains that are applicable to other research efforts with similar reporting requirements. Here we delineate which domains were used for each data table. Corresponding SQL code is easily repurposed and made available via the VA CIPHER library or by request.

Table 1: Auditable Informatics Tables
Table Name	Relevant Data Domains	Table Description & Intended Uses
BCMA	• BCMAMedicationLog• BCMADispensedDrug• LocalDrug• DrugClass• IVSolutionIngredient• BCMAAdditive• BCMASolution	• Contains information on quantity, drug name, drug class, and method of delivery for medications administered during inpatient visits.• Was used to evaluate concurrent medications administered as part of inpatient standard of care treatment
RxOutpatFill	• RxOutpatFill• LocalDrug• DrugClass	• Contains information on drug name, drug class, dosage, and delivery form of outpatient prescriptions.• Was used to track treatment of co-morbidities and AE’s after discharge
CptProcedures	• InpatientCPTProcedure• OutpatVProcedure• CPT	• Contains information on coding and description of procedures performed during inpatient and outpatient visits. Procedures are defined by CPT codes.• Was used to track relevant procedures
InpatProcedures	• InpatientICDProcedure• ICD10Procedure• ICD10ProcedureDescriptionVersion• ICD9Procedure• ICD9ProcedureDescriptionVersion• InpatientSurgicalProcedure	• Contains information on coding and description of procedures performed during inpatient visits. Procedures are defined by ICD codes.• Was used to track relevant procedures
DiagnosisInpatient	• InpatientDiagnosis• Inpatient• ICD10• ICD10DescriptionVersion• ICD9• ICD9DescriptionVersion	• Contains information on coding and description of diagnoses made during inpatient visits. Diagnoses are defined with ICD9 and ICD10 codes depending on the date they were made.• Was used to assess the presence of relevant co-morbidities as well as outcomes and adverse events
DiagnosisOutpat	• OutpatVDiagnosis• ICD10• ICD9• ICD10DescriptionVersion• ICD9DescriptionVersion	• Contains information on coding and description of diagnoses made during outpatient visits. Diagnoses are defined with ICD9 and ICD10 codes depending on the date they were made.• Was used to assess the presence of relevant co-morbidities
LabChem	• LabChemTest• LOINC• Topography• Chem_LabPanel	• Contains information on lab type, component of interest, LOINC codes, and lab values• Was used to assess patient status and confirm certain diagnoses of interest
MicroOrderedTest	• MicroOrderedTest• LabChemTest• Topography	• Contains information on lab type, component of interest, LOINC codes, and microbiology test values• Was used to assess patient status and confirm infection related diagnoses of interest
VitalSign	• VitalSign• VitalType	• Contains information on vital readings including heart rate, blood pressure, temperature, oxygen saturation• Was used to assess patient status during hospitalization

For our purposes, additional data monitoring committee (DMC) ‘report ready’ database tables were created from the auditable Informatics Tables. These included tables for Medical Dx History, Inpatient Medications, Demographics and Baseline Vitals, Baseline Lab Chemistry, Symptoms on Admission and Adverse Events. The tables contained filters for medication type as well as time of delivery after hospital admission, laboratory value thresholds, date, and type of specific events and so on. Report table creation followed an iterative process wherein logic was implemented, and the accuracy of the data was evaluated by clinician chart review. Data quality was monitored regularly over the course of the study. Real-time enrollment reports are created using MS SQL Server Reporting (MS SSRS) services while scheduled reports of ongoing trial results are created using SAS software.

Discussion

Early during the pandemic, in April 2020, we created a Clinical Trial Informatics Solution (CTIS) to support a multi-site ePCT in COVID-19 patients hospitalized within the VHA. Our work demonstrates the value of an informatics approach to support the urgent needs of a large enterprise that is well entrenched in a legacy EHR system with limited options when under cost and schedule constraints. Through an environmental scan, needs assessment, and workflow analysis we were able to deliver a solution that facilitated the launch of a clinical trial from conception to implementation in one month. This was the fastest instantiation of a clinical trial at our institution which has decades of experience with clinical trials. Such speed is critical during a pandemic where lives may be saved by new trials for off-label drug use.

Our CTIS includes a custom developed (a) eligibility screening GUI, (b) consent tracking GUI, (c) a treatment randomization application, (d) outcome and adverse event reporting GUIs and, (e) auditable database tables for EHR data. We focused on the essential trial processes that have the most impact on safety and efficiency through automation. Our modular solution fits seamlessly within existing VHA computing infrastructures and maintains the flexibility necessary for rapid modification and reuse in other clinical trials. Rapid implementation of a new system during the early part of the pandemic was challenging and required changes in our software development framework. Here we describe issues encountered and lessons learned to meet the demands for rapid development during a pandemic from an ergonomic and practical standpoint.

Centralized information sharing and continuity of operations

Implementation of the modular CTIS required a multidisciplinary team of software engineers, health informaticians, statisticians, quality assurance personnel and infectious disease clinicians. Under normal circumstances, task delegation and motivation of staff occurs through in-person, group meetings at the start of the study. The pandemic prevented conventional group meetings from taking place and new methods of team management were required [15,16]. To effectively manage dispersed team members with complementary skill sets we employed ‘objective focused’ daily scrums and enforced use of virtual meeting platforms. To facilitate awareness, we reduced person-to-person emails and enforced maintenance of critical communications as well as meeting minutes and governance documentation within sharing platforms, i.e. MS SharePoint and the VA GitHub. We found this centralized maintenance of information facilitated information dissemination and enhanced shared understanding in our virtual teams.

Furthermore, during start-up, teleworking revealed many latent technological challenges including but not limited to system load issues. In our case, application server outages brought contingency planning to the forefront. Specifically, the Randomizer build schedule was adjusted to prioritize automatic email notifications to provide regular situational awareness for clinicians and statisticians regarding how many patients were randomized and their treatment assignment. Thus, in the case the Randomizer application became inaccessible, treatment assignment for new patients could still occur via secure messaging or telephone between statistician and the randomizing clinician.

Rapid prototyping, Audit mechanisms and tool Re-Usability

The urgency in starting our trial forced us to implement our tools quickly without extensive pre-build requirement gathering. To ensure forward momentum we employed rapid (3–5 day) design-prototype-test cycles [17]. For example, the Outcomes and AE GUIs, as shown in Fig. 3, were initially built based on broad requirements set within the trial protocol. Frequent meetings with clinicians and QA personnel helped pinpoint ways to increase usability and reduce data entry error, i.e., creation of drop-down variables and collection date reminders. Report table creation followed an iterative process wherein logic was implemented, and the accuracy of the data was evaluated by clinician chart review. These rapid yet effective development cycles allowed us to maintain productivity and meet implementation deadlines.

Furthermore, it is important to note that EHR data is not static, i.e., clinicians may modify a diagnosis or add corrections to previous entries. This can produce discrepancies in ongoing data analysis for intermediate outcome reporting. As CDW EHR data is available for research only as database views, it is good practice to implement auditable tables within the trial database itself. Thus, data change history is maintained, and analyses may be adapted accordingly.

During development we also noticed potential for tool re-usability. Specifically, although randomization strategies can differ between trials; the Randomizer can be readily repurposed by changing its treatment schedules. Furthermore, as clinical trials frequently overlap in the types of EHR data they require, the stored procedures for our Informatics Tables may be easily adapted to support other planned trials. Finally, our Outcomes and AE Report GUIs may also be redeployed by exchanging dropdown variables and collection date ranges.

Certain trial designs may be beneficial during a pandemic

The ePCT design was chosen partly in response to the restraints forced on research by the pandemic. Specifically, in many hospitals conventional research was halted until more was known about mitigating viral transmission. Fortunately, ePCTs are conducted during the course of usual clinical care [6,7] thereby allowing study endpoints, treatment deviations, and compliance data to be extracted from a patient's longitudinal electronic health record (EHR). This design can reduce research costs, enhance trial safety by minimizing staff-patient contact, and promote realization of a learning healthcare system by facilitating translation of research evidence into clinical practice. It can also help assess the relative effectiveness of treatments when standard of care is not well defined [9,18]. A detailed rationale of the benefits associated with a pragmatic adaptive design during the pandemic are described elsewhere [9,19].

Future directions

We have begun expansion of CTIS tools to include automation of patient eligibility screening workflows. Specifically, the incorporation of EHR data for automated patient-trial matching for oncology trials. Oncology clinical trial inclusion and exclusion criterion are often complex and manual patient evaluation is typically more labor intensive than those for other diseases. To this end, biomarkers such as targeted genomic sequencing results, are also utilized when available. Given the urgency of CTIS implementation for COVID-19, no formal ‘usability evaluation’ was conducted. However, as part of our ongoing expansion we plan to include usability evaluations that include but are not limited to comparison of manual vs. automated eligibility assessments.

Conclusion

We built a flexible clinical trial informatics solution (CTIS) to support a multi-site embedded pragmatic clinical trial in COVID-19 patients hospitalized within the VHA. Our CTIS includes tools for eligibility screening, consent tracking, randomization, patient outcome and adverse event tracking as well as data reporting. We hope our system descriptions will serve as a useful building block to others when creating their own solutions to support clinical trials and have made code used for our randomizer application and informatics tables available to others.

Author contributions

RD, RA, SL, NVD, FM, DE provided leadership for CTIS component development and/or management of its delivery. RD, AK, RA, RS, SP, DE were involved in implementation of CTIS components and/or maintenance of its infrastructure. SL, RF, MTB provided application requirements and trial procedure guidance. RD drafted the manuscript and SL, NVD, DE, SP contributed to revisions. All authors have read and approved the final manuscript.

Funding statement

All work described herein was supported by the VHA Boston CSPCC and conducted under an approved clinical trial research protocol (R&D Study 1597-X and VABHS IRB #3305) for trial NCT 04359901.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.