Optimizing clinical trial design using prospective cohort study data: a case study in neuro-urology.

STUDY
DESIGN: Simulations using data from a prospective cohort study.
OBJECTIVES: To illustrate how prospective cohort data can be employed in randomized controlled trial (RCT) planning to assess feasibility and operational challenges, using TASCI (Transcutaneous tibial nerve stimulation in patients with Acute Spinal Cord Injury to prevent neurogenic detrusor overactivity: a nationwide randomized, sham-controlled, double-blind clinical trial) as a case study.
SETTING: Spinal cord injury (SCI) rehabilitation centers in Switzerland.
METHODS: TASCI is nested in the multicenter Swiss Spinal Cord Injury Cohort Study (SwiSCI), which prospectively includes patients with acute SCI. In simulations, data from 640 patients, collected by SwiSCI, were used to investigate different scenarios of patient eligibility and study consent, as well as the performance of the randomization list. Descriptive analysis was used to describe the population of interest and the simulation results; multivariable logistic regression analysis was performed to identify predictors of discharge within the TASCI intervention time period.
RESULTS: The recruitment target of 114 patients is obtainable within the originally envisioned 3-year time period under the most favorable recruitment scenario examined. The distribution of the primary prognostic factor produced imbalance in the randomization lists and informed further discussion of the cut-off values used in stratification. Influxes of patients resulted in overlapping intervention periods for multiple participants, which guided resource allocation. Early discharge was related to the primary prognostic factor and study center, but is only anticipated in about 8% of participants.
CONCLUSIONS: Prospective cohort data are a very valuable resource for planning RCTs.

Introduction

Randomized controlled trials (RCTs) are often considered the ‘gold standard’ for producing decisive evidence regarding intervention efficacy, but they are also challenging to execute [1]. The successful implementation of an RCT is contingent on a thorough understanding of the targeted study population. During the study design phase, researchers commonly rely on informal projections of numbers of eligible patients, patient recruitment and dropout rates, as well as financial, human and material resources needed for trial execution. Yet, reviews of ethics protocol, trial registry, funding body and literature databases indicate that up to 25% of RCTs fail to meet projected recruitment targets, underestimate the overall trial duration and resource needs, and consequently may even be terminated prematurely [2-7]. Many of these forecast failures could have at least partly been avoided with a more rigorous study planning phase, including a thorough review of published evidence and a comprehensive evaluation of critical projections [2, 8, 9]. Observational data from cohort studies or disease registries can provide valuable information on the underlying study population in an RCT design phase [10-12], but it is not systematically utilized in the medical field.

Employing cohort data for RCT planning provides advantages for trial design. As a case in point, we present results from the planning phase of TASCI (Transcutaneous tibial nerve stimulation in patients with Acute Spinal Cord Injury to prevent neurogenic detrusor overactivity: a nationwide randomized, sham-controlled, double-blind clinical trial) [13, 14], which is nested in the Swiss Spinal Cord Injury Cohort Study (SwiSCI) [15]. TASCI aims to investigate the efficacy of transcutaneous tibial nerve stimulation (TTNS) in preventing the development of detrusor overactivity after spinal cord injury (SCI). The trial presents some challenges from a study design and implementation perspective, including a relatively small target population, a multicenter inpatient rehabilitation setting, an investigational procedure that involves an extended period of daily intervention, and repeated outcome assessments. We will use the cohort data to simulate, under different scenarios, anticipated recruitment numbers, and performance of the randomization scheme, as well as to investigate operational challenges in resource management (e.g., cumulative workload due to participant overlap) and participant dropout (e.g., discharge during the intervention window).

Methods

Design, Setting, and Participants

This simulation study was conceived to illustrate key trial characteristics under different scenarios of patient eligibility, recruitment and groupings of the primary prognostic factor for stratification. Accordingly, eligible patients were identified using data from the SwiSCI cohort, consent was modelled, and then the participants were assigned to randomization lists that were stratified by the primary prognostic factor and study center.

Data were extracted from the SwiSCI inception cohort, a multicenter, prospective, longitudinal study [15]. Swiss residents, age ≥ 16, with a newly diagnosed SCI, who were admitted for specialized rehabilitation to one of the four Swiss SCI centers were eligible. Patients with an SCI related to a congenital condition, neurodegenerative disorder, Guillain–Barré syndrome, or who had a new SCI in the context of palliative care were excluded. The present study used data collected 28 days (time frame: 16–40 days) after SCI diagnosis and at discharge (time frame: 10–0 days before discharge) from participants who were admitted between May 2014 and April 2017. Written informed consent was obtained from all participants, 91% of the eligible population consented to routine data collection from the medical record, and 52% of the population consented to additional assessments. Ethical approval was granted by the ethics committees of Northwest/Central Switzerland (PB 2016-00183), Vaud (032/13), and Zürich (2013-0249).

Based on the trial’s power calculation, TASCI aims to recruit a total of 114 patients receiving specialized rehabilitation in one of the Swiss SCI centers over a 3-year period. TASCI inclusion criteria that are verifiable using SwiSCI data are: age ≥ 18 years; diagnosis of traumatic SCI or sudden-onset non-traumatic SCI, at the cervical or thoracic level; and admitted within 40 days of diagnosis. Exclusion criteria such as development of detrusor overactivity that necessitates treatment in the acute period after SCI [16], neurological and urological conditions preceding SCI, treatment and medication-use, or participation in another study (for comprehensive listing, see NCT03965299 at ClinicalTrials.gov) could not be evaluated for lack of relevant detail. Participants are randomized using permuted block randomization lists with varying block sizes, stratified on study center and lower extremity motor score (LEMS) [17], the primary prognostic factor. LEMS was identified as a good predictor of a favorable bladder outcome (urinary continence and complete bladder emptying) one year after SCI [18, 19]. Participants undergo screening and baseline assessments between days five and 39 after SCI, and begin the sham-controlled neuromodulation intervention (TTNS) by day 40 at the latest. The supervised intervention is continued five days per week for 30 minutes, until all of the three-month follow-up assessments have been completed (days 81–101). Masking of participants, care-providers, and outcome assessors to the group allocation is maintained until the final follow-up assessment, one year after SCI. The primary outcome, development of detrusor overactivity that jeopardizes the upper urinary tract within the first year of SCI, is evaluated using urodynamic assessment.

Study Measures

SwiSCI extracts routine data regarding demographics and rehabilitation characteristics from the clinical record. These parameters include: gender, age at SCI, SCI etiology, dates of admission to and discharge from rehabilitation, and dates of all relevant clinical assessments. The International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) [17] protocol was used to assess neurological status including: SCI level, SCI completeness (American Spinal Injury Association (ASIA) Impairment Scale (AIS)) and LEMS. Lower urinary tract symptoms (LUTS) were assessed with the Spinal Cord Independence Measure (SCIM) III [20], item 6, ‘Sphincter Management – Bladder’ and the International Spinal Cord Injury Lower Urinary Tract Function Basic Data Set (available in patients that consented to additional assessments) [21].

Simulations

First, a basic trial eligibility database was generated by applying the inclusion criteria from TASCI to the population from the SwiSCI source database (Supplement 1). To ensure a conservative targeting of the simulation population, additional exclusion criteria were: absence of any LUTS within 40 days of SCI (n=36), lack of LEMS within 40 days of injury (n=4), or discharge before trial start (within 40 days of SCI, n=4). To address uncertainty about the suitability of the population with only routine data collected (n=52) for a clinical trial, as these persons tend to have a higher comorbidity load (unpublished analysis, as informed by discussions with SwiSCI personnel), a random number with a uniform distribution was used to select either 50% or 75% of that population for the eligibility database (Figure 1). Patients who consented to additional clinical assessments (n=133) were entered directly into the basic trial eligibility database.

Figure 1

Simulation flowchart. * = assignment to treatment groups is identical for all centers; LEMS = lower extremity motor score; SwiSCI = Swiss Spinal Cord Injury Cohort Study; TTNS = transcutaneous tibial nerve stimulation.

Next, consent to the trial was modelled by including either 50% or 66% of the patients based on random numbers from a uniform distribution. The rates were chosen based on discussion with the neuro-urologists, as well as the historical recruitment rates for similar procedures at the local study centers. Additional rationale for choosing these rates was: the intervention is non-invasive and has proven safe when used in other contexts [22], the inpatient setting and embedding in routine SCI follow-up reduces burden on the participants, and there is no alternative evidence-based non-invasive intervention for preventing the development of detrusor overactivity after SCI. Participants were assigned to a randomization list in order of admission to rehabilitation. Floor and ceiling effects in the distribution of the primary prognostic indicator (Figure 2A) lead to the implementation of three different models of cut-off values. The process was repeated over 20 different randomization seeds, with three different groupings for the primary prognostic factor and generating a new permuted block randomization list for each random seed.

Figure 2

LEMS: distribution in the patient population, and imbalance in the simulations. Distribution of LEMS in the population of patients eligible for TASCI (A) and across study centers, grouped according to LEMS Model 1 (B). In simulations, imbalance was seen in the middle LEMS categories across all models, but was especially pronounced in Model 1 (C). LEMS models represent different cut-off values that are categorized based on the percent probability of urinary continence and complete bladder emptying one year after SCI: Model 1: < 33% (LEMS 0-17) (light grey in B), 33%-66% (LEMS 18-32) (medium grey in B), and > 66% (LEMS 33-50) (black in B); Model 2: < 25% (LEMS 0-14), 25%-75% (LEMS 15-35)> 75% (LEMS 36-50); or Model 3: < 20% (LEMS 0-11), 20%-80% (12-37) and > 80% (38-50). LEMS = lower extremity motor score.

Statistical Analyses

Analyses were conducted using Stata version 14.2 for Windows (College Station, TX, USA). Descriptive statistics were used to describe the population of potentially eligible patients and the simulation results. Continuous variables were tested for normality. Chi-square, Fisher’s exact, and Kruskal-Wallis tests were used to test for differences across study years and models of LEMS cut-offs. Imbalance in the randomization lists was investigated and reported as the absolute difference between the proportion of participants assigned to either intervention [23]. To describe participant flow, the intervention start was defined as day 34 (middle of the baseline assessment window) and intervention end as day 91 (middle of three-month follow-up window). Multivariable logistic regression was used to identify predictors of early discharge from the hospital within the TASCI intervention time period (< 81 days after SCI diagnosis).

Results

Validity of the Cohort Data for Trial Planning

The cohort data showed minimal between-year variation in demographic and rehabilitation characteristics (including center), SCI characteristics (including LEMS), and LUTS (p > 0.10 for all, Table 1). Further investigation into the urological management (bladder emptying method, p = 0.009) on a study center level revealed that all centers were stable across the three year period (center 1: p = 0.53; center 2: p = 0.70; center 3: p = 0.17; center 4: p = 0.07). Thus, these results support the use of the SwiSCI data in the context of TASCI planning.

Table 1

Characteristics of the study population. AIC = assisted intermittent catheterization; AIS = American Spinal Injury Association (ASIA) Impairment Scale (AIS); C = cervical; LEMS = lower extremity motor score; NTSCI = non-traumatic spinal cord injury; SCI = spinal cord injury; SIC = self intermittent catheterization; T = thoracic; TSCI = traumatic spinal cord injury.

Characteristic [Missing]	Overall	Year 1	Year 2	Year 3	p
Study population	N = 185	n = 54	n = 61	n = 70
Continuous variables	Median (Q1-Q3)	Median (Q1-Q3)	Median (Q1-Q3)	Median (Q1-Q3)
Age at SCI [0]	58 (43-69)	58 (34-73)	58 (49-68)	58 (43-70)	0.95
Length of stay (days) [1]	177.5 (118-239.5)	187 (112-267)	187 (130-239)	187 (111-216)	0.21
Categorical variables	n (%)	n (%)	n (%)	n (%)
Gender [0]					0.90
Female	45 (24.3)	13 (24.1)	16 (26.2)	16 (22.9)
Male	140 (75.7)	41 (75.9)	45 (73.8)	54 (77.1)
Lesion etiology [0]					0.35
NTSCI	40 (21.6)	8 (14.8)	15 (24.6)	17 (24.3)
TSCI	145 (78.4)	46 (85.2)	46 (75.4)	53 (75.7)
Neurological category[a] [1]					0.32
C1-C4 (AIS A,B,C)	22 (11.9)	10 (18.5)	8 (13.1)	4 (5.7)
C5-C8 (AIS A,B,C)	20 (10.8)	5 (9.3)	6 (9.8)	9 (12.9)
T1-T12 (AIS A,B,C)	66 (35.7)	18 (33.3)	25 (41)	23 (32.9)
AIS D	76 (41.1)	20 (37)	22 (36.1)	34 (48.6)
Lower Extremity Motor Score (LEMS)[a] [0]					0.70
0 - 17	94 (50.8)	29 (53.7)	34 (55.7)	31 (44.3)
18 - 32	23 (12.4)	7 (13)	7 (11.5)	9 (12.9)
33 - 50	68 (36.8)	18 (33.3)	20 (32.8)	30 (42.9)
Bladder emptying method[a] [3]					0.009
Indwelling	142 (76.8)	44 (81.5)	39 (63.9)	59 (84.3)
AIC	24 (13)	4 (7.4)	16 (26.2)	4 (5.7)
SIC	10 (5.4)	2 (3.7)	3 (4.9)	5 (7.1)
Spontaneous voiding[b]	6 (3.2)	3 (5.6)	2 (3.3)	1 (1.4)
Incontinence[a] [11][c]					0.35
No	81 (60.9)	33 (64.7)	22 (53.7)	26 (63.4)
Yes	18 (13.5)	8 (15.7)	5 (12.2)	5 (12.2)
Unknown	23 (17.3)	8 (15.7)	11 (26.8)	4 (9.8)
Center [0]					0.14
Center 1	46 (24.9)	16 (29.6)	10 (16.4)	20 (28.6)
Center 2	28 (15.1)	9 (16.7)	11 (18)	8 (11.4)
Center 3	23 (12.4)	9 (16.7)	4 (6.6)	10 (14.3)
Center 4	88 (47.6)	20 (37)	36 (59)	32 (45.7)
Discharge within 81 days of SCI [1]					0.16
No	170 (91.9)	46 (85.2)	59 (96.7)	65 (92.9)
Yes	14 (7.6)	7 (13)	2 (3.3)	5 (7.1)

patient status within 40 days of spinal cord injury (SCI)

physiological and/or reflex voiding, not catheterizing

consent to additional data collection needed, n=133

Recruitment

Inclusion Criteria

To evaluate whether the eligibility criteria generate a pool of patients that is sufficiently large to meet recruitment targets TASCI inclusion criteria were applied to the 640 patients from the SwiSCI source database. A total of 185 eligible patients were identified (Supplement 1), indicating that testable inclusion criteria produce a large enough pool of eligible participants for the trial to be feasible. Characteristics of the study population are described in Table 1.

Recruitment Rate Scenarios

In the simulations, the number of participants recruited in three years varied over the different randomization seeds, and across the four different recruitment rate scenarios. The median number of participants recruited ranged from 79 (range: 67–96) under the most conservative scenario (50% recruitment rate, 50% routine data collection eligibility) to 116 (range: 104–136) for the most favorable scenario (66% recruitment rate, 75% routine data collection eligibility). Thus, only the latter scenario met the recruitment target of 114 participants, alerting the team to the possibility that the trial may need to run longer than originally planned.

Intervention Group Balance

Baseline equivalence of the intervention groups in terms of the primary prognostic factor was also investigated. The original cut-off values for the stratification represented population average probabilities of approximately < 33% (LEMS 0-17), 33%-66% (LEMS 18-32), and > 66% (LEMS 33-50) probability of a favorable bladder outcome, respectively (Model 1). To account for the variable distribution of LEMS in the patient population between the study centers (Figure 2B), two additional models of LEMS cut-offs were investigated in the simulations. The additional models had population average probabilities of a favorable bladder outcome of: < 25%, 25%-75% and > 75% (Model 2), or < 20%, 20%-80% and > 80% (Model 3) [18].

Across all recruitment scenarios and LEMS models, in 223/240 runs (92.9%) the overall imbalance in the intervention groups was ≤ 0.05 (e.g. an imbalance of 0.05 = |0.525Verum - 0.475Sham|). The most extreme imbalance, 0.123, occurred in the most conservative recruitment rate scenario. Imbalance in the middle LEMS category was sensitive to variation in cut-offs across all recruitment scenarios (Fig. 2C, most favorable recruitment scenario shown). Imbalance within the centers did not vary substantially between LEMS models (all p ≥ 0.5), but the center with the fewest admissions showed an imbalance of ≥ 0.10 in 37.5% of simulation runs. Based on these results, the trial team re-discussed the choice of LEMS cut-offs in light of the evidenced trade-off between a higher probability of problematic imbalance occurring in LEMS categories (favoring Model 3) and the more precise control for the primary prognostic factor (favoring Model 1), eventually choosing Model 3. The possibility of combining the two centers with the fewest patient admissions was discussed but rejected, as the risk of imbalance did not offset the intrinsic risk of bias from between-center differences.

Patient Admission Flow

Operational challenges could also be anticipated and evaluated using the cohort data. Patient admission flow, including waves of patients and the resulting overlapping intervention periods require dynamic resource planning. Waves of patients were observed across all centers (Figure 3A). In the most favorable recruitment scenario, the median number of participants receiving the intervention per week throughout the entire study period was 6 (maximum: 12) (Figure 3B). The individual centers, in numerical order, had medians of 2 (maximum: 5), 1 (maximum: 4), 1 (maximum: 3), and 2 (maximum: 9), participants receiving the interventions per week (Figure 3 C-F). The above findings informed choices involving the allotment of study personnel and equipment, for example indicating that all centers should have an on-site research assistant. Moreover, TASCI is planning to mobilize equipment and personnel to optimize resource use.

Figure 3

Description of patient flow. 3A – Number of eligible patients in the intervention window per day, in all study centers. 3B-F – In simulations, count of the mean number of weeks with a given number of participants receiving the TTNS intervention, in all four centers (B) and in each individual study center (C-F, centers 1-4, respectively), under the most favorable recruitment rate scenario. N = number; TTNS = transcutaneous tibial nerve stimulation.

Early Discharge

A further operational challenge for TASCI is participant discharge before intervention completion, three months after SCI. Daily home self-administration of TTNS by the participant is not an option due to the risk of allocation unmasking. In the cohort data, 14 patients (7.6%) were discharged before day 81, the early end of the intervention completion window. Two main predictors for early discharge were identified, LEMS and study center (Table 2). To minimize attrition bias, strategies have been developed for covering the remaining weeks in an outpatient setting or via home visits where operationally feasible.

Table 2

Predictors of early discharge from rehabilitation. Results from a logistic regression model, where early discharge (=1), is defined as discharge within 81 days of SCI. C = cervical; CI = confidence interval; LEMS = lower extremity motor score; SCI = spinal cord injury; T = thoracic.

Predictor	Adjusted Odds Ratio(95% CI)	p
LEMS		0.064
0-17	ref
18-32	3.20 (0.14 - 70.95)
33-50	17.08 (1.45 - 201.53)
Center		0.004
Center 1	16.32 (1.85 - 143.64)
Center 2	1.77 (0.08 - 37.41)
Center 3	6.44 (0.34 - 122.48)
Center 4	ref
Age at SCI	0.98 (0.93 - 1.03)	0.36
Gender		0.25
Female	2.27 (0.56 - 9.12)
Male	ref
SCI etiology		0.87
Non-traumatic	1.13 (0.25 - 5.16)
Traumatic	ref
SCI Level		0.86
C1-C4	0.67 (0.11 - 4.28)
C5-C8	1.15 (0.16 - 8.2)
T1-T12	ref
Days from SCIdiagnosis torehabilitationadmission	0.95 (0.88 -1.02)	0.68

Discussion

This case study demonstrated that prospective cohort data can be used in RCT planning to evaluate the feasibility of recruitment targets in relation to eligibility criteria, the performance of the randomization scheme, and inform operational planning regarding human and material resources. Recruitment and resource use are well-established challenges for RCTs [2, 24–27]. Poor recruitment is consistently identified as the primary factor contributing to termination in approximately 40% of prematurely discontinued trials [3-5]. Eligibility criteria should be carefully weighed, as restrictive requirements reduce generalizability and patient availability, while poor targeting increases the sample size necessary to detect an intervention effect and thereby the risk of incorrectly rejecting a beneficial intervention [2, 26, 28]. Notably, eligibility criteria have been identified as one of the major factors affecting RCT recruitment in the context of SCI [29]. Previous studies have reported using observational data in RCT planning in intensive care units and the pediatric acute care setting to define primary outcomes, inform the sample size calculation, and provide supplemental information on current clinical practices and equipoise regarding the trial intervention [10, 12]. In our case, the primary outcome and sample size calculations were derived from a combination of previous experience and the literature. To the knowledge of the authors, cohort-based RCT planning such as in the present study, is novel to the field of SCI.

The conceptual framework and methodological approach to the prospective planning of an RCT presented here is conditionally applicable to other settings and research questions. A critical requirement is that the projected size and constitution of the patient population over the anticipated trial period is accurately reflected in the available cohort data. If this condition is fulfilled, such as tentatively indicated in the present study by the limited between-year variation in patient numbers and characteristics, planning is likely superior to that of the alternative approach of using external data, because inference based on other patient populations may be limited due to generalizability concerns. In particular, different clinical management approaches, including timing of admission and discharge, are relevant to RCT planning in the context of specialized rehabilitation for SCI. Furthermore, data from a prospective cohort can only be used if relevant information has been collected, with an acceptable level of detail, at relevant time points. Often cohort studies capture a broad selection of data from a larger portion of patients, while RCTs typically capture very specific information in a small subset of the patient population [30]. Accordingly, and in part due to lack of urodynamic investigation data, in our case study we were not able to fully investigate the impact of the TASCI exclusion criteria on the available patient population, so we cannot exclude the possibility that we have overestimated the number of available patients. Moreover, any association between the exclusion criteria and either of the stratification factors (study center or LEMS) could decrease the reliability of the imbalance analysis. In our view, the use of LUTS reflects the best possible option to approximate the trial inclusion process (screening and baseline exams) in our simulations. Also, in the case that a high comorbidity load, implying an increased burden of care, is related to trial eligibility, this may reduce the reliability of the imbalance analysis. Furthermore, SwiSCI does not collect urodynamic data, and records with missing data were excluded, introducing uncertainty into the targeting of eligible patients.

Key feedback to SwiSCI and other future prospective cohort studies includes the establishment of accommodating data models, which combine permanent basic data elements of general relevance (e.g., demographics; ISNCSCI assessment) with temporary data elements reflecting contemporary research priorities, such as the implementation of a clinical trial. In the case of TASCI, the collection of urodynamic data reflects such a priority data element. In addition, trial planning might benefit from staffing data for the centers to allow for a more precise identification of periods where supplementary resources might be needed from the trial team. Collaboration between cohorts and RCT planners should be initiated as early as possible to ensure that the relevant information is being collected, and thereby to maximize the benefits for trial planning.

The approach to trial design outlined here is intended to supplement piloting, but it cannot serve as a full replacement. A review of recruitment in RCTs in SCI identified a range of factors that affect the choice to participate in a specific trial such as the intervention under investigation, presence of competing studies, and the burden the study intervention places on the participants [29]. This implies that assumptions about recruitment rates and intervention windows are best investigated through piloting in the local patient population with the local investigators and resources [31]. However, confirming previous reports [32], we have observed that one of the most valuable aspects of these analyses has been the clarification of the assumptions, expectations and limitations underlying the trial.

Conclusions

Here we demonstrate how thorough evaluation of the targeted patient population, as well as alternative trial scenarios using cohort data, supports informed decision-making and trial design. Advantages to this approach include the optimization of resource use and increased chances that the trial will produce decisive evidence regarding treatment efficacy. From an ethical perspective, thoughtful trial planning decreases the chances of subjecting patients to unnecessary burden or risk when there is little chance of gaining beneficial knowledge [33]. However, the reality-check offered by piloting is still an essential part of the implementation of the RCT, as assumptions can be further investigated and adjustments can be made [6, 7, 31]. The high level evidence produced by well-designed RCTs provides a foundation for high-quality, evidence-based patient care. Prospective cohort data holds very valuable information, and comprehensive utilization of this data could improve trial planning in the field of spinal cord injury.