Managing work flow in high enrolling trials: The development and implementation of a sampling strategy in the PREPARE trial.

INTRODUCTION: Pragmatic trials in comparative effectiveness research assess the effects of different treatment, therapeutic, or healthcare options in clinical practice. They are characterized by broad eligibility criteria and large sample sizes, which can lead to an unmanageable number of participants, increasing the risk of bias and affecting the integrity of the trial. We describe the development of a sampling strategy tool and its use in the PREPARE trial to circumvent the challenge of unmanageable work flow.
METHODS: Given the broad eligibility criteria and high fracture volume at participating clinical sites in the PREPARE trial, a pragmatic sampling strategy was needed. Using data from PREPARE, descriptive statistics were used to describe the use of the sampling strategy across clinical sites. A Chi-square test was performed to explore whether use of the sampling strategy was associated with a reduction in the number of missed eligible patients.
RESULTS: 7 of 20 clinical sites (35%) elected to adopt a sampling strategy. There were 1539 patients excluded due to the use of the sampling strategy, which represents 30% of all excluded patients and 20% of all patients screened for participation. Use of the sampling strategy was associated with lower odds of missed eligible patients (297/4545 (6.5%) versus 341/3200 (10.7%) p < 0.001).
CONCLUSIONS: Implementing a sampling strategy in the PREPARE trial has helped to limit the number of missed eligible patients. This sampling strategy represents a simple, easy to use tool for managing work flow at clinical sites and maintaining the integrity of a large trial.

Introduction

Pragmatic trials in comparative effectiveness research assess different treatment options in routine clinical practice. They are characterized by broad eligibility criteria, which allows for variation between participants and permits evaluation of how a treatment works in a real world setting. This is in contrast to explanatory trials, which are conducted under ideal settings and enroll participants who have well defined characteristics for the clinical condition of interests [1,2]. In explanatory trials, researchers are often challenged by slow enrollment and a paucity of eligible patients. In contrast, researchers conducting pragmatic trials may have unmanageable volumes of eligible patients and enrolled participants as a result of broad eligibility criteria.

High volumes of potentially eligible patients may lead to unmanageable work flow. This can increase the risk of bias and decrease the integrity of the trial. Potential risks of high patient volumes include the enrollment of non-consecutive patients (missed patients), poor data quality, delayed data entry, missed follow-up visits, and loss to follow-up. Additionally, high patient volumes may contribute to personnel burnout. To maintain the integrity of pragmatic trials with high patient volumes, researchers should consider using a sampling strategy implemented in a non-biased manner. Researchers need to ensure that it does not reduce the integrity and generalizability of the trial [3].

Most clinical trials use convenience sampling, in which researchers screen and enroll participants who are most readily accessible to them (e.g., patients presenting to the participating clinic for treatment) [3]. When using convenience sampling, it is critical that consecutive patients be screened and enrolled to ensure that the trial participants are representative of the overall population. However, in high volume pragmatic trials, clinical sites may enroll patients only during pre-specified periods (e.g., patients who present on weekdays during daytime hours). This method is fraught with potential bias as patients who present on different days may be different [3]. For example, patients who present on weekends (and are excluded) may be sicker than patients who present during the week (and are included). Systematic sampling (e.g., enrolling every fourth patient) is another simple sampling strategy [4]; however, it is associated with a risk of manipulation or error, potentially introducing bias. The gold standard is random sampling, in which eligible patients have an equal chance of being selected to participate [5]. Few resources are available to guide researchers who wish to implement random sampling in trials of this nature.

To circumvent this challenge in the PREPARE (A Pragmatic Randomized trial Evaluating Pre-operative Alcohol skin solutions in FRactured Extremities) trial, study personnel developed a novel sampling strategy tool based on the random sampling framework. We describe the sampling strategy tool, its implementation, and preliminary data on its use. Specifically, the objectives are: 1) Describe the number of clinical sites initially elected to use a sampling strategy; 2) Describe changes in the sampling strategy used over time across clinical sites; 3) Determine the number of participants excluded due to the use of a sampling strategy; and 4) Determine if the use of a sampling strategy was associated with fewer missed patients.

Methods

The PREPARE trial

The PREPARE trial (A Pragmatic Randomized trial Evaluating Pre-operative Alcohol skin solutions in FRactured Extremities) is a pragmatic cluster randomized crossover trial that compares alcohol-based pre-operative antiseptic skin preparation with iodine povacrylex (0.7% free iodine) (DuraPrep™) versus 2% chlorhexidine gluconate (ChloraPrep™). Each cluster (clinical site) is initially randomized to use one of the two pre-operative alcohol surgical skin preparation solutions for fracture surgeries. Each cluster starts with the initially allocated study solution for a period of 2 months and subsequently crosses over to the other solution for their second recruitment period. This process repeats every 2 months for the duration of the 24-month enrollment period. The PREPARE trial will enroll at least 1540 patients with open fractures and 6280 patients with closed lower extremity and pelvic fractures across 20 clusters in North America. The primary outcome is surgical site infection within 90 days of the fracture, which includes superficial incisional surgical site infections within 30 days and deep incisional or organ/space surgical site infections within 90 days of fracture surgery, as defined by the Centers for Disease Control and Prevention (CDC) [6] and unplanned fracture-related reoperations within 12 months to manage infection, wound healing problems, and fracture healing problems. The open fracture cohort and the closed fracture cohort will be analyzed separately as they are two distinct patient populations. PREPARE is registered on clinicaltrials.gov (NCT03523962) and the master protocol has been published [7].

Unmanageable fracture volume

The clinical sites participating in the PREPARE trial are mostly level one trauma centers that treat a high volume of open fractures and closed fracture [8]. For example, the lead clinical site in the PREPARE trial, R Adams Shock Trauma Center, operatively treats approximately 400 open fracture patients and 1600 closed fracture patients each year. Given the pragmatic nature of the trial, the majority of these patients will meet the eligibility criteria for PREPARE. Additionally, due to the minimal efforts required from study participants, eligible patients tend to provide informed consent for participation [7]. Given the potential consequences associated with a high volume of eligible patients a pragmatic sampling strategy was needed to ensure that clinical sites could maintain acceptable work flow.

PREPARE sampling strategy

The sampling strategy used in PREPARE follows a random sampling framework and was built into the trial's REDCap Cloud (RCC) electronic data capture (EDC) system [7]. The sampling strategy randomly determines if each eligible patient should be approached for consent (included) or not approached for consent (excluded). It includes various sampling ratios of patients included to patients excluded (i.e., 1:3, 1:2, 1:1, 2:1, 3:1) for both the open fracture cohort and the closed fracture cohort. The sampling strategies are stratified by clinical site as well as cohort (open fracture cohort versus closed fracture cohort). The sampling strategies may differ between the open fracture cohort and the closed fracture cohort since the number of eligible patients is inherently different. Additionally, the ratios can be adjusted over the course of the trial. During a one-month run-in phase immediately prior to the trial initiation, each clinical site tracked open fracture and closed fracture volumes. This helps to inform the selection of their initial sampling ratio. As the trial progresses, clinical sites may choose to increase or decrease their sampling ratio depending on the number of patients enrolled, study personnel assigned to the trial, and workflow. Changes to the sampling strategy are made upon request of the clinical site in consultation with the Methods Centre and are approved by the Methods Centre Principal Investigator [7]. As a general rule, sampling strategies should only be changed every four months (e.g., after each enrollment period). However, following the pragmatic nature of the trial and to ensure high quality data, the sampling strategy may be adjusted within the four month enrollment period if work flow becomes unmanageable at a clinical site.

Fig. 1 shows the patient flow through the sampling strategy.

Fig. 1

Patient flow through the sampling strategy.

To use the sampling strategy, clinical site research personnel screen all patients with fractures that require operative management as per the PREPARE protocol [7]. If a patient meets all of the eligibility criteria the study personnel logs into RCC to access the sampling strategy (Fig. 2). Specifically, they select sampling strategy link from the home screen in the RCC system, creates a randomization form, and randomizes the potential participant to either “Approach for consent” or “Exclude” due to the sampling strategy. If the potential participant is randomized to approach for consent, the study personnel proceed with the consent process. If the potential participant is randomized to exclude due to sampling strategy, a screening form is completed indicating that the participant is “excluded due to sampling strategy”.

Fig. 2

Sampling strategy in RCC

Data analysis

All analyses were conducted on participants screened between the date of trial initiation on September 24, 2018 and January 15, 2020 (date of analysis). The initial and ongoing use of the sampling strategies are described descriptively. The number of participants excluded due to the use of a sampling strategy are also described descriptively in the closed fracture cohort. A chi-squared test was used to determine if the use of a sampling strategy was associated with fewer missed eligible participants in the closed fracture cohort. All analyses were conducted using SPSS, v21.

Results

The sampling strategy has been primarily implemented in the closed fracture cohort of PREPARE. 7/20 (35%) of the clinical sites elected to use a sampling strategy in the closed fracture cohort at the time of their site's initiation (Table 1). Clinical sites who implemented a sampling strategy were high volume centers that expressed concern over having adequate research personnel to screen and enroll consecutive closed fracture patients and manage data collection and participant follow-up. A number of sampling ratios (patients included: patients excluded) have been used by clinical sites (Table 1).

Table 1

Implementation of sampling strategies in PREPARE over time.

Clinical Site	PREPARE Cohort (Open Fracture vs. Closed Fracture)	Initial Sampling Strategy (Included: Excluded)	Changes to Sampling Strategy
1	Closed fracture cohort	1:1	2:1 at 2 months1: 3 at 14 months
	Open fracture cohort	None	Added 1:1 at 14 months
2	Closed fracture cohort	None	Added 3:1 at 4 monthsRemoved sampling strategy at 10 months
3	Closed fracture cohort	1:3	Removed sampling strategy at 8 months
4	Closed fracture cohort	2:1	1:3 at 10 months
5	Closed fracture cohort	1:3	1:2 at 8 months
6	Closed fracture cohort	3:1	No changes
7	Closed fracture cohort	2:1	No changes
8	Closed fracture cohort	1:1	No changes

No clinical sites initially adopted a sampling strategy for the open fracture cohort, as the volume of open fractures is lower than closed fractures and clinical sites deemed the work flow to be manageable. One clinical site implemented a sampling strategy in the open fracture cohort after 14 months of enrollment as the volume of work became unmanageable and data quality and follow-up were suffering (Table 2).

Table 2

Patients screened, included, and excluded due to the sampling strategy by cluster and overall in the closed fracture cohort.

Clinical Site	PREPARE Cohort (Open Fracture vs. Closed Fracture)	Number of Patients Screened by Clinical Site Using a Sampling Strategy	Excluded Due to the Sampling Strategy	Included Due to the Sampling Strategy
1	Closed fracture cohort	773	324	449
1	Open fracture cohort	46	30	16
2	Closed fracture cohort	78	26	52
3	Closed fracture cohort	60	30	30
4	Closed fracture cohort	261	65	196
5	Closed fracture cohort	730	183	547
6	Closed fracture cohort	358	268	90
7	Closed fracture cohort	487	354	133
8	Closed fracture cohort	656	289	367
Total		3403	1539	1864

In the closed fracture cohort, 2 clinical sites have changed their sampling strategy ratio (e.g., 1:1 to 1:2) so that more patients are excluded due to sampling, 2 clinical sites have changed their sampling strategy ratio (e.g., 1:2 to 1:1) so that fewer patients are excluded due to sampling, and 2 clinical sites have removed their sampling strategy (Table 1). An eighth clinical site elected to use a sampling strategy after 4 months of enrollment.

To date there have been 7745 patients screened across all clinical sites in the closed fracture cohort. The sampling strategy has been used on 3403 potentially eligible patients, and of these, 1864 patients (54.8%) were randomized to approach for consent and 1539 patients (45.2%) were randomized to not approach for consent (Table 2). This represents 30% of all excluded patients in the PREPARE closed fracture cohort and 20% of all patients screened for participation in the PREPARE closed fracture cohort across all clinical sites.

Use of the sampling strategy in the PREPARE closed fracture cohort was associated with lower odds of missed eligible patients (297/4545 (6.5%) versus 341/3200 (10.7%)) when a sampling strategy was not in use (p < 0.001).

Discussion

We have successfully developed and implemented a sampling strategy for the PREPARE trial which is a large, pragmatic cluster randomized crossover trial with a high volume of eligible fracture patients. The sampling strategy is based on the random sampling framework which is considered the gold standard when consecutive sampling is not feasible [9]. Random sampling methodologies are commonly used in epidemiological and population surveys; however, the strategies used in these settings are more challenging to implement in the clinical trial setting. For example, when conducting a follow-up survey in a sample of individuals who received a new public health intervention, it is possible to identify all eligible individuals at the onset of the research study and then select a random sample to survey. However, this approach cannot be applied in a clinical trial setting in which patients present one at a time to the clinical site and a decision must be made at that time regarding their inclusion in the trial. In the absence of established guidelines for implementing random sampling, clinical trials may elect to use convenience sampling or systematic sampling when consecutive sampling is not feasible [3]. Prior to developing the sampling strategy for PREPARE, our team was unable to find detailed descriptions of random sampling strategies similar to the one we developed for PREPARE being used in large, high-volume trials. Consequently, to the best of our knowledge, this represents a unique approach.

The sampling strategy was built into the PREPARE trial EDC system for ease of use by clinical sites. This allows clinical sites to use a single system for sampling and for subsequent data collection. RCC's randomization function is ideally suited to this application and minimal programming was needed to implement the sampling strategies. The RCC system also allows for transparent data collection as well as easy tracking, analysis, and reporting of this information. Finally, it allows all trial data to be stored within a single system, which is ideal for future analyses and archiving.

One of the advantages of using RCC's randomization function for this purpose is the flexibility of being able to easily adjust the sampling ratios throughout the enrollment period. Ideally, clinical sites should only make changes at the end of a four-month treatment period to ensure treatment group balance. However, to remain pragmatic and provide optimal flexibility, we have allowed clinical sites to make changes within a treatment period. Given the large sample size we do not believe this will result in an overall treatment group imbalance at the conclusion of the trial. The ability to adjust sampling ratios allows individual clinical sites to balance enrollment with workflow and patient volumes on an ongoing basis. As demonstrated in this study, the PREPARE sites have used this flexibility to increase or decrease sampling ratios, as well as to discontinue or implement sampling strategies during the course of the trial. We anticipate that clinical sites will continue to make these adjustments as the trial progresses and more fracture patients are enrolled.

There are multiple benefits to using a sampling strategy in high enrolling trials. Firstly, use of a sampling strategy by PREPARE clinical sites was associated with a reduction in the number of missed patients during screening and enrollment. This in turn equates to a reduction in the risk of bias, as patients are randomly excluded from the trial based on using the gold standard sampling strategy, as opposed to being systematically missed. Selection bias is also reduced by randomly excluding some participants that would otherwise be considered “ideal candidates” for enrollment. In the absence of a sampling strategy, clinical site personnel may selectively approach these ideal candidates, leading to their over-representation among study participants. Another advantage to this system is that it facilitates participation of clinical sites whose research capacity cannot keep up with their high volumes of fracture patients, which is a common issue at level I trauma centers participating in pragmatic trials. Additionally, the use of a sampling strategy may improve data quality and follow-up rates through more manageable workloads. Finally, anecdotally, the use of a sampling strategy may improve research personnel job satisfaction and reduce the risk of burnout by maintaining a manageable workflow.

Despite the multiple advantages, there are several limitations associated with including a sampling strategy. Firstly, it adds an extra step to the recruitment process, which requires a couple of extra minutes per patient. There is also the need for adequate training of clinical site personnel to ensure that the sampling strategy is being implemented consistently and correctly. Additionally, there is the potential to create imbalances if not implemented properly (e.g., trials with only one treatment crossover, changes to sampling strategy within treatment periods). In PREPARE, ideally the sampling strategy should only be changed after every four months (after the completion of two treatment periods) to maintain balance between the two treatment groups. There have been occasions on which the sampling ratio has been changed mid treatment period to address immediate challenges with workflow. Another limitation is that using a sampling strategy decreases enrollment, which in turn has feasibility and fiscal consequences. To date, 1539 patients have been excluded due to the use of the sampling strategy. Had these patients been enrolled, the PREPARE trial would be five months closer to meeting the enrollment target.

While the sampling strategy is working well in a very pragmatic trial with broad eligibility criteria and minimal burden to participants, it may be more challenging to implement in standard pragmatic RCTs in which potential participants are more likely to decline to participate or in which they may screen fail at a later point in time. Writing a protocol which allows for the flexibility in the use of a sampling strategy and allowing for different sampling strategy ratios and timelines, can help to address this issue [7].

There are several limitations to the current analysis. The first limitation is that the sampling strategy has only been implemented at 8 of 20 PREPARE clinical sites and we are still in the early stages of the trial. Secondly, we do not collect any data on potential participants that are excluded, and therefore we are unable to determine if there are any differences in characteristics between the potential participants that are approached for consent compared to those who are excluded due to the sampling strategy. We have also not explored the impact of the sampling strategy on rates of follow-up since the trial is still in its early stages. Additionally, we did not report on data quality (e.g., number of queries), as it changes rapidly and can be challenging to quantify while the trial is ongoing. However, in the interest of early knowledge dissemination, we feel there is merit in presenting these early results.

Conclusion

Implementing a sampling strategy in the PREPARE trial has helped to enable high volume clinical sites to achieve a balance between enrollment and workflow, which may in turn lead to higher data quality and less burnout for research team members. We anticipate that clinical sites will continue to implement, remove, and adjust their sampling strategy ratios as the trial progresses. This sampling strategy represents a simple, easy to use tool for managing work flow at clinical sites and maintaining the integrity of a large trial.

Funding sources

The PREPARE trial is funded by the () (PCS-1609-36512) and the () (Foundation Grant); the Aqueous-PREP trial is funded by the (W81XWH-17-1-070) and the (Foundation Grant). Surgical Associates funded start-up activities at the Methods Centre and The Physician Services Incorporated provided funding to the Methods Centre and for the Aqueous-PREP trial. The views in this publication are solely the responsibility of the authors and do not necessarily represent the views of the PCORI, its board of governors or methodology committee.