Embedded point of care randomisation for evaluating comparative effectiveness questions: PROSPECTOR-critical care feasibility study protocol.

INTRODUCTION: Many routinely administered treatments lack evidence as to their effectiveness. When treatments lack evidence, patients receive varying care based on the preferences of clinicians. Standard randomised controlled trials are unsuited to comparisons of different routine treatment strategies, and there remains little economic incentive for change.Integrating clinical trial infrastructure into electronic health record systems offers the potential for routine treatment comparisons at scale, through reduced trial costs. To date, embedded trials have automated data collection, participant identification and eligibility screening, but randomisation and consent remain manual and therefore costly tasks.This study will investigate the feasibility of using computer prompts to allow flexible randomisation at the point of clinical decision making. It will compare the effectiveness of two prompt designs through the lens of a candidate research question-comparing liberal or restrictive magnesium supplementation practices for critical care patients. It will also explore the acceptability of two consent models for conducting comparative effectiveness research. METHODS AND ANALYSIS: We will conduct a single centre, mixed-methods feasibility study, aiming to recruit 50 patients undergoing elective surgery requiring postoperative critical care admission. Participants will be randomised to either 'Nudge' or 'Preference' designs of electronic point-of-care randomisation prompt, and liberal or restrictive magnesium supplementation.We will judge feasibility through a combination of study outcomes. The primary outcome will be the proportion of prompts displayed resulting in successful randomisation events (compliance with the allocated magnesium strategy). Secondary outcomes will evaluate the acceptability of both prompt designs to clinicians and ascertain the acceptability of pre-emptive and opt-out consent models to patients. ETHICS AND DISSEMINATION: This study was approved by Riverside Research Ethics Committee (Ref: 21/LO/0785) and will be published on completion. TRIAL REGISTRATION NUMBER: NCT05149820. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Randomised trials integrated into clinical workflows have shown promise but require further feasibility testing to determine acceptability to patients and clinicians.

A mixed-methods approach allows combination of quantitative outcomes with explanatory qualitative data, increasing understanding of reasons underpinning success or failure of the intervention.

Testing study feasibility allows estimation of randomisation compliance, judges acceptability of the candidate research question, and allows optimisation of electronic prompt design prior to embarking on an adequately powered main trial.

As the study examines variation at an individual patient and clinician level, it is unclear how generalisable future study results will be outside the study centre.

Introduction

Every day, clinicians collectively make hundreds of thousands of decisions regarding the application of treatments and interventions in the care of patients. While some of these treatments will be guided by robust evidence from randomised controlled trials (RCTs), many ‘routine’ aspects of clinical care continue to lack a strong evidence base.1 Braithwaite et al describe this as the ‘60–30–10’ challenge—approximately 60% of administered treatments conform to evidence, 30% may be wasted or ineffective and 10% result in harm.2

When evidence for an intervention is absent, clinicians vary in their decision making according to their experience and preferences.3 This variation is manifestly observable and can be seen across multiple domains from choice of surgical procedure,4 5 management of heart failure or diabetic ketoacidosis6 7 or administration of antibiotics and intravenous fluids.8 9

Another commonly used treatment which varies in practice is the administration of supplemental magnesium for the prophylaxis of atrial fibrillation in critical care patients. While this practice is commonplace, the only evidence as to its effectiveness comes from the cardiac surgery population.10 Over time, this has been extrapolated to all critical care patients, without additional evidence of benefit. As such, clinicians vary in their threshold for routinely supplementing magnesium. Clinician behaviour will be consistent at extremes of serum magnesium measurements (never/always supplement), but within a ‘normal’ range the decision to supplement will have a random component linked to the clinician’s preference.11

Variation in practice does not necessarily imply substandard care—it may be that the clinician’s experience offers benefits in optimising treatment delivery, or it may be that there is no meaningful difference between treatment choices. Under ideal conditions, clinicians would be able to learn from variation and improve the quality and coverage of evidence for future patients. Ineffective yet costly treatments could be minimised, and strategies demonstrating effectiveness targeted to ever smaller subgroups of patients.

Unfortunately, generating new evidence from routine clinical decision making has proven difficult using existing research methodologies. RCTs, whilst well suited to demonstrating treatment efficacy in homogenised cohorts, under rigid treatment protocols, have proven costly and difficult to conduct in more pragmatic settings.12 While the classical RCT remains ideal for evaluating novel therapies, for treatments already in widespread use, with likely small effect sizes, the expense of conducting comparative effectiveness trials becomes untenable. In most cases, researchers rely on observational methods, which lack the validity derived from prospective randomisation.13 Therefore, to properly evaluate the comparative effectiveness of multiple treatment strategies, an element of randomisation is essential, together with a mechanism to deploy this efficiently.14

Electronic health record systems (EHRSs) offer a potential solution. Increasingly widespread and comprehensive, they have renewed interest in integrating clinical trials into routine care.15 While embedding trial infrastructure has improved efficiency, the requirement for point-of-care consent and randomisation remains. Predominantly, this continues to be delivered by a research nurse, partnered with the treating clinician, a process which remains time intensive and financially costly.16

Two barriers to implementing routine comparative effectiveness research standout—(1) how to fully integrate randomisation into EHRSs, ensuring that patient safety and the scientific integrity of the study is maintained; and, (2) what is the correct way for patients to consent to the randomised delivery of routine treatments? Central to these issues is the principle of clinical equipoise—the idea that without evidence, every clinical decision comes with a degree of uncertainty. When the benefits and risks of the treatment are balanced (or unknown), then it becomes justifiable to randomise, in order to learn what decision is best.17 18

Flexible electronic point-of-care randomisation

To learn effectively from clinical decisions, a rapid and responsive randomisation mechanism is required. To achieve this, we propose a two-stage innovation: (1) to embed the randomisation process into the EHRS and link randomisation to the moment of clinical decision making and (2) to make that randomisation optional for the clinician. The first step ensures that the prompt to randomise is presented to the clinician at the point of potential equipoise, ensuring relevance and minimising disruption to normal care processes. The second step means that the clinician–patient dyad only access the randomisation process if they share equipoise with the trial.

Our design builds on Vickers and Scardino’s concept of the clinically integrated randomised trial, as well as work by Fiore and colleagues in point-of-care trial design.19 20 To this we add concepts from preference trials, which are designed to explicitly acknowledge treatment preferences to minimise bias.21 While most of these trials target patient preferences, we believe that the concepts are equally applicable to clinicians.

A preference approach has the advantage of allowing clinicians to follow their preferred course of action when they feel strongly, while simultaneously allowing randomisation under conditions of equipoise. In this manner, the clinician retains overall responsibility and control over the patient’s treatment—ensuring safety is maintained. This is key for integrated trials where, by definition, oversight from research teams is minimised.

We propose to modify existing functionality within the EHRS to intercede at the point of clinical decision making. Many EHRSs use clinical decision support systems (CDSSs), based on series of logical rules, to deliver information to clinicians under pre-defined circumstances. These logical rules may be used to emulate inclusion and exclusion criteria within a trial. Once designated conditions are met, an electronic prompt can be displayed to the clinician, at the point of clinical decision making to highlight both the opportunity to randomise and the predetermined treatment group allocation.

Our design of electronic point-of-care randomisation (ePOCR) prompt will invite the clinician to consider whether they have equipoise for the treatment decision. In this way, the prompt simply externalises and makes explicit the normal decision-making process.

If the clinicians have equipoise, the ePOCR prompt allows them to view the randomised allocation, which can then be followed, and the patient contributes data to the randomised arm of the study. However, if the clinician lacks equipoise, they remain free to follow their preference. In a classical RCT, declining to follow randomisation may represent a protocol violation and result in the participant being excluded from the final analysis. However, in a preference approach, the participant continues to contribute data into the parallel observational study arms determined by the clinician’s preference.

Where the clinician declines randomisation, the parallel observational arm of the study continuously evaluates external validity and can identify previously unrecognised subgroups where clinicians have strong preferences that may require modification of the trial. In addition, where preferences are known, these observational arms may be used to identify preference and selection effects, adding extra information to that gained from the treatment effect estimation in the randomised arm.22 This flexible approach to delivering randomisation is depicted in figure 1.

Figure 1

Flexible randomisation as an expression of clinical equipoise. ePOCR, electronic point-of-care randomisation; RCT, randomised controlled trial.

To integrate randomisation into clinical workflows requires understanding of how clinicians interact with EHRS and how data are used to make decisions. While the use of interruptive prompts based on modified CDSS is an attractive method for accomplishing this, the possible disruption to care processes must be considered. The concept of alert fatigue in this setting is well documented.23 As such, ePOCR prompts must be designed to be minimally disruptive, while permitting the data collection required by the study. To this end, our feasibility study will compare two designs of interruptive prompt, a simple ‘Nudge’ design, and a more complex ‘Preference’ design. The Nudge prompt encapsulates the simplest version of the study design, while the Preference design allows the collection of additional treatment preference data for use in the observational study arm.

Pre-emptive and opt-out consent

There is ongoing debate as to the most appropriate consent mechanisms for facilitating comparative effectiveness research, specifically, for treatments with demonstrable variation already present in their routine use. Faden and colleagues highlight the strong ethical arguments in favour of streamlining consent procedures in this area and the acceptability to stakeholders of the same.24–26

In this study, we will investigate moving the point at which consent is obtained proximally, away from the final application of eligibility criteria and randomisation. A future model might see patients routinely consented for a range of potential trials (under a specific operational framework such as that suggested by Fiore and Lavori14) on admission to hospital, before it is known whether or not they will be eligible. This single point of contact would decrease the burden of identifying and consenting patients and minimise disruption to clinical workflows.

Study objectives

The overall study aim is to ascertain the feasibility of conducting a future clinical trial using infrastructure integrated into the EHRS and using a system of ePOCR. Feasibility will be judged by combining outcome data related to (1) the effectiveness of the ePOCR system and (2) the acceptability of ePOCR to clinicians. Since the feasibility of scaling future large scale trials using ePOCR and preference design approaches is reliant on a streamlined consent model, we will also evaluate the acceptability of both pre-emptive and opt-out consent models to patients. Finally, we will collect pilot data specific to the candidate research question of magnesium supplementation to inform design of a future trial.

Methods and analysis

Study design and setting

This single centre, mixed-methods feasibility study will follow an explanatory-sequential design, which allows supplementation of quantitative data on the effectiveness of ePOCR with qualitative data to aid interpretation.27 The study will run across four critical care units within University College London Hospitals (UCLH) NHS Trust from January to August 2022. These critical care units care for a mix of surgical patients including colorectal, urology and thoracics but excluding cardiac and neurosurgery. UCLH has used the Epic EHRS since 2018.

We will recruit patients aged 18 years and over, undergoing elective surgery of sufficient complexity to warrant postoperative admission to critical care. This cohort was selected opportunistically to facilitate obtaining informed consent pre-emptively during hospital visits prior to surgery. Potentially eligible participants will be identified through a combination of algorithmic screening of the EHRS by surgical procedure code, and by manual identification from booked critical care admissions.

We will recruit a cohort of critical care clinicians to undertake the qualitative interview programme. The intervention is targeted to bedside critical care nurses. There are approximately 300 critical care nurses working across all the study sites. Neither clinicians nor patients will be compensated for participating in the study.

Exclusion criteria will be applied at two stages. Patients unable to provide written informed consent, or who are pregnant will be excluded. Following postoperative admission to critical care, patients whose initial documented heart rhythm is atrial fibrillation will be excluded. Prior to the deployment of the ePOCR prompt, the EHRS will screen against the following criteria: (1) no documented allergy or intolerance to any preparation of supplemental magnesium, (2) no active treatment for bronchospasm (defined as active treatment administration indicating bronchospasm and screening of active problem list) and (3) the most recent serum magnesium result prior to prompt deployment lies between 0.5 and 1.5 mmol/L. This final criterion ensures that the prompt does not facilitate randomisation for magnesium values outside the scope of reasonable clinical equipoise. For example, serum magnesium values <0.5 mmol/L would normally always be supplemented, and vice versa for values >1.5 mmol/L.

On successful conclusion of the screening process, the ePOCR prompt will display to the bedside critical care nurse. The screening process repeats for each new serum magnesium result received. Screening and overall participant flow through the study are illustrated in figure 2.

Figure 2

Anticipated participant flow through study. EHRS, electronic health record system; ePOCR, electronic point-of-care randomisation.

Qualitative assessments will be conducted in three stages. A random sample of all critical care clinicians involved in routinely caring for this patient cohort will be invited to undertake an initial interview. Two further interviews will focus specifically on the bedside critical care nurses exposed to the ePOCR prompts.

Patient and public involvement

We sought opportunities to engage with patients and the public from study inception. To this end, two focus groups were conducted. The first addressed utilisation of electronic clinical data for research and the presence of naturally occurring variation in practice for evidence-light treatments. The second focused on the premise of flexible ePOCR and the need to investigate alternative consent models for comparative effectiveness research. These groups highlighted a general lack of awareness regarding evidence gaps for routine treatments. Both groups agreed that this is a priority area for future research. The authors are grateful to members of both groups for their feedback in improving the clarity of communication regarding a complex study design. We plan to disseminate study results to consenting participants on completion.

Interventions

Electronic point-of-care randomisation prompts

This study will compare two ePOCR prompts—Nudge versus Preference designs, illustrated in figure 3. The Nudge design is characterised by its passive nature and requires minimal interaction from the clinician. The intention is to ‘nudge’ the clinician to consider their level of equipoise for the decision to supplement magnesium and follow the randomised treatment where they have no preference. In contrast, the Preference design facilitates the explicit recording of the clinician’s treatment preference, while simultaneously allowing randomisation under conditions of equipoise. Preference options are presented as three possible choices—a strong preference for or against administering supplemental magnesium, and no preference. If no preference is selected, the randomised action is provided. If the clinician selects a strong preference, they are advised to continue with their preferred treatment. While this design is more burdensome because it requires interaction, it will allow the derivation of preference and selection effects as described above.

Figure 3

Examples of Nudge (top) and Preference (bottom) ePOCR prompt designs. ePOCR, electronic point-of-care randomisation.

Both prompt designs will be constructed using the Epic build module designed for ‘Best Practice Advisory’ creation, essentially a form of clinical decision support. Construction of a system of logical rules will allow screening of eligibility criteria as described. The technical aspects of both prompt designs will be tested in a sandbox environment prior to live deployment.

Deployment follows the same pathway for both ePOCR prompt designs. Following recruitment, participants will be randomised to either Nudge or Preference design. They will then be randomised again to either liberal or restrictive magnesium supplementation strategies (figure 4). The liberal magnesium arm will encourage supplementation at a serum magnesium value <1.0 mmol/L. The restrictive arm will encourage supplementation at a serum magnesium value <0.75 mmol/L. These values were determined from an observational study of supplementation practices at the study centre and fall within the boundaries of observed variation in practice.11

Figure 4

Two stage randomisation process

Randomisation will be conducted using the EHRS, which conducts simple randomisation using an internal number rule.28 For this feasibility study, basic randomisation without additional covariate balancing will be used. Randomisation will remain the same for both prompt design and magnesium strategy throughout study participation.

Both prompts will display to the bedside critical care nurse under either of two conditions: (1) accessing of the participant’s blood test results or (2) accessing the supplemental magnesium prescription within the EHRS. The prompt will deploy once for each new serum magnesium result, for five postoperative days or the end of the participant’s critical care admission, whichever is sooner.

This study has been designed to be highly pragmatic. At our institution, it is normal practice for all patients admitted to critical care to be issued with an ‘as required’ prescription for either intravenous or oral magnesium. In both study arms, the method and frequency of magnesium supplementation, and frequency of serum magnesium measurement remain at the discretion of the clinical team.

Qualitative assessments

Both clinician and patient interviews follow a semistructured design. Prior to ePOCR deployment, critical care clinicians will be invited to undertake an interview exploring general attitudes towards EHRS research and their current interactions with existing electronic alerts. The interview will feature guided simulation introducing both prompt designs and encouraging initial feedback. The use of simulation to introduce the prompts acknowledges the logistical difficulty in ensuring that each critical care nurse participating in the study is exposed to each prompt design at least once during the study period.

Critical care nurses will undertake a further interview following exposure to an ePOCR prompt to gather immediate feedback. A final follow-up interview will invite nurses to give a preference on prompt design, having experienced the intervention in a clinical setting.

Patients participating in the study will undertake a semistructured interview following discharge from critical care. This will explore attitudes towards pre-emptive and opt-out consent models. Interview schedules are included in online supplemental material S1.

Outcome, data collection and analysis

We will collect descriptive data on ePOCR performance. The primary study outcome will be the proportion of prompts of either design which result in compliance with randomisation by the clinician. Estimates of prompt compliance will be generated for both liberal and restrictive magnesium strategies in addition. Compliance is defined as either: (1) the appropriate administration of magnesium following prompt deployment, where the measured serum magnesium is less than the randomised threshold or (2) the appropriate withholding of supplemental magnesium following prompt deployment, where the serum magnesium is greater than the randomised threshold. The potential outcomes following prompt deployment are illustrated in figure 5, using the Nudge design as an example.

Figure 5

Derivation of compliance with randomisation from observed clinician action.

For the Preference design, descriptive data will be presented across the range of possible responses. We will link events where the clinician declines randomisation and expresses a strong preference to the subsequent action (administration of magnesium or not). The observation period for assessing compliance will be defined as the time from exposure to prompt to the subsequent shift change in clinical team. We will assess between group differences in proportion using a χ2 test.

All quantitative data pertinent to addressing the study outcomes will be extracted from the EHRS. The study will not require any additional documentation or data entry by clinical teams. We will extract routinely collected clinical data from the EHRS for study patients to aid planning a future main study. This will include baseline rates of atrial fibrillation in the study population, frequency of serum magnesium measurement, frequency of supplementation and estimates of treatment group separation (difference in mean serum magnesium values between liberal and restrictive groups).

The semistructured interview programme will contribute qualitative data to both primary and secondary outcomes. Overall study feasibility will be judged through a combination of ePOCR prompt compliance rates and acceptability to clinicians. ePOCR compliance data will contribute to further simulation work designed to estimate plausible ranges of samples sizes for a main study, which will be used to further demonstrate study design feasibility.

We will use a thematic analysis approach to analyse interview data as described by Braun and Clarke and illustrated in a recent analysis by McNulty et al.29 30

The primary objective of the study is to determine the feasibility of using ePOCR prompts. Overall feasibility will be judged using technical aspects around design and implementation and the experience of clinicians, which will both be assessed qualitatively.

While overall study feasibility will be contingent on the rate of prompt compliance (the proportion of alerts where the clinician complies with randomisation), we will also estimate differences in compliance between nudge and preference designs to add to our qualitative assessments. We propose a non-inferiority approach based on the premise that the preference design has improved research utility relative to the nudge design through estimation of treatment and selection effects. Therefore, if the preference prompt proves non-inferior in terms of observed compliance, and qualitatively acceptable, it would be demonstrably the preferred design. We will not seek to test between group differences for each magnesium strategy in addition.

To this end, we estimate the required sample size based on hypothesised equal compliance rate of 50% in both groups. We hypothesise a non-inferiority margin of −25% to be justifiable relative to the additional data the preference design would provide. This produces a sample size of 50 prompts per design, with a power of 80% and a 5% significance level.31 Using an average of two prompts per patient, this results in a sample size of 25 patients per group.

We will aim to recruit 20 clinicians identified as key informants relevant to the study question to undertake baseline interviews.32 We will employ a purposive sampling strategy as used by Connell et al to evaluate a complex digital intervention in a similar healthcare setting and justified by international consensus guidance for mixed-methods research.33 34 We will aim to interview all bedside nurses who receive a prompt and use guided simulation to aid the evaluation of preference for either prompt design.

Ethics and dissemination

This study protocol was approved by the NHS Riverside Research Ethics Committee (Ref: 21/LO/0785) and sponsored by University College London (Ref: 142382).

Potentially eligible patients will be approached during their anaesthetic pre-assessment clinic visit. After confirming initial eligibility, a member of the research team will discuss the study and issue the Participant Information Sheet (online supplemental material S2), which includes research team contact information and mechanisms to withdraw from the study at any point. The patient will be able to give written consent (online supplemental material S3) at any point from initial approach to immediately prior to surgery.

By approaching patients in pre-assessment clinic, we evaluate a pre-emptive approach to providing consent which may be transitioned to an opt-out approach in the future if acceptable. We justify obtaining consent at the initial visit in three ways. First, the study premise and intervention carry minimal risk to the participant, second, the burden on the participant is low (one follow-up interview following surgery). Third, we provide multiple routes to discontinue participation with multiple checks throughout the perioperative journey.

We ensure that participant data are protected by extracting on data pertinent to the study from the EHRS. All clinical data obtained during the study will remain within UCLH computer infrastructure and firewall. Data extracted from the EHRS will only be accessible by designated members of the research team and presented as summary level data. Interview data will be audio recorded and uploaded via secure email to UCLH computer systems for analysis. Only anonymised quotes will be used in the study reports. We plan to disseminate results by publication in peer-reviewed journals and will also prepare reports for patients and clinicians involved in the study on completion.