Pragmatic randomised trials using routine electronic health records: putting them to the test.

Ten years ago, in a paper called Britain’s Gift, the editor of the BMJ at the time and the director of the UK Cochrane Centre outlined a vision of medicine for the 21st century: easy access to good quality reviews of clinical evidence, and the streamlined recruitment of patients into randomised trials as a matter of routine whenever there is uncertainty about choice of treatment.

“For example,” they explained: “we still do not know which treatments are useful for acute stroke, but if every patient in the world experiencing a stroke were admitted to trials we would have enough patients within 24 hours to answer many of these questions.”1

The first goal of easy access to good quality reviews of evidence is on its way to being realised. Trials, however, remain exceptional in everyday clinical care, and sometimes address comparisons that are irrelevant to doctors and patients because they compare new treatments with placebo rather than with the best treatments currently available. Furthermore, trials are often conducted in idealised or unrepresentative patient groups.2 Because of these problems, randomised trials commonly fail to inform decisions in everyday clinical care: they address the abstract question of an intervention’s efficacy under ideal conditions, rather than its effectiveness when used in usual clinical practice, on outcomes that are important to patients.3 4

Here we describe a UK project to implement randomised trials as unobtrusively as possible in the everyday clinical work of general practitioners (GPs), comparing treatments that are already in common use, and using routinely collected electronic healthcare records (EHR) both to identify participants and to gather results. We discuss the rationale for this approach, the potential for improving clinical evidence at low cost, and the barriers encountered.

Opportunities for using EHR data for randomised trials

Reports from both the Council for Science and Technology5 and from the Academy of Medical Sciences6 in 2005 and 2006 highlight the potential of EHR data for translational health research, and research with EHR data has been recognised as a key activity in the Department of Health’s national health research strategy.7 Healthcare records are routinely stored on computers in UK general practice (most people in the UK are registered with a general practitioner). Some GP databases can now be linked anonymously to other healthcare datasets, including hospital admissions records, death certificates, and disease registries. This record linkage system has been implemented within the general practice research database (GPRD) used in the trials presented here, and could be implemented more widely. It allows long term, anonymous, unobtrusive follow-up for major clinical outcomes, at low cost, and with no extra time burden for the clinician, health service, or patient.

Conventional trial recruitment is often problematic, with many trials failing to meet their recruitment targets.8 The EHR database may also be used to recruit patients into trials: it is searched to compile a list of potentially eligible patients, which is sent securely to the clinician’s desktop computer. When a patient on the eligibility list attends the practice for events related to the trial, a flag appears on screen to notify the clinician that the patient may be eligible for recruitment, with a link to the study website. If patient and clinician agree to participate and the GP confirms eligibility, the patient is randomised.

Table 1 outlines the research questions, interventions, and measurements in the first two feasibility trials for the randomised evaluations of accepted choices in treatment (REACT) trials that we are initiating. For these projects, there are daily downloads of GP EHR into the GPRD. The trial database can be compared periodically to the full research database for fraud detection and generalisability of the randomised population.

Table 1

Research questions, interventions, and measurements in two feasibility REACT trials initiated within the GPRD

RETRO-PRO: the effectiveness of simvastatin compared to atorvastatin—a feasibility study (ISRCTN33113202)
Research questions	Feasibility of REACT trials; pilot for comparative effectiveness of simvastatin and atorvastatin in patients with primary hypercholesterolemia and high cardiovascular disease risk
Intervention	Randomisation between simvastatin and atorvastatin in 300 patients; non-blinded
Outcome measures	Recruitment rates and technical challenges; changes in lipid levels at three months; duration of statin treatment over time; long term incidence of myocardial infarction, stroke, and death (as measured in the GPRD, linked hospital data, disease registry data, or death certificates)
eLUNG: the effectiveness of antibiotics compared to no antibiotics for exacerbations of chronic obstructive pulmonary disease: a feasibility study (ISRCTN72035428)
Research questions	Feasibility of REACT trials; pilot for comparative effectiveness of antibiotics in patients with an exacerbation of chronic obstructive pulmonary disease and non-purulent sputum
Intervention	Randomisation between antibiotic (whichever the general practitioner uses as first line) or usual care in 150 patients; non-blinded
Outcome measures	Recruitment rates and technical challenges; patient diary over four weeks of the exacerbations of chronic pulmonary disease tool (EXACT-PRO) as completed on an electronic device; hospital admission over three months (as measured in GPRD and linked hospital data); long term incidence of mortality (as measured in GPRD or linked death certificates)

A key requirement for the REACT trials is that so called usual conditions apply as far as possible. Application of usual conditions is important to ensure external validity, and also to promote recruitment and retention: incompatibilities between the protocols for randomised trials and usual clinical practice can act as a barrier to recruitment.8 The studies we have outlined are open label and non-blinded, with patients’ progress monitored as usual in clinical practice, and these follow-up data extracted from the EHR. The only added feature in pragmatic randomised evaluations is that, among currently accepted treatments where there is no evidence on comparative effectiveness, treatment choices are based on random allocation rather than on arbitrary decisions by clinicians.

Such uncertainty in choice of treatments remains common. For many treatments in common use there is no evidence to inform choice between available options, and, for most new medicines, evidence based assessment of any added therapeutic value is not published at the time of market authorisation.9 The UK database of uncertainties about the effects of treatments (DUETs) was established to publish uncertainties about the effects of treatments that cannot currently be answered by referring to reliable, up to date, systematic reviews of existing research evidence.10 Where there is no evidence comparing two commonly used treatments for the same condition, clinicians and patients have no way of knowing which is the more effective. In these circumstances, treatments are chosen arbitrarily, through a non-scientific, haphazard process. Treating patients in this arbitrary manner generates no new evidence to improve clinical practice.

Where there is no evidence to guide the decisions of doctors and patients, it is ethically acceptable and actively desirable to offer willing patients the option of randomisation to assess which treatment is preferable. General Medical Council guidance requires doctors to resolve uncertainties about the effects of treatments,11 and good medical practice requires that doctors communicate evidence clearly to patients. Randomisation with systematic data collection is the most rational and ethical way to resolve uncertainties.12 13 Embedding randomised evaluations within usual clinical practice can achieve this goal, and increase the likelihood that clinicians will declare honestly to their patients when there is uncertainty about the relative merits of alternative treatment options.

In principle, as an ultimate goal, in every situation where there is genuine uncertainty about which of two or more widely accepted treatments is best, all willing patients could be offered randomisation as part of routine clinical care, and their progress followed up through EHR. If comparisons of accepted treatments could be conducted within the routine clinical practice setting in this fashion, the benefits would be considerable in terms of new evidence, and cost effectiveness in research.14

Challenges with using EHR data for randomised trials

The REACT approach does, however, face substantial challenges (table 2 lists opportunities and challenges, with strategies to address them, for REACT trials conducted within EHR databases). Firstly, and most importantly, are current norms in research governance. The requirements for informed consent and regulatory oversight in all trials are time consuming and expensive, even for trials comparing two interventions that have already been shown to be safe and are in widespread and routine use. This is a problem that has raised concerns over almost two decades.15 Clinicians who admit there is uncertainty in a choice between two interventions, and wish to address the uncertainty by offering treatment in the context of a randomised evaluation, are subject to intense regulatory scrutiny. Yet during routine clinical care—in situations where there is no comparative effectiveness research to guide treatment choice, so that decisions are equally arbitrary—no such constraints apply.12 Experimentation by politicians on the delivery of health services also does not suffer from this intense regulatory scrutiny.16

Table 2

Potential opportunities and challenges with REACT trials conducted within EHR databases

Opportunities	Challenges
Long term follow-up at low cost—EHR database and linked datasets can be used to follow study participants over the long term for major clinical outcomes and mortality	Ethical and regulatory approvals—Approval has been achieved for pilot studies; risk adapted regulatory processes may expedite approval for trials of routine treatments in future
Easy identification of eligible patients—Candidates are identified automatically through the EHR database from a pool of all patients: clinician is alerted when a patient they are treating meets eligibility criteria	Lengthy consent process—Ongoing research is necessary into the optimum length of consent processes for informed patients; current practice will adversely affect recruitment of clinicians and patients
Highly representative study populations—Randomisation at point of routine care means safety and effectiveness of intervention is assessed in usual clinical practice	Research approval at multiple local sites—Different regions have varying requirements for research approval, which is resource intensive
Representativeness is measurable—Study population can be compared to patients not enrolled in the trial	Availability of desired outcome data in EHR—Feasibility of collecting additional patient outcome data being assessed (eg, an electronic diary); REACT trials not suited for studies that require major study specific data collection
Adverse event monitoring—Daily transfer of EHR records into database: (i) analyses in trial centre of suspected unexpected serious adverse reactions; (ii) comparisons of event rates with those in patients not enrolled in trial	Data quality of EHR data—Recruitment can be restricted to patients with baseline completeness of key covariates; linkages to external data sources permits validation; outcomes can be restricted to outcomes well recorded in EHR
Evaluation of research questions of direct relevance to clinicians—Trials only of treatments already in routine use	Trial drug supplies—Research focus is on current therapies prescribed as usual by clinicians: no special supplies needed
Validation of major clinical outcomes—Confirmation of outcomes through the linkages and/or by the patient’s clinician; blinded review of complete EHR by experts	Compliance with conventional good clinical practice (GCP) quality standard requirements—With electronic records, there is no difference between data held centrally and locally; a review is ongoing into optimum scrutiny methods for dispersed electronic trials
Recruitment for rarer conditions—Multiple sites offer a broader pool of potentially eligible patients	Clinician training in protocol and GCP—Online GCP training package is provided for participating GPs
Adaptive designs—Potential to incorporate minimisation during treatment allocation	Clinician time to recruit patients—New IT systems and strategies minimise time and disruption; qualitative research of participating GP feedback is ongoing
Testing of study strategies—Cluster randomisation of sites will allow evaluation of study strategies (such as method of collection of additional data)	Lack of blinding of treatment allocation—REACT trials are best suited to measuring major clinical outcomes with clear diagnostic criteria (such as death)
Fraud prevention—Newly registered patients not eligible; eligibility and recruitment checks all recorded in the trial IT system; strategy to recruit few patients at many sites	Cross over of study treatments over time—A challenge in most long term trials; cross over may be outcome of interest (indicating treatment failure); statistical techniques may partly deal with this
Fraud detection—Clinical records of participants prior to and after the trial are available to the trial investigators; outcomes from linked external sources not controlled by local investigators (such as hospital episodes, mortality register)	Local prescribing rules and performance indicators­—­­­­­­­GPs may operate under mandatory or incentivised prescribing rules, without any exception for research studies
Reduced loss to follow-up—Linkages will ensure that outcomes leading to hospitalisation or death will be captured, even after a patient has left study site	Poor recognition of uncertainty by clinicians—If clinicians are unaware that current practice lacks good quality evidence this may be a challenge for recruitment
Linkage of patient data to EHR—Information collected by patients (for example, using smart devices or electronic diaries) could be linked to outcome data recorded in EHR	Uncertainty faced by clinicians not recognised by researchers or funding agencies—Clinicians need to be involved in setting a relevant research agenda

Several justifications have been suggested for so called research exceptionalism—the phenomenon whereby more stringent rules are applied to research than to usual clinical practice, even for treatments in widespread use. One frequently raised justification is that research does not in general specifically aim to benefit the participants, but rather to generate knowledge; study participants may take the risks while others accrue the benefits.17 However, this justification often does not apply to patients with chronic conditions, whose treatment next year may well benefit from knowledge gained in the randomised evaluations they participate in today. It is also unclear how a trial presents extra risk, where the randomisation is between two routine treatments already in widespread use, and with no evidence presently available to inform a choice between them. This asymmetrical approach to regulation can be traced back to the establishment of the Declaration of Helsinki18 following the Nuremberg war crimes trials.16 Informed consent is fundamental to medical ethics, but the regulations designed to prevent abuse were never intended to prevent evaluation of safety in routine practice.

In the REACT trials no new risks are introduced, but the alternative—the current situation—has demonstrable ethical problems. Good quality evidence to improve patient care cannot be reliably generated from arbitrary treatment decisions made in usual clinical practice, and patients may continue to suffer through being exposed to interventions that are later found to be inferior. Furthermore, where there is uncertainty clinical decisions are often made without fully acknowledging their arbitrariness to patients.

The extent to which research exceptionalism will restrict the benefits of the REACT trials is not yet clear. All consent forms currently cover a great deal of information normally not provided when the same treatments are routinely given outside a randomised evaluation, and are extremely time consuming to complete. UK government guidelines presently recognise that one size will not fit all with respect to the information that is required to make autonomous decisions.19 However, the guidelines also state that all randomised trials must comply with the good clinical practice quality standard, and this includes a list of 22 different topics to be covered in the information sheet.20 Research ethics committees may further add to this barrier, often including idiosyncratic administrative requirements, such as a duty on participants to inform a private health insurer (as happened in our trials).

We have concerns over these barriers to research on routine treatments, which will reduce recruitment of clinicians and patients. Requirements for informed consent should ideally be based on empirical evidence on what kind of process best informs participants, and be designed in collaboration with patients. However, the good clinical practice quality standard, which has come to be viewed as canonical, was based on expert opinion, and has little empirical evidence. A systematic review has found that evidence for the optimal amount of information to enhance patient understanding is inconclusive and limited.21 UK government guidelines state that “any researcher is faced with considerable difficulty” in selecting information for informed consent, “given the disagreement on how much information potential participants in research want.”19 There are also frank contradictions. For example, the guidelines recommend that draft versions of patient information sheets should be passed to patients in disease support groups for comments. But the same guidelines also require that all informed consent procedures in trials must adhere to current good clinical practice requirements,19 which mandate extensive content,20 so support groups are prevented from reducing information overload, for example, should they recommend this in their comments.

This extra burden may reduce recruitment and retard research throughout clinical medicine. The largest review conducted on strategies to improve trial participation found that concerns about extra effort and workload are barriers to recruitment for both doctors and patients.22 The review also recommended that trials should be framed and organised in ways that minimise differences between research and clinical practice, using simple and clear entry criteria, and address questions of clear relevance to clinical practice.22

A second major challenge of using EHR data for trials is data quality, which is of paramount importance. The REACT trials will not be suited to evaluating every type of research question. A study requiring detailed, study specific data collection at regular intervals may not be best suited to a trial using EHR data. There may also be specific outcomes that a trialist would prefer to measure that are not routinely collected in EHR. However, by definition many major outcomes are recorded in routine medical notes. Furthermore, mortality and other major clinical outcomes can now also be measured in EHR databases, and then verified across several other data sources. As an example, a heart attack in the REACT trials can be measured in the GPRD, linked hospital data, disease registries, and death certificates (if fatal). Where there is doubt, the patient’s clinician can also be asked to confirm a diagnosis for an outcome of interest.

Current progress of REACT trials

The first two REACT trials have recently been approved by the research ethics committee, along with a qualitative study that seeks feedback from GPs and patients to assess and improve the implementation. The main point of discussion by the committee was our proposed consent form. The original one page patient information draft submitted for ethical review was considered by the committee to be the “skimpiest ever” and missing much of the standard clinical trial information (the informed consent template of the UK National Research Ethics Service lists a large number of items to be covered). We resubmitted the patient information sheet, now twice the length and amended to meet only the minimum ethics committee requirements. It could be argued that the single most important consideration for informed consent in the REACT trials should be the replacement of clinician’s uncertainty with randomisation. After all, the patient could have received any of the interventions in a REACT trial by consulting another clinician.

The IT system has taken considerable time to develop and is currently undergoing testing. Once implemented, the system will provide instantaneous trial recruitment and daily analysis of EHR data that can easily be adapted to future studies. GP recruitment is also continuing. Of English GPRD practices, 42% approached have expressed interest and 15% declined (recruitment in Scotland has just started). Together with recruitment by the Primary Care Research Network, we now have over 200 interested practices, and study details (protocol and contract) have now been sent to practices.

The main challenge now will be to obtain research and development approvals from the 150 local NHS bodies that cover UK general practices. Our experience with a GPRD cluster trial23 and pharmacogenetic study found that this takes enormous effort and time, even for low risk studies, replicated at multiple sites, and often with differing systems. Our goal of maximising representativeness is achieved by recruiting at multiple sites with few patients in each, so such fragmented and diverse local administrative systems present a challenge. Finally, the project also has undertaken a review on how best to comply with the good clinical practice quality standard, since this dispersed model with electronic data collection means that site visits for scrutiny are of very limited value.

Conclusions

EHR databases contain a wealth of information, and their utility for randomised evaluations should be fully exploited. A revolution is long overdue in the technical and research governance frameworks for testing widely used interventions whose relative merits are unknown. Narrowly restricted studies with questionable external validity need not be the norm. Our suggestion for large scale randomisation in usual clinical practice may face several challenges, some of them technical, but most of them related to research governance procedures. We hope that these barriers will be overcome, by providing proof of concept for a streamlined simple framework for undertaking REACT trials, in which recognition of widespread uncertainties about the effects of treatments will motivate clinicians and patients to participate in randomised evaluations as a matter of routine.