Real-World Data and Randomised Controlled Trials: The Salford Lung Study.

Traditional efficacy double-blind randomised controlled trials (DBRCTs) measure the benefit a treatment produces under near-ideal test conditions in highly selected patient populations; however, the behaviour of patients and investigators in such trials is highly controlled, highly compliant and adherent, and non-representative of routine clinical practice. Pragmatic effectiveness trials measure the benefit a treatment produces in patients in everyday "real-world" practice. Ideally, effectiveness trials should recruit patients as similar as possible to those who will ultimately be prescribed the medicine, and create freedom within the study design to allow normal behaviours of patients and healthcare professionals (HCPs) to be expressed. The Salford Lung Study (SLS) was a world-first, prospective, phase III, pragmatic randomised controlled trial (RCT) programme in patients with chronic obstructive pulmonary disease and asthma to evaluate the effectiveness of a pre-licensed medication (fluticasone furoate/vilanterol) in real-world practice using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research. The real-world aspect of SLS was unique, requiring careful planning and attention to the goals of maximising the external validity of the trials while maintaining scientific rigour and securing suitable electronic processes for proper interpretation of safety data. Key learnings from SLS that may inform the design of future pragmatic effectiveness RCTs include: (1) ensuring the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad patient population with characteristics as close as possible to patients in routine clinical practice, to maximise the generalisability and applicability of trial results; (3) ensuring that patients and HCPs are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the trial data.

RCT Entities:   Population Interventions Outcomes

Key Summary Points

Introduction

Double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines, and are designed to achieve maximum internal validity with minimal potential for confounding factors [1]. Frequently conducted for the purpose of obtaining data to support regulatory approvals, DBRCTs underpin the evidence base informing treatment guidelines and healthcare decisions [1-3]. However, as efficacy DBRCTs are conducted in highly selected patient populations and under highly controlled conditions (Fig. 1) optimised to demonstrate the effect of the medicine, the generalisability of their findings to the overall disease population may be limited [3-7].

Fig. 1

Design aspects of traditional efficacy DBRCTs compared with pragmatic effectiveness RCTs. DBRCT double-blind randomised controlled trial, RCT randomised controlled trial

Pragmatic randomised effectiveness trials are designed to evaluate medicines in the “real-world” setting across a broad patient population [8, 9] (Fig. 1) and offer the opportunity to address issues faced by patients and healthcare professionals (HCPs) on a daily basis [3, 6], while retaining the benefits of random treatment allocation. Randomised effectiveness trials can complement traditional efficacy DBRCTs by filling the evidence gaps surrounding patient and physician experience, treatment adherence, and healthcare resource utilisation (HRU) and care costs, all of which are key to informing healthcare decision-making [9].

The phase IIIb, pragmatic effectiveness Salford Lung Study (SLS) RCT programme was designed to evaluate a pre-licensed inhaled corticosteroid/long-acting β2-agonist combination, fluticasone furoate/vilanterol (FF/VI), in patients with chronic obstructive pulmonary disease (COPD) and asthma in UK primary care [10]. SLS was a world first, embracing the novel use of electronic health records (EHRs) to comprehensively enrol a broad spectrum of patients from across Salford and surrounding areas. Here, we discuss key features and advantages/limitations of pragmatic effectiveness RCTs versus traditional efficacy DBRCTs (focusing on respiratory trials), using SLS as an illustrative example. We also describe key learnings from SLS and discuss how these might help inform the design of future pragmatic effectiveness RCTs and respiratory treatment guidelines/healthcare policies.

Efficacy and Effectiveness RCTS: Overview and Major Differences

Table 1 compares features of traditional efficacy DBRCTs versus pragmatic effectiveness RCTs. The strict inclusion/exclusion criteria of traditional efficacy DBRCTs are designed to maximise internal validity and reduce the impact of biases. However, in the real-world setting, patients with COPD and asthma arrive at the doctor’s office with many confounding/complicating factors not assessed in DBRCTs, which can have profound effects on the likelihood of a medicine causing benefit or harm (Box 1).

Table 1

Comparison of traditional efficacy DBRCTs and pragmatic effectiveness RCTs

	Traditional efficacy DBRCTs	Pragmatic effectiveness RCTs
Patients	Strict inclusion/exclusion criteria Patients with significant comorbidities and severe disease typically excluded Highly selected (“ideal”) patient population Limited relevance to patients in routine clinical care setting Good inhaler technique mandated High adherence mandated	Broad inclusion criteria, minimal exclusion criteria Patients with comorbidities and severe disease included Broad, heterogeneous patient population Greater relevance to patients in routine clinical care setting Variable inhaler technique Variable (and frequently poor) adherence
Study design/conduct	Designed to test efficacy under near-ideal conditions (i.e. where confounding factors are minimised) Provide data on the efficacy and safety of a medicine, albeit in a highly controlled setting Typically recruited in ambulatory care/outpatient centres Randomisation (± stratification) and masking (e.g. double-blind, double-dummy) to limit bias due to systematic differences between treatment groups Can assess experimental medicine versus a placebo or “gold standard” comparator Treatment per protocol; generally treatment modifications are not permitted Frequent study visits/monitoring Adherence to treatment actively monitored and encouraged	Designed to test effectiveness in the presence of real-world factors Provide data on the overall treatment strategy in a real-world setting Conducted in routine clinical practice in primary care; patient management reflective of usual clinical care Randomisation (± stratification) to limit bias due to systematic differences between treatment groups; typically open-label in design Usually assess experimental medicine against usual care or established standard-of-care Treatment modifications permitted based on physician’s clinical opinion Few mandatory study visits; limited disruption to patients’ normal routine No monitoring or active encouragement of treatment adherence; patients’ health behaviours as normal
Outcomes/data	Data have high internal validity, limited external validity Endpoints often designed to enable regulatory approval/licensing	Data have high external validity Often include additional endpoints of interest, e.g. healthcare resource utilisation, patient-reported outcomes
Transferability/generalisability	Treatment effect in the real world has to be estimated Culturally accepted as most informative and therefore transferable, although external validity is weak	Data more generalisable to the overall disease population Effect of healthcare systems and access to medicines and cultural factors may need to be considered

DBRCT double-blind randomised controlled trial, RCT randomised controlled trial

Box 1

Examples of patient confounding/complicating factors in the real-world clinical practice setting

Existing diagnosis (often pragmatic or clinical) Access to medical care Non-adherence to prescribed medication (over- or under-treating) Poor inhaler techniquea Poor compliance with treatment advice and follow-up Comorbidities/coexisting medical conditions Polypharmacy Cigarette smoking and/or recreational drug use Variability in health literacy Diversions and distractions caused by life and social events, crises, shift-work patterns, accidents and injuries

Many of these factors preclude patients from being eligible, or wanting, to participate in traditional, highly controlled DBRCTs

aSpecific to the respiratory setting

Almost all of these factors are excluded/altered by the strict eligibility criteria, conduct, and need for protocol compliance in traditional efficacy DBRCTs. Consequently, only a low proportion of primary care patients with COPD and asthma would be eligible for participation in typical efficacy trials [4, 11–14] and the relevance of DBRCT results to patients in routine practice is limited [1, 7]. Effectiveness RCTs can therefore supplement data from efficacy DBRCTs by providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.

The inherent design differences between traditional efficacy DBRCTs and pragmatic effectiveness RCTs result in different strengths and limitations of the acquired data (Box 2).

Box 2

Traditional efficacy DBRCTs versus pragmatic effectiveness RCTs: impact of study design aspects on acquired data

	Scenario	Pros	Cons
Process of care	In DBRCTs: Protocols demand atypical conduct from patients and HCPs, often requiring frequent, rigorous and prolonged study assessments Optimal treatment compliance and inhaler device technique is strongly encouraged Treatment pathway is strictly defined, and deviations from prespecified treatment may result in patients being withdrawn from study (restricting freedom of choice around patient care)	Rigorous assessment in efficacy DBRCTs allows for robust data acquisition and for patient safety to be closely monitored By strongly encouraging optimal treatment compliance, DBRCTs are able to provide an accurate profile of treatment efficacy and safety for an intended dose	A downside of the highly controlled environment in which efficacy DBRCTs are conducted is the impact on the behaviour of both patients and physicians, which does not truly reflect the characteristics of a medicine when used in the real-world setting
	In pragmatic effectiveness RCTs: Care is aligned with that received in routine clinical practice, with minimal scheduled study visits/assessments and minimal disruption to patients’ everyday lives Physicians have freedom of choice to modify patients’ treatment as deemed necessary	Study design permits freedom of choice around patients’ care and allows the normal behaviours of patients and HCPs to be expressed; thus, acquired data may be more generalisable to patients seen in everyday clinical practice	Data from pragmatic effectiveness trials more likely to be confounded by extraneous variables, not controlled for by virtue of the trial being conducted in a real-world setting
Data collection	In efficacy DBRCTs, patients’ data are typically entered into eCRFs by investigators/dedicated study team members during or following scheduled study visits The challenge for pragmatic effectiveness trial design is to adequately balance the delivery of highly accurate and complete data while minimising the level of interference that data entry and verification pose to routine practice. Effectiveness trials may rely on data extraction from patients’ EHRs or other spontaneous reporting systems, as well as eCRFs for data capture In contrast to traditional efficacy DBRCTs, where neither investigator nor patient has knowledge of the assigned treatment, pragmatic effectiveness RCTs are typically open-label in design	Compared with eCRFs, EHR-based data capture has advantages in terms of allowing for remote data collection in real time (avoiding recall or transcription bias) and in providing the opportunity for long-term follow-up of the trial population after the study has completed	Disadvantages of EHR-based data capture include missing data, lack of representation of endpoints of interest, and potential issues with accessibility for research purposes in certain regions/countries Conducting effectiveness trials in routine clinical care means that some investigators may be inexperienced in clinical research and processes must be implemented to manage this to ensure robust data collection (accordingly with resource, logistics, and cost implications) An open-label study design creates the potential for bias in the acquired data due to behavioural effects that arise as a consequence of knowledge of the treatment that has been administered, e.g.:    Physicians may be biased in assigning causality of AEs to an investigational medicine rather than to a well-established standard-of-care treatment where common AEs are well known—this could have a positive or negative effect on AE reporting    There may be increased HCP/patient vigilance with a new medicine, resulting in higher rates of healthcare contacts than with a more familiar treatment option    Patients may merely have a preference for, and revert to taking, a more familiar treatment, thus impacting on adherence to the assigned study treatment The Hawthorne effect—where individuals modify an aspect of their behaviour in response to their awareness of being observed—may apply and confound data collected in both traditional efficacy DBRCTs and pragmatic effectiveness RCTs; however, this effect may be less likely in effectiveness trials that are designed to minimise disruption to everyday clinical care
Trial eligibility criteria/patient population	The stringency of eligibility criteria for traditional efficacy DBRCTs versus pragmatic effectiveness RCTs will dictate the nature of the patient populations recruited, and this can have a profound effect on the data collected during such trials Patients in traditional efficacy DBRCTs are usually recruited in ambulatory care/outpatient centres, tend to be healthier than the non-trial disease population, and often participate in multiple trials In high-recruiting research centres, investigators may hold a database of patients who are “ready to enrol”. Such patients will meet trial inclusion/exclusion criteria as standard, are quick to learn and maintain excellent inhaler technique, intellectually capable, highly compliant with study protocols and procedures, and are familiar with a range of treatment devices and study assessments Pragmatic effectiveness RCTs seek to recruit a broad participant population with characteristics as similar as possible to patients who will eventually be prescribed the medicine in routine clinical practice. To achieve this, such trials typically employ minimal inclusion/exclusion criteria	The highly selected patient populations in traditional efficacy DBRCTs allow for testing of the efficacy of a medicine under conditions where confounding factors are minimised; thus, data have high internal validity	The strict entry criteria/requirements for adherence to protocol in traditional efficacy DBRCTs may preclude otherwise eligible patients from participating (e.g. patients from deprived areas, for reasons including difficulties with/costs of getting to the research site, or working and family commitments). This has led to the concept of “persistent participators” in efficacy DBRCTs—a population that is non-representative of patients treated in real-world practice Findings from efficacy DBRCTs have limited applicability/generalisability of the acquired data to the wider disease population—low external validity    An example would be the collection and interpretation of AEs. In a pragmatic effectiveness trial, by virtue of enrolling a broader population of patients (including those with comorbidities and more severe disease), it is likely that a higher incidence and/or wider variety of AEs will be reported than in an efficacy DBRCT evaluating the same medicine These limitations are somewhat circumvented in pragmatic effectiveness RCTs In pragmatic effectiveness RCTs, recruiting a broad participant population may introduce additional variability to the data set, and a greater number of patients may need to be enrolled to power the study to demonstrate treatment effect, compared with traditional efficacy RCTs
Outcomes	Endpoints in traditional efficacy DBRCTs (registrational trials in particular) are often dictated by outcomes of interest to regulatory authorities, often require frequent assessments and diary cards/electronic diaries, and serve as constant reminders of disease state and treatment response In pragmatic effectiveness RCTs, it is desirable to select endpoints that are relevant to patient-centric goals for treatment and that physicians routinely use to assess patients and make treatment decisions, so as to optimise external validity and transferability of the data, and enhance value to clinicians, payers, and policymakers    In respiratory trials, for example, such endpoints would include exacerbations, hospitalisations, mortality, validated patient-reported outcomes, and quality-of-life measures	In effectiveness trials, it is desirable to minimise the impact of study assessments by selecting endpoints and a frequency of measures that ideally can be gathered with little or no impact on the patient or HCP, and where observer bias is controlled—crucial for an open-label study design	In pragmatic effectiveness RCTs, certain endpoints may be precluded because of the intensive monitoring that would be required (e.g. serial lung function, daily diaries)
Data analysis/interpretation	In routine clinical practice, a patient’s treatment will be adjusted at the discretion of the treating physician Treatment modifications are rarely permitted in traditional efficacy DBRCTs, but are allowed (albeit with potential restrictions) in pragmatic effectiveness RCTs, with implications for the analysis and interpretation of the study data (particularly important for safety evaluation) In effectiveness RCTs where treatment can be modified, careful consideration must be taken as to whether specific study endpoints will be evaluated as ITT (i.e. according to randomised treatment group) or by actual treatment received    Effectiveness endpoints will typically be analysed as ITT, which is equivalent to comparing the treatment strategies being investigated in the effectiveness RCT    Safety data ought to be presented both by randomised treatment group and also by actual treatment	In traditional DBRCTs, efficacy and safety endpoints are typically analysed according to the ITT principle. Interpretation of data is more straightforward; randomisation group equates to treatment group and data can be analysed accordingly (e.g. safety events can be attributed to randomised treatment) In effectiveness RCTs, analysis by actual treatment received allows for assessment of true exposure risk of a medication	In effectiveness RCTs, variation in the treatment being taken produces an additional level of complexity for data analysis, in that it precludes the direct comparison of randomisation groups from being equated with the safety of treatment A compared with treatment B

AE adverse event, DBRCT double-blind randomised controlled trial, eCRF electronic case report form, EHR electronic health record, HCP healthcare professional, ITT intent to treat, RCT randomised controlled trial

The Salford Lung Studies in COPD and Asthma: What Were Their Novel “Real-World” Aspects?

The SLS programme comprised two concurrent, 12-month, open-label, phase IIIb pragmatic RCTs designed to evaluate the effectiveness and safety of initiating once-daily inhaled FF/VI, compared with continuing usual care (UC) in patients with COPD or asthma in UK primary care [10, 15–18]. All patients provided written informed consent for participation in SLS and the trials were conducted in accordance with the International Conference on Harmonisation Good Clinical Practice guidelines and the provisions of the 2008 Declaration of Helsinki. The trial protocols were approved by the National Research Ethics Service Committee North West, Greater Manchester South (approval numbers 11/NW/0798 and 12/NW/0455).

When SLS commenced, a full regulatory submission for FF/VI was under consideration by the European Medicines Agency that was based on extensive efficacy and safety data from completed RCTs [10]. SLS was conducted to meet the need for effectiveness data for FF/VI to complement existing evidence from efficacy RCTs. SLS was the first study in the UK to take advantage of joint advice from the Medicines and Healthcare products Regulatory Agency (MHRA) and The National Institute for Health and Care Excellence (NICE). MHRA, responding positively to NICE’s enthusiasm for pragmatic data in a broad community, approved the study design with a pre-licensed medicine—a key factor enabling the study to proceed.

SLS evaluated the effectiveness of a pre-licensed medication in the real-world setting using EHRs [10, 19] and collaboratively engaged general practitioners (GPs) and community pharmacists in clinical research. The SLS trial designs have been reported previously [10, 15–18] (Fig. 2). The studies employed broad eligibility criteria to recruit large, heterogeneous populations of patients with COPD and asthma. There were few protocol-mandated clinic visits and data were collected continuously and remotely from patients’ EHRs using a primary/secondary care-linked database system (as well as via electronic case report forms, eCRFs). Patients were recruited and managed by their usual GPs, who prescribed as normal, and patients ordered and collected repeat prescriptions in the usual way and collected their study medication from their usual community pharmacist. Treatment modifications were permitted at GPs’ discretion during the study; patients randomised to initiate FF/VI could modify their treatment to any other appropriate treatment and remain in the FF/VI randomisation group. Those randomised to continue UC were also allowed to modify their treatment to any other appropriate treatment (except for FF/VI) and remain in the UC randomisation group. The real-world design aspect of SLS was unique, requiring careful planning and attention to the goals of maximising the external validity of the trials while maintaining scientific rigour, as well as securing suitable electronic processes for proper interpretation of safety data.

Fig. 2

SLS trial designs. a SLS COPD, b SLS asthma. ACT Asthma Control Test, COPD chronic obstructive pulmonary disease, FF/VI fluticasone furoate/vilanterol, GP general practitioner, ICS inhaled corticosteroid, LABA long-acting β2-agonist, LAMA long-acting muscarinic antagonist, SLS Salford Lung Study, UC usual care, y years

Maximising External Validity

Careful design and much background work went into ensuring that the delivery of SLS was aligned with routine care (e.g. normality of medicines supply; patient and HCP behaviours consistent with everyday clinical practice; interplay between patients, GPs, pharmacists in the community setting), while ensuring that the study conduct and data collection met the requirements of a phase IIIb RCT.

Collecting patients’ data via EHRs allowed us to measure the SLS COPD primary effectiveness endpoint (moderate/severe exacerbations) through surrogates triggered in the EHR (prescription of antibiotics and/or systemic corticosteroids or hospital admissions/visits associated with a respiratory cause [17]). This ensured that patients and doctors were essentially unaffected by the study for the entire follow-up period. In SLS asthma, the same primary effectiveness endpoint was not feasible because of the expected low frequency of exacerbations based on a pilot study (recruitment numbers would have been too high to achieve the required statistical power); instead, response on Asthma Control Test (ACT) was selected as the primary outcome. ACT was completed by patients at the baseline (randomisation) and week 52 (end of study) scheduled visits, and was additionally administered via telephone at weeks 12, 24, and 40 [18]. Processes were implemented to ensure that ACT was administered with minimal interference to normal care (e.g. GPs were aware of ACT scores at baseline, but not thereafter; telephone ACT was administered by a study nurse blinded to treatment and who was trained not to provide advice to the patient, except under life-threatening circumstances).

All HCPs involved in SLS (GPs, nurses, pharmacists, and their staff) were trained to allow routine clinical practice to proceed, although consultation rates were higher in the FF/VI randomisation group than the UC group during the first 12 weeks [20, 21], predominantly for non-respiratory reasons, suggesting that GPs and patients did undergo an initial familiarisation period with the new therapy. Importantly, by extracting HRU data directly from EHRs, we were able to obtain a complete picture of the burden associated with COPD and asthma without bias due to recall or transcription of data, and were able to demonstrate a disproportionately high resource use for non-COPD/-asthma reasons [20, 21].

This “hands-off” approach really allowed normal patient and HCP behaviours to play out in SLS—quite unusual for a phase III trial, and a very positive aspect of the trial design, as it provides valuable information about how FF/VI performs when used in routine practice.

Maintaining Scientific Rigour

Through its prospective design, baseline randomisation/stratification procedures, and extensive a priori analysis plan, SLS achieved the scientific rigour characteristic of a traditional efficacy RCT. Much consideration went into the decision to allow asymmetric treatment modification in the trial design and the subsequent impact this would have on the data analyses (analysis by randomised treatment group or by actual treatment).

Furthermore, for the purpose of statistics and programming, the sponsor remained blinded to study treatment while the trial was ongoing and was only unblinded after all data had been collected and the study database had been locked, thus mimicking the approach taken in a typical DBRCT.

Safety Data Collection

SLS commenced with a pre-licensed medicine and our intention was to vigilantly collect and evaluate safety events in real time through patients’ EHRs. In recruiting a population inclusive of patients with comorbidities and severe disease, we anticipated that the study would accrue a large volume of safety data and that patients would experience multiple serious adverse events (SAEs) that would not be seen in trials where such comorbidities are excluded. An innovative approach for identifying potential AEs without interfering with patients’ normal routines was essential. The SLS safety data collection process has been published previously [19], but a key aspect of this was the creation of a consultant-led specialist safety team, who were alerted to review potential safety events in real time.

Over 7000 patients participated in SLS and both trials met their primary effectiveness endpoints, demonstrating the benefit of initiating FF/VI versus continuing UC [17, 18]. In SLS COPD, there was a statistically significant reduction for FF/VI versus UC in the mean annual rate of moderate/severe exacerbations, without increased risk of SAEs [17]. In SLS asthma, the odds of patients being ACT responders (ACT total score ≥ 20 and/or improvement from baseline ≥ 3) at week 24 were significantly higher for FF/VI versus UC, without increased risk of SAEs [18]. Consistent benefit of FF/VI over UC has also been demonstrated for various other endpoints, as demonstrated in secondary analyses of SLS COPD and asthma [20-29].

SLS Approach to Effectiveness RCTS: Advantages and Limitations

Conducting effectiveness RCTs such as SLS in routine primary care requires access to the patient population of interest and good infrastructure, operational management, training/good clinical practice, and site engagement. It could be argued that such requirements could preclude the conduct of similar effectiveness studies in other geographical locations [30].

As SLS comprised phase IIIb trials of a pre-licensed medication requiring detailed safety monitoring, the studies were time- and resource-intensive to design and the operational logistics were highly complex. There exists a perception that effectiveness studies are simpler to design and less expensive to implement than traditional efficacy RCTs, but our experience with SLS suggests quite the opposite; however, this may not necessarily reflect requirements for all real-world effectiveness studies, which should be designed on a case-by-case basis.

SLS commenced with a phase of pre-licensed FF/VI in the UK primary care setting and this had implications for the acquired trial data set, which should be considered when designing similar future effectiveness RCTs. For example, in SLS COPD, higher rates of all-cause primary care contacts for FF/VI versus UC were observed in the first 3 months of the study, which may have been driven by additional scrutiny of the then-unlicensed FF/VI [20]. Patients and physicians were allowed to modify treatment, and this required an additional level of consideration around the underlying reasons for modification and had implications for data analysis. In SLS, we were able to determine actual HRU and care costs (as opposed to the usual modelled costs) for patients with COPD and asthma, which is highly relevant for routine clinical practice in the UK. Furthermore, patients from deprived areas, who may be ineligible for, or unwilling to participate in, traditional phase III RCTs were recruited to the SLS. Salford, UK, is an urban location with areas of deprivation. Asthma patients were equally represented in deprivation categories, but COPD patients are over-represented in deprived areas. In an analysis of outcomes by deprivation, we found that deprivation did not impact the main outcomes of the SLS trials, thus supporting the recruitment of participants from all socioeconomic strata to provide data that are generalisable to routine clinical practice [26-28].

To reflect the routine clinical care setting, the UC randomisation group comprised many different inhaled maintenance therapies as the comparator for initiation of FF/VI, which could also be varied over the course of the study. Caveats of this design aspect include a limited capacity to prospectively evaluate FF/VI against a specific UC treatment, and inability to equate a UC option with an established standard-of-care in COPD or asthma. Allowing broad UC therapies and treatment modifications in SLS has limitations for data analysis (precludes direct, head-to-head comparisons of FF/VI versus UC, or of FF/VI versus one particular UC treatment), but advantages in that analysis by actual treatment could be conducted (e.g. safety reporting).

How Might SLS Inform HCPS and Decision-Makers?

For most newly approved medicines, evidence from efficacy DBRCTs is insufficient to fully guide physicians in choosing optimal treatment for their patients. Pragmatic effectiveness trials can fill the gap by providing data on the overall treatment strategy in routine clinical practice while maintaining the strength of an RCT [3] and are a valid option for addressing issues that patients, clinicians and policymakers face on a daily basis. Knowledge of the overall effectiveness of a medicine in the intended patient population, taking into account real-world factors related to patients, actual medication use, and healthcare systems, will ultimately help HCPs make more-informed treatment choices for their patients. Furthermore, this study is the first to provide HCPs with answers on how initiating FF/VI (having been treated previously with other medicines) may impact exacerbation rates and other outcomes versus continuing on those other medicines.

In respiratory clinical care, there has tended to be a focus on symptom management; however, patients are often more concerned with how their symptoms make them feel and the impact of symptoms on their everyday lives [31]. Health care is increasingly adopting a patient-centric approach, which considers patients’ perspectives regarding the impact of disease and its treatment. Clinical trials should, therefore, incorporate appropriate patient-reported outcomes (PROs) and quality of life (QoL) outcomes in their design. In SLS asthma, several PRO effectiveness endpoints were prospectively assessed (including ACT, Asthma Quality of Life Questionnaire, Work Productivity and Impairment Questionnaire: Asthma, and the EuroQol 5-Dimensions 3-Levels questionnaire). Initiation of FF/VI was associated with consistent benefits in PROs versus continuing UC, demonstrating that the observed improvement in asthma control (measured by ACT) for FF/VI translates into patient-perceived benefits in health-related QoL [23]. Follow-up interview-based studies conducted in subsets of patients who completed SLS have also provided important additional findings on patient-centred outcomes relevant to respiratory care in routine clinical practice [24, 32, 33].

For healthcare policy decision-makers, data from effectiveness trials can provide a more balanced view of the overall benefit/risk of a medicine, including HRU and cost-effectiveness—critical factors for consideration by resource-limited health services [10]. However, despite Health Technology Assessment groups expressing a desire to see more pragmatic studies describing effectiveness, for many, their dossier restrictions do not allow unblinded studies to be included in their assessments. There have been few studies like SLS in phase III and so payers and regulators have little experience of the nuances of such data sets and how to respond to them within their regulation. However, unless they do so, this will have a negative impact on sponsors’ willingness to fund effectiveness studies.

Learnings From SLS: How Can These Be Applied to Future RCTS?

Pragmatic real-world study design requires careful consideration of the setting, patient population, intervention, comparator, and outcomes [3]. Here, we discuss key learnings from SLS and how these might help inform the design of future effectiveness RCTs. Considerable effort and time was spent on aspects of the study design and operationalisation beyond our experience in DBRCTs; these aspects are summarised in Table 2.

Table 2

SLS design aspects—issues requiring greater effort/input to resolve compared to DBRCTs, based on opinion of SLS investigators and operational staff

SLS design aspect	Issue(s)	Solution(s)
Recruitment of GP investigators	Almost all GPs had not previously taken part in clinical trials GPs are busy and do not have space in their practices or the time to conduct clinical studies	Appointed GP ambassadors who recruited practices and “sold” the value of the study in their locality Recruited a team of approximately 50 research nurses to support GP investigators
Patient recruitment was initially slow in the SLS	No previous experience of recruiting in this environment GPs very busy, limited time for recruitment	Personal contact from patient’s own GP critical for recruitment Adopted a project management approach to patient recruitment and consent, local advertising and pharmacy approaches to patients
Medicines supply and GCP management, pharmacy involvement	Pharmacies are usually in competition for business Few pharmacists had prior clinical research experience Requirement for pharmacy study-specific SOPs	Created a pharmacy steering group to oversee training and SOP development, endorsement of pharmacy chain superintendent pharmacists [34]
Study endpoints, analysis and powering	No prior data on which to base our power calculations, our statisticians had not previously dealt with studies like this Endpoints had to be of value but be measured with minimal interference to patient care	Considerable debate with the SLS Scientific Committee to decide on endpoints Numerous reviews of statistical plans and endpoints—very different to a DBRCT, where our confidence is higher
Randomisation and stratification of patients by asthma severity	Since very few baseline investigations were performed, it was impossible to stratify according to lung function or usual measures of disease severity	Novel approaches to stratification were developed, such as the issuing of a “dummy prescription” by GP at baseline assessment, which allowed us to stratify according to intended treatment
Electronic collection of pharmacy dispensing data	Pharmacy systems in the UK are primarily stock control and labelling systems, and many different systems are used	Bespoke solution created, which took an incredible amount of work
Safety monitoring to GCP standards	This had not been done previously and as we were using EHR triggers to detect certain study endpoints and safety signals, we had to think completely differently to safety monitoring in a DBRCT	Worked with the sponsor’s pharmacovigilance team to build a robust safety system Had a consultant physician-led safety team (two physicians and four nurses) monitoring signals on a daily basis [19]
Data quality and standards	Use of EHRs and effort required to ensure that data was of high enough standard to meet GCP requirements Sponsor’s experience from data governance was a need to take EHR to a higher level	Implemented a much higher than usual investment in data cleaning and data quality The EHR needed an additional programme to collect relevant (and delete irrelevant) data. Level of quality and governance not adequate to start with and required an audit → fix → audit approach

DBRCT double-blind randomised controlled trial, EHR electronic health record, GCP Good Clinical Practice, GP general practitioner, SLS Salford Lung Studies, SOP standard operating procedure, UK United Kingdom

Setting and Infrastructure

The underlying operational infrastructure was the key to delivering two large phase IIIb trials evaluating a pre-licensed medicine out of local GP practices and pharmacies in and around Salford, UK. Salford was an ideal location for the trials owing to its relatively static population served by a single hospital with an integrated, real-time EHR connection with surrounding primary care practices and linking with patient-level prescription information (Salford Integrated Record, SIR) [10]. A bespoke information technology infrastructure was developed by NorthWest eHealth to extract data from the SIR for the purpose of effectiveness research [10, 19]. Over 2100 GPs, nurses, and pharmacy staff in and around Salford were trained in good clinical practice in SLS.

Critical to the delivery of SLS was the unique involvement and collaboration of community pharmacies. A key component of the integrity of the effectiveness design was maintaining the normality of repeat prescribing and dispensing in a situation where a pre-licensed medicine was being evaluated. Extensive training and process development permitted pharmacists/pharmacy staff to participate in a phase III trial, despite most having no prior clinical research experience. SLS demonstrated that pharmacies normally competing for prescription business can work in a collaborative and supportive manner for the benefits of patients. Specific aspects of how pharmacy involvement was successfully achieved in the SLS will be the subject of another publication [34].

Patient Population: Inclusiveness and Applicability

SLS showed that by employing limited eligibility criteria, it is possible to recruit to a phase IIIb effectiveness RCT a broad population of patients that are representative of those in everyday clinical practice, including from socioeconomically deprived areas.

SLS COPD enrolled approximately half of all eligible patients with COPD in the target geographical area [7]. These patients had a high disease burden and more symptoms, more frequent exacerbations, more comorbidities, and more SAEs/pneumonia SAEs compared with patients in large, registrational, efficacy RCTs in COPD [7]. Furthermore, over half of SLS COPD patients were categorised in the most deprived quintile by postcode [26, 28]. Notably, only at most 30% of SLS COPD patients would have been eligible for a typical regulatory phase III COPD exacerbation study [7], definitively demonstrating that COPD patients enrolled in traditional efficacy RCTs are not representative of patients in primary care.

The applicability of SLS findings to patients in routine clinical care is supported by a previous study demonstrating similarity in the characteristics of SLS COPD patients with other primary care patient populations across Europe [35]. Further support comes from a recent observational study demonstrating that patients in the SLS COPD UC group were similar to a matched cohort of non-trial COPD patients in England from the Clinical Practice Research Datalink database [36]. Evidence for a Hawthorne effect was observed, with a higher frequency of COPD exacerbations recorded in SLS patients than in non-trial patients; however, the largest effect was observed through behavioural changes in patients and GP coding practices [36]. SLS data have, therefore, contributed to the development of novel methods for evaluating the presence of an operating Hawthorne effect for future effectiveness trials conducted in everyday clinical practice.

Patient and HCP Engagement

Our experience with SLS underscores the importance of carefully designing pragmatic effectiveness RCTs to maximise chances of success in routine practice while ensuring operational feasibility. Engaging patients and HCPs in effectiveness research is extremely challenging. Initially, patient enrolment was slow in the SLS and we had to revisit our strategies for recruitment. The key was the involvement of patients’ own GPs in recruitment and obtaining of consent. Following on from SLS, additional qualitative research has been conducted to understand the drivers for patient and HCP engagement in the studies and how participation in future effectiveness trials might be enhanced [37]. Though key learnings from SLS will be the subject of a separate publication, our findings should be highlighted around the overall positive experience of patients and HCPs who participated in SLS and the importance of (1) local advertising to raise community awareness of study recruitment; (2) site/investigator engagement, ensuring that through extensive training on Good Clinical Practice, the study design and delivery was aligned with routine care; (3) provision of research nurse support at study sites, which was key to study delivery; (4) ease and convenience of study assessments; and (5) a need for improved study results dissemination [37].

Outcomes

SLS incorporated outcomes of interest outside those typically included in traditional respiratory efficacy DBRCTs (e.g. HRU/care costs, PROs, patient exit interviews), which has provided valuable supporting data for the benefit of initiating FF/VI versus continuing UC in patients with COPD and asthma in routine practice [20–24, 32, 33].

Furthermore, allowing asymmetric treatment modifications in the study design necessitated a novel approach to safety evaluation: highly comprehensive safety analyses were conducted both by randomised treatment group and by actual treatment at the time of an event.

The scientifically rigorous collection of real-world data in SLS offers major opportunities for future studies examining new research questions.

Conclusions

The real-world design aspect of SLS was unique, requiring careful planning and attention to the goals of maximising external validity while maintaining scientific rigour and securing suitable electronic processes for safety data collection. Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and HCPs are suitably engaged in the trial, to maximise the chances of successful trial delivery; (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers; and (5) using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.

Traditional efficacy double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines; however, their conduct in highly selected patient populations and in highly controlled settings limits the generalisability of their findings to patients seen in everyday clinical practice.

Pragmatic effectiveness trials conducted in the routine clinical care setting allow for the evaluation of the effectiveness of medicines in the presence of real-world factors related to patients, actual medication use, and healthcare systems, thus providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.

In this article, we discuss the key features and advantages/limitations of pragmatic effectiveness randomised controlled trials (RCTs) compared with traditional efficacy DBRCTs, using the Salford Lung Study (SLS) programme as an illustrative example.

SLS was the world’s first prospective, phase III, pragmatic RCT to evaluate the effectiveness of a pre-licensed medication in a primary care setting using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research.

Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and local healthcare professionals (HCPs) are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.