Effectiveness trials: critical data to help understand how respiratory medicines really work?

Most of the information about the benefits, safety aspects, and cost effectiveness of pharmacological treatment in the respiratory field has been obtained from traditional efficacy studies, such as randomised controlled trials (RCT). The highly controlled environment of an RCT does not always reflect everyday practice. The collection, analysis, and application of effectiveness data to generate Real World Evidence (RWE) through pragmatic trials or observational studies therefore has the potential to improve decision making by regulators, payers, and clinicians. Despite calls for more RWE, effectiveness data are not widely used in decision making in the respiratory field. Recent advances in data capture, curation, and storage combined with new analytical tools have now made it feasible for effectiveness data to become routine sources of evidence to supplement traditional efficacy data. In this paper, we will examine some of the current data gaps, diverse types of effectiveness data, look at proposed frameworks for the positioning of effectiveness data, as well as provide examples from therapeutic areas. We will give examples of both previous effectiveness studies and studies that are ongoing within the respiratory field. Effectiveness data hold the potential to address several evidentiary gaps related to the effectiveness, safety, and value of treatments in patients with respiratory diseases.

Introduction

Diverse evidence for medicines is needed in the world of today. Stakeholders such as regulatory agencies, payers, patient organisations, health care practitioners are increasingly requesting robust and reliable information about the benefits, safety aspects, and cost effectiveness of medicines. The foundation of this information is obtained from traditional efficacy studies, often required for regulatory approval, which demonstrate if a medicine is efficacious, safe, and has an appropriate benefit/risk ratio. However, the highly controlled environment of randomised controlled trials (RCT) does not always reflect everyday practice [1-4].

Collection, analysis, and application of effectiveness data to generate RWE have the potential to improve decision making by regulators, payers, and clinicians. The advantage with effectiveness data is that it gives a better picture how a treatment works in everyday practice compared with an RCT, accounting for a range of factors, such as unclear diagnosis, co-morbidities, and patient behaviours. This data also helps to reduce the uncertainty of the expected outcomes of various treatments. To that end, there has been a call for collection of more RWE data in the respiratory field [5].

Definition of effectiveness

There are many, varied, definitions of what constitutes ‘real-life’ data and terms like pragmatic trials, RWE, everyday clinical practice and effectiveness are used interchangeably [6]. In this review, the term effectiveness will be used.

Effectiveness can be thought of as the interaction of a medicine’s efficacy (usually demonstrated in near-ideal conditions in double-blind RCTs) [7] with factors related to patients, actual medication use, and health care systems, that results in the effects observed in patients in the everyday clinical setting. It may also be helpful to consider the different questions that efficacy and effectiveness trials seek to answer; an efficacy trial basically answers the question: ‘Does it work, is it safe?’ whereas and effectiveness trial responds to: ‘Will it work, what is the benefit and risk in the population where it will ultimately be used?’

RCTs are undertaken in well-characterised, highly selective populations, and managed in tightly

controlled settings. For example, respiratory RCTs have high internal validity but often represent fewer than 5% of patients treated in routine care [8]. Out of 334 patients with asthma who visited the doctor’s office, only 11 (3.3%) would have been eligible for inclusion in traditional RCTs. When applying the typical inclusion criteria for participation in efficacy RCTs, such as specific lung function (FEV1), reversibility, no co-morbidities, no smokers, treatment with inhaled corticosteroids (ICS) and symptoms – the attrition is very high [9-11]. Selection bias in efficacy trials can lead to exclusion of patients both with the most severe disease, as well as those with mild, well-controlled illness.

Though RCTs are and will remain the cornerstones of evidence necessary for regulatory approval, the extent to which RCT efficacy can be extrapolated to indicate outcomes achievable in real-life populations and routine care settings is unclear. Efficacy and effectiveness trials sometimes arrive at different conclusions about the benefit of a treatment. Examples of this is in the respiratory field data looking at whether ICS in combination with long acting beta agonist improve survival in COPD [11,12] or the safety of using tiotropium mist haler in COPD patients with concurrent heart diseases [13]. The reason for these differences may be due to several factors, such as selection of patients in the RCT or influence of confounders in effectiveness studies.

Types of effectiveness trials

Whether or not a study realistically represents real-life conditions can be unclear. A study might involve intensive patient follow-up yet include a broad population fairly representative of the true treated population. On the other hand, an observational study can focus on outcomes in a highly selected patient population yet involve no clinical intervention beyond usual care. The challenge is to recognise and describe which elements of a study represent everyday clinical practice [14]. While the intervention should be described precisely for both types of trial, in effectiveness trials this does not mean that the same treatment is necessarily offered to each patient.

Several different study designs can also be applied in effectiveness trials: prospective randomised controlled effectiveness trials to retrospective database studies, cross-sectional surveys, and post-RCT follow up studies [14]. Previous attempts to classify effectiveness trials include the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS)-wheel [15]. PRECIS identifies key domains that distinguish effectiveness trials from RCTs. Roche and colleagues [14] have proposed a framework based on two dimensions; the ecology of care (from constrained to free) and the population (from narrow to broad) – see Figure 1.

Figure 1.

Conceptual framework for therapeutic research.

Adapted from Roche et al. [14] (Reproduced with permission from the publisher).

Within the respiratory field retrospective studies using databases and or electronic medical records have dominated [16-19]. Comparisons between different treatment regimens have been made by matching patients using different statistical methods. The advantage of such an approach compared to an RCT is that the problem with only patient selection is avoided. The problem is that any possible difference between treatment might be caused by other factors (confounders) not considered in the model. Another problem with these kind of investigations is that the diagnostic criteria for the included patients are not always clear. Pragmatic randomised effectiveness trial is a design that has been introduced to avoid the problem with RCT-related patient selection and the potential problems with confounding in retrospective observational studies [20-23]. Examples of these kind of studies are presented below.

Hierarchy and levels of evidence

As the name suggests, evidence-based medicine (EBM) is about finding evidence and using that evidence to make clinical decisions. A cornerstone of EBM is the hierarchical system of classifying evidence. Physicians are encouraged to find the highest level of evidence to answer clinical questions [24]. Systematic reviews, meta-analyses, and RCTs are usually considered as the highest level of clinical evidence. The hierarchies rank studies according to the probability of bias, as for example in the GINA guidelines (Table 1) [25]. RCTs are given the highest level because they are designed to be unbiased and have less risk of systematic errors. For example, by randomly allocating subjects to two or more treatment groups, these types of studies also randomize confounding factors that may bias results. At the bottom of the hierarchy, we find case series or expert opinion, which are often biased by the author’s experience or opinions and the lack of control of confounding factors. Though it has been recognised that different clinical specialities may have diverse needs and therefore the type and level of evidence need to be modified, there is no clear ranking or grading of data from effectiveness studies in international treatment guidelines.

Table 1.

Hierarchies of evidence – GINA.

Evidence level	Sources of evidence	Definition
A	Randomised controlled trials (RCTs) andmeta-analyses. Rich body of data	Evidence is from endpoints of well-designed RCTs or meta-analyses that provide a consistent pattern of findings in the population for which the recommendation is made. Category A requires substantial number of studies involving substantial numbers of participants
B	RCTs andmeta-analyses. Limited body of data	Evidence is from endpoints of intervention studies that include only a limited number of patients, post hoc or subgroup analysis of RCTs or meta-analysis of such RCTs. In general, Category B pertains when few randomised trials exist, they are small in size, they were undertaken in a population that differs from the target population of the recommendation or the results are somewhat inconsistent
C	Non-randomised trials, observational studies	Evidence is from outcomes of uncontrolled or non-randomised trials or from observational studies
D	Panel consensus judgement	This category is used only in cases where the provision of some guidance was deemed valuable but the clinical literature addressing the subject was insufficient to justify placement in one of the other categories. The panel consensus is based on clinical experience or knowledge that does not meet the above-listed criteria

From: GINA, Global Strategy for Asthma Management and Prevention 2018.

Regulatory aspects

Regulatory policy makers have had an increasing focus on the use of effectiveness data [26,27] Despite this, what has not changed is that the RCT remains the primary evidentiary source for regulatory and payer decision making, while effectiveness data provide supportive information [27]. Regulatory bodies, including the European Medicines Agency (EMA) and the Food and Drug Administration (FDA), have referred to effectiveness data as data on patient health and/or delivery of healthcare that are generated in routine clinical practice. However, this does not capture the nuance of some study designs (pragmatic trials where treatment is not determined within routine practice) nor data that can be generated using new technologies (e.g. sensors) outside of a controlled clinical setting.

While there are significant opportunities for leveraging effectiveness data in premarket development programs, full approval for a wholly new drug or biologic based solely on RWE remains difficult to envision. Both the EMA and the FDA recognise the potential value of effectiveness and welcome these data as important but also raise concerns around their interpretation as there are many different kinds of effectiveness study designs with varying degrees of scientific rigour.

Examples of effectiveness data from different therapeutic areas

The Salford lung study

The Salford Lung Studies in asthma and COPD evaluated the effectiveness and safety of initiating once-daily inhaled fluticasone furoate/vilanterol (FF/VI) versus continuing usual maintenance inhaler therapy [28-30]. In these open-label trials, the only intervention was the introduction of FF/VI, whereby one study group was randomised to the experimental medicine and the other was randomised to remain on usual care. An e-database was employed to enable general practitioners to function as study investigators, with changes in care during the study permitted based on their clinical opinions. Both studies demonstrated significant differences favouring FF/VI over usual care. In asthma this translated into consistent improvements in asthma control and quality of life. At week 24, the odds of being a responder were higher for patients that initiated treatment with FF/VI than for those on usual care (71 vs. 56%, p < 0.0001) [31]. In COPD the rate of moderate or severe exacerbations was significantly lower, by 8.4% (95% confidence interval, 1.1–15.2), with FF/VI therapy than with usual care (P = 0.02) [30].

The REDOX trial

The REgistry-based RCT of treatment Duration and mortality in long-term OXygen therapy (REDOX) is a pragmatic, open label, effectiveness study of long-term oxygen therapy (LTOT) prescribed 24 h/day compared with 15 h/day. The trial is registered with ClinialTrials.gov (NCT03441204) and was opened for recruitment in May 2018. At participating centres that prescribe LTOT nationwide, patients who fulfil current blood gas criteria for LTOT, are deemed able to participate by the responsible physician and who give their written informed consent, are randomized upon registration in the national Registry for Respiratory Failure (Swedevox) between the study treatments. Clinical follow-up and other treatments are in accordance with clinical routine practice. The primary outcome is all-cause mortality at one year. Secondary outcomes include hospitalizations (overall and from respiratory and cardiovascular disease), incident diagnosed diseases (such as infections, respiratory exacerbations, ischemic heart disease, and heart failure), symptoms and health-related quality of life. The primary outcome and main secondary outcomes are assessed using national registry data with near complete follow-up. Patient reported outcomes are assessed using a postal questionnaire at 3 and 12 months administered by the Uppsala Clinical Research Centre (UCR). Thus, outcome assessment does not burden the clinical units, and registries are used to identify, recruit, randomize and follow study participants. The trial hope to randomize 2,126 patients over a 3-year period. Besides being the largest planned study in chronic hypoxemia and the first registry-based RCT in respiratory medicine, the methodology can hopefully facilitate increased and cost-effective research to take forward improved evidence-based treatment of people who suffer from very severe respiratory disease.

The BRONCHIOLE trial

The Beta-blockeRs tO patieNts with CHronIc Obstructive puLmonary disease (BRONCHIOLE) study (Clinical Trials.gov no. NCT03566667) is a pragmatic RCT examining the benefit of beta-blocker therapy in COPD without overt comorbid cardiovascular disease. The pilot phase started in July 2018, and 1700 patients will be included at some 10 centres until the end of 2020. Inclusion criteria are a diagnosis of COPD confirmed by spirometry, age >40 years, and sinus rhythm 50–120/min, and exclusion criteria only include existing cardiovascular disease or beta-blocker therapy at baseline and contraindications towards beta-blockers, such as untreated atrioventricular block or severe asthma. The primary outcome is a composite measure of all-cause mortality, incidence of cardiovascular events and COPD exacerbations within one year. Endpoint data are obtained mainly from national registries, completed by history and record review at follow-up visits.

The pragmatic design of BRONCHIOLE denotes randomization to metoprolol at an aimed dose of 100 mg in addition to standard COPD care, or to standard COPD care only. The intervention drug is prescribed through the ordinary digital system, and follow-up is limited to one visit and one telephone call to a research nurse and the ending doctor visit after one year. The simple design with few exclusion criteria and follow-up visits has so far resulted in a very high acceptance rate of participation, with patients of COPD in all different stages, performance status and ages. Subsequently, a high external validity and generalizability of the study is expected. The BRONCHIOLE design is an example of how high feasibility can be achieved even in academic studies.

Discussion

In this review, we propose that pragmatic randomised effectiveness studies have an important role to play when evaluation treatment in respiratory diseases. Although there are circumstances when retrospective observational effectiveness investigations are appropriate, it is clear that for major regulatory decisions randomisation in a prospective trial is needed to limit potential bias. With increasing availability of large health datasets from electronic health records the opportunities to access and analyse effectiveness data have increased and similarly the abilities to conduct prospective, randomised effectiveness trials with relatively high scientific rigour have increased the robustness of the data. Pragmatic randomised studies such as the Salford Lung Study, REDOX, and BRONCHIOLE can provide important contextual data to assist a payer in determining how to generalise data from multi-centre clinical trials to a local setting when treating patients with respiratory diseases.

Effectiveness studies are, by design, specific to the healthcare setting in which they are set, and care needs to be taken in translating findings from one setting to another. Additional work should be undertaken to confirm that findings are relevant to other settings. On the other hand, overall similarities between various geographical settings and similarities in disease management between different health care systems suggest that many effectiveness trials are relevant also outside their locality [32]. It could also be argued that in terms of patient representativeness – effectiveness studies often better reflect the patients that will receive the treatment than traditional efficacy RCTs – irrespective of geographical location.

Where this is not feasible or not desirable, replicate studies in different settings should be considered. An attractive trait of real-world studies is that they allow the exploration of the impact of a particular healthcare setting on the outcomes achieved.

We believe that an integrated approach should be taken which refers to using all relevant data from controlled trials, as well as other sources, to enhance the understanding of the overall evidence of effectiveness. A clearer distinction between different types of effectiveness studies and their strengths and weaknesses is also called for. In this context an internationally recognised hierarchy would be valuable. The BTS/SIGN guidelines have levels of evidence that open up for this [33] by assessing both the grade of recommendation and the level of evidence where e.g. grade B opens up for the inclusion of effectiveness data (Table 2). Professional organisations, regulators and payers alike should be encouraged and supported to recognise the role of different study types in the evidence generation chain. A revised and globally adopted framework that looks at the integrated evidence available from RCTs, pragmatic trials and retrospective observational trials combined would help create a more complete picture of the value of a therapy.

Table 2.

Hierarchies of evidence – BTS/SIGN.

Grade of recommendation
A	At least one meta-analysis, systematic review, or RCT rated as 1++,and directly applicable to the target population; orA body of evidence consisting principally of studies rated as 1+,directly applicable to the target population, and demonstrating overall consistency of results
B	A body of evidence including studies rated as 2++,directly applicable to the target population, and demonstrating overall consistency of results; orExtrapolated evidence from studies rated as 1++ or 1+
C	A body of evidence including studies rated as 2+,directly applicable to the target population and demonstrating overall consistency of results; orExtrapolated evidence from studies rated as 2++
D	Evidence level 3 or 4; orExtrapolated evidence from studies rated as 2+

From: British Thoracic Society/Scottish Intercollegiate Guidelines Network (BTS/SIGN). British guideline on the management of asthma. 2016. Available at: www.brit-thoracic.org.uk [accessed September 2018].

In this way, effectiveness studies can complement traditional RCTs through phase III-IV studies and provide information on comparative effectiveness, long-term safety, and everyday clinical practice (Figure 2). In addition, effectiveness studies can help to inform the design and objectives of classical randomised clinical trials by posing hypotheses for testing in classical RCTs, identifying target patient groups and appropriate comparators, as well as informing implementable clinical strategies (Figure 2).

Figure 2.

Positioning and role of effectiveness studies in the evidence generation chain.

Conclusion

Despite increasing calls for effectiveness studies, the understanding and use of data generated from such trials in respiratory medicine remains limited. Consequently, we may be ignoring a crucial aspect of medicine assessment and, therefore, denying patients with respiratory diseases the opportunity for more effective therapies, while also discouraging effectiveness and patient-focussed medicine development. Effectiveness trials should be more readily used to complement RCTs through a framework for critical evaluation of the full available dataset for a medicine. This is ultimately likely to benefit patients through encouraging patient-focussed drug development, which includes consideration of the drivers of effectiveness and making more effective medicines available to patients.