Measuring Health-Related Quality of Life in Randomised Controlled Trials: Expected and Reported Results Do Not Match.

Purpose: The assessment of health-related quality of life (hrQoL) may need to be reconsidered due to important differences between efficacy (the effect of a treatment under experimental study conditions) and effectiveness (the effect of a treatment under real-world conditions). We presume that most researchers intend to describe effects under real-world conditions when investigating hrQoL as an endpoint. Unfortunately, most studies are designed to confirm two theories: the efficacy of a new intervention under experimental study conditions and the real-world effectiveness of this intervention on hrQoL under non-experimental study conditions. Conflicting information emerges when the outcomes are supposed to describe effects under real-world conditions, but the assessment generates results obtained under experimental conditions. This paper examines the existing conflict between efficacy and effectiveness in a sample of 100 studies investigating hrQoL.
Methods: We analysed a sample of freely available publications of clinical studies listed in PubMed between April 2015 and August 2016 which assessed quality of life as an outcome. We assessed the following four characteristics that should differ in studies measuring either efficacy or effectiveness: 1) specification of the study as a randomised controlled trial or not, 2) description of the study design as pragmatic or not, 3) classification of the study as an efficacy or an effectiveness study and 4) number of selected inclusion and exclusion criteria.
Results: 91% of the studies assessed hrQoL under experimental conditions (in a randomised controlled trial), but not under real-world conditions. The important difference between efficacy and effectiveness was not described in 60% of the studies. Only 6% of studies classified the study as a pragmatic trial. The difference between inclusion and exclusion criteria was not addressed in any of the investigated studies.
Conclusion: The results of the four criteria confirmed our hypothesis that hrQoL studies are conducted mainly as experimental, but not pragmatic, trials indicating that the meaningfulness of the important difference between efficacy and effectiveness requires further discussion. Keywords: pragmatic trial, experimental study conditions, real-world conditions, efficacy, effectiveness, pragmatic.

Introduction

Health-related quality of life (hrQoL) is one of the essential outcome dimensions in clinical research. The most frequently discussed dimensions are physical health, mental well-being and social integration. The focus is primarily on the subjective patient perspective of his/her health status.1 Standards for reporting hrQoL have been described by the International Society of Quality of Life Research (ISOQOL).2,3 In this paper we focus on aspects which describe the differences of efficacy and effectiveness studies that were not yet considered in these guidelines.4

Improvement or maintenance of hrQoL is considered an important indicator for the success of interventions, especially in chronic diseases,5,6 but assessment methods are still controversial.7–12 Examples of discussed problems are the inadequate selection of dimensions to be measured, as well as the choice of the wrong measuring instrument.

We presumed that researchers investigating the effects of a new treatment for a particular disease will be interested in two outcomes: first, the effect of the new treatment on survival, and second, the effect on hrQoL.13 The demonstration of the effect on survival requires an experimental study design to confirm efficacy, ie, to confirm proof of principle, although demonstrating the principle effect on hrQoL will not provide the expected answer for two reasons. First, the result of hrQoL is generally a secondary endpoint in experimental studies which is considered as a hypothesis which needs to be confirmed in a separate study. Second, the effect of an intervention on hrQoL assessed under experimental study conditions may be completely different from the effects of the same intervention when assessed under real-world conditions.14,15

This paper addresses the existing challenge of describing two outcome dimensions (survival and hrQoL) that should be assessed under different conditions and proposes a solution to this challenge.

Methods

This paper focusses on four characteristics of clinical studies which are related to the difference between efficacy and effectiveness studies that have not yet been considered in guidelines and standards for reporting hrQoL data.2,3 The reported results are part of F. Wiedemann’s medical thesis.16

Evidence is provided that the criteria suitable for distinguishing between efficacy and effectiveness studies are not taken into consideration in studies on hrQoL. We examined a sample of 100 randomly selected publications of clinical studies assessing quality of life as an outcome listed in PubMed () between April 2015 and August 2016.

We conducted a PubMed search with the search parameters “trial” and “quality of life” in their titles. The format searched for was “Summary”, and the search algorithm was “Best Match”. The first 100 studies were numbered according to the order in which they were listed in the PubMed database by descending publication date and included in the study. We decided to include only freely available papers to facilitate easy access to other colleagues. Clinical studies on quality of life are almost exclusively related to hrQoL unless otherwise specified.

Inclusion criteria of papers in detail were the publication between April 2015 and August 2016 and including the terms “trial” and “quality of life” in the title of the paper. A paper was excluded if it was not freely available.

Some of the publications were updated during the project or were no longer freely available. In such cases, the updated versions were compared with the previous versions; we discovered that aspects relevant to us had not been changed. The complete references of the 100 articles can be found in the .

In these 100 publications the following characteristics were assessed:

1. Specification of randomised controlled trial (RCT) or other than randomised controlled trial (non-RCT). If a study was not explicitly named as a RCT, it was classified as a non-RCT.

2. Classification of the assessed outcome as efficacy or effectiveness. A study was classified as an efficacy or an effectiveness study if the term was used (mostly, but not exclusively) in the methods section. If the term efficacy or effectiveness was not used, it was classified as “not defined”.

3. Specification of the type of the study as pragmatic or other than pragmatic trial. A study was designated as pragmatic when this term was used in the title of the study or the methods section. If the term pragmatic was not explicitly stated by the study’s authors, we applied the term other than pragmatic.

4. Numbers of described inclusion and exclusion criteria. The criteria were identified and counted in the methods section of the publication.

Results

Fifteen of the 100 publications were only study protocols, and 85 were reports of completed studies. Fourteen were published in 2017 (updated versions), 64 in 2016 and 22 in 2015.

All four characteristics of the 100 studies on hrQoL are summarized in Table 1. The bibliography and the reported results are listed in the .

Table 1

Results of the Characteristics of the Analysed Publications Assessing Health-Related Quality of Life (n=100)

Analysed Characteristics of Form and Function of Clinical Studies Assessing hrQoL		Number of Studies
Study design	RCT	91
	Non-RCT	9
Classification of the assessed outcome as efficacy or effectiveness	Efficacy	14
	Effectiveness	25
	Both terms used synonymously	1
	Not defined	60
Type of study as pragmatic or other than pragmatic trial	Pragmatic	6
	Other than pragmatic	94
Number of selected inclusion criteria (n=75 with completely reported inclusion criteria)	Average number	4
	Interquartile range	3–5
Number of selected exclusion criteria (n=69 with completely reported exclusion criteria)	Average number	4
	Interquartile range	2–6

Abbreviations: HrQoL, health-related quality of life; RCT, randomised controlled trial.

Specification of Randomised Controlled Trial (RCT) or Other Than Randomised Controlled Trial (Non-RCT)

The study design was classified in 91 (91%) of the studies (including 15 study protocols) as an RCT and in nine trials (9%) not classified as an RCT.

Classification of the Assessed Outcome as Efficacy or Effectiveness

Fourteen (14%) of the examined studies (including one study protocol) claimed that the efficacy of a therapy was measured. Twenty-five studies (25%, including seven study protocols) reported that the effectiveness of the therapy was measured. Most of the studies (60%, including six study protocols) specified neither efficacy nor effectiveness. The terms efficacy and effectiveness were used synonymously in one study protocol.

Specification of the Type of Study as a Pragmatic or an Other Than Pragmatic Trial

The authors of six studies (6%, including two study protocols) used the term pragmatic (mostly in the methods section) to describe the type of study. These six trials also included the term randomised in their titles. Five studies were called “Randomised Controlled Trial”,17–21 and one study was called “Randomized Clinical Trial”.22 One study17 was named a “Pragmatic Randomized Trial”, three studies18–20 (including two study protocols) a “Pragmatic (…) Randomised Controlled Trial”, one study22 a “Pragmatic (…) Randomized Clinical Trial” and one study21 a “Pragmatic (…) Controlled Trial”. None of these six studies explained the meaning of pragmatic.

The authors did not describe the type of study as pragmatic in 94% of the studies, so these are summarized as other than pragmatic trials according to the classification described in the methods section. These 94 studies included 85 RCTs and nine non-RCTs.

Numbers of Described Inclusion and Exclusion Criteria

Both inclusion and exclusion criteria were reported in 69 (69%) of the 100 studies (including 15 study protocols). Only inclusion, but no exclusion, criteria were reported in six studies (6%). No publication reported only exclusion, but no inclusion, criteria; 25 (25%) publications reported neither inclusion nor exclusion criteria. Inclusion criteria were reported in detail in 75 (75%) of the 100 publications (including 15 study protocols). The median number of explicitly reported inclusion criteria in these 75 documents was four with an interquartile range (IQR) of three to five. Exclusion criteria were described in 69 (69%) of the 100 publications; the median number of explicitly reported exclusion criteria was four with an IQR of two to six.

The definition of inclusion criteria is a characteristic of experimental studies, but not of pragmatic studies. In experimental studies, patients with additional risks can distort the results and are therefore excluded from the study. These risks cannot be excluded in pragmatic studies because they are characteristic of real-world conditions.

A more detailed version of the data on inclusion and exclusion criteria in the 100 hrQoL studies, especially concerning the incomplete reporting of this criteria, has already been published.23

Concordance of Terminology: Efficacy vs Effectiveness and RCT/ vs Non-RCT

Table 2 shows the concordance of the use of the terms efficacy or effectiveness to describe the assessed outcome and the classification of the study as a RCT or non-RCT study.

Table 2

Numbers of Studies Assessing Health-Related Quality of Life (n=100) Using Either the Term Efficacy or Effectiveness, Both Terms Synonymously or None of These Terms

Terms used to Describe the Assessed Outcome Dimension	RCT	Non-RCT	Total
Efficacy	14	0	14
Effectiveness	23	2	25
Both terms used synonymously	1	0	1
Not defined	53	7	60
Total	91	9	100

Abbreviation: RCT, randomised controlled trial.

This table shows that the important difference between efficacy and effectiveness is not addressed in studies assessing hrQoL. The term effectiveness describes the effects of a treatment under real-world conditions and should not be used to describe experimental studies, such as RCTs.14,24–30

Concordance of Terminology: Efficacy vs Effectiveness and Pragmatic vs Other Than Pragmatic

Table 3 shows the concordance of the terms efficacy or effectiveness to describe the assessed outcome and the classification of the study as either pragmatic or other than pragmatic.

Table 3

Numbers of Studies Assessing Health-Related Quality of Life (n=100) Using Either the Term Efficacy or Effectiveness, Both Terms Synonymously or None of These Terms in a Pragmatic (Observational) or Any Other Type of Study

Terms used to Describe the Assessed Outcome Dimension	Pragmatic	Other Than Pragmatic	Total
Efficacy	1	13	14
Effectiveness	4	21	25
Both terms used synonymously	0	1	1
Not defined	1	59	60
Total	6	94	100

The reason for classifying a study as an efficacy or an effectiveness trial cannot be derived from this table, which demonstrates that the difference between efficacy and effectiveness was not recognized. Effectiveness should be measured in pragmatic studies, while experimental studies measuring efficacy may not be classified as pragmatic.14,24–30

Summary of the Results

Our results show that most of the studies (91%) were conducted under experimental study conditions in a RCT. No classification of the assessed study outcome as efficacy or effectiveness was specified in 60% of all studies. In 23% of the randomised studies, the authors claimed to assess the effectiveness of a treatment, which cannot be measured under experimental study conditions, but only under pragmatic study conditions. Studies claiming to measure the effectiveness of a therapy must apply a pragmatic study design.

Tables 2 and 3 confirm that there was no relationship between the study design (RCT vs non-RCT), classification of the assessed outcome (efficacy vs effectiveness) and type of the study (pragmatic vs other than pragmatic) in the examined studies. This suggests that the important difference between efficacy and effectiveness was not addressed in a majority of the examined studies. This confirms our study hypothesis.

Discussion

According to Sir Archie Cochrane and Sir Austin Bradford Hill, we should differentiate among three objectives of a clinical study: efficacy, real-world effectiveness and value.15 Efficacy describes whether an innovative intervention can work at all under ideal, experimental study conditions (proof of principle, PoP). Effectiveness addresses whether an innovative intervention that was efficacious will also be effective ie, useful when assessed under everyday conditions (real-world conditions). The third of these objectives describes the observed value from an individual or a societal perspective.28,29 The first objective must obviously be assessed under experimental study conditions (ESC) because all confounders that could influence the effect of an intervention were excluded for PoP. The gold standard for this type of study is a RCT. The results of the second and third objectives will only be meaningful when assessed under real-world, but not under experimental study conditions. A pragmatic study design is necessary to achieve these objectives.

Table 4 shows the differences between experimental study conditions and real-world conditions.

Table 4

Characteristics of Experimental and Pragmatic Studies.*

		Experimental Study	Pragmatic Study
Primary aim		To gain information on the working principle of an intervention	Standardized reporting of real-world effectiveness
Confirmation of		Proof of principle	Effects achievable under real-world conditions
Assessment of		Efficacy	Real-world effectiveness
Conditions for	Applying the intervention	Experimental study (standardised)	Real-world conditions (not standardised)*
	Measuring the outcomes	Experimental study (standardised)	Observational study (standardised)*
Structural features	Inclusion/exclusion criteria	Both need to be defined	Exclusion criteria do not exist
	Method of allocation	Randomisation	Preference-based allocation
	Definition of endpoints	Primary and secondary endpoints	Multiple equivalent endpoints

Notes: The combination of non-standardised and standardised conditions in a common study design was the methodological challenge that was solved by the pragmatic controlled trial14.

Authors of hrQoL studies presumably expect the results of their investigations to generate clinically significant outcomes. This expectation cannot be fulfilled because almost all studies are conducted under experimental study conditions, which are not suitable to describe real-world effectiveness.

Our results confirm our hypothesis that the two terms efficacy and effectiveness are not applied according to their meaning in studies claiming hrQoL as an outcome. If efficacy merely demonstrates proof of principle and effectiveness the suitability of an intervention for everyday use, several undesirable consequences of incorrect terminology can be expected. These consequences are misunderstandings in: scientific discussions, the design of recommendations for action, guidelines, regulations and legal decisions. To avoid these consequences, proposals should be discussed that describe both the conditions and methods required to describe either the results of an experiment or the results of everyday patient care.

The basic effect of a new intervention can be described by a single parameter on survival, but not on hrQoL. Therefore, it makes sense to measure the overall effect of an intervention on survival as the primary endpoint of an experimental study. Measuring the overall effect of an intervention on a common hrQoL endpoint is uninformative for two reasons. The construct of hrQoL consists of different dimensions (eg physical, psychological, social), which are influenced differently by different interventions. The importance of the same quality of life impairment (eg a hand injury) can have different consequences for different people (lawyer or pianist). Therefore, measurements of hrQoL are only meaningful if they consider specific impacts related to the challenges of everyday life. The simultaneous investigation of different equivalent endpoints is possible in a controlled observational study, but hardly possible in an experimental study. Therefore, it should be discussed that the effects of interventions on different dimensions of hrQoL should preferably be measured in effectiveness studies.14,29 For demonstration of effectiveness an experimental study will be inappropriate because randomisation and exclusion of patients do not correspond to everyday conditions.23,29

Strengths and Limitations

One of the strengths of our work is that we address an issue that has not yet been addressed in guidelines for clinical trials reporting results on hrQoL, namely failing to distinguish between efficacy and effectiveness.

Our conclusions are limited by the fact that we cannot offer a definitive solution, although an approach to a solution has been discussed. We only chose a sample of 100 papers. It is unlikely that this problem occurred only in our small sample, since the problem for distinguishing between efficacy and effectiveness has only rarely been addressed elsewhere.