How big does the effect of an intervention have to be? Application of two novel methods to determine the smallest worthwhile effect of a fall prevention programme: a study protocol.

INTRODUCTION: This project concerns the identification of the smallest worthwhile effect (SWE) of exercise-based programmes to prevent falls in older people. The SWE is the smallest effect that justifies the costs, risks and inconveniences of an intervention and is used to inform the design and interpretation of systematic reviews and randomised clinical trials. METHODS AND ANALYSIS: This study will comprise two different methodological approaches: the benefit-harm trade-off method and the discrete choice experiment to estimate the SWE of exercise interventions to prevent falls in older people. In the benefit-harm trade-off method, hypothetical scenarios with the benefits, costs, risks and inconveniences associated with the intervention will be presented to each participant. Then, assuming a treatment effect of certain magnitude, the participant will be asked if he or she would choose to have the intervention. The size of the hypothetical benefit will be varied up and down until it is possible to identify the SWE for which the participant would choose to have the intervention. In the discrete choice experiment, the same attributes (benefits, costs, risks and inconveniences) with varying levels will be presented as choice sets, and participants will be asked to choose between these choice sets. With this approach, we will determine the probability that a person will consider the effects of an intervention to be worthwhile, given the particular costs, risks and inconveniences. For each of the two approaches, participants will be interviewed in person and on different occasions. A subsample of the total cohort will participate in both interviews. ETHICS AND DISSEMINATION: This project has received Ethics Approval from the University of Sydney Human Ethics Committee (Protocol number: 14404). Findings will be disseminated through conference presentations, seminars and peer-reviewed scientific journals.

To determine the smallest worthwhile effect (SWE) of exercise programmes to prevent falls and investigate the relative importance of various factors that influence older people's decision to participate or not in these programmes.

To determine the extent of trade-offs between harms and benefits that older people are willing to accept in making decisions about participation in exercise programmes to prevent falls.

To investigate whether the benefit-harm trade-off method and the discrete choice experiment yield similar estimates of the SWE of an exercise programme to prevent falls.

Despite the clear evidence that the rate of falls in older people can be reduced with exercise interventions, there has not yet been any formal evaluation of whether potential recipients of such interventions consider effects of the magnitude observed in randomised trials to be worthwhile to justify the costs and inconveniences they experience in participating.

Trade-offs between the potential benefits and harms of exercise programmes to prevent falls should be weighed by older people deciding whether to participate or not in these exercise programmes.

The findings of this study should enable the construction of fall prevention programmes with high participation rates, high levels of adherence and high levels of perceived benefit.

This is the first study to use robust stated-preference methods (discrete choice methods and benefit-harm trade-off method) to estimate the smallest worthwhile effect of exercise-based programs to prevent falls.

The results will inform clinical practice, research and policy, as the attributes of an exercise to prevent falls most valued by consumers will be identified.

The estimates of the smallest worthwhile effect should be interpreted in the context of the included population and interventions only (ie, exercise programs to prevent falls in older people compared to no treatment).

Introduction

Randomised controlled trials have become the method of choice for determining the effects of health interventions. More than 500 000 randomised trials have been conducted in health interventions, almost all in the last 50 years. There is a high degree of consensus about how randomised trials should be conducted.1 Nonetheless, several important methodological issues remain unresolved. One of the most persistent issues concerns how to estimate the smallest worthwhile effect (SWE) of an intervention.

The SWE is, as its name suggests, the smallest beneficial effect of an intervention that justifies the costs, risks and inconvenience of that intervention. Estimates of the SWE of an intervention are needed to design powerful and efficient clinical trials and to determine whether interventions produce effects that are large enough to be worthwhile.2 Several approaches have been used to estimate the SWE of intervention. These have been critically reviewed by Barrett et al3 and Ferreira et al.4 The authors argue that any valid measure of the SWE of intervention must have three characteristics: judgements about whether the effects of intervention are large enough to be worthwhile must be made by recipients of care, the estimate must be intervention-specific and the method must focus on the effects of intervention (between-group differences) rather than on changes over time (within-group differences).

Many randomised clinical trials and systematic reviews have investigated the impact of interventions targeted at preventing falls in older people and now provide clear evidence that both the rate of falls and risk of falling in older people can be reduced with exercise interventions, which for instance involve challenge to balance abilities and are performed frequently (ie, at multiple times a week over a long period).5 6 However, there has not yet been any formal evaluation of whether potential recipients of such interventions consider effects of the magnitude observed in randomised trials to be worthwhile to justify the costs and inconveniences they experience in participating.

This paper presents the protocol of a study designed to determine the SWE of a fall-prevention programme. The study will use a modified contingent valuation method (the ‘benefit-harm trade-off method’) and the discrete choice experiment to determine what older people who have previously fallen consider to be the SWE of exercise to prevent falls from their own perspective.

The aims of the study are to answer the following questions:

What do older people who have previously fallen consider to be the SWE of an exercise-based fall prevention programme?

To what extent is the SWE of an exercise-based fall prevention programme influenced by expectations of the costs, risks or inconveniences of intervention?

What characteristics of exercise-based fall prevention programmes do older people who have previously fallen value most?

Do the benefit-harm trade-off method and discrete choice experiment yield similar estimates of the SWE of an exercise-based fall prevention programme?

Methods and analysis

Overview of approach and methods

Different face-to-face interviews will be conducted for each experiment. The first interview concerns the benefit-harm trade-off method and will be used to address aim 1. The second interview will contribute to the discrete choice experiment and will address aims 1–3. A subsample of 60 participants will participate in both interviews to address aim 4. A systematic review of qualitative studies will inform the development of both designs. The review seeks information on the perceptions and experiences of older people on barriers and facilitators of exercise programmes to prevent falls. Data from this qualitative review will be used to determine the potential attributes for both the benefit-harm trade-off and the discrete choice experiment, such as benefits, costs and inconveniences associated with an exercise-based fall prevention programme.

In the benefit-harm trade-off method, these attributes will have fixed levels and participants will decide the smallest expected benefit of intervention for which he or she would consider choosing to have the intervention. In contrast, in the discrete choice experiment, participants are asked to choose between alternatives defined by a set of attributes with varying levels. In this case, we can identify which attributes are driving participants’ preferences, the trade-offs participants make between attributes and how changes in attributes can lead to changes in the patients’ willingness to participate in the intervention.

Participants

Participants will be recruited via newspaper, radio and online media advertisements as well as through community organisations that target older audiences. To be eligible to participate in both the benefit-harm trade-off study and the discrete choice experiment, study participants must meet the criteria below:

Community-dwelling people aged 60 years or over,

Able to comprehend and read English fluently,

Have experienced one or more falls in the past,

Present no obvious cognitive impairment,

Present no serious neurological, cardiovascular or musculoskeletal condition that might hinder their participation in exercise programprogrammes.

Participants will not be excluded based on their participation in exercise programmes (ie, participants who are participating and who are not participating in exercise programmes will be eligible).

Benefit-harm trade-off method

For this approach, the interviewer will describe, for each participant, the potential benefits as well as the costs and risks and inconveniences associated with an exercise-based programme to prevent falls. The participant will then be told to assume that the treatment effect will be of a certain size. Participants will then be asked if, given this expected effect of treatment, they would choose to have the intervention. Subsequently, estimates of the SWE will be elicited by asking the participant to consider the situation in which, hypothetically, the expected effect of intervention is larger or smaller. The size of the hypothetical benefit will be varied up and down in progressively smaller increments until it is possible to identify the threshold benefit of intervention for which the participant would consider choosing to have the intervention. This is the SWE for that participant.

Discrete choice experiment

The discrete choice experiment will be conducted in a way that is consistent with current recommendations.7 This part of the study will use a survey which includes the same attributes associated with an exercise-based programme to prevent falls. Participants will be presented with multiple choice sets of two hypothetical programmes, where the levels of each attribute vary systematically between alternatives and scenarios, and will choose the optimal alternative in each choice set. As it is not feasible to present all participants with all possible combinations of attribute levels, each participant will be presented with a subset of all possible choice sets. An efficient design will be used.8 With this approach, the aim is to present participant choice sets that minimise the elements of the asymptotic variance–covariance matrix of the statistical methods used to analyse the data, that is, the aim is to maximise the precision of estimates of the value of attributes.

From participants’ choices, a mathematical function that describes numerically the value that respondents attach to different choice options will be estimated. This is one way to quantify patients’ preferences for healthcare programmes. The results will be used to estimate the relative value attached to attributes by examining trade-offs that people are willing to make between attributes. As a consequence, it is possible to determine the probability that a person will consider the effects of an exercise-based programme to prevent falls worthwhile, given the particular costs, risks and inconveniences.

Table 1 is an example of a discrete choice study. The example includes nine attributes (costs, transportation alternatives, travel time, type of exercise, frequency, time per day, improvement in the ability to undertake daily tasks at home, improvement in the ability to leave the house to undertake tasks and to socialise and falls risk reduction), with their specific levels. Participants would be asked to choose programme 1 or 2.

Table 1

An example of a discrete choice study

Scenario 1	Exercise option A	Exercise option B
Out of pocket cost ($ per session)	$100	$5
Is transport provided	No transport is provided; you need to provide your own	No need for transport
Travel time	About 45 min	5 min or less
Type of exercise	Combination of different types of exercise (balance and strength training),	Yoga
How often do you exercise per week?	1×/week	1×/week
Time per day	30 min	10 min or less
Improvement in the ease with which you can undertake daily tasks at home (daily tasks at home include bathing, dressing, preparing meals and cleaning the house)	10%	30%
Improvement in the ability to leave the house to undertake tasks or socialise (Tasks include shopping, banking, walking outdoors, using public transport)	30%	60%
Falls risk reduction On average, 30 of 100 older people fall at least once each year. Exercise can reduce the number of people who fall to...	20 of 100	10 of 100

Which would you choose? □Prefer option A □Prefer option B.

The initial design will be tested with a pilot study including community-dwelling older people living in Australia to assess comprehension and understanding of attributes and their levels. From the findings of this pilot study, we will be able to estimate prior parameters, which will inform the efficient design of the final study. Strategies to facilitate appropriate interpretations of the DCE questionnaire will be used, such as diagrams and plain language. If the final design results in a large number of scenarios, the design will be blocked into two blocks to ensure that respondents are not overly burdened by the number of questions.

Sample size

For the benefit-harm trade-off experiment, our simulations indicate that, at least with normally distributed data, a sample size of 60 participants will provide expected CI widths of ±≤0.4 SDs.

For the discrete choice experiment, the current theory of sampling determines that sample sizes are based upon the characteristics of the design of the study, such as the number of attributes included, the attribute level range, the number of hypothetical scenarios presented and the number of alternatives in each choice set.9 Consequently, the sample size cannot be determined until the attributes to be included in the final design are identified. An efficient design, which minimises D-error, will be estimated using NGENE software. D-error is a measure of statistical efficiency of the design (lower D-error indicates greater design efficiency). With an efficient choice design, we expect that a sample size of approximately 200 respondents will be sufficient to answer our questions of interest.

Data analysis

Benefit-harm trade-off method

The distributions of estimates of the SWE elicited in the benefit-harm trade-off will be plotted as frequency histograms. Non-parametric bootstrap methods10 will be used to generate 95% CIs for effects that are considered to be worthwhile by 20, 50 and 80% of participants.11

Discrete choice experiment

A summary of descriptive statistics will be calculated for respondent samples. Data will be analysed with a mixed multinomial logit model (MMNL). With this approach, it is implicitly assumed that preferences do not vary between responses made by an individual respondent, but variation in preferences between respondents is modelled explicitly. Mixed models allow for dependence of observations provided by the same respondent.12–14 The use of MMNLs relaxes the statistical assumptions made with more commonly used fixed effect multinomial logit models and is likely to better explain choice behaviour than fixed effect models.15 16 Interactions between attributes and between attributes and population characteristics (eg, age, gender, education and prior recent experience with falls) will be explored by including the appropriate interaction terms in the model. The analysis willprovide an estimate of the probability that participants consider an intervention to be worthwhile (or, more directly, of the log odds that participants would choose to have the intervention) based on any particular combination of attributes, and also will allow us to examine the trade-offs between attributes that participants are willing to accept. In accordance with the recommendations of Lancsar and Louviere, responses deemed by researchers as ‘irrational’ will not be excluded from analysis, and all available responses will be included in the model.17

Discussion

This project will determine, from the older person's perspective, the SWE of an exercise programme designed to prevent falls. Estimates of the SWE can be used to determine whether the effects of new and existing fall prevention programmes are large enough to justify their implementation.

The outcomes will be important for falls researchers and policy makers because they will provide the first rigorous analysis of the features of falls prevention programmes that might increase the participants’ participation and adherence to these programmes. Therefore, the results could be used to optimise interventions, as they will identify attributes of falls prevention programme that maximise value to the recipients.

This project also has a broader significance as it involves the development of novel methods that are applicable to the design and interpretation of clinical trials across all areas of healthcare. Benefit-harm trade-off studies are simple enough to be routinely conducted prior to clinical trials. This would provide robust, justifiable, empirical estimates of the SWE for use in sample size calculations and the interpretation of trial findings. It is possible that, if these simple procedures were widely adopted, clinical trials could be very different in size; some clinical conditions might be managed very differently, and health resources might be radically redistributed.