Preferences for engagement in health technology assessment decision-making: a nominal group technique with members of the public.

OBJECTIVE: To identify characteristics (factors) about health technology assessment (HTA) decisions that are important to the public in determining whether public engagement should be undertaken and the reasons for these choices.
DESIGN: Focus groups using a nominal group technique to identify and rank factors relevant to public engagement in HTA decision-making. Thematic analysis was also undertaken to describe reasons underpinning participants' choices and rankings.
SETTING: Members of the Australian general public. PARTICIPANTS: 58 people, aged 19-71 years participated in 6 focus groups.
RESULTS: 24 factors were identified by participants that were considered important in determining whether public engagement should be undertaken. These factors were individually ranked and grouped into 4 themes to interpret preferences for engagement. Members of the public were more likely to think public engagement was needed when trade-offs between benefits and costs were required to determine 'value', uncertainties in the evidence were present, and family members and/or carers were impacted. The role of public engagement was also seen as important if the existent system lacked transparency and did not provide a voice for patients, particularly for conditions less known in the community.
CONCLUSIONS: Members of the public considered value, impact, uncertainty, equity and transparency in determining when engagement should be undertaken. This indicates that the public's preferences on when to undertake engagement relate to both the content of the HTA itself as well as the processes in place to support HTA decision-making. By understanding these preferences, decision-makers can work towards more effective, meaningful public engagement by involving the public in issues that are important to them and/or improving the processes around decision-making. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This study uses both nominal groups ranking and thematic analysis to interpret focus group data.

Purposive sampling was undertaken to include a diverse range of people.

Participants, however, were drawn from a metropolitan area in Australia and it may be that the views expressed in this study are not representative of those living outside such areas.

This was a complex topic for participants to understand, and other research in this area would be helpful to explore this topic.

Introduction

Health technology assessment (HTA) is a multidisciplinary process used by healthcare or policy-making bodies (to be referred to here as HTA organisations—HTAOs) to assist with decisions regarding the use, or reimbursement of, a health technology or service. Increasingly, public engagement is seen as imperative in HTA to ensure the legitimacy, relevance and fairness of this decision-making.1–3 Patients, consumer advocacy groups or lay citizens may be involved in this ‘public’ engagement process.4 Approaches to public engagement vary and range from provision of information, opinion surveys, committee representation to more deliberative methods such as consensus panels and citizens’ juries5—with most HTAOs undertaking some level of public engagement.6

Judgements are made by the particular HTAO as to who to engage, when to engage and the method of engagement. The rationale behind these decisions is often not explicit, however, it has been suggested that these judgements are influenced by a number of factors including the topic to be assessed, the culture of the HTAO,7 the stage of the HTA process at which engagement is sought,8 as well as time and resource pressures facing the particular organisation.9 It is also increasingly acknowledged that evidence regarding the effectiveness and/or impact of public engagement on HTA decision-making is lacking.10 As such, there are still many gaps in our knowledge as to what constitutes best practice for public engagement.

Most public engagement research has focused on the experiences of patient and consumer advocacy groups, rather than the lay citizen.11 12 Reviews of public engagement practices4 8 have shown that lay citizens are more likely than patients or consumer advocacy groups to be involved in policy and deliberation processes. The role of the citizen in these processes is often to provide a perspective on overarching methodological, social value or ethical issues. In the field of HTA, this often takes the form of a standing panel13 that provides guidance to an HTAO. By contrast, the views of patients and consumer advocacy groups are often actively solicited in order to add to the understanding of harms and benefits on a particular health technology or service.4 While lay citizens (or what we will hereafter refer to as the ‘public’) are not excluded from such processes, it is often unclear what role they can play in such circumstances given the input of other groups.

Certainly for some decisions it would seem that the public do wish to have input.14 15 These are often where the content of the HTA is controversial, such as where access to pharmaceuticals requires large out-of-pocket expenses to an individual, the condition is severe and/or there is a lack of treatment alternatives, but are these the only circumstances? Many of these factors are similar to the ones identified as important in determining allocation of resources within the HTA process.16–19 We speculate that this may be because the values that underpin both decision-making and engagement are similar,20 with public engagement providing a mechanism to legitimise decisions around priorities in healthcare. The factors for public engagement, however, are likely to have a different focus and may be more about assisting in situations of uncertainly and where trade-offs are needed.21 Furthermore, we anticipate that it will not only be the content of the HTA itself, but how decisions are made (ie, the process) that will influence perceptions about when public engagement is needed.22 This study aims to identify the decision characteristics (or factors) that are important to the Australian public in determining whether engagement should be undertaken in the HTA decision-making process, and the reasons underpinning these choices.

Methods

Study design and participants

Participants were recruited by a market research company. Focus group participants were English-speaking lay members of the Australian public aged 18 years or older. To reflect a broad cross-section of the public as might be recruited for processes such as citizens juries,23 participants were purposively selected to include a range of sociodemographic characteristics, such as education, employment, cultural background, sex and parental status (whether or not the person is a parent, ie, has children). The market research company recruited participants to each group from their existing database to ensure a range of these characteristics. The group meetings were held at two venues and at various times during June–July 2014 to accommodate different schedules. The groups were split according to age (18–35, 36–54 and 55–74 years) to promote rapport in the focus group component as it was anticipated that preferences might vary according to age.24 We aimed to recruit a minimum of six groups as this was expected to achieve data saturation, defined as when subsequent groups do not identify new criteria or concepts relating to the study question.25 Generally, 6–8 participants per focus group is recommended to enable diverse discussion, and allows individuals and opportunity to participate in the discussion. We confirmed 10 participants per group to account for potential non-attendance.

As outlined in the ethics submission, all participants signed a written consent form prior to participating in the focus groups.

Data collection

Following initial pilot testing of the focus group guide, we conducted a series of focus groups with members of the general public. The groups were facilitated by one of the authors (SW). A researcher observer (KH) was present at all the group meetings to take field notes on the group discussions and dynamics. Each 2 h focus group included four parts: (1) open discussion on public engagement practices in HTA, (2) questions about factors that are important in HTA decision-making, (3) a broad discussion and identification of factors affecting decision-making and public engagement and (4) an individual ranking (nominal group) exercise involving the factors identified from the latter discussion26 (see online supplementary file). To give some context and information to the participant, four recent Australian HTA decisions were presented as vignettes.27–30 We included these context vignettes as pilot testing of the focus group questions indicated that participants needed concrete examples of HTA decisions. These vignettes were not intended to be the focal point of discussions but to give participants an idea as to the content and process of Australian HTA, and to stimulate discussion. As such, the emphasis was on national HTA decision-making.

In the nominal group exercise, each member of the group was asked to suggest the factors that they considered the most important in determining whether public engagement in HTA should be undertaken, and to give a reason for this choice. These were listed on a whiteboard or flipchart. Once each group member had contributed, the facilitators discussed each factor to ensure that participants had a consistent understanding of the listed factors.31 Depending on the group, some reframing for clarity and/or amalgamation of overlapping or duplicative factors occurred at this stage. The notes from this clarification phase were also used later to develop the master list where some further grouping of duplicative items occurred. Following this phase, participants were asked to rank on a sheet provided, what they thought was the most important factor (ranking of ‘1’) to the least important factor (ranking of ‘10’) influence public engagement from the list generated by their group.

All groups were recorded and transcribed verbatim. Data collection continued until no new factors or concepts emerged. This occurred by the sixth focus group, when no new factors were identified.

Data analysis

Nominal group ranking

To calculate the nominal group ranking, scores were assigned by one of the researchers to the factors listed by a participant. The most important factor was assigned a score of 10, the next factor a 9, and so on; with the last factor (the ‘least important’) having a score of 1. Items of the groups’ common list that were not included in the top 10 by an individual were given a 0. A master list of all factors was created by combining the common lists of all six groups. Analysis of the multiple group data was undertaken comparing the frequency and proportion of participants who included a factor in the top 10. A mean priority score for each factor was also calculated,31 by adding the scores from the ranking exercise for each factor and dividing this by the maximum score possible for the factor.32 We did not compare across group stratification due to the small sample size.

Thematic analysis

A thematic analysis33 was undertaken on the qualitative data from this exercise, and the broader discussion around decision-making and public engagement prior to this exercise. This has been shown to be helpful in interpreting data between nominal groups31 and has been used by other studies to assess preferences in healthcare settings.32 34 Using the methods of Braun, Clarke and Terry,30 one researcher (SW) read and re-read the transcripts. The preliminary analysis was discussed with coinvestigators KH and AT who were present at the focus groups, and/or had read the transcripts (investigators triangulation) to ensure that the analysis reflected the full breadth and depth of the data collection. Through a process of comparison and review of the data, themes were then inductively developed and refined in terms of definition and scope until agreement was reached.

Results

Patient characteristics

Of the 60 participants recruited, 58 participants attended six focus groups. In four groups, there were 10 participants each, with the remaining two groups having nine participants due to non-attendance of 2 participants. No reason was given for non-attendance. The age of participants ranged from 19 to 71 years (mean age 46 years), 28 (48%) were women and 36 (62%) were in full-time employment. Most participants had completed tertiary study (71%), and just over half of participants (52%) indicated that they had children and/or dependents (table 1).

Table 1

Participant characteristics

Characteristics	Number of participants (n)	Percentage
Gender
Male	30	52
Female	28	48
Age, years
18–24	6	10
25–34	11	19
35–44	12	21
45–54	9	16
55–64	14	24
65–74	6	10
Cultural background identified as Australian
Yes	40	69
No*	18	31
Employment
Full time	36	62
Part-time	13	22
Not working/studying/retired	9	16
Highest level of education
High school	17	29
University/college	27	47
Postgraduate	14	24
Parental status with dependents/children
No	28	48
Yes	30	52
Total	58

*Those who were born overseas, or those born in Australia who identify with another cultural heritage.

Nominal group ranking

Each group generated 9–15 factors, with 24 unique factors identified across the six groups (table 2). The only factor common across all groups was size of eligible population. The majority of factors was identified by three or more groups. Only one factor (background for assessing health technology) was not ranked by any participant as being in the top 10 most important (priority score of 0). It is worth noting that while table 2 gives an indication of the factors that the public consider important in determining whether public engagement should be undertaken, it does not indicate any directionality of the factor. For some factors, the size or ‘level’ of the factor determined whether people wanted more or less engagement (marked by * in table 2).

Table 2

Factors determining importance of public engagement in health technology decision-making: modified nominal group results sorted by mean priority score

Participant-identified factor	Number of groups	Number of participants voting on factor	Frequency of factors in top 10	Proportion of participants ranking in top 10 (%)	Mean priority score
1. Effectiveness (success rate) of the health technology	4	38	35	92	6.9
2. Potential health gain from health technology	5	48	46	96	6.2
3. Quality of HTA decision-making process	3	30	27	90	6.2
4. Comparative cost and benefit	2	20	17	85	5.5
5. Size of eligible population to have access*	6	58	54	93	5.2
6. Purpose of the technology (life saving, relieving symptoms)	4	38	28	74	5.0
7. Changes in quality of life	5	48	37	77	4.8
8. Reason for the condition (role of lifestyle)	1	9	5	56	4.8
9. Cost to patient (out of pocket)	4	38	29	76	4.6
10. Societal impact	3	30	25	83	4.3
11. Cost to taxpayer	4	38	32	84	4.0
12. Quality of evidence	5	48	30	63	4.0
13. Type of heath condition	1	9	6	67	3.9
14. Severity of condition	4	39	30	77	3.8
15. Likelihood of side effects	3	28	21	75	3.6
16. Age of eligible population*	5	49	36	73	3.6
17. Awareness of condition in community	1	10	8	80	3.5
18. Prevention (avoiding future costs)	3	28	17	61	3.2
19. Availability of alternatives	5	48	34	71	2.9
20. Productivity gains	2	20	14	70	2.6
21. Health priority	2	20	11	55	2.5
22. Uncertainties in evidence (benefit)	3	29	13	45	1.9
23. Characteristics of the health technology	1	10	3	30	1.0
24. Background for assessing the health technology	1	9	0	0	0.0

*Key factors where the directionality varied.

HTA, health technology assessment.

On the basis of frequency, the top-ranked factor was size of the eligible population affected by the decision, with 54 participants ranking this in the top 10. Other factors included potential health gain, changes in quality of life, age of the eligible population, effectiveness (success rate of health technology), availability of alternatives, cost to taxpayer, quality of evidence, severity of condition and cost to patient. On the basis of the proportion of participants voting for factors and the frequency, six common factors emerged: potential health gain (46 participants ranked this in the top 10 representing 96% of participants ranking this factor in the top 10), size of eligible population (93%), changes in quality of life (77%), effectiveness of health technology (92%), cost to the taxpayer (84%) and severity of the condition (77%). There were four factors that while not as frequently mentioned across the groups, were factors that particular groups regarded as important and were ranked in the top 10 by 80–90% of participants in those groups (table 2). This included awareness of condition in community which was only identified by one group, but was included in the top 10 by almost all (80%) the participants in that group.

Similarly, when looking at the mean priority score across the factors, a slightly different set of factors emerge. While potential health gain (mean priority score 6.2), size of eligible population (mean priority score 5.2), changes in quality of life (mean priority score 4.8) and effectiveness (mean priority score 6.9) are still in the top-ranked factors, purpose of the technology (eg, is it life saving or for relieving symptoms), and reason for condition are also included. This indicates that for a small number of people these factors were assigned a greater value as they were particularly important. This was particularly evident for the factor pertaining to quality of the existing process (mean priority score 6.2). Issues around the quality of the HTA decision-making process were raised in three groups, with 90% of respondents (27/30) ranking this in the top 10. Out of those 27, 8 (30%) voted this factor as the most important in determining whether public engagement should be undertaken.

As noted above, the ranking results do not give an indication of any directionality of the factor which is an important part of the interpretation. For example, in discussing size of the eligible population, some respondents felt that engagement was better placed for HTA topics that impacted on a large section of the population. Others felt that for these topics, the community did not need to be involved, but instead, engagement was needed for rare conditions or those with less of a ‘voice’ (less awareness in the community). The above analysis also does not take account of the interconnectedness between some of the factors, such as size of population and awareness of the condition in the community. These issues are addressed in the thematic analysis.

Thematic analysis

Four themes explaining participant's choices and rankings, and ultimately preferences regarding the extent and type of public engagement in Australian HTA decision-making were identified. These were: perceived value of the technology; impact of the HTA decision on both the individual and society; confidence in the procedural justice of the HTA decision-making process (ensuring fairness, transparency and quality of the process); and dealing with uncertainties in the HTA assessment process. These themes could be categorised more broadly as relating to either the content of the HTA itself or to the context and process of decision-making. Each theme is discussed below with illustrative quotes from participants.

Perceived ‘value’ of the health technology

The term value was used by participants in describing the trade-off between clinical benefits (and potential harms) and the costs (individual and society) of the health technology, and was often framed in terms of the evidence on a health technology. Some participants thought public engagement would be more appropriate in decisions where the size of the potential health gain was small and/or had a positive effect on quality of life including an individual's ability to work. ‘Public’, in these circumstances, was interpreted as patients providing detail of the benefits of treatment. However, this perspective was broadened when factors were discussed in the context of the disease, particularly the size of the population and the purpose, as well as the relative cost.

[When]. it's not very big effect, and we're talking about a lot of money, I think then it's more important to have public engagement to make the decision. (age 18–35 years)

Impact of the HTA decision on both the individual and society

Participants explained their choices in relation to the effect or ‘impact’ of the condition—and potential decision—on an individual, the individual’s family, and the community. Demographic and epidemiological characteristics of a disease, such as the severity of the disease and the size of population were raised, as well as characteristics and clinical place of the technology (purpose, availability of alternatives, quality of life). In general, participants were less inclined to think public engagement was warranted when there were alternatives available, and when the condition was neither severe nor impacting on an individual's survival.

When there is loss of life then certainly there should be more questions, more engagement from the general public. (age 18–35 years)

Participants were also more inclined to consider public engagement in decisions where a health technology had an impact on an individual’s family, for example, if a health technology resulted in a decrease in carer burden or the ability to work. Large out-of-pocket expenses were also viewed as a potential trigger for public engagement given such costs could have an impact on an individual's disposable income and, therefore, quality of life. Public engagement was also considered more valuable in situations where public reimbursement would have a substantial financial impact on the government and taxpayers. Conversely when costs to patients and government were small, some felt public engagement was less warranted.

When it's sort of such small bickies in terms of the overall costs…maybe stop wasting time even talking about that one and focus on something else. (age 55–74 years).

Confidence in procedural justice of the HTA decision-making process

Ensuring fairness in the system

A number of participants expressed that public engagement should be used to make sure there is a fair chance for all. This encompassed issues, such as awareness of the condition of the community, age, social responsibility or reason for the condition.

A common sentiment across the groups was that only those with a personal ‘stake’ in the technology would be interested in the decision-making process, and as such conditions that did not have a high profile in the community, and where patients’ views may be difficult to obtain, could potentially be disadvantaged. In such circumstances, participants indicated a willingness to be involved.

I would become involved with those where there are not many affected but need a voice. (age 55–74 years)

Transparency of the process

Some participants were of the opinion that the public should be entitled to have a say in decisions involving distribution of taxpayers’ money.

We're taxpayers, so I think that we assume that we should be informed about where out taxpayer dollars go. (age group 36–54 years)

However, others felt that additional public engagement was not needed if there was already a fair and transparent system in place.

Well the process has to be transparent, we have to know what's going on, and maybe we're finding out for the first time there is a process. So what about the rest of the community, they would have no knowledge of this so how can they have a representative, or how can they actively and aggressively debate the issues. (age 55–74 years)

If I had confidence in the system then I'd happily sit back and go well you know I'm not going to agree with every decision but competent people have been discussing the issues the way they should. (age 55–74 years)

This also extended to the quality of the public representatives. Some participants expressed that additional public engagement would not be necessary if the public representatives in the HTA process were vocal, connected to the community that took into account patients’ views and had the same level of influences as professionals within the decision-making process.

It's probably more about the quality of the representative. (age 36–54 years)

Dealing with uncertainties in the assessment process

Public engagement was deemed more important if the existing evidence was weak, conflicting or where there were uncertainties in the evidence. For example, inconsistencies around the direction of effect, or only short-term follow-up. Other participants noted that when evidence was of good quality then public engagement may not be necessary.

I'm happy to come back in six months’ time when you've got more evidence, and we might approve your drug. (age 36–54 years)

When there is better quality evidence I will be less inclined for the need to have public engagement, because you can say “We have this evidence, we don't need to ask you, it's kind of obvious, let's do it”. (age 18–35 years)

Alternatively, others felt that comments from the public might serve as another form of evidence to identify those issues that really matter to patients, such as changes in quality of life.

Discussion

Our study shows that there are multiple factors that are important to the Australian public in determining whether public engagement should be undertaken in HTA decision-making. Our study also shows that these factors are not solely about the content of the HTA but also about the HTA decision-making process. Participants indicated that they thought public engagement was important in situations where there were trade-offs between effectiveness and costs, and where such decisions had an impact on individuals and their quality of life. Public engagement was also considered to be more important when the existing decision-making processes did not already take into account the views of those impacted.

Few empirical papers have been published in the HTA field on this topic.35 36 Often studies address factors in isolation2 37 without acknowledging the interaction between the factors. One of the strengths of this study is the linkage between quantitative rankings and the exploration of participant's reasons for ranking of factors. Perhaps related to this work is the field of priority setting in HTA, which has grown substantially in recent years.38 Most of the reported findings in this field have been in relation to ‘content’ values. This encompasses issues around clinical effectiveness, cost effectiveness and social value judgements around equity such as age of population and severity of disease.22 For example, in a recent study39 using a citizens jury to develop a set of criteria to guide priority-setting, three key criteria were identified; potential to benefit a number of people, potential to extend life with quality, and potential to improve quality of life. A recent Australian study undertaken to assess public preferences for funding new health technologies also found that individuals preferred technologies that benefited large numbers of people, as well as providing value for money.40 Similar results were also found in a New Zealand review conducted to assess the factors that the public wanted considered in funding health technologies.41 The final report listed four dimensions (need, health benefits, costs and savings and suitability), as well as three levels of impact (to the patient/family, society and health system).

While the results from the nominal group ranking and thematic analysis echo some of these findings, they also highlight that engagement is needed in situations such as where ‘value’ needs to be determined, and this ‘value’ needs to be determined by the public, and take into account both the benefits of a technology and the potential impact of the health technology on the individual and society. Uncertainty also appears to be a driver for the public to want to be engaged in HTA decision-making. Other authors have noted that as the demand for new innovative technologies increases, discussing uncertainty will need to become a fundamental part of any transparent and accountable healthcare system,42 as will issues around affordability. The thematic analysis also highlighted the impact of process in influencing preferences around engagement. For decision-makers and policy-makers this is important as it indicates that by building a system that the public trusts to be transparent and fair, that additional public engagement may not be needed.

Further methodological exploration of the trade-offs is needed to help identify those values that are important to decision-making, and whether these trade-offs and values differ between different countries’ health systems.22 This study is part of a larger mixed-methods study on public engagement, and HTA decision-making26 that aims to determine quantitatively, using discrete choice methodology, which factors influence the public's preferences on when and how public engagement should be undertaken for HTA decisions. The factors identified in the focus/nominal groups will be used to construct a DCE questionnaire where individuals will need to consider multiple factors. Additional items around ‘how’ the public is engaged in HTA decision-making will also be included in the DCE questionnaire. This will allow us to ascertain if the public considers that different public engagement approaches are needed depending on the particular content and process values underpinning a HTA decision. Despite a significant body of literature of public engagement, little empirical and evaluative data exists for decision-makers to draw on in order to make such choices around engagement approach. Public engagement is a resource-intensive exercise,43 and it would be good to have evidence that some of the approaches being proposed in the HTA field44 have the support of the public.

Limitations

This was a complex topic to cover with participants unfamiliar with the HTA decision-making area. Pilot testing of the questions with a convenience sample indicated that participants needed a significant amount of guidance to understand the topic. While the focus group questions were centred on public engagement, they were situated in the context of HTA decision-making. For this reason, the criteria for whether the public should be engaged may reflect some of the criteria for decision-making. There had also been considerable media attention to health system changes at the time the focus group meetings were conducted, which may have led to an overemphasis on ‘process’ issues. Further empirical work using other approaches, such as interviews, might offer additional understandings in this area, and could be used as a comparison to our results. The population, though purposively sampled to reflect a diversity of views, did not include people from rural or remote areas, non-English speakers, and thus, it is uncertain if the findings are transferable to these populations.

Conclusions

Public preferences for when engagement should be undertaken involves consideration of factors relating to both the perceived value of the health technology and uncertainties in the HTA evidence assessment process. This is in addition to the impact of the HTA decision on both the individual and society and the public's confidence in procedural justice of the existing HTA processes. Research acknowledging the complexity of HTA decision-making and the interactions between factors is needed in order to better determine the drivers influencing the public preferences for the extent, or type of public engagement. In determining this, healthcare agencies and decision-makers can better tailor public engagement methods to optimise input from the public to where it matters most.