Stakeholder engagement in economic evaluation: Protocol for using the nominal group technique to elicit patient, healthcare provider, and health system stakeholder input in the development of an early economic evaluation model of chimeric antigen receptor T-cell therapy.

INTRODUCTION: Chimeric antigen receptor T-cell (CAR-T) therapy is a class of immunotherapy. An economic evaluation conducted at an early stage of development of CAR-T therapy for treatment of adult relapsed or refractory acute lymphoblastic leukaemia could provide insight into factors contributing to the cost of treatment, the potential clinical benefits, and what the health system can afford. Traditionally, stakeholders are engaged in certain parts of health technology assessment processes, such as in the identification and selection of technologies, formulation of recommendations, and implementation of recommendations; however, little is known about processes for stakeholder engagement during the conduct of the assessment. This is especially the case for economic evaluations. Stakeholders, such as clinicians, policy-makers, patients, and their support networks, have insight into factors that can enhance the validity of an economic evaluation model. This research outlines a specific methodology for stakeholder engagement and represents an avenue to enhance health economic evaluations and support the use of these models to inform decision making for resource allocation. This protocol may inform a tailored framework for stakeholder engagement processes in future economic evaluation model development. METHODS AND ANALYSIS: We will involve clinicians, healthcare researchers, payers, and policy-makers, as well as patients and their support networks in the conduct and verification of an early economic evaluation of a novel health technology to incorporate stakeholder-generated knowledge. Three stakeholder-specific focus groups will be conducted using an online adaptation of the nominal group technique to elicit considerations from each. This study will use CAR-T therapy for adults with relapsed or refractory B-cell acute lymphoblastic leukaemia as a basis for investigating broader stakeholder engagement processes. ETHICS AND DISSEMINATION: This study received ethics approval from the Ottawa Hospital Research Institute Research Ethics Board (REB 20200320-01HT) and the results will be shared via conference presentations, peer-reviewed publications, and ongoing stakeholder engagement. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Three stakeholder-specific focus groups will be conducted using an online adaptation of the nominal group technique to incorporate diverse perspectives in an early economic evaluation of a novel cancer therapy.

Stakeholder engagement, including the non-traditional engagement of patients and their support networks, can provide additional insight into variables that should be considered in an early economic evaluation, which may enhance its validity and generalisability.

Incorporating stakeholder input in an early economic evaluation will inform future research and development, and enable a realistic commercial viability estimate of a new cancer therapy.

Additional stakeholders, such as the general public, would need to be included for the implementation of the health economic model results and is outside the scope of this study.

Introduction

Global projections estimate that the cancer burden will exceed 29 million new cases by 2040,1 more than doubling the number of new cases worldwide in 2012.2 With significant increases in the cost and extent of cancer therapies,3 4 the annual direct costs for cancer therapies is likely to mirror these growing trends. In Canada, the total direct costs of cancer care have increased substantially from $2.9 billion in 2005 to $7.5 billion in 2012 (in constant 2015 dollars),5 and the incidence rate of cancer in Canada is projected to increase by over 80% between 2012 and 2042.6 The nexus of increasing cancer rates, rising healthcare costs, and limited funds emphasises the growing importance of robust and rigorous assessments of cancer treatment value.

Health economic (HE) evaluation can support healthcare decision-makers in allocating limited resources in a way that maximises the health of the overall population and optimises healthcare spending. Outputs from HE models can be used by healthcare payers, including regional government agencies—such as Cancer Care Ontario in the Canadian context—to facilitate determination that new interventions represent or do not represent ‘good value for money’. The validity of such models is of significant importance to ensure good modelling practice and well-informed decision making.7–12 Current guidelines for HE modelling emphasise the scientific credibility of HE models, such as through the transparent methodological reporting of model development, application, and validation; however, evidence from other fields (eg, environmental sciences and conservation) demonstrates the importance of involving key stakeholders in good modelling practice,13–22 including the incorporation of stakeholder perceptions of model salience and legitimacy.13 Parallels exist between environmental and HE modelling in that these models bridge the science-policy gap and support decision-making. The benefits reaped in the field of environmental modelling from stakeholder engagement, such as increasing model validity, advancing methods in the co-production of knowledge, and contributing to shared decision making, facilitating easier and improved decisions, represent untapped potential in HE modelling; however, the relevant stakeholders in HE modelling may include specific, and potentially very small subsets of the population, and with a dearth of evidence, it is unclear whether these benefits will translate to this field of modelling.

In most cases, HE models are applied to established health technologies and treatments that are ready for introduction into clinical practice; however, there is a growing literature advocating for the application of HE models in the early phase of healthcare innovation development as a means to enhance research and development, refine the size and breadth of the target population for treatment, and inform reimbursement scenarios for future market access.23–25 This research suggests that economic evaluation should be a continuous (iterative) process, and should start from an early phase evaluation and progress to a late phase evaluation based on comparative effectiveness research evidence, rather than waiting until the latter to consider HE implications.23 Resultantly, early HE evaluations, a central component of early health technology assessments, have applications to inform decisions made by health policy-makers and industries. In the case of health policy, early assessments provide insight into the potential impact of emerging technologies to inform future policy and market access, whereas industry gains from insight used to inform product research and development decision making.26

Chimeric antigen receptor T-cell (CAR-T) therapy is a class of immunotherapy that relies on re-engineering a patient’s T-cells to target tumour-expressing antigens in the treatment of cancer. The US Food and Drug Administration has approved two CAR-T therapies, axicabtagene ciloleucel and tisagenlecleucel.27 28 Anti CD19 CAR-T therapy has shown promising results with durable responses in adult relapsed or refractory (r/r) diffuse large B-cell lymphoma and paediatric B-cell acute lymphoblastic leukaemia.29–34

An early economic evaluation can support product investment decision-making, which is pertinent to CAR-T therapy, as it exists in a complex intellectual property landscape and funding for clinical trial support in Canada is likely to rely on non-commercial resources. The application of this model will help identify particular patient and intervention characteristics that will make CAR-T therapy reimbursable. Additionally, there is growing interest in investigating the effect of CAR T-cells in other types of cancer, as well as in autoimmune and other diseases. A database search for open CAR-T therapy trials for cancer on ClinicalTrials.gov conducted in mid-2020 returned over 300 results. If additional therapies are approved, healthcare systems may not be able to meet the costs of the potential increase in the number of eligible patients, as the promising therapeutic benefits will come at a cost. For example, the market price for one-time administration of tisagenlecleucel is US$475 000 and US$373 000 for axicabtagene ciloleucel in the USA.27 35 These prices are, however, for the drug products alone. When other related costs, such as the fees for hospital stays, supportive care, or physician visits are considered, the total cost of CAR-T therapy increases.36

Individuals with direct experience, such as patients and their support networks, are the most familiar with other related costs of treatment that can often go overlooked in HE evaluation. A growing recognition of the need to improve patient and public engagement in research has resulted in the proliferation of frameworks that aim to characterise engagement processes37–41 and classify the scope of engagement.42–46 The International Association for Public Participation (IAP2) presents one example of a continuum of engagement. Using the IAP2 continuum of public engagement, involve is defined as working ‘… with the public to make sure concerns and aspirations are considered and understood’.45 Involving knowledge users and individuals with a vested interest in the research findings as stakeholders in the conduct of an economic evaluation is thus understood as a form of engagement that aims to ensure their perspectives are reflected in the evaluation, its associated outputs, and its subsequent contribution to healthcare decision-making. In the case of models and frameworks designed to conceptualise engagement processes, irrespective of where they may fall on a continuum of engagement scope, evidence of limited transferability suggests value in the development of context-specific frameworks.47

Traditionally, stakeholder engagement in economic evaluations has focused on consulting industry representatives, policy-makers, and reimbursement decision-makers. These stakeholder groups represent the key knowledge users of the outputs of HE evaluations. Whereas these professional groups are dominant in the use of HE evaluation results, other stakeholders, such as patients and their support networks, are directly affected by the resulting decision-making of these groups. Involving stakeholders, such as clinicians, patients, and their support networks in the development of HE evaluation models can provide additional insight into a model structure, input parameters, and assumptions that may enhance the validity and generalisability of the economic model.

Stakeholder engagement in the broader field of health technology assessment, with which HE evaluation is a part, is not novel and represents an area of work that has received significant focus in health technology assessment literature and practice alike. Resultantly, there have been calls to involve health professionals, as well as patients and their support networks in HE evaluation studies; to date, few have done so.48–51 Overcoming the predominating value-free ideal that posits epistemic values (ie, predictive accuracy, internal coherence, external consistency, unifying power, fertility, simplicity) at the centre of scientific inquiry, while excluding non-epistemic values (eg, political, moral, social or religious values), is cited as a significant hindrance to involving stakeholders and identifying value judgements in HE evaluation modelling.50 Despite validation of HE evaluation models representing a point of progression, challenges to opening the ‘black box’ that is HE evaluation modelling to lay audiences contributes to a relative stagnation in the involvement of all stakeholders, including those not traditionally considered, in HE modelling processes.49

Without comprehensive input from all key stakeholders, a HE evaluation model may use insufficient or even inappropriate assumptions.52 For instance, oncologists and haematologists are familiar with therapeutic options for treating blood cancer and the sequences of therapy, and can inform the scope and inputs of the economic model with nuances that require clinical experience. Additionally, patients and their support networks are familiar with the resources they require and use to receive treatment, and are best placed for describing outcomes beyond recurrence and mortality, for example, side effects and quality of life. Involving these key stakeholders in early HE evaluation model development will provide additional insight into the assumptions, inputs, and outputs of the model, which may enhance model validity and help product developers and providers to consider the downstream cost of factors not otherwise considered.

Study objectives

This research aims to involve healthcare professionals, policy-makers, and patients and their support networks to inform the development of an economic model for an early HE evaluation (including cost–minimisation analysis, cost–benefit analysis, cost-effectiveness analysis, and cost–utility analysis). We will use this opportunity to explore which stakeholder groups should be engaged in early HE evaluation processes and the scope and nature of this engagement, which may set the foundation for such inclusive approaches in future HE evaluations. This will be achieved by the following study objectives:

Verify the comparators/treatment strategies being considered in the early HE evaluation, as well as the proposed HE evaluation model assumptions.

Generate stakeholder-specific knowledge to inform model development based on the experiences and expertise of key stakeholders (clinicians and researchers, healthcare payers and policy-makers, and patients and their support networks).

Identify the steps and modality with which key stakeholder groups can and want to be engaged in the development of early HE evaluation models, including the anticipated barriers and enablers to doing so.

Methods

We will engage stakeholders to generate stakeholder-specific knowledge to inform the development and verify the economic evaluation model of CAR-T therapy. We will use three different stakeholder-specific online video forums and in-person forums (when permitted considering limitations to congregation during the 2020 global COVID-19 pandemic); at least one each with (1) clinicians and researchers, (2) healthcare payers and policy-makers, and (3) patients and their support networks. An adapted version of the nominal group technique (NGT), a structured, consensus-building discussion approach, will be used to strengthen, identify, and prioritise additional model considerations in a collaborative manner.53 This engagement is highly structured, emphasising a balance across participants in providing opportunities to equally contribute to the conversation and diminishing power dynamics between the research team (which includes stakeholders) and stakeholders, and among stakeholders themselves. The NGT provides a format to generate relevant qualitative data through participants’ responses to predetermined questions, supporting the presentation of different ideas to achieve consensus in the generation of solutions, ideas, and priorities. The feasibility and utility of this approach will also be explored with each of the stakeholder groups; alternative modalities of engagement may be explored dependent on stakeholders’ preferences and availability and the realities of the COVID-19 pandemic. Integrating priorities from stakeholders with a vested interest and first-hand experience with cancer therapies, such as CAR-T therapy, will enhance understanding of diverse considerations in the development of a relevant and comprehensive economic model.

The group discussion will be conducted using real time, online synchronous communication with videoconferencing software (ie, Zoom) while limitations exist to in-person group congregation as a result of the COVID-19 pandemic. Online group discussions are prone to technology challenges, which may result in lagging, internet drop-out, and interruptions. As a result, a smaller number of participants is optimal when using a video-based format,54–56 aligning well with standard NGT methods. Considerations to enhance group discussions using videoconferencing have been highlighted in the literature and will be incorporated into discussion planning; these include limited group numbers, accounting for participants’ technology preferences, encouraging environmental contexts conducive to research participation, and ensuring the confidentiality and consent of research participants.54–56 While certain factors contribute to unique challenges that might be faced in using videoconferencing software, such as limited participation of those who do not have access to the internet or videoconferencing software, this platform offers advantages in reducing geographical constraints to recruitment and eliminating travel time to physical spaces for the discussions. Moreover, data richness has been demonstrated to be comparable between videoconference and in-person group discussions and greater for videoconferencing than online text-only focus group discussions,55 57 the current leading approach in applying the NGT using multimodal online platforms.58 59

The nominal group question guides were drafted and reviewed by research, patient, and knowledge user study team members, piloted, and modified to ensure clarity, sensitivity, and focus (online supplemental appendix 1). The questions are designed to elicit reflection on the important considerations, including model inputs and outputs, that should be taken into account in the development of the HE evaluation model of CAR-T therapy to ensure stakeholders’ experience and expertise are represented. The discussion guides are tailored to each stakeholder group to reflect differing levels of knowledge and exposure to economic evaluation processes and stakeholders’ areas of expertise.

Each stakeholder group will be introduced to CAR-T therapy, the proposed HE evaluation and decision-analytical model, the existing model assumptions as they stand based on standard HE modelling, inputs, and outputs considered, as well as the objectives of the session. Stakeholder discussions will be audiorecorded and are expected to last 2–3 hours to allow for adequate discussion and generation of comprehensive responses to the posed questions, using the following steps for each group, consistent with standard NGT methods.

Idea generation: Initial silent brainstorming of ideas.

Round Robin: The facilitator will go around the (virtual) table and ask each participant to list one factor to be considered in the development of the model (the facilitator will write participant responses down for all to see using a shared screen/poster board). This will proceed for each participant, one at a time, until no new ideas emerge. Participants can think of new ideas as they hear what others say, but must wait their turn before sharing them. This step is intended to rapidly capture ideas and provides everyone with the same level of input; there is no discussion at this point.

Clarification and grouping of similar content: Ideas are clarified and adjusted where needed, providing an opportunity to group/combine similar ideas. We are not looking for agreement at this point.

Ranking: Participants are asked to individually rank all of the proposed additional inputs, outputs, and model adjustments (ie, the ideas captured in the previous step) after the group discussion using an online feedback form.

Results discussion: Participants are invited to provide their comments and thoughts on the final ranked list once everyone has completed the online ranking exercise via email.

Following the stakeholder group discussion and based on the discussion outputs, a short follow-up feedback form will be developed specific to each stakeholder group (online supplemental appendix 2). Using the stakeholder inputs generated from the group discussions, the research team will document the ideas presented; this list will be cross-checked with our knowledge user and patient partners. Participants will then be asked to individually rank the idea groupings that were generated in the discussion, as well as the individual proposed inputs, outputs, and model adjustments (ie, ideas) using the feedback form. This form will serve to prioritise stakeholder-identified model considerations, as well as to gather feedback about the experience and modality of stakeholder engagement in the HE evaluation of CAR-T therapy. Respondents will also be asked to provide demographic information (eg, for patient stakeholders, information about their diagnosis, disease status, treatment(s), and socioeconomic status) to better describe our sample. Following completion of the feedback form, the results of the participants’ rankings will be tallied and shared with participants who indicate their interest for asynchronous comment via email to provide insight into the intricacies underlying the prioritised list.

Patient and public involvement

The research project was developed in close cooperation with our patient partner throughout several project meetings; this included collaboration in setting the research question, determining the study design, and informing the recruitment strategy, consent process, and planned analysis to produce meaningful results and limit the undue burden on potential participants. We have reported the patient and public involvement in the development of the study protocol using the Guidance for Reporting Involvement of Patients and Public (GRIPP) 2 short form (online supplemental appendix 3).60 Ongoing collaboration with patients and their support networks will be a central component to enable effective mobilisation of the research findings. We will use the GRIPP2 form to report patient and public involvement in the subsequent results papers.

Sample

Stakeholder group discussions will be held separately with three different Canadian stakeholder groups: (1) clinicians and researchers (ie, medical oncologists and haematologists, and clinical, health services, and health policy researchers), (2) healthcare payers and policy-makers (ie, public and private healthcare payers and individuals who influence drug reimbursement), and (3) patients with haematological cancer and members of their support networks. If additional stakeholder groups are identified during these discussions, the research team will remain flexible in hosting these discussions and will report this with the research findings. Research participants will be based in Canada and, as such, will provide perspectives rooted in the context of Canadian and Provincial/Territorial healthcare systems. Consistent with NGT methods, we will aim to recruit approximately seven participants for each stakeholder group discussion.53 We will employ a recruitment approach that aims to encourage participation across a diverse range spanning age (18+), gender, ethnicity, and socioeconomic status to develop an exhaustive list of stakeholder-generated ideas.

For healthcare professionals, payers, and policy-makers, we will aim to include individuals with diverse experience levels and from various roles to ensure we capture the full breadth of stakeholders’ inputs to be incorporated into the development of a comprehensive HE evaluation model.

Recruitment procedure

We will leverage virtual conferences and the mailing lists of national academic and clinical conferences dedicated to haematological cancer-related topics to facilitate study recruitment. Haematological cancer-related conferences bring together health professions, researchers, payers and policy-makers, as well as patients and their support networks.

We will supplement this recruitment strategy with convenience sampling using the project team’s professional networks. We will use existing contact lists and the professional networks of the clinical and research collaborators on the project to identify a master list of prospective healthcare clinicians, researchers, payers, and policy-makers with experience and expertise in working with haematological cancers from hospitals and research institutes across Canada. The project collaborators will contact health professionals in their networks by email to introduce the study and invite them to participate. We will also use snowball sampling, whereby each contacted health professional will be asked to invite any of their colleagues who may be interested in study participation and may have differing views. This will be supplemented with the identification of clinical investigators using our systematic review of trials of CAR-T therapy and a review of active CAR-T therapy trials on ClinicalTrial.gov.61

We will use several recruitment strategies developed in close collaboration with our patient coinvestigator (TH) to identify and recruit haematological cancer patients (with and without experience with CAR-T therapy due to the novelty of this treatment) and members of their support networks:

Via ongoing clinical care: Patients will be approached through their circle of care. Using snowball sampling, healthcare professionals and primary care physicians within the project team’s networks will be provided with an information sheet they can share with prospective patients and members of their support network who meet the study criteria. Interested participants can then contact the research team if they would like to participate. Patients will also be able to give members in their circle of care permission to have their contact information shared with the research staff, who would then get in touch with them.

Via survivor, advocacy, and support networks: Patients and members of their support networks will be recruited through the Leukemia & Lymphoma Society of Canada, a national voluntary health agency, using their local chapters across Canada and via the research project patient partner. Additionally, patients will be recruited through publicly available patient groups and listservs, such as the Chronic Lymphocytic Leukemia Patient Advocacy Group. The patient and family information sheet will be shared with boards of directors and listserv administrations to be distributed to their membership base on behalf of the study team via email, social media (eg, Twitter, Facebook) or newsletter.

Via coenrolment with the trial, ‘Chimeric antigen receptor T cells (anti-CD19 CAR-T) for the treatment of patients with relapsed/refractory CD19 positive hematologic malignancies’: Members of the research team are also conducting a trial of CAR-T therapy for the treatment of patients with r/r CD19 positive haematological malignancies (NCT03765177). This trial involves follow-up with patients from infusion date daily in the first 7 days, on day 14 and 28, in months 2, 3, 4, 5, 6, 9, and 12 (in person), then annually for up to 15 years (in person or over the phone). Individuals involved in the ongoing trial will be asked during their follow-up if they are interested in receiving information about this study.

Analysis plan

The NGT stakeholder group discussions will yield rapidly generated data, including focused qualitative data directly relevant to the posed research questions. Following NGT stakeholder group discussions and using rankings from the online feedback form, a priority list will be generated and input into a table. Data on the sum of scores for each idea generated, its relative importance, and voting frequency will be used to inform the ranked priority list based on each of these measurements and for each NGT stakeholder group.62 The lists generated from stakeholder discussions will then be assessed by members of the research team with expertise in HE evaluation in terms of whether data exist to operationalise the stakeholder-identified priorities to be included in the CAR-T therapy HE model development. Where challenges emerge in the incorporation of stakeholder priorities, the research team, including our patient and knowledge user partners, will develop recommendations for collecting such data in the future.

Thematic analysis will guide comparisons across stakeholder groups whereby similar priorities will be grouped under corresponding themes and disparities will be highlighted. The thematic analysis will be guided by the following steps: (1) data familiarisation, (2) code generation, (3) theme identification, (4) theme review, (5) theme characterisation, and (6) report synthesis.63 The emerging thematic framework will be assessed by a second analyst and debriefed with the research team to ensure accuracy and reflectiveness of the analysis. Audio recordings of the stakeholder groups will provide insight into the intricacies, context, and rationale with which group consensus was achieved and will be used to back-check the data analysis.

Quantitative data obtained from the stakeholder group discussion feedback form will be analysed to describe participant experience of engaging in the conduct of a HE evaluation, their level of interest in engaging in such processes, and recommendations to facilitate this engagement with stakeholders in the future. Data will be analysed descriptively (ie, using frequencies and contingency tables) and compared across the different stakeholder groups.

Discussion

Similar to other fields of research wherein challenges exist in translating research into practice, the limited utilisation of economic evaluations in healthcare resource allocation has been acknowledged.64 The barriers to uptake of HE evaluations in policy decision making have been discussed in terms of both their accessibility and acceptability.64 65 In addressing these barriers, it has been suggested that models should provide evidence that is credible, legitimate, and salient. Moreover, research aimed to bridge the evidence-to-policy gap suggests that involving knowledge users (ie, healthcare payers and policy-makers) throughout the model development process will ensure their information needs are met and enable economic evaluation models to effectively contribute to healthcare decision-making.66–68 The inclusion of input from key stakeholders throughout the economic evaluation process thus represents an important avenue to enhance evidence and facilitate the use of HE evaluation models to inform policy decision-making.

This study is reflective of the increasingly prominent call from the literature for stakeholder involvement in HE modelling.48 49 This form of involvement has demonstrated benefit in improving model transparency, validity, and credibility through the insight key stakeholders can provide, for example, in learning about the problem, identifying appropriate model objectives and requirements, and facilitating subsequent model use.48 49 69 The proposed study will enhance the HE model for CAR-T therapy through the validation of model assumptions and the generation of stakeholder-specific priorities rooted in both the relevant lived-experience of patients and their support networks and the content-specific expertise of other stakeholders. These priorities can be used to inform future development of the therapy and processes for its implementation to optimise resource utilisation. These stakeholders can contribute important knowledge to inform model development; however, additional stakeholders, such as the general public, industry, and politicians would need to be included for the implementation of the HE model results and is outside the scope of this study.

While the importance of stakeholder engagement in HE modelling is widely recognised, a consensus on the modality for such engagement does not exist, and work in this area is trailing behind several other scientific disciplines.49 Existing guidelines on how to engage stakeholders in fields such as environmental science can provide some insight14 15 65; however, these may or may not be applicable in the healthcare context and focus primarily on the stages of model development with which stakeholders could be a part of. Frameworks for patient engagement in broader health technology assessment call for more concrete methodologies to guide practice and provide a foundation on which this research outlines a rigorous step-by-step approach for stakeholder involvement in an early HE evaluation.40 41

This study will apply a novel approach to generate stakeholder-specific knowledge and priorities in the conduct and development of an early HE evaluation model. The results of this research will support the validation of the proposed CAR-T therapy economic model and inform a realistic commercial valuation of CAR-T therapy, inclusive of the knowledge and experience of diverse stakeholders, including patients and their support networks. This research explores a specific modality of stakeholder engagement in the conduct of an early HE evaluation, which could inform a tailored framework for engagement processes in future HE evaluation studies.

Ethics approval and consent to participate

All participant recruitment and consent materials were reviewed and received ethics approval from the Ottawa Hospital Research Institute Research Ethics Board (REB 20200320-01HT). Verbal consent will be obtained from study participants as a result of restrictions to in-person congregation during the global COVID-19 pandemic, and has been approved by the REB.