Developing and piloting an instrument to prioritize the worries of patients with acute myeloid leukemia.

BACKGROUND: Acute myeloid leukemia (AML) is a rapidly progressing blood cancer for which new treatments are needed. We sought to promote patient-focused drug development (PFDD) for AML by developing and piloting an instrument to prioritize the worries of patients with AML. PATIENTS AND METHODS: An innovative community-centered approach was used to engage expert and community stakeholders in the development, pretesting, pilot testing, and dissemination of a novel best-worst scaling instrument. Patient worries were identified through individual interviews (n=15) and group calls. The instrument was developed through rigorous pretesting (n=13) and then piloted among patients and caregivers engaged in this study (n=25). Priorities were assessed using best-worst scores (spanning from +1 to -1) representing the relative number of times that items were endorsed as the most and the least worrying. All findings were presented at a PFDD meeting at the US Food and Drug Administration (FDA) that was attended by >80 stakeholders.
RESULTS: The final instrument included 13 worries spanning issues such as decision making, treatment delivery, physical impacts, and psychosocial effects. Patients and caregivers most prioritized worries about dying from their disease (best minus worst [BW] score=0.73), long-term side effects (BW=0.28), and time in hospital (BW=0.25).
CONCLUSION: Community-centered approaches are valuable in designing and executing PFDD meetings and associated quantitative surveys to document the experience of patients. Expert and community stakeholders welcomed the opportunity to share their experiences with the FDA and strongly endorsed implementing this survey nationally.

Introduction

Acute myeloid leukemia (AML) is a form of cancer characterized by clonal expansion of abnormally differentiated cells in the bone marrow and peripheral blood.1 It is a relatively rare disease, with an incidence of ~4.1 per 100,000 people.2 While diagnosis of AML can occur at any age, the median age of diagnosis is 67 years, and it is most frequently diagnosed among people aged 65–84 years.2 AML can be cured in 35%–40% of adult patients aged <60 years and in 5%–15% of patients aged >60 years.1 Older patients who are unable to withstand the standard therapy can expect a median survival of only 5–10 months.3

There have been significant advances in understanding the cytogenetic heterogeneity of AML, but it is still too soon for this progress to have translated into improved therapeutic options.4 Though numerous agents for the treatment of AML are in clinical development, the general care strategy has remained unchanged for the past three decades.5 Depending on an initial clinical assessment, patients are most likely to undergo intensive induction chemotherapy. If complete remission is achieved following this regimen, essential post-remission therapy can be pursued via conventional chemotherapy as well as hematopoietic cell transplantation.3 It has been documented that the majority of patients with AML will eventually relapse and die of their disease or its associated complications.1 Given the relative paucity of innovation and lack of efficacious treatment options, more needs to be done to ensure that novel treatments are developed, tested in clinical trials, and brought to market.

To address the lack of patient-centered treatment options, there is growing consensus that regulatory evaluations should systematically incorporate the perspectives of patients and caregivers.6–9 This concept was solidified when the US Food and Drug Administration (FDA) was congressionally mandated to engage patients through the Patient-Focused Drug Development (PFDD) initiative.10,11 Although this program is groundbreaking in that it offers patients and caregivers a formal mechanism through which to contribute to the regulatory process, advocates who did not represent one of the few chosen disorders were left with little guidance about how to participate.12 Even for advocates involved in FDA-organized PFDD meetings, existing models for FDA engagement are largely limited to providing qualitative testimonials. As patient engagement evolves and “enhanced benefit risk assessment” becomes increasingly incorporated into the regulatory process, researchers have been tasked by key decision makers with developing scientific ways to measure the patient perspective and to report findings in a way that is actionable.10,13–15 Quantitative preference elicitation methods have emerged as a technique to introduce formal evidence-based decision making into the regulatory process.16

We sought to demonstrate how a partnership between a patient advocacy organization and academic researchers (aided by diverse stakeholders, including patients, and caregivers) could lead to the development of a quantitative stated-preference instrument focused on assessing the worries of patients and caregivers. This is important, as patient/caregiver worries have received significant attention during previous PFDD meetings, but are frequently only addressed qualitatively.11,17 Furthermore, we sought to demonstrate a fully integrated community-centered approach to survey development that can be used to make the PFDD processes more generalizable.

Patients and methods

This study was initiated by the Leukemia & Lymphoma Society (LLS), the largest organization dedicated to patients with blood cancer. An executive team – comprising both LLS staff and academic researchers – was engaged in leading the study. A community-centered approach was a core aspect of this partnership, and two different stakeholder groups were intimately involved in each step of the research process. The first was an expert stakeholder committee comprising health care providers, academic researchers, industry representatives, and disease advocates. The second was a community stakeholder committee comprising patients with AML and caregivers. The research team regularly communicated with these stakeholders on both group and individual levels. Figure 1 demonstrates the overlapping nature of the governance structure of this study. All three groups had ownership over the study, but had different roles.

Figure 1

Project governance.

Guided by recent advancements in the development of stated-preference instruments, Figure 2 illustrates how our PFDD effort was organized across five stages: engagement, development, pretesting, pilot testing, and presentation.18–21 These stages were intended to guide the different phases of development and dissemination of the survey, and we can also see these activities as a continuous, snowballing advocacy effort. The PFDD meeting was scheduled before these efforts commenced, which motivated everyone to complete the project on time.

Figure 2

Key stages of the study.

Abbreviation: FDA, US Food and Drug Administration.

Engagement

To initiate the project, the expert and community stakeholder committees were engaged to identify areas of greatest need and to determine the scope of the project. We sought to ground our survey in the lived experience of patients and caregivers and in the current policy/scientific/clinical environment. During this process, it was also necessary to educate all stakeholders about the methods we intended to use.

Members of the expert stakeholder committee (n=14) were initially convened on an orientation call in November 2015. Following this group call, the research team scheduled individual calls to gather information about the priorities and preferences of AML experts. These calls were valuable in identifying what type of research questions would be most valuable to the field and to assess the existing needs of the AML community at large.

Following another orientation call in January 2016, members of the community stakeholder committee (n=15) were each engaged in a one-on-one semi-structured cognitive interview about their experience with AML and its treatments. The participating patients and caregivers were located throughout the country and were each interviewed for ~1 hour through phone. All interviews were recorded and transcribed.

Development

The next stage of the project was development of the survey instrument. This included identifying the best approaches to capture the patient experience and determining which aspects of the patient experience we should include in the survey. While the executive team played an important role in shaping the survey, both the expert and the community stakeholder committees continued to be engaged.

While we intended to use a stated-preference instrument to capture the lived experience of patients, we considered two possible framing vignettes, one focused on the greatest unmet needs and the other on what factors patients worried about the most. Both committees endorsed a worries paradigm and were engaged in identifying and refining the different possible factors that could be included in the survey. Diverse stated-preference techniques such as a 2^K conjoint analysis were considered; however, best–worst scaling (BWS; Case 1) was determined to be the best analytic technique to prioritize patient worries.22 This approach has been used in several recent health care applications and has the benefit of a more parsimonious experimental design.12,23

In BWS Case 1, respondents must select a pair of items that are most and least preferred. BWS has become an increasingly popular way to measure the priorities of patients and caregivers and forces a choice between objects in a profile that are furthest apart from one another. This allows researchers to gain information about the best and worst selections as well as those that are not selected. In this way, BWS maximizes analytic efficiency while being relatively simple to design and interpret. Attributes included in BWS are typically developed through qualitative stakeholder engagement, and the attribute set is able to include items that are each expected to be highly relevant to the majority of the studied population. BWS also has the advantage of being less cognitively demanding than alternative stated-preference methods such as discrete choice or conjoint experiments.24

Our initial qualitative interviews yielded >15 potential “worries” that patients and caregivers faced throughout their experience with AML. Several potential worries were rejected by the executive team because they were redundant or lacked clarity. The multidisciplinary stakeholders ultimately agreed to pretest 12 attributes that fell into four experience domains (Table 1).

Table 1

Worry domains and items

Domains	Worry items
Decision making	Knowing about all of my treatment optionsChoosing a treatment in a short amount of timeHaving enough information about my disease
Treatment delivery	My doctors understanding my wants and needsHaving access to the best possible medical careHow much my medical care would cost
Physical impacts	Short-term side effects outweighing the benefitsSpending a significant amount of time in hospitalReturning to the activities that are important to me
Psychosocial effects	Understanding my chances of survivalCoping with the emotional demands of my situationBecoming a burden to those who care for me

Pretesting

The goals of pretesting a stated-preference instrument are to identify ways to improve the survey, identify any gaps in the survey, and to assess the burden of the survey on potential respondents. Given our timeline and effort to engage patients and caregivers in all aspects of the survey, we pretested our survey instrument with the patients and caregivers on our community stakeholder committee.

During the phone-based pretest, participants were guided through the survey by a member of the research team. Individual pretest calls were conducted with 13 patients and caregivers of varying ages and disease experiences. These individuals were employed to assess survey comprehension, refine attribute terminology, and test the acceptability and usability of the instrument. Each respondent was asked to indicate what they thought each worry meant and which worries were relevant to their own experience, and to note whether any key components were missing. Respondents “thought aloud” while completing the survey instrument, and the investigator also used verbal probes to explore anticipated problem areas, assess the level of understanding, and evaluate willingness to trade among the different worry attributes. In addition, participants were asked to provide feedback on the BWS design and its functionality. Feedback was incorporated in an iterative and continuous manner. This resulted in four different versions of the survey being pretested over the course of 2 weeks in February and March 2016. Interviews were recorded and transcribed.

As a result of the feedback from the 13 respondents and supplementary consultation from the executive team, one new attribute was added and three other attributes were substantially refined. Based on feedback from patients and caregivers when asked whether anything was missing from the original list of 12 items, “Long-term side effects impacting my everyday life” was added to better address worries related to the long-term harmfulness of available AML treatments. The following is an example of a response that motivated this decision:

You have like short term side effects outweighing the benefits but it doesn’t say anything about worrying about long term side effects which I did a lot of. [CR, Pretest 1]

The attribute “My doctors understanding my wants and needs” was rephrased to “Communicating openly with my doctors”. This change was made in response to many participants expressing that the patient–doctor relationship is incredibly important, but hinges on the quality of the underlying communication:

On a regular basis [I was] going to see a physician at [hospital name], and I felt that she just wasn’t listening to me, she wasn’t giving me the time I need. She was rushing me out the door. So I went to [different hospital name] […] to a doctor [who] relate[d] to me and gave me all the time I needed. And I think that was really important […] I don’t know if you want to somehow use the word communication, which I think is really the key for me. [GR, Pretest 2]

The attribute “How much my medical care would cost” was changed to “The overall monetary cost of my disease” to better capture both direct and indirect medical costs of AML, such as lost wages:

[…] all of a sudden, I was thrown into a situation where both my wife and I were working, making a good income and living to the standard, now, it was just my wife. How were we going to make it […] and how long would it be before I was able to go back to work? And so that was a tremendous, tremendous concern […]. Number one, survivability. Number two, what would the treatment cost and my time away from work do to us financially. [BV, Pretest 11]

Finally, “Understanding my chances of survival” was revised to “The possibility of dying from my disease”. This change was made to better reflect fear of dying, rather than concerns related to understanding the statistical prognosis. The distinction between understanding the odds and still experiencing fear was conveyed by one participant:

[…] there was so much confusion [about] the chance of survival that […] even if you told me there was a 90 percent chance of survival there’s always that 10 percent, and in my experience I got to see the 10 percent too often, so it was like, well, chances of survival stopped meaning anything for me once I got diagnosed. Thirty percent, 50% percent, 100, it didn’t matter. [RA, Pretest 8]

These 13 revised items were incorporated into a final version of the BWS survey that was approved by the executive team for piloting (Figure 3).

Figure 3

Worries about living with AML (best–worst score).

Abbreviation: AML, acute myeloid leukemia.

Pilot testing

In preparation for the PFDD meeting, we pilot tested the final survey instrument to gather data to demonstrate how the survey and analysis worked. We also sought to demonstrate the benefits of quantitative approaches – as opposed to the predominantly qualitative approaches that have been used previously.

The refined set of worries were incorporated into a BWS survey instrument. Following good research practices, a balanced incomplete block BWS Case 1 experimental design was developed to accommodate 13 worry-related attributes. In each BWS task, respondents were asked to judge the one attribute they worried about the most and the one they worried about the least during their experience with AML. Patients and caregivers were presented with 13 repeated subsets, which had four attributes from which to choose (Figure 4).

Figure 4

Example of a best–worst scaling choice task.

Participants were patients with AML or caregivers of patients with AML. They lived in the USA, were aged at least 18 years, and were able to complete an online survey in English. The survey was administered electronically using the Qualtrics survey system from March 24, 2016 to March 31, 2016. Each participant was recruited by the LLS or was already an active member of one of the stakeholder committees.

The dependent variable in a BWS analysis is a participant’s judgment about the extremes in each profile presented to them. The simplest approach to understanding these data is to create a best minus worst (BW) score, which focuses on the number of times an attribute was chosen as the most or the least worrying over all of the choice tasks. This is calculated by subtracting the number of times an object was selected as the most worrying from the number of times it was selected as the least worrying, then dividing by the total number of times it was presented to the participant.

This approach has demonstrated high levels of correlation with conventional multinomial logistic regression and conditional logistic regression choice models.25,26 Similar to other methods to estimate ordinal, multinomial outcomes, scoring assumes equal spacing between things chosen as the most worrying (BW score=1) and things chosen as the least worrying (BW score=−1). The BW score is estimated as a mean across the surveyed sample, allowing the reporting of standard errors (SEs) for these values.

Presentation

One of the primary goals of this study was to share our results with the FDA and other stakeholders at a PFDD meeting. We aimed to not only inform the FDA about the lived experience of patients with AML, but also sought buy-in for a national roll out of the study. To achieve this, we intended to use a combination of testimony, direct interaction with the FDA via a question and answer (Q&A) session, and a formal presentation of our results.

The goals of the meeting were discussed at length with both stakeholder committees; based on these conversations, the meeting was organized into four key themes: “Diagnosis”, “Quality of Life”, “Current Treatment Options”, and “Benefits and Risks of Future Treatments”. For the first three themes, questions were crafted by the executive team and circulated to the community stakeholder committee for feedback and editing. A day before the PFDD meeting at the FDA, the executive team convened with the community stakeholder committee, so that participants could meet in person for the first time. During this meeting, as a group we also reviewed the agenda for the PFDD meeting and conducted a rehearsal. As was practiced, throughout the first three parts of the FDA meeting, a moderator asked the community stakeholder committee members open-ended questions relating to the given theme. Following the conclusion of each set of questions, a dialogue and separate Q&A were encouraged between the committee members and the FDA. During the fourth part, “Benefits and Risks of Future Treatments”, the quantitative findings were presented.27

Following the conclusion of the meeting, patients and caregivers were asked to evaluate the FDA meeting and comment on their experience of the day. The intent was to assess the extent to which the community stakeholder committee members felt they were able to contribute to the meeting dialogue and whether their opinions were accurately conveyed to the FDA.

Compliance with ethical standards

This research was conducted in accordance with the Declaration of Helsinki, and the study protocol was reviewed and approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (IRB 6858). This study was deemed to be non-human subjects research. Hence, all the stakeholders involved in this study were not research subjects, but rather research partners who were engaged in multiple (if not all) stages of the study.

Results

Pilot study

In total, 18 patients and 7 caregivers of patients with AML were included in the analysis. Table 2 summarizes the characteristics of the sample. Participants were predominantly Caucasian, female, and well educated. On average, participants had received their AML diagnosis 5.8 years previously, and approximately half had been enrolled in a clinical trial at some point during their treatment. Throughout their entire treatment experience, 96% of patients had received chemotherapy and 56% had undergone an allogeneic or autologous transplantation.

Table 2

Pilot testing participant characteristics

Characteristics	n	%
Total participants	25	100
Relation to AML
Patient	18	72
Caregiver	7	28
Sex
Male	9	36
Female	16	64
Mean age, years (range)
50.84 (24–81)
Race/ethnicity
White/Caucasian	19	76
Black/African American	2	8
Hispanic	4	16
Education
High school/GED	1	4
Some college	1	4
College or higher	21	84
Annual household income ($US)
>$75,000	17	68
Between $25,000 and $74,999	7	28
<$25,000	1	4
Past treatment (select all that apply)
Chemotherapy	24	96
Radiation	8	32
Targeted therapy	3	12
Immunotherapy	2	8
Allogeneic stem cell transplant	13	52
Autologous stem cell transplant	1	4
Palliative care	1	4
Clinical trial
Have participated	13	52

Abbreviations: AML, acute myeloid leukemia; GED, general educational development.

Figure 3 displays the results of the AML-related worries stated-preference instrument via item-specific BW scores. Overall, respondents worried the most about “the possibility of dying from my disease” (BW=0.73, SE=0.12). Patients and caregivers also highly prioritized “long-term side effects impacting my everyday life” (BW=0.28, SE=0.13) and “spending a significant amount of time in the hospital” (BW=0.25, SE=0.13); both these items were categorized to the domain of physical effects. Following these, “coping with the emotional demands of my situation” (BW=0.10, SE=0.14) and “becoming a burden to those who care for me” (BW=0.06, SE=0.15) were prioritized most highly. These two factors belong to the psychosocial effects domain. All items belonging to the decision-making and treatment delivery domains, such as “the overall financial cost of my disease” (BW=−0.31, SE=0.15) and “choosing a treatment in a short amount of time” (BW=−0.08, SE=0.15), were modestly prioritized. Finally, the least prioritized item was “communicating openly with my doctors” (BW=−0.60, SE=0.11).

The results of this pilot study were presented to the FDA members at an independently organized PFDD initiated by the LLS. This meeting was well attended not only by members of the FDA, but also by most of the members of the study’s expert and community stakeholder committees, who traveled from all over the USA to attend.

Much of what was presented in the quantitative portion of the meeting, “Benefits and Risks of Future Treatments”, provided empirical evidence to support the qualitative testimony provided throughout the earlier components of the meeting. Reflecting on the meeting and presentation of these results, 72% of attendees from the community stakeholder committee felt their perspective was “very well” communicated to the FDA, whereas the remaining 28% felt their perspective was “fairly well” conveyed. In considering what next steps should be taken by the executive team, 16 committee members reported that publishing the findings from patient-centered research in reputable scientific journals is “very important”.

Discussion

Identifying and prioritizing the worries of patients and caregivers of patients with AML are necessary to understand the experience of those affected by the disease and are important in the development of better care and treatment options. In our study describing patient and caregiver AML-related worries, respondents ranked the possibility of dying from their disease as the greatest concern. This was followed by worries related to the long-term impact of their treatment and the amount of time they would have to spend in hospital. Items related to the treatment delivery or decision-making domains were less prioritized than the items in the physical impact and psychosocial effects domains. In interpreting BWS results, this means that although access to information about AML or the financial burden of AML might worry patients and caregivers, they are not as important as the long-term collateral damage of available treatments or the amount of time patients must spend in hospital.

There are several limitations to the study. First, as raised by a participant during the qualitative pretesting phase, worry does not have a one-to-one correlation with importance. For example, even if a patient thought having access to the best possible care was the most important thing, they may not have had to worry about it if they felt they already had access to it. Though this does not discount the importance of measuring worries, it is helpful in further honing the operational definition of the attributes that were studied. Second, the recruitment of the stakeholder groups through the LLS, while efficient, carries a significant risk of selection bias. Specifically, the individuals who were selected to participate were diagnosed with AML an average of 5.8 years ago. This could introduce a recall bias, in which patients and caregivers are unable to accurately recall what they were most worried about during treatment. To combat this, we chose to focus on the entire disease experience. Additionally, as AML is a very severe disease, in working with patients who had survived treatment, it is possible that our results are biased toward the preferences of patients who have done well with therapy. We attempted to remedy this by including caregivers of patients who had passed away, but would also implore future researchers to survey patients who are currently undergoing treatment. Lastly, due to the relatively small sample size, we were unable to analyze the pilot data stratified by patient and caregiver or by noteworthy characteristics such as age or disease state. Future research should consider a larger sample size to allow for meaningful stratification and segmentation analysis to examine the existence of preference heterogeneity.

Conclusion

This study demonstrates a novel way to identify disease-relevant attributes and prioritize patient and caregiver worries. Additionally, this study can serve as an example of how a patient group can partner with researchers to fully and systematically engage a disease-specific community and develop a stated-preference instrument. This mixed-methods and collaborative approach yielded a quantitative way to understand the experience of patients and caregivers and provided researchers with a scientific way to communicate priorities to regulatory decision makers. The presentation of a combination of quantitative evidence with traditional patient testimony was considered an important advancement for the PFDD movement. Methodologically, this study demonstrated that BWS Case 1 is a feasible way to elicit the priorities of patients and caregivers. The expansion of this pilot study to a national study of patients and caregivers is strongly endorsed by all stakeholders.