Development and First Use of the Patient's Qualitative Assessment of Treatment (PQAT) Questionnaire in Type 2 Diabetes Mellitus to Explore Individualised Benefit-Harm of Drugs Received During Clinical Studies.

INTRODUCTION: Individualised benefit-harm assessments can help identify patient-perceived benefits and harms of a treatment, and associated trade-offs that may influence patients' willingness to use a treatment. This research presents the first use of a patient-reported outcome measure designed to assess patient-perceived benefits and disadvantages of drugs received during clinical studies.
METHODS: The Patient's Qualitative Assessment of Treatment (PQAT) was developed in English and cognitively tested with US (n = 4) and Canadian (n = 3) patients with type 1 and type 2 diabetes mellitus (T2DM). The revised version of the PQAT comprises three qualitative open-ended questions focused on the benefits and disadvantages of treatment and reasons why patients would choose to continue/discontinue treatment. A final quantitative question asks patients to evaluate the balance between benefits and disadvantages using a 7-point scale. The revised version of the questionnaire was administered as an exploratory endpoint in a phase II clinical trial for a new injectable treatment for T2DM. Qualitative data were analysed using thematic analysis, and relationships between qualitative and quantitative data were identified.
RESULTS: Patient-reported benefits of treatment administered during the clinical trial included clinical markers of efficacy and subjective markers. Disadvantages reported by patients were mainly related to drug adverse effects or to the mode of administration. Of the 57 patients completing the PQAT, 70.2% reported being willing to continue treatment, with 59.6% reporting that the benefits outweighed the disadvantages. The reported benefits of feeling better and improved energy levels were more likely to be associated with a more positive ratio (70% and 71.4%, respectively), while the disadvantages of fatigue, headaches, and stomach pain were associated with a negative ratio and patients not being willing to continue the treatment.
CONCLUSIONS: The PQAT is a unique patient-reported outcome tool designed to aid understanding patients' real experience of benefits and disadvantages of a treatment. It combines the richness of qualitative data with quantitative data-information valuable for various stakeholders to make well-informed treatment decisions. TRIAL REGISTRATION: ClinicalTrials.gov identifier: NCT02973321.

Key Points

Introduction

Patients are experts in what it is like to live with a medical condition. As such, there is an ever-increasing drive to capture and better incorporate patients’ experiences, perspectives, needs, and priorities into healthcare decisions. These include decisions concerning drug development, evaluation, licensing, and prescription. Consideration of the benefits and harms of medical interventions is integral to decisions made by key stakeholders, such as pharmaceutical companies, regulatory authorities, reimbursement agencies, health technology assessment (HTA) bodies, healthcare professionals, and patients.

Historically, such benefit–harm assessments have been done at a population level, with decisions reflecting the opinion of a specific population when the medicinal product is used as intended. Such assessments typically use data from clinical trials and expert clinical opinion to consider benefits and harms separately and to inform benefit–risk frameworks in appraising interventions. Decision makers such as the US Food and Drug Administration (FDA) and European Medicines Agency have demonstrated their interest in understanding the patient’s perspective regarding treatment benefits and harms, and methods for gathering such information. Particularly important for the FDA, this is reflected within the Patient-Focused Drug Development (PFDD) initiative and new guidance documents in development on patient experience and Clinical Outcome Assessments [1-7]. Reimbursement agencies and HTA bodies have also shown an interest in understanding how new products are received by patients [8]. HTA bodies consider benefit–harm assessments in their benefit–risk framework for determining the value of competing therapies: “What are the overall benefits and harms of the technology in health outcomes?” is one of the components of the HTA Core Model recommended by the European Network for HTA to standardise HTAs [9].

The era of precision and genomic medicine has given rise to more heterogeneous patient needs and complex and diverse treatment options [10-12]. This has resulted in an increased need to understand how a product is perceived at the individual patient level [1]. Benefit–harm assessments conducted at the individual level can be influenced by several factors, including how risk-averse a patient is, experience, and circumstances, revealing subgroups of patients with differing opinions of benefit–harm attributes. Understanding this heterogeneity in patients’ preferences and thresholds for risk can be used to individualise patient care and improve medication adherence [1, 13].

There are currently few standardised tools for conducting benefit–harm assessments at the individual level. A Drug Information Association (DIA) workstream was convened in 2018 to discuss how to construct and use a comprehensive individual-level benefit–risk assessment for evaluating treatment as a supplement for aggregated population-level data [1]. This workstream highlighted the potential use of stated-preference methods, patient-reported outcome (PRO) questionnaires, and qualitative research methods to gather data from individuals.

Stated-preference methods are used to assess patient preference for medical interventions or certain attributes of these interventions. The workstream acknowledged the strength of such methods in that they can be used with large and diverse populations to identify subgroups of individuals with similar preferences and also rare benefits and harms. However, these methods gather information on risks (anticipated harms) based on hypothetical scenarios as opposed to actual patient experience [1]. To capture patients’ actual experiences of medical interventions, the workstream suggested the use of PRO questionnaires, qualitative research, or a combination of both methods as part of a mixed-methods approach [1]. A mixed-methods study addresses a set of research questions that require both qualitative and quantitative evidence and methods, as defined by the FDA [1, 14].

PRO questionnaires are used in research and clinical practice to evaluate the safety or effectiveness of a medicine from the patient’s perspective, although both safety and effectiveness are rarely measured together. PRO questionnaires typically include standardised response scales or predefined response options [15]. Qualitative data can provide insights into the context of quantitative responses to PRO questionnaires and aid interpretation of findings. Such combinations of both qualitative and quantitative data (i.e. mixed-methods research) are now increasingly used in clinical research studies.

There are currently no PRO questionnaires or accepted approaches available to assess the benefits, risks or harms, and associated trade-offs of treatments administered as part of clinical studies from the patient’s perspective. The Patient’s Qualitative Assessment of Treatment (PQAT), a mixed-method benefit–harm questionnaire, has been designed as per the DIA workstream recommendation. This questionnaire combines qualitative and quantitative questions focusing on the benefits and disadvantages of treatments, as well as the trade-off between them to capture, individuals’ personal perspectives [1].

As the questionnaire primarily explores the patients’ experience of treatment, it could be classed as a patient-reported experience measure (PREM), as opposed to a PRO measure [16]. PREMs are used to gather information on the process of care [17], aimed at improving recruitment, retention, and long-term patient engagement with trials [18]. The PQAT is able to record data on both patient experience and outcome measurements, and therefore should be defined as a ‘patient-reported instrument to elicit experiences and outcomes’.

Patient experience research is rarely dedicated to the development of tools to measure the experiences of patients with a new treatment. The purpose of this article is to discuss the development and initial use of a questionnaire, the PQAT. This questionnaire has been developed to better understand patients’ own perspectives regarding the benefits and disadvantages of medicinal products received within a clinical trial.

Methods

Development of the Patient’s Qualitative Assessment of Treatment (PQAT)

The rationale for the development of the PQAT was the need for a simple tool to assess the benefits and disadvantages of a treatment from the patient’s perspective using simple and generic questions. Initial items were developed by the authors (who are experienced in PRO research) and were based on knowledge and discussions relating to benefit–risk assessments. The initial draft of the PQAT was developed in English and comprised four questions asking about the ‘benefits’ and ‘downsides’ of treatment. Cognitive testing of this version was conducted among US (n = 4) and Canadian (n = 3) patients with type 1 and type 2 diabetes mellitus (T2DM) to assess respondent comprehension and understanding.

Use of the PQAT in the DRI13940 Study

The revised PQAT was administered as an exploratory endpoint in a dose-ranging phase II clinical trial (DRI13940, ClinicalTrials.gov identifier: NCT02973321 [32]). The purpose of this study was to assess a novel dual glucagon-like peptide-1 receptor (GLP-1R)/glucagon receptor (GCGR) agonist developed for the treatment of overweight/obese patients with T2DM inadequately controlled by diet/exercise and/or metformin.

Participant Sample

Eligible patients had been diagnosed with T2DM at least 3 months prior to screening and had experienced an inadequate response to diet/exercise and/or treatment with metformin (≥ 1500 mg/day or at least the maximal tolerated dose). Trial treatment was self-administered via subcutaneous injection. All patients were between 18 and 80 years of age. Full exclusion criteria for the study can be found at ClinicalTrials.gov (NCT02973321). The PQAT was only available in English and so was limited to English-speaking participants in the USA and Canada.

PQAT Assessment Schedule

The PQAT was administered at the clinical site at the end of the 26-week treatment period or at early discontinuation. Participants completed the questionnaire on an electronic tablet device when they arrived at the clinical site before any other procedures and tests. Participants were instructed to complete these questions on their own and without help from study site staff, friends, or relatives. Participant answers were not visible to site employees.1

PQAT Analysis

Analysis of Qualitative Data

All identifiable information was removed from the PQAT data prior to analysis. Participant free-text answers were coded and grouped according to concept using methods derived from thematic analysis [21-23]. Codes were used to capture the relevant concepts described by participants in relation to each of the three qualitative questions. The data were coded manually by two experienced qualitative researchers (authors KB and AF) in Microsoft Excel® (Microsoft Corp., Redmond, WA, USA). Data from the first 10% of participants (n = 6) were coded by both researchers independently. Assigned codes were then compared and discussed to reach a consensus on a provisional code list. The accuracy and consistency of the coding across data were assessed by an additional researcher (author AG). The existing code list was reviewed regularly to remove or merge any redundant or equivalent codes. A list of final codes used during the analysis can be found in the Electronic Supplementary Material (Online Resource 1). Qualitative data were documented at the concept level and subconcept level in study-specific data extraction tables. Verbatim quotes were also included within these extraction tables to provide context for the codes.

Analysis of Quantitative Data

Data extraction tables were populated with categorical responses to PQAT quantitative items at the individual-participant level. Data from the entire sample were summarised descriptively by presenting the number and percentage of subjects in each category.

Relationships Between Variables and Group Differences

Patterns or associations between participants’ responses to the different qualitative and quantitative PQAT questions (e.g. certain benefits or disadvantages) were explored by analysing relationships between variables and co-occurrence of concepts. Additionally, responses of patients who discontinued treatment early were compared with patients who continued the treatment to see if certain concepts were associated with early treatment discontinuation.

It was hypothesised that those who discontinued treatment or who reported being unwilling to continue treatment (in response to question 3) would indicate that the disadvantages of treatment outweighed the benefits of treatment (in response to question 4). It was also expected that participants reporting no benefits (in response to question 1) would be more likely to discontinue treatment and to indicate that the disadvantages of treatment outweighed the benefits (in response to question 4). Similarly, it was expected that participants reporting ‘no disadvantages’ (in response to question 2) would be more likely to complete the study and to report that the benefits of treatment outweighed the disadvantages.

Results

Development of PQAT

Feedback from participants indicated that all four questions were understood by participants, and all participants answered the questions as intended. However, minor amendments to the wording of questions and response options were recommended by participants as a means to simplify the measure and maximise comprehension.

The term ‘benefit’ was understood as “what positive things have you experienced from this drug?”, “advantages or good points of taking this drug”, “something that helped patients or make something easier” and “it means any assistance I received from the drugs in lowering my blood glucose levels”. The term ‘downsides’ was also well understood (“what negative effects have you noticed from this drug?”, “downside to me means drawbacks or negative conditions experienced”, “problems or disadvantages”, “any side effects or negative reactions to the drug”, or “issues you have with the drug taken within the trial”). However, three participants spontaneously suggested replacing the word ‘downsides’ with ‘disadvantages’ to improve understanding and ensure consistent interpretation among patients. An additional participant indicated a preference for the word disadvantage when specifically asked. As such, the PQAT was revised to replace ‘downsides’ with ‘disadvantages’.

The questionnaire was generally well received—participants gave positive feedback on the clarity and small number of questions. This included the following comments: “Questions look pretty good to me, in general” (male diabetes US patient, 60–69 years old); “I love the fact that it is a small number of questions” (female T2DM US patient, 30–34 years old); “As they were originally written, these questions made it pretty clear what you were seeking, so I don’t believe the questions need a lot of tweaking” (female T2DM Canadian patient, 50–59 years old).

The revised version of the questionnaire comprised three open-ended questions with free-text responses (focused on benefits, disadvantages of treatment, and reasons for willingness to or not to continue the treatment after the trial). A quantitative question focused on participants’ overall evaluation of the balance between benefits and disadvantages of treatment experienced during the trial (rated using a 7-point Likert scale). The full PQAT is available for use and can be requested from the Mapi Research Trust (https://eprovide.mapi-trust.org/).

Study Sample

A total of 296 participants were enrolled into the DRI13940 study. The PQAT was administered in English to English-speaking patients from the USA and Canada (n = 70); 57 of these participants (81.4%) completed the questionnaire. Reasons for eligible participants not completing the questionnaire included technical issues with the electronic device or patients who were lost to follow-up during the trial. This sample of PQAT respondents included participants from all treatment arms. A total of 14 participants (n = 14/57; 24.6%) who completed the PQAT discontinued treatment prior to the end of the study and completed the PQAT at their early discontinuation visit. All other participants completed the PQAT at their week 26 visit. A descriptive summary of the study sample is included in Fig. 1. Demographics and clinical characteristics of PQAT respondents at baseline were comparable with the full trial sample. Details of patients’ demographics and clinical characteristics are included in Table 1. The sample size (n = 57) was deemed insufficient to support the conduct of inferential statistical tests.

Fig. 1

Descriptive summary of the study sample. PQAT Patient’s Qualitative Assessment of Treatment

Table 1

Demographic and clinical characteristics of Patient’s Qualitative Assessment of Treatment respondents in DRI13940

Variable	All participants from DRI13940 trial (n = 296)	All participants who completed the PQAT (n = 57)	Participants who completed the PQAT and completed trial (n = 43)a	Participants who completed the PQAT and discontinued trial (n = 14)
Mean age [years (range)]	55.6 (29–79)	56.6 (33–77)	56.6 (33–74)	56.6 (35–77)
Sex [n (%)]
Females	143 (48.3)	24 (42.1)	23 (53.5)	4 (28.6)
Males	153 (51.7)	33 (57.9)	20 (46.5)	10 (71.4)
Country
Canada	19 (6.4)	5 (8.8)	3 (7.0)	2 (14.3)
USA	108 (36.5)	52 (91.2)	40 (93.0)	12 (27.9)
Duration of diabetes (years) [n (%)]
< 10	224 (75.7)	45 (78.9)	36 (83.7)	9 (64.3)
≥ 10	72 (24.3)	12 (21.1)	7 (16.3)	5 (35.7)
Weight [kg (mean; range)]
Baseline	95.3 (50.8–150.6)	97.4 (66.2–141.9)	97.4 (66.2–141.9)	97.1 (79.8–119.2)
HbA1c (%) [mean (range)]
Baseline	8.2 (7.0–10.0)	8.2 (6.5–10.3)	8.1 (6.5–10.2)	8.3 (6.6–10.3)

HbA glycosylated haemoglobin, PQAT Patient’s Qualitative Assessment of Treatment

aVisit 12; 26 weeks

Question 1: Benefits of the Drug Received During the Trial

Question 1 of the PQAT was intended to explore the main benefits participants experienced from taking the drug during the trial. Figure 2 presents concepts most frequently reported by at least seven participants among PQAT respondents (n = 57). These include improved blood glucose levels and blood glucose control (n = 38), weight loss (n = 26), feeling better (n = 10), and improved energy levels (n = 7). Seven participants reported experiencing no benefits. Example quotes from participants related to the main reported benefits are provided in Table 2.

Fig. 2

Main reported benefits for the full PQAT sample (n = 57), participants who continued all trial procedures (n = 43) and participants who discontinued treatment (n = 14). The main benefits presented here were reported by at least seven participants. The percentage of participants was calculated as a proportion of participants in each group (i.e. 57 for the full PQAT sample, 43 for participants who completed all trial procedures, and 14 for participants who discontinued treatment). The n number shown corresponds to the number of participants reporting each concept from the full PQAT sample. PQAT Patient’s Qualitative Assessment of Treatment

Table 2

Example participant quotes for main benefits reported

Concept	Quote [sic]a	Patient ID
Improved glucose levels/control	“Blood sugar was lower and more controlled”	005
	“My sugar levels were kept at bay”	008
	“I believe my blood sugar levels were lower than they would have been without the drug”	018
Weight loss	“I lost some weight which was good”	029b
	“Lost significant amount of weight nearly 50 lb”	041
	“More energy and slimming of the waist”	048
Feel better	“I feel better less tired”	017
	“Because I feel great with my sugar in a normal Range”	044b
	“I felt healthier while on the medication”	048
Improved energy	“Lost significant amount of weight nearly 50 lb and as a result gained energy reduced blood sugars and blood pressure”	041
	“Increased energy and mobility”	005
No benefits	“I have not noticed a difference”	025b
	“No benefits”	032b
	“None”	049b

aQuotes are reported exactly as they were submitted on the electronic tablet by the respondent

bParticipant discontinued treatment

Beyond the most frequently reported benefits, participants (n = 18/57; 31.6%) reported a total of 15 additional benefits. Each additional benefit was reported by three or fewer participants, and these benefits included changes to appetite (e.g. loss of appetite; n = 3), improvements in general health (e.g. improved blood pressure; n = 3), physical activity (n = 3), mode of administration (n = 2), sleep (n = 2), and appearance (n = 1). Two participants reported perceived benefits linked to blood sugar numbers (better understanding and more frequent assessment of sugar numbers), which are associated with their participation in a clinical trial as opposed to benefits of the medication.

The main benefits reported by participants who completed all trial procedures (n = 43), excluding those who discontinued from the trial early (n = 14), were largely comparable with the full PQAT sample (Fig. 2). A slightly higher proportion of those participants who discontinued treatment early reported ‘weight loss’ as a benefit (n = 8/14; 57.1%) than those participants who completed all trial procedures (n = 18/43; 41.9%). As hypothesised, there was also a higher proportion of participants who discontinued treatment early (n = 4/14; 28.6%) who reported ‘no benefit’ than of those who completed all trial procedures (n = 3/43; 7.0%).

Question 2: Disadvantages of the Drug Received During the Trial

The disadvantages most frequently reported by PQAT respondents are shown in Fig. 3 and include nausea (n = 16/57; 28.1%), vomiting (n = 11/57; 19.3%), diarrhoea (n = 6/57; 10.5%), fatigue (n = 5/57; 8.8%), mode of administration (n = 5/57; 8.8%), stomach pain (n = 4/57; 7.0%), and headaches (n = 4/57; 7.0%). Interestingly, over a quarter of participants reported no disadvantages (n = 15/57; 26.3%). Table 3 provides example quotes from participants related to the main reported disadvantages.

Fig. 3

Main reported disadvantages for the full PQAT sample (n = 57), participants who continued all trial procedures (n = 43) and participants who discontinued treatment (n = 14). The main disadvantages presented here were reported by at least four participants. The percentage of participants was calculated as a proportion of participants in each group (i.e. 57 for the full PQAT sample, 43 for participants who completed all trial procedures, and 14 for participants who discontinued treatment). The n number shown corresponds to the number of participants reporting each concept from the full PQAT sample. PQAT Patient’s Qualitative Assessment of Treatment

Table 3

Participant quotes for main disadvantages reported

Concept	Quote [sic]a	Patient ID
No disadvantages	“None that i know of”	056
	“There were no disadvantages”	005
	“I didn’t see any disadvantages with taking the drug”	020
Nausea	“Nausea all the time”	049b
	“Persistent mild nausea up until about the last 3 weeks when it finally cleared up”	018
	“Intense nausea for the first four weeks. Occasional nausea after the first 4 weeks”	057
Vomiting	“Nausea, vomiting, burping, stomach pain”	042
	“Nausea and difficulty keeping food down. Most noticeable when the drug was taken close to the same time as metformin”	041
	“I experienced nausea in and vomiting in early part of this study”	029b
Diarrhoea	“I experienced some vomiting, dehydration and diarrhea”	037b
	“Vomiting, headaches, lethargy, diarrhea, regurgitation”	032
	“In the beginning, loose stools”	015
Fatigue	“Light headedness and general lethargy”	023b
	“I was always tired”	026b
	“The way I felt. Stomach pain, fatigue, irritability”	025b
Stomach pain	“Stomach pain 2 to 3 h after eating. The pain would be especially painful night causing me to toss and turn all night”	026b
	“Nausea, vomiting, burping, stomach pain”	042
	“Stomach pain”	054
Headaches	“Vomiting, headaches, lethargy, diarrhea, regurgitation”	032b
	“Stomach blowit and head aches”	044b
	“Headaches pretty quickly after taking medicine, but that improved a bit over time”	043
Mode of administration	“None, other than the mild unpleasantness of the injection itself”	040
	“Do not liking taking the shot. Would rather take a pill”	024
	“Poking with the needle in the tummy were bit uncomfortable but for the sake of controlling the sugar I guess it was ok”	027

aQuotes are reported exactly as they were submitted on the electronic tablet by the respondent

bParticipant discontinued treatment

Beyond these most frequently reported disadvantages, participants (n = 22/57; 38.6%) reported 24 additional disadvantages. Each of these additional disadvantages was reported by three or fewer participants and covered a diverse range of concepts, including other digestive problems (e.g. constipation; n = 3), diet (e.g. loss of appetite; n = 3), pain and discomfort (n = 3), general health (n = 3), cold/flu symptoms (n = 2), sleep disturbances due to pain (n = 2), urinary problems (n = 1), feeling different (n = 1), emotional (irritability; n = 1), excessive weight loss (n = 1), and aversion to medical treatment (n = 1).

The main disadvantages reported by participants who completed all trial procedures (n = 43) and excluding those who discontinued from the trial early (n = 14) were also evaluated (Fig. 3). No participants who discontinued treatment reported ‘no disadvantages’, whereas more than a third of participants who completed all trial procedures reported ‘no disadvantages’ (n = 15/43; 34.9%). A larger proportion of participants who discontinued treatment reported ‘nausea’, ‘vomiting’, ‘diarrhoea’, ‘fatigue’, ‘stomach pain’, and ‘headaches’ than those participants who completed all trial procedures (Fig. 3). ‘Fatigue’ (n =5/14; 35.7%) and ‘sleep disturbances’ (n = 2/14 [14.3%]; data not shown in Fig. 3) were only reported by those participants who discontinued treatment early. A further 14 disadvantages were reported by eight participants, all of whom discontinued treatment early. Each disadvantage was reported by one participant. These concepts were ‘less liquid intake’, ‘irregular eating cycles’, ‘heartburn’, ‘chest pain’, ‘dehydration’, ‘less urinating’, ‘irregular bowel movement’, ‘regurgitation’, ‘bloating’, ‘light headedness’, ‘irritability’, ‘difference felt in body’, ‘flu-like symptoms’, and ‘pancreas enzyme high during trial’.

Co-occurrence of Benefits and Disadvantages as Reported via PQAT Questions 1 and 2

Participants’ responses to questions 1 and 2 of the PQAT were examined to identify potential relationships between concepts (Table 4). A large proportion of participants who reported weight loss also reported improved or controlled blood glucose levels (n = 18/26; 69.2%). Over half of participants who reported feeling better also reported improved glucose levels/control (n = 7/10; 70.0%). Similarly, half of participants who reported feeling better also reported weight loss (n = 5/10; 50.0%). Over half of participants who reported improved energy levels also reported improved or controlled glucose levels (n = 4/7; 57.1%).

Table 4

Co-occurrence of main reported benefits and disadvantages

Key disadvantages were also commonly reported alongside key benefits (Table 4). For example, over half of participants who reported nausea and vomiting also reported improved or controlled glucose levels (n = 11/16 [68.8%] and n = 8/11 [72.7%], respectively) and weight loss (n = 10/16 [62.5%] and n = 7/11 [63.6%], respectively). As expected, a large proportion of participants who reported vomiting also reported nausea (n = 7/11; 63.6%), and a large proportion of participants who reported diarrhoea also reported vomiting (n = 4/6; 66.7%). Half of participants who reported stomach pain also reported fatigue (n = 2/4; 50.0%).

Question 3: Willingness to Continue with the Drug Received During the Trial

Overall, 40 participants (70.2%) responded that they would be willing to continue treatment. Of the 14 participants who discontinued treatment, six (42.9%) reported that they would be willing to continue with treatment.

The most commonly reported reasons for being willing to continue with treatment were improved glucose levels/control (n = 8/25; 32.0%), weight loss (n = 4/25; 16.0%), feeling better (n = 4/25; 16.0%), and treatment efficacy (n = 4/25; 16.0%). Participants did not provide any further detail as to how the treatment was effective. The most commonly reported reasons for not being willing to continue with treatment were vomiting (n = 4/14; 28.6%) and nausea (n = 3/14; 21.4%). One participant stated that he would prefer a natural solution. Other disadvantages previously reported in response to question 2, such as headaches or sleep disturbances, were not reported as reasons for being unwilling to continue with treatment.

Participant responses to question 3 were considered alongside the benefits and disadvantages participants previously reported in response to questions 1 and 2 of the PQAT. All participants who reported that they ‘feel better’ and had ‘improved energy levels’ responded that they would be willing to continue with treatment. Most participants who reported the main disadvantages in response to question 2 reported that they would not be willing to continue with treatment. This was particularly noticeable for participants who reported fatigue (n = 5), stomach pain (n = 4), and headaches (n = 4), for whom 75–80% reported that they would not be willing to continue with treatment. Conversely, 60% of participants who reported mode of administration as a disadvantage (n = 5) reported they would be willing to continue with treatment. Interestingly, 50% of the patients who reported diarrhoea (n = 6) reported that they would be willing to continue with treatment.

Six participants reported a conditional statement: that they would continue with treatment if a pre-specified condition was met. These conditions included cost—two participants reported that they would be willing to continue taking the treatment if the cost was affordable—and dose—two participants reported that they would continue treatment if the medication was given at a lower dosage. One participant stated that he would continue taking the medicine if his doctor allowed it, and one participant reported that he would continue treatment if the nausea and flu symptoms stopped.

Question 4: Balance Between Benefits and Disadvantages

When asked to compare the benefits of the drug received to the disadvantages experienced, 59.6% of patients reported that the benefits outweighed the disadvantages (scores of + 1 to + 3), 15.8% reported that the benefits and disadvantages were equal (0), and 24.6% reported that the disadvantages outweighed the benefits (− 1 to − 3).

Comparison of Specific Benefits and Disadvantages Reported by Participants and Responses to PQAT Question 4

Figure 4 indicates how participants reporting key benefits and disadvantages responded to question 4. At least 50% of participants reporting key benefits endorsed the highest positive response to question 4 (+ 3), indicating that the benefits of the drug received significantly outweighed the disadvantages. For example, 70% of patients who reported feeling better (n = 7/10) and 71.4% of participants who reported improved energy levels (n = 5/7) gave an answer of + 3 to question 4. On the contrary, only a small number of participants reporting weight loss (7.7%) or improved/controlled blood glucose (10.5%) selected − 3 as a response to question 4 (which would indicate that the disadvantages of the drug received significantly outweighed the benefits).

Fig. 4

Comparison of the reported benefits (question 1) and disadvantages (question 2) with participant responses to question 4

In contrast and consistent with a priori hypotheses, the majority of patients discontinuing treatment early (n = 9/14) reported that the disadvantages of treatment outweighed the benefits. All participants who reported ‘no disadvantages’ in response to question 2 selected ‘0’ in response to question 4 (indicating that the benefits were equal to the disadvantages) or provided a positive response, with 73.3% of participants selecting ‘3’ (indicating that the benefits of the drug they received significantly outweighed the disadvantages). In contrast, the majority of participants who reported ‘no benefits’ in response to question 1 selected ‘0’ or provided a negative response, indicating that the disadvantages outweighed the benefits. All patients who reported fatigue (n = 5), headaches (n = 4) or stomach pain (n = 4) reported that the disadvantages of the drug received outweighed the benefits (including improved/controlled blood glucose or weight loss, which were reported by more than half of these participants).

Concordance Between Participants’ Responses to Questions 3 and 4

The majority of participants (n = 29/40; 72.5%) who responded ‘yes’ to question 3 (indicating they would be willing to continue with treatment) provided a positive response to question 4, with 57.5% (n = 23/40) reporting that the benefits of the drug significantly outweighed the disadvantages (+ 3). In contrast, responses to question 4 among participants who responded ‘no’ to question 3 (indicating that they would not be willing to continue with treatment) were more widely distributed across the response scale. Among these participants, the majority responded with either − 3 (n = 6/17; 35.3%), indicating that the disadvantages significantly outweighed the benefits, or 0 (n = 4/17; 23.5%) indicating that the benefits were equal to the disadvantages. This is largely consistent with the hypotheses stated in Sect. 2.3.3, that participants who reported being unwilling to continue treatment would indicate that the disadvantages of treatment outweighed the benefits of treatment. Further details are provided in Fig. 5.

Fig. 5

Comparison of participants’ response to question 3 (willingness to continue with treatment) and question 4 (benefit–disadvantage ratio). The percentage of participants was calculated as a proportion of participants in each group (i.e. 57 for the full PQAT sample, 40 for participants who responded ‘yes’ to question 3, and 17 for participants who responded ‘no’ to question 3). PQAT Patient’s Qualitative Assessment of Treatment

PQAT Completion Time

The length of time it took each participant to complete the PQAT was recorded on the electronic tablet device. The mean time for all PQAT respondents (n = 57) to complete the PQAT was 5 min 21 s, but the majority of participants (n = 40/57; 70.2%) responded within 2–5 min. The median time taken to complete the PQAT was 3 min 46 s (n = 57).

Total word counts were calculated for participants’ responses to each qualitative question of the PQAT. Overall, word counts for questions 1 (benefits) and 3 (willingness to continue treatment) were greater than question 2 (disadvantages). Figure 6 presents the association between the PQAT completion time and the total word count in responses to questions 1, 2, and 3. As expected, the general trend observed indicated that shorter responses to the PQAT questions (i.e. fewer words) were associated with shorter completion times.

Fig. 6

Correlation of time to complete the PQAT and total word count. PQAT Patient’s Qualitative Assessment of Treatment

As shown in Fig. 6, a subgroup of participants (n = 13) required longer than 6 min to complete the questionnaire. This subgroup had a higher mean age than the whole population, with all respondents over 50 years old. The mean (range) age for participants taking longer than 6 min to complete the PQAT was 61.2 (50–71) years, compared with 56.6 (33–77) years in the whole sample.

Discussion

Integrating the patient perspective into drug or health technology development is of increasing interest to treatment decision makers [9, 24–26]. The PQAT is a unique patient-reported instrument designed to collect individualised patient-perceived benefits and disadvantages of a treatment and how these are viewed in balance by patients.

The first example of integration of the PQAT as part of a clinical trial is illustrated with the DRI13940 study. Participant’s responses to question 1 of the PQAT highlighted key benefits of treatment, including objective clinical markers of efficacy (improved or controlled blood glucose levels, weight loss) and more subjective markers not generally considered in clinical decision-making (feeling better, improved energy levels). The benefits of improved or controlled blood glucose levels and weight loss have been previously reported with GLP-1R agonists treatment [27-29]. Disadvantages of treatment reported by participants in response to question 2 were mainly related to drug adverse effects (nausea, vomiting, diarrhoea, fatigue, stomach pain, or headache) commonly associated with this therapeutic class or to the injectable mode of administration.

Conceptual saturation was assessed for questions 1 and 2 of the PQAT. This enables researchers to demonstrate that they have covered their topic adequately by having a sufficient number of cases to explore their concepts [30]. No new concepts were elicited in the final group of respondents. This demonstrates that although the sample size for this study was relatively small (n =57), it was sufficient to capture the breadth of participants’ responses.

Using the PQAT, the most common reasons reported by participants for being willing to continue their diabetes treatment included improved glucose levels/control, weight loss, and feeling better. Main reasons reported by participants who were not willing to continue with treatment included vomiting and nausea. These findings suggest that physical adverse effects, such as nausea and vomiting, are key reasons for a participant not to pursue the treatment. More subjective concepts, such as feeling better or improved energy levels, could be key to understanding what encourages participants to adhere to treatment.

Approximately 70% of participants who reported feeling better and improved energy levels, as well as approximately half of the participants who reported improved blood glucose levels/control and weight loss, endorsed the highest positive ratio in response to question 4. All participants who reported fatigue, headaches, and stomach pain reported a negative ratio, indicating that these disadvantages of the drug significantly outweighed any benefits.

Largely, participants’ responses to questions 3 and 4 were consistent with each other and with a priori hypotheses (i.e. participants who responded they would be willing to continue the treatment in question 3 also gave a positive benefit–disadvantage ratio to question 4). However, for some participants (19.3%) responses were not consistent, e.g. where patients reported being willing to continue with treatment yet endorsed a negative benefit–disadvantage ratio indicating that the disadvantages of the drug they received outweighed the benefits, and vice versa. Such discrepancies may suggest that participants would be willing to continue with treatment regardless of the disadvantages outweighing the benefits, or that they may not have fully understood the question. Alternatively, participants’ willingness to continue with the drug may be influenced by other factors independent of the specific benefit–disadvantage ratio of the drug itself. Responses provided within the additional free-text space for question 3 provide some insight into additional factors that may influence willingness to continue treatment (beyond perceived benefits and disadvantages). These include considerations regarding the affordability of the treatment, whether participants’ doctors would allow them to take the treatment, and patients’ aversion to medical treatments overall (one patient reported they preferred a natural solution). As such, certain patients may have acknowledged that the benefits of treatment outweighed the disadvantages (or vice versa), but were influenced by other, external reasons for wanting to continue to take the treatment. An updated version of the PQAT (PQATv2) includes two additional quantitative questions asking participants to rate the level of the benefits and disadvantages reported in response to questions 1 and 2. This will help to understand participants’ responses to questions 3 and 4 (willingness to continue with treatment and benefit–disadvantage ratio).

Overall, the completion time for the PQAT was short, with a mean completion time of approximately 5 min. A subgroup of participants did require longer than 6 min to complete the questionnaire. This may be related to age; the mean age of this subgroup of participants was higher than that of the whole sample (61.2 vs. 56.6 years), with all participants being over 50 years old.

Strengths of the PQAT

Developed as a simple generic questionnaire, the primary strength of the PQAT is that it can be employed in large samples across multiple trials and therapeutic areas. This methodology trades a degree of depth of individual patient data for a breadth of understanding across a larger population.

Secondly, the PQAT has a short completion time, with a mean time to complete of approximately 5 min. Combined with the absence of irrelevant answers, this reinforces that the majority of participants found the instrument easy to understand and complete in the electronic format provided in this study (electronic tablet). Electronic administration of the PQAT also facilitates quick access and transfer of data, and minimises the likelihood of secondary data-entry errors and missing datapoints. Additionally, there were no issues with illegible writing, and all responses were interpretable, despite some spelling and grammatical errors.

Participants completing the PQAT have experienced the medical intervention in question, meaning that their answers will provide an informed opinion of the treatment, based on their own real-life experiences. This is an advantage compared to traditional stated-preference methods such as discrete-choice experiments, which rely upon a hypothetical or decontextualised situation [31].

PROs typically utilise close-ended questions with pre-defined response options. When evaluating novel therapies in investigational studies, it is not always possible to identify or pre-specify all concepts that are most relevant or least relevant for participants beforehand. This is particularly true for concepts such as treatment-associated adverse effects. As such, designing PROs that assess all possible concepts can be both inefficient and overly burdensome to participants. Participants’ qualitative responses to PQAT items 1, 2, and 3 included concepts that may not be traditionally captured in PROs that are typically considered within benefit–risk frameworks. For example, the concept of ‘feeling better’ was reported by a significant number of participants and frequently reported among participants who reported the positives of treatment to outweigh the disadvantages (i.e. those who indicated a positive benefit–disadvantage ratio). However, as a heterogeneous and general concept, ‘feeling better’ would likely not be captured in a PRO designed to be a reliable and valid assessment of disease symptoms or impacts. Similarly, mode of administration was reported as both a benefit and a disadvantage by participants. While mode of administration may be considered in terms of benefit within current benefit–risk frameworks, it would rarely be considered under risk, which is typically dominated by the adverse effects of an intervention.

Combining the richness of patient-reported qualitative free-text data with quantitative data allows associations between responses to the quantitative and qualitative questions to be identified. This provides a deeper understanding of the drivers behind how patients balance benefits and disadvantages, and their willingness to continue treatment or not. Therefore, the PQAT has significant potential as a valuable means of eliciting information regarding the most meaningful patient-perceived benefits and disadvantages of treatment.

Limitations of the PQAT

It is important to acknowledge limitations of this first application of the PQAT in the DRI13940 trial. The PQAT was only available to participants in English-speaking countries (i.e. the USA and Canada). Evidence suggests that the sample of PQAT respondents was largely representative of the broader DRI13940 population, and results from the conceptual saturation analysis demonstrate that the sample size was sufficient to capture the breadth of participants’ qualitative responses. However, the sample of PQAT respondents in the study (n = 57) did limit the scope of potential quantitative analyses that could have been used to evaluate and explore the PQAT data. In particular, sample sizes were not considered sufficient for supporting inferential statistics to explore group differences (e.g. analysis of variance [ANOVA]) or predictors of individual harm–benefit evaluations (e.g. through the use of logistic regression models).

It is important to acknowledge that responses to the free-text questions in the PQAT were short in length and often lacked detail. As a self-completed measure based on standard questions, the version of the PQAT used in this study does not afford the opportunity to probe further to understand the relative importance of elicited concepts or to ensure that respondents’ answers are accurately articulated and interpreted. For example, very few participants in the present study provided answers that elaborated beyond a simple reference to the concept. Descriptive terms related to frequency, severity, duration, bothersomeness, or importance of elicited concepts were rarely used by participants. The lack of detail provided by respondents may also be due to the use of an electronic tablet, which some respondents may not find particularly user-friendly. Therefore, it has been decided to use a weblink for future trials, so that patients can complete the PQAT from their device of choice (computer, tablet, or smartphone). Further, due to its brevity, the qualitative data, while informative, did not provide any indication on the perceived magnitude of the benefits and disadvantages from the patients’ perspective. For example, one respondent reporting that the benefits outweigh the disadvantages on question 4 may have experienced small benefits and no disadvantages, whereas another with the same score may have experienced large benefits and some significant disadvantages.

Next Steps

The findings reported here from the PQAT could prove valuable to a range of stakeholders involved in treatment decision-making. For example, knowing how the new treatment is perceived by a patient within phase II clinical trials can be used by biopharmaceutical companies to inform phase III clinical trials and better understand the impact of adverse events associated with the treatment. This information could also be used for dose selection and to identify or develop support programs that can be helpful for patients taking certain treatments, increasing adherence and therefore efficacy, or treatment success [1]. This is particularly important for chronic conditions such as diabetes.

Data from the PQAT may also be of interest to regulatory authorities, who are becoming increasingly interested in understanding the patient’s perspective when considering benefits and harms as part of their benefit–risk frameworks [2-7]. Similarly, reimbursement agencies and HTA bodies have shown an interest in understanding how new products are received by patients [8]. Finally, the data gathered could be used to identify differences in patient experiences and perceptions of treatment according to demographic, clinical, and attitudinal factors. This information could be used to make more personalised and effective treatment decisions by healthcare professionals and patients, improving adherence and satisfaction with treatment.

Completion of the PQAT among larger samples of participants will allow future exploration of different concepts among participants with a range of demographic and clinical characteristics. Inferential statistical tests and analysis of relationships between variables and elicited concepts, using correlational analyses or more complex modelling analyses, can be employed.

A second version of the PQAT (PQATv2) is currently being administered at the patient’s home in other diabetes trials via a weblink, allowing the patient to complete the questionnaire on the device of their choice. The two additional questions included in the PQATv2 will provide further information to capture the level of benefits and disadvantages identified on 0–10 numeric rating scales.

In the future, it is envisaged that the PQAT could be used to develop strategies for comparison with other treatments by directly addressing benefits and disadvantages identified by patients. The generic nature of the questions means that the questionnaire can be used in other disease conditions. Such application may require additional testing depending on the therapeutic area (i.e. in oncology the balance between benefits and adverse effects may be different). It is also envisaged that the tool can be used beyond a clinical trial context following minor adaptation of the text (e.g. removal of references to the drug administered during the trial). For example, the PQAT could be used in the outpatient setting to support the development of a personalised treatment plan during shared decision-making between patient and physician.

Conclusion

The PQAT is a novel methodology that utilises a mixed-methods approach, combining open-ended free-text questions (qualitative) and quantitative questions with fixed-choice response options. Findings from study DRI13940 demonstrate that the PQAT is simple and easy for participants to understand. The information gathered provides sufficient information to gain an understanding of the benefits and disadvantages of a treatment from the patient’s perspective and how patients balance these benefits and disadvantages when deciding whether or not to continue with treatment. Such information can be used to identify subgroups of patients with differing opinions of benefit–harm attributes, which can be used to individualise patient care and improve medication adherence. Future application of the PQAT within clinical trial and real-world settings presents an opportunity to explore the value and utility of individualised treatment benefit/disadvantage assessment, which is likely to be of interest to a wide range of stakeholders (including regulators, payers, healthcare professionals, and patients).

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 336 kb)

Key stakeholders involved in decision-making are recognising the value in understanding how patients perceive and balance the benefits and harms of a treatment.

The Patient’s Qualitative Assessment of Treatment (PQAT) is a new patient-reported outcome measure designed to identify the benefits and disadvantages of treatments received during clinical studies as perceived by individual patients.

The purpose of this article is to discuss the development and initial use of the PQAT as an exploratory endpoint in a phase II clinical trial for a new treatment for type 2 diabetes mellitus.