Data Mining of Free-Text Responses: An Innovative Approach to Analyzing Patient Perspectives on Treatment for Chronic Rhinosinusitis with Nasal Polyps in a Phase IIa Proof-of-Concept Study for Dupilumab.

PURPOSE: Patient perspective is an important and increasingly sought-after complement to clinical assessment. The aim of this study was to transcribe individual patients' experience of treatment in a dupilumab clinical trial through free-text responses with analysis using natural language processing (NLP) to obtain the unique perspective of patients on disease impact and unmet needs with existing treatment to inform future trial design. PATIENTS AND METHODS: Patients with chronic rhinosinusitis with nasal polyps (CRSwNP) who were enrolled in a Phase IIa randomized controlled trial comparing dupilumab with placebo (NCT01920893) were invited to complete a self-assessment of treatment (SAT) tool at the end of treatment, asking, "What is your opinion on the treatment you had during the trial? What did you like or dislike about the treatment?" Free-text responses were analyzed for the overall cohort and according to treatment assignment using natural language processing including sentiment scoring. In a mixed-methods approach, quantitative patient-reported outcome (PRO) results were utilized to complement the qualitative analysis of free-text responses.
RESULTS: Of 60 patients enrolled in the study, 43 (71.6%) completed the SAT and responses from 37 patients were analyzed (placebo, n = 16; dupilumab, n = 21). Word analyses showed that the most common words were "smell," "improve," "staff," "great," "time," and "good." Across the whole cohort, "smell" was the most common symptom-related word. The words "smell" and "experience" were more likely to occur in patients treated with dupilumab. Patients treated with dupilumab also had more positive sentiment in their SAT responses than those who received placebo. The results from this qualitative analysis were reflected in quantitative PRO results.
CONCLUSION: "Smell" was important to patients with CRSwNP, highlighting its importance as a patient-centric efficacy outcome measure in the context of clinical trials in CRSwNP. TRIAL REGISTRATION: ClinicalTrials.gov, NCT01920893. Registered 12 August 2013, https://www.clinicaltrials.gov/ct2/show/NCT01920893.

Introduction

To comprehensively understand illness and the effects of treatment, it is necessary to supplement clinical, radiologic, laboratory, and other evidence reported by clinicians with the patient’s own perspective. Together, these elements support a fundamental role of medical science: “to transcribe the individual patient’s experience of illness; to make a doctor’s story out of the patient’s own”.1 Each patient has their own narrative of their illness and constructive engagement with this narrative, termed “narrative medicine”, may uncover valuable diagnostic and therapeutic options.2 In the context of clinical trials, patient experiences, perspectives, needs, and priorities are increasingly sought by healthcare regulators, payers, and physicians.3,4 The patient’s perspective can assist in determining disease impacts, identifying unmet medical needs for existing and new treatments, informing new subpopulations and targets for drug discovery, and evaluating endpoints of relevance to patients.4 Recent guidance recommends that patient experiences and qualitative research should be incorporated early in the drug development process,5 and the use of patient-reported outcome measures (PROMs) is increasing.6

Patient satisfaction is one type of PROM; it refers to the patient’s evaluation of the process of taking the medication and the outcomes associated with the medication.7 While PROMs have been developed and validated to quantitatively describe patients’ treatment satisfaction,8 the concept is complex, subjective, and inherently heterogeneous, and can benefit from capture and analyses of unstructured responses to questions.9 Allowing the patient to give unstructured responses to questions without the constraints of closed questions not only builds awareness and insight of patient perspectives on disease and treatment, but is also potentially complementary to quantitative PROM data by providing information on experiences not captured using these tools.6,10 Methods that combine both quantitative and qualitative approaches have not often been utilized in clinical trials.

Mining patients’ “free-text” responses to open questions is an innovative approach to free-text analysis that provides relevant, interpretable, and actionable information10 on patient experience. In natural language processing (NLP), algorithms are utilized to derive patterns within the textual data, to allow for evaluation and interpretation of findings. The approach has been used successfully to explore the patient voice in atopic dermatitis, cancer, and psychiatry research.6,11,12

Chronic rhinosinusitis with nasal polyps (CRSwNP) is a chronic inflammatory disease of the nasal and paranasal sinuses that is associated with a high symptom and economic burden and poor health-related quality of life (HRQoL).13–17 CRSwNP predominantly displays a type 2 inflammatory signature, which is associated with the prominent expression of the cytokines interleukin (IL)-4, IL-5, and IL-13.18,19 Information on patient perspective of CRSwNP treatment during a randomized clinical trial is lacking.

Dupilumab, a fully human VelocImmune®-derived20,21 monoclonal antibody, blocks the shared receptor component for IL-4 and IL-13, cytokines that are key and central drivers of type 2 diseases such as atopic dermatitis, asthma, CRSwNP, allergic rhinitis, and food allergies, which are often associated as comorbidities,22 thus inhibiting their signaling. Dupilumab is approved for patients with type 2 inflammatory diseases, including atopic dermatitis, asthma, and CRSwNP.23,24

In the Phase IIa proof-of-concept study (NCT01920893) and in two Phase III studies (NCT02912468; NCT0289454) in patients with CRSwNP refractory to intranasal corticosteroids, dupilumab on a background of mometasone furoate nasal spray (MFNS) significantly improved both subjective and objective assessments, including symptoms such as sense of smell, endoscopic nasal polyp (NP) score, and computed tomography (CT) scan score, respectively.25,26 Dupilumab also showed improvement in PROMs including disease severity and HRQoL.27

The objectives of the present study were to (i) explore the feasibility of capturing patient narratives of illness and treatment in a dupilumab clinical trial in CRSwNP using free-text responses and (ii) investigate the potential of transcribing these narratives using natural language processing (NLP) to gain insights to inform the design of future clinical trials of treatments for CRSwNP. Such insights might include the identification of subgroups of patients with high unmet needs, or outcomes most important to patients in this study population, which has different selection criteria and thus different characteristics from the populations used in the development of validated PROMs. A mixed-methods approach was utilized, in which the qualitative analysis of free-text responses was complemented with quantitative PROM results.

Patients and Methods

Population

This analysis was based on a Phase IIa, multicenter, randomized, double-blind, placebo-controlled, parallel-group study in 60 adults with CRSwNP, details of which are reported elsewhere.25 Briefly, patients were assigned 1:1 to add-on therapy with subcutaneous dupilumab (a 600 mg loading dose followed by 300 mg once weekly for 16 weeks) or matched placebo, on a background of MFNS.

Data Collection

As part of the study, patients had the option of completing a voluntary self-assessment of treatment (SAT) tool at the end of treatment. Patients were asked to respond to an open-ended question with a free-text input that asked, “What is your opinion on the treatment you had during the trial? What did you like or dislike about the treatment?” The trial was conducted in the USA, the Netherlands, the UK, Sweden, and Spain, and patients provided a handwritten response to the SAT in their native language. Responses in Dutch, Swedish, and Spanish were digitized, transcribed, and linguistically translated into English using forward and backward translations. Responses missing the anonymized Subject ID were not analyzed, as treatment arm could not be determined.

Analyses

Natural Language Processing

Responses to the SAT were analyzed using NLP, the process of automated extraction and structuring of text to identify patterns and derive meaningful conclusions from unstructured text through analysis of word occurrence, frequency, and linguistic features.28 Analysis was performed on the translated dataset.

The frequencies of specific words were tabulated and word clouds constructed for the overall population and the two treatment arms (dupilumab, placebo) to derive a visual representation of the most frequently reported words, in which font size within the cloud indicates word frequency.29 In order to avoid visual noise, the word “trial” was removed, and only words appearing at least twice were included.

Co-occurrence identified words with the highest degree of centrality. The list of co-occurrences was then ordered by decreasing frequency to construct a network (using R software). Using this network, the number of subcommunities and words belonging to each subgroup were determined. Finally, community detection was used to define a “natural” division of a network into groups of nodes using the Fastgreedy algorithm.30,31

Sentiment analysis, which helps determine overall sentiment in terms of emotions or opinions expressed within an unstructured piece of text, was used to enable a deeper understanding of the attitudes of a person with respect to a topic in a piece of text. Participants’ sentiments of their treatment experience were determined by matching individual words to their sentiment scores, which was provided by an external sentiment reference dataset. Sentiment labeling as positive, neutral, or negative was performed manually for all available samples. Sentiment analysis was performed for the following outcomes: participants’ feelings about their medication (Medication); participants’ feelings about their treatment (Treatment); and participants’ overall experience (Overall experience).

Disease Outcome Measures

The 22-item Sino-Nasal Outcome Test (SNOT-22) score for assessment of impact on HRQoL and University of Pennsylvania Smell Identification Test (UPSIT) score, and daily patient-reported loss of smell score for assessment of impact on loss of sense of smell, were evaluated in the subgroup of patients who completed the SAT and included their Subject ID (for whom treatment allocation could be assigned) and for the subgroup who did not complete the SAT or did not provide Subject ID. PROMs were analyzed using a mixed-effect model with repeated measures (MMRM) approach as described previously.25 The model included change from baseline as response variable with fixed-effects factors for treatment, stratification (comorbid asthma, biopsy performed), visit, treatment by visit interaction, nasal polyp score baseline value, and baseline by visit interaction. The model did not impute missing data points.

Results

Dataset

Of 60 patients enrolled in the Phase IIa study, 43 (71.6%) completed the voluntary SAT, and responses from 37 patients were analyzed (placebo, n = 16; dupilumab, n = 21; six responses were not analyzed due to missing Subject ID). SATs were completed in Dutch (n = 13), English (n = 11), Spanish (n = 14), and Swedish (n = 5). Text responses varied between four and 226 words. Demographics and baseline clinical characteristics are summarized in Table 1. The treatment groups were generally well matched at baseline, although patients assigned to placebo had a longer mean duration of disease than those assigned to dupilumab. Patients who did not complete the SAT had worse baseline UPSIT and loss of smell score than patients who completed the SAT.

Table 1

Baseline Demographics and Clinical Characteristics

	Patients Who Completed the SAT		Patients Who Did Not Complete the SAT
	Placebo/MFNS (n = 16)	Dupilumab/MFNS (n = 21)	Placebo/MFNS (n = 14)	Dupilumab/MFNS (n = 9)
Age, mean (SD), years	50.4 (10.5)	46.4 (8.5)	48.0 (7.4)	49.9 (12.4)
Male sex, n (%)	8 (50.0)	13 (61.9)	8 (57.1)	5 (55.6)
NP duration, mean (SD), years	10.2 (8.0)	7.5 (5.6)	12.9 (9.5)	7.8 (7.7)
Bilateral endoscopic NPS, mean (SD), range 0–8a	5.8 (0.7)	6.0 (0.7)	5.6 (1.1)	5.7 (1.5)
Lund–MacKay total CT score, mean (SD), range 0–24a	18.4 (4.6)	19.0 (4.5)	19.1 (6.6)	17.8 (6.3)
Smell test (UPSIT) score, mean (SD), range 0–40b	13.9 (4.5)	11.8 (7.0)	17.6 (10.4)	15.1 (10.8)
Sense of smell loss, mean (SD), range 0–3a,c	2.9 (0.3)	2.5 (0.9)	2.7 (0.7)	2.2 (1.1)
SNOT-22 total score, mean (SD), range 0–110a	41.1 (22.2)	43.1 (19.8)	39.9 (17.7)	37.6 (14.1)
CRSwNP disease severity (VAS), mean (SD), range 0–10 cma	5.4 (3.1)	6.3 (2.5)	7.7 (1.5)	6.7 (3.4)
Patients with comorbid asthma, n (%)	10 (62.5)	13 (61.9)	9 (64.3)	3 (33.3)
Patients with comorbid NSAID-ERD, n (%)	5 (31.3)	6 (28.6)	5 (35.7)	1 (11.1)

Notes: aHigher scores indicate worse status. bHigher scores indicate better status. c0 = no symptoms, 1 = mild symptoms, 2 = moderate symptoms, 3 = severe symptoms.

Abbreviations: CRSwNP, chronic rhinosinusitis with nasal polyps; CT, computed tomography; MFNS, mometasone furoate nasal spray; NP, nasal polyposis; NPS, nasal polyp score; NSAID-ERD, non-steroidal anti-inflammatory drug-exacerbated respiratory disease; SAT, self-assessment of treatment; SD, standard deviation; SNOT-22, 22-item Sino-Nasal Outcome Test; UPSIT, University of Pennsylvania Smell Identification Test; VAS, Visual Analog Scale.

Word Clouds

Construction of a word cloud based on word frequency is shown for the overall cohort of 37 patients who completed the SAT in . Words that occurred at high frequency included “smell,” “improve,” “staff,” “great,” “time,” and “good.” Word clouds based on word frequency were also constructed for completers who were assigned to dupilumab (Figure 1A) or placebo (Figure 1B). The word cloud for dupilumab differed from that for placebo. In the dupilumab arm, the most frequent words to follow “good” were “smell” and “not.” In the placebo arm, in contrast, the most frequent words to follow “good” were “not” and “improve.”

Figure 1

Word clouds in patients treated with (A) dupilumab or (B) placebo. Word cloud based on frequency (higher frequency = bigger font size). Only words that occur twice or more are shown. Stop words and the words “study” and “treatment” are omitted.

Word Co-Occurrences

Word clusters reporting the co-occurrences of words used by patients treated with dupilumab or placebo are shown in Figure 2. These word clusters differed between dupilumab and placebo. In the dupilumab arm, examples of words that co-occurred were “smell, taste, sense, study, satisfy, problem, part,” “experience, effect,” and “polyp, breathe, improve, help, nose, runny, hope, shot, feel, drug.” In the placebo arm, examples of words that co-occurred were “treatment, staff,” “study, part,” “great, shot,” “breathe, nose, trial, test,” and “good, professional, feel, sense, smell, time.” The context of these co-occurring words can be seen in samples of full-text responses from individual patients ().

Figure 2

Co-occurrence of words in patients completing a self-assessment tool treated with (A) dupilumab, (B) placebo. Words in the same cluster were more frequently reported together. Colors identify word clusters.

Sentiment Analysis

Participants’ feelings related to their medication, how they were treated during the study, and their overall experience were explored in sentiment analysis in the overall cohort of SAT completers, and in patients assigned to placebo and dupilumab (Figure 3). More instances of positive sentiment were recorded for each domain in patients treated with dupilumab compared with placebo. The percentage of patients expressing negative sentiment in relation to medication was especially apparent in placebo-treated patients.

Figure 3

Sentiment analysis of patients completing a self-assessment tool: overall, dupilumab group, and placebo group.

Disease Outcome Measures

PROMs including loss of smell and HRQoL in the group of patients who completed the SAT are shown in . At Week 16, patients treated with dupilumab compared with placebo had greater improvement in SNOT-22 total score (least squares mean difference versus placebo [95% confidence interval]: −20.75 [−29.64, −11.85]; p < 0.0001), UPSIT score (14.50 [9.61, 19.39]; p < 0.0001), and loss of smell score (−1.35 [−1.94, −0.77]; p < 0.0001). Significant improvements with dupilumab versus placebo were also seen for UPSIT and loss of smell score in patients who did not complete the SAT, despite the small subgroup size (n = 23) (). SNOT-22 score also improved with dupilumab versus placebo in patients who did not complete the SAT, but the difference did not reach statistical significance.

Discussion

In CRSwNP, the burden of disease and value of interventions have previously been investigated from the patient perspective using PROMs.25,32,33 Studies measuring HRQoL in patients with CRSwNP have reported improvements in subjective outcomes following surgical intervention33 and drug therapy.33 However, no randomized studies involving patients with CRSwNP have used NLP to glean clinically important meanings from free-text patient narratives obtained within clinical trials. Free-text narratives allow patients to give unrestricted feedback without the constraints of structured instruments and are complementary to quantitative PROM data. Analysis of free-text responses using NLP has the potential to provide meaningful conclusions about experiences that are specific to the target population, and which may not be captured using traditional PROM questionnaires, which use “forced choice” answers.

Free text data analysis is increasingly used to evaluate patient perception, experience, and satisfaction with care delivery.34,35 The approach presented here to capture and analyze free-text responses in the setting of a clinical trial illustrates the insights that can be achieved to identify subpopulations with high unmet medical needs, targets for treatment, and outcomes that are of importance to patients for inclusion in clinical trials. To our knowledge, this is the first study to investigate free-text responses in the context of a clinical trial in patients with CRSwNP. More than 70% of patients provided a response on the SAT, showing the feasibility of this method for capturing patient experience. This approach also introduces an element of “narrative medicine” into the clinical trial process, which as well as being patient-centric, provides a window on the personal connections between patient and treatment that may not otherwise be captured and may help inform aspects of future trial design, such as subpopulations of patients with high unmet medical needs or the hierarchy of PROMS.

The present analysis provided patient “bedside feedback” on patients’ experience of CRSwNP treatments by evaluating the frequency with which words occurred in the free-text responses. “Smell” was the most common symptom-related word in the word cloud analysis, indicating the importance of smell to patients with CRSwNP, and its centrality, both in terms of how patients perceive the disease burden and the improvement in symptoms resulting from dupilumab therapy. The importance of smell perception is illustrated in the free-text response “I just hope and pray that my smell and taste lasts a long time. I love being able to smell my sweet two year old and my yummy food ☺. Thanks a million!”. In particular, the findings show that patients treated with dupilumab emphasized the improvement in smell and breathing. Patients treated with placebo, in contrast, were hopeful of smell improvement, which further reinforces the importance of this symptom for CRSwNP patients, and also more often expressed the care they received from medical personnel during the study.

The PROM instruments were scheduled for completion at study visits before any procedure and we cannot be certain of the sequence in which each individual patient completed the instruments. The potential for the SAT responses to be influenced by the PROM instruments was limited by the SAT being administered at the last visit and asking the patient to provide an overall impression of the treatment. The clear signal indicating the importance of smell in the SAT emerged despite smell being a relatively minor part of the PROM items. The negative influence of olfactory disorders on HRQoL has previously been highlighted using PROs.36–38 Further to the NLP analysis in this study, PROs including loss of smell and HRQoL reflected the findings of the text-mining approach. Patients treated with dupilumab, who demonstrated a qualitative improvement in smell and breathing in the SAT, had significantly greater improvements in patient-reported sense of smell and HRQoL than patients receiving placebo, who reported hope of smell improvement in the SAT. These findings support the value of utilizing a mixed-methods approach.

The present findings complement these earlier findings in providing qualitative support for the value that patients place on their sense perception and the impact its loss has on HRQoL. Word co-occurrences highlighted the different patient perspectives based on treatment assignment. With dupilumab, words related to sense perception had frequent co-occurrences with words such as “improvement” and “better” as in “I can smell and taste better than I have in the past 10 years.” Although patients who were assigned to placebo also expressed words related to sense perception, these words were more likely to co-occur with “hope” as in “I hope that if the medicine comes on the market that I will be eligible for it” and “satisfied” as in “I am very satisfied with how the study went.” From a general perspective, this self-assessment of treatment of CRSwNP using NLP allowed patients to respond without predefined choices, which are subject to possible investigator bias10 and do not generate the kind of in-depth data that may be gleaned from patient narratives.39 The text-mining approach in the present context provided this in-depth insight into patients’ treatment perspectives, and in particular provided patient-level evidence for smell as an important symptom for NP. NLP is thus an invaluable method for understanding how patients comprehend the impact of disease and its treatment on HRQoL.40,41 Having confirmed the importance of sense of smell perception from a patient perspective, an immediate application of this finding will ensure that sense of smell is adequately captured as a PRO measure in future studies of patients with CRSwNP. The approach used in the current analysis could be valuable for the evaluation of patient perspectives in other disease areas and to identify unmet medical needs, and aspects of treatment of importance to patients. The information obtained in this way will be useful to inform patient-centric study design as well as to evaluate change over time in patient perceptions.

Limitations of this analysis include the small sample number and potential for interpretation errors resulting from cultural and language differences in the study population. Due to the voluntary nature of the SAT, responders were self-selected. Additionally, there was no cognitive debriefing of study participants to elicit feedback regarding the relevance of the study question and their understanding of the question. The selected approach for this study also limits the ability and utility of quantifying the results of the natural language processing analyses.

Conclusion

This study demonstrates the feasibility and utility of using NLP of patient free-text responses to open questions to capture patient experience of CRSwNP and its treatment in their own voices. NLP of free-text responses provides qualitative data, which are not only useful in terms of “bedside feedback” in the specific population studied during the trial, but introduces an element of “narrative medicine” to discover perspectives that are important to the patient and should therefore be a focus of treatment. The results provide support for this approach to identify the most relevant concepts and related outcome measures for use in clinical trials of treatments. The present study found that “smell” was among the most frequently used words in patient free-text responses, highlighting sense of smell as particularly important to patients and identifying this as an important efficacy outcome measure in the context of future clinical trials.