A non-interventional observational study to identify and validate clinical outcome assessments for adults with phenylketonuria for use in clinical trials.

INTRODUCTION: Current clinical outcome assessments (COAs) are not effectively capturing the complex array of symptoms of adults with phenylketonuria (PKU). This study aimed to identify concepts of interest relevant to adults with PKU. Based on these concepts, COAs for patient-reported outcomes (PROs), observer-reported outcomes (ObsROs), and clinician-reported outcomes (ClinROs) were selected or developed and content validity was assessed.
MATERIALS AND METHODS: Concept-elicitation interviews were conducted with an international cohort of adults with PKU (n = 30), family member observers (n = 14), and clinical experts (n = 8). Observers and clinical experts were included to overcome the risk of lack of self-awareness among adults with PKU. The concepts of interests endorsed by ≥30% of patients, observers, and/or clinical experts were selected, mapped to items in existing COAs, and used to develop global impression items for patients, observers, and clinicians. Next, the content validity of the COAs and global impression items was evaluated by cognitive interviews with patients (n = 22), observers (n = 11), and clinical experts (n = 8). All patients were categorized according to blood phenylalanine (Phe) levels (i.e., <600 μmol/L, 600-1200 μmol/L, and >1200 μmol/L).
RESULTS: Concepts of interests were identified across four domains: emotional, cognitive, physical, and behavioral. After mapping, eight existing COAs were selected based on the concept coverage (six PROs, one ObsRO, and one ClinRO). The six PRO measures were considered as potentially fit-for-purpose. The ObsRO measure was not deemed relevant for use in observers of adults with PKU and only a subscale of the ClinRO measure was considered valid for assessing adults with PKU by clinicians. Due to the lack of existing COAs covering all concepts of interests, global impression items for symptom severity and change in symptoms were developed, which were limited to one question covering in total 14 concepts. Upon validation, some of the patient and observer global impression items were excluded as they were subject to lack of insight or could not be reported by observers. Due to the limited interaction time between clinician and patient, use of the clinician global impression items was not supported.
CONCLUSION: Existing COAs relevant to adults with PKU were selected and PKU-specific global impression items were developed by mapping the most frequently identified concepts of interests from internationally-conducted in-depth interviews. Future studies should address the appropriateness of the selected COAs and global impression items to assess if these can be used as efficacy endpoints in PKU clinical trials.

Introduction

Phenylketonuria (PKU) is a rare, autosomal recessive disorder caused by pathogenic variants in the phenylalanine hydroxylase (PAH) gene [1]. Despite the availability of pharmacological treatments, i.e., sapropterin dihydrochloride (Kuvan®) and recently pegvaliase (Palynziq®), the mainstay of PKU treatment remains a Phe-restricted diet supplemented with medical food [2], [3]. Management should be lifelong aiming to maintain blood Phe levels of adults with PKU below 360 and 600 μmol/L, as recommended by the US and EU guidelines, respectively [2], [3]. However, strict blood Phe control by dietary management is challenging, posing a significant burden to the adult living with PKU [4], [5]. As a result, adherence to the dietary restrictions decreases over time, leading to blood Phe levels above guideline-recommended thresholds in the majority of adult patients [6], [7].

Most severe neuropsychological complications can be prevented when initiating dietary management immediately upon diagnosing PKU during newborn screening [8]. However, early- and continuously-treated adults with PKU still experience higher rates for neuropsychological comorbidities than the general population, which can include a wide range of clinical manifestations, such as mood disorders (anxiety, depression), hyperactivity/inattention, and other deficits in executive functioning [9], [10], [11], [12], [13], [14], [15]. Due to the neuropsychological comorbidities and demanding dietary management, it is expected that PKU has an impact on the quality of life (QoL) of adult patients. The generic health-related quality of life (HRQoL) tool is, however, not sensitive enough to detect this impact as it was designed to assess the HRQoL of the general population, lacking specificity in PKU [16], [17]. Nevertheless, the emotional impact of PKU and its dietary management can be assessed by the PKU-specific QoL questionnaire (PKU-QOL) [17], confirming findings from other studies showing that the burden of diet is indeed contributing to some of the neuropsychological symptoms (e.g., mood) experienced by adults with PKU [9], [18], [19].

Due to the rarity and complexity of the disease, there are currently no valid and fit-for-purpose neuropsychological assessment tools that can be used as adequate endpoints in clinical trials to allow for full evaluation of the impact of lowering blood Phe levels by novel treatments from the patient perspective. This is further complicated by the heterogeneity among adults with PKU in overall functioning and the occurrence of neuropsychological symptoms [11]. Furthermore, adults with PKU may lack self-awareness due to neuropsychological impairments. This lack of self-awareness hampers self-rating of symptoms, which may seem evident to a clinician or familiar observer [20]. Additionally, patients with PKU may be reluctant to attribute symptoms that they are experiencing to their PKU.

Measures or tools developed prior to the Food and Drug Administration (FDA) Guidance for Industry (2009) did not consider patient input and were lacking demonstrated content validity. In accordance with the updated FDA guidance, a non-interventional, cross-sectional, observational, qualitative study was conducted in adults with PKU, observers, and clinical experts to first identify symptom, function, and HRQoL concepts of interest (COIs) that are important, relevant, and meaningful to adults with PKU (NCT03505125). In the first part of this qualitative study, the COIs were used to inform the identification of existing clinical outcome assessments (COAs) that are important and relevant to adults with PKU and that are sensitive to changes in self-reported blood Phe control. Additionally, PKU-specific global impression items were developed for patients (PGI), observers (OGI), and clinicians (CGI). In the second part, the content of the identified COAs and PKU-specific global impression items was examined by patients, observers, and clinical experts to determine the appropriateness of these assessments as potential efficacy endpoints in PKU clinical trials.

Materials and methods

Overview of the study design

The study design has been depicted in Fig. 1. During this non-interventional, cross-sectional, observational study, qualitative concept-elicitation and cognitive interviews based on best practice methods and regulatory guidelines were conducted with Palynziq-naïve adults with PKU, observers of these patients (i.e., family members), and clinical experts [21], [22], [23], [24], [25]. In the first part of the study, participants were questioned about concepts related, but not limited to, symptom descriptions, impacts, functional outcomes, HRQoL, important sequelae of PKU relative to blood Phe, and how performance may be affected by fluctuations or changes in blood Phe level. Based on the concept-elicitation interviews, the most frequently reported COIs were identified. The resulting COIs were mapped to a battery of existing COAs for patients, observers, and clinicians. In addition, global impression items were developed, which captured the most important PKU symptoms/signs/behaviors and assessed the severity of and changes in PKU symptoms in the patients as reported by patients (PGI), observers (OGI), and clinicians (CGI). In the second part of the study, the content validity of the identified COAs and global impression items was evaluated using cognitive interviews with adults with PKU, observers of these patients, and clinical experts. The study was approved by the local Institutional Review Boards (IRBs) and Ethics Committees in all participating study countries.

Fig. 1

Study flow diagram. A concept-elicitation interview study (Part A) was conducted with adults with PKU, observers of these patients, and clinical experts to identify concepts that are relevant to adults living with PKU. In addition to mapping of these key COIs to existing COAs, PKU-specific global impression items were developed for adult patients, observers, and clinicians. In a cognitive interview study (Part B), the content of these COAs and global impression items was validated by adult patients, observers of these patients, and clinical experts.

Participants

For the concept-elicitation interview study, Palynziq-naïve adults with PKU and observers of these patients were recruited from 10 clinical sites in Canada (n = 1), the United States (USA; n = 4), Germany (n = 2), Turkey (n = 2), and the United Kingdom (UK; n = 1). Patients (n = 30) and their observers (n = 14) were recruited using Phe category quota sampling per the following blood Phe categories: <600 μmol/L (<10 mg/dL), 600–1200 μmol/L (10–20 mg/dL), and > 1200 μmol/L (>20 mg/dL, i.e., classical PKU). These classifications do not reflect the clinical categorization of blood Phe but were employed to ensure that interviews were conducted with adults with PKU (and their associated observers) across a range of blood Phe levels, aiming to include more patients in the 600–1200 μmol/L and > 1200 μmol/L Phe categories as those are most likely to benefit from PKU therapies. Key eligibility criteria for patients and observers included 18 to 70 years of age and absence of significant impairment that (in the opinion of the investigators) would interfere with informed consent or participation in the interviews. Additionally, patients required a clinically-confirmed diagnosis of PKU and a blood Phe test within eight weeks of screening. Observers of patients had to be a partner, parent, adult child, sibling, or any other person who had at least eight hours per week of interaction with the affected participant enrolled in this study. The observers needed to be able to reliably report on the patient's current PKU symptoms, impacts, and QoL. Clinical experts (n = 8) had to have the equivalent of an MD, MS, or PhD with a minimum of 10 years of overall experience and be a currently practicing clinician treating adults with PKU with at least five years of experience in evaluating, prescribing, treating, and/or interacting with patients. Exclusion criteria for clinicians included any intellectual property and/or financial interest in COAs used for PKU individuals. For the cognitive interview validation study, adults with PKU (n = 22), observers of these patients (n = 11), and clinical experts (n = 8) were recruited from eight clinical sites across Europe and North America. Clinical sites in Turkey that contributed to the concept-elicitation interview study could not participate in the cognitive interview study because Turkish translations of the selected COAs were not available. Blood Phe category quota and eligibility criteria for patients, observers, and clinical experts were similar to the concept-elicitation interview study.

Procedures

Concept-elicitation interview study

The concept-elicitation interview study was conducted consistent with the FDA Guidance and the ISPOR Task Force recommendations for developing the content validity for newly developed patient-reported outcomes (PROs) [21], [22], [23], [24], [25]. Concept-elicitation and cognitive interviews were conducted by telephone in the patients' and observers' native language using a semi-structured interview guide. Clinical expert interviews were conducted in English. Interviewers were experienced in qualitative interviewing methodology and trained on the study interview guide. During the first half of the concept-elicitation interviews, symptoms, signs, behaviors, and impacts of PKU were elicited from patients and observers. After the open-ended portion of the interview, the interviewer asked the patient or observer to complete the PKU symptom survey. The survey was developed de novo based on COIs identified in a targeted PKU literature review (Supplementary files 1–3). After evaluation of the identified COIs, items were drafted and subsequently reviewed for clarity and conciseness. The PKU symptom survey was a 60-item questionnaire completed by patients during the telephone interview either via web-survey format or paper format. Using the 60-item PKU symptom survey, patients were instructed to select the symptoms they were experiencing in the past 8 weeks and symptoms over the past 12 months, describing if these symptoms fluctuated with blood Phe levels. Observers were also asked to complete a 60-item PKU symptom survey. They were instructed to rank what was most important to them as the observer and not to rank the importance from the patient's perspective. During the survey completion, the interviewers probed the interviewee with any symptoms endorsed in the survey that had not been raised previously during the concept-elicitation interviews to ascertain whether they were also endorsed as relevant and important. Upon completion of the interviews, participants completed demographic and medical history questions for sample descriptive purposes. The clinical experts were sent a similar 60-item survey prior to the interviews, aiming to identify the symptom experience of patients with PKU including information about frequency, intensity, and duration of symptoms. The clinical experts were asked to review and endorse signs, behaviors, and impacts on an inventory report covering emotional, cognition, behavioral, and physical domains identified from a targeted literature review of PKU in adults. The sample size was estimated to achieve saturation, which was assessed by blood Phe category and was considered achieved if no new concepts, descriptions of a concept, or terms emerged over two or more interviews [26], [27]. Only concepts that met saturation were considered COIs for consideration for the COA measurement.

Cognitive interview study

The cognitive interviews were conducted using a standardized interview guide to increase consistency across interviewers. Patients, observers, and clinical experts were asked to provide feedback on the COAs and global impression items to understand their thoughts on the instructions, response options, recall period, and item concepts. For the global impression items of change, patients, observers, and clinical experts were asked to review the items without completing them given that the questionnaire examines change over time for treatment in a clinical trial. Likewise, during the cognitive interviews, clinical experts were asked for feedback generally about the relevance of clinician-reported questionnaires for use in patients with PKU. All interviews were audio-recorded and transcribed with participants' permission obtained in accordance with IRB approved protocols. Typically for cognitive interview studies, a sample size of 10–15 participants is considered sufficient per subgroup of participants [26].

Analysis

Qualitative data analysis was completed in ATLAS.ti version 8 [28] and quantitative analysis was performed using SAS® statistical software version 9.4 (SAS Institute Inc., Cary, NC). The concept-elicitation interview data and PKU symptom survey results were analyzed to assess the most commonly reported PKU COIs. The a priori COI threshold was set at ≥30% of endorsement by patients, observers, and/or clinical experts. A priori cut-off criteria for prioritizing the most relevant concepts for item development have been used in various COA development studies [29], [30], [31]. COAs were identified through previous studies, PKU literature, ad hoc searches (i.e., conference abstracts, Cochrane, and AHRQ review on PKU), and expert opinion. COAs mentioned on FDA label claims (using Evidera's FDA PRO label database and FDA COA Compendium for adults) for indications, such as depression, anxiety, concentration issues, and attention deficit hyperactivity disorder (ADHD) were also considered. The concepts of the identified COAs were mapped to the most relevant PKU COIs. Those COAs with the best conceptual overlap with PKU COIs based on established criteria were selected (Supplementary Table 1). The global impression items went through an iterative review and revision process to address reading level, attempting to be as close to an 8th grade reading level as possible, while maintaining authentic patient language. Next, the cognitive interview data analysis assessed the clarity and relevance of instructions, items, response scale, and recall period based on respondent input, identifying the COAs and global impression items with the strongest supportive evidence of content validity for adults with PKU.

Results

Demographics and clinical characteristics of adult patients with PKU

The demographic characteristics of the patients (n = 30) participating in the concept-elicitation interview study, grouped by blood Phe category, are displayed in Table 1. Patients in the >1200 μmol/L blood Phe group were slightly older (37.7 ± 5.4 years) compared with patients in the <600 μmol/L (31.0 ± 6.3 years) and 600–1200 μmol/L blood Phe groups (30.8 ± 12.5 years). The <600 μmol/L and 600–1200 μmol/L blood Phe groups consisted mostly of female patients, while 83% of the patients in the >1200 μmol/L blood Phe group were male. Sixty-six percent of patients were either part-time or full-time employed and 63% of patients had completed a higher education. Most patients rated their overall health as very good (47%) or good (30%).

Table 1

Self-reported demographics of patients participating in the concept-elicitation interview study.

Characteristic	Total(n = 30)	Blood Phe <600 μmol/L(n = 7)	Blood Phe 600–1200 μmol/L(n = 11)	Blood Phe >1200 μmol/L(n = 12)
Age (years)
Mean (SD)	33.6 (9.2)	31.0 (6.3)	30.8 (12.5)	37.7 (5.4)
Range	18.0, 54.0	23.0, 40.0	18.0, 54.0	27.0, 47.0
Sex, n (%)
Female	19 (63%)	7 (100%)	10 (91%)	2 (17%)
Country, n (%)
Canada	1 (3%)	0 (0%)	1 (9%)	0 (0%)
USA	14 (47%)	3 (43%)	7 (64%)	4 (33%)
Germany	7 (23%)	2 (29%)	3 (27%)	2 (17%)
Turkey	4 (13%)	2 (29%)	0 (0%)	2 (17%)
UK	4 (13%)	0 (0%)	0 (0%)	4 (33%)
Current marital status, n (%)
Single	13 (43%)	2 (29%)	6 (55%)	5 (42%)
Married	15 (50%)	5 (71%)	3 (27%)	7 (58%)
Divorced	2 (7%)	0 (0%)	2 (18%)	0 (0%)
Employment status, n (%)a
Employed full-time	19 (63%)	5 (71%)	4 (36%)	10 (83%)
Employed part-time	1 (3%)	0 (0%)	0 (0%)	1 (8%)
Homemaker	2 (7%)	2 (29%)	0 (0%)	0 (0%)
Student	4 (13%)	0 (0%)	3 (27%)	1 (8%)
Unemployed	2 (7%)	0 (0%)	2 (18%)	0 (0%)
Other b	2 (7%)	0 (0%)	2 (18%)	0 (0%)
Education status, n (%)a
Primary school	1 (3%)	0 (0%)	0 (0%)	1 (8%)
High school	6 (20%)	0 (0%)	3 (27%)	3 (25%)
Vocational school	3 (10%)	1 (14%)	2 (18%)	0 (0%)
Some college or post-high school education or training	7 (23%)	0 (0%)	4 (36%)	3 (25%)
College degree	8 (27%)	3 (43%)	1 (9%)	4 (33%)
Postgraduate degree	3 (10%)	2 (29%)	0 (0%)	1 (8%)
Other c	2 (7%)	1 (14%)	1 (9%)	0 (0%)
Overall health, n (%)
Excellent	3 (10%)	2 (29%)	1 (9%)	0 (0%)
Very good	14 (47%)	3 (43%)	2 (18%)	9 (75%)
Good	9 (30%)	1 (14%)	6 (55%)	2 (17%)
Fair	4 (13%)	1 (14%)	2 (18%)	1 (8%)

Not mutually exclusive.

Other employment included trainee/apprentice (n = 1) and medical leave (n = 1).

Other education included “Diploma”, slightly higher than a bachelor's degree (n = 1), and lower secondary school (n = 1).

Phe: phenylalanine

Almost all adults with PKU (97%) reported being seen in a PKU, metabolic, or genetics clinic within the past 12 months, with 83% having their blood Phe levels checked at least once a year (Supplementary Table 2). Most patients (90%) reported being diagnosed and initiating treatment at an early age (i.e., two months of age or earlier) but metabolic control generally deteriorated over time. At the time of the interview, 73% of patients reported being adherent to a Phe- or protein-restricted diet, with 40% of patients indicating that they were being highly adherent to the dietary restrictions and only 20% of patients reported using sapropterin. Throughout life, only 30% of patients were highly adherent to the Phe- or protein-restricted diet and 33% were highly adherent to a medical food/amino acid supplement (Supplementary Table 2).

Supplementary Table 3 shows the demographic characteristics of the patients (n = 22) participating in the cognitive interview study, out of which half of the patients (n = 11) were also interviewed in the concept-elicitation interview study. Similar to the concept-elicitation interview study, the mean age of the patients in the <600 μmol/L blood Phe group (29.4 ± 6.5 years) was slightly younger than the 600–1200 μmol/L (33.2 ± 6.8 years) and > 1200 μmol/L (36.6 ± 6.3 years) blood Phe groups. The majority of PKU participants were female (59%). Patients primarily reported being employed full-time (64%) and having a college or postgraduate degree (59%). The overall self-reported health state was similar between patients participating in the concept-elicitation and cognitive interview study.

The majority of patients (96%) reported attending a PKU, metabolic, or genetics clinic within the past 12 months (Supplementary Table 4). Forty-five percent reported having their blood Phe levels checked 4 to 6 times a year or monthly, most of whom were from the <600 μmol/L or 600–1200 μmol/L blood Phe groups (80%). All patients reported being diagnosed with PKU at birth to two months of age. During childhood, most patients (96%) reported having well-controlled Phe levels, while blood Phe control declined over time, with 36% and 27% reporting moderately and poorly controlled Phe levels in adulthood, respectively. At the time of the interview, most patients (77%) reported taking medical food or amino acid supplements, eating special low-protein foods (64%), and/or following a Phe/protein-restricted diet (59%). No patients in the >1200 μmol/L blood Phe group reported being highly adherent to following a Phe/protein restricted diet or consuming medical food/amino acid supplements. Only a minority of the patients (18%) was using sapropterin for the treatment of their PKU (Supplementary Table 4).

Demographics of observers

The average age of the observers (n = 14) participating in the concept-elicitation interview study was 46.1 ± 9.0 years, ranging between 30 and 63 years. Half of the observers were male. Fifty percent of observers were recruited from the USA followed by 29% from Germany and 21% from Turkey. The observers were either spouses (50%), parents (43%), or other relationships (7%) to the adults with PKU. Almost all observers (93%) reported living with the affected participant and knowing the patient for more than five years (93%). The observers interacted with the affected participants for a mean of 44.9 h/week ranging between 8 and 99 h. Most observers (79%) reported being married and working either full- (43%) or part-time (21%). Forty-nine percent of observers reported completing some college or post-high school education or training (21%), college/university degree (14%), or postgraduate degree (14%).

The observers (n = 11) of the patients included in the cognitive interview study were on average 39.6 ± 11.9 years of age, ranging from 23 to 56 years of age. Five observers were interviewed in the concept-elicitation and cognitive interview study. Sixty-four percent of the observers were female. Thirty-six percent of observers were recruited from the USA followed by 27% from Germany and Italy, and 9% from Canada. Nearly all observers (91%) reported living with the patient with PKU. Most observers reported being married (82%), employed part- or full-time (54%), and 54% completed some sort of higher education. The mean hours/per week observers reported interacting with the patient were 51.8 h with a range of 15–80 h, with most observers (82%) reporting that they knew the patient for more than 5 years.

Demographics of clinical experts

The same clinical experts participated in the concept-elicitation and cognitive interview studies. Six clinical experts from the USA, one from Canada, and one from the UK were interviewed. Clinical experts had between 20 and 40 years of medical experience and training, and had 10 to 21 years of clinical experience with PKU. The clinical experts reported a total enrollment ranging between 30 and 150 adults with PKU per clinical expert. This represented between five to 10 patients a week with two to four clinics per month specifically for adults with PKU.

Identification of key concepts of interests

Both the concept-elicitation interview and PKU symptom survey data were evaluated to assess the COIs most important to patients, observers, and clinical experts and relevant to patients with different blood Phe levels. The identified COIs were reviewed and ranked by endorsement level. The COIs with at least 30% endorsement for any blood Phe category were retained and categorized at the concept level by domain (i.e., emotional, cognitive, physical, and behavioral). Based on the PKU symptom survey data and results from the interviews, Fig. 2 provides the comprehensive list of the most commonly reported COIs (N = 16) with ≥30% endorsement across blood Phe categories and averaged across patients, observers, and clinical experts. The overlap of these key COIs between patients, observers, and clinical experts is shown in Supplementary Table 5, demonstrating that eight additional COIs were endorsed by ≥30% of patients, observers, or clinical experts without reaching an endorsement level of ≥30% when averaged between the three groups.

Fig. 2

Most frequently identified concepts of interest (COIs) endorsed by a mean of ≥30% across adults with PKU, observers, and clinical experts according to the PKU symptom surveys and interviews and across blood Phe categories. Concepts of interests are grouped by domain and subdomain. The area of the box is proportional to the average percentage reported across patients, observers, and clinical experts.

Mapping of concepts of interests to existing clinical outcome assessments and development of global impression items

Forty-two COAs (Supplementary Table 6) were evaluated to assess their recall period, response options, whether patient input had gone into the initial development, and if the tool had sufficient information about its reliability and validity in adult populations (Supplementary Table 1). Many of the evaluated COAs were self- and observer-reported psychological and cognitive rating scales. A number of the reviewed COAs were found to be widely used in clinical and research settings and applied across many diseases. Many COAs were deemed ineligible for use as assessment tools owing to the limited overlap with the key COIs (i.e., subdomain and concept level), which were elicited from the concept elicitation interviews. Additionally, measures having any of the following characteristics were excluded: attribution included in item, non-relevant domains, long or vague recall period, double-barreled or complex items, and/or willingness of the developer to license the instrument.

Considering the above-described eligibility criteria, six PRO measures were identified as potentially fit-for-purpose for assessing part of the key PKU-specific COIs (Table 2). In addition to the PRO measures, the Conners' Adult ADHD Rating Scales-Observer Report Long Version (CAARS-O:L) and ADHD Rating Scale-IV (ADHD RS-IV) were identified as potentially fit-for-purpose ObsRO and ClinRO measures, respectively (Table 2).

Table 2

List of clinical outcome assessments (COAs) and PKU-specific global impression items, identified based on the outcomes of the concept-elicitation interview study.

	Patient	Observer	Clinical Expert
COAs	• Neuro-QOL short forms: 1) Emotional and Behavioral Dyscontrol 2) Anxiety 3) Cognitive Function 4) Fatigue 5) Sleep Disturbance • PROMIS Emotional Distress – Depression	• CAARS-O:Long	• ADHD-RS-IV
	Patients, observers, and clinical experts
Global impression items	Item 1. Short-tempered
	Item 2. Easily irritable
	Item 3. Anxious
	Item 4. Easily frustrated
	Item 5. Mood swings
	Item 6. Difficulty paying attention
	Item 7. Difficulty concentrating
	Item 8. Easily distracted
	Item 9. Easily forgetful
	Item 10. Low or lack of energy
	Item 11. Easily tired
	Item 12. Sleep problems
	Item 13. Headaches or migraines
	Item 14. Level of discomfort in social situations

ADHD-RS-IV: Attention Deficit Hyperactivity Disorder-Rating Scale-IV; CAARS-O: Long: Conners' Adult ADHD Rating Scales-Observer Report Long Version; Neuro-QoL: Quality of Life Outcomes in Neurological Disorders; PROMIS: Patient Reported Outcome Measurement Information System.

Due to the lack of existing COAs covering all key COIs, PKU-specific PGI/OGI/CGI items were developed to assess the severity of symptoms/signs over a 7-day recall period. Additionally, the PGI/OGI/CGI items were evaluating if these symptoms/signs would change in response to treatment in a clinical intervention trial. After evaluation of the COIs against key COI selection, 14 global impression items were drafted. The global impression items were limited to one question deemed most representative of all concepts from each domain's most highly endorsed subdomains (Table 2).

The general structure of the global impression items is shown in Table 3, using anxiety as an example of a PGI item. Each global impression item has five response options to assess severity of symptoms/signs and seven response options to assess the change in symptoms/signs. The OGI and CGI items are similar to the PGI items, asking how the person or patient with PKU was during the past 7 days to evaluate the severity of symptoms, attempting to understand if observers and clinicians receive patient report about these symptoms. Additionally, the OGI and CGI items evaluate how these symptoms would change compared to before the beginning of a clinical intervention trial. The concepts, response options, and recall period assessed are consistent across PGI/OGI/CGI items, ensuring symmetry and allowing for cross-responder analyses.

Table 3

Structure of the patient global impression (PGI) item evaluating symptom/sign severity and change in symptoms/signs using anxiety as an example. Response options are the same for the PGI/OGI/CGI items.

PGI-severity item structure	PGI-change item structure
Rate how anxious you were during the past 7 days	Rate how anxious you are now compared to before the study began
Response options severity items	Response options change items
• No symptoms • Mild • Moderate • Severe • Very severe	• Very much improved • Moderately improved • Slightly improved • No change • Slightly worse • Moderately worse • Very much worse

Content validity of the clinical outcome assessments and global impression items for patients, observers, and clinicians

Based on the outcomes of the cognitive interview study, patients and observers found the instructions, recall period, and response options of the global impression items clear, relevant, and easy to understand. However, there were some concerns about the lengthy recall period for the PGI-Change items, considering greater than a three-month recall as challenging for accurate recall of symptoms prior to treatment. Additionally, some patients and observers would opt for a five-point improvement scale, removing either the slightly or moderately response due to similarity of interpretation of the adverbs. Both patients and observers generally considered a moderately improved or very much improved response to be meaningful. After review of the cognitive interview results and triangulating the feedback from patients, observers and clinical experts, five of the PGI items were considered to be subject to lack of insight while two of the OGI items were found to be solely reliant on patient-report (Table 4). Due to the inability to continuously observe patients, clinical experts were uncertain if they could complete the CGI-Severity or CGI-Change items, which did not support their use by clinicians managing adults with PKU.

Table 4

Summary of the reliability of patient (PGI) and observer global impression (OGI) items.

Global Impression Items	Accuracy of patient-report	Accuracy of observer-report
Item 1. Short-tempered	Subject to lack of insight	Observable
Item 2. Easily irritable	Subject to lack of insight	Observable
Item 3. Anxious	Self-report acceptable	Observable
Item 4. Easily frustrated	Subject to lack of insight	May be observable
Item 5. Mood swings	Subject to lack of insight	Observable
Item 6. Difficulty paying attention	Self-report acceptable	May be observable
Item 7. Difficulty concentrating	Self-report acceptable	Reliant solely on patient-report
Item 8. Easily distracted	Self-report acceptable	May be observable
Item 9. Easily forgetful	Subject to lack of insight	May be observable
Item 10. Low or lack of energy	Self-report acceptable	Observable
Item 11. Easily tired	Self-report acceptable	Observable
Item 12. Sleep problems	Self-report acceptable	May be observable
Item 13. Headaches or migraines	Self-report acceptable	Reliant solely on patient-report
Item 14. Level of discomfort in social situations	Self-report acceptable	May be observable

Regarding the COAs, the Quality of Life Outcomes in Neurological Disorders (Neuro-QoL) and Patient Reported Outcome Measurement Information System (PROMIS) measures all had support for content validity for use in adults with PKU. Both PRO measures were reported to have clear instructions and their recall periods, response scales, and item concepts were well-understood and relevant to adults with PKU. Even though many of the Neuro-QoL and PROMIS items were relevant to PKU patients, there were items in each of these measures that some of the patients did not deem important and relevant to their PKU experiences, for example “controlling my behavior” and “my worries overwhelmed me”. For the observers of adults with PKU, the CAARS-O:L was not considered a valid measure due to imprecise recall period, confusing response options, and concepts not relevant to adults with PKU. On the other hand, the content validity of the ADHD-RS-IV inattention subscale was confirmed by clinicians treating adults with PKU. However, there was no content validity of the ADHD RS-IV items pertaining to hyperactivity and impulsivity as these items were found to be more relevant for children and adolescents than for adults while other items were not deemed relevant in PKU. Overall, clinical experts reported potential problems with the reliance on patient-report (although this is an instrument requiring clinical judgment based on patient report or observation) and the lack of clarity as to how clinicians determine severity of behaviors, requiring proper training to standardize clinician scoring of patient-reported symptoms.

Fig. 3 summarizes the selected PRO measures and PGI items that established content validation from this study showing how these interrelate with the different domains and subdomains.

Fig. 3

Summary of the results from the concept-elicitation and cognitive interview study, showing the patient-reported outcome (PRO) measures and patient global impression of severity (PGI-S) and change (PGI-C) items by domain and subdomain. The PGI items corresponding to the numbers in this figure are shown in Table 2.

Due to the CAARS-O:L not being identified as a valid ObsRO measure, assessment of adults with PKU by their observers will solely rely on the OGI-Severity and OGI-Change items listed in Table 4. For clinicians, the ADHD-RS-IV inattention subscale was found to be the only valid COA to be used when evaluating adults with PKU.

Discussion

The complex array of symptoms of adults with PKU is not completely understood but there appears to be a higher incidence of depression, anxiety, and impaired social functioning even if adults have been early- and continuously treated [11], [12], [13], [32]. Although the relationship between metabolic control and PKU-associated neuropsychological comorbidities requires further studying, a recent study demonstrated the correlation between blood Phe levels >360 μmol/L and neuropsychological symptoms arguing for lifelong metabolic control [15]. Nevertheless, these mental health problems may not only be attributed to elevated blood Phe levels but also to the burden of diet and stress related to living with a chronic disease, impairing the QoL of adults living with PKU [5], [9], [18], [19], [33]. As the burden of diet among other PKU-specific QoL domains are not considered in generic HRQoL questionnaires, a PKU-specific QoL questionnaire (PKU-QOL) was previously developed to understand the impacts of PKU and its treatment on the emotional, social, and physical aspects of patients with PKU [34]. Since its publication in 2015, three studies have evaluated the validity of the PKU-QOL in adults with PKU [17], [35], [36]. Although metabolic control was found to be negatively associated with the patients' HRQoL, results obtained with the generic and PKU-specific measures were generally comparable [17], [35]. Furthermore, the PKU-QOL mainly focused on the burden of diet with relatively minimal emphasis on the neuropsychological symptoms.

Currently, there is no consensus on the standardized battery of measurements to evaluate symptoms, functional deficits, and HRQoL impairments of adults with PKU. Therefore, this study was designed to identify potential tools to measure outcomes in clinical trials that are relevant and meaningful to adults with PKU. During the first part of this study, PKU-related COIs were identified through internationally conducted in-depth interviews and symptom surveys. By including adults with PKU across the range of blood Phe control, a varied representation of the adult PKU population was ensured. However, the small number of subjects in each category did not allow specific interpretation of the COIs between blood Phe groups. In addition, blood Phe categories were based on a single blood Phe measurement not considering fluctuations in blood Phe over time. Therefore, COIs were identified across blood Phe categories. The COIs endorsed by ≥30% of patient, observer, and clinical expert groups were mapped to existing COAs. Through this mapping exercise, a total of eight COAs relevant to adults with PKU were selected as potentially fit-for-purpose for patients, observers, and clinicians. Because the selected COAs did not completely match the key COIs, PKU-specific global impression items were developed as well. Based on the most highly endorsed COIs, PKU-specific global impression items were compiled to assess the severity of PKU signs/symptoms and determine how these signs/symptoms would change in clinical intervention trials. Generally, global impression items are well-validated, easily applied, brief, quantifiable, and considered practical measurement tools [37]. Because of the PKU-specificity, the global impression items could facilitate the evaluation of novel treatments for lowering blood Phe levels over time. To minimize burden and maximize utility, each PKU-specific global impression item was limited to one question covering a total of 14 COIs that were identified to be relevant for adults with PKU and potentially sensitive to changes in blood Phe. As the global impression items will likely not be sensitive to immediate changes in blood Phe, the recall period was set at 7 days for all severity items. This recall period will allow regular for follow-up without overburdening the patient, observer, or clinician.

An important strength of this study is the involvement of adults with PKU, observers of the affected participants, and clinical experts in the identification of the key COIs. In accordance with the FDA Guidance for Industry (2009), the PRO measures were based on qualitative concept-elicitation interviews to reflect the experiences of adults with PKU [21]. In this study, the input from clinical experts also was considered when developing the PRO measures. PKU clinical experts can offer a unique perspective on the patient disease experience, understanding the genetic and biomedical aspects of the disease as well as reflecting on a diverse sample of their adult patients with PKU. Additionally, observers of patients were included to minimize the risk of underestimating problematic symptoms that may not be perceived by the adult with PKU. This lack of self-awareness may be especially relevant for adults with PKU who are having poorly controlled blood Phe levels, impairing self-rating of symptoms due to neuropsychological deficits. The lack of self-awareness was also apparent in this study with differences between patient, observer, and clinical expert survey results, demonstrating that patients may lack insight on the mood domain (mood swings, short-temper, frustration, irritability) and cognitive domain (forgetfulness).

The second part of this study consisted of cognitive interviews with adults with PKU, observers, and clinical experts to validate the content of the existing COAs and newly developed global impression items that were identified by the concept-elicitation interview study. Although 11 patients, 5 observers, and all clinical experts participated in both the concept-elicitation and cognitive interview study, it is unlikely that the participants could remember with any detail what was discussed during the concept-elicitation interview study given that no results from the first part of the study were shared and there was a significant time lag between the studies. Generally, the instructions, recall period, response options and item concepts of the PGI and OGI items were understood and considered relevant for adults with PKU and their observers. However, the CGI items were not deemed valid because the frequency of clinician visits with patients with PKU was below what would have been required. Although the PRO measures (i.e., Neuro-QoL and PROMIS) all had support for content validity for use with adults with PKU, the ObsRO measure (i.e., CAARS-O:L) was not considered appropriate to be used by observers of adults with PKU. For clinicians, only the ADHD-RS-IV inattention items were deemed valid as ClinRO measures, on the condition that clinicians would be trained on how to administer the tool.

As adults with PKU often become lost-to-follow-up, most observational or prospective studies are including patients with relatively good metabolic control who are generally more engaged and likely to participate in clinical studies [6], [33], [38], [39]. Accordingly, the patient sample interviewed in this study was biased by including patients who were regularly seen at clinic. Therefore, the data collected in this study likely does not reflect the entire range of symptoms experienced by the adult PKU population at large and particularly those patients who are non-adherent with attending appointments and obtaining blood Phe testing. Nevertheless, patients with good metabolic control and more regular follow-up may become more prevalent in the future due to recent treatment advances, such as Palynziq, allowing the majority of adults with PKU to achieve blood Phe levels below 600 μmol/L [40]. Despite these treatment improvements, it will be important to develop inclusion and exclusion criteria such that trial participants have a level of symptom severity to be able to assess and validate the multi-item scales of the identified COAs and global impression items in clinical trials preventing minimal floor effects (i.e., little or no symptoms). Additionally, female and male patients, participating in the cognitive interview study, were not equally divided by blood Phe category. In the cognitive interview study, women made up 100% of the <600 μmol/L blood Phe group, 91% of the 600–1200 μmol/L blood Phe group, but only 17% of the >1200 μmol/L blood Phe group, suggesting higher treatment adherence among female patients, possibly related to the known risk of maternal PKU. However, the predominance of females in the <600 μmol/L and 600–1200 μmol/L blood Phe group was not that apparent in the cognitive interview study. Another limitation inherent to the study design was the subjective and self-reported collection of data through one-on-one telephone interviews that lasted 90 min. Nevertheless, interviews were conducted in the participants' native language and the interviewers were experienced in qualitative interviewing methodology and trained on the study interview guide. To avoid bias in questioning, interviewers were blinded to the blood Phe category of the patients. Additionally, the 60-item PKU symptom survey was specifically designed for this study, warranting further research if one were to consider using it outside of this study.

In conclusion, concept-elicitation interview responses and symptom survey data from adults with PKU, observers of these patients, and clinical experts were analyzed across a wide range of blood Phe categories to assess the most common and important symptoms, signs, and behaviors of adult patients. The most commonly endorsed COIs were mapped to existing COAs and used to develop global impression items for patients, observers, and clinicians, allowing cross-responder analyses. The content validity of the selected COAs and PKU-specific global impression items was evaluated by cognitive interviews. Based on these outcomes, the list of existing COAs was refined, excluding the ObsRO measure (i.e., CAARS-OL:L) and shortening the ClinRO measure by only including the ADHD-RS-IV inattention subscale. Additionally, the reliability of the PGI/OGI/CGI items was found to support use of the PGI and OGI items. The COAs and global impression items identified and validated through this study are believed to comprehensively address concepts that are important, relevant, and meaningful to adults with PKU. The relevance of the identified concepts may be further increased by determining if these are more significantly affected in adults with PKU as compared with a control group. In future clinical trials, these COAs and individual PGI and OGI items could be selected for use based on the endpoints of interest to evaluate outcomes providing further information on their validity. Additionally, these studies may determine whether some of the COAs could be excluded due to overlap between COAs and global impression items, shortening the final list of assessments that can be used in clinical practice. Next steps may include discussion with regulatory authorities to determine if further content validation of the COAs and PGI/OGI items is needed before implementing these tools as efficacy endpoints in future PKU clinical trials. One of these follow-up studies could aim to prospectively evaluate the identified COAs and PGI/OGI items relative to a variety of clinical measures beyond blood Phe as well as assess the validity of the 7-day recall period before use as endpoints in PKU clinical intervention trials.

Funding

This study and support in the process of manuscript development were funded by BioMarin Pharmaceutical Inc., Novato, CA, USA.

Authorship statement

Barbara K. Burton, Christoph Baerwald, Deborah A. Bilder, Cary O. Harding, Nicola Longo, H. Serap Sivri, Gisela Wilcox, and Janet Thomas participated to the study and data collection. Anne Skalicky performed the data analyses together with Aaron B. Ilan, Elaina Jurecki, David T. Madden, and Kathleen Delaney. All authors contributed to the writing of the manuscript. All authors have reviewed and approved the final manuscript.

Declaration of Competing Interest

All clinical experts were financially compensated by BioMarin for their participation to the study. Additional disclosures: AS is an employee of Evidera. CB has received consulting fees from and and was a speaker for BioMarin, Genzyme, Nutricia, Shire, and Vitaflo. BKB has received personal fees from BioMarin, Homology Medicines, Takeda, Horizon, Denali, JCR Pharma, , and Moderna outside the submitted work. DB has received grants and personal fees from BioMarin outside the submitted work. COH has received personal fees from BioMarin, grants from StrideBio, grants and personal fees from Synlogic, and personal fees from Sanofi-Genzyme outside the submitted work. JT has received support from BioMarin outside of the submitted work. GW has received travel grants from Sanofi-Genzyme, BioMarin, Shire/Takeda, and , received speaker honoraria from Sanofi-Genzyme, BioMarin, Shire/Takeda, and Nutricia, was awarded research grants from the , received advisory board membership with BioMarin and Meta Healthcare, received Medical Advisory Panel membership for the National Society for PKU (NSPKU), and performed consultancies for Dimension Therapeutics/Ultragenyx. KD, ABI, EJ, and DTM are employees of BioMarin Pharmaceutical Inc.