Initial development of a patient-reported outcome measure of experience with cognitive impairment associated with schizophrenia.

Cognitive impairment is a serious, often distressing aspect of schizophrenia that affects patients' day-to-day lives. Although several interview-based instruments exist to assess cognitive functioning, a reliable measure developed based on the experiences of patients facing cognitive difficulties is needed to complement the objective performance-based assessments. The present article describes the initial development of a patient-reported outcome (PRO) measure to assess the subjective experience of cognitive impairment among patients with schizophrenia, the Patient-Reported Experience of Cognitive Impairment in Schizophrenia (PRECIS). The phases of development included the construction of a conceptual model based on the existing knowledge and two sets of qualitative interviews with patients: 1) concept elicitation interviews to ensure face and content validity from the perspective of people with schizophrenia and 2) cognitive debriefing of the initial item pool. Input from experts was elicited throughout the process. The initial conceptual model included seven domains. The results from concept elicitation interviews (n=80) supported these domains but yielded substantive changes to concepts within domains and to terminology. Based on these results, an initial pool of 53 items was developed to reflect the most common descriptions and languages used by the study participants. Cognitive debriefing interviews (n=22) resulted in the removal of 18 items and modification of 22 other items. The remaining 35 items represented 23 concepts within six domains plus two items assessing bother. The draft PRO measure is currently undergoing psychometric testing as a precursor to broad-based clinical and research use.

Introduction

Cognitive impairment is a serious, often distressing aspect of schizophrenia. Among the array of symptoms of schizophrenia, cognitive difficulties contribute significantly to the burden of the disorder because they interfere with the ability to manage day-to-day tasks. Cognitive impairment has been shown to be the strongest predictor of functional impairment in schizophrenia, typically predicting more accurately than psychiatric symptoms: a patient’s poorer response to psychosocial interventions, employment status, and social functioning.1

Currently, no effective pharmacological treatments are available for cognitive impairment in schizophrenia.2 Traditional antipsychotic medications have no beneficial effect and may worsen cognitive functioning due to sedation and other adverse effects. The second-generation antipsychotic medications also do not benefit cognitive functioning, and the high anticholinergic effects in several of these medications can significantly impair attention and memory functioning3 and reduce response to cognitive remediation.4

Several instruments can be used to assess the components of functional capacity or cognitive performance in schizophrenia. For example, the University of California Performance Skills Assessment (UPSA)5 and Specific Levels of Functioning (SLOF) scale6,7 are measures of functional capacity; the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS),8 Cambridge Automated Neuropsychological Test Battery (CANTAB),9 Brief Assessment of Cognition in Schizophrenia (BACS),10 Repeatable Battery for the Assessment of Neuropsychological Status (RBANS),11,12 and CogState13 batteries can be used to assess cognitive performance. However, none of these instruments were developed based on patient reports of their subjective experiences of and reactions to cognitive impairment.14

Although several measures of the subjective experience of cognitive impairment in schizophrenia are available, all of them have shortcomings that preclude their use as patient-reported outcome (PRO) measures. For example, the Schizophrenia Cognition Rating Scale (SCoRS)15 is an interview-based assessment of cognitive functioning that is meant to be used with patients and informants. However, it is not truly a patient-reported measure because the items used in this assessment were either modifications of items from dementia rating scales or developed by researchers based on data on neuropsychological impairment in schizophrenia. There was no patient input into the development of this scale, and none of the SCoRS items included, or were based on, patients’ reports of their own subjective experiences of cognitive impairment. Similarly, the Self-Assessment Scale of Cognitive Complaints in Schizophrenia (SASCCS) was constructed by experts without patients’ input.16 The Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS)17 is a brief questionnaire, but its format does not allow for verbal input from the patient; in addition, it focuses primarily on memory and concentration, and internal consistency of subscale scores is not strong. The Subjective Cognitive Impairment Scale (SCIS)18 was developed with input from patients; however, it has been validated only in patients with first-episode psychosis, and an English version of this scale is not available.

Importantly, the subjective experience of cognitive difficulties may not be correlated directly with the performance on neuropsychological tests.18 Moreover, performance in real-world settings may be uncorrelated with scores on laboratory measures of neuropsychological functioning, leading to the distinction in the brain injury literature between impairment (reflecting abnormally low test scores) and disability (reflecting difficulty in performing everyday tasks).19 This dissociation between impairment and disability can also be observed in patients with schizophrenia.14 Because cognitive impairment associated with schizophrenia is likely to affect a patient’s quality of life, distress, motivation for treatment, and perhaps other outcomes, a mechanism is needed to capture the patient’s experience of cognitive difficulties in a sensitive and reliable way. Providing a truly patient-reported, English-language instrument was the motivation for the development of the PRO measure that is described in this article.

As recommended by the US Food and Drug Administration (FDA) guidance for PRO measures,20,21 the initial development of the new PRO measure incorporated the existing knowledge, expert input, and patients’ perspectives. Qualitative research methods were used to extract the most common and relevant themes from patient narratives regarding their own experiences of cognitive disability, and these qualitative data were used in creating the final item pool. The resulting draft PRO measure is currently undergoing psychometric testing in a clinical trial as a precursor to broad-based clinical and research use.

Methods

Design

The PRO instrument was developed from multiple sources of information. First, a conceptual model was constructed based on the existing knowledge. Next, two sets of semi-structured qualitative interviews were conducted to elicit the perspectives of people living with schizophrenia: 1) concept elicitation interviews to understand patients’ experiences, leading to the development of draft items and 2) cognitive debriefing interviews, resulting in item revision and reduction. Input from experts in the field was elicited throughout the study.

Conceptual model development

In order to develop the conceptual model, existing literature was reviewed, and expert input regarding neurocognition and schizophrenia was incorporated. A comprehensive literature search in PubMed using Medical Subject Heading terms related to schizophrenia, cognitive impairment, and psychometric testing yielded 121 published articles. Based on a systematic review of the articles, unpublished results from a qualitative study of people with schizophrenia and their family members, and expert input, an initial conceptual model of seven domains contributing to the subjective experience of cognitive impairment associated with schizophrenia was developed (Figure S1).

Concept elicitation interviews

Based on the initial conceptual model, a semi-structured interview guide was developed with open-ended questions to allow for unanticipated concepts to emerge. Discussion topics included the patient’s experience with attention, memory, executive functioning (eg, planning, organization, and flexibility); communication and social cognition; “flat” cognition (ie, lacking sharpness of thought); metacognitive abilities (eg, self-awareness, distinguishing self from others, and application of information); impaired visual and auditory perception; and an “other” category to elicit unanticipated experiences associated with cognitive difficulties. Aligned with the iterative process of qualitative methods, one additional concept (“unmooring of thought from context and one’s surroundings”) was added based on the unpublished findings from a doctoral dissertation completed during the study.22

During the interviews, special attention was paid to the specific terminology that the respondents used to describe their experiences, respondents’ reports of the frequency of these experiences, conditions under which the experiences typically occurred, and the effect of cognitive difficulties on their day-to-day lives, including emotional reactions. The time duration of interviews ranged from 45 minutes to 2 hours. After each interview, the interviewer completed a field note about the patient’s level of cognitive insight and engagement in the interview to assist with an accurate analysis.

Due to varying levels of cognitive insight among patients and the abstract nature of some of the domains, four probing strategies were used. The interviews were begun by asking patients directly about the concepts in an open-ended manner. As required, particularly for abstract topics, examples from either the interview guide or the patient’s life were used. As a third strategy, scenarios related to particular domains were used, and the patients were asked about how they would respond. If these three strategies were not yielding meaningful responses, we occasionally presented a prescripted scenario, provided two options for how someone could respond, and asked the participant to tell us which response was closest to his/her experience; because this strategy was less open-ended than the others, it was used infrequently.

Item generation

Based on the results from the concept elicitation interviews, initial items were generated to reflect the most common descriptions and language participants used to describe their experiences with cognitive difficulties and the most common conditions under which these experiences occurred. For concepts with variation in common respondent descriptions, multiple items were developed per concept. A third-grade reading level was targeted throughout to accommodate a variety of literacy levels.

Cognitive debriefing interviews

The objectives of the cognitive debriefing phase were to evaluate whether the draft items reflected the patients’ experiences (ie, face validity); to elicit suggestions for other items that could be included to better represent their experiences (ie, content validity); and to assess comprehension of items, instructions, and response options. The time duration of cognitive debriefing interviews ranged from 52 minutes to 2 hours and 16 minutes. Similar to the concept elicitation phase, the interviewers completed a field note after each interview to assist with an accurate analysis.

The interviewer began by asking the participants to complete the draft PRO measure in its entirety. The participants were then asked an open-ended question about their overall impressions of the questionnaire. A series of general questions followed regarding the clarity of the instructions and response options as well as the participants’ perceptions on the length of the questionnaire. Then, the participants were asked several questions about each item in the questionnaire: what they understood it to mean, the ease of answering the item, whether their answers would have changed for a different time frame, how closely the item described their actual experiences, and, for those who indicated that an item did not apply, whether the time frame of the item mattered for this determination. In addition, they were also asked about missing topics related to their experiences with cognition and the degree of bother associated with the categories included in the questionnaire.

Setting and participants

The interviews were conducted in person by staff members at clinical sites serving people with schizophrenia in the Northeastern United States. Four sites conducted the concept elicitation interviews between November 2013 and April 2014, and three of these sites also conducted the cognitive debriefing interviews between July and August 2014. Clinical staff members were chosen to conduct interviews due to their experience in interacting effectively with people with schizophrenia.

Training of the interviewers included practice with open-ended interviewing techniques and nonleading probing, review of the conceptual model, conduct of mock interviews, and practice with multiple strategies for engaging patients in discussion about cognitive difficulties. Throughout the study, the interviewers across all the study sites participated in quality assurance sessions to receive feedback and facilitate consistency in interviewing styles.

People with schizophrenia were recruited to achieve a sample with the demographic and clinical characteristics typical of patients receiving treatment for schizophrenia. Clinical sites recruited patients from clinic databases and community events. Patients participating in a clinical trial within 30 days prior to screening were not eligible. Potential participants were screened to achieve a sample including both older and younger people (with at least 10% aged 18–25 years and at least 10% aged 40–55 years), both genders (with at least 30% women), and both white and nonwhite racial groups (with at least 25% nonwhite). Patients of Hispanic ethnicity were included in either racial group.

Eligible patients were clinically stable for at least 8 weeks with mild or moderate disease severity, defined by Positive and Negative Syndrome Scale item scores <5 for hallucinations, delusions, and conceptual disorganization as well as scale scores <21 for the negative syndrome total score. Patients with mild or moderate comorbid depression (assessed by using the Calgary Depression Scale) were considered for inclusion, although efforts were made to limit the extent of comorbid psychiatric illnesses, particularly depression, in the study sample. Eligible patients were maintained on stable doses of antipsychotic and concomitant psychotropic medications for at least 6 weeks and on a current dose for at least 2 weeks. This study was approved by the New England Research Institutes Institutional Review Board, and written informed consent was obtained from all of the study participants.

Analytic approach

All the interviews were audio-recorded and transcribed verbatim. ATLAS.ti qualitative analysis software (Scientific Software Development GmbH, Berlin, Germany) was used to facilitate data management and analysis.

For both rounds of interviews, two analysts used consensus-based coding. In order to develop codebooks, the analysts independently conducted line-by-line open coding23 on an initial set of transcripts, met to determine an initial code list and definitions, and entered these into ATLAS. ti to facilitate consistency. In order to verify the code lists, they independently coded additional sets of transcripts (two sets for concept elicitation interviews and one for cognitive debriefing interviews) and met to compare coding and reach a consensus about additional codes and modified definitions.

Coding also followed a similar approach for both rounds of interviews. For the concept elicitation interviews, the analysts independently coded three sets of five transcripts and met after each to resolve discrepancies, at which point inter-rater reliability for the most recently coded transcript exceeded 90%. Consensus coding continued with one analyst applying the codebook to subsequent sets of transcripts and the second analyst reviewing the full coding; again, the analysts reviewed discrepancies. This process continued for three additional sets of five transcripts, at which point inter-rater reliability for the most recently coded set was 89%. For the remaining transcripts, one analyst applied the code list and reviewed the areas that were unclear or may have required a new code with the second analyst.

During concept elicitation, saturation was assessed by using sets of five transcripts. Saturation of concepts24 was considered achieved when no new concepts arose within the previous two sets of transcripts (ie, 10 transcripts). Throughout the coding process, the analysts wrote memos to capture emerging connections between categories. Thorough analysis of quotations identified the most common ways in which the participants described their experiences with cognitive difficulties and conditions under which the difficulties occurred. Results from this process formed the basis of the initial PRO item pool.

For the more straightforward cognitive debriefing interviews, coding proceeded with one analyst applying the code-book, the second analyst reviewing the full coding, and both the analysts discussing to reach a consensus. The analysis entailed calculating frequencies of responses and categorizing quotations to assess face validity and comprehension of each item and overall content validity. Particular attention was paid to identifying ways to revise items to more closely reflect the participants’ experiences.

Throughout the analysis and interpretation of both rounds of interviews, the analysts implemented procedures to follow Green and Thorogood’s general principles for enhancing rigor and credibility: transparency of method, maximization of validity (including attention to deviant cases), maximization of reliability (including frequency counts of themes), constant comparison within the data set and within a case, and a reflexive approach to analysis.25

Item reduction

In order to begin the iterative process of item reduction, response thresholds to indicate adequate comprehension and ease of response were preselected. For categories expected to apply to all people to some degree (ie, difficulties with memory, communication, control, understanding others, planning, handling problems, and attention), items with <90% endorsement of comprehension were omitted from further consideration. For categories in which some people may not experience difficulties (ie, sharp thinking and remaining connected with one’s surroundings), items with <80% endorsement as understandable were omitted. In addition, items with <90% of respondents reporting that the item was “easy” or “very easy” to answer were also omitted. Items that met or exceeded the response thresholds received further discussion to address any difficulties in interpretation that the respondents reported; if an adequate revision was identified, the item was modified and retained.

Results

Concept elicitation

In total, 80 people with schizophrenia participated in the concept elicitation interviews (Table 1). This relatively large sample for a qualitative study was recruited to ensure that saturation of relevant concepts was achieved,24 particularly given anticipated variations in subjective experiences of cognition related to an illness with heterogeneous symptoms and potential communication difficulties. Overall, the results supported the domains of the original conceptual model but yielded substantive changes in the concepts within domains and the terminology used in order to more closely reflect the respondents’ experiences (Figure 1).

Table 1

Participant characteristics

Characteristics	Concept elicitation (n=80)	Cognitive debriefing (n=22)
Gender
Women	25 (31%)	10 (45%)
Race and ethnicity
Minority	30 (38%)	12 (55%)
Age (years)
18–25	10 (13%)	3 (14%)
26–39	18 (23%)	7 (32%)
40–55	52 (65%)	12 (55%)

Figure 1

Revised conceptual model of CIAS based on concept elicitation interviews.

Note: Two general categories (“Memory-general” and “Sharpness of Thought” – unspecified) are not represented as distinct concepts.

Abbreviation: CIAS, cognitive impairment associated with schizophrenia.

The initial item pool included the full range of reported difficulties with cognition that demonstrated three criteria: 1) emerged with some consistency, indicating that it is a common experience for people with schizophrenia; 2) addressed difficulties that respondents were aware of (a prerequisite for accurate reporting by patients) given varying levels of insights into cognitive impairment among people with schizophrenia; and 3) were central to the overall concept of interest. Based on the analysis of interviews and these criteria, one domain and six concepts were omitted from the development of items.

The domain Intermittent Impaired Perception (including perceptual difficulties with visual, auditory, olfactory, time, and body) was omitted owing to the overall low frequency of participants reporting difficulties, as well as concerns by some members of the development team that this domain may be perceived by patients as overlapping with positive symptoms of schizophrenia and did not fit conceptually with the other domains. The concept of “planned movement/motor skills” (originally in the domain of Impaired Perception and later moved to Executive Functioning) was also omitted owing to lack of centrality to the overall concept of interest.

The concepts of “intrusive thoughts” (domain: Attention), “unspecified descriptions of lack of sharpness of thought” (domain: Sharpness of Thought), and difficulties with “self-awareness/self-reflectivity” (domain: Metacognitive Abilities) were omitted because an insufficient number of respondents reported difficulties in these areas. Within the domain of Executive Functioning, “abstract thinking” was omitted because most respondents with difficulties in this area were not aware of them and, therefore, would not be able to report accurately about them. Within the domain of Metacognitive Abilities, the concept of “mastery/application of information” was omitted because this concept is more appropriately measured through objective testing.

Initial item pool

In total, 26 concepts across six domains were retained for item generation based on the participant descriptions (Table 2). A draft instrument comprising 53 items was developed, including two items to assess the level of bother (refer to “Item Tracking Grid” in Table S1).

Table 2

Initial item pool domains and concepts with illustrative quotations

Domain/concept	Illustrative quotation per concept
Memory
Forgetfulness during conversation	“I, I tend to forget sometimes with where I’m, with what I was talking about and what I was gonna say.” [Respondent 333025]
Acquisition of information	“Like today, I read something; I forget what I read … It’s not a problem. I just forget what I read … I mean, it bothers me because, you know, if, if, people ask questions, about the paper and all that and they read the same paper, no conversation there because I don’t remember what I read….” [Respondent 444001]
Location of items	“I’m notorious for putting my keys down and forget where I put them.” [Respondent 222035]
Agenda/tasks	“But sometimes, like if I’m supposed to do something, somebody reminds me to do something, and I do forget. Like if they tell me, ‘You have to buy this’ or ‘You have to do this within a certain time’, I may forget. I do have a forgetful memory.” [Respondent 111010]
Directions	“I, I did, I try to get to Walmart and I had to ask somebody. I, I remembers Georges, we have to take Georges Road but I took, I took Livingston so I forgot how to get to Walmart.” [Respondent 333004]
Long-term retention	“Like, like another thing is … remembering just events and places; I don’t remember my last birthday.” [Respondent 444013]
General difficulties	R: “It’s hard for me to remember certain things and how to express myself on it.” (I: “Okay. What kind of things?”) R: “I don’t know, it could be anything.” [Respondent 333001]
Attention
Mind wandering	“And I’m afraid of going [to college] because … I can’t focus, my mind just drifts off in so many different places…” [Respondent 333008]
Screening out distractions	“But it’s like if I … hear the voices of other people as I’m trying to take the test, and that really distracts me. Even though it’s supposed to be quiet, I still hear the pencils moving.” [Respondent 333013]
Ability to let go of thoughts	“You know, a lot of stuff gets in my head and all, it’s kind of hard to let go.” [Respondent 111008]
Racing or disorganized thoughts	“I’ll be thinking about one thing and several other things are racing through my mind at the same time.” [Respondent 111011]
Completion of tasks	“My difficulty’s staying focused and concentration, sometimes setting out goals and staying on task. Like trying to do it by myself and staying on task and trying to do it is a little bit difficult.” [Respondent 222027]
Executive functioning
Planning and organization of steps	“I don’t really know how, how to plan things out. … I, I think, I think that would be hard for me to do. I, I don’t think I would be able to do that very well. I mean, I, I, if I had someone else to do that with and let them do all the work [LAUGHTER] that would be the best thing.” [Respondent 111010]
Flexibility	“It’s not a good thing. Like, it’s like your whole life is being thrown off balance … Because you’re used to things happening a certain way and then when something is out of place, it’s not good.” [Respondent 333024]
Seeing the full picture	“Usually I don’t even think about it. I just, I, I do it anyway. So I don’t, I wouldn’t think that further down the line how it would affect me in the future.” [Respondent 222036]
Creativity in problem-solving	“Sometimes I don’t come up with the solutions so it’s kind of frustrating.” [Respondent 333004]
Sharpness of thought
Mentally blocked/blank	“Yeah, because nothing really comes to mind, so you really don’t like know what to say sometimes.” [Respondent 333020]
Fogginess/cloudiness/lack of clarity	“Well, actually, I’m, sometimes my head gets all cloudy and foggy.” [Respondent 111008]
Slow processing	“Sometimes when I get tired things stop … Umm, everything just slow down.” [Respondent 222012]
Communication/social cognition
Missed social cues	“I’m not good at like certain social situations. I’m not good also at like reading cues and getting the hint about things. I mean unless someone says it.” [Respondent 222031]
Self-expression: meaning	“I have a hard time explaining myself so somebody can understand what I’m saying.” [Respondent 111002]
Self-expression: speech	“Yeah, it’s sometimes hard to find words, yeah. … Like sometimes you’ll be talking to a guy and then like you, you know the word’s like on the tip of your tongue and you just can’t get it out.” [Respondent 333026]
Difficulty interacting/relating with others	“It’s hard to meet new people too, for them to understand what I’m trying to say or where, understand where I’m coming from.” [Respondent 222017].
Metacognitive abilities
Understanding/distinguishing self from others	“I, I do take things people say very personally…” [Respondent 444009]
Self-control	“No, no, no. It just slips. … Just I don’t want them to know anything at all, but sometimes it comes out even if I don’t want it to.” [Respondent 444009]
Unmooring of thought from context	“… Like when I’m so spent, like … I’m so, so, so, so tired. Yeah, that’s a point of disorientation completely. That’s part of the problem, you know, ‘cuz that’s when you start to feel like, altered is such a different feeling, you know. Your brain isn’t working.” [Respondent 222034]

Response categories were modeled based on the validated Patient-Reported Outcomes Measurement Information System (PROMIS) response options;26 specifically, a 5-point response scale ranging from “not at all” to “very much” was used. Because the concept of interest was the degree of difficulty that patients have with cognition (rather than self-reported functioning), the PROMIS response categories were modified to include the word “hard”; that is, the response categories for most items were presented on a 5-point scale, ranging from “not at all hard” to “very hard.” Items not amenable to this format were constructed with a stem that indicated difficulty (eg, “I had trouble with…”) or to clearly indicate that the experience was unwanted (eg, “I kept thinking about things even when I wanted to let them go”).

Questionnaire instructions were developed to provide respondents with a clear definition of the concept of interest and to distinguish it from the positive symptoms of schizophrenia. The instructions also directed respondents to consider their overall experience with an item, rather than focusing on one particular difficult experience. A recall period of the past week was selected to optimize recall while accounting for the frequency of occurrence of difficulties with cognition associated with schizophrenia (ie, the finding from the concept elicitation interviews that these difficulties do not necessarily occur daily). Expert advisors reviewed the draft instrument to consider whether additional modifications were required based on content validity considerations (refer to “Item Tracking Grid” in Table S1).

Cognitive debriefing

In total, 22 respondents participated in cognitive debriefing interviews (Table 1). As the goal of these interviews was narrower and more straightforward than the goal of the concept elicitation phase, a smaller sample was sufficient to reach saturation.25 The participants addressed their overall impressions of the questionnaire, its length and comprehensiveness, the clarity of the instructions and response categories, and their comprehension of individual items (refer to Table 3 for illustrative quotations).

Table 3

Illustrative quotations for cognitive debriefing results

Topic	Illustrative quotation
Overall impression of questionnaire
Positive	“I thought it was really good. It had very interesting questions. I like how it asked me questions that had to do with my everyday life, and that I could really relate to, and actually answer because it made sense.” [Respondent 333029]
Mixed/neutral	“I think it was good to adhering to what like schizophrenia is in terms of that. … I mean some of it didn’t really apply to me. I felt like … about maybe like 60–70 percent was for people with more serious cases or just more disabled than I am.” [Respondent 222055] “That wasn’t too bad.” [Respondent 222046]
Negative	“It was very frustrating for any person who can’t concentrate. But, I think I’m in remission. I didn’t have problem with it.” [Respondent 222058]
Length of questionnaire
Acceptable	“I mean, I was able to pay attention to this whole survey with no problem. But if it was longer I would have lost my mind.” [Respondent 222044]
Too long	“… I think that they have to expect the person to take at least 40 minutes to fill it out because everybody does differently and they might read into it and have to think about whether or not they fit into those categories.” (I: “… how many questions would you have on it? I think this had about 50.”) R: “Probably about 25, 30.” [Respondent 333033]
Comprehensiveness of items
Missing topics	“Medications. How medications affect cognitive thought, cognitive thinking. You know, what you’re on now, you know, versus what you were on then – past and present.” [Respondent 444024] “… Like ‘How much did you like plan or rely on an alternative like way of reacting to your symptoms?’… or ‘how much did you think like the cause of this was?’ or something like that.” [Respondent 222055]
Clarity of instructions
Clear	“Yeah. I think that as far as the, the thinking experiences, I think the wording was very on point.” [Respondent 222047]
Suggestions	“For me it was [clear]. It might not have been for everybody though. … They might have ran into it and thought maybe it did mean some of the delusions and paranoias. They may have thought that.” [Respondent 333033]
Suggestions to enhance the clarity of response options
Order	“I think ‘quite hard’ and ‘somewhat hard’ is like same difference.… I would say just take one of them out.” [Respondent 222044]
Scale size	“I would use probably one to ten rather than one to five.” [Respondent 444022]
Wording	“In case somebody didn’t really know what they meant by ‘somewhat’, they could have put ‘partly’.” [Respondent 333033]

The majority of participants (13 of 22; 59%) reported a positive overall impression of the questionnaire. The main reasons were that the questions were meaningful and relevant to their experiences and were easy to understand. Eight of 22 participants (36%) reported a neutral or mixed reaction, and one participant (5%) had a more negative response due to the concern that completing the questionnaire would be difficult for people with impaired concentration. The average time for completing the questionnaire was 9.4 minutes (range: 4.6–14.7 minutes). Most participants (19 of 22; 86%) stated that the time required to complete the questionnaire was acceptable, with the remaining (3 of 22; 14%) reporting that it was too long.

Regarding comprehensiveness, most participants (19 of 22; 86%) indicated that there were no missing topics related to their experiences with cognition; the remaining three participants (14%) identified the effects of medication, changes over time, and an ability to explain the impacts of each item as areas where coverage could be more detailed or complete.

As an additional indicator of comprehensiveness, 19 participants rated the completeness of the questionnaire on a 5-point Likert scale from “very complete” to “not at all complete.” Sixteen of these 19 (84%) participants rated the questionnaire coverage as “very complete” or “quite a bit complete.” Of the three participants rating the questionnaire as “somewhat complete” or “a little bit complete,” the explanations provided indicated that some of the questions were “a little hard” to answer, suggesting that their responses reflected their experiences with answering some items rather than an assessment of how well the questionnaire covered their difficulties with thinking.

Overall, a majority of participants (17 of 22; 77%) reported that the instructions were clear and easy to follow. The main reason for lack of clarity involved the instruction to separate difficulties with cognition from delusions, hallucinations, or paranoia, which participants indicated, may be difficult for some. As a result, the instructions were modified to include further explanation and examples to clarify the distinction between cognitive difficulties and positive symptoms.

Among the respondents who addressed the definition of “bother” provided in the instructions (ie, being “annoyed” or “concerned” by their experiences), 95% (20 of 21) of them reported that the definition made sense. When asked about alternative words to explain what being bothered by difficulties with thinking feels like, the only additional term to be endorsed by more than one participant was “uncomfortable,” which was added to the definition.

Response options were considered easy to use by 77% of the participants (17 of 22) commenting on Section 1 (response options: “not at all hard” to “very hard”) and 79% (15 of 19) commenting on Section 2 (response options: “not at all” to “very much”). The other participants suggested minor wording changes (eg, to use the word “difficult” instead of “hard”), preferred a 10-point scale instead of a 5-point scale, or were not clear that the response options were orderable. As a result, instructions were added to indicate that response options ranged “from the lowest to the highest” and to choose the “not at all” option if an item did not apply.

The analysis of responses for individual items resulted in 18 items being omitted (Table 4). The participants demonstrated lack of adequate comprehension of 15 items. These items ranged across nine concepts and five domains and were omitted from further consideration. Three additional items were also omitted: one item was identified by the participants as being redundant (concept: general difficulties with memory), and two items were not well understood by some participants and were redundant with other items that had better comprehension.

Table 4

Number of items by domain, concept, and development phase

Domain and concept	Phase of development
	Initial item pool (n=53)	Cognitive debriefing		Draft PRO measure (n=35)
		Modified (n=22)	Omitted (n=18)
Memory
Forgetfulness during conversation	2	1	–	2
Acquisition of information	2	1	–	2
Location of items	1	1	–	1
Agenda/tasks	2	1	–	2
Directions	1	1	–	1
Long-term retention	1	1	–	1
General difficultiesa	1	–	1	0
Attention
Mind wandering	2	–	1	1
Screening out distractions	1	1	–	1
Ability to let go of thoughts	1	–	–	1
Racing or disorganized thoughts	1	1	–	1
Completion of tasks	1	1	–	1
Executive functioning
Planning and organization of steps	2	1	1	1
Flexibility	2	1	–	2
Seeing the full picture	2	1	1	1
Creativity in problem-solving/thinking ‘outside the box’	2	1	–	2
Sharpness of thought
Mentally blocked/blank	1	1	–	1
Fogginess/cloudiness/lack of clarity	2	–	1	1
Slow processing	4	2	2	2
Communication/social cognition
Missed social cues	1	–	–	1
Self-expression: meaning	1	1	–	1
Self-expression: speech	3	1	2	1
Difficulty interacting/relating with others	5	1	2	3
Metacognitive abilities
Understanding/distinguishing self from others	4	–	4	0
Self-control	4	1	1	3
Unmooring of thought from context	2	–	2	0
Additional itemsb
Bother	2	2	–	2

Notes:

One item for general memory difficulties was included for cognitive debriefing but omitted from the draft PRO measure due to redundancy, as reported by the respondents.

Items related to the level of bother were added to capture the perceived impact of cognitive difficulties related to schizophrenia.

Abbreviation: PRO, patient-reported outcome.

The remaining items exceeded the minimum threshold for comprehension. Minor modifications based on participants’ feedback were made to 22 items to increase the level of understanding (Table 4). Further expert review yielded additional minor clarifications to four items. A total of 35 items were retained for psychometric testing, representing 23 concepts within six domains plus two items about the level of bother (see “Draft PRO” in Table S2).

Discussion

This report describes the development phases of a new PRO measure to assess the subjective experience of cognitive impairment associated with schizophrenia, the Patient-Reported Experience of Cognitive Impairment in Schizophrenia (PRECIS). Existing neurocognitive performance measures assess a patient’s cognitive functioning abilities but have not been developed based on the patients’ experiences and do not capture the extent to which cognitive tasks are hard or bothersome for them. The development of the instrument included the construction of an initial conceptual model based on the existing literature, concept elicitation through in-depth qualitative interviews with patients, item generation based on the most common descriptions and language participants used to describe their experiences, and item reduction resulting from a second set of cognitive debriefing interviews with patients.

The resulting 35-item draft PRO instrument represents 23 concepts from six domains and two items assessing bother. The factor structure and other psychometric properties of the draft PRO measure are being assessed and will be reported in a future article. A final version of the instrument and scoring algorithm will be proposed based on the results of psychometric testing.

Although following FDA guidance for developing PRO measures yielded a study design that elicited patients’ opinions, the study has several limitations. For example, the presence of depression among the participants could have affected their responses during qualitative interviews or the ability of the interviewers to build rapport. Although efforts were made to limit the number of participants with comorbid depression, the frequency of this condition among people with schizophrenia meant that it was not feasible to fully exclude these patients. Similarly, some people with schizophrenia lack insight into cognitive difficulties, making it difficult for them to report on experiences with those difficulties in an interview. As noted previously, trained clinical interviewers used four strategies to adjust to a participant’s level of cognitive insight, and field notes included interviewers’ assessments of each participant’s level of awareness to aid analysts. It is unknown how the participants’ medications may affect cognitive insight, although there is no evidence to suggest that medications decrease awareness, and the stabilizing benefit of medication may increase it. Future studies could stratify the analyses by the presence of comorbid depression and level of cognitive insight in order to further account for these conditions.

Even with these limitations, the cognitive debriefing results suggested that the PRO measure reflects many patients’ experiences. The majority of participants in this phase reported a positive overall impression of the questionnaire because the items were meaningful and relevant to their experiences and were easy to understand. Cognitive impairment associated with schizophrenia typically impacts many areas of patients’ everyday lives, including their education, work lives, and interactions with friends and family. Cognitive difficulties can also affect self-esteem and generate distress among patients. The new PRO measure will provide a patient-based perspective on this clinically important area of schizophrenia. It will complement the existing measures of cognitive functioning by providing a tool to describe patients’ statuses in an open-ended manner and with a language that captures their own experiences.

Given the importance to patients of receiving understandable information about their conditions and treatment choices,27 this new measure could prove valuable in building a therapeutic alliance. Following validation, this PRO measure and the conceptual model on which it is based could serve to define key patient-based endpoints for future clinical trials, along with the development of a new endpoint model incorporating both objective and subjective endpoints for assessing cognitive impairment in schizophrenia.