Refinement of the Long-Term Conditions Questionnaire (LTCQ): patient and expert stakeholder opinion.

PURPOSE: It is a key UK government priority to assess and improve outcomes in people with long-term conditions (LTCs). We are developing a new patient-reported outcome measure, the Long-Term Conditions Questionnaire (LTCQ), for use among people with single or multiple LTCs. This study aimed to refine candidate LTCQ items that had previously been informed through literature reviews, interviews with professional stakeholders, and interviews with people with LTCs.
MATERIALS AND METHODS: Cognitive interviews (n=32) with people living with LTCs and consultations with professional stakeholders (n=13) and public representatives (n=5) were conducted to assess the suitability of 23 candidate items. Items were tested for content and comprehensibility and underwent a translatability assessment.
RESULTS: Four rounds of revisions took place, due to amendments to item structure, improvements to item clarity, item duplication, and recommendations for future translations. Twenty items were confirmed as relevant to living with LTCs and understandable to patients and professionals.
CONCLUSION: This study supports the content validity of the LTCQ items among people with LTCs and professional stakeholders. The final items are suitable to enter the next stage of psychometric refinement.

Introduction

It is a key UK government priority to assess and improve patient-valued outcomes in people with long-term conditions (LTCs).1,2 Over 15 million people in England are living with at least one LTC, and this number is expected to rise.3 It is also increasingly common for people to have more than one LTC, adding to the complexity of people’s needs, the management of these needs, and their interactions with services.4–7 In order to assess and target mechanisms to improve outcomes in people with LTCs, health and social care services require an appropriately developed instrument that is valid and reliable for use among individuals who have one or more LTC. Currently, there are no instruments available that can capture outcomes of importance to those with more than one LTC. To date, the EQ-5D8, an instrument measuring generic health status, has been used in England among people with LTCs, eg, in the GP Survey,9 but it is thought it may not capture the full array of outcomes important to both patients and health care professionals when used among those with one or more LTC.10 In the absence of an appropriate instrument, this paper reports one stage of work to refine and further develop a new patient-reported outcome measure (PROM): the Long-Term Conditions Questionnaire (LTCQ). The LTCQ aims to be suitable for use among people with single or multiple LTCs and physical and/or mental health conditions and to be relevant and meaningful for health and social care providers for monitoring and improving health management and outcomes.

To inform the content of the LTCQ, two stages of development have been carried out to date. In stage 1, professional and lay stakeholders (n=31) were consulted10 on their opinions and views on the need for a PROM for people with LTCs. The results indicated broad support for a single PROM suitable for use across LTCs and across different health and social care services that would incorporate both traditional (eg, physical functioning) and non-traditional domains, such as empowerment and social participation. In stage 2, in-depth qualitative interviews (n=42) were carried out with people with one or more LTCs. Interview transcripts were analyzed along with six transcripts taken from a study on treatment outcomes in schizophrenia.11 The interviews, together with a literature review of relevant concepts, served to incorporate people with LTCs’ views into the development of the new instrument. A conceptual framework with three broad main organizing themes (impact of LTCs, experiences of services and support, and self-care) was developed from the qualitative interviews. Draft questionnaire items were constructed to reflect these organizing themes to form an item pool of 23 candidate items for the new questionnaire. The current paper reports the third stage of the development of the LTCQ through the testing and refinement of 23 draft items in cognitive interviews with people with LTCs, a professional stakeholder consultation, and a translatability assessment.

The importance of incorporating the patients’ perspective from an early stage in PROM development is twofold. Firstly, the instrument will only be useful if it captures patients’ perspectives and priorities accurately. Patient input in the selection and modification of items is therefore an integral part of identifying and measuring the relevant construct/s.12 Secondly, patient involvement helps to identify patients’ language and terminology when discussing their health. This is a simple yet important step in ensuring items are easy to understand among the target population.

In the case of the LTCQ, ensuring the items capture the impact of living with LTCs (ie, content validity) and the terminology used by patients was supported through the stage 2 analysis of the in-depth interviews carried out to inform the items. After drafting items, however, it is important that their content is further supported through evaluating patient understanding of candidate items and checking whether these interpretations are relevant to the construct being measured.13–15 Minor changes to improve the comprehensibility and completeness of the questionnaire item pool can be made using a structured and systematic style of cognitive interviews.12,16

While increasing attention has been given to incorporating the patients’ perspective in the development of PROMs,13,17 less emphasis has been placed on the importance of incorporating health professionals’ or other stakeholders’ (eg, commissioners, policy makers) perspectives in early item development. Including professional stakeholders or end users in the refinement of a new instrument can add a further valuable step to ensuring content is not missed, and most importantly that the resulting instrument is both meaningful and interpretable within the applied setting.10,18 The authors thus endeavored to include both patient and expert opinion in the iterative refinement of the draft LTCQ.

Materials and methods

Several iterative rounds were undertaken in the refinement of the 23 candidate items for the LTCQ, including expert consultation, public involvement, cognitive interviews, and a translatability assessment. The methods for refinement are outlined in the following sections. Figure 1 shows the sequence in which they took place, with some methods described taking place in multiple rounds and/or in parallel. Ethical approval for this research was obtained through the National Research Ethics Service (14/LO/0834) and Research and Development approval was obtained from participating sites.

Figure 1

Sequence of steps taken to assess LTCQ candidate items.

Abbreviations: LTCQ, Long-Term Conditions Questionnaire; DH, Department of Health; NHSE, National Health Service England; PIIG, Public Involvement Implementation Group.

Expert consultation and public involvement

Consultation with a range of stakeholder professionals and public representatives was sought through a variety of methods.

Consultation with professionals

Professional stakeholders (n=31) who had previously participated in stage 1 interviews regarding their experiences of using PROMs10 were contacted via email and asked to comment on the draft LTCQ items. This consultation aimed to evaluate whether items were appropriate and meaningful from a health professional’s perspective, with a particular focus on whether any content was absent from the draft questionnaire items. Thirteen stakeholders responded to our request: 12 via email and one via telephone. Stakeholders’ backgrounds included representation from primary care (n=5), secondary care (n=2), social care (n=1), policy (n=4), and patient-focused voluntary organizations (n=1) in England.

Consultation with Department of Health and National Health Service England

Members of the Department of Health (DH) and National Health Service (NHS) England (n=4) with expertise in LTCs were consulted to ensure the questionnaire would be useful for implementing health policies. Information on the development of the draft questionnaire items was presented at a meeting, which had been arranged to discuss the progress of the questionnaire, with those present providing feedback.

Consultation with patient and public representatives

Thirty public representatives were contacted via email through the research unit’s Public Involvement Implementation Group (PIIG), a group established to oversee and monitor implementation of public involvement in the unit’s research. Five members provided feedback via email in relation to comprehensibility, appropriateness for use in health and social care services, presentation, and length.

Cognitive interviews with people with LTCs

Face and content validity of questionnaire items can be supported through pretesting the draft items for understanding and ease of completion among the relevant population, via cognitive interviews.19 Face validity ensures that the questionnaire superficially makes sense and can be easily understood,20 while content validity ensures that all aspects of the domain being measured are represented by the items.14 In addition to using a conceptual framework derived from earlier patient interviews to inform items, content validity is supported through evaluating patient understanding of draft items and determining whether these interpretations are relevant to the concept being measured.13,14 Items that are not found to be directly related to the concept of interest can then be removed or amended as appropriate. Participants taking part and the procedure to conduct the cognitive interviews are outlined in the following sections.

Participants

Men and women were required to be at least 18 years old and have a confirmed diagnosis of one or more LTCs. Participants were recruited based on the diagnosis of at least one of the following LTCs: cancer, chronic obstructive pulmonary disease, depression, diabetes, inflammatory bowel disease, ischemic heart disease, multiple sclerosis, osteoarthritis, schizophrenia, and stroke. These conditions were chosen to represent a range of physical and mental health conditions affecting key bodily systems, as well as having different symptoms, trajectories, and prognoses. The LTCs were selected by a panel of PROM experts (researchers) and lay advisors knowledgeable in LTCs.

The panel were also signposted to the WHO Global Burden of Disease,21 a primary care study on multimorbidity,22 and the Quality and Outcomes Framework23 to aid their selection process. Participants were required to have used health, social care, and/or voluntary care services for their LTC in the last 12 months, to have the ability to communicate in English, and to be living in the community. If participants did not meet these requirements, or if they were deemed too unwell by their GP, they were excluded from the study. The same sampling strategy was followed as for the previously reported qualitative interviews (Peters et al11 for details).

Procedure

Recruitment

Participants were contacted using two methods: 1) the research team contacted participants who had previously taken part in an in-depth interview (stage 2) to inform the items and who consented to be recontacted for this research and 2) additional participants were recruited from two Thames Valley primary care practices solely for the purpose of taking part in the cognitive interviews.

The sample size intended to reflect the need to identify problems participants may experience when completing the questionnaire, not the frequency of a particular problem in the general population.24 A target sample size of 20–25 participants over a series of interview rounds (five to ten participants per round to allow for testing of revisions) was thus likely to be sufficient.13,24–28

Recruitment materials and consent

Invited participants were provided with a participant information sheet explaining the research. Before taking part, participants were given the opportunity to ask questions about the research and asked to complete and sign a consent form. Participants were given a voucher of £20 to thank them for their participation.

Interviews

Interviews lasted for an average of 79 minutes, were audio-recorded, and notes of participant comments were taken throughout. The “verbal probing” method of cognitive interviewing was used, allowing participants to complete the questionnaire in its entirety, followed by a focused interview. An interviewing guide was developed to probe participants on the clarity of items, to prompt participants to recall their thought processes when interpreting items, and to gain a deeper understanding of the reasoning behind their answer, eg, “How did you arrive at that answer?”28,29 This process also investigated interpretations of the instructions and the response options of the questionnaire.25

Analysis

Interviewers (CP, CH, and LK) summarized participants’ comments regarding aspects of the questionnaire using their field notes and audio recordings. To facilitate a collaborative19 and systematic analysis among the authors, a charting approach was used to organize summaries in Microsoft Excel. Areas for inquiry (such as questionnaire instructions, items, response options, format, content, and missing items) were assigned to the column tabs, while each participant summary was entered successively in rows below. This enabled within-case analysis to assess items within the context of the overall questionnaire and between-case analyses to investigate if interpretations were consistent across participants and condition groups. Problems with specific terminology or response options were particularly highlighted.30 A traffic-light system facilitated discussion of problematic items among the authors. Ease of interpretation by patients was indicated as follows: green indicated no difficulties, amber indicated minor difficulties, and red indicated substantial difficulties. In total, three rounds of cognitive interviews took place with patients. The number of rounds was not predetermined. Interviews were conducted until no new minor or substantial problems were identified.12

Translatability assessment

Ease of item translation into other languages is an important consideration when developing a new instrument.31 LTCQ draft items were assessed for their cultural and linguistic translatability by PharmaQuest Ltd, a company specializing in the translation and cultural adaptation of measures used in health and medical care settings. This process did not serve to translate the existing items into selected languages, but aimed to identify and rectify potential translation difficulties that items might cause in future translations. To do this, a number of steps were carried out. A concept-elaboration document, where instructions and items are defined, was produced by PharmaQuest and discussed with the authors. Once finalized, this document was distributed along with the draft LTCQ items to five “in-country” translators. Translators reviewed each item to assess how easily it could be translated into their native language. Six languages were chosen on the basis of their diversity, as well as their frequency of use in the UK: Arabic (Egypt), French (France), Polish (Poland), Punjabi (India), simplified Chinese (China), and Urdu (Pakistan). Translators referred to the concept-elaboration document to clarify the intended meaning of items and to suggest alternative language when direct translation was not possible. In cases where questionnaire items were altered after the initial translatability assessment, amended items entered a further translatability assessment round, as illustrated in Figure 1.

Results

Four rounds of revisions of the LTCQ item pool took place. Questionnaire items were refined in each round and retested for understanding in subsequent rounds until no further changes were required. Figure 1 shows the sequence in which refinement and testing took place. In round 1, expert and public consultation took place in parallel with cognitive interviews. In round 2, professional consultation took place in parallel with a translatability assessment. In round 3, the refined items were tested again during cognitive interviews with people with LTCs. Final amendments were tested in a final round of cognitive interviews and a translatability assessment to ensure understanding of the amended items. In total, 32 participants took part in the cognitive interviews. Table 1 shows the participant characteristics and time taken to complete the questionnaire.

Table 1

Participant characteristics

Characteristics	Round			Total(n=32)
	1 (n=13)	2 (n=9)	3 (n=10)
Sex
Male	6	4	4	14
Female	7	5	6	18
Age, years
Mean	64.9 (SD 13.3)	62.4 (SD 13.2)	56.9 (SD 18.9)	61.72 (SD 15.1)
Range	36–88	45–80	30–79	30–88
LTC*
COPD	1	1	0	2
Diabetes	6	1	1	8
IBD	1	2	0	3
IHD	2	1	2	5
MS	3	0	0	3
OA	2	4	6	12
Stroke	2	1	2	5
Cancer	1	4	1	5
Depression	0	3	1	4
Time taken to complete LTCQ (minutes)	3–10	1–7	2–9

Note:

Participants may have more than one LTC. Other reported LTCs: hiatus hernia, chronic kidney disease, borderline personality disorder, sciatica, chronic renal failure, skin problems, gout, asthma, epilepsy, hypothyroidism, rheumatoid arthritis, vision problems, recurrent kidney stones, hypertension, fibromyalgia, suspected IBD, transient ischemic attack, atrial fibrillation, mild hearing loss, primary progressive aphasia, chronic pain.

Abbreviations: LTC, long-term condition; COPD, chronic obstructive pulmonary disease; IBD, inflammatory bowel disease; IHD, ischemic heart disease; MS, multiple sclerosis; OA, osteoarthritis; LTCQ, Long-Term Conditions Questionnaire.

Round 1: LTCQ version 1 (23 items)

Consultation with DH and NHSE

The LTCQ was presented to four members in the DH and NHS England. Those present showed broad support for the instrument and its use in health and social care settings. Particular emphasis was placed upon maximizing ease of interpretation for end users, such as policy makers. Alternative response scales to the frequency options (never–always) were also discussed by the group. Emphasis was placed upon finding the optimal recall period in which people with LTCs could respond in a meaningful way; the initial two week period was thought to be too short. Finally, the importance of professional and patient input was encouraged, as it would take patients’ views into account yet enable professionals to get the most useful information from scores.

Cognitive interviews (n=13) and PIIG (n=5)

Results from the first round of cognitive interviews and PIIG feedback were considered in tandem. While most items presented were considered to be relevant and appropriate for use in people with LTCs, a number of modifications were made. Four participants and two PIIG members indicated they would prefer longer than the two week time frame when responding to questions, so that they would have an adequate time to reflect on their health. The recall period was thus increased from two to four weeks.

Following feedback, six questionnaire items (Table 2) were modified to reduce the item length and improve sentence structure while retaining the original meaning. Next, participant and PIIG interpretations for eight of the 23 items indicated that they were slightly ambiguous. Two of these eight items were deleted, as they measured areas already covered by other existing items, and five items were amended to improve clarity. For instance, examples were added to expand on the meaning of “roles and responsibilities”. The sixth remaining ambiguous item asked the respondent about “coping with treatments and services”, and comments indicated that these were considered two separate concepts. Therefore, this item was divided into two items, the first to ask about treatment and the second asking about services. Table 2 shows the changes.

Table 2

Modifications to LTCQ versions 1–4

Version 1
Reason for revision	Instructions and format
Participants experienced difficulties giving a true reflection of health in two week time frame	Two-week recall period changed to four weeks
Cognitive difficulties experienced with frequent changing from positive to negative item wording	Item-order revision to “positive” and “negative” item blocks
Reason for revision	Items selected for revision
Revision to six items to improve sentence structure and reduce item length	Able to live the life that you want toAble to be as physically active as you wanted to beAble to take part in activities that were enjoyable for youMore dependent on others than you wished to beYour health condition(s) made you unhappy about your lifeHappy about your life
Five items deleted due to similarity with other retained items (ie, duplication)	Had the support you needed to live your lifeAble to be as socially involved as you wanted to beHealth condition(s) make(s) you a burden to othersAble to make a valued contribution to othersHappy about your life
Six items revised to remove ambiguity (one item* split into two items)	Had the support you needed to cope wellAs much contact with other people as you would have likedHad the information you neededConfident in taking actionsAble to fulfill your roles and responsibilities*Difficult to cope with the treatments and services
One item removed as item was not directly relevant to concept	Life that gave you dignity and self-worth
Version 2
Reason for revision	Instructions and format
Revision to improve clarity and facilitate future translations	A number of structural changes suggested and expansion on the term “long-term conditions”
Reason for revision	Items selected for revision
Revisions to improve clarity and facilitate future translations	Treatment(s) too demanding to deal withServices you receive too demanding to deal with
Items revised to reduce ambiguity and improve sentence structure	Your home met your needs in relation to your health condition(s)Health condition(s) made you unhappyAble to fulfill your responsibilities (eg, family, work, community)Confident in managing your health condition(s)
Version 3
Reason for revision	Format
Questionnaire order changed to decrease risk of bias	Two items’ order changed to decrease the potential of previous items influencing responses
Reason for revision	Items selected for revision
Terminology	Treatment(s) too much of a burdenServices(s) too much of a burden
Revision to improve clarity (one item* split into two items)	Information you wanted to deal with your health condition(s)Able to be as physically active as you wanted*Safe and secure in your daily life
Additional item to improve content	Bothered by symptoms
Version 4
Reason for revision	Instructions
Ambiguity	Defining period related to not receiving treatment or services
Reason for revision	Items selected for revision
Ambiguity	Range of services you use in relation to your health condition(s) difficult to manage
Consistent item wording	Treatment(s) (eg, medications, therapies) very difficult to cope with

Abbreviation: LTCQ, Long-Term Conditions Questionnaire.

In addition to the two items deleted, three further items were deleted, as interview participants and PIIG members considered them to duplicate other items. In these cases, the item that was interpreted most easily by participants and PIIG members was retained. One further item relating to “dignity” was deleted, as comments indicated it was vague and represented direct experiences of services rather than evaluating the impact of services. Modifications to the draft LTCQ resulted in an 18-item version of the LTCQ.

While satisfied with the response options, some participants had difficulty completing items in the manner intended, due to items frequently changing from positive to negative wording, eg, posing a “positively” worded item asking the respondent about being able to be as socially involved as they would like to be followed by a “negatively” worded item asking the respondent about being worried that they were treated differently due to their LTC. Alternating between positive to negative wording required a shift in cognitive processing by the respondent, as the interpretation behind the response options also changed (ie, responding “strongly agree” in the first example produced a positive score, yet responding “strongly agree” in the second example produced a negative score. The item order was thus amended to start and finish with blocks of positive items, while a block of negatively worded items was placed between them. Participant and PIIG observations also indicated that items that were easy to answer should be placed at the beginning of the questionnaire and then transition into items addressing higher-level concepts, such as “able to live the life that you want”.

Round 2: LTCQ version 2 (18 items)

Consultation with professionals and translatability assessment

Thirteen professional stakeholders from primary care, secondary care, social care, policy and, patient-focused voluntary organizations in England responded to our request for feedback. There was a consensus among respondents that the LTCQ included items of value and importance to people with LTC(s) and covered a range of appropriate social, psychological, and physical issues.

The challenge of developing an instrument for use in LTCs was discussed by a number of stakeholders. For example, one stakeholder (a GP and academic) discussed the difficulty of finding an appropriate recall period to reflect on. Four weeks were considered quite short for patients to have had the opportunity to access different services (eg, a GP might see a patient with high needs every six to eight weeks, while seeing others once a year). The authors retained the four week recall period, as it was considered appropriate by the majority of stakeholders and patients.

Two stakeholders suggested further improvements should be made to the items relating to treatments and services. One (a manager in a social care setting) suggested including examples of what is meant by the terms “treatment” and “services”, while the second (a GP and academic) commented on the sentence structure of the items. The translatability assessment, undertaken in parallel with the stakeholder consultation, further reinforced the stakeholder comments by highlighting the difficulty of translating these items, which included the phrase “too demanding to deal with”. The items were reworded for testing in subsequent cognitive interviews. The translatability assessment proposed minor revisions to four further items to improve clarity and sentence structure. Finally, minor modifications to the introduction and instructions were made to enhance clarity. For example, the introductory text describing what is considered an LTC was simplified.

Round 3: LTCQ version 3 (18 items)

Cognitive interviews (n=9)

Participants confirmed that all items were relevant to living with LTCs and contained no direct duplication. The newly revised items regarding the burden of treatments and services were not well received by participants. Seven participants considered the term “burden” to be too strong in relation to the inconvenience or irritation of treatments, while six participants gave similar feedback for the services item. In addition, three people had not had experiences of services or treatments in the previous four weeks and were unsure of how they should respond. The authors thus amended the wording and added a response option for participants to indicate if they had not used services or treatments in the past four weeks.

Three participants described problems with interpreting the term “feeling safe and secure” in the context of their health. Two of the three participants had a mental health condition and were uncertain whether the item should be answered in the context of their mental or physical health. Following discussions within the research team, the item was split into two items (reflecting inside and outside the home) to minimize ambiguity. This decision was supported through both the professional consultation and the translatability assessment, which queried if this item incorporated safety both inside and outside the home. Minor modifications were made to two further items to improve clarity. Finally, an additional new item relating to symptoms was developed in response to concerns that content validity may be compromised through not capturing the full range of impacts associated with living with LTCs. The number of items thus expanded from 18 to 20.

Round 4: LTCQ version 4 (20 items)

Cognitive interviews (n=10) and translatability assessment of new items

Participants considered all questions to be relevant to living with LTCs, with no duplication. One participant did not see the tick box to indicate that they had not received services in the previous four weeks. Greater emphasis was thus placed on the tick box to alert participants to this option. Following patient feedback, a minor adjustment was made to both the treatment and services items to make them consistent with each other. Improvements were observed on the understanding of the revised safety items, which distinguished between inside and outside the home, and the new item asking about symptoms was understood by all participants. Finally, the four rephrased items and the two additional items developed after the initial translatability assessment were submitted for a translation review. No changes were recommended, resulting in the retention of 20 items for the next planned survey phase.

Representation of dimensions

Following expert and patient refinement, 23 items were reduced to 20 items. All remaining items were considered relevant to living with LTCs and understandable to patients. The three broad categories identified in the stage 2 patient interviews were confirmed and represented by two or more items (Table 3). The theme “dignity” and its respective item were removed from the LTCQ item pool as a consequence of participant feedback.

Table 3

LTCQ candidate items confirmed by patient and expert opinion

Impact of LTCs
LTCQ domain	Item
Achieving personal goals	Able to live your life as you want, despite your health condition(s)
Dependency and being a burden	More dependent on others than you wanted, because of your health condition(s)
Impact of LTC(s)	Bothered by symptoms of your health condition(s)Health condition(s) made you unhappy
Loneliness	Lonely due to your health condition(s)
Physical activity	Physically active as you wanted, despite your health condition(s)
Roles and responsibilities	Able to fulfill responsibilities (eg, at home, at work, in your local community), despite your health condition(s)
Safe environment	Safe at home, despite your health condition(s)Safe outside your home, despite your health condition(s)
Social participation	Able to take part in activities you enjoy, despite your health condition(s)Enough social contact with other people, despite your health condition(s)
Stigma	Worried about being treated differently
Suitability of home	Home is suitable for your needs in relation to your health condition(s)
Experiences of services and support
LTCQ domain	Item
Burden of treatment and services	Services you use in relation to your health conditions difficult to cope withTreatment(s) (eg, medications, therapies) difficult to cope with
Support	Enough support to cope well with your health condition(s)
Self-care
LTCQ domain	Item
Coping	Able to cope with your health condition(s)
Empowerment/sense of control	In control of your daily life, despite your health condition(s)
Information and knowledge	Knew enough about your health condition(s)
Confidence to manage LTC(s)	Confident in managing the day-to-day aspects of your health condition(s)

Abbreviation: LTCQ, Long-Term Conditions Questionnaire.

Discussion

LTCs need to be managed over long periods of time, which can often involve care provision and interactions with a number of organizations and professionals from health and social care, as well as work by those with LTCs to maintain quality of life. This research refined items to inform a new PROM, the LTCQ, to assess the impact of living with single or multiple LTCs. The aim of this stage of the work was to enhance content validity of the LTCQ. The changes made have led to a draft LTCQ that was valid and acceptable, both in terms of contents and comprehension, to potential respondents.

Four revisions of the LTCQ took place, resulting in amendments to the recall period, improvements to item clarity, removal of duplicate items, and revisions to enable easier translation of the LTCQ. Content validity was enhanced through the addition of one item relating to symptoms following participant feedback. Item order was amended to improve the patient experience when completing the questionnaire and to reduce completion errors when changing from positively to negatively worded items. Some items that required more thought, eg, “able to live the life that you want”, were placed toward the end of the questionnaire, so that the respondent had a more gradual introduction to higher-level concepts. The final 20 items reflected the three categories identified as important to people living with LTCs: impact of LTCs, experiences of services and support, and self-care.

The methods used in this study enabled input from both people living with LTCs and professional stakeholders in the refinement of the new PROM. The incorporation of a wide range of people living with both physical and/or mental single or multiple morbidities ensured items are suitable for use across a range of LTCs. The varied sample was achieved through a comprehensive and purposeful search strategy influenced by recent research in the field.22 Fifteen participants who had taken part in interviews for a previous phase of the study were asked to take part in a cognitive interview. This provided the opportunity to check whether the LTCQ items accurately reflected how they had described living with LTCs. In addition to these 15 participants, 17 new participants were recruited, adding the potential for new insights into the impact of living with LTCs, which might have been otherwise missed. The inclusion of a range of stakeholders from different professional backgrounds also served to gain perspectives from end users in a multitude of applied settings, to ensure that the content and application of the instrument was suitable for their varied needs.

Public representatives were given the opportunity to provide input into the instrument’s development. This ensured maximum acceptability to patients and professionals, enhanced content validity, and facilitated the cross-checking of results. Finally, a translatability assessment ensured that items were suitable for future translations of the instrument and helped to simplify the language and grammar of the LTCQ.

Achieving consensus on the optimal form of the instrument was challenging, due to the range of participants and professionals that were involved in the refinement of the LTCQ. For example, defining an appropriate recall period that was short enough for patients to respond accurately to items while also facilitating a clinician gaining a meaningful score for use in decision making meant that a compromise needed to be reached. Being sympathetic to both the responder and the end users’ needs, however, has arguably resulted in a more useful instrument for both parties. Although there has been an increase in good-quality PROM-development studies that incorporate and report cognitive interviewing and expert opinion in the initial drafting of questionnaire items,32,33 the addition of expert opinion at the item refinement stage has been lacking. As was the case in this study, additional items are often added while others are deleted at the item refinement stages. Therefore, the authors of this paper recommend additional professional input at these stages of questionnaire development, in particular for instruments that are intended for use in clinical practice.

As recommended by other authors in the field,31 the verbal probing method was used to evaluate the LTCQ’s content and comprehensibility. Retrospective probing was particularly successful, as the draft questionnaire was relatively short, allowing participants to recall their thought processes. An alternative method, the “think aloud” method, was not used, as it can impose greater cognitive demands on participants and upset the questionnaire flow. Retrospective probing, as opposed to concurrent probing, was employed to minimize interruption to the response process, and allowed researchers to observe how participants initially engaged with the questionnaire.16,32

In addition to reporting the improvements made to the LTCQ items, this paper sheds light on the process of analysis during the cognitive interview process, which is not well documented in the literature. A charting approach facilitated systematic analysis and enabled all members to have informed input to modifications of the questionnaire. This approach was preferred over alternative methods of analysis, such as successive aggregation, where individual researchers independently analyze their own interviews and then bring interpretations together.19

Some limitations of this study must be acknowledged. While a comprehensive randomized search strategy was used to sample patients through primary care, patients who responded and took part in a cognitive interview were self-selecting. Furthermore, those with mental LTCs were less well represented in comparison to those with physical LTCs, as were those with experience of social care use. This was a similar experience to our previous study to inform the initial questionnaire items, which included data supplementation via a secondary analysis of previously conducted interviews with people who had schizophrenia, since none was recruited directly into this study. While this research may have benefited from further input by people with mental health conditions, the authors were reassured by the involvement of people with mental health LTCs in the previous stage of the study contributing to content, some representation in the cognitive interviews, and input from stakeholders who work in the mental health sector. The next stage of research will also test the questionnaire further in this population, as well as in people with physical health conditions.

Conclusion

This paper reports one stage in the development and refinement of a new instrument, the LTCQ, which is intended for use in people living with LTCs and suitable for use in health and social care settings. To date, this instrument has undergone several stages of research, including interviews with stakeholders, literature reviews, and in-depth interviews of people with LTCs, to support its conceptual underpinning, followed by patient and expert opinion and a translatability assessment to refine the candidate items. Our analysis suggests that 20 LTCQ items are suitable for inclusion in the next stage of psychometric refinement of the questionnaire.