Patient involvement in the development of patient-reported outcome measures: a scoping review.

BACKGROUND: Patient-reported outcome measures (PROMs) measure patients' perspectives on health outcomes and are increasingly used in health care. To capture the patient's perspective, it is essential that patients are involved in PROM development
OBJECTIVE: This article reviews in what ways and to what extent patients are involved in PROM development and whether patient involvement has increased over time. SEARCH STRATEGY: Literature was searched in PubMed, EMBASE, MEDLINE and the Cochrane Methodology Register. INCLUSION CRITERIA: Studies were included if they described a new PROM development. DATA EXTRACTION: Basic information and information regarding patient involvement in development phases was recorded. MAIN
RESULTS: A total of 189 studies, describing the development of 193 PROMs, were included. Most PROMs were meant for chronic disease patients (n = 59) and measured quality of life (n = 28). In 25.9% of the PROM development studies, no patients were involved. Patients were mostly involved during item development (58.5%), closely followed by testing for comprehensibility (50.8%), while patient involvement in determining which outcome to measure was minimal (10.9%). Some patient involvement took place in the development of most PROMs, but in only 6.7% patients were involved in all aspects of the development. Patient involvement did not increase with time.
CONCLUSIONS: Although patient involvement in PROM development is essential to develop valid patient-centred PROMs, patients are not always involved. When patients are involved, their level of involvement varies considerably. These variations suggest that further attention to building and/or disseminating consensus on requirements for patient involvement in PROM development is necessary.

Introduction

Patient‐reported outcome measures (PROMs) assess health outcomes from the patients’ perspective.1, 2 PROMs were originally developed for the use in clinical research as a way to measure treatment effectiveness. However, PROMs are now increasingly used in clinical practice to monitor and to improve the care for individual patients and in health policy and management, for example in the English National Health Service (NHS), to measure the performance of health‐care providers2, 3 or by the American Centres for Medicare and Medicaid Services (CMS) to award incentive payments for eligible professionals.4 Patients are more and more considered to possess important experiential knowledge on health and health care, a source of information that is relevant for improving quality of care.2, 3, 4, 5

To truly capture the patient's perspective, it is essential that patients are involved in PROM development,5, 6, 7, 8, 9 as only patients can determine which health outcomes are relevant for them9, 10, 11 and whether the questionnaire captures these outcomes in a comprehensible and understandable manner.12, 13, 14, 15 Besides, if a questionnaire fails to represent the patients’ perspective, it may result in patients failing to complete the questionnaire and a negative impact on the validity.16 Only a few studies investigated patient involvement in the development of PROMs, and only for a small number of PROMs, or for a specific disease.7, 17 Nevertheless, there are concerns that there are many PROMs in use where no patients were involved during the development process. Haywood investigated the quality and acceptability of PROMs used in chronic fatigue syndrome/myalgic encephalomyelitis and found no clear evidence of patient involvement in the development of PROMs.17 Other research using interviews with patients uncovered conceptual difficulties and questionnaire design problems among a few well‐known PROMs which might have been prevented by involving patients in the development.7 It appears that although patient involvement is increasingly accepted as an essential part of the development process, we are still using PROMs which were developed without patient involvement.

Although patient involvement appears to be neglected in PROM selection and at least in the development of earlier PROMs, there is some agreement in the literature on how patients should be involved. There are several phases in the development where the input of patients can be used to create a more patient‐centred PROM. To ensure that the health outcomes and domains measured with PROMs are relevant for patients, patients may partake in identifying core outcomes,10, 11 for example by participating in a reference group or focus groups.18 After establishing which outcomes should be measured, patients should be involved in the generation of items by either focus groups or interviews.12, 15, 18 While interviews can be used to obtain a number of personal feelings and opinions on a subject, which can be especially useful for eliciting the opinions of minority groups, focus groups may be useful for obtaining opinions that are likely to reflect the majority.1, 19 Choosing between these methods may also be dependent upon the patient population or practical considerations.20 Item development may have some overlap with the phase where core outcomes are identified, because although they have different purposes, both could take place using the same qualitative methods. After item development, it is important to determine that the questionnaire is comprehensible and that the content is valid for patients.12, 13, 14, 15 Structured cognitive interviews are an established way to ensure this12, 13 by enabling researchers to determine how items are interpreted by potential respondents and how a response is formed.13, 15

It appears that there are at least three phases in which patients can be involved in the development of PROMs, determining important health outcomes, item generation and checking for comprehensibility and content validity. However, as mentioned before, PROM development procedures are not standardized regarding patient involvement and the few studies who looked at patient involvement in the development of PROMs reported variations in whether and to what extent patients are involved in the development.7, 17 As patient involvement is such an important factor in the development of PROMs,5, 9, 10, 11, 12, 13, 14, 15, 21 it is of the utmost importance that all the parties involved are aware of the level of patient involvement of a PROM if it comes to selecting a PROM or interpreting PROM results. Additionally, awareness of the current situation regarding patient involvement is necessary so that all parties can act accordingly to ensure the development and use of valid patient‐centred PROMs. Therefore, by conducting a scoping review, this study aims to review the level of patient involvement in PROMs development. We expect to find more patient involvement in recently developed PROMs as compared to older PROMs, as the importance of the patients’ perspective is increasingly stressed.2, 3, 4, 5 In summary, our research questions are as follows:

In what ways and to what extent have patients been involved in the development of PROMs?

Has patient involvement in the development of PROMs increased with time?

Methods

Scoping review

This study is a scoping review. Scoping reviews are used to give a unique overview, in this case of patient involvement in the development of PROMs. For a scoping review, studies are not excluded based on type of study (as long as a description of a PROM development was included), type of participants, type of health care or development techniques. Furthermore, studies are not assessed on aspects of methodological quality. Characteristics of a scoping review can be a lack of a narrow review questions, the inclusion of studies which have employed a range of data collection and analysis techniques, the quality of the included studies is not assessed and the subject has not been subjected to a review before.22 Although we did not conduct a systematic review, the PRISMA statement23 was followed where possible.

Search strategy

The search was conducted on 15 May 2014 by author BW. The databases used were PubMed, the Cochrane Methodology register, MEDLINE and EMBASE. The search terms were the mesh terms: (‘Outcome Assessment (Health Care)/methods ‘[Mesh] OR proms[tiab] OR ‘patient reported outcome measures ‘[tiab]) AND (questionnaires[mesh] OR questionnaire*[tiab]).1 The terms were determined after an initial search of the literature and advice from a librarian with expertise in health services research and systematic reviews. Studies describing translations or alterations of PROMs were checked for references to original PROM development studies. Initially, all languages were included and no time restriction was used.

Literature selection

Inclusion criteria

Studies were included if they described the development of a new PROM. A PROM is a questionnaire which measures patients’ perspectives on health outcomes.1, 2 We considered the development of a PROM the process from establishing which outcomes to measure until the psychometric testing. This includes determining which outcome to measure, item development and testing the questionnaire on comprehensibility. Studies were included if they described (a part of) these phases.

Exclusion criteria

Studies which describe the development of a short version, alteration or translation of one already existing questionnaire do not usually go through all the development phases and were therefore excluded. Short versions and adaptations were considered closely to ensure that only studies were excluded which adapted an existing PROM very slightly or shortened a PROM without making any fundamental changes.

Post hoc exclusion criteria

Studies were excluded if they did not concern a published manuscript or if the manuscript was written in a language other than English, German, Dutch or French. As only two manuscripts were written in a different language, the impact was limited.

Study eligibility

Author DB independently scored ten per cent of the abstracts on eligibility, after which disagreements were discussed. The inter‐rater agreement on study eligibility, calculated using the kappa statistic, was 0.71 (95% CI: [0.50, 0.92]). Full‐text selection by a second reviewer was deemed unnecessary as the reasons for exclusion were mostly very clear. However, authors DD and DB independently assessed the eligibility of 10 full texts. Additionally, full texts which were excluded because the questionnaire that was developed measured the process of care instead of outcomes, were discussed among the authors. There were no disagreements. See Fig. 1 for details on the literature selection process.

Figure 1

Search flow.

Data extraction

The basic information that was collected from each article included the first author, the year of publication, the country, the health problem or treatment associated with the PROM, whether the patient group concerned adults, parents or children, whether the PROM is generic or specific, the health outcome measured, and the care type associated with the health problem.

To gain insight into patient involvement in the development of PROMs, additional information was recorded which is displayed in Table 1.

Table 1

The categories for abstracting data regarding patient involvement

Category	Explanation	Example
Determining which health outcome should be measured	During this phase, it is determined which outcome is going to be measured, or domains or a framework are developed
Patient involvement in determining which outcome is measured by including all the patients’ suggestions	Although patients were not actually asked which outcome should be measured, the patients’ suggestions were not restricted	‘Underlying the development of the questionnaire was a concept analysis and a description of patients’ postoperative recovery from the perspective of patients, registered nurses and surgeons’42
Patient involvement in determining which outcome is measured by letting patients help develop frameworks or domains	Patients were restricted to an outcome, but participated fully in developing domains or frameworks	‘As outlined in Fig. 1, the study involved four main steps: (1) qualitative concept elicitation interviews of children/adolescents with RLS and their parents…’43
Item development	During this phase, items are developed. This phase may have some overlap with the first phase as the same methods can be used for both framework development and item development. In case of overlap, both categories were scored
The use of focus groups with patients	Focus groups with patients contributed to the development of items	‘Second step: to elicit relevant verbal material: Patients with hip or knee OA, and relevant health professionals, were recruited to take part in focus groups…’44
The use of interviews with patients	Interviews with patients contributed to the development of items	‘A pool of potential scale items was generated from semi‐structured interviews of 27 people with PSP’45
Patient involvement using other methods	Patients were involved in the item development using a different method than interviews or focus groups	‘Patients were first provided with an open‐ended free text space to comment broadly on what “quality of life” meant to them as they coped with MM’46
The use of other PROMs, literature or professionals	Items were developed (partly) using other sources than patients, such as experts, results of a literature review or other questionnaires	‘The Palliative Care Outcome Scale (POS) was developed using data from a review of other outcome measures used, or proposed for use in evaluating the palliative care of patients with advanced cancer’47
Testing for comprehensibility	During this phase, the developers ensure that the questionnaire is understandable and the questions are interpreted correctly
The use of cognitive interviews with patients	Cognitive interviews with patients took place	‘Additionally, the questionnaire will be pretested with a variety of patients in cognitive interviews using the format outlined by Willis’48
The use of other methods involving patients	Patients were involved using a different method than cognitive interviews for testing the PROM for comprehensibility	‘The HDQoL was then pre‐tested on a group of 20 participants from pre‐symptomatic to late‐stage disease and reviewed in the light of their feedback’49
The use of professionals or other non‐patient groups	Other, non‐patient, groups were asked to assess the questionnaire for comprehensibility	‘To assess the questionnaire content validity (the extent to which a measurement reflects the specific intended domain of content and to test the items face validity (extent to which a measure seems to calculate a phenomenon on face value, or intuition, e.g. test/survey items are sent to experts to obtain suggestions for modification, comprehensiveness and relevance to haemophilia, a draft version of the questionnaire (75 items) was sent with a standardized evaluation form to a group of experts in the treatment of patients with haemophilia (n = 20) from the collaborating centres’50

Authors DD and DB extracted the data for 10 studies. Disagreements were discussed, after which the categories were further specified. Main discussion points were as follows:

Generic or specific? Initially, we defined a PROM as generic if everyone in the general population could answer the questions and a PROM as specific if it covered a specific disease. However, several PROMs were not generic, but covered health problems relevant for more than one disease, such as pain or fatigue. Conventional definitions24 would label this category as specific as it measures a specific concept. However, most of these questionnaires can be used for a far larger group of patients than a disease‐specific questionnaire could, which is why we gave this group of questionnaires the label ‘semi‐generic’.

What is regarded as patient involvement in determining which health outcome to measure? Although ideally patients are involved from the start of the development by asking them what outcomes are important to them, this is extremely rare. Therefore, we broadened this category to include cases where patients contributed to the development of the framework or domains within an established outcome or where patient suggestions were not restricted in any way.

Statistics

A chi‐square test using the variables ‘Health outcome measured as outcome of surgery, cancer, chronic disease, mental health, other, lifelong disorder’ and ‘patient involvement (‘yes’ or ‘no’)’ was conducted to give insight into whether the level of patient involvement is influenced by the patient population for which the PROM is meant. To analyse patient involvement in PROM development over time, we distinguished between three time periods: before 2005 #bib2006–2010 and after 2010. The choice of these time frames was quite arbitrary, but it resulted in three groups of more or less equal size. The dependent variable described whether any patient involvement had taken place (‘yes’ or ‘no’). A chi‐square test was conducted to analyse whether patient involvement differs between these three time periods. Post hoc pair wise comparisons were conducted using a chi‐square test. Additional analyses were performed using different time periods: before and after 2006 (mean of PROM publication dates), before and after 2008 (median of PROM publication dates) and time as a continuous variable. Data were analysed using spss.25

Results

Search flow

As shown in Fig. 1 #bib4489 studies were identified from the databases PubMed, EMBASE, MEDLINE and the Cochrane methodology register. After a search for duplicates, 329 were removed, leaving 4160 studies. Of these studies, 219 studies remained after selecting abstracts which appeared to meet the inclusion criteria. Finally, a full‐text selection resulted in 189 studies. The studies excluded during full‐text selection were mostly presentations (n = 10), adaptations of already existing PROMs (n = 9) or patient‐reported experience measures (n = 4).

Study characteristics

The 189 studies were published from 1980 until 2014 and described the development of 193 PROMs. The PROMs were mostly developed in the USA (n = 69), the UK (n = 62) and Germany (n = 17). Most PROMs were developed for a specific patient group (n = 152) or few patient groups (n = 32). Nine questionnaires were generic. Of the 193 PROMs, 180 questionnaires were meant to be filled out by adults and one by parents. A range of patient populations were included, from haemophilia (n = 6) and mental health patients (n = 5) to diabetes patients (n = 4). The majority of the PROMs were meant for chronic disease patients (n = 59) or for patients undergoing surgery (n = 25). Most PROMs measured solely quality of life (n = 28), health‐related quality of life (n = 21) or health status (n = 15). Both relatively unknown PROMs and much used PROMs such as the SF‐36, the EQ‐5D and the KOOS were included in the review.

Patient involvement

In the development of 74.1% of PROM patients were involved in some way, leaving 25.9 per cent of PROMs without any patient involvement (Table 2). Patients were mostly involved in the item development (58.5%), especially using interviews (31.6%). Testing for comprehensibility closely followed item development (50.8%), while patients were never involved in determining which outcome should be measured. However, in some cases, patients were not limited to just the predetermined outcome and all suggestions were included (1.0%), and in some cases, patients were involved in developing the domains or a framework (9.8%). These examples of patient involvement together suggest some limited patient involvement in determining which outcome to measure in some PROMs (10.9%). In 6.7% of the PROM development studies, patients were involved in all the development phases we documented. The level of patient involvement did not differ dependent on whether the PROM was developed as a measurement of a health outcome of surgery, cancer, chronic disease, mental health or a lifelong disorder (χ2 (6) = 7.063, P = 0.315). There is a small significant difference between the levels of patient involvement for the three time periods (χ2 (2) = 6.511, P = 0.039). Patient involvement was significantly lower between the years 2006 and 2010 compared to after 2010 (χ2 (1) = 6.545, P = 0.011), but no significant difference was found between before 2005 and after 2010 (χ2 (1) = 2.872, P = 0.090) and between before 2005 and 2006–2010 (χ2 (1) = 1.048, P = 0.306). Sensitivity analyses using different time periods showed similar results.

Table 2

Patient involvement in the development of patient‐reported outcome measures (PROMs) (n = 193)

	n	%
Patient involvement in any phase of the development *	143	74.1
Patient involvement in one phase	67	34.7
Patient involvement in two phases	63	32.6
Patient involvement in three phases	13	6.7
Establishing which outcome to measure	21	10.9
Unrestricted input from patients	2	1.0
Patient involvement in the development of frameworks or domains	19	9.8
Item development	113	58.5
Focus groups with patients	42	21.8
Interviews with patients	61	31.6
Other methods involving patients	27	14.0
Establishing the comprehensibility of the questionnaire	98	50.8
Cognitive interviews	48	24.9
Other methods involving patients	55	28.5
Patient involvement before 2005	53	73.6
Patient involvement between 2006 and 2010	46	65.7
Patient involvement after 2010	44	86.3

*Questionnaire development was divided into three development phases: establishing which outcome to measure, item development and testing for comprehensibility.

Discussion

As patient involvement in the development of PROMs is essential for measuring patient relevant outcomes,5, 6, 7, 8, 9 this study aimed to review whether and to what extent patients have been involved in the development of PROMs. In most PROM development studies, some patient involvement is recorded, but only in a few cases, patients have been involved throughout the development process. Surprisingly, in more than a quarter of PROM development studies, patient involvement was not documented at all. Other research looking at patient involvement in the development of PROMs for certain health problems suggests similar or even less favourable results for patient involvement.7, 17

Besides patient involvement in development phases, this review also investigated whether patient involvement increased with time. We expected that, as the patients’ perspective is increasingly regarded as important,2, 3, 4, 5 patients would be more involved in recently developed PROMs. There appeared to be some differences in patient involvement, but none that suggest an increase of patient involvement over time. This is especially surprising because patient involvement is not only recommended by many researchers,5, 6, 7, 8, 9 a few years ago it has also become a requirement if the PROM is used in medical product development.26 However, in ISOQOL's reporting standards for the use of PROMs in randomized controlled trials,27 the issue of patient involvement was rejected after consideration. Similarly, the COSMIN checklist, which assesses the methodological quality of studies on measurement properties of PROMs,21 barely mentions patient involvement. It only asks to check whether the items are relevant to the study population and whether the questionnaire was pre‐tested, for example using cognitive interviews. Apparently, patient involvement is still not always considered important enough to be included in reporting formats. Additionally, although many benefits may be gained by involving patients during the development process, patient involvement can also result in budget and time issues,9, 28 which may prevent PROM developers from involving patients. Differences in requirements regarding patient involvement and negative consequences of patient involvement may explain the variation we found in the level of patient involvement and the lack of a significant increase in patient involvement over time.

If patients are still not always involved, and if the involvement varies in intensity (from letting some patients comment on the questionnaire in writing after completing it, to the whole process of organizing focus groups, interviews and cognitive interviews), what does this mean for the validity of the questionnaires and for the use of their results by patients, health professionals and health insurers? Patient involvement is seen as essential to ensure the content validity of the questionnaire.12, 13, 14, 15 Involving patients by focus groups or interviews during item development ensures a better understanding of illness experience20 and could help challenge tacit models on which PROM designs are based.29 Furthermore, the content of the questionnaire may be less relevant to patients if they are not involved in the development. This, in turn, may lead to a negative attitude towards the questionnaire and failure to answer the questions. All this can negatively affect the validity of a survey.16 Based on the findings of this review, it appears that the content validity of many PROMs can be improved. This is troubling as PROMs are increasingly used to inform patients, to improve the care of individual patients and to assist purchasers.2, 3, 30, 31 PROMs may be used to improve the care of individual patients by increasing health professionals’ awareness of, and ability to, address patients’ concern.32 However, to use PROMs for this purpose, it is essential that a PROM is used which reflects the patients’ needs and concerns accurately.33 Furthermore, patient information based on PROMs may not sufficiently cover aspects that are most relevant to patients.

In the Netherlands, as well as in the English NHS and the American Medicare and Medicaid system, the use of PROMs in models of payment by results is a particularly relevant issue. In the Netherlands in 2006, managed competition has been introduced. In this system, health insurers should contract providers based on quality and costs of care. To do so, health insurers need instruments for measuring the quality of the care delivered by providers.34 Besides measures of effectiveness and safety and patient experience measures,35 PROMs are slowly introduced in the Netherlands to assist health insurers in selecting the best providers for contracting. In theory, this is a good thing, as PROMs are meant to reflect the patient's perspective1 and research suggests that patients have a different view on which aspects of their care are important to them. Identifying and understanding which aspects of care are the most important to patients may be the key to good care.36 However, if the validity of PROMs may need to be improved and if PROMs do not always reflect the patient's perspective, health insurers may want to ask themselves whether they are going to select health providers for the right reasons.

Limitations

The present findings should be regarded with some caution because of study limitations. First, patient involvement was recorded using the documentation of the development process in the included articles. The results are therefore dependent on the quality and accuracy of the documentation. Patient involvement may go unnoticed if it is not documented in the article describing the development. Second, reporting standards may have improved over time and although patient involvement was recorded almost as many times in older publications as in recent publications, a lower standard of reporting may mean that levels of patient involvement in earlier studies have been underestimated. Third, not all development studies were documented in detail and sometimes PROM developers used one method for several development phases, which occasionally made it difficult to separate the development phases. However, to ensure that patient involvement was not underestimated, we recorded any suggestion that a method was used for more than one phase as patient involvement during several development phases. Fourth, as we wanted to compare PROMs on patient involvement during the three phases of development, we selected studies describing one or more of these phases. However, sometimes PROMs are further developed after the conclusion of these phases. As further development is very difficult to identify and the methods used for further development can vary greatly, we were unable to include this. Nevertheless, it is possible that for some of the PROMs included in this review, a higher level of patient involvement has been practised following the initial development. Fifth, although we were able to compare the PROMs on patient involvement in three development phases, we could not compare the PROMs on the level of patient involvement during these phases. The amount of information that is given in publications varies hugely and often lacks detailed information on patient involvement. Sixth, statistical analyses were performed on data which was not derived from a systematic review. Although systematic review guidelines were followed where possible, the included papers may not be a complete reflection of all PROM development papers, which may have implications for the interpretation of the results.

Implications

This review may have several implications for the use and development of PROMs. First, a consensus should be reached on how patients should be involved in the development of PROMs. In this review several studies, describing opportunities to involve patients were brought together to use as a guide for extracting data concerning patient involvement from the papers. However, to ensure that all PROMs incorporate patient involvement throughout their development phases, it is important that one, easy to use, guideline is created. Perhaps inspiration may be gained from best practice examples such as the Genetic Counseling Outcome Scale37 and the Breast‐Q,38 where patients were involved throughout the development process. In the development of an instrument for fatigue in rheumatoid arthritis, patients were even given the role of research partners.39 Preferably the guideline is developed involving researchers, professionals, relevant organizations and patients to ensure input from all stakeholders and a widespread acceptation of the guideline.

Besides a guideline for patient involvement in PROM development, the implementation of a guideline for how to report the methods used to involve patients in publications is also necessary. As patient involvement is necessary to ensure that the questionnaire measures the patients’ perspective,5, 6, 7, 8, 9 existing PROMs should be selected based on the level of patient involvement. However, certain information about the development process is necessary for comparing PROMs on patient involvement. Although an attempt at guiding researchers in reporting patient involvement has been made,40 studies still offer varying amounts of information about their development process. This makes it very difficult to select a PROM based on its level of patient involvement. A guideline that is supported by all relevant parties and enforced by journals should help ensure that all publications include information needed to make an informed choice for a PROM.

Third, besides patient involvement in the development of PROMs, patients may also be involved in the interpretation and presentation of PROM results. Patients may be involved in the interpretation of PROM results using importance ratings. Importance ratings may give an indication of patient's values, needs and expectations regarding health care.41 Importance ratings enable the weighting of the PROM results according to the importance patients allocate towards aspects of the PROM. Furthermore, as patients can use PROM data to make an informed decision concerning health‐care providers or treatments,30, 31 PROM results should be freely available and easy to read. Patient involvement in the presentation of the results could help easy use of PROMs by patients.

Conclusion

Although some patient involvement takes place in most PROM development studies, the level of patient involvement varies greatly. Furthermore, in more than a quarter of PROM development studies, no patient involvement was recorded. This lack of patient involvement throughout the development may have consequences for how well the questionnaire reflects the patient's perspective, which in turn may result in limited benefits for the use of PROMs in individual care, for making decisions by patients between treatments and health professionals and, finally, for rewarding health professionals or even selectively buying health care from certain health professionals. For PROMs to truly measure the patient's perspective, further attention to building and/or disseminating consensus on requirements for patient involvement in the development of PROMs is necessary and existing PROMs should be carefully selected on the level of patient involvement.

Conflict of interest

The authors have no conflict of interests to disclose.

Funding source

This study was funded by The National Health Care Institute, which is located in Diemen, the Netherlands. The National Health Care Institute did not have any role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Appendix S1. The results of the data abstraction from the included literature.

Click here for additional data file.