The selection and use of outcome measures in palliative and end-of-life care research: the MORECare International Consensus Workshop.

CONTEXT: A major barrier to widening and sustaining palliative care service provision is the requirement for better selection and use of outcome measures. Service commissioning is increasingly based on patient, carer, and service outcomes as opposed to service activity.
OBJECTIVES: To generate recommendations and consensus for research in palliative and end-of-life care on the properties of the best outcome measures, enhancing the validity of proxy-reported data and optimal data collection time points.
METHODS: An international expert "workshop" was convened and an online consensus survey was undertaken using the MORECare Transparent Expert Consultation to generate recommendations and level of agreement. We focused on three areas: 1) measurement properties, 2) use of proxies, and 3) measurement timing. Data analysis comprised descriptive analysis of aggregate scores and collation of narrative comments.
RESULTS: There were 31 workshop attendees; 29 recommendations were included in the online survey, completed by 28 experts. The top three recommendations by area were the following: 1) the properties of the best outcome measures are responsive to change over time and capture clinically important data, 2) to enhance the validity of proxy data requires clear and specific guidelines to aid lay individuals' and/or professionals' completion of proxy measures, and 3) data collection time points need clear identification to establish a baseline.
CONCLUSION: Outcome measurement in palliative and end-of-life care requires the use of psychometrically robust measures that are clinically responsive, with defined data collection time points to establish a baseline and clear administration guidelines to complete proxy measures. To further the field requires clinical imperatives to more closely inform recommendations on outcome measurement.

Introduction

Widening access to palliative and end-of-life (EOL) care services is advocated with corroboration of patient and carer benefits, greater potential for health service cost savings, and increasing demand with an aging population. A major barrier is the requirement for better selection and use of outcome measures to demonstrate the effectiveness of services. Commissioners of health and social care services increasingly require service providers to use patient, carer, and service outcomes to demonstrate a service's safety, effectiveness, and quality as opposed to detailing service activity. In research and clinical practice, a multitude of measures are used, and frequently these measures are not validated with palliative care populations. This hampers meta-analyses, limits responsiveness to change in outcome, evaluating service effectiveness, and developing the evidence base to inform best practice.

Trials and nonrandomized designs in palliative and EOL care are often compromised by the use of untested outcome measures, measures not developed for palliative care populations, uncertainty as to the best measurement time points, and the use of measures for symptom change with less use of measures encompassing the multiple domains of palliative care (e.g., Palliative care Outcome Scale [POS], Edmonton Symptom Assessment System). These limitations are not unique to palliative and EOL care. The COnsensus-based Standards for the selection of health Measurement INstruments is a comprehensive checklist for assessing the quality of the measurement properties of health status questionnaires. The checklist was developed in response to the rapid increase in health status questionnaires and the need for quality criteria to compare measures in systematic reviews, identify shortcomings, and design studies validating measures.

Palliative and EOL care involves people with increasing debility associated with advancing disease and has a broad mandate of care provision, with intended outcomes of improving quality of life for patients and their caregivers. The nature of palliative and EOL care requires measurement properties that accommodate the following: the multiple domains of palliative care; to change over time and increasing levels of fatigue; the use of proxies, particularly when individuals are near to death; and timing to detect change and monitor sustainment of change. These challenges are well reported, and international advancements to address them are evident: projects such as the Palliative Care Outcomes Collaboration in Australia that uses national standardized assessments of palliative care outcomes and a European collaboration, entitled PRISMA, focused on promoting best practice in the measurement of EOL care (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) working with a European Association of Palliative Care (EAPC) Taskforce on patient-reported outcome measures (PROMs) in palliative care to further the work and provide guidance on selecting and using measures. Internationally, there is increasing use of multi-domain palliative care measurement tools in research and clinical practice (e.g., POS and Support Team Assessment Schedule [STAS]).

No work, however, has considered specific recommendations for the use and selection of outcome measures in trials and nonrandomized studies of palliative and EOL care interventions. This article reports on an international outcome measurement workshop and online consensus survey that aimed to identify agreed best practice on selecting and using outcome measures. The work is part of the Methods Of Researching End of life Care (MORECare) project, developing evidence-based guidance on the design and conduct of evaluative research on palliative and EOL care. MORECare integrated data from three systematic reviews and five workshops to generate a statement (the MORECare statement) that sets clear standards on good research practice in evaluating services and treatments in palliative and EOL care. This paper reports on the outcome measurement workshop.

Methods

Setting

An invitation-only workshop was held as a joint initiative between the MORECare and PRISMA projects at the 2011 EAPC conference to further the European work undertaken by PRISMA and the EAPC Taskforce on PROMs in palliative care.

Participants

Holding the workshop at an international conference intended to enable participation by international experts in palliative care research, service provision, and policy. Participants were identified from the EAPC online conference program and considered experts from their publications. Fifty-nine individuals were identified and received a personal e-mail invitation to attend the workshop and a single reminder e-mail.

Design

We used the MORECare Transparent Expert Consultation method, which incorporates consensus methods of nominal group and consensus survey. The Transparent Expert Consultation comprised a staged process: Stage I—identifying critical issues from the literature; Stage II—a workshop with experts on the issues using nominal group technique to generate recommendations; and Stage III—an online consensus survey to prioritize and identify areas of contention/uncertainty (Fig. 1).

Fig. 1

Flow diagram of the study design.

Stage I: Identifying Critical Issues

Critical issues on selecting and using outcome measures in palliative and EOL care were identified from systematic reviews of intervention studies and outcome measurement. The workshop and consensus survey focused on three critical questions:

What are the properties of the best primary outcome measures for evaluation of palliative and EOL care services?

How can we best enhance the validity of proxy-reported outcome data?

When are the optimal time points to collect outcome data to evaluate the effectiveness of service models in palliative and EOL care?

Stage II: Workshop

Participants received a pre-workshop briefing pack detailing the aim, critical questions, and workshop format. The workshop format comprised presentations on the MORECare project and the project's interface with the PRISMA/EAPC Taskforce, followed by three structured groups of seven to 13 participants each focused on one of the critical questions. The group structure used a modified nominal group facilitated and scribed by members of the MORECare and PRISMA projects. The facilitators guided participants through a structured process of: 1) brief discussion, 2) individual writing of recommendations and ranking, and 3) individuals in turn reading out their highest ranked recommendations until individual lists were exhausted (or time exceeded). The scribes wrote the recommendations and ranking on flip-charts, and each small group discussed and agreed on the final priority order, which they presented and discussed with the whole group. Participants listed and ranked recommendations from one to five (highest to lowest) on structured A4 sheets detailing the respective group question, the ranking scale and boxes to list recommendations, rank, and detail rationale.

Stage III: Consensus Survey

Recommendations were posted online to the workshop participants (excluding the facilitators and scribes) and members of the MORECare Project Advisory Group. Participants received a personalized e-mail invitation and reminder after two weeks. The online participants ranked, from one to nine (strongly disagree to strongly agree), the extent they agreed with a recommendation and used free-text spaces to comment on each recommendation.

Data Analysis

Individual recommendations and ranking were entered into Excel spreadsheets with assigned participant identification numbers. Flip-chart records and the scribes' notes were typed. Three researchers (C.J.E., H.B., M.G.) drew on qualitative analysis to identify themes in the recommendations and assign codes and collate typed data. Coded recommendations were rearranged by theme in Excel spread sheets. Duplicates were combined and recommendations arranged by priority ranking. The recommendations retained participants' original language, with amendments to enhance clarity. A third researcher (I.J.H.) reviewed the analysis and proposed recommendations. After combining for duplicates, we included all recommendations in the online survey that participants had ranked three and above and/or participants' listed recommendations formed a prominent theme. The MORECare research team reviewed and agreed on the recommendations for the online survey.

Analysis of scaled data followed conventional rules of descriptive statistics (frequencies and medians) and plots (box and whisker plots of interquartile ranges to analyze and interpret levels of agreement [Table 1]). Narrative comments were collated by recommendation, themes identified, and rearranged in Excel sheets to enable comparison. The analysis of comments intended to aid understanding and provide illustrative examples of the issues raised by the proposed recommendations.

Table 1

Recommendation Agreement: Interpretation of Median Regions and IQR

Median Regions and IQR	Interpretation
7–9	Recommendations are indicated
4–6	Recommendations are equivocal
1–3	Recommendations not indicated
IQR in one region	Strict agreement for recommendation
IQR in any three-point region	Broad agreement for recommendation

IQR = interquartile range.

Ethics

The study was approved by the University of Manchester Research Ethics Committee (reference number: 10328).

Results

Thirty-one international and national experts attended the workshop comprising senior academics, policy makers, and clinicians, representing countries from Europe (e.g., Norway, Belgium, U.K., Italy, Germany), North America, and Australia. All held a title of professor or doctor. Twenty-eight experts completed the online survey.

Recommendations and Levels of Agreement

The workshop generated 155 individual recommendations; 29 recommendations were included in the online consensus survey after combining duplicates and completing analysis. The recommendations included in the online survey were either generic describing good practice in choosing and administering outcome measures in effectiveness research or focused on specific challenges in research on palliative and EOL care (Table 2). Overall, the median levels of agreement for the 29 recommendations showed that most recommendations (n = 26) were indicated with medians ≥7; none were not indicated (medians ≤3) (Table 2). However, the levels of agreement across the three areas varied. The recommendations on: 1) the properties of outcome measures showed mainly strict agreement that the recommendations were indicated (n = 6/11 [Fig. 2]), 2) enhancing proxy data showed mainly broad agreement (n = 6/9 [Fig. 3]), and 3) data collection time points all showed broad agreement (n = 9 [Fig. 4]). Recommendations in which the level of agreement was equivocal or broad were considered as uncertain or contentious. Qualitative analysis of the workshop discussions and the online survey free-text comments informed understanding on these areas. We present quotes to illustrate these debates.

Table 2

Recommendations on Selecting and Using Outcome Measures by Area and Level of Agreement

No.	Area 1: Properties of the Best Primary Outcome Measures in Evaluations of Palliative and EOL Care Should …	Median (IQR)
	Strict agreement that recommendation is indicated
1	Be easy to administer and interpret (e.g., short and low level of complexity).	8 (7–9)
2	Be applicable across care settings to capture change in outcomes by location (e.g., patient's home, hospital, hospice).	8 (7–9)
5	Be responsive to change over time and capture clinically important data.	8 (7.5–8)
6	Have demonstrated content validity.	8 (7–9)
9	Have demonstrated reliability.	8 (7–9)
10	Be integrated into clinical care.	8 (7–9)
	Broad agreement that recommendation is indicated
3	Work across a disease trajectory from diagnosis to death.	7 (6–8)
4	Be culturally sensitive to a respective population group (e.g., linguistically and culturally).	7.5 (6–8)
7	Have demonstrated construct validity.	8 (6–9)
8	Have demonstrated face validity.	8 (6–9)
11	Be appropriate for clinical practice, research, and audit uses.	7 (6–9)

EOL = end of life; IQR = interquartile range; PROMs = patient-reported outcome measures.

Fig. 2

Properties of the best primary outcome measures in evaluations of palliative and EOL care. Box and whisker plot of the interquartile ranges and medians of level of agreement for the 11 recommendations (box: 25th and 75th percentiles; whiskers: minimum and maximum).

Fig. 3

Enhancing the validity of proxy data in evaluations of palliative and EOL care. Box and whisker plot of the interquartile ranges and medians of level of agreement for the nine measurement recommendations (box: 25th and 75th percentiles; whiskers: minimum and maximum).

Fig. 4

Data collection time points in evaluations of palliative and EOL care. Box and whisker plot of the interquartile ranges and medians of level of agreement for the nine recommendations (box: 25th and 75th percentiles; whiskers: minimum and maximum).

Properties of the Best Outcome Measurements

Properties of the best primary outcome measures formed 11 recommendations (Fig. 2; Table 2). Overall, there was strict agreement that the properties of the best outcome measures required valid measures that were responsive to change over time, captured clinically relevant data, and were easy to administer across care settings (Recommendations 1, 2, 5, 6, 9, 10). The text data indicated that measures required flexibility in the administration of the measure (e.g., PROMs, observation) to ensure responsiveness to patients' fluctuating capacity and increasing fatigue, notably in the last days and hours of life. An area of less certainty was the recommendation that a measure was appropriate for clinical practice, research, and audit, with broad agreement indicated but with the widest spread (Recommendation 11) (Fig. 2).

Uncertainty centered on accommodating competing priorities for practice, research, and audit, summarized by a survey respondent as:Measures in research required greater detail than the comparatively brief measures required in clinical practice. However, there was agreement on the importance of research capturing clinically relevant data (Recommendation 5) that could be integrated into clinical care (Recommendation 10). A way forward discussed in the workshop and in the online survey comments was the use of “core outcomes” with additions depending on the context, as a survey respondent stated:

It is impossible to serve all 3 masters optimally. They need to be optimized for one or perhaps two goals. (ID OMs 6)

… core outcomes (for all settings) and others setting-specific, this will give a more accurate approach, for the process and for specific settings. (ID OMs 12)

Enhancing the Validity of Proxy-Reported Outcome Data

Greater uncertainty surrounded the seven draft recommendations on enhancing the validity of proxy-reported data, with wide variance in degrees of agreement (Fig. 3). There was strict agreement that to enhance proxy measures required clear specific guidelines to aid completion, the differentiation of patient and proxy data in datasets and that proxy measures captured patients' experiences of care (Recommendations 14, 17, 20) (Table 2).

Two recommendations were equivocal and concerned increasing the reliability of proxy measures and understanding bias (Recommendations 13 and 18). Recommendation 13 that proxy measures should only be used in conjunction with observer measures was viewed as useful in the absence of PROMs data. However, uncertainty surrounded the validity of using such measures in conjunction and the best method of statistical analysis. Best research practice meant using valid measures, but further research was required on the contribution of using measures in conjunction to enhance reliability. There was agreement that Recommendation 18 was indicated on the importance of investigating factors that may influence proxy raters to reduce bias. However, participants' comments questioned the feasibility of this in research practice because of the complexity in collecting and interpreting such data.

The main area of contention was the recommendation on the precedence of PROMs over proxy measures, indicated by the wide spread (Recommendation 12) (Fig. 3). The survey participants' comments indicated that using PROMs data alone negated that palliative care encompassed both patients and their families, would fail to capture data in the last days of life from, for example, observation, and the relevance of clinical data, for example, medication use. Survey participants advocated the requirement for multiple measures in research on palliative care. The workshop delegates advocated that proxy measures were “second choice” over PROMs, but proxy data were “indispensable” in palliative care because of patients' deteriorating capacity and increasing fatigue.

Optimal Data Collection Time Points in Clinical Trials and Evaluations of Interventions

There was broad agreement that the eight recommendations on time points were indicated (Fig. 4). The wide distribution and no strict agreement on the recommendations suggested uncertainty/contention. The recommendation of death as the endpoint in retrospective data analysis was equivocal (Recommendation 25). The online survey comments revealed that death as an endpoint was considered too narrow as it failed to capture the holistic nature of palliative care, which encompasses carers and family members across the disease trajectory and into bereavement. However, death could form an important endpoint in retrospective data analysis depending on the objectives of the study, as one workshop participant stated:

Use death as [an] endpoint and count burden from this – as time or referral/first assessment may vary so greatly. (ID G2.015)

Although the workshop participants used the term endpoint, this caused confusion in the online survey, and alternative terms of anchor point or starting point were proposed. The confusion likely contributed to wide range of scores and finding of an equivocal recommendation.

Discussion

Research on palliative and EOL care is beset by well-reported challenges rooted in involving individuals with advanced illness in research and their families, the multiple domains of palliative care provided across health and social care, and the intention to improve quality of life in the face of deterioration. These challenges have hampered the use and selection of outcome measures in trials and nonrandomized studies on palliative and EOL care services. We critically discuss three main areas to further this field of research: priority measurement properties, incorporating proxy data, and identifying time points.

Measurement Properties

The findings corroborate work undertaken by the PRISMA project that research on palliative care requires the use of short, brief outcome measures that accommodate patients' increasing levels of fatigue and minimize burden. This does not equate to “easy” to measure aspects of care, but areas important to patients and families that encompass desired benefit from a service or intervention. The challenge is to use measures that capture the multiple domains of palliative care, but in a format that is short and easy to interpret. Our findings support the requirement for measures used in palliative care to be applicable for both clinical practice and research. Work in Australia has demonstrated the feasibility of collecting national palliative care outcome data across specialist services using defined measurement tools incorporated in routine practice and the potential for national benchmarking for care quality and outcomes. However, our findings and the PRISMA work show that there are competing priorities between the requirements of a measure used in clinical practice and research. Time pressure in clinical practice necessitates the use of short measures (e.g., six to 10 items) that are easy to administer and interpret to inform clinical decisions. In research evaluating the effectiveness of an intervention or service, the measure(s) used needs to demonstrate the degree of change over time, not simply the presence or absence of, for example, a symptom. Both clinical and research settings, however, require the use of measures with validated psychometric properties to ensure scientific rigor and enhance clinical decision making. To accommodate these competing priorities (and similarities) of our findings, the PRISMA work suggests the use of core measures for clinical and research practice and further assessment of specific areas for research when indicated. Mularski et al. propose, from consensus work, that evaluation studies use an outcomes measurement strategy that encompasses three outcome categories: condition specific, patient outcomes, and family/caregiver outcomes. Measures such as POS (http://pos-pal.org), the Edmonton Symptom Assessment System, and Memorial Symptom Assessment Scale are examples of measures commonly used in both palliative care clinical practice and research, are validated with palliative care populations, and are short to administer and multidimensional.

Palliative care is provided internationally across health and social care settings and to patients with cancer and increasingly to those with nonmalignant illness. Our data indicate the requirement for measures to be culturally sensitive and be applicable across care settings to capture change in outcomes, particularly for patients who move between care settings, for example, at home, inpatient hospital, or care home. The development of existing measures is apparent, notably POS, with work to increase cultural sensitivity for countries both in Africa and Europe, and use with disease groups other than cancer. Further validation work of palliative care measures like POS, however, is required for specific patient groups, notably patients with dementia and for social care settings (e.g., care homes).

Incorporating Proxy Data

Our data showed the clinical complexity for palliative care in using PROMs as the “gold standard” in research. The complexity of palliative care service provision to meet diverse care needs of patients and families necessitates data collection from multiple sources. It is an essential requirement in research on palliative and EOL care to expect and plan for missing data because of deterioration. Proxy-reported outcome data are integral sources in palliative care, particularly at points of a patient's deterioration or unstable symptoms when physical and/or mental debility may prevent use of a PROM. At these points, patients likely experience increasing symptom distress, points at which it is imperative to understand the effectiveness of the care and treatment provided. PROMs, such as POS and Memorial Symptom Assessment Scale, are validated for use with a proxy.

However, the extent a proxy report reflects a patient response depends on the degree of agreement and the direction of agreement (e.g., proxies report high ratings when patients report low) between the proxy and patient; the greater the level of agreement, the greater the confidence in including proxy reports. Studies in palliative care have shown wide variance in agreement responses depending in particular on the variable measured and the patient-proxy dyad. Studies, for example, report high agreement for observable patient variables (e.g., ability to transfer) and moderate to low for psychological domains and subjective patient experiences (e.g., quality of life). In patient and caregiver dyads, higher agreement is reported than in patients and health care provider dyads, but over time, similar reporting is seen as practitioners' knowledge of their patients increases. Our findings correlate with studies on proxy and patient reporting that to further the inclusion of proxy reports requires greater understanding of the factors that influence reporting to improve reliability (e.g., degree of neutrality, knowledge of patient), further development of proxy measures to capture patients' subjective experiences (e.g., quality of life), and ways to enhance statistical analysis.

Identifying Time Points

Identifying optimal time points intends to maximize the number of patients available at consecutive points of measurement to demonstrate the effectiveness of the intervention. Our data show that best practice to identify time points was the area of most uncertainty, with broad or equivocal agreement surrounding the recommendations posed. Identifying time points in research on palliative and EOL care is a prominent methodological challenge rooted in minimizing patient burden, imprecision in identifying the state of “dying,” and attrition. In particular, attrition from increasing debility minimizes data collection for individuals with the greatest symptom burden and compromises a study's internal validity with overrepresentation of the “healthiest” patients. Hence, careful planning of time points is crucial.

Our data show broad agreement for the recommendations on establishing time points before the evaluation, and the frequency of measurement is determined by the degree of burden of the data collection tools. These recommendations are likely too narrow, hence the wide spread of responses. The timing and frequency of measurements is embedded within a conceptual framework for the intervention. The conceptual framework is developed from the research question and detailed understanding of the intended intervention outcome/benefit, the expected response pattern (e.g., two weeks post-consultation with the palliative care clinician), and the phase of illness (e.g., stable disease, deteriorating, dying phase).

The recommendation of death as an endpoint in retrospective evaluations was equivocal. This likely reflects misunderstanding of the term “endpoint.” The use of anchor point from which data are included may have increased clarity. Retrospective evaluations in palliative care are important study designs. Prospective designs are hampered by imprecision in the identification of the state of dying, particularly in measures of quality of life, which often deteriorate in the last days of life. A way to accommodate this is prospective observations and data analysis that uses both prospective analysis from the baseline and retrospective analysis from the last observation before death. Death is a fixed anchor point for the whole sample and provides more reliable data on the clinical outcomes over time.

Limitations

We undertook the consensus workshop at an international conference in palliative care. This enabled participation by international experts but limited invitations to those attending the conference and the time available, particularly for discussion. The use of clear questions and a structured process to generate the recommendations was intended to maximize the time. We limited the online consensus to a single round to enhance the response rate. This prevented wider consultation on the survey comments that detailed reasons for scoring as well as measures of central tendency and dispersion. There is no agreement on the best method for mathematical aggregation in consensus surveys. We used an implicit approach of medians and measure of dispersion and the Jones and Hunter established method for interpreting scaled data.

Conclusions

To enable the wide application and interpretation of outcome measures in palliative and EOL care, the measurement properties need to be validated for this population, capture multidimensional components, and be broadly applicable across health systems, populations, disease trajectories (including post-death for bereaved carers), languages, and cultures. The best measures have essential properties of simplicity of use and good interpretability, the ability to measure clinically important change, and reporting by a patient or proxy informant. To further the field requires clinical imperatives to more closely inform recommendations on outcome measurement.