Towards a standardized method of developing quality indicators for palliative care: protocol of the Quality indicators for Palliative Care (Q-PAC) study.

BACKGROUND: In recent years, there have been several studies, using a wide variety of methods, aimed at developing quality indicators for palliative care. In this Quality Indicators for Palliative Care study (Q-PAC study) we have applied a scientifically rigorous method to develop a comprehensive and valid quality indicator set which can contribute to a standardized method for use in other countries. METHODS AND
DESIGN: Firstly, an extensive literature review identified existing international quality indicators and relevant themes for measuring quality in palliative care. Secondly, the most relevant of these were selected by an expert panel. Thirdly, those prioritized by the experts were scored by a second multidisciplinary expert panel for usability and relevance, in keeping with the RAND/UCLA-method, combining evidence with consensus among stakeholders. This panel included carers and policymakers as well as patients and next-of-kin. Fourthly, the draft set was tested and evaluated in practice for usability and feasibility; the indicators were then translated into questionnaires presented to patients, next-of-kin and care providers. To encourage the acceptance and use of the indicators, stakeholders, including national palliative care organizations, were involved throughout the whole project.
CONCLUSION: Our indicator development trajectory resulted in a set of quality indicators applicable to all patients in all palliative care settings. The set includes patient and relative perspectives and includes outcome, process and structure indicators. Our method can contribute internationally to a more standardized and rigorous approach to developing quality indicators for palliative care.

Background

Quality of care is of interest to everyone receiving or providing palliative care [1]. Evaluation of quality informs care providers, administrators and policymakers about whether patients and their families are receiving care that fits their needs [2-4]. The challenge for quality improvement in palliative care is to develop effective ways for the quick and efficient assessment of service performance and outcomes, as this facilitates the modification of services and practices in order to improve the quality of care at individual and institutional levels [5-8]. For this purpose, quality indicators specifically for palliative care can be used to address issues unique to this type of care [9].

Although in the past decade a variety of studies have focused on quality indicators for palliative care, the methods found in the literature by which indicators were developed were not always clearly presented and not always seemed to follow a systematic approach involving different steps of the development process [10-26]. Two recent systematic efforts have developed a nationwide quality register system to measure outcomes and quality of palliative care services: the Palliative Care Outcomes Collaboration (PCOC)-study in Australia [27], and a similar registration project in Sweden [28]. While these projects have resulted in very useful national quality monitoring systems, they do not strictly make use of quality indicators but mainly focus on outcome measures. Additionally they did not seem to aim for a level of comprehensiveness necessary to evaluate the various dimensions of palliative care.

It appears that there is a need for a more standardized method of developing a comprehensive quality indicator set. Even at a more fundamental level, however, there is also a need for a common understanding of what quality indicators are. Quality indicators are well-defined and measurable aspects of care that give an indication of the quality of care delivered; they are generally expressed in a number or percentage, address a specific aspect of care or a related outcome, and are expressed at an aggregated level, often the level of care organisations [29-32]. They have to be clinically relevant, manageable and based on existing evidence, or consensus in the absence of such evidence [30]. Specific properties have to be described: a numerator (ie for which patients the indicator is positive) and a denominator (the group of patients being evaluated), a threshold value as a performance standard and exclusion criteria. Validated measurement instruments and relevant outcome measures are preferably used [33]. Confusion persists about outcome measures, measurement instruments and quality indicators, with these concepts often used interchangeably and mistakenly. Outcome measures are an essential component of quality, providing a way to evaluate patient- or family-level status and responses to treatment, measured on an individual level [34]. However, as quality indicators, they do not have standardized specifications detailing the eligible population, data collection procedures and types of analysis needed to calculate them [24] (see Table 1).

Table 1

Definitions and examples of outcome measures, quality indicators and quality measurement instruments

	Outcome measure/ variable	Quality indicator	Quality measurement instrument
Definition	An essential component of quality whereby the focus lies on patient’s or relatives outcomes, measured at an individual level.	Well defined and measurable aspect of care, generally expressed in a number or percentage, addressing a specific aspect of care or a related outcome. Quality indicators are usually expressed on aggregated level.	Instruments that can be used to monitor quality of care.
Characteristics	Concerns outcomes of care	Concerns structure, process or outcomes of care	Concerns structure, process or outcomes of care
	In palliative care, outcome measures provide a way to evaluate patient- and family-level status and response to treatment for symptoms and conditions in physical, psychological and other domains	Clinically relevant, manageable and based on existing evidence, or if not applicable on consensus	These instruments are used to measure outcome measures on individual level as well as quality indicators on aggregated level Individual and aggregated level
		Contains standardized specifications detailing the eligible population, data collection procedures and types of analyses needed to calculate the indicator
	Individual level	Aggregated level
Example	Pain intensity	Percentage of patients with moderate to severe pain [10]	Numerated rating scales (NRS)
	Quality of life Comfort	Extent to which patients indicate that caregivers respect their life stance [10]	McGill Quality of Life Questionnaire [44]
	Patient’s appraisal of the quality of care Relative’s appraisal of the quality of care	Extent to which direct relatives felt supported by the caregivers immediately after the patient’s death [10]	VOICES questionnaire [42,43]

Following the model proposed by Donabedian, quality indicators should be monitored in all three major areas of health care: structure, process and outcome [35]. Structure refers to the organization of care (eg presence of a written policy about visiting patients in the ICU), whereas process refers to the interaction between providers and receivers of care (eg psychosocial support being offered within the first 72 hours of admission to the ICU) [36]. Outcomes of care are at the level of the beneficiary of that care (eg adequate treatment of pain) [36].

With good examples of systematic and scientifically rigorous development trajectories for quality indicators for palliative care lacking in the literature, we intend this article to contribute to the establishment of a scientifically rigorous standardized and applicable method of measuring usable quality indicators. We do this by describing the protocol of a study developing a quality indicators set for palliative care for adults in Flanders (the Flemish speaking part of Belgium), applicable in all settings providing palliative care, thus providing an example usable by other countries interested in monitoring the quality of palliative care.

Methods and design

In order to develop a comprehensive set of quality indicators to monitor the quality of palliative care in Flanders, four phases were followed in the development process: 1) identification of existing quality indicators, 2) development of a framework for quality of palliative care, 3) indicator selection by expert consultations and 4) testing the draft quality indicator set in palliative practice (Figure 1). The protocol of the present study was approved by the Ethical Review Board of Brussels University Hospital of the Vrije Universiteit Brussel.

Figure 1

Flowchart of the 4 phases of the Flanders Study to develop Quality Indicators for Palliative Care.

Phase 1: identification of existing quality indicators

To get an overview of previous attempts to develop quality indicators for palliative care, a thorough systematic literature review was performed, building on a review done in 2007 by Pasman et al [36]. With the field of quality indicators and the literature on quality measurements in palliative care developing quickly, it was important to update the existing review using the same method and search strategies but including the most recent literature published between December 2007 and May 2009.

As in the original review, we searched four international databases (Pubmed, PsychINFO, EMBASE and CINAHL). Two independent reviewers (KL and JC) followed the same procedures as in the original review by Pasman et al (results of this review update are integrated into a paper published in Journal of Pain and Symptom Management [37]).

Phase 2: identifying domains and themes of quality palliative care

To achieve eventual comprehensiveness in the set of quality indicators we conceived a conceptual framework based on publications focussing on the determination of high quality palliative care consisting of several domains [38-40]: 1) physical, 2) psychological, social and existential, 3) information, communication, planning and decision making with patients, 4) with family and 5) with other carers, 6) type of care, 7) coordination and continuity, 8) support of friend or family carers and 9) structure of care (Table 2). Within these nine domains, we distinguished several themes (see Table 3 for an example of one domain with different themes).

Table 2

Selected domains for the quality of palliative care in Flanders

	Domain
1.	Physical treatment and care
2.	Psychological, social and existential treatment and care
3.	Information, communication, planning and decision making with the patient
4.	Information, communication, planning and decision making with the family
5.	Information, communication, planning and decision making with other caregivers
6.	Type of care at the end of life
7.	Coordination and continuity of care
8.	Support of family and informal caregivers
9.	Structure of care

Table 3

Example of themes within domains: Domain 1 ‘Physical treatment and care’

Process of care
1.	To measure or evaluate general symptom burden
2.	To measure or evaluate pain
3.	To measure or evaluate nausea and problems with digestion
4.	To measure or evaluate fatigue and insomnia
5.	To measure or evaluate decubitus
6.	To measure or evaluate appetite
7.	To measure or evaluate problems with respiration/tightness of the chest
8.	To measure or evaluate delirium
9.	To measure or evaluate complaints on the mouth
10.	To treat or care for physical symptoms once observed with the patient
11.	Mentioning in the patient’s file the presence of symptoms
12.	Mentioning in the patient’s file the offered treatment and/or medication for the purpose of physical problems
13.	Mentioning in the patient’s file the result of the offered treatment and/or medication for the purpose of physical problems
Outcome of care
1.	Low general symptom burden
2.	Absence of pain
3.	Absence of problems with respiration/ tightness of the chest
4.	Absence of delirium
5.	Absence of decubitus
6.	Absence of fatigue and insomnia
7.	Absence of complaints of the mouth
8.	Absence of nausea and problems with digestion
9.	A good appetite

In order to select the themes most important to quality palliative care we held a consultation of experts representing all relevant actors in palliative care, including patient and family perspectives. The panel consisted of nurses and other caregivers from all different specialized palliative care services, physicians from different specialities, coordinators of the Flemish Palliative Care networks, volunteers, spiritual and religous counsellors, bereaved family members and representatives of palliative patients i.e. large patient organisations such as the Flemish Cancer League and the Flemish Advisory Board for the Elderly.

The panel was asked to score each theme, as identified through the literature, on a scale from 1 (least important) to 5 (most important). Additionally we asked them to select the three most important themes per domain (questionnaires available upon request). Per domain themes were ranked from most important to least important based on the mean scores and the frequency with which the theme was indicated as most important within the domain. Based on the mean scoring (highest tertile across all domains) a number of themes were selected. To be selected, there had to be consensus among the panel members for each indicator, i.e. not more than four members scoring outside the mean range [1-9]. With every domain not then represented, we included one additional theme most often indicated as one of the three most important themes within the domains that were underrepresented. We choose to preserve the nine domains because the research team decided that all domains should be covered in order to have a comprehensive set. Additionally, experts were able to identify additional themes that did not come out of the literature review in a separate box for each domain and these were added if consensus among the experts was reached (i.e. more than one expert added this theme).

Phase 3: indicator selection by expert consultation

The selected themes were then translated into quality indicators (i.e. with standardized specifications detailing the eligible population, data collection procedures, numerator and denominator), using the quality indicators found in the systematic literature review in the first phase. Then a second consecutive multidisciplinary palliative care expert panel was organized for another selection round. This panel, like the first panel, consisted of researchers in quality measurement or in palliative care, nurses and other caregivers in the field of palliative care, caregivers with a policy task in palliative care, bereaved family members and patient representatives. The experts could add quality indicators important to them not presented in the list and could discard those they found less important. For social and existential aspects of care, no indicators were found in the literature so experts were asked to suggest well-defined and measurable quality indicators for further development.

Unlike with the first expert consultation, we put together the second panel following the appropriateness method of the Research ANd Development corporation in collaboration with the University of California at Los Angeles (RAND/UCLA) [41]. This is the only systematic method for selecting quality indicators combining evidence with consensus among stakeholders. We chose to follow this method because, particularly for palliative care, quality indicators have to be developed using other evidence alongside expert opinion [29] as this area of health care has a limited or methodologically weak evidence base. Within this RAND/UCLA-method we executed the five prescribed steps [42]:

1. Comprehensive literature review on the topic (i.e. systematic literature review in phase 1) and recommendation of a preliminary set of quality indicators within the quality framework obtained through the first expert panel.

2. Recruiting expert clinicians from professional organisations reflecting the variety of specialities in palliative care and inviting them to join a panel for a two-stage process to rate the validity of the indicators. These experts included a representative from a patient organisation and a next-of-kin (eg partner, relative or friend most closely involved), as in the first panel round.

3. Sending the draft indicators by post to the panel members, who rated the indicators in terms of appropriateness and relevance as measures of quality on a scale from 1 (not appropriate or not necessary) to 9 (very appropriate or very necessary) (questionnaires available upon request).

4. Feeding back first-round scores to the panellists for a second round of scoring in a face-to-face panel meeting. Only indicators with a median score of 4, 5, 6 or higher but not reaching consensus were discussed, modified where necessary and rescored. The indicators with a median score of 7, 8 or 9 were automatically added to the quality indicator set and modified where necessary by the researchers.

5. Using the second-round scores to select only those indicators rated highly for validity and on which panel members had reached consensus. Indicators rated low or not reaching consensus among panellists were discarded.

After this second face-to-face consultation a draft set of quality indicators was produced (Figure 1).

Phase 4: Developing and testing the draft quality indicator set in palliative practice

The next challenge was to test the draft indicator set in palliative care practice, in particular in order to assess its usability and feasibility (eg work load). Up until now, the draft set had been based on literature and expert opinion and hence needed operationalization and validation in practice, after which final adjustments could be made.

The indicators were transferred into questionnaires for patients, medical staff and next-of-kin (eg partner, relative or friend most closely involved) (Table 4). In order to obtain all information needed to calculate the selected indicators we composed our questionnaires using previously validated questions from existing questionnaires when these existing questions covered the quality indicator well. We for instance used questions of the Views Of Informal Carers – Evaluation of Services (VOICES) Questionnaire [43,44] and the McGill Quality of Life Questionnaire [45], particularly as these questionnaires are frequently used and already validated for a large group of respondents. For those indicators where no validated questions were available we composed questions ourselves. In a pre-test among patients, family members and caregivers, we tested this not yet validated questions and made an estimation of the time needed to complete each questionnaire.When patients were not able to complete the questionnaire themselves, a proxy was assigned.

Table 4

Distribution between observational units and study units during the testing phase

	Observational unit
	Patient	Proxy	Next-of-kin	Caregiver		Care institution
Study unit	before death	before death	after death	before death	after death	before death
Care for the patient	x	x	x
Care for the next of kin			x
Process of care				x	x
Structure of care						x

The questionnaires were divided over the observational units via four different palliative care services (representing all the different palliative care services in Flanders): the palliative home care team, the palliative care unit, the palliative support team in the hospital and the care home (only residents for whom a ‘palliative trajectory’ was initiated). A selection of services was recruited via the existing palliative care networks in Flanders: a call was sent out by the coordinators of these networks and services could volunteer to participate. After recruitment, a contact person for each service was appointed to organize the mailing of the questionnaires together with a researcher from the research team. The intention was to capture a cross section of the quality of palliative care in the different services at a particular moment, so one interrogation only was conducted. This demanded that a random selection was made of patients receiving palliative care services as well of those who had received palliative care services and had died between six weeks and six months earlier; a representative snapshot of the quality of existing care was thus created, taking into account coincidental patient mix confounders. With the quality of care susceptible to change over time, such a snapshot forms a basic starting point. The choice of a single measurement also makes the measurement of quality less burdensome for staff.

Four different types of questionnaire (available upon request), each measuring different quality indicators, were sent:

For adult patients currently receiving palliative care

For next-of-kin involved in the care of a patient who had died six weeks to six months previously

For the central formal care provider (a nurse in the palliative home care team and in the care home and a nurse or a physician in the palliative support team and the palliative care unit) about the care of a patient currently receiving palliative care

For general practitioner of patients who had died in the previous six weeks to six months

A maximum of 10 questionnaires was sent to any one nurse or physician.

We were able to list which patients, care providers and next-of-kin had already participated by the identifier numbers on the returned questionnaires attributed to each selected patient at the time of inclusion. In collaboration with the coordinator of each service involved, reminders were sent: one to patients and next-of-kin and up to two to care providers. As the researchers have not disposed of the list with patient, caregiver and family member contact details, identifying individuals through the identifier numbers on the questionnaires was not possible. After the reminders were sent and the questionnaires returned the collected data was analysed and quality indicators were calculated for each participating service.

The testing phase was then evaluated in terms of feasibility and usability. First telephone interviews with non-responding family members and family physicians were held to assess reasons for non-participation. This was done by a third independent party in order to preserve anonymity of patients and family members from the researchers. The contact person of the palliative service communicated the phone numbers of family and physicians directly to this independent person by email, based on the identifier numbers received by the researchers. Furthermore interviews were conducted with all coordinators and contact persons to evaluate the usability of the indicators and the procedure. Before the start of the interview all contact persons were asked to question the other caregivers about their experiences with the testing and the length of the questionnaire(s) they completed.

Based on the results of the test in terms of usability and feasibility, the indicator set, questionnaires and manual were adapted in order to be ready for use by the palliative services independently. We evaluated the instrument combining qualitative interviews with the caregivers involved about the workload and future use of the set and of quantitative psychometric analyses of the dataset gathered (e.g. number of missing answers per indicator). Furthermore, on request of the field, the indicator set was divided into a required minimal data set with complementary modules, each set usable separately or combined by the palliative services interested in quality measurement and improvement.

Discussion

This article presents a method followed in Flanders (the Flemish speaking part of Belgium) for developing quality indicators for palliative care. The method comprises four phases: 1) a literature review identifying existing quality indicators, 2) a literature review and expert consultation to identify the important themes and domains for quality palliative care, 3) quality indicator selection by expert consultation following the internationally validated RAND method and 4) testing the draft quality indicator set in practice. This four-phase method combines the different phases from previous development processes in other countries (ie literature review, expert consultation, practical test) and additionally includes patient and next-of-kin perspectives, structure, process and outcome indicators and applicability to all patients in all palliative care services.

The method developed here can contribute to the development of a standardized and functional set of quality indicators for palliative care in other countries as it meets the standards of scientific rigour and the level of comprehensiveness required.

Choosing a multi phased method leading to comprehensiveness

An important strength of the quality indicators development method described here is that it contains several phases to make the quality indicator set as comprehensive as possible, i.e. measuring quality of care for all adult palliative patients and across all domains relevant to palliative care. By combining a literature review on quality indicators with the building of a framework on important themes in palliative care, all aspects of palliative care have been taken into account. Previous sets of quality indicators for palliative care have tended to target specific populations such as cancer patients or vulnerable older people [9,16,18,46-48]. Our set, like that developed in the Netherlands is characterized by applicability to all setting and to all adult patient groups receiving palliative care [10] making it possible in the future, after assessing case mix adjusters, to compare scores of quality indicators not only over different disciplines but over different care settings.

Choosing a combination of evidence through literature review and expert consultation

As the thinking about quality indicators and the literature about quality of palliative care is developing quickly, it is necessary to review the field regularly, especially when beginning to develop a new set of quality indicators. In the literature we found researchers developing quality indicators who had not performed such a review but had started from an existing quality indicator set such as Saliba et al. [22] who had selected indicators feasible for nursing home quality assessment from the original ACOVE quality indicator set. This raises the question of whether completely new indicators need to be developed for a particular country or whether existing indicators can be adjusted and validated in a set for that country [37]. Either way, a literature review can help to reveal existing indicators for further development and quality monitoring in any country.

Consultation of experts in palliative care in order to select the most important elements and indicators for measuring quality is a key element in the methodology described here and scoring by experts is important in assessing the validity and usefulness of the proposed indicators. Additionally it has the advantage that consultation of relevant palliative care actors enhances the acceptance and face-validity of the resulting indicator set among the palliative care community [10]. The composition of the panel should reflect all relevant actors within palliative care: those in medical practice and palliative care, policymakers, researchers, patient representatives and family. The latter two categories are often forgotten in quality indicator development but are essential to the creation of a comprehensive set of quality indicators reflecting all perspectives in palliative practice. The expert consultation can best be done by the RAND/UCLA-method [42].

Choosing inclusion of outcome, process and structure indicators and involvement from all perspectives

Although measuring structure and process indicators from medical charts or administrative databases may be easier and cheaper, using only such sources for quality measurements may cause problems. Such databases can be limited, especially where indicators depend on precise documentation of issues such as communication, patient-reported outcomes or preferences [9]; such information can only be obtained from patients and their families themselves. On the other hand some researchers and care providers tend to see quality only through the perspective of the patient. This perspective does indeed provide the best indication of whether good quality care has been achieved [10], but data for calculating have to be derived from individual care users which, apart from the practical issues, also limits the possibility of measuring all relevant aspects of the quality of palliative care which might be susceptible to improvement. Therefore many authors [9,29,35,38,39,49] suggest that a quality indicator set, in order to be broad and comprehensive, must include structure and process as well as outcome indicators. We chose to translate our draft indicator set into questionnaires for the different respondent groups, patients, family and care providers, depending on which we considered would be the most reliable raters for selected quality indicators. Other quality researchers have based their evaluation of usability and feasibility on abstraction from patient records [11,14,23,25], thereby automatically limiting the comprehensiveness of an indicator set.

Choosing a manageable testing of the quality indicator set

A crucial stage in the development process is the testing and evaluation of the set for usability and feasibility (eg work load) in practice; before it can be taken up on a nationwide scale the set needs operationalization and validation in palliative practice. Because of its extensive character, we felt it necessary to perform this testing phase in a small sample of palliative care service organisations.

To stimulate acceptance and use of quality indicators, stakeholders from palliative practice should be involved as early as possible in the development process. In Flanders we also collaborated closely with the Flemish Federation for Palliative Care, which helped to promote the project and enhance participation from care providers.

Work in progress

The quality indicator set for palliative care for adults in Flanders is now fully developed and ready to be used in practice to monitor quality of palliative care. Before scores of quality indicators can be validly compared across different services and care providers, however, significant case mix adjusters and the discriminative power of the indicators need to be assessed. Only if future research in a larger sample confirms the discriminative power of the indicators can they be recommended for comparative information or benchmarking [10] while benchmarking itself can only be made possible by implementing the quality indicator set on a large scale in Flanders.

A standardized method for the development of quality indicators for palliative care can also offer opportunities on the international level; providing internationally comparable data [50].

Conclusion

There is an increasing recognition of the need for quality indicators for palliative care in order to develop quality programmes across countries and to provide evidence to policymakers about the quality of the palliative care they are providing. However, a standardized method of developing quality indicators for palliative care has been lacking. In this paper we propose a method which meets the required level of scientific rigour and creates a sound basis for achieving the comprehensiveness needed in a set of quality indicators. The method suggested here combines standard literature analyses with multidisciplinary expert consultations involving all relevant actors within palliative care, following the RAND/UCLA-method, the only validated systematic method for quality indicator selection. Those wanting to develop indicators in other countries can use this standardized method to further develop and validate quality indicators for palliative care.

Competing interests

The authors state that there is no conflict of interest.

Author’s contributions

KL performed the study, participated in its design and drafted the manuscript. JC conceived of the study, participated in its design and coordination and helped to draft the manuscript. ALF and RVS participated in the design of the study and revised the article critically for important intellectual content. SJJC revised the article critically for important intellectual content. LVDB participated in the design of the study and revised the article critically for important intellectual content. LD conceived of the study, participated in its design and coordination and revised the article critically for important intellectual content. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-684X/12/6/prepub