The effect and process evaluations of the national quality improvement programme for palliative care: the study protocol.

BACKGROUND: The nationwide integration of palliative care best practices into general care settings is challenging but important in improving the quality of palliative care. This is why the Dutch National Quality Improvement Programme for Palliative Care has recently been launched. This four-year programme consists of about 70 implementation trajectories of best practices. A large evaluation study has been set up to evaluate this national programme and separate implementation trajectories. METHODS/
DESIGN: This paper presents the protocol of the evaluation study consisting of a quantitative effect evaluation and a qualitative process evaluation. The effect evaluation has a pre-test post-test design, with measurements before implementation (month 0) and after implementation (month 9) of a best practice. Patients are eligible if they have a life expectancy of less than six months and/or if they are undergoing palliative treatment and provided they are physically and mentally capable of responding to questionnaires. Bereaved relatives are eligible if they have been involved in the care of a deceased patient who died after a sickbed between six weeks and six months ago. Three types of measurement instruments are used: (1) numerical rating scales for six symptoms (pain, fatigue, breathlessness, obstipation, sadness and anxiety), (2) the Consumer Quality Index Palliative Care - patient version and (3) the version for bereaved relatives.The process evaluation consists of analysing implementation plans and reports of the implementation, and individual and group interviews with healthcare professionals. This will be done nine to eleven months after the start of the implementation of a best practice. DISCUSSION: This mixed-method evaluation study gives more insight into the effects of the total programme and the separate implementation trajectories. However, evaluation of large quality improvement programmes is complicated due to changing, non-controlled environments. Therefore, it is important that an effect evaluation is combined with a process evaluation. TRIAL REGISTRATION: NTR-4085.

Background

Death comes to us all. Nowadays, acute deaths due to infectious diseases have largely been replaced by non-sudden deaths, caused by e.g. cancer, cardiovascular diseases and dementia [1]. Therefore, the need for palliative care is increasing. High-quality palliative care is important in improving the quality of life of patients and their families facing the problems associated with a life-threatening disease [2]. Since the 1990s, the Dutch government has consistently invested in the optimization of palliative care, and nowadays many best practices are available in the Netherlands. 'Best practices' are defined in this paper as practices, often developed among small groups of patients and professionals, that contribute to the improvement of palliative care and that are transferrable to other settings. Some examples of best practices in the Netherlands are:

The Dutch version of the Liverpool Care Pathway for the Dying Patient [3]. The pathway was developed to aid members of a multi-disciplinary team in matters relating to continuing or discontinuation of treatments and regarding comfort measures during the last days and hours of a patient's life.

PaTz – a systematic approach to optimize palliative care based on the Gold Standard Framework (http://www.goldstandardsframework.org.uk/). Its elements include collaborative meetings of general practitioners and community nurses, the timely identification of patients in need of palliative care and the drafting of an advance care plan [4].

Informare, a tailored method that enables health care professionals to provide timely information about palliative care to patients and relatives [4].

‘Signal box for nursing assistants’, a tool that enables nursing assistants to identify palliative care needs in their patients and pass this information on to other professionals [4].

These best practices are known for their contribution to high-quality palliative care, although so far most of the knowledge about the effects of these best practices on the quality of palliative care is practice-based rather than evidence-based. At the moment, most evidence is available with regard to the Dutch version of the Liverpool Care Pathway (LCP) for the Dying Patient and regarding PaTz. Research has indicated that use of the Dutch version of the LCP leads to a decrease in the symptom burden for patients in the last days of life, a decrease in the grief burden for relatives and an improvement in the documentation of care [5-7]. Furthermore, a recent review showed that the Gold Standard Framework (the British equivalent of PaTz) improves processes in general practice, co-working and the quality of palliative care. Although the direct impact on patients and carers is unknown at present, the Gold Standard Framework has considerable potential to improve palliative primary care [8].

Although large-scale implementation of such best practices appears to be important for improving palliative care, implementation in general health care settings (e.g. general hospitals, general practices, residential elderly care and home care) is challenging, e.g. because professionals in general health care settings are often not specialized in palliative care and often provide care to a variety of patient categories. Moreover, hospital settings in particular are traditionally strongly focused on cure and life prolongation, which influences how palliative care needs are attended to [9].

National quality improvement programme for palliative care

The Dutch National Quality Improvement Programme for Palliative Care was launched in 2012 with the aim of promoting the implementation of best practices in general health care settings. This four-year programme receives financial and practical support from ZonMw (The Netherlands Organisation for Health Research and Development) and was commissioned by the Dutch Ministry of Health. The programme’s key objectives are to promote that:

•Patients die at their preferred place.

•Patients and relatives feel they are in control regarding palliative care.

•Patients and relatives see palliative care as being coordinated.

•Patients and relatives feel care to be concordant with their needs, preferences and values.

•Patients and relatives receive care for their needs in the physical, psychosocial and spiritual domains.

This programme enables care organizations to improve their palliative care by implementing one or more of the available best practices. The best practices are pre-selected by an independent committee of experts, professionals and patient representatives, taking account of the available evidence, the usability and the transferability of the best practice to general health care settings. The committee assesses potential new best practices every year to include in the National Quality Improvement Program Palliative Care. The nine best practices that are currently pre-selected for implementation are listed in Additional file 1.

Every year during the programme period (2012–2016), there is a call for applications. Representatives of regional networks of palliative care providers are invited to choose a specific best practice from the pre-selected list and to submit a project proposal for an implementation trajectory. Each year, financial support is given by ZonMw for approximately 17 implementation trajectories. The grant applications are assessed by an independent quality working group. The criteria for granting this support include the requirement that the applicant must be a representative of the regional palliative care networks, the care organizations involved must reach a substantial number of patients with palliative care needs and the best practice chosen must be implemented within one year [10].

It is important to determine the effects of the National Quality Improvement Programme for Palliative Care, since a lot of professionals and financial resources are involved and ultimately patients and relatives must benefit from this great effort. Other large-scale improvement programmes have already been implemented and evaluated in other domains, e.g. regarding safety in health care and social care [11-15]. However, so far such large-scale programmes are a new feature within palliative care. Earlier evaluations in other health care domains not only focused on the effectiveness of the programmes but also described the implementation processes in order to enable an understanding of how the programmes take shape in practice, as well as of barriers and facilitators. Such a process evaluation can be referred to as opening the ‘black box’ of interventions [16], which is important in increasing the likelihood that interventions will continue to be used in practice after the research has finished. In palliative care too, it is important to bridge the gap between research and practice [17,18] and to maximize the likelihood of the long-term adoption of quality improvement projects in practice [19]. Therefore, it is important to describe both the effects and the process of this large-scale implementation of palliative care best practices.

Objectives

This paper presents the study protocol of the evaluation study. The primary objective of this study is:

•To gain insight into the effects of the National Quality Improvement Programme for Palliative Care as a whole on the quality of palliative care at a national level.

Its secondary objectives are:

•To gain insight into the effects of the separate implementation trajectories of best practices on the quality of palliative care within the participating regional palliative care networks or organizations.

•To elucidate the measured effects by describing the implementation process and the barriers and facilitators within the separate implementation trajectories.

The specific research questions being addressed in the evaluation study are as follows:

1. What are the effects of the National Quality Improvement Programme for Palliative Care as a whole on the quality of palliative care? More specifically, what are the programme’s effects on:

a. The percentage of patients who die in their preferred place?

b. The extent to which patients and relatives feel in control regarding palliative care?

c. The extent to which patients and relatives experience well-coordinated palliative care?

d. The extent to which patients and relatives feel that the palliative care is concordant with their preferences and values and meets their needs in the physical, psychosocial and spiritual domains?

2. What are the effects of the separate implementation trajectories on the quality of palliative care at the level of the regional palliative care networks or care organizations involved? (with equivalent subquestions to questions 1a to 1d)

3. How and to what extent are the best practices implemented? More specifically:

a. Does the implementation of the best practices take place as planned?

b. What barriers and facilitators are there for the implementation of the best practices and what conditions have been created to ensure the best practices are maintained?

c. Are there any other factors or developments – inside or outside the participating organizations – that might have contributed to the measured effects of the programme as a whole or within the separate implementation trajectories?

Methods

Evaluation study design

The National Quality Improvement Programme can be considered as a complex intervention involving various implementation trajectories of various best practices within a continuously changing, uncontrolled environment. Therefore we opted for a quantitative effect evaluation with a pre-test post-test design (to answer research questions 1 and 2) in combination with a qualitative process evaluation with a post-test design (to answer research question 3). The combination of quantitative and qualitative methods is considered useful in gaining insight into the effects of the programme, as well as the implementation processes, barriers and facilitators for the implementation and internal or external factors that may influence the outcomes.

Study population

All implementation trajectories receiving a grant in the National Quality Improvement Programme will participate in the evaluation study. Within the participating organizations, data will be collected from patients and bereaved relatives (see Figure 1). In implementation trajectories with more than five participating organizations, a purposive sample will be selected of care organizations with the highest number of patients meeting the inclusion criteria and representing all the different settings involved in the specific implementation trajectory.

Figure 1

Organogram of the national quality improvement program palliative care.

An existent set of quality indicators measured among patients and bereaved relatives will be used for the effect evaluation [20]. The inclusion and exclusion criteria for patients and bereaved relatives are described in Additional files 2 and 3. Per trajectory we aim to include 15–20 patients and 30–40 bereaved relatives in each assessment period. As there are two assessment periods for the approximately 17 planned trajectories per year, this will result into a minimum of about 510 participating patients and 1020 participating bereaved relatives per year, which gives a total number of about 2040 patients and 4080 bereaved relatives over four years.

For the process evaluation, data will be collected among professionals. For each implementation trajectory, the project manager will be interviewed and a group interview will be done with other stakeholders involved. Assuming 17 implementation trajectories per year over four years, this will result in 68 interviews with project managers and 68 group interviews with stakeholders.

Effect evaluation

Pre-test and post-test measurements will take place in the effect evaluation, in month 0 and month 9 respectively (see Figure 2). During these two measurement months, eligible patients and bereaved relatives will be asked to participate.

Figure 2

Chronological overview of one year of the program.

Data collection

In the pre-test and post-test measurements, a set of quality indicators will be measured [20], making use of three types of measurement instruments:

•Numeric Rating Scales of six symptoms

•The Consumer Quality Index Palliative Care for patients

•The Consumer Quality Index Palliative Care for bereaved relatives

Numeric Rating Scales (NRS) will be used to measure the quality indicators ‘The % of patients with moderate to severe pain’, ‘The % of patients with moderate to severe fatigue’, ‘The % of patients with moderate to severe shortness of breath’, ‘The % of patients with moderate to severe constipation’, ‘The % of patients with moderate to severe anxiety’ and ‘The % of patients who feel moderately to severely depressed’. These indicators are associated with one specific programme objective, namely ‘Patients receive care for their needs in the physical and psychosocial domains’.

Patients will be reminded by one of their health care professionals to complete the NRS for the six symptoms (see Table 1) approximately the same time at three consecutive days. In principle, the patient will fill in the NRS scores. Only if the patient is not able to write will the health care professional assist the patient by writing down the NRS scores. The completed NRS forms will be collected by the health care professional after day 3 and send to the research team at the end of the measurement period.

Table 1

Numeric rating scales of 6 symptoms

Symptoms (What number would you give your …symptom… on a scale of 0 to 10?
No pain	0	1	2	3	4	5	6	7	8	9	10	Much pain
No fatigue												Very fatigue
Not short of breath												Very short of breath
Not obstipated												Very obstipated
Not sad												Very sad
Not anxious												Very anxious

The Consumer Quality Index Palliative Care (CQ Index) is a validated questionnaire assessing quality indicators from the perspective of patients and bereaved relatives [21,22]. The quality indicators measured by this CQ Index are also directly related to the key objectives of the National Quality Improvement Programme (see Table 2). CQ index questions for instance concern patients’ or bereaved relatives’ experiences regarding the control over palliative care, the coordination of palliative care, the concordance between the care given and needs, preferences and values, and the physical, psychosocial and spiritual care received. In the version for bereaved relatives, questions are also asked about the actual and preferred place of death of the patient. The questionnaire version for patients consists of 56 items, while the version for bereaved relatives consists of 64 questions. Both questionnaire versions take approximately 30–45 minutes to complete. For inpatient patients, the questionnaire will be completed on the basis of an interview with a trained interviewer. Patients who receive care at home and bereaved relatives will receive the questionnaire by post to be completed and returned in a reply envelope to the research team. No reminders will be sent to non-responding patients, while a reminder will be sent after two weeks to non-responding bereaved relatives.

Table 2

Key objectives of the national quality improvement program and associated quality indicators, measured with the CQ index palliative care

Key objectives of the programme	Quality indicators measured with CQ index version for patients	Quality indicators measured with CQ index version for bereaved relatives
Patients die at their preferred place	-	*Percentage of patients who died in the location of their preference
		*Extent to which patients in the last month before their death were in the location of their preference
Patients and relatives feel they are in control regarding palliative care	*Extent to which patients experience respect for their autonomy	*Extent to which relatives indicate that the patient was asked about her/his opinions with regard to end-of-life decisions
	*Extent to which patients receive information from their caregivers about the expected course of the disease	*Extent to which direct relatives received information about the advantages and disadvantages of various types of treatment
	*Extent to which patients receive information about the advantages and disadvantages of various types of treatments	*Extent to which direct relatives received information that was understandable and unambiguous at the time of the patient’s death
	*Extent to which patients indicate that they receive understandable explanations	*Extent to which, according to the bereaved relatives, their autonomy was respected
		*Extent to which direct relatives were informed about the possibilities of aftercare
Patients and relatives see palliative care as being coordinated	*Extent to which patients know who the contact person is for the care	*Extent to which the bereaved relatives knew who the contact person was for the care
	*Extent to which patients experience expertise of caregivers and continuity of care	*Extent to which direct relatives perceived expertise of caregivers and continuity of care
	*Extent to which patients receive contradictory information
Patients and relatives feel care to be concordant with their needs, preferences and values	*Extent to which patients are satisfied with “politeness” and “being taken seriously” by caregivers	*Extent to which the direct relatives felt that they were treated well in all respects by the caregivers
	*Extent to which patients experience respect for their privacy	*Extent to which direct relatives considered that the patient had the opportunity to be alone
	*Extent to which patients indicate that caregivers respect their life stance	*Extent to which the direct relatives had the opportunity to be alone with their relative
	*Percentage of patients who receive medical aids soon enough
Patients and relatives receive care for their needs in the physical, psychosocial, and spiritual domains	*Extent to which patients receive support for their physical symptoms (pain, fatigue, shortness of breath and constipation)	*Extent to which relatives indicate that the patient had access to a counselor for spiritual problems
	*Extent to which patients receive help with physical care	*Extent to which relatives indicate that the patient received support with preparations for saying goodbye
	*Extent to which patients receive attention from their caregivers	*Extent to which relatives indicate that there was attention for the psychosocial and spiritual well-being of the patient
	*Extent to which patients receive support when they feel anxious or feel depressed	*Extent to which, according to the direct relatives, attention was paid to their own psychosocial and spiritual well-being
	*Extent to which patients indicate that they have access to a counsellor for spiritual problems	*Extent to which direct relatives felt supported by the caregivers immediately after the patient’s death
		*Extent to which a final conversation or discussion was held to evaluate the care and treatment

Data analysis

Descriptive statistics (frequencies, percentages and means) will be used to analyse background characteristics.

The effects of the separate implementation trajectories (research question 2) will be analysed using descriptive analyses as well as univariate analyses (with paired t tests or McNemar’s test). To assess the differences between pre-test and post-test measurements for the whole programme (research question 1), multilevel analysis will be used, including the levels of the participating care networks or care organizations, and the type of best practice. SPSS, STATA and ML-WIN will be used for the statistical analysis.

Process evaluation

A qualitative process evaluation will be conducted nine to eleven months after the start of the implementation in a separate implementation trajectory (see Figure 2). The process evaluation will be used to explain the effects found (or not found) in the effect evaluation and provide insight into how the best practices are implemented.

First, an analysis of relevant documents (e.g. implementation plans, and interim and final reports of the separate implementation trajectories) will provide insight into the implementation process. Second, we will conduct individual semi-structured interviews with the project managers of each trajectory. The topic list for the interviews will include questions on whether the best practice was implemented as planned, what the barriers and facilitators were for the implementation, what implementation strategies were used and whether the project managers felt implementation has been successful and effective. Third, for each implementation trajectory we will organize semi-structured group interviews with relevant other stakeholders (such as representatives of clients’ participation councils, nurses, doctors or other care professionals involved). A similar topic list as described for the individual interviews will be used for the group interviews. Both individual and group interviews will be audio-recorded.

The audio-recorded interviews will be transcribed verbatim. Subsequently a qualitative thematic analysis will be done, starting immediately after conducting and transcribing the individual and group interviews for a specific implementation trajectory. All relevant documents and interviews will be coded, analysed and discussed by at least two researchers. The coding process will be inductive, in the sense that themes and associated codes will emerge from the interview data through repeated comparison of relevant fragments within and between interviews. The coding process will be supported by the software package MAXQDA [23].

Feedback and dissemination

The project manager and other stakeholders in a specific implementation trajectory will receive internal feedback reports on the results of the effect evaluation (after the pre-test measurements as well as after the post-test measurements) within two months after the measurement period. The prospect of getting feedback reports is important in order to increase the commitment of the project managers and stakeholders to participating in the measurements. In addition, the feedback reports will motivate the professionals involved to further improve the quality of their care, where needed.

Furthermore, interim reports about the progress of the programme as a whole will be published; these will also be made available to the general public. Also, additional international scientific publications about the effects and implementation of the programme are planned.

Ethical considerations

The study protocol has been approved by the Medical Ethical Committee of the University Medical Center in Leiden, the Netherlands. Written informed consent of all patients will be obtained for the assessment of their symptoms and inpatients also consented with the Consumer Quality Index Palliative Care questionnaire, as it has to be completed by a trained interviewer. Patients who are at home and bereaved relatives received this questionnaire by post and for this no written informed consent is needed. Furthermore, in data collection and analyses procedures we follow the rules of Dutch Personal Data Protection Act (Wet Bescherming Persoonsgegevens). All personal identifiers will be removed or disguised in the analysis process to safeguard the privacy and anonymity of participants.

Discussion

This evaluation study provides a unique chance to gain insight into the effects and implementation process of the National Quality Improvement Programme for Palliative Care and its separate implementation trajectories. The implementation of best practices within the framework of a long-term, broad nationwide programme can be considered as a complex intervention within a continuously changing, uncontrolled environment. Complex interventions are described by Pawson et al. as long journeys, embedded in multiple social systems, which are open and adaptive [24]. These characteristics also hold for the National Quality Improvement Programme for Palliative Care, which consists of the implementation of multiple best practices in different settings and with different starting points over a four-year period. Our programme is also relatively open and adaptive, for example as a result of the feedback reports and interim reporting. Feedback reports give the organizations insight into the current state of the quality of palliative care within their organization and might lead to additional actions to improve palliative care.

Randomised controlled designs (RCT) are often not applicable and feasible for the evaluation of complex interventions such as national quality improvement programmes [25,26]. When an RCT is not possible - as is the case in the evaluation of the National Quality Improvement Programme for Palliative Care - it is important to use a mixed-method design and to combine quantitative effect measurements with a qualitative process evaluation. Qualitative process evaluations, e.g. based on the qualitative analysis of interviews and documents, can help understand the effects (or sometimes the lack of effects) of interventions. Process evaluations are helpful in elucidating the implementation process and the interventions actually delivered (opening the ‘black box’) and in identifying other possible factors that may have contributed to the measured outcomes. One such factor may be an ‘investigation effect’ caused by the feedback reports. Therefore, during the process evaluation we will pay extra attention to possible additional actions of care organizations prompted by these feedback reports.

Evaluating complex interventions among patients with a limited life expectancy and among bereaved relatives is particularly challenging for researchers. In such populations, researchers have to take extra account of the burden of measurement for respondents, although many of them are willing to contribute to scientific research [27,28]. An additional challenge in our study in particular is the fact that palliative care patients are often 'scattered' throughout the organization or various teams or units. Due to the aim of the National Quality Improvement Programme for Palliative Care, i.e. to improve palliative care in general health care settings (like hospitals or home care), groups of patients with palliative care needs are less concentrated than in research within hospices, for instance, or specialist palliative care units. This means that an extra effort has to be made for the recruitment of patients and relatives as well as for informing all the professionals involved.

Abbreviations

CQ Index: Consumer quality index; SPSS: Statistical package for the social sciences; ML-WIN: A software package for fitting multilevel models; MAXQDA: A software package for supporting qualitative data analysis.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

NR, JH and AF conducted the study and drafted the manuscript. EN and LD participated in the design of the study and gave comments on drafts of the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-684X/13/5/prepub

Additional file 1

Currently selected ‘best practices’.

Click here for file

Additional file 2

Inclusion and exclusion criteria for patients.

Click here for file

Additional file 3

Inclusion and exclusion criteria for bereaved relatives.

Click here for file