A multi-stage process to develop quality indicators for community-based palliative care using interRAI data.

BACKGROUND: Individuals receiving palliative care (PC) are generally thought to prefer to receive care and die in their homes, yet little research has assessed the quality of home- and community-based PC. This project developed a set of valid and reliable quality indicators (QIs) that can be generated using data that are already gathered with interRAI assessments-an internationally validated set of tools commonly used in North America for home care clients. The QIs can serve as decision-support measures to assist providers and decision makers in delivering optimal care to individuals and their families.
METHODS: The development efforts took part in multiple stages, between 2017-2021, including a workshop with clinicians and decision-makers working in PC, qualitative interviews with individuals receiving PC, families and decision makers and a modified Delphi panel, based on the RAND/ULCA appropriateness method.
RESULTS: Based on the workshop results, and qualitative interviews, a set of 27 candidate QIs were defined. They capture issues such as caregiver burden, pain, breathlessness, falls, constipation, nausea/vomiting and loneliness. These QIs were further evaluated by clinicians/decision makers working in PC, through the modified Delphi panel, and five were removed from further consideration, resulting in 22 QIs.
CONCLUSIONS: Through in-depth and multiple-stakeholder consultations we developed a set of QIs generated with data already collected with interRAI assessments. These indicators provide a feasible basis for quality benchmarking and improvement systems for care providers aiming to optimize PC to individuals and their families.

Introduction

The goal of palliative care (PC) is to improve quality of life for individuals and their families facing the problems associated with a life-limiting illness, and to provide care that promotes dignity, respect, and comfort [1, 2]. Palliative and end-of-life care spans the disease process from early diagnosis to end-of-life, inclusive of bereavement. A majority of Canadians surveyed in 2016 supported resources being available for PC at home [3]. Recent data from Ontario—one of the sites of this study and the largest province in Canada with more than 14 million residents–found that among the approximately 50% of decedents who received PC, 43% (roughly 54,000 over the course of one year), had palliative home care services [4]. Despite this, little research has assessed the quality of home-based palliative services [5-7], instead focusing on utilization, organization, and cost-effectiveness of these services [8, 9]. Existing QIs in the literature tend to focus on hospital use [5, 10–12]. Currently, Canada does not have a standard set of quality indicators (QIs) for community-based PC. Having valid and reliable QIs, grounded in a community-based perspective, is essential to identify and monitor areas for improvement [13], and ultimately, contribute to the delivery of optimal care to individuals receiving PC. QIs have enormous potential for improving care by providing important information to a variety of users, such as care providers, consumers, accreditation organizations and researchers [14].

QI rates are often used to compare providers, at a systems level, a process known as benchmarking [15]. Benchmarking can also be thought of as a starting point to understand which factors contribute to quality improvement, to promote discussions among health care providers to encourage organizational change within the organizations being compared, and to learn from each other’s improvement strategies [16].

Although QIs have been proposed for those receiving PC, or individuals with life-limiting illnesses [17-23], they have several limitations. The majority of existing QIs tend to rely on administrative data to capture hospital admissions/emergency department (ED) visits and focus on the process or structure of care rather than on outcomes that matter most to individuals receiving PC [20, 24–26]. They also tend to be focused predominantly on individuals with cancer [17, 27, 28], despite the fact that cancer is the cause of death of less a third of decedents, and despite PC being increasingly available to those with other life-limiting conditions such as organ failure and dementia [29]. Therefore, QIs are needed that encompass a wide range of individuals who are receiving PC or who could benefit from such care, including those for whom prognosis may be less predictable [30, 31].

We outline the development of a set of proposed QIs that were explicitly defined based on data elements within the interRAI assessments. interRAI is a not-for-profit network of researchers, clinicians, and policy experts from just over 30 countries who develop and test standardized, clinical assessments for use in a variety of health and social services settings (e.g., home care, nursing homes/long-term care, inpatient mental health) [32-34]. These assessments are extensively used in multiple parts of Canada, and around the world, and the data are available to interRAI researchers and their trainees. For example, the interRAI Home Care assessment (interRAI HC) is currently used in 21 countries [35, 36] and for all long-term home care clients in Ontario. The assessments are performed by trained assessors (typically registered nurses) using all sources of information, including the person receiving care, informal caregivers (natural support persons) and families, clinical care providers and the medical record. The assessment is completed in Ontario for all home care clients who require home care services for a minimum of 60 days.

Utilizing the existing, routinely-collected, and population-based interRAI data to generate QIs is a very efficient use of this information and avoids the tremendous time and energy required using traditional approaches to quality improvement, such as using detailed chart reviews [37]. In fact, the interRAI data represents one of the only databases in Canada with a sufficient number of assessments and unique individuals to allow for the creation and testing of QIs. There is also a long history of QI development and testing by interRAI researchers [14, 38–40], with the earliest QIs proposed in the mid-1990s [41]. Earlier work by our team, using the interRAI assessment for home care, identified a set of preliminary QIs for community-based PC that could be generated with these assessments [42], and also explored the rates of these preliminary QIs by province/territory [43].

The main goal of this paper is to report on the first steps, of a larger study, to develop, test and validate a set of QIs for community-based PC. In this paper, we describe the creation of the QIs that were developed through in-depth consultations with multiple stakeholder groups and evaluation by a panel of experts.

Materials and methods

QI development took place over approximately four years (2017–2021) and is still ongoing (Fig 1). The research team included 18 researchers, clinicians or decisions makers from Canada, the US and Belgium with expertise in palliative medicine, health services research, epidemiology, and knowledge translation. The team was actively involved in all stages of the development of the QIs, which included three sequential phases. The study protocol was reviewed and approved by the Wilfrid Laurier University Research Ethics Board (REB #5683).

Fig 1

Summary of the steps in the QI development process.

Phase I: Qualitative input from key stakeholders

The first activity, in this phase, involved input from decision makers from across Ontario, Canada’s largest province. A group of 30 PC experts participated in a one-day workshop, the results of which have been published previously [44]. Participants included clinical leaders, researchers, front-line care providers as well as health and information system administrators. At the time of the workshop, the province was divided geographically into 14 Local Health Integration Networks (LHINs) and 12 of the 14 LHINs were represented. Other participants represented key organizations such as Hospice Palliative Care Ontario, the Ontario Ministry of Health and Long-Term Care, Ontario Palliative Care Network and the Canadian Institute for Health Information.

In summary, the qualitative analysis of the information provided by the workshop participants identified six key themes important for measuring quality in community-based PC: access to care, patient care, caregiver support, symptom management, spiritual care, and home as the preferred place of death. Where possible, QIs were created that directly related to these six themes (as described below in “Phase II”). For example, since participants discussed symptom management as an important area to assess, we created QIs to measure pain, shortness of breath, fatigue, and other troublesome physical symptoms. The indicator concepts, structure, and definitions were derived from the validated interRAI data elements.

Following the workshop, project Knowledge Users (KUs) helped the research team to recruit individuals receiving PC, their family members and decision makers from across Canada. It was considered inappropriate, and potentially a violation of research ethics, for the research team to directly contact individuals receiving PC and their families. As such, the KUs assisted the team in recruiting potential study participants. KUs represent individuals who are likely to be able to use research results to make informed decisions about health policies, programs and/or practices. The eight KUs on our team included individuals from five different provinces/territories. A series of interviews and focus groups were held with families (n = 9) and decision makers (n = 11), including one individual who was actively receiving PC. The primary goal of these interviews and focus groups was to understand participants’ experiences within the health care system and associate these experiences to measurable QIs that could be developed with existing interRAI data. For example, it was apparent from the interviews that caregivers often felt overwhelmed in their caregiving role and expressed that there was a clear disconnect between what the system could provide and what caregivers expected from the system. Palliative care typically strives to provide access to on-call practitioners during and after office hours. Despite this, some care caregivers felt that during a crisis, they had to rely on ambulances and the use of the emergency department (ED). Caregivers cited emotional and psychological needs as well as loneliness among those receiving PC [45]. This rich qualitative data was used to guide QI development in the next phase of the project.

Phase II: Defining potential QIs

The research team used the feedback from Phase I, along with current literature [46, 47], to define potential QIs that captured the important domains related to PC quality. For example, based on the workshop results, a QI was drafted to capture the rate of caregiver distress, and other QIs were created to capture issues related to symptom management (e.g., QIs related to pain, breathlessness, falls, constipation, nausea/vomiting). Since the family caregivers and decision makers also discussed issues of accessing the ED, QIs were also developed related to ED visits and hospital admissions. QIs were defined to capture issues such as negative mood, anxiety and loneliness since these were discussed by caregivers and are supported as important for quality assessment in the literature [48, 49].

To generate these QIs, we focused on two interRAI assessment systems, namely the interRAI Home Care instrument (interRAI HC) and the interRAI Palliative Care (interRAI PC) tool, since these are currently used in care planning for home care clients, and palliative clients, in various parts of Canada. The interRAI HC, for example, is completed fully across Ontario, Newfoundland and Labrador and Yukon Territory, with partial coverage in British Columbia and Alberta [50], resulting in roughly 250,000 assessments annually. Research on the interRAI instruments supports the validity and reliability of these data and concludes that the overall quality can be trusted when used to support decision-making [51].

Both of these assessments provide several of the same validated health index scales, which are generated automatically once the assessment is completed. These scales include the Depression Rating Scale (DRS) [52], the Pain Scale [53], the Cognitive Performance Scale [54], the Caregiver Risk Evaluation [55], and the Pressure Ulcer Risk Scale [56]. Since these scales have confirmed validation, the scores on these scales were used in the QI definitions when possible. For example, for the QI on the prevalence of negative mood, the QI definition uses the DRS score of 3+, a cut-point with established predictive validity [52]. Other QIs are based on individual assessment items. Each QI has a distinct numerator and denominator (S1 Table). The QIs fall into two broad categories, namely, “follow-up prevalence” QIs and “failure to improve” QIs (Table 1). The first group captures the prevalence of the issue on re-assessment. For these QIs, the rate was based on re-assessments and admission assessments were excluded from the calculation. This is necessary since admission assessments would not truly reflect quality of PC at the time of the assessment. The “failure to improve” QIs assess the lack of improvement on the issue over two points in time. This is important to capture since individuals who come into contact with PC generally have an indication bias of high health needs. While the baseline function of these individuals is important to reflect those needs, future change in those needs (i.e., as addressed by PC) is also important to measure. In total, 27 preliminary QIs were created for further evaluation in Phase III. Of this list, 20 (74.1%) can be generated with both interRAI instruments, five (18.5%) others can be generated only with the interRAI PC data, and the remaining two QIs, can only be calculated with the interRAI HC data.

Table 1

List of the 27 preliminary QIs reviewed by the expert panel, how they relate to the 6 themes identified in Phase I and which interRAI assessment can be used to generate the QI.

Name and brief description of each theme and QIs that relate to that theme	Failure to Improve	Follow-up prevalence
1. Access to care: coordination/continuity of care, access to care providers, access to services at the appropriate time
Prevalence of emergency department visitsa,b		X
Prevalence of hospital admissionsa,b		X
2. Patient care: Discussion of preferred setting of death across the illness trajectory, advanced goals/care planning
Prevalence of clients feeling that progress is not being made regarding completion of personal goalsb		X
Prevalence of no advance directivesb		X
Prevalence of clients feeling a lack of completion of financial, legal and other formal responsibilitiesb		X
3. Caregiver support: How to cope with distress/burden/loneliness, education for caregivers/knowledge to keep client at home, caregiver supports (networks, respite)
Prevalence of caregiver distressa,b		X
4. Symptom management: Treating symptoms and also having a patient-centred approach to care
Prevalence of fallsa,b		X
Prevalence of severe or excruciating daily paina,b		X
Prevalence of pain that is not controlled by medicationsa,b		X
Failure for pain to improvea,b	X
Prevalence of constipationa,b		X
Prevalence of shortness of breath at resta,b		X
Prevalence of shortness of breath upon exertiona,b		X
Failure for shortness of breath to improvea,b	X
Prevalence of stasis/pressure ulcersa,b		X
Prevalence of a delirium-like syndromea,b		X
Prevalence of nausea or vomitinga,b		X
Prevalence of fatiguea,b		X
Prevalence of sleep problemsa,b		X
Prevalence of poor self-reported healtha,b		X
Prevalence of negative mooda,b		X
Failure for negative mood to improvea,b	X
Prevalence of declining social activities that causes distressa		X
Prevalence of lonelinessa		X
Prevalence of anxious complaintsa,b		X
5. Spiritual care: Patient and their family should be provided resources and have access to spiritual care
Prevalence of struggling with the meaning of lifeb		X
Prevalence of wanting to die nowb		X
6. Home as the preferred place of death: no QIs could be created to address this theme

a indicates a QI that can be generated with the interRAI HC data

b indicates a QI that can be generated with the interRAI PC data

Phase III: Modified Delphi panel to evaluate preliminary QIs

The third phase utilized a modified Delphi panel, based on the RAND/ULCA appropriateness method [57]. The main goal of the Delphi panel was to assess the level of agreement among a group of PC clinicians, researchers, and decision makers with respect to keeping or dropping any of the preliminary QIs. The Delphi method has been widely used in PC research [58]. Prior to the Delphi panel receiving the QI definitions and evaluation criteria, an Excel spreadsheet containing each of the QIs and evaluation criteria (along with an information letter/consent form) was shared with two researchers on our team, and a group of graduate students, for their feedback. We first consulted with two researchers with expertise in the Delphi process and a strong understanding of the goals of our study and a small group of graduate students (roughly 15–20), representing “non-experts,” to ensure that the materials were clear and the instructions were easy to follow. The students represented a mix of master’s and PhD level students who were all completing degrees within the School of Public Health Sciences at the University of Waterloo.

Decision makers who took part in an interview or focus group, during Phase I, were eligible to participate in the Delphi panel. Panel participants provided written informed consent prior to their participation. They were asked to evaluate each QI on four criteria: 1) Importance- the extent to which the indicator reflects an important outcome or issue for those receiving PC or their caregivers; 2) Validity- the degree to which the indicator truly captures some aspect of the quality of care (at a population level, not for an individual); 3) Evidence of improved outcomes- evidence that improvement in the indicator can have a positive effect on the individual; and 4) Usability- the extent to which the QI can be readily interpreted and used to improve care delivery. These criteria were based on previous research [59] and were rated on a scale of 1 to 9 (1 = low; 9 = high) as per the RAND/UCLA method. The evaluation spreadsheet also asked for input on the definition of the numerator/denominator. Finally, participants were given the opportunity to provide open-ended feedback on each QI, as well as space to suggest any additional QIs that the Delphi panelists felt were missing from the list, regardless of whether they could be measured with existing interRAI data.

Participants were given six weeks to complete the documents. If documents were not returned two weeks after the initial deadline, an individual reminder e-mail was sent, asking them to be returned within two weeks. After that time, if they still were not returned, a final reminder phone call was made. Any documents not returned then were considered a non-response and no further contact was made with the participant.

Determining consensus among raters

As outlined in the RAND/UCLA manual [57], the process to determine the level of agreement among the panel members involves multiple steps.

Step 1: focused on determining if there was “disagreement” or “agreement,” for each of the four criteria. This involved several calculations in order to arrive at the value of the interpercentile range adjusted for asymmetry (IPRAS). This method is ideal for panel sizes larger than 15, and therefore appropriate for our panel (n = 21) [57]. An example has been provided (S1 File) which outlines how each of the four criteria were determined to have agreement/disagreement.

Step 2: involved assessing the value of the median, for each of the four criteria, in conjunction with whether there was “agreement” or not from Step 1. Each of the four criteria were assigned into one of three mutually exclusive groups, namely “discard,” “retain,” or “review,” based on the work of Nakazawa et al. [60]. “Discard” was defined when the median value was between 1–3 AND there was agreement in Step 1. “Retain” was defined when the median was between 7–9 AND there was agreement from Step 1. “Review” was defined when the median was between 4–6 OR the median was another value AND there was disagreement in Step 1.

Step 3: In this step, each QI was assigned into one of three groups, namely “discard,” “retain,” or “review”, based on a review of the rating of the individual QI’s four criteria. For example, if any of the four criteria were rated “discard” in Step 2, then the QI would be discarded. If three or four of the criteria were considered “retain” then that QI was kept. Two scenarios were used to decide if a QI should be “reviewed.” First, if two of the criteria were considered “retain,” then we retained the QI for further review. Second, if two or more of the criteria were considered “review” then the QI would also fall into the “review” category. It should be noted that the research team utilized the Delphi results as a guide, to support decision-making, but the team also used their discretion when making the final decisions about whether or not to keep a QI for further consideration.

Results

A total of 33 individuals were invited to take part in the Delphi panel. They represented members of our research team (n = 12) and other experts who they suggested that we approach (n = 21). Of the 33 evaluation spreadsheets sent to the Delphi panel members, 21 were completed for a response rate of 63.6%. This group of 21 participants included three individuals who also provided input during Phase I. Among those who did not respond, one person felt that they did not have the necessary expertise to complete the evaluations, and the remainder (n = 11), did not respond after repeated emails. There were six individuals who consented, completed the demographic questionnaire, but then ultimately did not respond. They were very similar to respondents in terms of age (mean = 53.8; sd = 5.9), gender (female = 66.7%) and years of experience working in PC (>10 years = 66.7%). These individuals came from a variety of backgrounds including research (n = 3), nursing (2), and medicine (1).

The Delphi participants were mostly female (71.4%), with a mean age of 46.3 years (sd = 7.3) and the majority had been working in the area of PC for more than 10 years (66.7%; Table 2). The largest proportion (38.1%) came from Ontario, but there were also representatives from British Columbia, Alberta, Nova Scotia and Yukon Territory and two experts from outside of Canada. The majority had a clinical background in nursing or medicine (61.9%), with the remainder involved in PC research or working in the field in the role of a health care administrator or policy maker.

Table 2

Demographic characteristics of individuals who participated in the expert panel.

	Total sample (n = 21)
Mean age in years (standard deviation)	46.3 (7.3)
% (n)
Gender
Female	71.4 (15)
Male	28.6 (6)
Professional background a
Director/Senior Director/Project Lead	28.6 (6)
Physician	23.8 (5)
Registered practical nurse	19.0 (4)
Researcher	19.0 (4)
Other	14.3 (3)
Years of experience working in palliative care
<1 year	4.8 (1)
1–10 years	28.6 (6)
>10 years	66.7 (14)
Highest degree of education completed
University—graduate degree	90.5 (19)
College/Undergraduate university degree	9.5 (2)
Geographic location
Ontario	38.1 (8)
British Columbia	19.0 (4)
Nova Scotia	14.3 (3)
Alberta	9.5 (2)
Yukon Territory	9.5 (2)
Outside of Canada	9.5 (2)

aThese groups are not mutually exclusive as participants were able to select all that applied

Of the 27 preliminary QIs that were evaluated, 20 (74.1%) were classified as “retain” and the remainder, as “review” (Table 3). None of the proposed QIs had scores that would put them into the “discard” category. The team decided to keep all QIs where the Delphi panel suggested that the QI be retained. Since there were only seven QIs classified as “review,” a second Delphi panel was deemed unnecessary. Instead, an online meeting was held with the research team, who provided feedback on these QIs. The final decision was made to drop five of these QIs from further consideration, mainly based on the concern about whether these QIs were truly capturing quality of PC services. Panel members also provided suggestions for new QIs that the team should consider. These included topics such as the timely access to PC services, satisfaction with care, the place of death/preferred place of death, and details around the treatment for certain issues (e.g., for depression, for anxiety). Since there were no interRAI data elements to capture these suggestions, no additional QIs were developed.

Table 3

Summary of scores from the expert panel and final decision for each of the proposed QIs.

Quality indicatora	Median score from the expert panel 1 = low and 9 = high (and results from Step 1)b				Panel Decision	Final Decision
	Importance	Validity	Evidence of improved outcomes	Usability
Prevalence of falls	8	6	8	7	retain	KEEP
Prevalence of severe or excruciating daily pain	9	8	9	8	retain	KEEP
Prevalence of pain that is not controlled by medications	9	7	8	8	retain	KEEP
Failure for pain to improve	8	7	7	7	retain	KEEP
Prevalence of constipation	8	7	8	8	retain	KEEP
Prevalence of shortness of breath at rest	9	7.5	8	8	retain	KEEP
Prevalence of shortness of breath upon exertion	7	6	8	7	retain	KEEP
Failure for shortness of breath to improve	8	7	9	7	retain	KEEP
Prevalence of stasis/pressure ulcers	8	8	8	8	retain	KEEP
Prevalence of a delirium-like syndrome	9	7	8	7	retain	KEEP
Prevalence of nausea or vomiting	8	7	8	7.5	retain	KEEP
Prevalence of fatigue	7	6	8	7	retain	KEEP
Prevalence of sleep problems	8	7	8	7	retain	KEEP
Prevalence of poor self-reported health	6	4	6	5	review	DROP
Prevalence of negative mood	8	6	8	7	retain	KEEP
Failure for negative mood to improve	8	7	8	8	retain	KEEP
Prevalence of declining social activities that causes distress	7	6	7	6	review	KEEP
Prevalence of loneliness	8	6 (D)	8	7	retain	KEEP
Prevalence of caregiver distress	9	8	8	8	retain	KEEP
Prevalence of anxious complaints	8	6	8	6	review	KEEP
Prevalence of struggling with the meaning of life	6.5	5 (D)	6	5.5	review	DROP
Prevalence of clients feeling a lack of completion of financial, legal and other formal responsibilities	7	5.5	7	6	review	DROP
Prevalence of clients feeling that progress is not being made regarding completion of personal goals	7	5.5	6	6	review	DROP
Prevalence of wanting to die now	8	5	5.5 (D)	5 (D)	review	DROP
Prevalence of emergency department visit	7	7	7	7	retain	KEEP
Prevalence of hospital admissions	7	7	7	7	retain	KEEP
Prevalence of no advance directives	8	7.5	7.5	8	retain	KEEP

a The final list of 22 quality indicators that were kept are shown in italicised font

b All criteria had “agreement” following step 1 except those marked with “D” to indicate disagreement

The modified Delphi panel resulted in 22 QIs kept for further testing and validation. Within the QIs capturing clinical issues, the indicators with the highest scores related to importance were those related to pain, shortness of breath and delirium. Those with the highest importance scores in the “psychosocial” area included QIs capturing caregiver distress, mood and loneliness. Three other QIs were kept related to hospital or ED use and advance directives.

Discussion

To our knowledge, this is the first project to recommend a set of standardized QIs for community-based PC using existing interRAI data. The proposed set of 22 QIs was developed through a rigorous and multi-year process involving many stakeholders and researchers from across Canada and in two other countries. The QIs explicitly capture the issues cited as important by those receiving PC, their families, and those working in the field. The proposed QIs can be measured with existing interRAI instruments, currently used in more than 30 countries around the world. This allows for cross-country comparisons, which have previously been completed using the QIs for nursing homes [61]. Using the existing interRAI data is an efficient and cost-effective use of this information and avoids the additional effort that would be required if quality was assessed by using detailed chart reviews [37] or additional surveying of staff, individuals receiving PC, and families.

While it is important to understand the validity of individual indicators, it is also important to evaluate the content validity of the set of QIs. Several criteria have been proposed with which to carry out this evaluation [62]. For example, it is important that the QIs adequately cover the depth and breadth of the content of interest. One way to assess this criterion is to compare our QI set to the six key themes which were discussed during Phase I. In five of these themes, we were able to develop at least one QI in each theme. However, we could not create any QIs related to theme six, namely, the home as the preferred place of death, as we lack the data elements in the interRAI tools to capture this.

A second criterion of content validity relates to proportional representation, or the number of QIs in each domain that matches the importance of that domain in the assessment of PC quality. We treated each of the six themes as equally relevant since we had no other information with which to judge the importance of these themes. However, it is clear that given the type of clinical data captured within the interRAI assessments, it was easier to create QIs to address theme four, symptom management, versus the other themes. Another criterion is the costs of measurement which relates to the burden of data collection on providers. In this regard, these preliminary QIs have a very low cost since they are calculated using existing data and no additional data collection is required. The QIs were rated, during the Delphi panel, in terms of importance, providing insight into the criterion that captures the priority or ranking of the QIs. We feel that the QI list does not include redundant QIs, another important criterion, since none of the QIs were suggested to be dropped from the original list during the Delphi process. This provides evidence that all of the 27 preliminary indicators were rated high enough to warrant further consideration. Finally, the QI development did a very good job in addressing stakeholder involvement. The QIs were developed with input from multiple stakeholders which improves the confidence that the QIs have content validity. We were, however, limited to input from only one individual, with lived experience, who was receiving PC, despite nearly a year of effort in attempting to recruit other care recipients from across Canada.

Although the interRAI data represent a very rich source of information, the study team was limited to creating QIs that could be measured using existing items within two interRAI assessments and were unable to create QIs to reflect some salient issues mentioned during the qualitative interviews, many of which have also been cited in the literature as important aspects when assessing the quality of PC. For example, we were unable to create QIs to assess issues related to communication between the person, their family and members of the health care team [63], the use of hospice/PC services [64], the extent to which the person’s wishes were met [64], and access to resources and services. We were also unable to determine what specific treatments were received for certain clinical issues like depression and anxiety [18] and how satisfied people were with the PC services they received [65]. This information is important to assess as part of ongoing quality improvement efforts, although it was not directly germane to the clinical rationale for the interRAI assessments, which is care planning. As a result, this type of information would have to be captured using alternative means.

The proposed QIs provide an efficient means to capture key quality issues using existing interRAI data. Since the interRAI assessments are used widely in Canada, and in multiple other countries, these data provide a cost-effective source of information for testing and validating QIs. Although their main function is to assess the overall care needs of the individual, in order to drive care planning, interRAI assessments are useful for case-mix measurement [66-69] and quality assessment [39, 70–72]. In Canada, public reporting on select QIs already exists for long-term care homes [73], and those receiving PC and their families deserve a similar level of transparency. The proposed QIs can contribute to improvements in quality by providing detailed information to individual care providers (e.g., home care agencies) to drive internal continuous quality improvement efforts. The QIs can also provide a solid basis for quality benchmarking and learning, when organizations are compared at a systems-level. Finally, the QI data can be used to educate consumers and to guide health care policy.

The next steps in this project will involve analyzing the existing interRAI data to understand the properties of these QIs (e.g., magnitude of the issue, level of variation between geographic regions) as part of the ongoing validation process to decide which ones should be kept in the final list. Our team has access to approximately 3.7 million home care assessments from five provinces and one territory [50]. In addition, the study team has access to approximately 110,000 interRAI PC assessments, which are completed with palliative home care clients in Ontario only. We also plan to create client-level risk adjusters for each of the proposed QIs to account for differences in risk factors across patient populations [74-76]. This step is very important since complex illnesses, multiple co-existing conditions and case-mix differences can influence the QI measures, irrespective of quality. Organizations that provide care to more impaired individuals will tend to have higher unadjusted rates, regardless of the quality of care they provide [74]. Risk adjustment methods are therefore needed to maximize the ability to make fair comparisons between providers [77].

As a result of this work, we have identified a set of 22 validated palliative care QIs capturing multiple issues that are important to individuals receiving PC, families and decision makers. This work fills an important gap as many other sectors of the health care system in Canada have access to interRAI-based QIs to assist in decision support and quality improvement [70, 71, 78, 79], but this has been lacking in the PC sector. Once the QIs are finalized, they can be readily embedded into existing software systems for use by Canadian provinces and health authorities who are using the interRAI assessments. They can also be calculated in other countries using these interRAI tools (e.g., the 21 countries using the home care instrument). The final set of QIs will be useful for the purposes of benchmarking performance across different subpopulations of interest, such as health planning/funding regions. The QIs will also provide community-based PC providers, and health system and policy decision makers, with real-time data to support them in targeting their quality improvement efforts and evaluating client outcomes.

Operational definitions for each of the 27 quality indicators (QIs).

(DOCX)

Click here for additional data file.

An example of applying the interpercentile range adjusted for asymmetry (IPRAS).

(DOCX)

Click here for additional data file.

7 Jan 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Jason Scott

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf  and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

3. Thank you for stating the following in the Competing Interests section:

“I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. A. Sinnarajah has received research grants (last 5 years) on palliative care (as principal investigator or co-investigator), from the Canadian Institutes of Health Research, MSI Foundation, Canadian Cancer Society, Applied Research in Cancer Control, College of Family Physicians of Canada, Choosing Wisely Alberta, Alberta Innovates Health Research, Alberta Cancer Foundation, Alberta Health Services, University of Calgary, Canadian Frailty Network, Alberta Health and Campus Alberta Health Outcomes and Public Health. He has an academic appointment for palliative care research with Queen’s University and Lakeridge Health (currently), and University of Calgary (last 5 years). Lastly, he is/has paid medical administrative positions with Alberta Health Services and Lakeridge Health. The remaining authors declare that no competing interests exist.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I found this article very interesting and I am sure that other readers in the field will also find it so. Observations: The Q.Is were explicit although no real comment was made about the lack of expert by experience or service user input here. What is apparent is the nature of the Q.Is that were dropped. Although these were dropped because of Q.I scores they were focused on psycho social elements. The others were all medical which could be expected however, there are close links between both these elements. I am sure you agree. I understand that this would be a focus of the next stage of research? However, having a rational for this and ensuring that this is a key consideration would have reflected your statement of intention to rectify this. As you say, there was only one family member involved and this could be considered a gap.

Reviewer #2: This manuscript is about a set of quality indicators for community-based palliative care by using interRAI data. Although the author successfully described total 22 QIs developed by panel evaluations.

Major

1. The criteria the author used was slightly different from the criteria in the previous literature the author cited (Importance, Scientific acceptability, Usability, and Feasibility). Furthermore, generalizability of the indicator is also one topic of the criteria for QI. How about the generalizability of these indicators outside the interRAI network? Please discuss.

2. Modified Delphi approach usually includes two or more rounds excluding item generation phase. Why did the author perform only one round? Please discuss.

3. How to determine the domain in this study group? That information would also be useful for readers to capture the process of consensus development.

4. The description about panel members were slightly complex to capture. The relationship between 33 Delphi panel members and 30 PCs and other stakeholders was unclear. Who were these 33 members?

5. Disagreement and panel decision about QI seemed to be complex to understand.

5-i. First, according to the Step 2 in this manuscript, three groups (“discard”, “retain”, or “review”) seemed not to be collectively exhaustive. How to deal with the indicator when the median value was between 7-9 and no agreement in STEP1?

5-ii. Next, in STEP3, if the criteria were considered exact two “review” and two “retain”, which category should the QI be in?

5-iii. Third, how to determine “DROP” or “KEEP” in the Table3? This process was not described in Method section.

5-iv. Finally, Table3 only included appropriateness score for each QI. Disagreement/agreement of each category is necessary to understand the panel decision of each QI precisely. It seemed that the words (“review”, “retain”, and “discard”) were used both for the four criteria and for the individual QI, which could make readers confuse.

6. There is no data about the numerator and denominator of each QI. Furthermore, the detail explanation of each QI should be described.

Minor

1. In supplemental file, “Interpercentile range (IPR)= Upper IPR – Lower IPR” is correct? Upper limit IPR?

Reviewer #3: 1. Line 344: The authors write that “the proposed QIs provide a relatively comprehensive picture of the issues that are 345 widely accepted as key metrics when assessing the quality of PC services[61].” What does “comprehensive” mean in this context? I strongly suggest that you critically discuss the content validity not only of single indicators, but also of the indicator set as a whole (see e.g. Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. Int J Qual Health Care. 2021 Jul 31;33(3):mzab107. doi: 10.1093/intqhc/mzab107. PMID: 34282841; PMCID: PMC8325455.)

Specifically, it would be helpful to explicitly report on the degree to which the proposed QI cover the 6 themes of palliative care that you mention at the beginning of your paper, the gaps in your indicator set and whether the proposed QI represent a balanced view of what is important for palliative care patients, e.g. with respect to the 6 themes.

1. Table 1: Why is the distinction between “clinical” and “psychosocial” indicators relevant? I would expect more content-oriented distinctions between indicators, e.g. with respect to the 6 themes of palliative care you mention.

2. While a Delphi study can help to capture judgements by experts, the qualitative reasons put forward for quantitative ratings remain a “black box”. Also, what does the following sentence mean: “It should be noted that the research team utilized the Delphi results as a guide, to support decision-making, but not as a constraint” (line 272 f.)? To enhance transparency of your findings, I strongly suggest reporting for each QI why you dropped or kept them, and how you made that decision.

3. Why did you decide to drop important indicators only because no data was available? If these indicators were important, wouldn’t it be valuable to develop the required data, e.g. expand on existing interRAI assessments?

4. It remains unclear whose quality is to be measured by the indicators – are specific providers accountable for the features measured? Or a regional health system as a whole? Please specify.

5. Please specify what RN (line 114) means – research nurse?

6. What exactly are “knowledge users” (line 160) and why is it important that they did the recruitment of patients and family members?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Atsushi Mizuno

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: 2022-01-03_QI-palliative-care.docx

Click here for additional data file.

27 Jan 2022

Our responses to the reviewers' comments have been uploaded with our cover letter.

Submitted filename: response to reviewers_jan2622.docx

Click here for additional data file.

23 Feb 2022

14 Mar 2022

Reviewer #1:

Following amendments reviewer concerns have been addressed. It is evident that manuscript authors have considered all comments to some extent. Although accepted, I would still be interested in seeing a clearer statement about ethics. Line 160 identifies KUs recruited those in receipt of PC and their families. Line 161 because it was inappropriate for the research team to do so. Line 165 identifies 8 KUs on the research team? I imagine KUs have stringent ethical probity however, this is not declared and lines 160, 161 and 165 appear to contradict each other.

We apologize if this was confusing. The 8 KUs on our team acted as a “conduit” to families and individuals receiving palliative care and helped to link us to these individuals. The KUs and our team felt that it would be inappropriate for us, as a research team, to reach out directly to families/care recipients. Instead, the KUs helped to share our project with potential participants who could then contact us IF they were interested. For example, the KUs helped us by posting flyers on their web pages or by emailing study information to potential participants. In this sense, they were not participants themselves, but were simply helping us to identify individuals who were willing to take part in the research. We have modified line 161 and added one new sentence (line 164-165) to help clarify this process.

Reviewer #2:

1. As I described, the generalizability of this results should be discussed. First of all, the author did not discuss about actual dataset of interRAI but interRAI data elements (e.g. data components) only. Thus, these indicators by authors are just only agreement of these elements could be useful for palliative care through consensus strategy. Therefore, the author should describe not only advantage of these indicators but also disadvantages such as lack of information compared with the previous literatures. The advantages of these indicators could be more contrasted. For example, “Prevalence of falls” is unique as indicators for palliative care, especially under sub-theme of symptom management.

To clarify, the “interRAI dataset” is made up of the data elements within the assessment and nothing else, so the two things are identical. To be more explicit about which QIs have been cited in the literature, but which we could not measure, we have added references to the literature for those specific QIs where we mention them on lines 376-382 (please note that these added references do not show up in the “track changes” in the word document, but include ref #18, 63, 64 and 65).We feel that we have already described the unique advantages of the QIs derived using the interRAI dataset in the discussion section and feel it would make the discussion somewhat repetitious to add further comment on this.

2. As far as I am correct, the panel members for this unique Delphi manuscript, especially for Phase III seemed to be completely different from Phase I and II. This should be clarified not only in Results section but also in Method section. As it has not still been improved, it could make readers confuse. The selection bias of panel members should also be described in limitation of this study.

It was possible for decision makers participating in the Delphi panel to have also taken part in an interview/focus group during Phase I. We did not explicitly exclude them from the Delphi panel. We have modified lines 242-243 in the methods, and lines 291-292 in the results, to make this more explicit.

We do not feel that there was selection bias operating in the panel members. We had 6 individuals who consented, provided a demographic questionnaire but then did not complete the Delphi exercise. They were very similar to Delphi panel members in terms of age, gender, years of experience and clinical background. We have added additional text to describe this group (lines 294-298).

3. The word about “domain” and “theme” could be used carefully not to make readers confuse in this manuscript. It would be better to describe how to determine these domains and themes by this study team (including all Phase I, II and III).

We see these as two distinct concepts. We have used the term “domain” when discussing the QIs and the extent to which the indicators capture important information to assess quality. We found one instance where the term “domain” likely was inappropriate (line 329) and switched it to the word “area” to be more clear.

On the other hand, we have used the term “theme” specifically as it relates to the qualitative analysis of the workshop results, the focus groups, and interviews. We have added some text (line 153) to make it more clear with respect to the analysis of the workshop data. A quick search confirms that the term “theme” has always been used in this context in the manuscript. This is the accepted wording used in qualitative research to highlight a topic of discussion that was raised by multiple participants. It reflects the way in which the key issues raised by participants “emerge” during the qualitative analysis. As such, we feel that these two terms are indeed unique and have been used appropriately throughout the paper.

Submitted filename: response to reviewers_mar322.docx

Click here for additional data file.

23 Mar 2022

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

PONE-D-21-34279R2

Dear Dr. Guthrie,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jason Scott

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: The authors have answered all the comments and improved manuscripts for readers appropriately. As I asked previously, I would recommend the authors to describe how to make "domain" of these quality indicators.

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

29 Mar 2022

PONE-D-21-34279R2

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

Dear Dr. Guthrie:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jason Scott

Academic Editor

PLOS ONE