Nurse-sensitive outcomes in district nursing care: A Delphi study.

OBJECTIVES: To determine nurse-sensitive outcomes in district nursing care for community-living older people. Nurse-sensitive outcomes are defined as patient outcomes that are relevant based on nurses' scope and domain of practice and that are influenced by nursing inputs and interventions.
DESIGN: A Delphi study following the RAND/UCLA Appropriateness Method with two rounds of data collection.
SETTING: District nursing care in the community care setting in the Netherlands. PARTICIPANTS: Experts with current or recent clinical experience as district nurses as well as expertise in research, teaching, practice, or policy in the area of district nursing. MAIN OUTCOME MEASURES: Experts assessed potential nurse-sensitive outcomes for their sensitivity to nursing care by scoring the relevance of each outcome and the ability of the outcome to be influenced by nursing care (influenceability). The relevance and influenceability of each outcome were scored on a nine-point Likert scale. A group median of 7 to 9 indicated that the outcome was assessed as relevant and/or influenceable. To measure agreement among experts, the disagreement index was used, with a score of <1 indicating agreement.
RESULTS: In Delphi round two, 11 experts assessed 46 outcomes. In total, 26 outcomes (56.5%) were assessed as nurse-sensitive. The nurse-sensitive outcomes with the highest median scores for both relevance and influenceability were the patient's autonomy, the patient's ability to make decisions regarding the provision of care, the patient's satisfaction with delivered district nursing care, the quality of dying and death, and the compliance of the patient with needed care.
CONCLUSIONS: This study determined 26 nurse-sensitive outcomes for district nursing care for community-living older people based on the collective opinion of experts in district nursing care. This insight could guide the development of quality indicators for district nursing care. Further research is needed to operationalise the outcomes and to determine which outcomes are relevant for specific subgroups.

Introduction

Worldwide, healthcare services are challenged by the rapidly growing ageing population [1]. Moreover, the majority of older people desire to continue living at home, resulting in a rise in the total number of community-living older people. In Europe, the majority of older people live independently at home, either alone or with a spouse or other family members [2]. However, with increasing age, adverse consequences such as frailty, disability, chronic diseases, and multiple complex long-term conditions are present among these community-living older people [3, 4]. Because of these adverse consequences, community-living older people often need assistance with their daily life activities to be able to live at home as long as possible. Professional care assistance at home is provided through district nursing care, next to other healthcare professionals such as the general practitioner and other (paramedic) professionals in primary care [5]. The funding, organisation, definition, and delivery of district nursing care vary between countries worldwide [6-8]. For the purpose of this paper, district nursing care is defined as any technical, medical, supportive or rehabilitative nursing care and the provision of assistance with personal care [7]. This definition is in line with the definition used for community care nursing in Europe [7, 9] and reflects district nursing care in the Netherlands [10].

In many European countries, the quality of care at home is under pressure, as demands on district nursing care are increasing due to the ageing population, the increase in care complexity, and the shortage of district nursing care professionals [11, 12]. Therefore, it is crucial to monitor the quality of district nursing care in terms of patient outcomes. Insight into patient outcomes is necessary to measure the effect of healthcare services on patient health and wellbeing [13, 14]. However, patient outcomes to measure the quality of district nursing care in clinical practice on patients’ health status and wellbeing are currently scarce [15].

For district nursing care, it is necessary to determine nurse-sensitive outcomes, i.e., patient outcomes that are relevant based on nurses’ scope and domain of practice and that are influenced by nursing inputs and interventions [16]. The Nursing Outcome Classification (NOC) provides a set of nursing outcomes that can be used across the care continuum to assess the outcomes of care following nursing interventions [17]. However, in this overview, it is unclear what outcomes are relevant for district nursing care. Two studies, one by the International Consortium for Health Outcomes Measurement (ICHOM) [18] and the other by Joling et al. [15] have already been conducted on outcomes that are potentially relevant to district nursing care. The ICHOM developed a set of standard health outcome measures to guide the improvement of the quality of care for the general population of older people [18]. While this study provided a meaningful overview of relevant outcomes for this population, it remains unclear whether these outcomes are nurse-sensitive outcomes specifically for district nursing care because they were developed by teams of physician leaders, researchers and patient advocates [18]. The systematic review by Joling et al. [15] identified 567 quality indicators for older people in the community care setting (i.e., primary care and district nursing care). Most of these indicators refer to care processes (80%), while only 33 indicators focus on 18 unique patient outcomes regarding health status and wellbeing (5.8%) [15]. However, it is unclear which of the proposed outcomes in the literature could be used as nurse-sensitive outcomes for district nursing care. Before quality indicators can be developed and operationalized, it is necessary to determine what outcomes are relevant to measure.

The aim of this study was to determine nurse-sensitive outcomes for district nursing care for community-living older people. Measuring nurse-sensitive outcomes for district nursing care is important because it can contribute to understanding the internal quality of teams and organisations. It provides insight into the quality of delivered care, which consequently could guide monitoring and improve the quality of district nursing care. Moreover, public transparency regarding outcomes allows patients to compare and choose a desired organisation. Finally, insight into nurse-sensitive outcomes could guide health insurers in contracting district nursing care organisations based on the quality of delivered care.

Materials and methods

Design

A Delphi study following the RAND/UCLA Appropriateness Method (RAM) [19] was performed. The objective of the RAM is to detect when experts agree rather than to reach consensus among experts [19]. The RAM is focused on combining available scientific evidence with the collective judgement of experts to provide a statement regarding the appropriateness of delivered care [19]. This focus fits the aim of this study to determine nurse-sensitive outcomes for district nursing care based on the collective opinion of national experts. Because of the specific national context of district nursing care, this study focused on the situation in the Netherlands. To enhance the robustness of this study, the guidance on conducting and reporting Delphi studies (CREDES) was followed [20]. In accordance with the RAM, the following steps were conducted: questionnaire development, identification of experts, two rounds of data collection (an online questionnaire and an expert panel meeting including a paper questionnaire), and data analysis after both rounds. Attrition bias due to the exhaustion of the experts was prevented by limiting the number of Delphi rounds to two rounds.

Questionnaire development

The questionnaire was developed by reviewing the literature. Scientific and grey literature were searched using the following keywords and their accompanying synonyms: “patient outcomes,” “district nursing care,” and “quality indicators.” For scientific literature, MEDLINE/PubMed and CINAHL/EBSCO were searched. For grey literature, international and national websites and reports of governments and research institutions were searched. Additionally, Dutch reports on what older people find important in the care that they receive at home were identified and analysed to include the patient perspective and guide the identification of important patient outcomes for district nursing care [21, 22]. The literature was reviewed until no new outcomes for district nursing care were identified. In total, 41 patient outcomes were identified. The 41 outcomes were clustered following the domains used in the nursing outcomes classification by Moorhead et al. [17]: Functional health (n = 4), physiologic health including neurocognitive health (n = 16), psychosocial health (n = 4), health knowledge and behaviour (n = 6), perceived health (n = 2), and family health (n = 1). Additionally, the domains death (n = 2) and healthcare utilization (n = 6) were added. These outcomes were extracted from systematic reviews; peer-reviewed scientific publications, including those from the ICHOM; and reports on potentially preventable complications (see S1 Appendix). Different references were used for defining the outcomes. The outcomes were defined based on the definition used by one references or–in case definitions were incomplete, inconsistent between references, or not suitable for district nursing practice–a combination of multiple references. Because the participants were from the Netherlands, mostly Dutch literature has been used. Because the study aims to determine what outcomes are nurse-sensitive to district nursing care rather than developing and operationalizing quality indicators, the definitions of the outcomes were not constructed as quality indicators.

To determine the sensitivity of the identified outcomes to nursing care, the relevance and influenceability of the outcomes were scored. Relevance was operationalised as “being a relevant patient outcome to measure the quality of district nursing care,” and influenceability was operationalised as “the extent to which district nursing care has an influence on the patient outcome.”

At the beginning of the developed questionnaire, information was provided about the study. The background information of the participants regarding their age, sex, years of experience in district nursing care, and area of work was collected. Next, all 41 potential nurse-sensitive outcomes were presented along with their definitions. Participants were asked to score both the relevance and influenceability of each outcome on a 9-point Likert scale, with 1 being completely not relevant/influenceable and 9 being completely relevant/influenceable. An example question is shown in S2 Appendix. Participants had the opportunity to propose additional outcomes in case outcomes had been omitted. The complete questionnaire is available upon request.

Identification of experts

A purposive sample of national participants was selected for the expert panel of this Delphi study. To ensure the diversity of the district nursing care professionals, the following inclusion criteria were used: 1) the participant had current or recent clinical experience as a district nurse, and 2) the participant had experience in research, teaching, practice, or policy with regard to district nursing care. The aim was to purposively create a balance between people currently working in district nursing care and those with recent experience in practice yet currently fulfilling a role in research, teaching, practice or policy regarding district nursing care. With the requirement of the nurses to have an (additional) role in research, teaching, practice, or policy, it was assumed that the nurses would be accustomed to critical thinking and reflection, which was necessary given the challenges of defining outcomes of care [16]. Participants (hereafter referred to as experts) from a diversity of organisations across the Netherlands were selected. Based on the RAM, the aim was to include a panel of 10–15 experts, which would allow the expert panel to have sufficient diversity while also ensuring that all experts would have a chance to participate [19]. To take into account the possible decline in participation during the multiple rounds, a total of 20 experts were approached via the Dutch nurses’ association and the researchers’ networks. Experts were informed about the study and invited to participate by email.

Data collection

Delphi round one: Online questionnaire

The first Delphi round started with an online questionnaire using the online tool Qualtrics [23]. The experts received a personal invitation to the questionnaire by email. A letter including information about the study and providing consent for the study was provided within the questionnaire. The experts were asked to complete the questionnaire within two weeks. Two reminders were sent to increase the response rate. After the deadline, the online questionnaire was closed, and the results were analysed. New outcomes proposed by the experts were reviewed by a part of the research team (JDV, NB, MJS). The team discussed if the outcomes focused on patient outcomes or were relevant for measuring the quality of care. Decisions were made based on the expertise of the research team. Five outcomes were included in the next round: a meaningful life, duration of district nursing care, the intensity of district nursing care, total time at home, and quality of dying and death. Two outcomes focusing primarily on process or structure of care (providing preventive care and accessibility of district nursing team) were not included. The newly added outcomes were defined using the literature and by insights of the experts. (S1 Appendix).

Delphi round two: Expert panel meeting and paper questionnaire

After the analysis of the results of round one, the content from the online questionnaire was supplemented with the five newly added outcomes in a paper questionnaire. In the second Delphi round, the experts participated together in a three-hour face-to-face expert meeting. During this meeting, the findings from the questionnaire from round one regarding the relevance and influenceability of the outcomes were discussed, with special attention to the outcomes that lacked agreement (disagreement index (DI) ≥1), the outcomes that had an uncertain rating (group median 4–6), and the newly added outcomes. Additionally, the definitions of the newly added outcomes, formulated by the research team were discussed and concluded with the experts in the second Delphi round to assure that this corresponded to what the experts initially meant. After discussion of the outcomes in the expert meeting, the paper questionnaire was completed. In this questionnaire, the experts’ individual scores from the first round; the group median score; and the DI, as an indication of the level of agreement, were provided (S2 Appendix).

After the analysis of the results of round two, a draft of the results was shared with the participating experts as a member check to confirm the credibility of the results.

Data analysis

All analyses were guided by the RAM. The relevance and influenceability of each potential nurse-sensitive outcome was scored on a nine-point Likert scale. For each outcome, a group median score was calculated to determine the degree of relevance and influenceability, and the DI was calculated to determine the level of agreement. As described in the RAM, the DI is the ratio between the interpercentile range (IPR) and the IPR adjusted for symmetry (IPRAS), which can be calculated following the equation in S3 Appendix [19]. A DI <1 indicates agreement, with a score closer to zero indicating stronger agreement. A group median score of 1–3 with agreement (DI<1) indicated that the outcome was not relevant/influenceable, a lack of agreement (DI≥1) and/or a group median score of 4–6 with agreement (DI<1) on an outcome indicated that the relevance/influenceability of the outcome was uncertain, and a group median of 7–9 with agreement (DI<1) indicated that the outcome was relevant/influenceable [19]. Scores were analysed using SPSS version 24.

Ethical considerations

The experts were informed that participation was voluntary and that all data would be processed anonymously and only for research purposes. The experts’ consent was assumed upon their return of the completed questionnaires. Because participants in this study were not subjected to physical and/or psychological procedures, no approval was needed according to the Dutch Medical Research Act (WMO). This study was conducted in accordance with the principles of the Declaration of Helsinki, and data were handled according to the General Data Protection Regulation.

Results

Demographics of the expert panel

In total, 16 of the 20 contacted experts (80%) agreed to participate, 15 of whom completed the online questionnaire in round one (93.8%) (Table 1). Of the experts who completed the questionnaire in round one, 11 were able to participate in the expert meeting and questionnaire in round two (73.3%). In both rounds, seven experts indicated that they worked in multiple areas of district nursing care. Reasons for non-response were a lack of time for participation and illness.

Table 1

Characteristics of the expert panel.

	Delphi round 1 N = 15	Delphi round 2 N = 11
Response rate, n (%)	15/16 (93.8)	11/15 (73.3)
Age in years, mean (minimum-maximum; sd)	40.3 (27–65; 12.2)	35.5 (27–53; 9.2)
Female, n (%)	13 (86.7)	9 (81.8)
Years of clinical experience in district nursing care, mean (minimum-maximum; sd)	12.3 (3–20; 6.4)	10.3 (3–20; 6.0)
Current area of workA
District nurse, n (%)	7 (46.7)	7 (63.6)
Researcher, n (%)	5 (33.3)	3 (27.3)
Teacher in a bachelor of nursing program, n (%)	5 (33.3)	4 (36.4)
Practice or policy (manager, professional association), n (%)	7 (46.7)	6 (54.5)

A The percentages do not add to 100% because some experts worked in multiple area

Delphi round one

The 41 potential nurse sensitive outcomes identified in the literature were assessed by the experts in round one. The group median scores and DIs for the relevance and influenceability of the potential nurse-sensitive outcomes are provided in Table 2. Based on the median scores and DIs <1, the experts assessed 22 outcomes as relevant (53.7%) and two outcomes as not relevant (multimorbidity and planned hospital admission) (4.9%). For the remaining 17 outcomes (41.5%), there was uncertainty; for four of these outcomes, the uncertainty was due to a lack of agreement among experts.

Table 2

Median scores and DIs of the relevance and influenceability of outcomes per Delphi round.

	Relevant		Influenceable
	Round 1 Group median (DI)A	Round 2 Group median (DI)A	Round 1 Group median (DI)A	Round 2 Group median (DI)A
Functional health
Activities of daily living	8 (0)	8 (0)	6 (0.21)	7 (0)
Frailty	7 (0)	7 (0.22)	6 (0.22)	7 (0)
Instrumental activities of daily living	7 (0.13)D	7 (0.16)	6 (0.72)	6 (0.21)
Mobility	7 (0.32)	7 (0.16)	6 (0.21)	7 (0)
Physiologic health including neurocognitive health
Bladder continence	6 (1.36)B	4 (0.97)	4 (0.32)	4 (0.32)
Bowel continence	5 (0.93)	4 (0.52)	4 (0.32)	4 (0.32)
Cognitive functioning	6 (0.95)	4 (0.97)	5 (0.32)	5 (0.32)
Communication	6 (0.86)	4 (0.21)	5 (0.72)	6 (0.85)
Decision making	8 (0.13)	8 (0)	7 (0.16)	8 (0.16)
Decubitus	8 (0.16)	8 (0)	7 (0.16)	7 (0.16)
Dehydration	8 (0.33)	8 (0)	7 (0.22)	7 (0)
Delirium	6 (0.86)	7 (0.16)	5 (0.97)	7 (0.21)
Dyspnoea	6 (0.95)	6 (0.52)	5 (0.85)	6 (0)
Fatigue	6 (0.18)	7 (0.16)	6 (0.32)	7 (0)
Fracture and wounds other than decubitus	6 (0.52)	7 (0.22)	6 (0.25)	6 (0)
Infection	7 (0.22)	7 (0)	6 (0)	6 (0)
Multimorbidity	3 (0.33)	2 (0.16)	2 (0.16)	2 (0.16)
Pain	7 (0.16)	7 (0.16)	7 (0.22)	7 (0)
Polypharmacy	5 (1.70)B	3 (0.37)	4 (0.98)	4 (0.32)
Unintentional weight loss	7 (0.33)	8 (0.16)	6 (0.45)D	7 (0.37)
Psychosocial health
Anxiety	6 (0.52)	7 (0.32)	5 (0.52)	7 (0.22)
Loneliness	7 (0.22)	7 (0)	5 (0.86)	6 (0.22)
Participation in social activities	7 (0.22)D	7 (0)	6 (0.18)	7 (0.22)
Signs of depression	6 (0.52)	6 (0.51)	5 (0.72)	6 (0.22)
Health knowledge and behaviour
Autonomy	8 (0)	8 (0)	7 (0.13)	8 (0.16)
Compliance	8 (0.16)	8 (0.16)	7 (0.13)	8 (0.16)
Falls	7 (0.32)	8 (0.16)	6 (0.52)	7 (0.21)
Knowledge of the patient	6 (0.49)	2 (0.16)	5 (0.72)	4 (0.52)
Problem behaviour	5 (0.85)	4 (0.21)	5 (0.72)	5 (0.32)
Substance use	4 (0.97)D	3 (0.16)	4 (0.32)	4 (0)
Perceived health
Quality of life	8 (0.16)	8 (0.16)	6 (0.22)D	7 (0)
Satisfaction with district nursing care	8 (0.23)	8 (0)	8 (0.16)	8 (0.16)
Meaningful lifeC	-	8 (0)	-	7 (0.16)
Family health
Informal caregiver burden	8 (0)	8 (0)	7 (0.16)	7 (0)
Death
Death	5 (1.36)B	3 (0.16)	4 (0.86)	3 (0)
Place of death	8 (0.16)	8 (0.16)	7 (0)	7 (0.16)
Quality of dying and deathC	-	8 (0)	-	8 (0.16)
Healthcare consumption
Emergency department or service use	7 (0.37)	7 (0)	6 (0.42)	7 (0)
General practitioner visit	5 (0.85)	5 (0.52)	6 (0.72)	6 (0.52)
Nursing home admission	6 (2.38)B	5 (0.96)	6 (0.93)	7 (0)
Planned hospital admission	2 (0.37)	2 (0)	3 (0.59)D	3 (0)
Unplanned hospital admission	8 (0.65)	8 (0.16)	6 (0.32)	7 (0)
Unplanned hospital readmission	8 (0.33)	8 (0)	6 (0.22)	7 (0.22)
Duration of district nursing careC	-	7 (0.22)	-	7 (0.16)
Intensity of district nursing careC	-	7 (0.22)	-	8 (0.16)
Total time at homeC	-	5 (0.96)	-	6 (0.22)

Notes

ADL: activities of daily living; IADL: instrumental activities of daily living

Green: Indicates the outcome is relevant/influenceable based on a median score between 7–9 and a DI <1.

Yellow: Indicates the uncertainty of the relevance/influenceability of the outcome based on a median score between 4–6 and/or a DI ≥1.

Red: Indicates the outcome is not relevant/influenceable based on a median score between 1–3 and a DI <1.

A DI: disagreement index, with a DI <1 indicating agreement.

B No agreement based on a DI ≥1.

C Newly added outcomes after Delphi round one.

D In an additional analysis, the median scores and DIs of round 1 with all experts (N = 15) were compared to those of round 1 with only the experts who participated in the expert meeting (N = 11). This comparison revealed the following deviating results for N = 11 compared to N = 15, as described in this table:

IADL: DI 1.61 (uncertain relevance)

Substance use: median 3 (not relevant)

Participation in social activities: median 6 (uncertain relevance)

Unintentional weight loss: median 7 (influenceable)

Quality of life: median 7 (influenceable)

Planned hospital admission: median 4 (uncertain influenceability)

Regarding influenceability, the experts assessed nine outcomes as influenceable (22.0%) and two outcomes as not influenceable (multimorbidity and planned hospital admission) (4.9%). The remaining 30 outcomes were assessed as uncertain (73.2%), with none lacking expert agreement. After round one, the following five outcomes were added as new outcomes: meaningful life, duration of district nursing care, intensity of district nursing care, total time at home, and quality of dying and death.

Delphi round two

After the face-to-face discussion in round two, the experts assessed 30 of 46 outcomes as relevant (65.2%), which were mainly distributed among the domains of functional health (4/4), perceived health (3/3), family health (1/1), psychosocial health (3/4), and outcomes regarding death (2/3). (Table 2). Six outcomes were assessed as not relevant (13.0%). The remaining 10 outcomes were assessed as uncertain (21.7%), of which none lacked expert agreement. The discussion during the expert meeting led to changes in the assessment of the relevance of eight outcomes.

Regarding influenceability after Delphi round two (Table 2), the experts assessed 27 outcomes as influenceable (58.7%), which were mainly distributed among the domains of perceived health (3/3), family health (1/1), functional health (3/4), healthcare consumption (6/9), and outcomes regarding death (2/3). Three outcomes were assessed as not influenceable (6.5%), and 16 outcomes were assessed as uncertain (34.8%). The expert meeting discussion led to changes in the assessment of the influenceability of 15 outcomes.

To determine whether the different compositions of the experts in the two rounds resulted in deviating overall results regarding the relevance and influenceability of the variables, the median scores and DIs of round 1 with all experts (N = 15) were compared to those of round 1 with only the experts who participated in the expert meeting (N = 11). This comparison revealed deviating results for the following six variables: the relevance of instrumental activities of daily living (IADL), substance use, and participation in social activities and the influenceability of unintentional weight loss, quality of life and planned hospital admission. The relevance of IADL and participation in social activities changed from relevant to uncertain, and that of substance use changed from uncertain to not relevant; the influenceability of unintentional weight loss and quality of life changed from uncertain to influenceable, and that of planned hospital admission changed from not influenceable to uncertain. All other variables (92.6%) had minor changes that did not influence the overall results.

In total, the experts agreed that 26 outcomes (56.5%) were nurse-sensitive, i.e., both relevant and influenceable. From high to low, the nurse-sensitive outcomes were distributed among the following domains: perceived health (3/3), family health (1/1), functional health (3/4), death (2/3), healthcare utilization (5/9), health knowledge and behavior (3/6) psychosocial health (2/4), and physiologic health (7/16). Table 3 shows an overview of the nurse-sensitive outcomes, listed in order of most relevant and influenceable (left column) to least relevant and influenceable (right column) based on the group median and the overall DI. The nurse-sensitive outcomes with the highest median scores were the autonomy of the patient, the patient’s ability to make decisions regarding the provision of care, the patient’s satisfaction with delivered district nursing care, the quality of dying and death, and the compliance of the patient with needed care (i.e., the extent to which the behaviour of a patient matches the established care).

Table 3

Nurse-sensitive outcomes according to district nursing care experts.

Outcomes with a group median score of 8 for both relevance and influenceability (N = 5)	Outcomes with a group median score of 8 for relevance and 7 for influenceability (N = 12)	Outcomes with a group median score of 7 for both relevance and influenceability (N = 9)
Autonomy Decision making Satisfaction with district nursing care Quality of dying and death Compliance	ADL Dehydration Informal caregiver burden Decubitus Meaningful life Quality of life Unplanned hospital readmission Falls Unplanned hospital admission Place of death Unintentional weight loss Intensity of district nursing careA	Emergency department or service use Pain Mobility Fatigue Participation in social activities Frailty Delirium Anxiety Duration of district nursing care

ADL: activities of daily living

A Median score of 7 for relevance and 8 for influenceability

Intensity of district nursing care

Discussion

This study is the first to provide insight into nurse-sensitive outcomes for district nursing care based on the collective opinion of experts who represent the district nursing profession. After two Delphi rounds, the experts determined that 26 of 46 outcomes (56.5%) were nurse-sensitive outcomes for district nursing care. The nurse-sensitive outcomes that were assessed as the most relevant and influenceable (i.e., with a median of 8 and a DI between 0 and 0.16) were patient autonomy, the ability of the patient to make decisions regarding the provision of care, the patient’s satisfaction with delivered district nursing care, the quality of dying and death, and the compliance of the patient with needed care.

In the comparison of our results to the outcomes of care for district nursing care described by previous studies by Joling et al. [15] and the ICHOM [18], similarities were found in 14 of the 26 nurse-sensitive outcomes. Activities of daily living, falls, pain, participation in social activities, and informal caregiver burden were considered important outcomes by all three studies. Additionally, overlap with Joling et al. [15] was found for outcomes including decubitus, unintentional weight loss, emergency department or service use, and unplanned hospital (re)admissions. Additionally, overlap was found with the ICHOM study in relation to outcomes including autonomy, frailty, decision making, and place of death [18]. An important difference was that the experts agreed that polypharmacy and mortality were not suitable as nurse-sensitive outcomes for district nursing care. A possible explanation for the differences between our study and those by Joling et al. [15] and the ICHOM [18] lies in the focus of this Delphi study on nurse-sensitive outcomes. The other two studies did not study the relevance of these outcomes to measure the quality of district nursing care specifically and the influence nurses could or could not have on these patient outcomes. Additionally, our Delphi study determined 12 additional nurse-sensitive outcomes that were considered important and that were added by the experts after round one or were mentioned in other relevant literature on patient-reported outcomes for adults in general [24], home care quality indicators [25], or effect measures for primary care [26]. All outcomes identified in our study as nurse-sensitive outcomes for district nursing care are available as nurse outcomes in the nursing outcome classification, except for the outcomes regarding healthcare utilization, which are not included in this classification [17]. In our study, healthcare utilization was used as an outcome following other literature [15, 18].

Strengths and limitations

To enhance the robustness of this study, the RAM and the guidance on CREDES were followed [19, 20]. An important strength was the high response rates for both rounds (93.8% and 73.3%). The differences in characteristics between the experts in the two rounds were minimal, and additional analyses showed that these differences did not influence the results for 92.6% of the variables. Additionally, the member check did not result in any comments. Furthermore, through the inclusion of experts who had clinical experience as district nurses and who had fulfilled additional roles in research, teaching, practice, or policy, the full scope of the district nursing care profession were reflected. In the interpretation of the results, some limitations should be considered. First, only Dutch experts were included in this study because of the specific district nursing context in the Netherlands. This approach limits the generalisability of the results. Second, patients were not included as experts because of the challenges regarding defining outcomes of care [16]. To incorporate their meaningful views, however, we included Dutch reports on what patients find important in receiving care at home [21, 22]. Last, the identification and definitions of the outcomes have some limitations. It is possible that outcomes and quality indicators were missed since no systematic review has been conducted. This risk was minimised by letting experts add and define missing outcomes. However, the definitions by the experts may not be comprehensive and requires further research. Additionally, the outcomes used in this study focus on older people which may limit application in district nursing care which also include care for children and middle-aged people. However, 75% of the people receiving district nursing care in The Netherlands is 67 years or older, and the mean age of the people receiving district nursing care is 75 years [27].

Conclusion and implications

This study provides insight into nurse-sensitive outcomes based on the collective opinion of experts who represent the district nursing profession. In total, 26 nurse-sensitive outcomes were identified that could guide the development of quality indicators for district nursing care. Measuring nurse-sensitive outcomes provides insight into the impact of district nursing care, which is a first step in monitoring and improving the quality of care. This contributes to the major call to action internationally on prioritizing the development of the evidence base for district nursing care [6]. At the national level, policy makers, the Dutch Nurses Association and healthcare organizations are working together to define quality indicators for district nursing care. The results of this study contribute to this development by determining 26 nurse-sensitive outcomes. To use nurse-sensitive outcomes as quality indicators, outcomes should be made measurable in a way that is feasible for current practice. Although the outcomes were defined based on the literature, they were not operationalized as quality indicators with a denominator and numerator. Making these nurse-sensitive outcomes measurable as quality indicators requires further research and development before their implementation in practice. In addition, the nurse-sensitive outcomes may differ between different groups of patients in various types of district nursing care, such as palliative care, rehabilitative care, and chronic care. The distinction between these groups and the accompanying relevant and influenceable outcomes for the quality of district nursing care require further research. Lastly, careful consideration is needed regarding the influenceability of the outcomes. None of these outcomes was assessed as completely relevant or influenceable (median 9), the uncertainty of the influenceability of the outcomes is relatively high (34,8%) and the overall medians of the influenceability of the outcomes are lower compared to the assessment of the relevance. This could be explained by the multidisciplinary role of district nurses in practice. Care for community-living older people is not only provided by district nurses, but also by the general practitioner and other (paramedic) professionals in primary care. Most of the outcomes are indeed often not completely influenceable by the delivered district nursing care. Coordinated care by interdisciplinary teams is associated with better outcomes regarding hospitalizations, emergency department visits, and long-term care admissions in community-living people [5]. Therefore, close collaboration between professionals in district nursing practice is needed to influence and achieve the best possible outcomes for people receiving district nursing care.

Overview of identified potential nurse-sensitive outcomes, corresponding definitions and references.

(DOCX)

Click here for additional data file.

Examples of questionnaire questions round one and round two.

(DOCX)

Click here for additional data file.

Equation to calculate disagreement index (DI).

(DOCX)

Click here for additional data file.

CREDES checklist.

(DOCX)

Click here for additional data file.

8 Dec 2020

PONE-D-20-16928

Nurse-sensitive outcomes in district nursing care: a Delphi study

PLOS ONE

Dear Dr. Veldhuizen,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

In particular, this paper would benefit from additional rigour in reporting and this should include justification for the methodological choices made within the Delphi rounds.  The reviewers also comment that this paper needs international appeal, so a clearer explanation of the role of the District Nurse in other countries or contexts would be helpful.

Please submit your revised manuscript by Jan 14 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Fiona Cuthill, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Your paper is interesting and i enjoyed reading it. Thank you! I did not see the supporting information and perhaps some of my comments are covered in that. I offer you some comments and signposts to strengthen the paper. I hope you find them helpful.

For international appeal, identify different similar roles. The definition given does not resinate with the UK DN role, so articulate which country this reflects.

I don’t know the study is conducted in the Netherlands until p6, so perhaps this needs to be in the aim

In questionnaire development section, i’d like to understand how these indicators are worded. i know it will be in supplementary information, but perhaps including a box with some examples might help. I understand what you mean by nurse-sensitive outcomes, but i want to be convinced that the outcomes of the indicators are for patients/families living at home. On pg 13 in the Delphi Round table 2, the list on the left hand side (with no title) - are these the indicators? If so, how do e.g. falls or instrumental activities of daily living measure patient experience, health and well-being etc that you said you wanted to measure? on p17, i can see that the outcomes in the first column would be outcomes to strive for, but in the middle and the last column, would an outcome really be dehydration?

I don’t understand the terminology, instrumental activities of daily living.

p7L119 - what original reference are you referring to?

p7 L120 - If no definition was provided, other references were used - I’m not sure what this means. can you please clarify.

p7 121 - how did the authors decide the themes? Link back to what you think is missing in the outcomes literature

P8 L130 - 138 - i don’t think this fits here as experts are referred to but they haven’t been defined.

P8 L 131- who was the letter given to and how were they sampled? Why are they considered experts? these questions cos you haven’t identified experts yet.

P8 - were experts from a diversity of services/communities/organisations? Please identify.

Demographics of the expert panel on pg 11/12 would be better moved to p8 as it answers these questions. You have achieved a good balance of participants.

p9 L165 - how many outcomes were added after round one and how did the team make the decision to include them?

p12 L 214 - remind the reader how many outcomes were considered.

p15 - I can’t see C Newly added outcomes after Delphi round one in the table

p17 - Table header hat needs separated

Discussion - in the background section you have offered some critique on Joling’s and ICHOM use of outcomes and suggest that they are not dN sensitive and that they do not emphasise outcomes for patients’s health status/well-being. Rather than comparing your findings with these papers, emphasis where your findings do exactly that. So difference, rather than similarity. More literature could be used to unpack findings in this section. I think this would strengthen your discussion. What would also strengthen your discussion is how these findings could be useful in the global context to improve the quality of DN and community services. You have put this in the conclusion but it might be better here.

Limitations - there may be some limitations in the way the outcomes are written and how they were themed (this isn’t clear) I’d place the acknowledgement that these outcomes are for older people only and therefore some limitation in application to enhance DN service.

Conclusion - its great to hear these have the potential to be included in policy development. Well done!

Reviewer #2: Hello

Thank you for the opportunity to review your manuscript.

I have a few general comments and then some more specific additional comments.

Delphi studies generally require some justification as to how they choose the "expert panel". You have provided some appropriate discussion on this. I am interested to know however whether your panel were appropriate to answer your research questions. Less than half of participants in Round 1 were District Nurses (46.7% as per Table 1). Given the small numbers of participants in your Delphi rounds I think this requires additional discussion in your methods. How did you decide on numbers and sample size? What is your justification for the approach you have used? The rigour of your project relies on asking the questions to the "right" people"? Years of experience in district nursing and current area of waork does not make individuals experts. Some additional context is required to suppiort your choices.

You have used Medians to analyse all data. Do you have a justification for this? Why use Medians rather than Means? I would recommend a methodological reference to support this approach.

Providing the definitions in the Supplementary material was helful. Thankyou.

Presenting the example of Mobility provided clarity about how this was presented. I am not sure however, how "sensitive" mobility is to district nursing care. Our physiotherapist and occupational therapist colleagues would question whether mobility is truly sensitive to nursing care. When we rate the relevance of mobility as a nursing sensitive outcome for district care - what are we rating? Mobility is most frequently considered a characteristic of the person receiving care (and all patients are different with different mobility impairments). Is making someone more mobile a nursing outcome? It might be but I think we are oversimplifying this. I think improving mobility is an appropriate outcome for the healthcare team. What makes it nursing-sensitive?

I can see that an additional 5 outcomes were added after Round 1. What was the process used to define these additional outcomes? They are included as definitions in supplemental material 1 but I am unsure of the source of this definition.

See "Quality of death and dying". The definition here is "Discuss timely the options and take care of counselling in the palliative and terminal phase". This is not a very objective definition and may not actually be about "quality" of death and dying from a patient's perspective. Clarity is needed here.

I also have the following additional comments:

- Line 49. There is a 1 at the end of this first sentence? Is this meant to be a reference?

- Line 125. Nurse-sensitiveness is not an actual word. i would suggest you use 'sensitivity to nursing care' throughout your paper

- The CREDES checklist needs to be updated to reflect the page numbers for the required content (rather than the manuscript section)

There are a number of other manuscripts published that have used Delphi methods to develop a list of potential nursing-sensitive indicators. It may be useful to look at these to guide the reporting of your work.

Kind regards

The Reviewer

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Caroline Dickson

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

3 Feb 2021

REVIEWER(S)' COMMENTS TO AUTHOR

Reviewer: 1

Reviewer name: Caroline Dickson

Your paper is interesting and I enjoyed reading it. Thank you! I did not see the supporting information and perhaps some of my comments are covered in that. I offer you some comments and signposts to strengthen the paper. I hope you find them helpful.

RESPONSE: Thank you for reading this paper carefully and for your critical and helpful comments.

1. For international appeal, identify different similar roles. The definition given does not resonate with the UK DN role, so articulate which country this reflects.

RESPONS: Following your advice, we changed the definition of district nursing by adding the context of district nursing care in the introduction (page 4 line 58-63): “The funding, organisation, definition, and delivery of district nursing care vary between countries worldwide (2–4). For the purpose of this paper, district nursing care is defined as any technical, medical, supportive or rehabilitative nursing care and the provision of assistance with personal care (3). This definition is in line with the definition used for community care nursing in Europe (3,5) and reflects district nursing care in the Netherlands (6).”

2. I don’t know the study is conducted in the Netherlands until p6, so perhaps this needs to be in the aim.

RESPONS: Thank you for your suggestion. Based on this, we have mentioned the Netherlands now in the abstract (page 2 line 28) and introduction (page 4 line 63). We did not add the country to the aim because the study generates insight into nurse-sensitive outcomes potentially suitable in other countries, providing district nursing care.

3. In questionnaire development section, I’d like to understand how these indicators are worded.

a. I know it will be in supplementary information, but perhaps including a box with some examples might help.

b. I understand what you mean by nurse-sensitive outcomes, but I want to be convinced that the outcomes of the indicators are for patients/families living at home.

c. On pg 13 in the Delphi Round table 2, the list on the left hand side (with no title) - are these the indicators? If so, how do e.g. falls or instrumental activities of daily living measure patient experience, health and well-being etc. that you said you wanted to measure? on p17, I can see that the outcomes in the first column would be outcomes to strive for, but in the middle and the last column, would an outcome really be dehydration?

RESPONS: Thank you for your comments.

a. SI Appendix 1 provides all information about the definition of the outcomes. We decided to move the information about the domains to earlier in the document (page 7 line 126-130) to get an idea of what kind of outcomes are identified. We decided not to add an example to the article for it to be clear and short.

b. We want to point out that this study focuses on determining outcomes rather than operationalizing quality indicators. Because this might be unclear until it is explained in the method section, we added the following to the introduction (page 6 line 88-89): “Before quality indicators can be developed and operationalized, it is necessary to determine what outcomes are relevant to measure.” With our study, we focused on which outcomes are nurse-sensitive for patients living at home. The outcomes included at the start of our Delphi round might indeed not be nurse-sensitive for this setting, as this is the questions we addressed to our experts. Since our experts are expert in district nursing care, and our question in the Delphi rounds are formulated with a focus on the nurse-sensitivity for this particular setting, we assume that our findings relate to those patients living at home.

c. Table 2 show the outcomes that are assessed by the experts. We added a title on the left-hand side of table 2 to clarify (page 17, line 263). We first wanted to identify important nurse-sensitive outcomes for district nursing care (i.e. subjects/topics that are important to measure the quality of the delivered care) prior to developing and operationalizing quality indicators (which are used to make outcomes measurable). We also like to underline that there is a difference between the outcome that we want to measure (e.g. presence or degree of dehydration) and the goal we strive for in the care we deliver (absence of dehydration). Further research is needed to operationalise these outcomes to quality indicators, as is described in the discussion section (page 26 line 378-383). To make this more clear and transparent, we added the following sentence to the method section (page 8 line 137-140): “because the study aims to determine what outcomes are nurse-sensitive to district nursing care rather than developing and operationalizing quality indicators, the definitions of the outcomes were not constructed as quality indicators”.

4. I don’t understand the terminology, instrumental activities of daily living.

RESPONS: We described all definitions in SI Appendix 1 because most outcomes are common in (district) nursing care. Instrumental activities of daily living is defined as “The extent to which the patient (together with the people around the patient) is independent in carrying out instrumental activities of daily living (IADL) such as housework, shopping, preparing meals, and making telephone calls”. We hope that we provided sufficient clarification with the definition from SI Appendix 1.

5. Discussion

a. in the background section you have offered some critique on Joling’s and ICHOM use of outcomes and suggest that they are not DN sensitive and that they do not emphasize outcomes for patients’ health status/well-being. Rather than comparing your findings with these papers, emphasis where your findings do exactly that. So difference, rather than similarity. More literature could be used to unpack findings in this section. I think this would strengthen your discussion.

b. What would also strengthen your discussion is how these findings could be useful in the global context to improve the quality of DN and community services. You have put this in the conclusion but it might be better here.

RESPONS: Thank you for your suggestions.

a. Joling and ICHOM identify important quality indicators and outcomes for community-living older people as described in the introduction (page 5 line 76-88). However, Joling and ICHOM do not focus on nurse-sensitive outcomes. To emphasize our study’s focus on nurse-sensitive outcomes, we added our study’s aim at the beginning of the discussion (page 22 line 314-315). We also connected the results of our study to the nursing outcome classification (NOC) by Moorhead et al. (7) to show that all outcomes identified in our study are nurse-sensitive (page 24 line 338-342): “All outcomes identified in our study as nurse-sensitive outcomes for district nursing care are available as nurse outcomes in the nursing outcome classification, except for the outcomes regarding healthcare utilization, which are not included in this classification (7). In our study, healthcare utilization was used as an outcome following other literature (8,9)”. The NOC is developed using various and an extensive amount of literature (7). We added background information about the NOC to the introduction (page 5 line 73-76): “The Nursing Outcome Classification (NOC) provides a set of nursing outcomes that can be used across the care continuum to assess the outcomes of care following nursing interventions (7). However, in this overview, it is unclear what outcomes are relevant for district nursing care”.

b. Because the conclusion is part of the discussion and to prevent repeating information, we decided to leave the usefulness of our finding in the global context in the conclusion.

6. Limitations - there may be some limitations in a. the way the outcomes are written and b. how they were themed (this isn’t clear) I’d place the acknowledgement that these outcomes are for older people only and therefore some limitation in application to enhance DN service.

RESPONS:

a. We agree that this study’s focus on older people may be a limitation of the study. Therefore we added the following sentences to the description of the study’s limitations (page 24 line 362-366): “Additionally, the outcomes used in this study focus on older people which may limit application in district nursing care which also include care for children and middle-aged people. However, 75% of the people receiving district nursing care in The Netherlands is 67 years or older, and the mean age of the people receiving district nursing care is 75 years (10)”.

b. In the previous version of the manuscript, we divided the outcomes in themes used in ICHOM and other references. However, based on reviewer comments and new insights, we have decided to use the NOC domains (7) instead of themes because the nursing outcome classification domains fit better in our study focusing on nurse-sensitive outcomes. We have changed “themes” into “domains” throughout the manuscript and changed the order of the outcomes in table 2 and SI Appendix 1. We added the following to the method section (page 7, line 126-130): “The 41 outcomes were clustered following the domains used in the nursing outcomes classification by Moorhead et al. (7): Functional health (n=4), physiologic health including neurocognitive health (n=16), psychosocial health (n=4), health knowledge and behaviour (n=6), perceived health (n=2), and family health (n=1). Additionally, the domains death (n=2) and healthcare utilization (n=6) were added.”

7. Conclusion - it’s great to hear these have the potential to be included in policy development. Well done!

RESPONS: Thank you very much.

Minor edits

8. p7L119 - what original reference are you referring to?

RESPONS: The original references are described in SI Appendix 1. We added the explanation provided in SI Appendix 1 to the method section as follows (page 8 line 132-140): “Different references were used for defining the outcomes. The outcomes were defined based on the definition used by one references or – in case definitions were incomplete, inconsistent between references, or not suitable for district nursing practice – a combination of multiple references. Because the participants were from the Netherlands, mostly Dutch literature has been used. Because the study aims to determine what outcomes are nurse-sensitive to district nursing care rather than developing and operationalizing quality indicators, the definitions of the outcomes were not constructed as quality indicators”.

9. p7 L120 - If no definition was provided, other references were used - I’m not sure what this means. can you please clarify.

RESPONS: To clarify the process of defining outcomes, we changed the text in the manuscript, as mentioned in our response to reviewer 1 - comment 8. We followed the definitions provided in the original references, for example, the outcome “place of death” was defined following the description described by ICHOM. However, definitions provided were sometimes incomplete, inconsistent between reference, or not suitable for district nursing practice. Therefore, when no reference provided a (suitable) definition, we used multiple references to combine definitions of similar outcomes into one definition to define the outcome. For example, we defined falls as “The presence of fall incidents, where a fall incident is defined as an unintended change of body position that results in a fall on the ground or another lower level” combining the definitions of ICHOM and those provided by Bakker et al. (11). This has been done before by a previous study by van den Bulck et al. (12). With the author’s permission, we used some of the definitions provided by van den Bulck et al. (12). SI Appendix 1 provides an overview of all literature used for identifying and defining the outcomes. We added the sources of the outcomes to table 1 in SI Appendix 1 (page 5-12, line 104).

10. p7 121 - how did the authors decide the themes? Link back to what you think is missing in the outcomes literature

RESPONS: We would like to refer to our response at point 6, where we clarified this point.

11. P8 L130 - 138 - I don’t think this fits here as experts are referred to but they haven’t been defined.

RESPONS: We agree that using the term “experts” may be confusing. Therefore we changed “experts” to “participants” in page 9 line 149-170. After line 170 we decided to use “experts”, following the RAM. After introducing the participants as experts, we called them experts to adhere to the terminology of the RAND/UCLA Appropriateness Method (RAM) User’s Manual (13). We added this to the method section as follows (line 170): “Participants (hereafter referred to as experts) […]”.

12. P8 L 131-

a. who was the letter given to and how were they sampled?

b. Why are they considered experts? these questions cos you haven’t identified experts yet.

RESPONS: We changed the order of the method section to make this more clear.

a. We moved information about providing consent for the study from “questionnaire development” to “data collection” (page 10 line 182-184). Sampling has been described under “identification of experts” (page 9 line 160-176).

b. We understand that the wording of experts may be confusing before the experts are identified. To reduce confusion, we changed “experts” to “participants” before the experts are identified (page 9 line 149-170). After introducing the participants as experts, we called them experts to adhere to the terminology of the RAND/UCLA Appropriateness Method (RAM) User’s Manual (13). We added this to the method section as follows (line 170): “Participants (hereafter referred to as experts) […]”.

13. P8 –

a. were experts from a diversity of services/communities/organisations? Please identify.

b. Demographics of the expert panel on pg 11/12 would be better moved to p8 as it answers these questions. You have achieved a good balance of participants.

RESPONS: Thank you for your compliment regarding the balance of participants.

a. Experts were from a diversity of organisations across the Netherlands. Because we agree that this was not clear, we added the following to the method section (page 10 line 170-171): “Participants (hereafter referred to as experts) from a diversity of organisations across the Netherlands were selected”.

b. We agree it would be helpful to provide the demographics of the experts to the method section. However, to distinguish between study methods and the study results, we decided to leave this information in the results section.

14. p9 L165 - how many outcomes were added after round one and how did the team make the decision to include them?

RESPONS: We agree that we could elaborate more on this subject. Therefore we added the following information regarding proposed and newly added outcomes under the heading Delphi round one, page 10 line 186-193: “New outcomes proposed by the experts were reviewed by a part of the research team (JDV, NB, MJS). The team discussed if the outcomes focused on patient outcomes or were relevant for measuring the quality of care. Decisions were made based on the expertise of the research team. Five outcomes were included in the next round: a meaningful life, duration of district nursing care, the intensity of district nursing care, total time at home, and quality of dying and death. Two outcomes focusing primarily on process or structure of care (providing preventive care and accessibility of district nursing team) were not included. The newly added outcomes were defined using the literature and by insights of the experts.” In the description of round two, the following was added: “Additionally, the definitions of the newly added outcomes, formulated by the research team were discussed and concluded with the experts in the second Delphi round to assure that this corresponded to what the experts initially meant”.

15. p12 L 214 - remind the reader how many outcomes were considered.

RESPONS: We added the total number of outcomes at the beginning of Delphi round 1 (results section) to remind the reader (page 14, line 247-248).

16. p15 - I can’t see C Newly added outcomes after Delphi round one in the table

RESPONS: the newly added outcomes are provided in table 2 and indicated with a C behind the outcome’s names in the first column. This relates to five outcomes included in the table, i.e. a meaningful life, duration of district nursing care, the intensity of district nursing care, total time at home, and quality of dying and death. The outcomes can be found in the table underneath the domains ‘Perceived health’, ‘Death’, and ‘Healthcare consumption’.

17. p17 - Table header hat needs separated

RESPONS: Thank you for your attention to the details. We changed the header’s format in table 3 from centre align to align left to enhance the table's readability.

Reviewer: 2

Reviewer name: not available

Hello,

Thank you for the opportunity to review your manuscript. I have a few general comments and then some more specific additional comments.

RESPONSE: Thank you very much for taking the time to review our paper.

1. Delphi studies generally require some justification as to how they choose the "expert panel". You have provided some appropriate discussion on this. I am interested to know however whether your panel were appropriate to answer your research questions.

a. Less than half of participants in Round 1 were District Nurses (46.7% as per Table 1). Given the small numbers of participants in your Delphi rounds I think this requires additional discussion in your methods. How did you decide on numbers and sample size? What is your justification for the approach you have used?

b. The rigour of your project relies on asking the questions to the "right" people"? Years of experience in district nursing and current area of work does not make individuals experts. Some additional context is required to support your choices.

RESPONS: It is a valid point to check if the sample of experts and choice for experts were appropriate to answer the research question.

a. The sample size has been chosen based on RAM (13), this has been described in the method section (page 10 line 171-173): “Based on the RAM, the aim was to include a panel of 10-15 experts, which would allow the expert panel to have sufficient diversity while also ensuring that all experts would have a chance to participate”. With 15 experts in the first round and 11 in the second round, we complied to this. We elaborate on the choice of the experts in point 1.b below.

b. To support our choices regarding choosing the right people, we added the following to the method section: (page 9 line 165-167): “The aim was to purposively create a balance between participants currently working in district nursing care and those with recent experience in practice yet currently fulfilling a role in research, teaching, practice or policy regarding district nursing care”. The experts were selected based on two criteria: they had to work as a district nurse or have recent experience as a district nurse. Moreover, those who were not currently working as a district nurse were selected given their work regarding district nursing in research, teaching, practice, or policy. In our opinion, 100% of the participants were appropriate to answer the research question. We agree that years of experience does not make a person an expert. Therefore, we added the experts' criteria to have an additional role in research, teaching, practice, or policy. With this, it was assumed that the nurses would be accustomed to critical thinking and reflection, which was necessary given the challenges of defining care outcomes (14). Additionally, we purposively selected participants from a diversity of organisations. This was also added to the method section as follows (page 10 line 170-171): “participants (hereafter referred to as experts) from a diversity of organisations across the Netherlands were selected”.

2. You have used Medians to analyse all data. Do you have a justification for this? Why use Medians rather than Means? I would recommend a methodological reference to support this approach.

RESPONS: We followed the RAM instructions (13), in which they work with medians. It is also common in Delphi studies to analyse data using medians (15). Means are more sensitive to outliers with a small sample size than medians. We agree that it could be more clearly described that the RAM guided us. Therefore, we added the following sentence to the data analysis section, page 12 line 212: “All analyses were guided by the RAM.”

3. Providing the definitions in the Supplementary material was helpful. Thank you. Presenting the example of Mobility provided clarity about how this was presented. I am not sure however, how "sensitive" mobility is to district nursing care. Our physiotherapist and occupational therapist colleagues would question whether mobility is truly sensitive to nursing care. When we rate the relevance of mobility as a nursing sensitive outcome for district care - what are we rating? Mobility is most frequently considered a characteristic of the person receiving care (and all patients are different with different mobility impairments). Is making someone more mobile a nursing outcome? It might be but I think we are oversimplifying this. I think improving mobility is an appropriate outcome for the healthcare team. What makes it nursing-sensitive?

RESPONS: This is a valid point. We agree that the identified outcomes are not only sensitive to nursing interventions but may also be relevant outcomes for other disciplines or the whole healthcare team. This reflects in the median scores: the experts agree that the outcomes are nurse-sensitive (with a median of 7 or 8 regarding the outcomes' relevance and influenceability), but none of the outcomes is assessed as completely nurse-sensitive (median 9). We believe that this is in line with district nursing practice, in which nurses fulfil a multidisciplinary role within primary care. Care for community-living older people is provided by various professionals, such as district nurses, the general practitioner, physiotherapist, occupational therapist, or other disciplines. Therefore, most of the outcomes are indeed often not completely influenceable by the delivered district nursing care only. We believe that this reflection is essential to share. Therefore we added this point to the discussion section, page 26 line 387: “Lastly, careful consideration is needed regarding the influenceability of the outcomes. None of these outcomes was assessed as completely relevant or influenceable (median 9), the uncertainty of the influenceability of the outcomes is relatively high (34,8%) and the overall medians of the influenceability of the outcomes are lower compared to the assessment of the relevance. This could be explained by the multidisciplinary role of district nurses in practice. Care for community-living older people is not only provided by district nurses, but also by the general practitioner and other (paramedic) professionals in primary care. Most of the outcomes are indeed often not completely influenceable by the delivered district nursing care. Coordinated care by interdisciplinary teams is associated with better outcomes regarding hospitalizations, emergency department visits, and long-term care admissions in community-living people (16). Therefore, close collaboration between professionals in district nursing practice is needed to influence and achieve the best possible the outcomes for clients who receive district nursing care”. Additionally, we put emphasis on other disciplines in the introduction, page 4 line 56-58: “Professional care assistance at home is provided through district nursing care, next to other healthcare professionals such as the general practitioner and other (paramedic) professionals in primary care (16)”.

4. I can see that an additional 5 outcomes were added after Round 1. What was the process used to define these additional outcomes? They are included as definitions in supplemental material 1 but I am unsure of the source of this definition.

RESPONS: The experts proposed the new outcomes during the first online questionnaire. The expert explained the newly proposed outcomes in this questionnaire. Based on this, the research team defined the outcomes using literature. During the expert meeting, the definitions were discussed with the experts. To explain and justify the choices made, we added the following to the method section (page 10 line 186-193): “New outcomes proposed by the experts were reviewed by a part of the research team (JDV, NB, MJS). The team discussed if the outcomes focused on patient outcomes or were relevant for measuring the quality of care. Decisions were made based on the expertise of the research team. […] The newly added outcomes were defined using the literature (SI Appendix 1)”. Additionally, we added the following sentence to the description of round 2 (page 11 line 202-204): “Additionally, the definitions of the newly added outcomes, formulated by the research team were discussed and concluded with the experts in the second Delphi round to assure that this corresponded to what the experts initially meant”.

5. See "Quality of death and dying". The definition here is "Discuss timely the options and take care of counselling in the palliative and terminal phase". This is not a very objective definition and may not actually be about "quality" of death and dying from a patient's perspective. Clarity is needed here.

RESPONS: We agree that this definition should be studied further. As explained above, the definitions were based on the literature and insights from the experts. Regarding this outcome, the experts' insights and opinions were used because there is a lack of clarity and consistency in the literature regarding this construct (17). However, we agree that the experts in our study might not be experts regarding the subject “quality of death and dying”. In line with this, we would like to emphasise that the study aims to identify important outcomes for district nursing care. The operationalisation of the outcomes is not determined in this study and requires further research. To underline this limitation, we added the following to the limitations in the discussion section (page 25 line 357-361): “Last, the identification and definitions of the outcomes have some limitations. It is possible that outcomes and quality indicators were missed since no systematic review has been conducted. This risk was minimised by letting experts add and define missing outcomes. However, the definitions by the experts may not be comprehensive and requires further research”.

Minor edits

6. Line 49. There is a 1 at the end of this first sentence? Is this meant to be a reference?

RESPONS: Thank you for your attention regarding the details. The 1 at the end of the sentence is indeed supposed to be a reference. We changed it to the correct formatting.

7. Line 125. Nurse-sensitiveness is not an actual word. i would suggest you use 'sensitivity to nursing care' throughout your paper

RESPONS: We changed “nurse-sensitiveness” to “sensitivity to nursing care”.

8. The CREDES checklist needs to be updated to reflect the page numbers for the required content (rather than the manuscript section)

RESPONS: Thank you for pointing this out to us. We updated the CREDES checklist to reflect the page numbers.

9. There are a number of other manuscripts published that have used Delphi methods to develop a list of potential nursing-sensitive indicators. It may be useful to look at these to guide the reporting of your work.

RESPONS: Thank you very much for your suggestion. We looked at other manuscripts to guide the reporting of our work in combination with the guidelines for Conducting and Reporting Delphi Studies (CREDES) (1).

References:

1. Jünger, S; Payne, SA; Brine, J; Radbruch, L; Brearley S. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine. 2017;31(8):684–706.

2. Jarrin, OF; Pouladi, FA; Madigan E. International priorities for home care education, research, practice, and management: Qualitative content analysis. Nurse Education Today. 2019;73:83–7.

3. Van Eenoo L, Declercq A, Onder G, Finne-Soveri H, Garms-Homolova V, Jonsson PV, e.a. Substantial between-country differences in organising community care for older people in Europe—a review. The European Journal of Public Health. 2016;26(2):213–9.

4. World Health Organization. Home care across Europe: current structure and future challenges. World Health Organization. Regional Office for Europe; 2012.

5. Tarricone R, Tsouros AD. Home care in Europe: the solid facts. WHO Regional Office Europe; 2008.

6. Maurits EEM. Autonomy of nursing staff and the attractiveness of working in home care [PhD Thesis]. Utrecht University; 2019.

7. Moorhead S, Johnson M, Maas ML, Swanson E. Nursing Outcomes Classification (NOC)-e-book: Measurement of health outcomes. Elsevier Health Sciences; 2018.

8. Joling KJ, Van Eenoo L, Vetrano DL, Smaardijk VR, Declercq A, Onder G, e.a. Quality indicators for community care for older people: A systematic review. PloS one. 2018;13(1):e0190298.

9. Akpan A, Roberts C, Bandeen-Roche K, Batty B, Bausewein C, Bell D, e.a. Standard set of health outcome measures for older persons. BMC geriatrics. 2018;18(1):36.

10. Vektis. Factsheet Wijkverpleging [Internet]. 2020 [geciteerd 4 januari 2021]. Beschikbaar op: https://www.vektis.nl/intelligence/publicaties/factsheet-wijkverpleging

11. Bakker AJEM, Habes V, Quist G, others. Klinisch redeneren bij ouderen: functiebehoud in levensloopperspectief. Bohn Stafleu van Loghum; 2016.

12. van den Bulck AO, Metzelthin SF, Elissen AM, Stadlander MC, Stam JE, Wallinga G, e.a. Which client characteristics predict home-care needs? Results of a survey study among Dutch home-care nurses. Health & Social Care in the Community. 2019;27(1):93–104.

13. Fitch, K; Bernstein, SJ; Aguilar, MD; Burnand, B; LaCalle, JR; Lázaro, P; van het Loo, M; McDonnell, J; Vader, JP; Kahan J. The RAND/UCLA Appropriateness Method User’s Manual. 2001.

14. Doran D. Nursing outcomes. Jones & Bartlett Learning; 2010.

15. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PloS one. 2011;6(6):e20476.

16. Stall N, Nowaczynski M, Sinha SK. Systematic review of outcomes from home-based primary care programs for homebound older adults. Journal of the American Geriatrics Society. 2014;62(12):2243–51.

17. Hales S, Zimmermann C, Rodin G. The quality of dying and death: a systematic review of measures. Palliative Medicine. 2010;24(2):127–44.

Submitted filename: 210113 Response to Reviewers_Delphi_DN_Outcomes_jdv.docx

Click here for additional data file.

29 Apr 2021

Nurse-sensitive outcomes in district nursing care: a Delphi study

PONE-D-20-16928R1

Dear Dr. Veldhuizen,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Fiona Cuthill, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you very much for addressing my comments and re-submitting your manuscript.

I wish you well with your work and hope the paper achieves some impact.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr Caroline Dickson

3 May 2021

PONE-D-20-16928R1

Nurse-sensitive outcomes in district nursing care: a Delphi study

Dear Dr. Veldhuizen:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Fiona Cuthill

Academic Editor

PLOS ONE