Development and pilot testing of quality improvement indicators for integrated primary dementia care.

BACKGROUND: Implementation of integrated primary care is considered an important strategy to overcome fragmentation and improve quality of dementia care. However, current quality indicator (QI) sets, to assess and improve quality of care, do not address the interprofessional context. The aim of this research was to construct a feasible and content-wise valid minimum dataset (MDS) to measure the quality of integrated primary dementia care.
METHODS: A modified Delphi method in four rounds was performed. Stakeholders (n=15) (1) developed a preliminary QI set and (2) assessed relevance and feasibility of QIs via a survey (n=84); thereafter, (3) results were discussed for content validity during a stakeholder and (4) expert consensus meeting (n=8 and n=7, respectively). The stakeholders were professionals, informal caregivers, and care organisation managers or policy officers; the experts were professionals and researchers. The final set was pilot-tested for feasibility by multidisciplinary dementia care networks.
RESULTS: The preliminary set consisted of 40 QIs. In the survey, mean scores for relevance ranged from 5.8 (SD=2.7) to 8.5 (SD=0.7) on a 9-point Likert scale, and 25% of all QIs were considered feasible to collect. Consensus panels reduced the set to 15 QIs to be used for pilot testing: 5 quality of care, 3 well-being, 4 network-based care, and 3 cost-efficiency QIs. During pilot testing, all QIs were fully completed, except for well-being QIs.
CONCLUSION: A valid and feasible MDS of QIs for primary dementia care was developed, containing innovative QIs on well-being, network-based care and cost-efficiency, in addition to quality of care QIs. Application of the MDS may contribute to development and implementation of integrated care service delivery for primary dementia care. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

The complexity of primary dementia care, caused by the complex nature of this clinical syndrome, often associated comorbidity, and the multiple professionals involved poses a great challenge to healthcare systems.1 2 As a result, dementia care is often fragmented, which leads to poor quality and inefficient care, because different professionals provide unaligned services, resulting in patients’ and caregivers’ low satisfaction with the services offered.3–5 Care integration is considered important to overcome fragmentation and improve quality of care.6–9

Integrated care is defined as the delivery of a continuum of care, designed to meet multidimensional needs of the population and the individual, by a coordinated multidisciplinary team of professionals.10 To achieve dementia care integration, a transition towards network-based care is needed.11–13 DementiaNet, a healthcare innovation consisting of 36 multidisciplinary networks of primary care professionals, facilitates such a transition.11 14 The essential characteristics of DementiaNet are summarised in box 1. To evaluate the effectiveness of such an integrated care approach in practice and to identify areas for improvement, a tailored, feasible and valid set of quality indicators (QIs) is needed.15 This need is based on extensive literature search and expert consensus on QIs based on patient-reported outcomes of physiotherapy and geriatric patient care showing that, when connected firmly with feedback loops to the healthcare, it can improve outcomes and efficiency on the levels of direct patient care and health services.16 Several QI sets have been developed to assess quality of primary dementia care, including a set of six QIs used to assess DementiaNet networks’ quality of care.17–21 However, these sets do not sufficiently adopt the interprofessional context. Furthermore, existing integrated QI sets do not include well-being issues,22 or their large number of indicators limits feasibility.23 Therefore, the aims of this study were to construct a content-wise valid minimum dataset (MDS) of QIs to measure quality of integrated primary dementia care in co-creation with stakeholders and to pilot test its feasibility in practice.

DementiaNet is a Dutch healthcare innovation focusing on primary network-based care. Networks of professionals from medical, care and social disciplines, caring for the same persons with dementia (PWDs), are formed.14 Currently, the DementiaNet approach is being applied in more than 40 networks. DementiaNet facilitates gradual development of self-organised primary care networks providing high-quality integrated dementia care. Strategies to achieve this include clinical leadership and quality improvement cycles. With DementiaNet, we address needs of professionals (increase knowledge, skills and collaboration); PWD and informal caregivers (personalised care, integration of care and welfare, care coordination and continuity of care); and the healthcare system (local level, tailor-made, system approach, cost-effectiveness and sustainable approach). Full description can be found in the paper of Nieuwboer et al,14 and the first quantitative evaluation of effectiveness is described by Richters et al.21

Methods

Design

Between May 2018 and July 2019, a modified Delphi method was conducted. This methodology is particularly useful to reach agreement on content validity and feasibility; inclusion of a large number of participants prevents domination during the consensus process.24 25 In four rounds, we (1) developed a preliminary QI set, (2) sent out a survey to stakeholders, (3) discussed results during a stakeholder consensus meeting and (4) discussed results during an expert consensus meeting. Thereafter, the final set was pilot-tested for feasibility.

Delphi method

Participants

Participants were stakeholders and experts in the field of primary dementia care. Stakeholders were healthcare professionals, informal caregivers, and care organisation managers or policy officers. Experts were professionals and researchers, including all authors, specialised in research or care. Professionals from each discipline were balanced for each round.

Stakeholders for the first physical meeting (round 1) and survey (round 2) were invited by email and recruited via the DementiaNet newsletter database to ensure their familiarity with integrated primary dementia care. Additional caregivers for the survey were recruited via the outpatient clinic or through project contacts. Stakeholders interested to participate in round 1 and participants who showed interest during round 2 were invited for the second physical meeting (round 3). Purposeful sampling was used for the meetings (rounds 1 and 3); we made sure all relevant primary care disciplines were represented, and the representatives were seen as experts by their colleagues. To include non-expert opinions as well, random sampling of a large group of stakeholders was used for the survey (round 2).

Participants for the expert consensus meeting (round 4) were selected based on their involvement in the DementiaNet project; all were employees at the Radboudumc Alzheimer Centre or at the General Practitioner Organisation of Nijmegen.

Delphi rounds

1. Development of the preliminary set

In a brainstorm session with stakeholders, relevant and easy-to-implement indicators were pre-selected, covering three predefined topics: (1) quality of dementia care (process indicators, patient level); (2) well-being of person with dementia (PWD) and their caregiver (outcome indicators)15 26; and (3) network-based care (process indicators network level).27 Additionally, the existing quality of care indicators used by DementiaNet networks were included in the discussion.11 21 Afterwards, the authors reviewed previous QI sets developed by their group17 21 and subsequently composed the preliminary set for the stakeholder survey.

2. Survey to stakeholders

In an electronic survey, the preliminary QI set was presented to stakeholders. Consent was asked at the beginning of the survey. Caregivers were only asked to comment on quality of care and well-being indicators. A glossary was included to explain medical jargon. Caregivers could request a paper version of the survey.

Stakeholders were asked to rate all indicators individually on relevance by using a 9-point Likert scale from 1 (not relevant) to 9 (highly relevant), to identify the three most important indicators per topic and to add missing indicators. Professionals were also asked to assess QIs on measurement feasibility (yes/no).

Coded questionnaire responses of professionals and caregivers were analysed separately to determine the mean relevance, top 3 indicators and feasibility of all indicators. An indicator was judged feasible when over 50% of respondents indicated the data feasible to collect.28 Descriptive analyses were performed using IBM SPSS V.25.0.

3. Stakeholder consensus meeting

Survey results, for professionals and caregivers separately, were presented to a stakeholder panel for their judgement of content validity. Inclusion of additional indicators, revision of QIs, and exclusion based on relevance and feasibility were discussed.

4. Expert consensus meeting

Results of the survey and stakeholder consensus meeting were distributed to the experts a week prior to the meeting. The expert panel composed the MDS for pilot testing, based on the rated relevance, arguments from the consensus meeting, content validity, feasibility and correspondence with current guidelines.

Pilot testing

DementiaNet networks were selected to pilot the new set of indicators (MDS) based on their proactive attitude and timing of their networks’ yearly assessment.11

Pilot test

Networks reported their experiences via an open-ended questionnaire, including QI relevance, feasibility of data collection and added value of outcomes.

The authors narratively analysed the questionnaires and evaluated the collected data to determine measurement feasibility. Acceptable measurement feasibility was defined as data collection per QI completed for over 80% of the cases. Thereafter, the set of indicators was adjusted accordingly.

Results

We invited 312 stakeholders for round 1; 19 were willing to participate, and 15 were able to participate. Thirty-one were invited for round 3; 8 were able to participate. Four stakeholders participated in both rounds. Diversity in stakeholders’ characteristics was large, with general practitioners, practice nurses, case managers, community nurses and caregivers being most represented. All invited experts agreed to participate. Table 1 describes participants’ characteristics.

Table 1

Participants’ background and years of experience in numbers (and percentages for round 2) defined per round

Characteristic	Round 1	Round 2	Round 3	Round 4
Number of participants	15	114	8	7
Female, n (%)	11	100 (88%)	6	6
Dominant background, n (%)
Researcher	–	1 (1%)	–	3
Healthcare professional	8	94 (82%)	5	2
General practitioner	1	10 (9%)	1	1
Practice nurse	1	12 (11%)	–	–
Case manager	2	21 (18%)	1	–
Community nurse	1	29 (25%)	1	–
Other	3	22 (19%)	2	1
Manager/policy officer	5	7 (6%)	–	2
Informal caregiver	2	12 (11%)	3	–
Years of experience dominant background, n (%)
<5	5	16 (17%)*	1†	1
5–10	5	25 (27%)*	3†	–
>10	5	51 (56%)*	3†	6

*Years of experience of professionals and managers/policy officers; n=92.

†n=7.

Indicator development

Figure 1 summarises the process of QI development.

Figure 1

Subsequent stages of the QI development process for integrated primary dementia care. QI, quality indicator; QoC, quality of care.

1. Development of the preliminary set

Fifteen stakeholders developed the preliminary QI set. The meeting yielded a set of 40 indicators; 11 for quality of care, 12 for well-being of PWD and their caregiver and 17 for network-based care, including all 6 QIs previously used by DementiaNet.11 21 No indicators were added by the authors after reviewing previously developed QI sets.

2. Stakeholder survey

Eighty-four stakeholders fully, and 30 stakeholders partly completed the assessment of the 40 QIs (table 1). Mean relevance scores per indicator ranged from 6.0 (SD=2.1) to 8.5 (SD=0.7) for professionals and from 5.8 (SD=2.7) to 8.3 (SD=0.9) for caregivers. Professionals considered collection of 25% of the indicators feasible. Table 2 shows relevance, top 3 and feasibility scores per indicator.

Table 2

Relevance (by professionals and caregivers) and feasibility (by professionals and managers/policy officers) of QIs to measure integrated primary dementia care assessed via a survey and final consensus

Category	Short notation QI*	Healthcare professionals and managers/policy officers						Informal caregivers				Final consensus, yes/no	Remark
		n	Relevance, mean (SD)	n	Priority, n (%)	n	Feasibility, yes/no	n	Relevance, mean (SD)	n	Priority, n (%)
Quality of care	Assessment of the caregivers’ care	101	8.2 (1.1)	99	28 (9%)	87	Yes	11	7.8 (0.8)	36	4 (11%)	No
	Case manager involved	102	8.3 (0.9)	99	53 (18%)	90	Yes	11	7.3 (2.9)	36	6 (17%)	Yes
	Future treatment wishes discussed with PWD and caregiver	101	7.9 (1.2)	99	39 (13%)	88	Yes	11	8.1 (0.7)	36	3 (8%)	Yes
	Geriatric assessment	102	7.0 (1.8)	99	16 (5%)	88	No	11	7.4 (1.6)	36	4 (11%)	No
	Goals of the caregiver discussed	100	8.0 (1.0)	99	17 (6%)	87	No	11	6.1 (2.9)	36	0	No
	Goals of the PWD discussed	100	7.6 (1.5)	99	29 (10%)	87	Yes	11	5.9 (3.0)	36	1 (3%)	No
	PWD discussed in a multidisciplinary meeting	102	7.6 (1.5)	99	42 (14%)	88	Yes	11	7.0 (2.9)	36	4 (11%)	Yes
	PWD discussed in a pharmacotherapeutic meeting	102	7.05 (1.9)	99	6 (2%)	88	No	11	7.0 (2.2)	36	3 (8%)	Yes
	Psychoeducation offered to caregiver	100	7.7 (1.5)	99	18 (6%)	87	No	12	6.5 (2.7)	36	4 (11%)	No
	Support offered to caregiver	99	7.8 (1.4)	99	26 (9%)	86	Yes	12	6.7 (2.3)	36	2 (6%)	Yes
	Changing relationships discussed with PWD and caregiver	NA						NA				No	Added round 3
Well-being	Caregiver feels supported	83	8.5 (0.7)	83	48 (19%)	80	No	12	7.8 (1.0)	35	4 (11%)	No
	Caregiver involved in care PWD	83	8.2 (1.2)	83	22 (9%)	80	No	12	8.3 (0.9)	35	6 (17%)	No
	Caregiver involved in own care	83	8.1 (1.3)	83	10 (4%)	80	No	9	8.2 (0.8)	NA		No
	Caregivers knows involved professionals	83	8.1 (1.0)	83	16 (6%)	80	No	12	7.7 (1.3)	35	3 (9%)	No
	Formulated goal caregiver	83	7.0 (1.6)	83	0	80	No	NA				No
	Formulated goal for PWD	83	6.0 (2.1)	83	3 (1%)	81	No	NA				No
	PWD feels supported	83	8.3 (1.1)	83	60 (24%)	80	No	12	7.5 (2.3)	35	7 (20%)	No
	PWD involved in own care	83	8.0 (1.1)	83	21 (8%)	80	No	12	7.0 (2.3)	35	2 (6%)	No
	PWD knows contact professional	83	7.8 (1.6)	83	20 (8%)	80	No	12	7.1 (2.2)	35	3 (9%)	No
	Perseverance time of the caregiver	83	8.2 (1.3)	83	33 (13%)	80	No	12	7.7 (2.2)	35	6 (17%)	Yes
	Psychoactive medication for more than 2 weeks	83	6.0 (2.1)	83	5 (2%)	80	Yes	12	5.8 (2.7)	35	1 (3%)	No
	PWD satisfied with his/her daily activities	NA						NA				Yes	Added round 3
	Well-being related goal for PWD	NA						NA				Yes	Added round 4
Network-based care	Activity to involve broader network (eg, PWD and caregivers, and dementia initiatives)	78	7.0 (1.7)	77	5 (2%)	74	No	12	7.5 (2.0)	NA		No
	Agreements with municipality on medical support indications	78	7.4 (1.6)	77	11 (5%)	75	No	NA				No
	Case review with network	78	7.0 (1.8)	77	10 (4%)	74	No	NA				No
	Discussion of mutual interests	78	6.8 (1.7)	77	4 (2%)	74	No	NA				No
	Discussion of mutual personal relationships	77	6.4 (1.6)	77	1 (0%)	74	No	NA				No
	Document present with expertise and contact information of all professionals	78	7.9 (1.1)	77	12 (5%)	75	No	NA				Yes	Merged round 4
	Informal activity with network	78	6.4 (2.0)	77	1 (0%)	75	No	NA				Yes
	Interprofessional education together with network	78	7.1 (1.8)	77	6 (3%)	75	No	NA				No
	Network agreements on early signalling	78	7.4 (1.8)	77	13 (6%)	75	No	NA				No
	Network goals discussed	78	7.1 (1.7)	77	16 (7%)	74	No	NA				No
	Network has one or two network leaders	79	7.6 (1.5)	77	25 (11%)	75	No	NA				Yes	Merged round 4
	Network worked on at least one challenge or celebrated one success	78	6.8 (2.0)	77	9 (4%)	74	No	NA				Yes
	Network working arrangements discussed	78	7.2 (1.4)	77	12 (5%)	74	No	NA				No
	PWD and caregiver consulted on experience with network-based care	77	7.1 (1.6)	77	10 (4%)	74	No	12	8.2 (0.6)	NA		No
	Representative of a welfare organisation present	79	8.0 (1.3)	77	44 (19%)	75	Yes	NA				Yes
	Representative of PWD or caregiver present in network	79	7.5 (1.7)	77	30 (13%)	75	No	12	7.7 (2.1)	NA		No
	Work agreement for the shared caseload	78	7.7 (1.3)	77	22 (10%)	75	No	NA				Yes	Merged round 4
Cost-efficiency	Diagnosed in the primary setting	101	6.0 (2.4)	99	23 (7.74%)	90	Yes	11	6.0 (2.8)	36	5 (14%)	Yes
	Number of emergency consultations	83	6.0 (2.3)	83	11 (4%)	80	Yes	12	6.3 (3.2)	35	3 (9%)	Yes
	PWD care is reimbursed from the long-term care finance system	NA						NA				Yes	Added round 4

*All QIs concern the past 12 months.

NA, not available; PWD, person with dementia; QI, quality indicator.

3. Stakeholder consensus meeting

Eight stakeholders discussed the survey results. Stakeholders generally agreed with the survey respondents on relevance and prioritisation. Disagreement existed on the QI ‘recent pharmacotherapy check’, which had a low relevance and feasibility score, though stakeholders considered the indicator crucial for assessing quality of care. They argued that polypharmacy negatively influences quality of life and morbidity. Two new well-being QIs were proposed for inclusion: ‘discussing changing relationship between PWD and caregiver’ and ‘PWD satisfaction with daily activities’. The QI about PWDs’ goals was nominated for exclusion as it was considered not specific and not feasible.

4. Expert consensus meeting

A geriatrician, two general practitioners, a nursing scientist, two researchers and a coordinator of General Practitioner Organisation of Nijmegen decided on the composition of the MDS for pilot testing considering primarily relevance but also feasibility. The panel agreed with the stakeholders on adding the QI on PWD satisfaction with daily activities. Three network QIs about task division and communication were merged. Experts added the QI: ‘whether the network has formulated a well-being related goal for the PWD’, which replaced some of the excluded network and well-being QIs. During the discussion, cost-efficiency was added as a separate topic, and QIs were arranged accordingly. An additional QI was included for cost-efficiency: ‘whether care of PWD is reimbursed from the long-term care finance system’. Twenty-six QIs were excluded, resulting in a pilot-testing set of 15 QIs: five quality of care, three well-being, four network and three cost-efficiency QIs.

Three DementiaNet networks voluntarily pilot-tested the QI set. Regarding relevance and added value, they preferred the new set above their currently used QIs, as they stated that it represented more aspects of the network collaboration. This information helped them to identify new improvement goals. The QIs focusing on PWD care wishes and on support for caregivers were considered of added value. Measurement feasibility was good; it took one to two hours, depending on the size of the caseload, to complete the QIs. QIs were fully completed, except for the well-being QIs; on average, these were completed for 14% (perseverance time) and 42% (satisfaction with daily activities) of their PWD (range: 11–31). These well-being indicators were difficult to collect as they required caregiver consultation as an additional time-consuming action. Participants suggested the case manager should complete these questions. Furthermore, pilot answers suggested ambiguity of two QIs, which were therefore reformulated (table 3).

Table 3

Final set of QI descriptions for integrated primary dementia care

Category	QI description
Quality of care	A case manager is involved.
	The PWD has been discussed in a multidisciplinary meeting in the past year.
	The PWD has been discussed in a pharmacotherapeutic meeting in the past year.
	Future treatment wishes were discussed with the PWD and their informal caregiver in the past year.
	Support is offered to the informal caregiver in the past year.
Well-being	Assessment of the perseverance time of the informal caregiver.
	Informal caregiver states that PWD is satisfied with his/her daily activities.
	The network has set at least one well-being related goal for the PWD.
Network-based care	At least one representative of a welfare organisation is part of the network.
	Task division and communication:1. There is a document with expertise and contact information of all the healthcare professionals available for the network in caring for frail older adults, which is updated yearly.2. The network has one or two network leaders.3. The network has composed written working agreement for their shared caseload of PWDs.
	The network worked on at least one new challenge or celebrated one success in the past year.
	There was an informal activity for network participants in the past year.
Cost-efficiency	The PWD is diagnosed in the primary setting.
	Number of emergency consultations in the past year.
	PWD care is reimbursed from the long-term care finance system.

PWD, person with dementia; QI, quality indicator.

Discussion

An MDS of 15 QIs was developed to measure the quality of integrated primary dementia care; QIs for quality of care, well-being, network-based care, and cost-efficiency were included. Most QIs were overall judged positively regarding content validity and feasibility by primary dementia care stakeholders. During pilot testing, most indicators were considered relevant, of added value and feasible to collect. However, collection of the well-being QIs still needs specific attention.

Strengths and limitations

An important novelty of this set is the inclusion of well-being, network-based care and cost-efficiency indicators, which supports the triple-aim ambitions; improve quality of care, population health and cost-effectiveness.29 Previous sets mainly included quality of care indicators.19 Moreover, this novel MDS for integrated primary dementia care is substantially shorter than existing sets.23 The comprehensiveness of the set, in combination with its high relevance and feasibility, make the MDS useful for daily practice; based on the pilot test future acceptance is expected to be high.

Another strength of the set is its fit with the validated Rainbow Model of Integrated Care.27 This model’s four levels of care provision, the service, professional, organisational and system levels, are all represented in the MDS. QIs on quality of care and well-being refer to the service level: processes and outcomes of care for the PWD and caregiver (eg, QIs on multidisciplinary meetings and satisfaction of PWD with daily activities). Network QIs refer to the professional and organisational levels, the required collaboration between professionals and organisations (eg, QIs on task division and communication). Cost-efficiency QIs refer to the system level, the impact of regulations on collaboration between professionals and organisations (eg, QI on PWD care reimbursement).

Last, stakeholder meetings and the survey included a diverse group of participants regarding background, age and informal caregiver’s roles. Their interaction and discussions created broad support for the QIs included; acceptability of the MDS is therefore expected to be high. However, the stakeholders were mainly active in DementiaNet networks; the MDS is therefore highly applicable to the Dutch network-based dementia care context. Adaptations to the MDS might be necessary when using the set for other healthcare systems in other countries.30 However, because of their generic content, the QIs can be highly transferable to other populations, for example, vulnerable older adults. Collection of the well-being QIs proved difficult. Evaluation of new strategies to collect these QIs is needed since their relevance is well recognised in practice.

Future implications for research and practice

By addressing various levels of integrated care, the MDS is highly useful for performance feedback in an interdisciplinary setting to encourage professionals to improve the quality of services for their joint caseload. Care coordination, interdisciplinary teamwork and personalised care can be improved by setting improvement goals and assess change over time. Moreover, the complexity in dementia care is captured by clustering QIs in levels related to each other; better network collaboration leads to better quality of care, better patient and caregiver well-being and/or lower healthcare costs.

The MDS can inform policy makers on best practices to achieve (cost-)effective network-based dementia care. As most QIs in the MDS are not dementia specific, it is applicable in other populations, for example, vulnerable older adults, which will increase acceptability by preventing fragmentation due to disease-specific QIs.

Future research should include the use of the MDS in long-term follow-up evaluations of integrated primary dementia care programmes such as DementiaNet. However, first reliability testing and assessing sensitivity for relevant change are needed.

Conclusion

A concise set of QIs with proven feasibility and content validity was developed to assess integrated primary dementia care. Innovative QIs on different relevant aspects of integrated care, well-being, network-based care and cost-efficiency were included next to quality of care QIs. By monitoring changes in QI scores over time and subsequent care improvement cycles, professionals in interdisciplinary primary dementia care collaborations can improve the quality of service for their joint caseload. As such, the set may contribute to the implementation of guidelines and care pathways for integrated primary dementia care.