Creating effective quality-improvement collaboratives: a multiple case study.

OBJECTIVE: To explore whether differences between collaboratives with respect to type of topic, type of targets, measures (systems) are also reflected in the degree of effectiveness. STUDY
SETTING: 182 teams from long-term healthcare organisation developed improvement initiatives in seven quality-improvement collaboratives (QICs) focusing on patient safety and autonomy. STUDY
DESIGN: Multiple case before-after study. DATA COLLECTION: 75 team leaders completed a written questionnaire at the end of each QIC on achievability and degree of challenge of targets and measurability of progress. Main outcome indicators were collaborative-specific measures (such as prevalence of pressure ulcers). PRINCIPAL
FINDINGS: The degree of effectiveness and percentage of teams realising targets varied between collaboratives. Collaboratives also varied widely in perceived measurability (F=6.798 and p=0.000) and with respect to formulating achievable targets (F=6.566 and p=0.000). The Problem Behaviour collaborative scored significantly lower than all other collaboratives on both dimensions. The collaborative on Autonomy and control scored significantly lower on measurability than the other collaboratives. Topics for which there are best practices and evidence of effective interventions do not necessarily score higher on effectiveness, measurability, achievable and challenging targets.
CONCLUSIONS: The effectiveness of a QIC is associated with the efforts of programme managers to create conditions that provide insight into which changes in processes of care and in client outcomes have been made. Measurability is not an inherent property of the improvement topic. Rather, creating measurability and formulating challenging and achievable targets is one of the crucial tasks for programme managers of QICs.

Introduction

There is a strong demand for quality improvement, which is often pursued by setting up quality-improvement collaboratives (QICs). Some studies on QICs suggest positive effects in participating organisation, but results are mixed.1–4 Moreover, most studies evaluating QICs investigate their effectiveness rather than describing how the collaborative was designed by programme managers and how conditions for establishing effectiveness were constructed, which seems crucial for interpreting the effectiveness results. QICs differ greatly in topic, improvement methods3 and functioning in different countries and healthcare sectors, which is suggested to influence their effectiveness.5 6 Øvretveit et al3 7 suggest that ‘success’ depends on the ways in which programme managers of QICs deal with challenges interwoven with the topic under study.

A first challenge for collaborative organisers is selecting the topic or quality problem.3 5 It has been suggested that the availability of best practices and evidence of effective interventions for selected topics is essential for creating effective collaboratives.3 Evidence of this suggestion is however lacking.

A second crucial challenge is related to teams' targets, which should be both challenging and achievable.3 Collaborative organisers may consider results from best practices to define what is achievable. The degree to which they establish the right balance between challenge and perceived achievability may determine the effectiveness of the improvement efforts. Moreover, for teams to be successful in the end, they should define their targets early and measure progress regularly.3 This leads us to the third challenge: measurability.

Improvements in processes of care are to be achieved by Plan–Do–Study–Act cycles: stepwise changes to care practices guided by measured results for which appropriate measures and usable data-collection tools should be available. Although the philosophy of Breakthrough is that teams set local goals and report on local indicators, agreeing on a common set of measures which all teams will register enables teams to learn from each other and keeps teams focused on the collaborative target. As effectiveness is based both on the interventions carried out and on the way the improvements are measured,8 it is crucial to explore to what extent collaborative organisers succeed in creating measurability.

Until now, no empirical studies were found comparing different collaboratives to investigate to what extent programme managers meet these challenges in creating effectiveness and how this is associated with variations in success of collaboratives. Our study involves an evaluation of a quality-improvement programme for the long-term care in The Netherlands, called Care for Better, and deals with seven QICs focusing on patient safety and client autonomy. The aim of this multiple case study is to explore whether differences between collaboratives with respect to type of topic, type of targets and measures (systems) are also reflected in the degree of effectiveness.

Methods

Setting and design

This study included quality-improvement teams from nursing and residential care homes, home care, and care for mentally and physically disabled participating in a QIC.9 Programme management was in the hands of the long-term care knowledge institute Vilans. The Breakthrough method10 including use of Plan–Do–Study–Act cycles and small-scale testing was used.

Data collection

Outcome data on targeted measures

Table 1 gives an overview of the target, targeted measures, type of measurement and level of data collection of each collaborative as formulated by programme management (more information in appendix 1). In total, 182 teams participated in one of the collaboratives with one or more pilot wards or locations in the same collaborative (n=233 pilot locations). Complete baseline and end-measurement data were available for 139 locations.

Table 1

Overview of care for better collaboratives

Collaborative	Level of evidence	Target	Measures	Pilot locations with complete data
Pressure Ulcers	+	Lowering the prevalence of pressure ulcers by 50%	Incidence degree 2 or higher Prevalence degree 2 or higher	16 (31.4%)
Eating and Drinking	+	Lowering the prevalence of ill-nutrition by 40%	Prevalence ill-nutrition	38 (100%)
Prevention of Sexual Abuse	±	Score 8 at a scale of 1–10 at each measure	Attitude Competence Steering	20 (71.4%)
Medication Safety	+	Lowering the prevalence of medication errors by 30%	Prevalence medication errors	27 (77.1%)
Problem Behaviour	−	Lowering the prevalence of problem behaviour	Prevalence problem behaviour for two or three clients Prevalence problem behaviour for overall pilot location	9 (64.3%)
Fall Prevention	+	Lowering the prevalence of fall incidents by 30%	Prevalence fall incidents	24 (100%)
Autonomy and Control
Physically Disabled Clients	±	Substantial improvement in autonomy and quality of life of clients	Client questionnaires on quality of life and quality of (client-centred) care	5 (83.3%)
Nursing and Residential Care Homes	±	Substantial improvement in autonomy and quality of life of clients	Client questionnaires or observations on quality of life	0 of 28 (due to incomplete data/incomparable measures
Mentally Disabled Clients	±	Substantial improvement in autonomy and quality of life of clients	Client questionnaires on baseline and qualitative portfolios on end-measurement	0 of 13 (due to no end-measurement)

Level of evidence was assessed by two researchers independently based on collaboratives' plans of action. The highest level (+) was assigned if evidence-based guidelines, measurements instruments and national measurements were available. A moderate level of evidence (±) was assigned if improvement efforts and good examples were described. The lowest level (−) was assigned if hardly any improvement efforts or good examples were described in the literature.

Survey data

Seventy-five (41%) team leaders returned a end-measurement questionnaire which was sent as part of our evaluation study. Almost 70% were nurses, about 10% were quality workers, 10% were occupational or physiotherapist, and about 10% were executive manager of the pilot location. Unfortunately, we had outcome as well as survey data of only 52 teams, and therefore present the results of these two sources separately.

The team leaders rated several statements on their collaborative on three dimensions: measurability, achievability of the targets and whether or not the targets were challenging (seven-point scale ranging from their ‘strongly disagree’ to ‘strongly agree’). Factor analyses confirmed these three dimensions (data not shown, but available on request). Measurability was assessed by: (1) measuring indicators helps to monitor progress, (2) there were clear agreements on measuring central indicators, (3) programme management offered a standardised set of indicators to monitor progress and compare results, and (4) timely and accurate progress information was available at all times.11 The reliability was 0.86. Achievability was assessed by: (1) collaborative targets are achievable, (2) programme management made clear how to achieve collaborative targets, (3) programme management offered good practices and evidence on achievable results, and (4) programme management gave specific instructions on how to improve interventions.11 The reliability was 0.77. The extent to which the targets were conveyed as challenging by programme management was measured with the item: ‘Programme management set high expectations with regard to performance and improvement possibilities.’11

Analysis

Mean changes between measurements were examined using paired-samples t tests. To assess teams' effectiveness in terms of achieving collaborative targets, teams were assigned to one of three groups on the basis of changes in a particular measure: improvement (target achieved), deterioration and stable. Teams that started at baseline with a prevalence of 0% were not included in the analyses, since no relative change score can be calculated.

Results

Measurability, achievable and challenging targets

Significant differences in perceived measurability and achievability of targets between the various collaboratives were found (F=6.798 and p=0.000 and F=6.566 and p=0.000 respectively; see table 2). No significant differences between the collaboratives were found with regard to challenging targets. Team leaders in the Problem Behaviour collaborative and the collaboratives on Autonomy and Control scored significantly lower on measurability and challenging targets. Also, with respect to achievability, the Problem Behaviour collaborative scored significantly lower.

Table 2

Descriptives measurability, achievability and challenging targets

	Measurability			Achievability			Challenging
	N	Mean	SD	N	Mean	SD	N	Mean	SD
Pressure Ulcers	8	5.59	.86	9	4.61	.88	9	4.44	1.33
Eating and Drinking	16	5.61	.92	16	5.52	.76	16	4.56	1.41
Prevention of Sexual Abuse	12	5.94	.91	12	5.73	.96	12	5.00	1.71
Medication Safety	10	5.20	.65	11	5.30	.44	11	4.36	1.50
Problem Behaviour	7	4.38	.64	7	3.64	1.14	7	4.29	.76
Fall Prevention	8	5.25	1.20	8	4.82	.62	7	4.71	1.11
Autonomy and Control	11	3.75	1.32	12	4.46	1.12	11	4.00	1.48
Total	72	5.16	1.18	75	4.99	1.04	73	4.49	1.39
F		6.798			6.566			.559
p		.000			.000			.761

Values in bold indicate a significant difference from the total mean score.

Collaborative characteristics and effects on targeted measures

The following section describes whether these differences between collaboratives are also reflected in the degree of effectiveness of each collaborative.

Pressure ulcers

Although the level of evidence on preventing pressure ulcers is high, and team leaders perceive a high measurability, the collaborative target is not perceived to be achievable and challenging. Accordingly, overall effectiveness is moderate. On average, prevalence rates decreased from 18% to 10% (table 3). Only six of the 16 teams improved prevalence by more than 50% and achieved the collaborative target (table 3 last three columns).

Table 3

Descriptive results for different collaboratives

						No (%) of teams achieving target†
	T0 mean (SD)	T1 Mean (SD)	Absolute change (T1–T0)	Relative change (T1–T0/T0 ×100) (%)	p Value paired t tests*	improved	stable	deteriorated
Pressure ulcers (N=16)
Prevalence of Pressure Ulcers	18.16 (17.59)	10.21 (11.22)	−7.95 (10.92)	−42.83 (43.49)	.011	6 (37.5%)	9 (56.3%)	1 (6.3%)
Eating and Drinking (N=38)
Prevalence Malnutrition	30.67 (13.35)	23.71 (12.27)	−6.97 (10.77)	−19.93 (46.12)	.000	18 (47.4%)	12 (31.6%)	8 (21.1%)
Prevention of Sexual Abuse (N=20)
Attitude (1–10)	7.82 (0.82)	8.53 (0.73)	.65 (0.85)	8.89 (11.98)	.001	8 (40.0%)	11 (55.0%)	1 (5.0%)
Competence (1–10)	6.30 (1.69)	8.50 (1.18)	2.09 (2.20)	41.42 (39.97)	.000	13 (65.0%)	5 (25.0%)	2 (10.0%)
Steering (1–10)	2.64 (1.66)	7.62 (1.97)	5.01 (2.42)	319.47 (284.20)	.000	13 (65.0%)	7 (35.0%)	–
Medication Safety (N=27)
Prevalence of Medication Errors	36.15 (26.49)	10.26 (9.16)	−25.89 (21.68)	−68.31 (25.77)	.000	26 (96.3%)	–	1 (3.7%)
Problem Behaviour (N=9)
Problem-Behaviour Selected Clients	33.56 (48.63)	7.41 (8.51)	−26.14 (48.58)	−62.06 (26.98)	.145	9 (100%)
Problem Behaviour Pilot Location	2.52 (1.18)	1.00 (0.70)	−1.52 (0.63)	−65.68 (17.18)	.001	7 (100%)
Fall Prevention (N=24)
Prevalence of Fall Incidents	22.67 (15.52)	8.00 (9.68)	−14.67 (16.64)	−59.78 (43.03)	.000	19 (82.6%)	2 (8.7%)	2 (8.7%)

p Values are obtained from two-sided paired-samples t tests; T0=baseline; T1=end-measurement.

Cut-off points were based on the improvement target of each collaborative.

Eating and drinking

For this collaborative, the level of evidence is high. Measurability and achievability are perceived as positive, which is in accordance with the positive results. Overall prevalence of malnutrition significantly decreased from 31% to 24%. Eighteen teams (47.4%) were able to decrease prevalence of malnutrition by 40% or more, thus achieving the collaborative target.

Prevention sexual abuse

Although only a few good examples of how to prevent incidents of sexual abuse are available, team leaders perceived high measurability and achievability of targets. Baseline mean scores for attitude and competence of professionals were moderate, respectively 7.82 and 6.30 on a 1–10 point scale. These scores improved at end-measurement by 9% and 41%, respectively. The baseline mean score on perceived steering of management was very low (2.64 on a 0–10 point scale). On average, participating teams were able to realise 320% improvement on this measure. The percentage of teams achieving the target ranged from 40% to 65%.

Medication safety

This collaborative distinguishes itself positively with respect to achievability of targets. Team leaders rated the achievability of targets with a score of 5.30 (SD 0.44), which is in line with the positive results. On average, teams were able to decrease the number of medication errors by 68%, which is considerably higher than the 30% aimed for. Twenty-six teams (96.3%) were able to realise the target. One team wanted to address the problem of under-reporting, and the team's local target was to increase the number of errors, the opposite of the generic collaborative target.12

Problem behaviour

For this collaborative, good examples for effective interventions were not available. Implementing suitable interventions for individual clients was rather a matter of trial and error. Team leaders perceived a low achievability of targets, thought that results were hard to measure and, targets being not defined, did not find them challenging, which may be related to why only nine of the 14 pilot teams collected complete data. On average, teams monitored 2.71 (SD 1.27) clients intensively. Significant changes between baseline and end-measurement were found (t=2.58 and p=0.016). Teams also collected prevalence data for the whole pilot location. On average, the number of incidents decreased significantly. Since the improvement target was undefined, all (or no) teams achieved the target.

Fall prevention

There was considerable evidence on interventions to prevent fall incidents. Team leaders rated measurability, achievability and challengeability of targets only moderately. At baseline, the average prevalence of incidents was 23%, which decreased significantly to 8%, corresponding with a relative improvement of almost 60%. Furthermore, 19 teams were able to realise the improvement target of 30% decrease.

Autonomy and control

Although there is some evidence on how to improve quality of life and autonomy of clients, and several measurement instruments are available, the Autonomy and Control collaboratives had some difficulty in formulating specific measurable achievable relevant time-based targets and setting up the measurement. Consequently, team leaders scored significantly lower on measurability than in the other collaboratives, which is also reflected in the low percentage of teams with complete data. Scores on achievability and challengeability of targets were lower as well. Client data on five of the six care organisations for physically disabled showed no significant changes in client-centred care. For the other projects on Autonomy and Control, no comparable measures across teams and measurements were used, and thus the effectiveness of these projects cannot be assessed.

Discussion

Based on a comparison of a wide range of QICs, our study is one of the first that empirically takes into consideration the influence of characteristics of the collaborative to understand the (construction of) effectiveness of different QICs.

One of the often mentioned criteria for being called a collaborative is having a topic with a large gap between current and best practice,3 4 13 and it can be debated whether all Care for Better collaboratives meet this criterion. For some of the topics, it was not clear how large the gap was between current and best practice, since best practices were unavailable. For example, within the Problem Behaviour collaborative, no good examples were available. The first round of this collaborative was used as a stepping stone for later rounds and functioned as a ‘learning laboratory,’14 in which instruments and tools were (co-) designed and tested by the health professionals themselves and resulted in a toolkit for other teams.

As was shown in the study by Benn et al6 and Nembhard,15 quality-improvement methodology, programme faculty support and monitoring are important for success. Our results are in line with this; programme management of several collaboratives experienced difficulties, especially with regard to formulating clear and measurable targets, since they mainly had to rely on literature based on other care sectors, making it hard to translate to long-term care.

The present study shows that the overall results of the Care for Better programme are mostly positive; nevertheless, there is considerable variation between teams and collaboratives. Our results showed, not surprisingly and in line with observations by Øvretveit et al3 and Wilson et al,5 that a high level of evidence for a certain topic does not guarantee effectiveness in improving quality. The converse finding is more revealing: little evidence base may still lead to improvements in processes of care on the condition that programme management reflects upon how achievable and challenging targets should be formulated and which instruments and indicators should be used to demonstrate whether targets are realised.

Limitations

The lower percentages of complete data for some collaboratives seemed to be indicative of the difficulty programme managers had in selecting indicators and questionnaires, and setting up measurement systems. Outcome data on these collaboratives should be interpreted with caution, although no significant differences in baseline scores between teams with complete data and incomplete data were found.

Some of the outcome indicators were based on self-reported data. The situatedness of improvement efforts calls for simple measurements that are relevant for a specific team, to monitor progress which is an essential part of the improvement method. From an evaluation perspective to enable comparison, however, the (external) validity and reliability of these data are uncertain. This friction is inherent to the evaluation of complex improvement programmes.16

Another limitation is the overall moderate response on the evaluation survey. There may be two explanations: (1) the collaboratives on Pressure Ulcers, Eating & Drinking and Prevention of Sexual Abuse were not informed beforehand about the evaluation; and (2) several team leaders left their job halfway through the collaborative.

As a final limitation, this study did not include control sites, which makes it difficult to rule out possible secular trends. However, randomised controlled trials are hardly an option given the changing and complex features of QICs. Evaluation studies should therefore combine effectiveness measures with detailed descriptions of the programme and its context to understand under what conditions participating teams are able to realise changes in quality of care.

Conclusion

This large multiple-case before–after study suggests that the effectiveness of a QIC is associated with the efforts of programme managers to create conditions that provide insight into which changes in processes of care and in client outcomes have been made. We suggest that in preparing and organising a QIC, programme managers should more carefully consider the type of quality problem or topic addressed and invest in formulating achievable and challenging targets. In the presence as well as the absence of evidence, a crucial task for programme management is to create measurability, as this proved not to be an inherent quality of improvement topics.