Coproduction in commissioning decisions: is there an association with decision satisfaction for commissioners working in the NHS? A cross-sectional survey 2010/2011.

OBJECTIVES: To undertake an assessment of the association between coproduction and satisfaction with decisions made for local healthcare communities.
DESIGN: A coproduction scale was developed and tested to measure individual National Health Service (NHS) commissioners' satisfaction with commissioning decisions.
SETTING: 11 English Primary Care Trusts in 2010-2011. PARTICIPANTS: Staff employed at NHS band 7 or above involved in commissioning decisions in the NHS. 345/440 (78%) of participants completed part of all of the survey. MAIN OUTCOME MEASURE: Reliability and validity of a coproduction scale were assessed using a correlation-based principal component analysis model with direct oblimin rotation. Multilevel modelling was used to predict decision satisfaction.
RESULTS: The analysis revealed that coproduction consisted of three principal components: productive discussion, information and dealing with uncertainty. Higher decision satisfaction was associated with smaller decisions, more productive discussion, decisions where information was readily available to use and those where decision-making tools were more often used.
CONCLUSIONS: The research indicated that coproduction may be an important factor for satisfaction with decision-making in the commissioning of healthcare services. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This study had a high-response rate to the survey which informed model development.

Several predictors were significantly associated with decision satisfaction.

A retrospective design was used which required participants to recall events in the past. This may have led to bias in responses received.

Predictors were not measured on the same scale as one another and so limited comparisons can be made of their relative effect sizes.

The relationship between decision satisfaction and decision quality is not yet tested. Therefore we cannot conclude that a decision with which a commissioner is satisfied is necessarily an independently verifiably ‘good’ decision.

Introduction

Commissioning, evidence and decisions in the National Health Service

In England in 2013, the responsibility for commissioning health services changed hands as Clinical Commissioning Groups (CCGs) took over the role from Primary Care Trusts (PCTs).1 It is the duty of National Health Service (NHS) commissioners to plan, fund and review a wide variety of health services ranging from emergency care to community-based interventions for their local populations.2 Previous research on commissioning suggests that commissioning decisions should involve the drawing together of different professionals and interests around the ‘common cause’ of services which can better meet patients’ needs.3–5 Previous research also suggests that commissioning is complex. Decision makers are required to take into account a number of factors including local need, available resources, funding opportunities and need for savings, as well as sources of information such as national policy directives and available evidence.6 7 Elliott and Popay8 in a previous investigation of decision-making by local NHS policy makers found that the ‘influence of research evidence on decision-making was tampered by factors such as financial constraints, shifting timescales and decision makers’ own experiential knowledge. They suggest that research is ‘more likely to impact on the (local) policy in indirect ways’, including for example shaping the policy debate. As Walshe and Rundall9 noted many managerial decisions in healthcare are ‘constrained, contested and political’.

Sainfort and Booske10 reported that the process of measuring satisfaction with a decision is fundamental for difficult situations where there is no ‘right’ decision and or where long-term consequences are uncertain. This is the case in healthcare, particularly in commissioning decisions which are frequently criticised due to the substantial variability observed across England.11 We used the background literature on organisational, clinical and individual satisfaction with healthcare decisions and case study evidence to develop the concept of decision satisfaction.

Often there is a lack of relevant data about existing populations and services and a scarcity of evidence about the outcomes of services which cause problems for those wishing to make effective decisions. These problems, coupled with the statutory national obligations that need to be fulfilled, increase the pressure on newly formed CCGs.

Evidence-based decision-making and coproduction

Most research on evidence has focused on its uptake in health services, investigating for example the extent to which clinical guidelines are used.12 13 Of equal importance is how evidence is used at a point when decisions have to be made and to what extent decision makers are satisfied with the decision outcome.

Baumbusch et al14 introduce the idea that the translation and utilisation of research in clinical settings is a process requiring collaboration and dialogue described as a ‘collaborative model’. Successful commissioning decisions are rarely made by a single individual or professional discipline.7 They are the product of multiple views, experiences and resources. For the purposes of this research we use the term coproduction as defined by Swan et al15 explaining this process of incorporating multiple views, experiences and resources into commissioning decisions.

Proponents of a coproduction perspective discuss the difficulties of making academic findings useful in practice.16 17 They view the gap between evidence and practice as a result of the way academics produce knowledge, leading to difficulties in its usefulness to practitioners. Hence this becomes a knowledge assimilation or translation problem, rather than a knowledge diffusion problem.18

To overcome this issue knowledge needs to be coproduced by the relevant communities involved in commissioning.19 20 There is no agreed definition of coproduction although we have identified several principles that are important for its success as shown in figure 1.15

Figure 1

Principles important for the success of coproduction as defined by Swan et al.15

The aim of this research was to undertake an assessment of the association between coproduction and satisfaction with decisions made for local healthcare communities by healthcare commissioners. To achieve this, we developed and tested a coproduction scale and measured individual commissioners’ satisfaction with commissioning decisions.

The conceptual model

We undertook an in-depth qualitative investigation of commissioners working in PCTs in England.21 This empirically grounded understanding of how evidence is utilised in commissioning decisions enabled the development of a conceptual model presented in figure 2.

Figure 2

Conceptual model of potential predictors of decision satisfaction. Adapted from Swan15 (NHS, National Health Service).

The model shows six decision predictors which could influence satisfaction with a decision. These are grouped into those related to the decision, that is its size; the characteristics of decision makers—such as their background (clinical or managerial); the type of evidence used in the decision—practical (based on previous experience) or empirical and the extent of coproduction in the decision. We hypothesised that these variables would be associated with decision satisfaction.

Methods

Survey design

This research was part of a larger study to examine the use of evidence for management decisions in PCTs.15

A cross-sectional survey of commissioners’ decision-making was designed to test the conceptual model. Prepiloting and piloting of the questionnaire were conducted with purposive samples of participants drawn from local NHS organisations (see online supplementary file 1 for a copy of the survey). The results were used to develop and refine the questionnaire and the process of administration.

Topic areas and questions were derived from published surveys, literature reviews and our own in-depth case study evaluation of commissioning processes in four PCT sites.7 15 The survey included subsections on demographic details, work role, sources of evidence use, decision characteristics (size, monetary value and tool use)7 21 and satisfaction with decision-making using an adapted Decisional Conflict Scale.22 We also drew on a scale measuring empirical evidence sources by Weatherley (see online supplementary appendix 1).

Commissioners were asked to select a recent commissioning decision which they had been involved in and where the decision-making process was largely completed. They were asked about the extent of coproduction in the decision using specific questions presented in figure 3.

Figure 3

The coproduction questions and the scoring system applied.

Sample

A sample size calculation indicated that approximately 300 respondents would be required to allow us to detect a 10–15% difference in proportions (with 80% power and a 95% CI) in responses by professional work role (ie, clinically vs non-clinically qualified commissioners).

Participants

We identified all potential participants in a random sample of 15 PCTs, stratified by the size and index of multiple deprivation of the population they served (from the total of 143 eligible PCTs excluding pilots). Contact details of all staff employed at NHS grade 7 (broadly team manager or advanced practitioner level) or above who were involved in commissioning decision-making were obtained from each identified PCT. This included staff from departments of public health, finance, purchasing, commissioning, contract monitoring and information services as well as the executive team.

Participants were given information sheets and details about how to participate. They could complete the survey via face-to-face meetings held at their office or by emails using an online electronic questionnaire. Four additional reminders were sent to non-respondents at two weekly intervals. Questionnaires completed manually and electronically were anonymised and transferred to an Excel database.7

Analysis

Reliability and validity of the coproduction scale

The scale was validated and checked for subscales. To do this, all coproduction items were entered into a correlation-based principal component analysis (PCA) model. We hypothesised that potential subscales would not be independent of each other; hence PCA with direct oblimin rotation was utilised to allow for low-factor correlation. Parallel analysis was conducted to identify how many components should be extracted from the model. Scale reliability was measured using Cronbach's α. PCA was conducted using R (http://www.r-project.org; please see online supplementary file 2 table S1 for factor loadings).

Decision satisfaction statistical model

Multilevel (ML) linear regression modelling was used to predict the decision satisfaction score using an adapted Decisional Conflict Scale,15 where lower scores denote higher decision satisfaction. ML modelling allows us to model individual responses while allowing for differences between the PCTs to also be modelled. The scores for the dependent variable and all predictors in both analyses were checked for normality using visual inspection of histograms and Q–Q plots, alongside measurements of skewness and kurtosis. MLwiN V.2.22 was used for the ML modelling.

Predictors considered for inclusion in the model are taken from the conceptual model shown in figure 2, which resulted from previous qualitative research in the field.21 To reduce the correlation between the predictors in the model, coproduction was modelled by the subscales (PCA components) rather than the individual items. These are detailed further in table 1. A likelihood ratio test was used to compare the null single-level model with the null ML model to determine the influence of PCT level effects. A separate model for each predictor was run to determine which would be included in the main model. Predictors found to be significantly different from the null model were then considered for inclusion in an overall model. This model was created by adding these predictors stepwise in descending order of individual impact on decision satisfaction (determined by the change in −2log likelihood in their separate models—representing the quantity of improvement of model fit). Predictors were retained in the main model if they improved the model fit significantly (at p=0.05 level).

Table 1

The effect of adding each predictor separately to the null multilevel model of decision satisfaction

Predictor	Improvement to model fit (change in −2log likelihood)	Coefficient B (SE)
Decision size (service cost)	51.8***	0.004 (0.053)
PCA1: productive discussion	48.5***	−0.170 (0.023)
PCA2: information availability and use	22.7***	−0.112 (0.023)
The number of decision-making tools used	17.4***	−0.0408 (0.0166)
Experience of NHS commissioning (years)	12.8***	−0.0102 (0.0049)
Sources of empirical evidence as defined by Weatherly et al23	10.7**	−0.037 (0.051)
Sources of evidence derived from our qualitative research21	10.4**	−0.014 (0.050)
Respondent medical qualification (yes/no)	6.4*	−0.1299 (0.0510)
Index of multiple deprivation of population served (IMD)	2.9	0.0056 (0.0032)
PCA3: dealing with uncertainty	0.7	−0.008 (0.024)
Size of population served (proxy for size of commissioning organisation)	0.1	0.0000 (0.0000)

*p<0.05; **p<0.01; ***p<0.001.

NHS, National Health Service; PCA, principal component analysis.

Results

In the first recruitment wave 6 of the 15 PCTs invited agreed to take part, and 9 were rerandomised by strata. In the second recruitment wave further 5 PCTs accepted resulting in a final sample of 11. The survey was circulated to 440 individuals across these 11 PCTs and 345 (78%) responded.7

Participant demographics

The median age band of the participants was 45–54 years, and 63% of the sample were female. Forty-seven per cent of respondents had 5 years or less experience in commissioning. Thirty-one per cent (n=107) of respondents were qualified health or allied health professionals, although only 1% (n=3) was currently employed in a clinical setting. The largest single group of respondents (43%, n=149) were working in commissioning and contract roles, and the remaining were working in public health (33%, n=114), finance (7%, n=24) or other related commissioning roles (15%, n=52).

Selected decisions

When asked to select a decision to frame their responses, the majority (n=189, 55%) of respondents selected ‘changing the organisation or design of a particular service’. The second most popular was a ‘major decision on strategic direction’ (n=83, 24%) and 30 (9%) participants selected Individual Funding Requests.

Principal components of the coproduction scale

The parallel analysis indicated that three principal components (PCs) would be sufficient and hence were extracted. Items were considered to be part of a PC if absolute item loadings were ≥0.45. Items 1 and 10 did not load onto any component and were excluded from the analysis. Items were predominately well explained by the model (item communality > 0.5), but the two items (Q13 and Q17) with low communalities (less than 0.3) were also removed. Cronbach's α for the remaining items was then calculated to be 0.84, indicating very good reliability. The reliability for the three subscales was also good, with α=0.77 for PC1, 0.80 for PC2 and 0.68 for PC3.

The three PCs explained three distinct subscales centred on ‘productive discussion’, ‘information’ and ‘dealing with uncertainty’, these are shown in figure 4. As these subscales have a maximum correlation of 0.36 with each other, this reduces the chance of overfitting when used as independent variables in a regression model.

Figure 4

The three distinct subscales explained by principal components (PCs) produced explained by three PCs of the coproduction scale.

Decision satisfaction

The modified decisional conflict scores were found to be non-normal (leptokurtic) and hence the data were transformed using a natural logarithm to meet the normality assumptions of the linear model.

The null ML model was an improvement in the null single-level model (change in −2LL=294.9−290.7, χ2(1)=4.2, p<0.05), with a variance partition coefficient of 0.1 indicating that 10% of the variation in decision conflict scores can be explained by the PCT to which the respondent belongs.

The effect of adding each individual predictor to the model is shown in table 1. The greatest model improvement was found by adding either the decision size or coproduction score to the model.

Overall, the results presented in tables 1 and 2 demonstrate that higher decision satisfaction was associated with smaller decisions, more productive discussion, decisions where information was readily available to use and those where decision-making tools were more often used. Furthermore respondents with a medical qualification, and with great experience in NHS commissioning, are more likely to report greater decision satisfaction.

Table 2

The final model for influences on decision satisfaction (model fit −2LL=157.7)

Predictor	Coefficient	(SE)
The size of the decision	0.021	(0.027)
PCA1: productive discussion	−0.16	(0.02)
PCA2: information availability and use	−0.11	(0.02)
The number of decision-making tools used	−0.007	(0.02)
Respondent years experience of NHS commissioning	−0.009	(0.005)
Respondent medical qualification	−0.09	(0.05)

Note that lower scores denote higher decision satisfaction. At an individual level, the coefficients can be interpreted as the change in decision satisfaction for a unit change in the predictor.

NHS, National Health Service; PCA, principal component analysis.

Discussion

The models produced here show that several predictors are significantly associated with decision satisfaction. We found that decision satisfaction was influenced by the cost implications of the decision, and the scores on PCs of coproduction 1 and 2 (productive discussion and information availability and use), and the number of decision-making tools used. The term coproduction can be interpreted in a variety of ways and it is not easily defined as a concept.15 20 By using the definition previously identified by Swan et al this research indicates that coproduction in commissioning may consist of three separate components: productive discussion, information availability and use, and dealing with uncertainty.

The third PC, ‘dealing with uncertainty’, was not found to influence decision satisfaction. This appears to indicate that uncertainty, characterised by pausing discussions to clarify the meaning behind certain terms and explaining unfamiliar concepts and terms where necessary, was not important in decision satisfaction. It was not significant when included as the only predictor in the model indicating that the lack of effect is not driven by a model containing similar or overlapping terms. The variability of the coefficient for ‘dealing with uncertainty’ in the model of decision satisfaction was similar to the variability for the coefficients of ‘productive discussion’ and ‘information availability and use’. Therefore the lack of effect is unlikely to be explained by uncertainty affecting some respondents in a positive way and other respondents in a negative way.

The results of our model demonstrate that increasing coproduction may be able to increase satisfaction with decisions made by healthcare commissioners. Many healthcare decisions are complex and difficult. The ‘right’ decision is not always obvious at the time the decision is made. This highlights the importance of facilitating and encouraging coproduced decision-making within the newly formed CCGs. This supports previous research that concluded that CCGs will need to make sure that they use “collaborative discussion and service planning in addition to transactional work particularly in times of financial austerity”.4

Strengths and limitations of the findings

This study had a high response rate to the survey which informed the model development. Nevertheless, the research used a retrospective design which required participants to recall events which had happened in the past. There may have been bias in responses received due to recall bias. Social desirability bias may also have been introduced as participants may have given answers which they deemed to be appropriate to the researchers—not necessarily which reflected their true beliefs about the situation. Given the nature of the data collection and our promises of anonymity, it was impossible to correlate reports by different managers on the same decision, although this would have lent additional validation to our findings. Also, we do not know the effect on the results if patients’ and providers’ satisfactions with the decisions had been surveyed.

Although the models show that several predictors are significantly associated with decision satisfaction, these predictors (with the exception of the three parts of coproduction) are not measured on the same scale as one another which means that limited comparisons can be made of their relative effect sizes. It is also possible that the elements identified might reflect an underlying latent factor, or that other important factors such as good working relationships are involved. These issues will be of interest for further research in this area.

Implications for practice and future research

The findings of this research are important for commissioners in the NHS today who operate in a difficult and changing environment. Commissioners must acknowledge the implications that their decisions have on the health of the population for which care is being commissioned. Those working in, or managing commissioning organisations need to:

Include the right individuals in the decision-making process;

Ensure access to the right data and resources;

Ensure that decision-making tools are available to commissioning groups. (Tools currently available include clinical guidelines, cost-effectiveness analyses and National Service Frameworks.7)

Facilitate productive discussion and exploration of different views throughout the decision-making process.

Ensuring that a decision is coproduced is a step in the right direction towards bridging the research to practice gap. Collaborative decision-making using the coproductive approach enables knowledge to be moved across boundaries between academic community and society, taking into account the interests and objectives of a range of stakeholders.13 24–26

Further research should validate and investigate measurements of decision satisfaction in this organisational setting and in other similar settings and in this context it would be of interest to investigate the relationship between commissioners’ patients’ and providers’ satisfaction with the same decisions.

Conclusion

Organisational decision-making to plan services is important in all healthcare systems but is often difficult in practice. Satisfaction with commissioning decisions in this research required coproduction in the form of collaboration, interaction and using the ‘right’ resources. Our data demonstrate that coproduction is comprised of three separate components (productive discussion, information availability and dealing with uncertainty). In this research, productive discussion appeared more important than the information availability and use for decision satisfaction. These findings will be of value to CCGs in commissioning decision-making and the use of evidence, as they make decisions for the benefit of their local populations.