Capability building for large-scale transformational change: learning from an evaluation of a national programme.

BACKGROUND: The Long Term Plan presents an ambitious vision for England's National Health Service which will require a sustained programme of transformational change. The Virtual Academy of Large-Scale Change (VALSC) was developed to build capability in health and care system teams involved in transformation or redesign programmes.
METHODS: To evaluate the VALSC, quantitative and qualitative data were collected and reviewed against the Kirkpatrick model. Quantitative data were collected via end-of-session surveys to assess individual knowledge before and after participating in capability-building interventions. Qualitative data were also collected and included post-intervention surveys and interviews. Interviews were transcribed and analysed using an inductive approach to identify themes that were subsequently assessed against the Kirkpatrick model.
RESULTS: Results suggest that the VALSC programme has helped build capability for large-scale change in terms of learning, behaviour change and impact. Participants' ipsative self-assessment of knowledge demonstrated a significant change (p<0.001) and qualitative data suggested three broad themes in which the VALSC made an impact. First, participants were empowered with transformation and change skills which they applied to local health and care challenges. Second, VALSC helped strengthen connections within and between transformational change teams. Third, VALSC helped transformational change teams to engage more effectively with their stakeholders.
CONCLUSIONS: The VALSC developed knowledge, skills, behavioural change and application impact that built capability in individuals and teams. Therefore, continuing to develop capability-building offers that empower and build agency in front-line staff working on service transformation and equip them with approaches, methods and tools to increase their chances of success, is recommended. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Background

Like many other healthcare systems internationally, England’s National Health Service (NHS) faces challenges associated with medical advances, population demographics, ageing populations and increasing public demands.1 To address these challenges, the NHS Long Term Plan (LTP) presents an ambitious vision to redesign care ‘to future-proof the NHS for the decade ahead’.2 Central to the delivery of the LTP is the ‘triple integration of primary and specialist care, physical and mental health services, and health with social care’ within place-based systems.2 This will require a sustained programme of transformational change to secure the major national improvements set out in the plan.

Transformational change has been defined as ‘the emergence of an entirely new state, prompted by a shift in what is considered possible or necessary, which results in a profoundly different structure, culture or level of performance’.3 4 This will require leaders and staff with the capability to deliver change and improvement on a scale perhaps not hitherto seen in the NHS.

Capability comprises an organisation’s ‘knowledge, experience and skills’.5 Furnival et al define improvement capability as ‘the organisational ability to intentionally and systematically use improvement approaches … to generate improved performance’,6 where improvement approaches can be defined as ‘approaches or methodologies [that provide] structured, systematic and well-established tools and techniques for continually improving service quality’.7 Furthermore, improvement capability has been identified as requiring effectiveness across several domains including: leadership; strategic alignment of goals; employee commitment and motivation; stakeholder engagement; and process improvement and learning. Improvement capability, therefore, requires core competence in these domains. This can be developed through building knowledge and skills that, overtime, become embedded within an organisation’s people, teams and culture.8 9

Over the past 20 years, there has been considerable focus in the NHS on building knowledge and skills in the process improvement domain.10 11 However, it has also been observed that successfully applying process improvement approaches to increase performance has been variable.6 11 Furthermore, knowledge and skills in other domains associated with managing and leading large-scale, transformational change have been identified as lacking.3 10 12 Therefore, there remains a gap in improvement capability building particularly at the system level.11 13

An early attempt to build capability for transformational change was undertaken by the former NHS Institute for Innovation and Improvement. The Academy of Large-Scale Change (ALSC) was established to equip regional and national leaders with world-class improvement and change knowledge and skills.12 The ALSC’s hypothesis was that, to be confident and effective in their change leadership actions, leaders needed a ‘grounded theory of large-scale change’.12 Some 80 participants from the regional and national regulatory landscape took part in the ALSC. This comprised 18 months of action learning to develop knowledge, models, theories and frameworks to enable emergent planning and design for sustainable large-scale change.12 14

The knowledge outputs from the ALSC were captured, collated and published in a practical guide. Specifically for the NHS, the guide brought together: a comprehensive review of the evidence base at the time; a theory and model of large-scale change developed by the ALSC; together with a set of tools and techniques to support action.12 14 In 2017, the Leading Large-Scale Change Guide was fully revised and updated by NHS England’s Sustainable Improvement (SI) Team to reflect the latest evidence and the current needs of leaders across health and care in relational to transformational change.15

To support the publication of the refreshed Leading Large-Scale Change Guide, the Virtual Academy of Large-Scale Change (VALSC) was developed. The VALSC aimed to build capability in health and care system teams involved in transformation or redesign programmes to intentionally apply the practical approaches and tools contained in the guide.

Drawing on learning from the original ALSC, but aiming to achieve greater scale, the VALSC comprised a blend of online and face-to-face activities (webinars and masterclasses) that supported teams across the health and care landscape to progress their transformational change programmes. Delivered by the SI Team (including authors IMS and EB), the VALSC ran from September 2017 (following the launch of the refreshed Leading Large-Scale Change Guide) to September 2019 (at which point NHS England and NHS Improvement’s improvement capability-building teams came together in a new improvement directorate to shape a new, aligned offer). This paper reports on the evaluation of the VALSC, describing its components, reach, reaction from participants and overall impact on leaders’ approach to change.

Methodology

Approach

The broad aim of the VALSC was to support health and care system teams involved in transformation or redesign programmes to apply practical approaches and tools of large-scale change. A mix of quantitative and qualitative data were collected (described below) and reviewed against the Kirkpatrick model to verify progress and evidence impact.

The Kirkpatrick model is commonly used to evaluate the impact of training and development programmes.16 17 The model provides a framework to evaluate the impact over four levels: reaction, learning, behaviour and results.18 The first level (reaction) is concerned with participants’ impressions of the programme. The second level (learning) is concerned with changes in participants’ knowledge and skills. The third level (behaviour) is concerned with the extent to which participants transfer their new knowledge and skills into practice in their work environments. The fourth level (results) is concerned with the overall impact of the training in terms of improvement.16 18 The Kirkpatrick model has also been proposed as applicable for the evaluation of healthcare improvement programmes.19 20

Data collection

A blend of data were used to evaluate the VALSC against the levels of the Kirkpatrick model. Levels 1 and 2 of Kirkpatrick emphasised the quantitative data. Quantitative data were collected via end-of-session surveys after each webinar and masterclass delivery. Using ordinal (1–10) scales, participants were asked to rate their experience of the session as well as their knowledge before and after the session. To support assessment against Kirkpatrick levels 1 and 2, qualitative data were also collected including: chat box comments from webinar sessions; participant observation of masterclasses (by IMS and EB); and masterclass artefacts (eg, photographs, videos and work produced by participants). At levels 3 and 4 of Kirkpatrick, greater emphasis was placed on the qualitative data which, in addition to that described above, also included post-intervention surveys and interviews. All webinar participants were invited to complete an online survey. The survey presented a combination of open and closed questions covering: participants’ reaction to and experience of the programme; what they had learnt; how they were using the learning; and its overall impact. Semi-structured interviews were conducted with a convenience sample of VALSC masterclass participants. Despite its convenience nature, the sample represented a variety of organisation types (foundation trust (FT), integrated care partnership (ICP), regulator and clinical commissioning groups (CCG)) from across the country (North West, North East, West Yorkshire, Midlands, South, South West and London) working on various changes (integrated care, self-care, primary care, long-term conditions and maternity care). The sample was, therefore, broadly representative of VALSC participants. Interviews were conducted by FM who had no role in the programme’s design or delivery and sought a range of views from interviewees. Nine interviews were conducted via telephone and audio recorded then transcribed for analysis. Quotes from interview participants presented in this report have been anonymised. The relationship of quantitative and qualitative data to the Kirkpatrick model’s four evaluation levels is shown in table 1.

Table 1

Data collected and levels of the Kirkpatrick model

Kirkpatrick level	Quantitative	Qualitative
1—Reaction	Participants’ ranking of training. Participants’ likelihood to recommend to colleagues.	Participant observation of training. Webinar chat box comments. Photographs of masterclass events. Interviews with sample of participants.
2—Learning	Ipsative self-assessment of knowledge before and after learning interventions.	Work produced by participants. Photographs of masterclass event outputs. Interviews with sample of participants.
3—Behaviour	Pre-training and post-training survey.	Pre-training and post-training survey. Interviews with sample of participants.
4—Results		Session/event videos. Interviews with sample of participants.

Data analysis

Before-and-after knowledge data were analysed using the Wilcoxon signed-rank test. This is used to test for difference between paired samples of non-parametric data.21 22 Non-parametric tests do not assume normal distribution and can be used with ordinal data.21–23 Paired samples arise when study participants are measured before and after an intervention.24 In this study, participants rated their knowledge before and after taking part in capability-building online webinars and/or masterclass workshops. Data were collected via survey questionnaires in which participants rated their knowledge from 1 (lowest) to 10 (highest) using a Likert-type ordinal scale.

Interview transcript content was analysed thematically. Thematic content analysis is a systematic approach used to analyse recorded communication and textual data.25 26 It uses subjective interpretation of text data to identify themes and patterns.27 Thematic analysis focuses on interpretation within the context of the data.25 26 Using an inductive approach, one of the authors (FM) identified themes and subthemes within the data.26 Within each major theme, subthemes were then assessed by another author (IMS) and matched against the Kirkpatrick model levels. Kirkpatrick-aligned subthemes were then reviewed, discussed and agreed by all authors.

Results

Participant reaction

Webinars

The VALSC offered a free webinar series open to all working across health and care. The series comprised six webinars delivered by members of NHS England’s SI Team (now part of NHS England and NHS Improvement’s new Improvement Directorate) and expert faculty. Each webinar in the series was certified to allow participants to receive continuing professional development (CPD) points. The topics covered by each webinar included: systems leadership (how to lead without hierarchical authority)28–30; the model of large-scale change (which identifies several key stages in the leadership of large-scale change—see figure 1)15; the change model for health and social care (which brings together important aspects of implementing sustainable change into a holistic framework)15 31 32; mobilising and organising (to connect with the core values and motivations of individuals to build commitment to a shared purpose)33–35; working through networks (cooperative structures where groups or individuals coalesce around a shared purpose)36; and measurement and impact (use of data to demonstrate impact and make decisions about targeting change efforts).37

Figure 1

The model of large-scale change.

Between November 2017 and December 2018, almost 1500 delegates from health and care systems across England participated in VALSC online capability-building webinars (see online supplemental data for more detailed descriptions of webinar topics and map of geographic distribution).

Participants’ reactions to webinars were assessed via rating of each session and likelihood to recommend to colleagues. Webinar participants were invited to share their thoughts about the value of the webinars in postwebinar session and postwebinar series surveys. Figure 2 presents webinar participants’ average rating (out of 10) of how well each webinar topic was explained.

Figure 2

Participants’ rating of how well each webinar subject was explained.

Participants were also asked to rate how likely they were to recommend the webinar series to colleagues. Ninety-six per cent reported that they would recommend the webinar series.

Qualitative data from webinar chat box and postcourse surveys were also assessed for participant reaction. Reaction was positive with typical responses including, for example, ‘great session[s]’, ‘can’t wait for the rest’, ‘inspiring’ and ‘this was very worthwhile’.

Participants reported that the webinars were appropriately pitched at the intended audience. They reported receiving the correct ‘level of complexity and learning’ in a way that was ‘informative, practical, clear how to apply locally’. Overall, the webinars were reported as motivating and thought provoking:

Thank you—this is a great way to share knowledge, expertise and to motivate. (Programme Director, Shaping our Future)

The series as a whole is making me think more about large-scale change and my perspective on it. (Improvement Manager)

Masterclasses

The VALSC masterclasses were two-day workshop events aimed at cross-sectoral transformation teams. The masterclasses were free to join and attendance was via a team application to a competitive expression-of-interest process. Masterclasses focused on the thinking and tools of systems leadership and large-scale change. They were very practical in nature, taking a learning in action approach to help teams develop action plans for the next steps in their transformational change programmes. Masterclasses were also CPD certified.

Between January 2018 and June 2019, over 100 system-wide transformation teams from across England comprising over 600 delegates attended VALSC masterclasses (see online supplemental data for map of geographic distribution). Teams attending the masterclasses included delegates from: integrated care systems and ICPs; sustainability and transformation partnerships; FTs; community, acute and mental health providers; local government; local authorities; education providers; CCGs; regulators and national arm’s-length bodies; service users; and the charitable sector.

Transformation programmes worked on by masterclass teams included, for example: priority national programmes underpinning the NHS LTP; community service pathway reviews; place-based system-wide transformation programmes; digital transformation programmes; new models of care programmes (including diabetes and community and inpatient bed-based models); mental health community provision and integration; emergency and urgent care services redesign; regional pathology network provision; pharmacy; local maternity system redesign; and primary care transformation.

Teams attending masterclasses comprised four to eight participants. Participants represented a ‘diagonal slice’ of roles from across organisations from junior to senior and executive level. Modelling the principles of systems leadership, masterclass teams worked in a non-hierarchical manner. Inclusion was based on willingness and ability to influence the transformational change.

Participant reaction to masterclasses was assessed via: observation of participants throughout the workshop; end-of-day evaluation form feedback; photographs of masterclass events; and interviews with sample of participants.

Observation of, and interviews with, participants suggested that the participants’ reaction to masterclasses included seeing them as a space to reflect and work together on their transformation challenge. Participants also gained greater clarity and collective understanding of shared goals. Some participants reported this leading to increased team bonding.

Very useful two days of learning. Good to have the head space to think and re-think the LSC planned. (Anonymous)

Excellent opportunity to stop fighting the fire and plan how to put it out, work[ing] through our system together. (Assistant director)

Participants also reported that independent and constructive challenge from peers and faculty experts was of benefit. This facilitated sharing ideas with, and learning from, others facing similar challenges leading to a sense of community that provided reassurance of not being alone in leading change.

Peer review with the group on our table gave us a light bulb moment. It helped us focus grass roots change. (LMS Lead)

Team [are now] reinvigorated and ready to go back to try new approaches. Fresh determination and commitment to make change happen. (Director, CCG)

Learning impact

Individual knowledge before and after participating in online webinars and/or masterclass workshops was assessed via end-of-session surveys using Likert-type ordinal scales in which participants rated their knowledge from 1 (lowest) to 10 (highest). Baseline data suggested a relatively low level of awareness of large-scale change approaches, methods and tools with few delegates reporting themselves as highly knowledgeable in large-scale change before taking part in the VALSC. Differences in participants’ ipsative, before and after, self-assessment of knowledge data were analysed using the Wilcoxon signed-rank test.21 Knowledge change before and after each webinar and masterclass showed a significant change (p<0.001)—data are summarised in table 2 (graphical distributions of before-and-after data are presented in online supplemental data).

Table 2

Participants’ ipsative self-assessment of knowledge

	Total participants	Evaluation responses	Response rate (%)	Average level of knowledge
				Before	After
Webinars
Systems leadership	857	239	28	4.9	7.5
Leading large-scale change	677	202	30	5.0	7.5
The change model	505	171	34	5.5	7.6
Mobilising	448	141	31	5.2	7.5
Working through networks	454	140	31	4.9	7.0
Measurement and impact	423	120	28	5.1	7.3
Masterclasses
Day 1—Complexity, systems leadership and large-scale change	667	567	85	5.0	7.7
Day 2—Leading large-scale change	664	491	74	5.5	8.3

Data for Virtual Academy of Large-Scale Change (VALSC) webinars from November 2017 to December 2018.

Data for VALSC masterclasses from January 2018 to June 2019.

Behavioural and application impact

The VALSC has helped empower participants with transformation and change knowledge and skills. Interviewees report gaining an increased understanding of how to work in complex contexts. This included appreciation of the interconnected and unpredictable nature of complex systems and, consequently, the challenges this posed for leading change. For some, this led to changes in behaviour which moved away from transactional, programmatic approaches towards the more emergent approach of large-scale change.

We [learnt] that we were not necessarily in control… We are just adapting within that, rather than trying to control something that is so complex. (Service Transformation Lead)

[It has] completely changed me from [taking] a programme/project approach to [adopting] an appropriate tool use for managing complex change. (Programme Manager, Foundation Trust)

In making the behavioural shift, VALSC participants reported as helpful being equipped with, and knowing how to use, a set of evidence-based tools for large-scale change. The tools were selected from the Leading Large-Scale Change Guide21 and assembled into a structured process taking participants through key stages of the large-scale change model.

What we're trying to use are the tools from the [VALSC] programme in actually bringing together staff. (ICP Managing Director)

[It] is about having a different approach to large-scale change and having the tools that are evidence based … and having … a team that takes you through how you can apply and use them. (Senior NHS Commissioning Manager)

The approach helped VALSC participants to know which tools to apply in different circumstances and stages of change. This gave them reassurance and increased confidence in their ability to apply the VALSC’s model and tools locally. This included creating their own permission to innovate, make adjustments and adapt to their context to progress their transformational change programmes.

We have a sense of confidence that we are on the right track to drive forward. (Service Transformation Lead)

Given me the permission to question, to go with my gut and to deliver. (General Manager, NHS Trust)

[We now have a] better understanding of how to engage key stakeholders and frame (reframe) messages. Permission to adapt and improve in light of changed circumstances and opportunities. (Clinical Director, Paediatrics)

By transferring their new knowledge gained through the VALSC into their work practice, VALSC participants developed skills and reported improvements in how their transformation programmes were progressing. In particular, approaches to engagement and mobilisation of stakeholders were reported as having helped to improve communication, secure buy-in and create traction for change. This involved framing in ways that connected with stakeholders and using narrative approaches that tap in to stakeholders’ emotions and raise intrinsic motivation to act in ways which move towards achieving the proposed vision.

We spend time every couple of months really thinking about our stakeholders and framing what we do that makes it relevant …[and] we focus on an agreed group of stakeholders thinking through our strategy and approach. (ICP Managing Director)

What they [the tools shared in the masterclass] have done is to help us hone our stakeholder communications which we’ve focussed on. Creating a metaphor for what we’re trying to do because the concept we’re working on is quite difficult. (Deputy Programme Director)

Furthermore, VALSC participants also reported that application of systems leadership tactics, stakeholder analysis and framing tools had helped to secure strategic leadership and management buy-in to move their change programmes forward.

We know what we have to achieve, who to influence (stakeholders) and the framework to use. (Volunteer, Diabetes UK)

It strengthened our team working across boundaries and we hope to be able to influence our organisation in a positive way. (Anonymous)

We had a Board away day and we actually used some of those [stakeholder mapping] tools again with the board team. (Divisional Director of Operations)

Supportive leadership has been identified as a key enabler for (and if absent a barrier to) successful change.3 38 It has also been proposed that key to successful large-scale change are approaches that can close the gap between stakeholders’ intrinsic motivation for change and the external motivation for change driven by organisational hierarchy.35 Masterclass participants reported that, by applying the methods and tools learnt, they were able to better articulate their vision and objectives and frame them to both motivate key stakeholders and secure the leadership support considered essential to affect successful change.

Summary of main themes

Across the levels of the Kirkpatrick framework, the VALSC contributed to three broad domains. First, it helped to strengthen connections within and between transformational change teams. Second, it empowered participants with transformation and change knowledge and skills which they have been able to apply to health and care challenges. Finally, it helped participating teams to engage better with their stakeholders. The main themes across these three domains and the levels of the Kirkpatrick framework are summarised in table 3.

Table 3

Summary of lessons learnt aligned to levels of Kirkpatrick model

	VALSC is strengthening connections within and between transformational change teams	VALSC is empowering workforce across the system with transformation and change skills which are being applied to health and care challenges	VALSC is helping teams to engage with their stakeholders better
Reaction	Increased team bonding and space to reflect. Independent and constructive challenge.
Learning	Greater clarity of shared goals. Sharing with and learning from peers.	Appreciation of how to effect change in complex contexts. Equipped with an approach and a set of tools.
Behaviour		Increased confidence about transformational change.
Results		Teams creating permission to innovate.	Improved communication with stakeholders. Gained strategic/management buy-in.

VALSC, Virtual Academy of Large-Scale Change.

Limitations

Several limitations to the evaluation have been identified.

Self-reported measures of phenomena can be susceptible to response and response-shift bias.39–41 Self-ratings of knowledge can be higher than if objectively assessed39 42 as people tend to report their own abilities positively.40 Such bias is a threat to reliability of self-reported data and this can be compounded by designs using pre-intervention and post-intervention data collection.41 Retrospective pretest ratings have been recommended as one way to attenuate this.41 Therefore, as pre-workshop ratings were collected from participants retrospectively in end-of-workshop surveys, this should guard against response-shift bias, at least to some extent.

Although ipsative self-assessment of knowledge used ordinal (1–10) scales, how each participant evaluated against those scales may not be consistent. That is, one person’s ‘10’ may, for example, be another person’s ‘8’. This poses a further threat to reliability. However, application of statistical tests to both ordinal and Likert-type data is common and there are competing opinions on appropriateness in the literature.22 There are also examples of similar statistical tests being applied to evaluate healthcare quality improvement capability-building programmes.43

Two of the authors, IMS and EB, worked on the design and delivery of the VALSC programme. There is, therefore, the potential for positivity bias in the interpretation of data. To mitigate this, data collection (including surveys and interviews) and analysis (including qualitative thematic analysis) were undertaken by FM. FM played no part in programme design or delivery and held a substantive role in SI’s Impact, Research and Evaluation team—with a specific remit to support objective evaluation of SI programmes.

Large-scale change can take years to achieve.3 Therefore, the study was limited by a relatively short time frame in which to evaluate implementation impact. Longer follow-up was not possible within the scope of the study and future evaluations may wish to consider the feasibility of longitudinal case studies over a greater duration.

Implications and conclusion

Using the Kirkpatrick model, we have evaluated the VALSC and found that it contributes to developing capability in individuals and teams. The reaction of participants in VALSC activities has been positive and a significant change in knowledge has been quantified. Qualitative feedback from participants has reported adopting new knowledge into practice to develop skills and suggests improvements in how transformational change programmes are being managed. Specifically, the VALSC developed knowledge, skills, behavioural change and application impact relating to several domains of improvement capability: stakeholder engagement; strategic alignment; leadership of change; and employee motivation. The VALSC also aimed to achieve this at large scale.

The VALSC’s online delivery mode offers an approach to developing and sustaining transformation knowledge and skills at scale. It is now over a decade since the original ALSC was established to develop transformational change skills in the NHS. However, baseline self-report data from the VALSC concur with previous studies suggesting that levels of transformational change knowledge and skill in the health and care sectors remain relatively low.10 12 The original ALSC focused intensively on a small group of managers from the regional and national regulatory landscape of the time. In the intervening years, many will have changed roles or left the NHS and it is unclear how, or if, the participants shared their experience and knowledge gained from the original ALSC with others. While there have been subsequent programmes to deliver transformational change skills in the NHS,44 they have, to the knowledge of the authors, continued to focus on discreet cohorts (similar to the original ALSC) and not comprised virtual or online learning elements. Several NHS policies now promote the use of online approaches to develop capability for improvement and change—including within clinical and managerial curricula.45–47 The evaluation may, therefore, be of interest to policy leads to consider how existing online programmes might contribute to achieving this aim.

Furthermore, the ability to work with change in complex adaptive systems is important for successful large-scale transformation.48 Complex contexts challenge the assumption that change can be managed in a transactional, machine-like, way.49 Snowden and Boone suggested that misclassifying complex contexts as simple and attempting to apply traditional management can lead to disorder or chaos.50 However, the data collected (via participant observation of masterclasses, by IMS and EB, and interviews conducted by FM) suggest many VALSC participants were using traditional, transactional programme management approaches to try to deliver complex transformational change. If this was widely replicated across the NHS, it may be a threat to successful change and the transformation envisaged by, for example, the NHS LTP. However, it must be noted that although circa 2000 delegates have participated in the VALSC’s webinars and workshops, this represents only a small proportion of the total health and care staff complement at which it was targeted. That said, the distribution of VALSC participants geographically and organisationally suggests application of transactional practices may be widespread—implying there are many others who could benefit from the programme. We therefore recommend continuing to develop capability-building offers (such as the VALSC) that empower and build agency in front-line staff working on service transformation and equip them with approaches, methods and tools that increase their chances of success and contribute to building improvement capability.