Benefits realization management in the context of a national digital transformation initiative in English provider organizations.

BACKGROUND: The Global Digital Exemplar (GDE) Programme is a national initiative to promote digitally enabled transformation in English provider organizations. The Programme applied benefits realization management techniques to promote and demonstrate transformative outcomes. This work was part of an independent national evaluation of the GDE Programme. AIMS: We explored how benefits realization management was approached and conceptualized in the GDE Programme.
METHODS: We conducted a series of 36 longitudinal case studies of provider organizations participating in the GDE Programme, 12 of which were in depth. Data collection included a combination of 628 interviews (with implementation staff in provider organizations, national programme management staff, and suppliers), 499 documents (of national and local implementation plans and lessons learned), and 190 nonparticipant observations (of national and local programme management meetings to develop insights into the broader context of benefits realization activities, tensions arising, and how these were negotiated). Data were coded drawing on a sociotechnical framework developed in related work and thematically analyzed, initially within and then across cases, with the help of NVivo 11 software.
RESULTS: Most stakeholders broadly agreed with the rationale of benefits realization in the GDE Programme to show due diligence that public money was appropriately spent, and to develop an evidence base supporting the value of digitally enabled transformation. Differing national and local reporting purposes, however, created tensions. Central requirements, for progress reporting and tracking high-level benefits, had limited perceived local value and were seen to impose an unnecessary burden on provider organizations. This was accentuated by the lack of harmonization of reporting requirements to different stakeholders (which differed in content and timing). There were tensions between the desire for early evidence of outcomes and the slow processes of infrastructural change (which created problems of attribution of benefits to causes as benefits emerged gradually and over long timeframes), and also between reporting immediately visible local changes and showing how these flowed through to high level organization wide benefits (eg, in terms of health outcomes or cost savings/return on investment). The attempt to fulfill these diverging agendas and informational needs within a single reporting tool had limited success. These difficulties were mitigated by efforts to simplify reporting requirements and to support targeted collection of key national outcome measures. Although progress was hampered by an initial lack of benefits realization expertise in provider organizations, some providers subsequently retained these skills for their own change management purposes.
CONCLUSIONS: There is a need to recognize the limitations and cost of benefits realization management practices in the context of healthcare digitalization where benefits may materialize over long timeframes and in unanticipated ways. Although diverse stakeholder information needs may create tensions, prior agreement about rationales for collecting information and a targeted approach to tracking local and high-level benefits may enhance local relevance, reduce perceived reporting burdens, and improve acceptance/effectiveness. A single integrated reporting mechanism is unlikely to fulfill both national and local requirements.

INTRODUCTION

Although there is an international drive towards implementing increasingly large-scale health information infrastructures internationally, outcomes are mixed, with many projects failing to deliver projected improvements in efficiency and quality and safety of care. Benefits realization management (BRM) techniques have been promoted and increasingly applied to such projects with the aim to demonstrate that health information technology investments are leading to desired outcomes. Here, 2 related aims are conflated: (1) to exercise accountability and demonstrate due diligence by showing that money is properly spent and delivers returns on investment; and (2) to manage change and ensure that benefits are realized. There are different definitions and interpretations of benefits and a lack of agreement on BRM approaches, but the overall aim is maximizing value and collecting evidence for this value. BRM is widely applied to IT implementations in commercial settings. It is an instrumental management approach, presuming that benefits can be attributed to particular sets of causes, that organizational goals can be unambiguously captured, that progress in reaching these goals is visible in short timescales, and that change is demonstrable through quantitative outcome measures. These kinds of retrospective cost–benefit assessment were effective in capturing benefits when initial computerization displaced discrete administrative tasks (eg, book-keeping and routinized administrative roles), but became problematic when complex organizational functions were being computerized, and where substantial investments in renewing core information infrastructures did not yield immediately visible returns.

While the literature shows that BRM can be helpful, particularly in helping to align agendas and in bridging the gap between strategic and implementation considerations in business settings,, there is limited evidence in the empirical literature of its effectiveness on implementation outcomes., Even where a decision is made to proceed with BRM, many organizations do not invest heavily in monitoring outcomes and there is limited actual use of BRM. This may be due to the complex social and organizational contexts in which attempts to measure benefits of IT are made; standardized methodologies not working across organizational settings; and the transformative nature of contemporary IT applications, which involves an extended period following implementation that can be exploited to improve processes and workflows. As a result, benefits are likely to take a long time to materialize, and emerge in often unanticipated ways and places which cannot be readily established by before-and-after comparisons. In addition, there may be tensions between stakeholders, as there are different purposes for information reporting, definitions of BRM approaches vary, and benefits to some may be disbenefits to others.

These limitations are exacerbated in healthcare settings, where the range of stakeholders is large, and existing information infrastructures are complex assemblages of multiple separately developed systems. BRM and other change management tools designed for discrete technology systems may not work in these settings, but they are still frequently applied. Empirical research in the area of BRM in digital transformation in healthcare settings is limited. Where it exists, the limitations of BRM have been raised as problematic, but there is currently no alternative conceptualization of BRM in these settings. Our work seeks to address this gap.

We draw on a recently completed formative evaluation of a national digitally enabled transformation initiative in English provider organizations (Table  1). The Global Digital Exemplar (GDE) Programme was launched in the context of prior experience of IT procurements that failed to deliver promised organizational transformation. BRM was seen as a way to emphasize national transformation goals and achievements. Our aim was to explore how BRM was approached and conceptualized in the context of this Programme and how the effectiveness of BRM approaches could be maximized going forward.

Table 1.

The Global Digital Exemplar Programme

The Global Digital Exemplar (GDE) Programme is an ambitious first-of-type national initiative with twin goals: to advance digitally enabled (service) transformation in selected exemplar NHS England provider organizations already characterized by relatively high levels of digital maturity and bring them up to an international level to create a national learning ecosystem to spread the knowledge acquired

The Programme was launched after the 2016 Wachter Review proposed a phased approach to digitization of the English NHS as the scale of investment required to bring all provider organizations to digital maturity greatly exceeded available resources. The Wachter Review therefore recommended creating a cohort of digitally advanced exemplar provider organizations (hereafter GDEs) intended to transmit learning to less mature ‘Fast Follower’ provider organizations (FFs) and thereby catalyze large-scale digitally enabled transformation of the wider English NHS. Provider organizations that were shortlisted were invited to propose ambitious portfolios of digital change (including core infrastructure upgrades and implementation of complex transformational systems), to be implemented over 2 to 3.5 years.

Provider organizations were selected in several tranches from December 2016. The GDE Programme involved 33 acute provider organizations, 15 mental health provider organizations, and 3 ambulance provider organizations, resulting in 51 provider organizations (23 GDEs and 25 FFs who paired up to share knowledge, 3 of these merged during the Programme). GDE acute provider organizations each received £10 million and FFs received £5 million. Mental health GDE organizations received £5 million and relevant FFs received £3 million. All organizations were required to secure internal matched funding.

The Healthcare Information and Management Systems Society (HIMSS) Electronic Medical Record Adoption Model (EMRAM) was chosen as one of the guiding benchmarking criteria for the GDE Programme, with the expectation that GDEs and FFs would respectively obtain HIMSS Level 7 and HIMSS Level 5 or equivalent by the end of the Programme. Limitations of this model and its applicability across the NHS were recognized, for example, by setting a lower EMRAM target (Level 5) for mental health providers. The HIMSS EMRAM 2018 acute version was used for all the assessments.

In addition to supporting digitally enabled transformation within selected provider organizations, the GDE Programme offered national support for the establishment of programme governance and delivery assurance arrangements as well as supporting various mechanisms for sharing learning, including the development of a GDE–FF partnerships and Blueprinting, and establishing various learning networks to capture and share implementation experiences.

METHODS

We conducted a longitudinal qualitative investigation exploring how BRM was conceptualized and executed as part of the GDE Programme. Our full study protocol is described elsewhere. The qualitative longitudinal nature of this study allowed us to explore how BRM was conceived, how it evolved, and how emerging tensions were negotiated from a range of stakeholder perspectives.

Data collection

Data collection occurred between January 2018 and December 2020. Organizations participating in the GDE Programme had also agreed to take part in the independent evaluation. Individual respondents were approached via email initially, provided with information about the study, and asked to respond if they were interested in participating. Most invited participants took part in the study, although we had some nonresponders and some participants declined participation as they were too busy with care-related activities (around 10% of the sample). The majority of data collection took place face-to-face, although particularly in the latter stages of the study during the COVID-19 pandemic, we conducted our data collection online or over the telephone.

The study involved a number of interrelated elements:

Detailed case studies of set of 12 provider organizations, chosen to include a range of GDE and fast follower (FF) acute provider organizations and mental health providers representative of the wider sample. These were sampled for maximum variation and included a range of core electronic health record systems, geographical locations, and sizes. We conducted interviews and repeated these over the 3 years of the evaluation. Participants included staff directly involved in running and implementing the Programme and its portfolio of digital change projects within the provider organization. A purposive sample of respondents included senior managers and clinical staff involved in implementing and/or planning the implementation of systems as part of local GDE Programme activities and sought to encompass a range of viewpoints, backgrounds, and levels of seniority. These were recruited through gatekeepers at each site (including Chief Clinical Information Officers) who recommended individuals based on their role. In order to mitigate risk of positivity bias, we also asked each participant to recommend further interviewees, in particular those with diverging viewpoints. This was further supported by collection of documents (eg, business plans, lessons learned) in order to gain insights into implementation plans, and observations of meetings that related to GDE Programme activities (in order to gain insights into social dynamics and planning activities). Data in detailed case studies were collected by lead academic researchers responsible for sites (MK, SH, and HN).

Broader studies of 24 provider organizations, representing the wider population of GDE and FF sites. These included the remainder of sites that were not sampled as part of the detailed case studies. Data collected were similar to the in-depth case studies but less detailed. This allowed us to test emerging findings and ascertain whether these applied to the wider study population. Data in the broader studies were collected by lead researchers responsible for broader sites (SE and WL). Interviewers were not academics but had strong backgrounds in health service planning and delivery which equipped them with some prior knowledge of the organizations and individuals involved.

Programme level study: We further conducted interviews with senior staff involved in designing, launching and running the GDE Programme, and with frontline staff responsible for monitoring and evaluating the performance of participating provider organizations. These were sampled purposively through national programme level gatekeepers, including the Senior Responsible Owner for the implementation of the overall Programme. We then also asked each participant to recommend further interviewees, in order to gain insights into a variety of perspectives, including dissenting opinions. Interviewees also included Engagement Leads and Benefits Leads (specialists employed by the Programme but linked to provider organizations in their region to support engagement of the provider organization and assist progress and outcome reporting). The Evaluation Team was given full access to the series of regular meetings charged with oversight of the GDE Programme including access to monthly teleconferences and documentation submitted to these. This included programme management and monitoring, benefit realization, and accreditation activities with provider organizations. RW and KC, experienced social scientists, led the majority of data collection in the programme level study, with some interviews conducted by HN.

Semistructured interviews with individuals and groups were audio-recorded and transcribed. The length of these was on average around 1 h. While we did have topic guides (which we piloted and revised in line with emerging findings), we sought to keep the conversation informal to establish rapport also allow participants to raise issues important to them.

During observations, researchers took notes. Table  2 provides an overview of interview topic guides and observation themes.

Table 2.

High-level interview topic guide and observation themes

Interviews Background of interviewee, site, and GDE Programme activities (prompt: overall views and experiences) Views of BRM in organization and nationally (prompt: perceived value, tensions, benefits and disbenefits) Suggestions for improvement of BRM approach (prompt: from this point forward, looking back) Attempts to consolidate perspectives and resolve tensions (prompt: highlight other perspectives)

Observations Observation of BRM plans and activities Execution of plans Ways to measure benefits Emerging tensions

Abbreviations: BRM: Benefits realization management; GDE: Global Digital Exemplar.

We collected data in 4 rounds: May 2018–February 2019, March 2019–May 2019, June 2019–March 2020, and August 2020–December 2020. We stopped data collection when we reached saturation and no new themes were emerging.

Analysis

We uploaded all transcripts, notes, and documents into NVivo 11 software and conducted a content analysis. Researchers (MK, SH, and HN) coded data individually within cases against a theory-informed coding framework that we have developed in related work. Here, we coded qualitative data against the following themes within the framework: technological (eg, usability), social/human (eg, integration with existing practices), organizational (eg, strategies to implement change), and macro-environmental (eg, political and economic considerations). Within this, we focused on benefits realization as a thematic strand and paid particular attention to emerging tensions between these overall categories. We also allowed additional themes to emerge. The sociotechnical perspective allowed us to explore how BRM approaches were applied to digitalization and also how digital tools were used to implement BRM methodologies and how these impacted on various stakeholders.

Detailed analysis within in-depth case study sites was followed by cross-case analysis and integration with data from broader case studies and programme level studies. Cross-case analysis identified common challenges experienced with BRM approaches, how these evolved over time and how they were shaped by individual contexts. In doing so, we focused on various perspectives of benefits realization within the Programme, including plans, experiences, and tensions. Emerging tensions between stakeholder groups were explored in most detail. In order to achieve this, we conducted detailed analysis workshops with the whole team, exploring rationales, approaches, and integrating these within wider Programme activities and sociopolitical environments. The results below reflect common challenges, tensions between stakeholder groups, and how these were negotiated over time.

Ethical approval

We obtained ethical approval from the School of Social and Political Science at The University of Edinburgh, UK.

RESULTS

Our dataset consisted of 556 interviews, 86 meetings, and 387 documents in 36 provider organizations (see Table  3). Programme level data collected included 72 interviews; observation of 104 national meetings, workshops, and conferences; and 112 documents.

Table 3.

Our sample

In-depth case study sites
12 provider organizations 8 GDEs: 6 acute, 2 mental health 4 FFs: 3 acute, 1 specialist FF	309 interviews (39 senior managers, 65 clinical digital leaders, 47 nonclinical digital leaders; 46 GDE Programme staff, 112 operational staff) 104 documents 67 meetings observed 247 interviews (32 senior managers, 78 clinical digital leaders, 65 nonclinical digital leaders, 44 GDE Programme staff, 28 operational staff) 283 documents 19 meetings observed
Broad case study sites
24 provider organizations 15 GDEs: 10 acute, 5 mental health 9 acute FFs
Programme level study
72 interviews (61 policy makers, 3 vendors, 4 engagement leads, 4 other stakeholders) Nonparticipant observations of 104 national meetings, workshops, and conferences 112 documents

Abbreviations: FF: fast follower; GDE: Global Digital Exemplar.

The overall BRM approach employed in the Programme is summarized in Box 1.

Box 1.

Overview of the BRM approach as part of the GDE Programme

Building on and learning lessons from previous national benefits realization approaches

Significant investment in benefits realization as central part of the Programme (with the aim to evidence returns on investment and justify future investments at both organizational and national levels)

View of benefits realization as part of quality improvement

Engagement with benefits realization activities was part of the funding agreement with provider organizations

Common framework and processes across participating provider organizations

To promote service enhancements and local engagement, organizations were required to draft a Statement of Planned Benefits before commencing implementation activities

Provision of guidance and learning to facilitate benefits realization capability and governance at local organizational levels

Focus on tracking benefits over time

Focus on safety/quality, clinical outcomes, staff and patient experience, resource sustainability as sources of value (not just financial impact)

Proactive engagement of organizations in managing and realizing benefits

We identified the following themes, which we discuss in detail below:

Tight coupling of national and organizational benefits realization.

Challenges associated with consolidating various information requirements in a single data collection procedure.

Broad agreement on the rationale for benefits realization and uses of data.

Embedding of benefits realization approaches in provider organizations over time.

Tight coupling of national and organizational benefits realization

The Programme initially envisaged light touch monitoring, but as it evolved, reporting requirements became stricter due to top-level pressure for tighter scrutiny. Benefit realization activities were tightly coupled with release of national funding and with accreditation targets for organizations participating in the Programme. Programme-wide reporting systems were put into place and various initiatives were developed to facilitate reporting (eg, electronic reporting tools, Benefits Realization specialists [‘Leads’] seconded to sites). However, provider organizations were already required to submit various sets of reports to their own boards and to national reporting systems. These varied considerably in terms of the content of information collected, frequency, and reporting dates. The requirements of the central reporting system inevitably differed from these many preexisting organizational reporting systems, as they had different informational goals and operated within different timeframes of realizing and baselining these. The new reports thus imposed a degree of additional burden.

Yes, there’s lots of duplication and I think what they’ve done is focused on the inputs rather than the outputs so it becomes very labour-intensive to keep it all up-to-date which you wouldn’t mind if there was a benefit to it, if it produced your reports, if it produced the outputs they’re looking for, but it doesn’t. (Site D, GDE, in-depth case study, deputy Chief Clinical Information Officer [CCIO])

Programme managers hoped that the new reporting tools could be used to fulfill existing reporting requirements. However, the diverse stakeholder information and reporting requirements were difficult to consolidate, and the content and timing of information collected and programme reports required inevitably differed from existing organizational reporting and project management systems. The apparent duplication of reporting and the fact that reporting systems did not meet local reporting requirements led local stakeholders to emphasize the perceived burden of reporting without obvious local benefit. The resources needed to collect benefits/outcome data locally were high and fell upon organization members who did not experience benefits from their use.

[Programme office] wouldn’t insist on that as part of the assurance and if that wasn’t linked to the release of funding, then I’m not sure it would get done to that degree, with that degree of rigor. With the best will in the world, you know, if there wasn’t a bit of a carrot and stick scenario, I’m not sure how far we’d get with that, so I can completely understand why they’ve done it, and it is important, just is a very bureaucratic process and takes a lot of time. (Site M, GDE, in-depth case study, project manager)

As a result, many provider organizations perceived benefit reporting to the programme office as burdensome, time-consuming, and resource intensive.

We knew we had to do benefits and stuff but the amount of work that it’s actually taken … It’s a bit like the governance or the documentation we have to do, or the benefits stuff we have to do, has almost outweighed the amount of work on projects. (Site F, GDE, in-depth case study, benefits lead)

Challenges associated with consolidating various information requirements in a single data collection procedure

Issues also arose about the different informational requirements and criteria of provider organizations and the array of external stakeholders involved in monitoring the GDE Programme (including Her Majesty's Treasury, see Figure  1).

Figure 1.

Stakeholder map of strategic governance structures in the GDE Programme.

NHS England and NHS Improvement: these work together to manage England's National Health Service; NHS Digital: provider of digital services in NHS England; Local Health and Care Record Exemplars: regional sharing of information and digital health and care services; NHSX: a public body that holds the budget for digitalization and commissions services from NHS Digital; Health Education England: national body coordinating training and education of the workforce.

The various local and national stakeholder groups had differing responsibilities/concerns. They therefore had varying rationales for benefits realization and informational needs, and placed value on different types of outcomes. For example, while provider organizations prioritized specific local service improvements, national programme managers sought to demonstrate high-level public benefits to justify future investments to the Treasury and secure prioritization for digital transformation. They wanted this information for both retrospective assessments of return on investment and prospective validation of business models for future investment. In doing so, they sought to link local programme outcomes to health system-wide missions (eg, improving population health and efficiency). These systems-wide benefits might, however, not be directly experienced by local organizations.

So this is all the cash releasing, non-cash releasing or public money, it's all the money intertwined so it may not necessarily mean cash back to the organisation, it covers it as a whole. (Site E, GDE, in-depth study, benefits realisation lead)

The benefits realization process operated as if these diverse needs could all be met within a single concerted information collection procedure. However, different constituencies sought different kinds of information to answer different questions. As a result, provider organizations, although understanding the rationale behind the overall approach, struggled with the decontextualized nature of benefits measurement and the demand to link local benefits to high-level national mission/targets. For instance, national stakeholders sought to validate overall outcomes of the Programme to demonstrate benefits of digital transformation investment. They were therefore keen to be able to point to aggregate changes, eg, in productivity and quality of care. In contrast, the local experience of qualitative benefits in care delivery, which was of great importance to site members, was more difficult to link to accreditation criteria.

I get the point of it and I understand why they do it and I understand that actually they’ve got to choose some way of measuring me and I can see why they would choose that way of measuring it but it’s so … some things are just so quality driven rather than quantity driven. (Site J, FF, in-depth case study, project lead)

It was hard to reconcile these different informational requirements in a single reporting structure—not least because some of the digital transformation benefits only materialized after a system had been embedded and optimized.

Also this whole thing around benefits realisation is really a bugbear of mine. Because if they want a good qualitative evaluation, then we need to do that separately rather than look at it from a milestone perspective and also give it time to embed to see whether it benefits people. (Site F, GDE, in-depth case study, CCIO)

Programme managers were acutely aware of the tensions emerging from the burden of central reporting requirements for provider organizations. In order to address this issue and consolidate national and local data collection, they revised the benefits realization process and changed the reporting tool over time from a spreadsheet to a specially commissioned automated reporting tool that was upgraded throughout the Programme.

Programme managers envisaged the reporting tool would become a real-time evidence base of IT benefits that could be used to guide national and local investment decisions. This rationale was largely understood and supported by provider organizations. There was a hope that it might be a means to harmonize reporting requirements, and thereby reduce duplication of data collection, but only limited progress had been made in this direction by the end of the evaluation.

Provider organizations needed training in how to use the new tool and also had to manually reenter some data.

We started last year before [new reporting tool] was ready, or some of that wasn’t ready, or we weren’t using it, so the benefits … we did it one way and then we did another. We did it all on a spreadsheet and we did it all … copy and paste all that in there so you get used to one way and then you do it another way; so with the other reporting it was done on paper and then it was done in the system and … so that was all still developing which didn’t help, and just confused me, frankly. (Site M, FF, in-depth case study, GDE Programme manager)

Issues were exacerbated by perceived shortcomings in the usability of the national reporting tool such as its lack of intuitive navigation. The system was perceived as cumbersome and difficult to complete with many mandatory fields. There were attempts to address these usability issues and to tailor the tool to local needs. These included reducing the number of fields that had to be completed and aligning existing field with fields required for local reporting, meaning that information could be reused for local BRM activities and associated reporting purposes.

… we’ve had a lot of issues with [tool]. I’d say that [tool] probably gives [programme managers] what it needs, but from a [provider organisation] perspective, we don’t like [tool]. It’s just the layout of it and everything else, it’s very, very different from what we’d probably previously looked at. I mean, it was a big change for us, we had the training and everything, we knew what we needed to do, and even now it’s not as user friendly as we would like it in a hospital setting. So it’s quite cumbersome really. (Site B, FF, in-depth case study, business change manager)

Broad agreement on the rationale for benefits realization and uses of data

Despite these challenges, we observed a broad overall agreement across stakeholder groups on the ethos of demonstrating the achievement of milestones/benefits to show due diligence that public money was appropriately spent, and to develop an evidence base supporting the value of digitally enabled transformation.

Provider organizations, with limited existing benefits realization capabilities, had to buy in people and employ intermediaries to assist with these activities and help to satisfy national reporting requirements.

… it almost becomes somebody’s full-time job just to manage feeding [programme management office] with information. (Site D, GDE, in-depth case study, deputy Chief Information Officer [CIO])

GDE programme managers had put in place various structures to facilitate benefits realization, some of which evolved during the Programme. These included a requirement for provider organizations to produce a Statement of Planned Benefits before embarking on programme activities, and the appointment of national and local benefits realization managers who managed the measurement and tracking of benefits over time. This national support and guidance was greatly appreciated by organizations, as benefits realization expertise was in short supply. Sites also appreciated the support of the national team to help them meet accreditation requirements.

… the support that I’ve had from the [national Programme] team has been really good in the fact that they’ve trained me and they’ve been a constant source of support and resource really when I’ve needed it. (Site B, FF, in-depth case study, business change manager)

In addition, benefits realization work was perceived to facilitate the setting of a common direction of travel.

You can have the best system in the world that all works, but unless you sell the story and direction of travel, I think sometimes there is a fascination to get, let’s just go for this bit here, and you don’t sell them the whole story … I don’t think we record the benefits as well as perhaps we should … I don’t think enough work goes, which you can then publish to allow others who are embarking down similar programmes to go, wow, that’s fantastic. (Site I, GDE, in-depth case study, digital lead)

The national team worked with provider organizations to collate and validate benefits information. Although this was an expensive process, the team leveraged value from the data and used it in new ways (eg, the opportunity was identified to add this data to Blueprints [formal documents capturing implementation experience] to help other sites considering such an implementation to develop and evidence business plans). Towards the end of the Programme, national programme managers had also established a map of baseline measurements and benefits associated with digitally enabled transformation initiatives at provider organizations that could help them to support national business cases for future funding of digital transformation programmes.

Embedding benefits realization approaches in provider organizations over time

Many burdensome benefits realization duties in provider organizations were given to temporary appointments, who left at the end of the GDE Programme. Some sites over time recognized the benefits of local capabilities and reported increasing embedding of benefits realization approaches driven by the GDE Programme recognizing the value of the approach in implementing change and pursuing quality improvement. This was particularly true for organizations that had coupled quality improvement with benefits realization from the start and invested in appropriate baselining of benefits information.

We’ve gone through the right processes and we are focusing on getting solid baseline data. (Site D, GDE, in-depth case study, deputy CCIO)

Some also mentioned that the benefits management approach encouraged reflection and enabled them to engage in detailed and integrated strategic planning activity, guided by what they wanted to achieve through digitally enabled transformation from the start.

I think as an IT department, the [provider organisation] has probably learnt that we need to consider how we’re putting systems in and not just to put systems in and launch them, so to consider all what’s needed, what are the benefits of putting it in, what will the benefits to the [provider organisation] be. So I think that has made us stop and think a bit more about that side of things, rather than just going out there and launching new systems and putting new pieces of kit into places. (Site F, GDE, in-depth case study, project support officer)

The nationally supported approach to benefits realization was also seen to help promote clinical engagement locally, as managers could demonstrate the achievement of clinical benefits attributable to new digital systems and thereby motivate clinical users to use these systems.

… when people are wary of change, we remind them of what a difference it can make to their lives as clinical operational people. So we found that really, really useful … (Site B, FF, in-depth case study, CIO)

Now, since then they’ve seen the benefits and realised that the 725 drug errors we didn’t have last year is because we’ve introduced this positive patient identification system so now they’re going, this is brilliant, we want to do this on all the sites. (Site AG 5, GDE, broader study, CNIO)

In addition, organizations realized over time that collecting data about benefits helped them—both in making cases for future local funding for digital technologies and in increasing local recognition of the need to pursue digitally enabled transformation as opposed to merely focusing on technology acquisition.

Well we have to do efficiency as well because they all cost and that’s … we know that if we’re going to put in closed loop medication to get to stage six, that’s going to become a huge cost and there’s going to be a revenue trail to that as well. (Site E, GDE, in-depth case study, CCIO)

DISCUSSION

Summary of findings

Although most stakeholders agreed on the overall importance of benefits realization approaches to support evidencing of investments and facilitating a common direction of travel, the tight coupling of national and organizational benefits created tensions. Here, different stakeholders prioritized different types of benefits and sought evidence about outcomes over different timeframes. These requirements were difficult to harmonize in one reporting tool, and as a result, organizations perceived recording data that was not immediately useful to them as burdensome—given that evidence collection was time consuming. This was exacerbated by a perceived lack of usability of the reporting tool, which was only partly mitigated by improvements to usability and the increasing BRM capability (and familiarity with the tool) within provider organizations.

Strengths and limitations

Our work has generated insights into the benefits realization approaches of a national digital transformation programme in healthcare settings over a period of 3 years. As far as we are aware, there is no comparable study in this setting that has examined benefits realization approaches from a variety of perspectives, including national programme managers and provider organizations, over such extended timeframes. However, there are also some limitations to our work. First, we have focused on national and local managers, who although with clinical backgrounds, may not have reflected the wider healthcare worker population. We obtained limited insights into the frontline staff perspective, which may have unearthed further complexities and unintended consequences associated with BRM. Second, the Programme focused on a cohort of already digitally and strategically mature organizations. Many issues uncovered, although context dependent, are likely to be transferable to other digitally advanced provider organizations in international settings, but may not be transferable to the wider pool of organizations that are less digitally mature. Third, although benefits realization was an important component of our work, it was not the sole focus. Other strands of investigation surrounding the wider evaluation work explored digitalization processes and outcomes, transfer of digitalization knowledge, and the creation of a learning ecosystem. On the one hand, this allowed us to explore developments in context, but on the other hand, it may have limited the depth of insights. Some specific administrative and policy dimensions are likely to be (at least partly) unique to the English setting. For example, the BRM approach employed has built on previous experience, and was in itself innovative as it emphasized value in the broader sense as opposed to financial returns. It also encompassed viewing BRM as an essential part of quality improvement, was driven by local planning of benefits, and made local engagement in BRM activities a requirement (Box 1). However, even in the context of a mature BRM model tensions emerged (see below). These must be managed across all BRM initiatives but the inherent trade-offs may be accommodated differently in different settings.

Integration of findings with existing literature

Our work supports the notion that various stakeholders need to be involved in developing and actively shaping benefits realization processes. However, in the context of large-scale digitally enabled transformation, these stakeholders need to include frontline healthcare workers and health system managers. In this context, our work has shown that BRM did not succeed as a single integrated reporting mechanism, as it could not cater for different purposes of diverse stakeholders. This in turn had consequences for what was reported (eg, high-level health system benefits that may not necessarily be relevant to local organizations) and when evidence was sought (eg, to make the business case for new investments). It supports the need for critical examination of which informational needs and purposes are and are not supported in BRM activities.

Our work shows that tensions emerged around due to the coupling of local and national benefits, where provider organizations were mandated to provide information in particular forms, level of detail, and frequency that were not of immediate value to them. This exemplifies an enduring tension, recognized in organizational theory, between tight coupling characterized by organizational control over and dependency of subsystems, and loose-coupling where organizational subsystems function as one but still have their own underlying identity and autonomy. BRM activities, aligned with technology procurement and implementation timetables are out of synch with the gradual emergence of transformatory benefits as systems are embedded and optimized.

We further observed the difficulty of consolidating the differing informational needs of various stakeholders within a single structured reporting tool. Benefits reporting was a dividing subject, as local organizations were charged with collecting and collating data but did not directly benefit from its use. Grudin’s Law suggests that such uneven distribution of the costs of running a system and the benefits of using it is likely to undermine acceptance of a system. This is also well-documented in healthcare settings. A shift from benefits realization to more evidence-based approaches to realize value associated with health information technology has already been advocated and may be a suitable way forward.

At the outset there was little expertise in benefits realization techniques within participating provider organizations and our work also points to benefits of benefits realization activities. Some organizations found these techniques were helpful (eg, in collecting information about the clinical benefits of a digital innovation that could be used to motivate engagement/adoption) and sought to retain BRM capabilities at the end of the Programme to assist in managing further digitization.

Implications for policy and practice emerging from this work

The provision of large-scale national funding perhaps inevitably introduces the expectation of demonstrating probity/value in spending tax payers’ money. Public health service provision offers a distinctive context for exploiting implementation experience with multiple semiautonomous organizational units receiving central funding. There is therefore scope to reuse benefits information and utilize this information more widely (particularly as capturing this data is often costly). There is also a need to prioritize between competing claims of different parts of the organization. For BRM in health services to be effective and align with various stakeholder needs, there must be active engagement and formative development of BRM processes. This process needs to reflect a wider understanding of care quality, business, and societal benefits.

As is often the case with national initiatives, there is a balance to be struck between what is required for oversight, transparency, and accountability versus the burden of local data collection and reporting. Mapping different stakeholders’ and informational needs, as well as benefits for various stakeholders emerging from BRM is therefore a necessary first step to maximize the chances of successful adoption.

In light of the high costs of data collection for organizations, there is also a need to keep data collection proportionate and focusing on core information needs (eg, to get national funding and to help organizations write business cases) with a limited number of metrics. The tensions arising from tight coupling of multiple reporting timeframes and purposes may be avoided by lightweight reporting to ensure accountability over progress without enforcing rigid compliance with planned timetables. GDE programme managers recognized these tensions, but their room to manoeuver was constrained both by annualized funding cycles and by the political prominence of the Programme which provoked additional central reporting demands. An alternative approach may be to develop some agreed high-level national measures and some more detailed local measures that vary across organizations and systems.

In addition, there is a need to develop user friendly tools for data collection, coproduced with users; to scope existing datasets that may contribute to routine data extraction; to invest in other forms of automatic data extraction, simulation, and verification; and to get these tools adopted as standard across the organization. Developing a benefits realization roadmap, which involves preparation for identification of benefits at various levels followed by realization at various points of time and auditing the achievements, may also be helpful.

We summarize some facilitators for effective acceptance of benefits reporting in Table  4.

Table 4.

Facilitators for effective acceptance of benefits reporting

Coproduce benefits realization methods with provider organizations Clarify reasons for data collection Harmonize reporting tools and timeframes Use tools to plan future changes rather than apply them in the course of the programme Be careful not to shift (or appear to shift) goalposts—communicate adjustments clearly Share benefits as part of the learning ecosystem Recognize that it is difficult (and expensive) to collect robust evidence needed to justify past investment or make the case for future procurement is difficult (especially within short timeframes) Recognize that benefits and cost savings from infrastructure renewal and optimization emerge gradually and are hard to attribute Encourage recording of unanticipated benefits and risks Shift towards a targeted approach which recognizes that some forms of benefit realization information are expensive to collect, validate, and curate—it may be necessary to decide which are worth reporting and resource their collection appropriately

These experiences from a national health system have important implications for approaches to BRM which is likely to have increasing salience in ongoing large-scale digitalization initiatives in healthcare internationally. First, the adoption of standardized approaches within and across nations may neither be necessary nor desirable as contexts, stakeholders, systems, and needs vary. The focus should be not on national standardization but on the alignment of national and organizational interests that need to be represented in the approach. Second, evaluation of systems goes beyond traditional BRM and needs to include formative work and not focus solely on cost–benefit. Our work has shown that a narrow focus on BRM in terms of cost–benefit or national indicators creates a tension between national and organizational stakeholder groups. This is recognized by some countries which adopt a holistic approach to evaluation including delivery and strategy, and exploring both positive and negative outcomes as well as their trade-offs. Thirdly, our work gives insights into the practicalities of implementing benefits measurement tools. It therefore provides directions as to how existing BRM methodologies and frameworks can be implemented and adopted by participating organizations.

CONCLUSIONS

BRM emerged and has been widely adopted in commercial settings, although questions have been raised about its value and the extent to which it is actually used. These questions are particularly pertinent when considering applying BRM in the complex institutional and technological setting of public health digitization programmes and when installing/upgrading core information infrastructures in which implementation/optimization is often protracted and outcomes (expected and unexpected) emerge gradually. Our study highlights differences between various stakeholders in both their information needs (in terms of content and timing of reporting) and purposes (relating to safety and quality of care, process management, and justification of investments). There is no easy way to solve the tensions in meeting these diverse needs. Most importantly, a single integrated reporting mechanism is unlikely to succeed. In addition, capturing reliable evidence is expensive. It may therefore be necessary to decide which benefit realization measures are worth reporting and resource their collection appropriately.

FUNDING

This article has drawn on a program of independent research funded by NHS England. The views expressed are those of the authors and not necessarily those of the NHS, NHS England, or NHS Digital. This work was also supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Care.

AUTHOR CONTRIBUTIONS

KC and RW conceived this paper. KC and RW led the drafting of the manuscript and all authors commented on drafts of the manuscript.