Twenty-five years of national health IT: exploring strategy, structure, and systems in the English NHS.

Objective: There is global interest in implementing national information systems to support healthcare, and the National Health Service in England (NHS) has a troubled 25-year history in this sphere. Our objective was to chronicle structural reorganizations within the NHS from 1973 to 2017, alongside concurrent national information technology (IT) strategies, as the basis for developing a conceptual model to aid understanding of the organizational factors involved. Materials and
Methods: We undertook an exploratory, retrospective longitudinal case study by reviewing strategic plans, legislation, and health policy documents, and constructed schemata for evolving structure and strategy. Literature on multi-organizational forms, complexity, national-level health IT implementations, and mega-projects was reviewed to identify factors that mapped to the schemata. Guided by strong structuration theory, these factors were superimposed on a simplified structural schema to create the conceptual model.
Results: Against a background of frequent NHS reorganizations, there has been a logical and emergent NHS IT strategy focusing progressively on technical and data standards, connectivity, applications, and consolidation. The NHS has a complex and hierarchical multi-organization form in which restructuring may impact a range of intra- and inter-organizational factors. Discussion: NHS-wide IT programs have generally failed to meet expectations, though evaluations have usually overlooked longer-term progress. Realizing a long-term health IT strategy may be impeded by volatility of the implementation environment as organizational structures and relationships change. Key factors influencing the strategy-structure dyad can be superimposed on the tiered NHS structure to facilitate analysis of their impact.
Conclusion: Alignment between incremental health IT strategy and dynamic structure is an under-researched area. Lessons from organizational studies and the management of mega-projects may help in understanding some of the ongoing challenges.

INTRODUCTION

The increasing scale and complexity of health information systems and the trend towards national (and even supranational) initiatives have been acknowledged. In the English National Health Service (NHS), national information technology (IT) strategies have a 25-year history and have evolved to keep pace with technological advances, expectations of service users, and anticipated benefits around safety, service effectiveness, and efficiency. In parallel, the NHS has undergone frequent restructuring with changes in organizational boundaries, reporting arrangements, funding mechanisms, and the location of care delivery. Within this volatile landscape, attempts to implement NHS-wide IT strategies have often fallen short of expectations.

This paper explores 25 years of NHS IT and offers a broader overview than prior studies, which tend to focus on individual elements of the evolving program. Based on this historical overview, we develop a framework to inform further explanatory research in this area. We first outline our methodology, then explain the evolution of the NHS structure. We next describe the sequence of NHS IT strategies, their principal objectives, and implementation approaches. We go on to discuss the implications of a complex and dynamic structure for delivery of an evolving national strategic program over an extended timeframe. Finally, using strong structuration theory (SST) as our lens, we synthesize key attributes of the NHS IT implementation environment into a conceptual model.

METHODOLOGY

We undertook an exploratory, retrospective longitudinal case study of the concurrent phenomena of evolving national IT strategy and dynamic structural changes in the English NHS., To chronicle the progressive developments in both areas, we reviewed strategic plans, legislation, and health policy documents, and constructed an analytical schema for each area.

We reviewed the literature on multi-organizational forms, complexity, national level health IT implementations, and mega-projects to identify factors that could be mapped to our structure and strategy schemata. As outlined in Figure 1 and guided by SST, we superimposed these factors on a simplification of the schema and created a conceptual model that integrates strategic and structural factors.

Figure 1.

Flow chart of methodology.

NHS STRUCTURE

Established in 1948 as a publicly funded free at point-of-care service, the NHS has a tiered structure with organizations at national (policy and executive), regional and local (intermediate) management, and provider levels. This section outlines the main structural changes in the NHS from the NHS Reorganisation Act (1973) to the present day. At the local level, we consider acute hospitals and general practice but have, for clarity, excluded other NHS services such as dentistry, opticians, community care, mental health care, and ambulance and paramedical services. Some other entities within the NHS (such as Public Health England and NHS Improvement) have, similarly, been omitted. The separate organizational structures responsible for formulating and delivering the national NHS IT strategies are considered in a later section.

NHS restructuring

The English NHS currently comprises some 8700 organizations, and the evolving structure over time can be seen in Figure 2, which also illustrates an increasing frequency of restructuring from the 1990s at a time when national IT strategies started to be formulated.

Figure 2.

Evolving structure of the NHS from 1973 to 2017. NHSME – NHS Management Executive. NHSE – NHS Executive. HA – Health Authority. DHSC – Department of Health and Social Care. PCGs – Primary Care Groups. Clinical CGs – Clinical Commissioning Groups.

Policy executive

In government, the Secretary of State for Health oversees a civil service Department of Health (DH). Between 1968 and 1988 and following a reorganization in early 2018, responsibility for both health and social care has been combined; at other times, these have been managed separately. The top level of the NHS has evolved from an NHS Management Board (1980s), through an NHS (Management) Executive (1990s), operating for a period through 8 regional offices, and then to a number of “arm’s length bodies” (executive agencies of the DH). NHS England, established in 2013, now incorporates these executive functions along with the regional responsibilities previously fulfilled by organizations described in the next section.

Intermediate: regional and local management

At the regional level, 14 Regional Health Authorities were established in 1973 and succeeded in 1996 by 8 regional offices of the NHS Executive and 95 Health Authorities. A further reorganization in 2002 abolished both of these and established 28 Strategic Health Authorities, later reduced to 10. In 2013, their functions were incorporated into NHS England.

Nearer the front line of healthcare, NHS organizations can be broadly categorized into secondary care (hospitals) and primary care (general practice). For secondary care, the NHS Reorganisation Act (1973) established 90 Area Health Authorities as well as 205 District Health Authorities that were later reduced to 189. The former were abolished in 1982 and the latter in 1996 after NHS Trusts had been established. For the primary care sector, the Family Practitioner Committees (FPCs) of the 1970s were replaced by Family Health Services Authorities (FHSA), then for a brief period by Primary Care Groups (PCGs), then Primary Care Trusts (PCTs) and, last, Clinical Commissioning Groups (CCGs).

Provider organizations

In 1990, NHS Trusts were established as public corporations to manage local hospitals and, more recently, some Trusts that meet defined performance criteria have achieved more financial and managerial autonomy as Foundation Trusts. General practitioners are self-employed contractors to the NHS, typically working from health centers with other GPs and healthcare professionals.

Summary of NHS structure

The NHS presents a complex and volatile structural and temporal context for national IT strategy implementation. At each level, organizations have responsibilities that include planning, resource and performance management, and stakeholder engagement. The frequent reorganizations typically disrupt senior management teams, various organizational functions, intra- and inter-organizational networks and knowledge-sharing channels, corporate memory, and reporting and governance arrangements. A new organization seldom replicates precisely the functions of any single predecessor and may be based in a new location. Evolving structures have tended to maintain a divide between primary and secondary care, against a backdrop of attempts to achieve a more integrated approach to care provision across sectors, including social care. The central management of NHS-wide IT initiatives (which will be discussed later) has been undertaken progressively by a separate set of organizations in a parallel reporting structure, adding further complexity.

HEALTH IT STRATEGY

Evolving NHS IT strategies

Pre-1992

Before the 1980s, the use of computers in the NHS was largely confined to the collection of statistical data (eg, hospital episode statistics) and management functions (eg, finance), though there had been some early attempts at computerizing medical records. As personal computers became available, enthusiasts began to explore their use in clinical care and, by the late 1980s, computing was widely established in UK General Practice (primary care or family practice) and used particularly to streamline business processes such as claims for item-of-service remuneration. Adoption of clinical computing in other sectors such as hospitals was slower, though a Hospital Information Support Systems (HISS) program was initiated in 1988. From these beginnings, the first coherent national NHS IT strategy appeared in 1992, followed by a series of advances that are summarized in Figure 3 and reviewed in the following sections.

Figure 3.

Evolving NHS IT strategies from 1992 to 2017. EHR – electronic health record.

1992–1998

The first national IT strategy, outlined in Getting Better with Information (1992), promised a patient-centered approach, with integrated operational clinical systems from which secondary management information could be readily derived. The emphasis was on using IT to support care and communications in the face of rising costs of implementing technological innovation and expectations of improved outcomes by patients and public. The overall approach aimed to balance national compliance against local choice by providing a national steer for locally led implementations. This was articulated as 4 overarching themes: undertaking national facilitating projects, developing a national infrastructure, maximizing value for money, and enabling people. The strategy was founded on 5 key principles: person-based information; integrated systems; derivation of information from operational systems; security and confidentiality safeguards; and information sharing across the NHS.

To help local organizations develop robust business cases and ensure value for money, central guidance on meeting users’ information requirements was provided. Adoption of national standards for inter-operability and quality was incentivized through reimbursement for procurement of accredited systems. The strategy also recognized the need for appropriate training in the use of IT and the challenge of behavioral change.

1999–2003

The earlier approach was re-emphasized in Information for Health, published in October 1998 as a 7-year strategy for the NHS in England. It outlined a range of measures to enable personal health information to be stored electronically and “to be communicated seamlessly to extended clinical teams across organisational boundaries, bridging health and social care.” Key components of the strategy included: lifelong electronic health records (EHRs); round-the-clock access to these for clinicians; seamless care for patients during healthcare encounters; better public access to health information; and provision of management information to optimize use of NHS resources. This was again promoted as a national strategy for local implementation.

The NHS Plan (2000) presented a new vision for a patient-centered NHS including electronic booking of appointments; electronic records accessible to both clinicians and patients; smart cards for patients; and electronic prescribing of medicines. In 2001, a further document, Building the Information Core – Implementing the NHS Plan, addressed some of the practical issues around the plan’s implementation and reaffirmed and updated the 1998 strategy.

2003–2012

Overall, delivery of the 1998 strategy proved challenging, and so Delivering 21st Century IT Support for the NHS (published in 2002) initiated an accelerated approach to implementation through a National Programme for Information Technology (NPfIT). While retaining some of the principles of the 1998 strategy, it adopted different technical approaches including a “spine” of national services for personal demographic information, a summary care record, and secondary use of anonymized data for purposes such as resource planning. It incorporated additional elements such as a Quality Measurement and Analysis Service and adopted a radically different centralized approach involving procurement of systems from a small number of large suppliers. NPfIT soon attracted criticism, however, particularly relating to a perceived lack of transparency, failure to engage adequately with clinical end users, and the risks associated with its scale and projected cost. The agency responsible for its delivery (Connecting for Health – CfH) was abolished in 2013 and residual activities transferred to its successor organization, NHS Digital.

After 2013

Following an earlier report, The Power of Information (2012), NHS Digital published its strategy Information and Technology for Better Care (2015) with objectives of data protection; shared architecture and standards; implementing national services; supporting local organizations in their use of technology, data, and information; and making better use of healthcare information. This reiterates many of the objectives of the 1992 and 1998 strategies (see Figure 3) and indicates a reversion to a more decentralized management model.

NHS-wide IT management

Organizations

The 1992 strategy was developed by the Information Management Group (IMG) of the NHS Executive, which reported to the DH. This group’s remit included the delivery of infrastructure projects (many involving standards development and testing) as well as some information services to the NHS. In 1999, as part of the implementation of the 1998 strategy, IMG was abolished and some of its units and functions incorporated into the new NHS Information Authority (NHSIA), a Special Health Authority and arm’s length body of the DH, which created an internal Information Policy Unit (IPU) to provide oversight. From 2003, functions of the NHSIA began to be directly integrated into the DH prior to its abolition in 2005 and replacement with Connecting for Health (CfH) and the Health and Social Care Information Centre (HSCIC). Subsequently, the responsibilities of these organizations have been combined into NHS Digital, a trading name of HSCIC.

Further complexity was added during the attempt to deliver NPfIT when 5 clusters were created, each covering a defined geographical area and with a single supplier (local service provider). Figure 4 illustrates the national delivery structures for NHS IT, along with strategic lead organizations and strategic focus.

Figure 4.

NHS-wide IT delivery structure. NHS ME – NHS Management Executive. IPU – Information Policy Unit. DHSC – Department of Health and Social Care. IMG – Information Management Group. NHSIA – NHS Information Authority. HSCIC – Health and Social Care Information Centre. CFH – Connecting for Health.

Implementation approaches

Seen in retrospect, the 25-year series of NHS strategic initiatives can be characterized as logical and incremental,, initially emphasising the definition and propagation of technical and data standards, next addressing issues of connectivity, and later specifying and delivering applications (Figure 4). This emergent approach allows for ongoing technological innovation, facilitates integration with evolving umbrella health strategies, and provides design and implementation flexibility to accommodate both a changing political agenda and a volatile multi-organizational environment.

From 2003, NPfIT deviated significantly from earlier approaches, particularly in shifting from decentralized socio-technical implementation efforts to a centrally imposed managerial framework of incentives and sanctions. This shift, first, assumed a degree of local professional and organizational acceptance that failed to materialize. A prescriptive (top-down) approach has been reported to reduce their receptiveness to technological and healthcare delivery developments. Second, it concentrated the risk by moving responsibility for realization of the strategy from local NHS organizations (monitored centrally) to a large-scale central program. Whereas predecessor initiatives can be regarded as strategic frameworks within which customized implementation programs (eg, for laboratory results reporting) were established, NPfIT had a strong emphasis on controlling systems delivery with considerable reshaping of earlier strategies.

NPfIT has been described as the world’s largest civilian IT project, and has been categorized as a mega-project with associated risks relating to long time-frames and multiple participant organizations with complex interfaces, ambiguous power relations, and conflicting interests and priorities. Further inter-organizational challenges include achieving consensus in planning and decision making, managing program execution, and information flows.

Some authors have offered a structural perspective on national IT in healthcare. The fragmented and decentralized system in the United States, which favors a bottom-up approach to national systems, has been contrasted with the top-down approach adopted by NPfIT, and in Canada and Australia., Based on experience in New Zealand, a “middle-out” approach to national systems has been proposed, characterized by central leadership, public–private sector collaboration, and local investment in solutions that achieve national goals. It has been suggested that the current NHS approach post-NPfIT, based on standards accreditation, a flexible socio-technical change model, and a shared learning environment, now resembles that of New Zealand. However, seen from the full 25-year perspective, rather than the limited frame of reference of NPfIT, the NHS appears to have adopted many of the middle-out processes throughout its strategic IT journey.

DISCUSSION

We next discuss program outcomes, both for general health IT and for the NHS, before characterizing the NHS structure as a complex multi-organization form. A synthesis of our analysis findings is then presented as a conceptual model.

Program outcomes

The difficulties in implementing information technology and systems across a healthcare domain continue to be reported, and even single organizations may struggle to achieve alignment between information systems and organizational objectives.

Studies from the health informatics and project management literature have used several theoretical lenses to study NHS IT projects, including boundary setting, diffusion of innovation, actor-network theory, SST, socio-technical systems, maturity models, and normalization process theory. The absence of any clear conceptual framework for other evaluations has also been noted. Various challenges have been identified from these studies including lack of engagement with frontline staff, failure to meet user needs or expectations, tensions between organizational and professional interests,,, communication issues between central bodies and local organizations,4,33 changing external environments, changing supplier relationships, familiar issues with schedules and costs in IT programs, and general problems in implementing large public sector IT.

With much attention directed at evaluating NPfIT, the conduct and outcomes of the longer-term NHS strategy have tended to be overlooked. While the UK Parliament’s Public Accounts Committee explicitly deems NPfIT a failure, some overall progress in the longer-term NHS-wide IT endeavor has been identified. One factor that has received little focused attention is the impact on overall progress of the NHS’s complex and dynamic organizational structure. In the extended 25-year context, the interplay between a progressive, incremental national strategy and a dynamic healthcare structure appears to be an under-researched area.

Our analysis has been guided by SST, which provides a 3-layered ontology of macro-level forces (government policy), micro-level practice (end-use of information systems), and an interposed meso-level of mediation. Structuration theory has a long history of application in information systems research generally and in health IT studies in particular,, including NPfIT. The theory facilitates the understanding of intra- and inter-organizational networks and of changes across time, and has been used to guide research on strategy as practice.

What is “the NHS”?

A multi-organization form

Recent literature,, has generally characterized the NHS structure as an organizational field, though it also has some properties of organizational forms described outside the healthcare environment, such as an M-form structure with a centralized parent organization and multiple semi-autonomous subsidiary divisions, and a meta-organization, where there is legal autonomy but pursuit of a common goal.

Various studies have explored multi-organizational forms within healthcare systems, and examples are illustrated in Table 1.

Table 1.

Multi-organization forms in healthcare

Form	Attributes	Healthcare examples
Field67	A number of organizations interacting to achieve common goals. The arrangement exhibits status hierarchies, networks, and shared values and identity.68	Healthcare in Alberta, Canada69
		NHS in England37,62,64
Federation70	Multiple semi-autonomous organizations that retain some legal autonomy while cooperating toward system goals and ceding some control to a central management body that provides inter-organizational coordination of activities. Within a complex environment, constituent organizations may be diverse and often dispersed geographically, though linked by physical and virtual networks.	U.S. hospitals71
		GP federations in England72
Quasi-firm73	A loosely coupled interorganizational arrangement of independent entities but with a shared and significant long-term strategic purpose. These arrangements are often established to manage interdependences resulting from technological innovation.	U.S. Healthcare sector73

The arrangement exhibits status hierarchies, networks, and shared values and identity.

A complex system

A systems approach facilitates a methodical exploration of factors relevant to implementing national programs across complex and dynamic organizational forms. Healthcare systems are both hierarchic, characterized by interrelated subsystems, and social, characterized by formal structures and elementary units. Within these systems, network relationships (configuration and effectiveness), and autonomy and control are important.

Complexity in healthcare systems has been investigated based on the degree of interrelatedness of system components, including the extent to which these are non-decomposable (or difficult to understand in isolation) and nonlinear (or exhibiting unpredictable responses to external influences). The strength of linkages between organizations has also been explored through a concept of loose coupling that focuses on relationship patterns., The properties of both horizontal and vertical relationships between institutions have been investigated in a U.S. healthcare environment, and the nature of such relationships may influence knowledge sharing as well as commitment to strategic direction from above.

Studies from sectors outside healthcare have investigated autonomy and control in complex hierarchical systems, noting horizontal tensions between component organizations, vertical tensions between organizations and the center, and tensions between central directives or advice and local autonomy. Difficulties in balancing autonomy and accountability have also been reported in the English NHS.

Synthesis

Strategy, structure, and complexity

The relationship between strategy and structure has long been a subject of interest in the management literature.,, The challenges in planning and implementing strategic change in a dynamic organizational environment have been well documented., Among these are the unpredictable consequences of reorganizing complex healthcare systems. There have been conflicting findings from empirical studies on the relationship between structural complexity and the adoption of technological innovation in organizations including in the healthcare sector, but little focused attention on the impact of the frequent restructuring seen in the NHS.

Over a 25-year evolution of NHS-wide IT initiatives, the NHS has undergone frequent restructuring affecting the constitution of component organizations and both horizontal and vertical relationships. These reorganizations have been prompted by various factors, including the inception of new public management in the NHS, heralded in 1983 by the Griffiths Report, the requirement to support purchaser and provider functions in the NHS internal market, and the progressive politicization of public healthcare and the resulting sensitivity of the NHS to changing governments, reflected in the tension between centralized and decentralized approaches, typically favored by Labour and Conservative administrations, respectively.

There is no direct evidence that restructuring has been undertaken to facilitate the implementation of IT, nor has the rollout of IT had any observable impact on restructuring decisions. In that sense, the evolving national IT strategy and service reorganizations have not been integrated processes.

The structural changes have affected political/executive, regional and local management, and provider levels, and have involved primary care and hospital sectors. However, the changes have been particularly marked in the intermediate layer of regional and local management. Within this dynamic landscape, local initiatives in health information management have attempted to integrate (or interface) with technical and data standards defined or imposed through the succession of national strategies. At the same time, lower levels of the tiered NHS hierarchy have had to respond to changing levels of centralized control of systems procurement, while also managing direct health care delivery functions including planning, resource allocation, and stakeholder engagement.

Conceptual model

The foregoing discussions have highlighted structural and strategic factors relating to NHS-wide IT programs.

Structure

The English NHS is a large, complex, and turbulent system comprising multiple organizations. Document and literature reviews have suggested that important intra-organizational factors include longevity, readjustment to new degrees of autonomy and channels of control, and internal stability of management teams and functions. Inter-organizational factors include geographical proximity, the effectiveness of network relationships between organizations and sectors, commitment to shared objectives around the national IT program, and the effectiveness of knowledge sharing. Interface factors among levels of the hierarchical NHS structure include the stability of the levels, the permeability of the interfaces in terms of upward and downward flows of information, the strength of centralized control, the extent to which strategic goals cascade effectively from policy level through the intermediate level and to provider organizations, and the effectiveness of knowledge sharing.

Strategy

The content of the national IT strategy has been logical and incremental, while the implementation approach has shifted from local to central before reverting to local. Overall, many attributes of a middle-out approach can be identified, though the NPfIT mega-project attempted to replace the earlier socio-technical implementation model with a top-down approach.

Model

In Figure 5, we consolidate as a conceptual model the principal structural and strategic (design and implementation) attributes from our analysis.

Figure 5.

Strategy and structure: vertical and horizontal relationships in the NHS environment.

CONCLUSION

Healthcare systems are complex, and the NHS structure has been remarkably volatile throughout progressive national IT strategy initiatives. We believe that the alignment between incremental strategy and dynamic structure is an under-researched area and that lessons from organizational studies may add to our understanding of important strategic management issues.

While the state-organized NHS might be considered atypical in its size, political oversight, and degree of structural instability, the general principles of a complex, networked healthcare system apply across a range of healthcare delivery systems. Whether formulated as prescriptive or emergent strategy, implemented through top-down, bottom-up, or middle-out approaches, and managed through socio-technical or more directive change processes, achievement of effective national IT systems is likely to require understanding and optimal management of the complex, and possibly dynamic, structural relationships between the heterogenous organizations involved.

Guided by SST, we have developed a conceptual model that aims to consolidate a range of factors from across the strategy–structure dyad. We suggest that there is, in this area, fertile ground for new lines of research enquiry around the impact of dynamic and complex structures on national IT systems design and implementation approaches, as well as evaluation of the role of the identified interface and intra- and interorganizational factors on achievement of strategic program outcomes.

CCG	Clinical Commissioning Group
	Established in 2013 to commission local health services from provider organizations.
CfH	Connecting for Health
	The organization responsible for delivering the National Programme for Information Technology from 2005 to 2013.
DH	Department of Health
	The government department responsible for health from 1988 to 2018.
DHSC	Department of Health and Social Care
	The government department responsible for both health and social care since 2018.
EHR	Electronic Health Record
FHSA	Family Health Services Authority
	Responsible for the administration of local primary care services 1990 to 1999.
FPC	Family Practitioner Committee
	Responsible for the administration of local primary care services 1973 to 1990.
GP	General Practitioner
	A family physician in the United Kingdom.
HA	Health Authority An administrative tier in the NHS responsible for a geographical area and including Regional (1974—1996), Area (1974—1982), District (1974–1996), and Strategic (1996–2013) levels. A special NHS management organization with national responsibilities eg, the NHSIA.
HISS	Hospital Information Support System
	A pilot project from 1988 to 1994 to investigate the benefits from integrated information systems in NHS hospitals.
HSCIC	Health and Social Care Information Centre
	Established in 2005 alongside CfH and NPfIT to manage the service elements of NHS IT. Now operates as NHS Digital.
IMG	Information Management Group
	A group reporting to the NHS (Management) Executive and responsible for NHS IT until 1999.
IPU	Information Policy Unit
	A unit within the Department of Health overseeing NHS IT from 1999 to 2005.
NHS	National Health Service
	The publicly funded free at point-of-care service established in 1948.
NHSE	NHS Executive
	Renamed senior level of NHS management in the 1990s.
NHSIA	NHS Information Authority
	The arm’s length special health authority responsible for delivering NHS IT from 1999 to 2005.
NHSME	NHS Management Executive
	The senior level of NHS management in the early 1990s.
NPfIT	National Programme for Information Technology
	The NHS strategic IT program from 2003 to 2013.
PCG	Primary Care Group
	Responsible for budget management for local general practices from 1999 to 2000.
PCT	Primary Care Trust
	Responsible for allocating local general practice funds and commissioning services from providers from 2000 to 2013.