STAT-HI: A socio-technical assessment tool for health informatics implementations.

This paper proposes a socio-technical assessment tool (STAT-HI) for health informatics implementations. We explore why even projects allegedly using sound methodologies repeatedly fail to give adequate attention to socio-technical issues, and we present an initial draft of a structured assessment tool for health informatics implementation that encapsulates socio-technical good practice. Further work is proposed to enrich and validate the proposed instrument. This proposal was presented for discussion at a meeting of the UK Faculty of Health Informatics in December 2009.

INTRODUCTION

This paper seeks to address the issue of practical application of socio-technical principles in health informatics implementations. In the UK, this issue has arisen mostly from concerns about the National Health Service (NHS) National Programme for IT (NPfIT) in England. This topic has been a focus for discussion by the UK Faculty of Health Informatics, an informal group comprising a mixture of academics and practitioners in the field.

What is the Problem?

NPfIT was established in 2002 to deliver “twenty-first century IT support” for the NHS, defined as a set of electronic services to provide lifelong health records, direct booking of outpatient appointments and transmission of prescriptions from primary care to pharmacy, all based upon a robust information architecture and secure broadband network [1].

There has been sustained and widespread criticism, not always well-informed, of the perceived failures of NPfIT and its overly centralist, secretive and monolithic approach [2, 3]. For example, the 2002 Gateway Review by the UK Office of Government Commerce (OGC) into the business justification for what was then called the Integrated Care Records Service (only recently published following requests made under the provisions of the Freedom of Information Act [4]) stated that they found “much talk of what the IT programme will achieve, but little recognition of the potential impact of this on current practices, procedures and systems, both technical and organisational” [5].

Similar themes continue to be echoed [6]. The evaluation report on the NPfIT Summary Care Record pilot implementation in North West England suggested that the project’s failure to adopt a socio-technical approach was one reason why the system was not embraced by busy clinicians [7].

Comparable problems have been reported in other major health informatics (HI) implementations [8, 9], so this issue is by no means unique to the English programme.

What is the Socio-Technical Systems View?

The roots of socio-technical thinking can be found in research studies by the Tavistock Institute in the 1950s. For example, investigations in the British coal industry found that autonomous groups of miners had re-invented surprisingly effective innovations in mining methods contrary to “obvious” techniques suggested by increasing mechanization [10]. The socio-technical approach was built upon a philosophy of improving worker satisfaction and quality of life, with the belief that this leads to improved productivity and effectiveness [11].

In this paper, the “socio-technical” perspective is understood to refer primarily to the insight that a technical system implementation inevitably affects and is affected by the interdependent social system within which and upon which it operates [12, 13]. This concept has been used to formulate design principles [14] (see Box 1) and methods [15, 16] for information systems.

This approach requires a more nuanced worldview than the positivist mindset typically associated with reductionist physical science and “hard” technology. It utilizes perspectives such as constructivist or interpretive epistemologies and a social interactionist or negotiated view of reality [17] involving complex adaptive systems [18]. This paper argues that these factors are particularly crucial in HI and introduces a tool that supports adoption of a socio-technical approach in HI.

How does this Tool Aim to Help?

The organization and practice of healthcare is one of humanity’s most complex endeavours. Clinical work has been described as mostly “unpredictable and non-routine” [19] and “highly contingent, ad-hoc and idiosyncratic” [20]. Dealing with multiple uncertainties is a challenge for clinicians [21, 22], so arguably even more so for planners, designers or implementers of HI solutions [23, 24]. In particular, the NHS is a paradox: nominally a centrally-controlled monolith but in fact a coalition of competing confederacies of autonomous interests.

Clegg’s Socio-Technical Design Principles [14]

Design is systemic.

Values and mindsets are central to design.

Design involves making choices.

Design should reflect the needs of the business, its users and their managers.

Design is an extended social process.

Design is socially shaped.

Design is contingent.

Core processes should be integrated.

Design entails multiple task allocations between and amongst humans and machines.

System components should be congruent.

Systems should be simple in design and make problems visible.

Problems should be controlled at source.

The means of undertaking tasks should be flexibly specified.

Design practice is itself a socio-technical system.

Systems and their design should be owned by their managers and users.

Evaluation is an essential aspect of design.

Design involves multidisciplinary education.

Resources and support are required for design.

System design involves political processes.

We argue that this irreducible complexity means that a socio-technical approach is vital to the success of innovations and transformations that are based on HI. The socio-technical approach would help to negotiate these complexities using principles such as minimal process specification, multi-stakeholder participation, user leadership, local adaptability and holistic evaluation [14].

Unfortunately, knowledge of socio-technical principles and their vital importance in HI does not seem to be widespread among practitioners. The socio-technical assessment tool for health informatics (STAT-HI) proposed here aims to help bridge the gap between research and practice by distilling key factors of good practice into a simple, readily accessible and applicable form.

METHODS

Checklist Format

The tool is expressed as a reflective checklist to validate the socio-technical soundness of an HI implementation approach. It does not aim to check every aspect of project fitness for purpose.

This approach seeks to utilize the ‘checklist effect’. Merely having a structured set of tasks or questions can in itself improve outcomes. This effect is so strong that it is a well-known source of bias in studies that compare interventions that use any checklist-based approach (whatever its merits) against those that do not [25]. Checklists have proved very powerful in improving aspects of clinical practice [26].

The checklist was conceived as a categorized summary of principles derived from a synthesis of critical success factors identified in the literature. It emulates the approach and certain principles of the AHRQ [27], GEP-HI [28] and STARE-HI [29] standards for designing and reporting system evaluation in health informatics.

Sources

The primary sources used were seminal UK reports on reasons for success or failure in major IT projects [30, 31] and a selection of HI studies [32-39] encountered in an ongoing literature review of health informatics theory by the present authors and colleagues [40]. This selection does not purport to be comprehensive or to form a systematic review of socio-technical principles in HI, but is presented as a starting point for discussion and development. The sources were selected on face validity as offering immediately relevant and directly applicable socio-technical assessment criteria.

The checklist emphasises particular elements of programme governance, cultural factors, the approach to changes in workflow and practice and the need for strong and balanced clinical leadership and governance. The headings echo the critical success factors identified by Whetton [17] (see Box 2) and Georgiou & Westbrook’s ten questions to consider before implementing computerized physician order entry (CPOE) [41] (see Box 3).

Whetton’s Critical Success Factors [17]

A federal approach.

Senior management support.

Project champion.

Project management.

Project team membership.

End-user input and participation.

Change management.

Communication.

End-user skill development.

PROPOSAL

Table shows the first published version of the STAT-HI checklist, comprising twenty items under the four headings of culture, governance, managing change and technology. The bracketed remarks are intended to illustrate and amplify the implied recommendation taken from the referenced source. Some checklist questions are taken verbatim from the cited source but most are paraphrases.

DISCUSSION

Suggested Application of STAT-HI

The STAT-HI checklist is presented as a tool for project/programme managers, project/programme boards and other professionals who provide external assurance and audit functions.

Georgiou & Westbrook’s Ten Key Questions for CPOE Implementation [41]

What does the organisation/department expect to gain by introducing the new system?

Who wants or needs this new technology and why?

Which groups are most involved in the decision making about implementation and use?

Will the system be technically compatible with current systems in use?

Can it be tailored to fit the specific needs of professionals?

How will the benefits of the new system be measured?

What changes to work practices and processes are required?

Are the lines of accountability for dealing with expected and unexpected problems clear?

What are the drawbacks and risks of system implementation?

Are they being addressed, and are there safeguards for dealing with problems?

It is suggested that a required reading list should be promulgated to HI practitioners alongside this checklist, comprising eight books, papers and electronic sources [30-32, 34, 37-39, 45].

The reading list supports the aim of the checklist to be reflective, rather than simply a tick-box exercise. The checklist is an opportunity to learn, to enlarge one’s worldview and to change practice. Schön [46] argued that reflective practice goes beyond problem-solving by considering problem setting: has it been ‘named and framed’ rightly, or is evidence emerging that it needs to be rethought?

Therefore we suggest that a reviewer (whatever their professional role) making an assessment using STAT-HI should give narrative justification for their considered responses to each of the questions, adducing data to substantiate their commentary as necessary to satisfy both their own ethical standard and the independent critical judgement of the relevant project/programme assurance bodies.

Is this Proposal Novel?

By definition, this checklist utilizes existing evidence and prior good practice recommendations. Its only claim to originality is the condensation and annotated checklist presentation of key guidance and the suggestion of a specific minimum knowledge base for HI practitioners.

Do Existing Methodologies Incorporate Socio-Technical Principles?

Arguably there are socio-technical principles embedded in numerous extant methods, even if not explicitly. Examples of this include the OGC programme management guidelines, Soft Systems Methodology and the ETHICS method.

OGC’s Managing Successful Programmes [45] acknowledges that change programmes involve ambiguity and the need to establish both organizational readiness for change and capability to manage and deliver change. Programmes are described as led by vision not specification. They are most effective when issues are debated freely and risks evaluated openly. The programme team needs staff with the relevant skills and experience to manage cultural and people issues involved in the business change.

Checkland’s Soft Systems Methodology is a process of enquiry that accepts and examines the multiple perceived realities of different stakeholders in a problem scenario. It envisages a group learning cycle that seeks to discover feasible improvements that can be accommodated within the continually re-negotiated social interpretations of individuals and organizations. Its philosophy directly rejects the idea that its conceptual models represent descriptions of an absolute reality, but regards “systems” as devices to think about the purposeful activities of various actors and thus formulate questions of the problem situation [47]. So if used for HI, the soft systems approach might in a sense be regarded as an application of socio-technical principles but with a more strongly constructivist interpretation of what we mean by “system”.

Mumford devised the ETHICS method (Effective Technical and Human Implementation of Computer-based Systems) as an explicitly socio-technical approach [15-16]. It emphasizes participatory diagnosis of problems, setting objectives for both efficiency and work satisfaction and ensuring compatibility of the technical and organizational systems. Mumford’s extensive socio-technical work seems to be little known outside academia, particularly since the 1980s [11].

Why are Socio-Technical Principles Not Routinely Followed in HI Projects?

Given the above, we echo Cobb’s paradox: "We know why projects fail, we know how to prevent their failure – so why do they still fail?" [30].

One factor is that health informatics and project management are not controlled professions with mandatory regulatory standards. Hence, the professional competence of some NHS HI project managers is limited to general IT project experience, which tends to bring the implicit but incorrect view that HI is "just another" industry sector. This has been another general criticism of the NPfIT “commodity” approach to HI. If anything, the unregulated nature of project management and health informatics means that their need for guidance is even greater than a formally controlled profession.

A converse aspect of the problem is information overload. There is simply so much guidance available on project and programme management in healthcare. In some respects this parallels medical practice, where for many years it has been recognized that practicing clinicians cannot be expected to keep track of every new piece of research evidence. Instead, important new knowledge is highlighted through summary publications, clinical guidelines and professional standards. In fact, the latency inherent in the medical profession is such that even solid research evidence only changes clinical practice slowly, patchily and when multiple interventions are used for reinforcement [38, 48].

By contrast, HI benefits and project management methodologies do not have a robust evidence base but are largely aspirational [42] or normative [49]. In fact there are legitimate grounds for caution with respect to HI, given the evidence that products are not equal [50], things can be made worse [51] and that the method of implementation can make all the difference to success or failure [34, 52].

At times HI is used instrumentally, beyond its functional purposes, to enforce “improvements” in ways of working as defined by the project sponsors [53]. This adds yet another layer of complexity and potential opposition, especially if the merits of the given changes are disputed.

Of course, the invidious effects of politics and commercial interests cannot be ignored in this context. The NHS is unavoidably part of the political game, given its funding and structure. A taxpayer-funded IT programme has legitimate expectations of optimum progress from both its political masters and its commercial suppliers. But when these interests seem to sacrifice good practice for the sake of apparent progress or simply meeting contractual milestones, then the taxpayers have sound reason for concern.

It must also be stressed that the anticipated benefits of planned change in healthcare systems, particularly when based on HI, are not deterministic. The inherent complexity of the socio-technical system in healthcare means that apparently minor changes can have profound unpredictable consequences [18, 37]. Indeed, it has been argued that any ICT “solution” actually adds complexity, and hence risk, by increasing the number of information and process interrelationships in the world [54].

Comparison with Organizational Readiness Guidance

NHS Connecting for Health (CFH), the English Department of Health agency responsible for NPfIT, publishes an “Organisational Readiness Assurance Guide” for HI planning [55]. This document seems to assume a linear model of change that comprises work process redesign, system deployment and establishing a new status quo. It does not address unintended consequences or complex adaptive change. Its subtitle “Are you ready?” reinforces the concept of the organization or individual as passively receptive to a “deployment”. It does suggest that new working practices and roles be subjected to an impact analysis and requires that both line managers and “stakeholders” agree to the new working practices and roles, but does not specify whether that includes the clinicians themselves. Arguably, its helpful direction on applying lessons learned from other HI implementations itself implies that the socio-technical perspective needs greater emphasis.

Limitations

The genesis and focus of this paper has been on HI in the UK. However, the essential principles discussed here would intuitively seem rather general across the global HI domain.

At best, this proposed tool can offer a selection of key messages and maxims to practitioners. This is significantly less than a formal theory of HI programme management, but is arguably an improvement on the status quo.

Obviously, this proposed tool will not remove political or contractual pressure to impose systems to meet arbitrary deadlines. However, it could provide a mechanism to create or channel balancing feedback pressure within or upon the governance structures of HI programmes where participants have concerns that good practice may be compromised by inappropriate commercial or governmental timescales.

Further Work

The suggested approach will be debated and the content and format tested with the UK Faculty for Health Informatics. If the tool seems to have face validity based on this expert consensus, the aim will be to apply it to current or recent HI projects to measure its utility. One extension may be to devise a numerically scored assessment rating. Further thought is needed about how in practice the rating would be conducted for it to be credible, and how any proposed remedial action would be validated. Further essential content for the checklist may be added from analysis of the recently published AMIA collection of case studies of failed HI projects [56].

CONCLUSIONS

There is a real need for improving socio-technical awareness among HI project teams. This tool offers a starting point to bridge the gap between research knowledge and routine practice.

Coiera described health informatics as the systems science of healthcare [57]. We argue that a socio-technical approach is fundamental to correct application of that science.

Table 1.

STAT-HI Checklist

Heading	Ref	Item	Source
Culture	1.1	Does the programme/project give sufficient attention to social factors, ethical considerations and practical workflow issues?(See principle 13 [14]).	[32-35, 37-39]
	1.2	Does the business case and programme/project plan critically evaluate the relevant supporting evidence (or its absence) from the literature? (Is the “treatment” evidence-based? [32])	[42]
	1.3	Is there an open and constructive relationship with suppliers?(Does the commercial contract make clear where functionality is constrained to a stated specification or what degree of flexibility there is for development or adaptation and how this is costed)?	[31]
	1.4	Is the default assumption that the implementation will be evaluated using STARE-HI (or a similar structured methodology) and that the evaluation will be published?(See principle 16 [14]).	[29]
	1.5	Is there an appropriate balance between standardization and respect for local autonomy?(See principle 13 [14]).	[32]
	1.6	Is there an environment in which those who feel the programme/project is starting to go wrong feel able to say so and then get a proper hearing?(See principle 2 [14]).	[30]
Governance and Risk	2.1	How clearly are the success criteria defined? How widely are they agreed?(Does the business case include funding to backfill clinicians for project governance and workflow design activities? Are the participating clinicians solely the enthusiasts, or have opponents been recruited to balance the discussions? See principles 5 and 18 in [14]).	[30, 37, 39]
	2.2	Has the programme been broken into manageable steps?(Have all stakeholders agreed that the stages are manageable? See principle 5 in [14]).	[30]
	2.3	Are there sufficient skills to deliver the full scope of the programme? (Has the business case and programme/project plan been independently peer reviewed by qualified health informatics practitioners? See principle 18 in [14]).	[30]
	2.4	Do the Senior Responsible Owner and programme/project manager have good relevant track records? (For instance, are they registered with UKCHIP [43] at level 3)?	[30]
	2.5	Does the risk management plan include assessment of how to address unforeseen consequences in workflow and emergent change following implementation?(Is the programme alert for new kinds of errors, negative emotions, disruption to well-established communication patterns? See principle 1 in [14]).	[37, 39]
	2.6	Is there a decision-making structure that will ensure strong and effective leadership of the IT-enabled business change?(Do eventual system users and operational managers own the changes? See principle 15 in [14]).	[31]
	2.7	Does the programme/project use an evolutionary approach with rapid learning cycles, recognizing that it is virtually impossible to fully understand and predict the behaviour of complex IT systems at the start of the project?	[30, 32, 37, 39]
	2.8	Beyond immediate technical success, how will wider benefits be secured?	[31]
Managing change	3.1	Does the programme have a complete understanding of its current business processes and how its stakeholders interact both with the business and between themselves and a clear understanding of what it wants the new business process to achieve?	[31, 37, 39]
	3.2	What incentives exist to drive performance? (Have clinicians agreed with the desired outcomes of the HI project, not seeing it as an end in itself that is being imposed upon them)?	[31][44]
	3.3	Does the programme have a good appreciation of the likely impact of the business process change on service levels, productivity and different stakeholders?	[31, 37, 39]
	3.4	Does the technology exist to deliver the change?(Is a feasibility study or pilot required to offer proof of concept or proof of technology)?	[31]
Technology	4.1	Does the design and the implementation plan include not just system functionality but how it will affect actual clinical workflow?(Is there sufficient attention to ‘messy’ real-world practice rather than deceptively neat abstract models? [23-24])	[33-35, 37-39]
	4.2	Will system performance be rapid and reliable enough for clinical usage?	[36-39]