A Sociotechnical Approach to Evaluating the Impact of ICT on Clinical Care Environments.

INTRODUCTION: Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care. However, clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety. Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood.
DESIGN: This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.
METHOD: Departmental culture, including clinical work processes and communication patterns will be thoroughly explored before system implementation using both quantitative and qualitative research methods. Findings will be compared with post implementation data, which will incorporate measurement of safety and workflow efficiency indicators. DISCUSSION: Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. However, sociotechnical and multimethod approaches to evaluation do not exist without criticism. Inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.

INTRODUCTION

Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care [1]. However, despite large-scale investments in Information Communication Technology (ICT), uptake has been slow [1, 2] whilst recent evidence of ICT-induced errors has called for a more critical examination of the impact of such technology in healthcare [3-6].

Grounded in both social theory and Information Technology (IT), sociotechnical approaches to ICT evaluation focuses on the interrelation between technology and its social environment [7]. Over time, ICT implementation models have developed an emphasis on the alignment between the clinical context where clinical work is carried out and the technology designed to improve the delivery of care [8-10], as well as the contextual differences both between organisations and within – at departmental and individual levels [11]. This paper will propose a methodology for the sociotechnical evaluation of the impact of a computerised information system on a clinical care department.

WHY SOCIOTECHNICAL THEORY?

Contemporary health organisations are complex, composed of deeply interdependent and interrelated social and technical elements where changes to one aspect will affect the other [12]; the introduction of any new technology into a setting will entail innovation in clinical roles, work processes, and culture change [13-15] whilst the attitudes and utility of technology is simultaneously socially shaped [13]. Rather than viewing computer systems as isolated entities within their intended clinical environment, the contemporary perspective portrays systems as an active component of the clinical team, constantly cooperating and dynamically interacting with the clinical staff and organisational routines [12, 16].

The sociotechnical approach is one which seeks to identify the dynamics between technology and the social, professional, and cultural environment in which it is used [17]. The limitations of a purely technocentric approach to systems evaluation became apparent when early implementation projects were less than successful and purported benefits of ICT rarely realised [17-19]. Causes of implementation failure extend beyond poor system design to the erroneous perceptions and theories about how medical work is conducted [12, 20, 21]. Clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety [3, 4, 15, 18]. Sociotechnical theory provides a paradigm against which workflow and perfusion of ICT in healthcare could be better explored and understood.

DESIGN

Any form of ICT evaluation in health may be distinguished as either summative or formative. Summative evaluations determine the overall quality or value of an intervention and have traditionally followed the objectivist view which dictates that the merits and worth of ICT should and could be quantified [18, 22]. In healthcare, such evaluations of ICT are conducted post hoc to account for the impacts of the system such as financial benefits or system effectiveness in terms of clinical outcomes [18]. Dominating the literature on health ICT evaluation, ICT is often conceptualised as the “intervention” in summative studies which allude, as in medicine, to the Randomised Controlled Trial (RCT) model of evaluation as the “gold standard” of evaluation [18, 23].

An issue with ICT evaluation in health is the complexity of the evaluation object [24]. The deep integration of ICT with the culture and work practices of the healthcare organisation requires an evaluative focus which expands beyond only software and hardware to the surrounding clinical situation and associated processes to capture the diverse and diffuse nature of system effects in the healthcare setting [24]. Whilst RCT-type studies are valuable for determining the cause and impact of a pre-specified effect, it assumes possibility of isolating system functioning from the social processes that surround it, whilst reality suggests that the design, implementation, and operation of ICT are deeply influenced by organisational and social factors [23].

Formative evaluations are conducted during the development of a product for the purpose of system improvement [22]. The increase in qualitative studies of health ICT has exhibited a shift towards the subjectivist theoretical orientation to systems evaluation which asserts the responsibility of social and organisational aspects in shaping the outcomes of objective measurement and endorsing a “multimethod” approach for both summative and formative evaluations [18]. The mixed or multimethod approach to health ICT evaluation is one which views qualitative and quantitative research methods as complementary [23, 25, 26]; the combination of methods allows exploration of the what, why, and how of a social phenomenon that qualitative methods can address, and the size, extent or duration (how much) of certain phenomena that quantitative methods establish [26]. Pleiotropic healthcare interventions such as ICT create a hybrid sociotechnical situation requiring a range of integrated multi-perspective research methods to achieve a holistic understanding [23, 27, 28].

ICT is also a dynamic and emerging process, continually evolving across the multiple phases of the implementation process [11, 23]. A longitudinal study design which follows the target of study over a substantial period of time will allow appreciation of this aspect by providing a means of monitoring changes caused by the system within the context it operates [22, 24].

This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.

To explore the context of the clinical department.

METHOD

Pre System Implementation

Process

Implicit in the sociotechnical approach to systems evaluation is consideration for the context in which the system is implemented [29]. Differing socio-organisational settings influence ICT adoption processes and system impact [10, 11, 30-33]. Departmental culture, including roles, clinical work processes and communication patterns will be thoroughly explored before system implementation to identify focus areas and facilitate interpretation of findings post implementation. Focus groups will be held pre system implementation with a purposive sample of participants across all clinical staff groups to obtain a broad representative cross-section. Discussion will concentrate on how work is currently conducted, issues of concern and expectations of the electronic system to gain a general understanding of the nature and needs of the department. Individual roles as well as the perceived roles and contributions of members of other staff groups will be explored and triangulated to establish a sense of departmental culture and dynamics. Significant points or themes arising from analysis of transcripts will be further explored during in-depth interviews with relevant participants or key informants identified from initial focus groups.

Whilst interview and focus groups examine how a situation is perceived, they are limited in their ability to understand and explain behaviour [34]. Observations of actual work processes in real situations divulge more about individual work, successes, failures, and preferences than consultation with participants alone [34]. Emerging themes from interviews and focus groups will also be explored through sufficient hours of non participant observation. This will be supplemented with video recordings which allow reviewing of specific work processes/behaviour multiple times, careful transcription and re-transcription of events to reveal and analyse “fine-grained, ‘seen but unnoticed’” aspects of conduct otherwise unidentified from the grosser examples of work phenomena obtained from plain observation alone [35]. Non-participant observations and video recordings will follow certain individuals or teams during routine work processes with focus on specific items raised during interviews and focus groups.

Whilst qualitative research methods such as interviews, focus groups, and observations yield context-dependent data from the perspective of the participants within their organisational or departmental context, quantitative means for measuring social and departmental characteristics also exist, and may be used in conjunction with qualitative measures to strengthen robustness of data through triangulation [18]. Social network analysis examines communication patterns and interactions between professional groups to elicit the patterns of connections that occur between staff [36]. This may be further quantified using the Team Climate Inventory (TCI) which measures team cohesiveness and support for work innovation which serves to predict the likelihood of departmental acceptance and successful adoption of ICT [37]. For the purposes of this study the entire department should be conceptualised as the team.

To determine the impact of the clinical information system on staff functioning and departmental dynamics

To quantify the impact of the clinical information system on patient safety, departmental productivity and efficiency indicators

Post System Implementation

Evaluation will maintain focus on the impact of ICT on departmental culture, work practices and communication post implementation. Focus groups, interviews, and observations conducted pre system implementation will be repeated post implementation to identify changes to the previously defined culture, roles, communication patterns and work processes. Questions will be informed by existing literature on the impact of information systems on culture and work in conjunction with pre implementation findings, including specific areas of concern identified from pre implementation study unique to the department. Social network surveys will be redistributed at this stage to identify changes in communication patterns and individual roles in the exchange of information in the network.

A central argument for the implementation of computerised patient information systems is their potential for improving patient safety and work practice efficiency [1]. Key performance indicators for safety and productivity (e.g. medication errors [25], diagnostic turnaround times [38], and patient waiting times [39]) exist from evaluation literature in the field, and will be selected and measured in conjunction with context-dependent issues or indicators identified from analysis of qualitative data both pre and post system implementation. To elucidate impacts of the computerised system, data from both before and after implementation of the system will be elicited from relevant administration databases as well as paper documentation. Quantitative measurements of indicator data supplement qualitative findings, adding rigour and validation whilst qualitative data place quantitative findings in an appropriate social setting that allows interpretation and a broader understanding of their implications [40].

ICT implementation is multi-staged [11, 23]. Data collection will be carried out immediately post implementation and continue at appropriate intervals across the various stages throughout the implementation process to monitor impacts and inform system customisation and development to improve system integration into existing work processes and culture. Findings and recommendations should be considered and formulated in light of influence from complex external factors such as resource limitations, government policy, influence from different stakeholders, the economic environment, and features of the health system and the information technology industry [11, 18].

Quantification of the effects of a clinical information system on a range of safety, productivity, and efficiency indicators

Identification of the underlying mechanisms which determine optimal systems functioning within the clinical department

DISCUSSION

The sociotechnical approach to ICT evaluation appreciates the highly complex healthcare setting and emphasises the relationship between the social and technical impacts of clinical information systems. ICT represent technical artefacts which embody the implicit theories about how work is conducted; success of the artefact in real clinical situations relies upon the accuracy and applicability of these theories [41]. ICT evaluation tools [42, 43] exist to guide the process of planning and execution of evaluation projects, with which the proposed protocol may be further developed and fine-tuned to accommodate individual settings; the methodology outlined in the paper supplements existing work in its offering of a unique, practical evaluation procedure which combines the theory and practice of the sociotechnical perspective to attain a more holistic understanding of the impacts of ICT and to ultimately achieve successful implementation.

However, sociotechnical theory does not exist without criticism. Examination of both the social and technical systems within a clinical department seeks to determine the different contextual factors operating in separate settings in order to identify the latent mechanisms that can influence outcomes [18]. Berg [44] cautions that understanding the interdependency of the social and technical systems within a particular context is insufficient for reconciling the social and technical aspects within that environment. Similarly, multi-method approaches to evaluation could potentially produce contradictory findings which do not, in fact, facilitate mutual interpretation of results produced from the separate methods [28]. The proposed methodology risks inheriting such limitations; efficacy of the process remains to be verified through independent field testing.

CONCLUSION

Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. Incorporating both interactionist and ethnomethodological methods, this paper has outlined a formative, multi-method, longitudinal model for ICT evaluation which aims to inform interpretation of objective system effects in light of how the IT is conceptualised and viewed by users, as well as its impact on staff work practices within the unique context of the clinical department. However, inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.