Towards an integrative cognitive-socio-technical approach in health informatics: analyzing technology-induced error involving health information systems to improve patient safety.

The purpose of this paper is to argue for an integration of cognitive and socio-technical approaches to assessing the impact of health information systems. Historically, health informatics research has examined the cognitive and socio-technical aspects of health information systems separately. In this paper we argue that evaluations of health information systems should consider aspects related to cognition as well as socio-technical aspects including impact on workflow (i.e. an integrated view). Using examples from the study of technology-induced error in healthcare, we argue for the use of simulations to evaluate the cognitive-socio-technical impacts of health information technology [36]. Implications of clinical simulations and analysis of cognitive-social-technical impacts are discussed within the context of the system development life cycle to improve health information system design, implementation and evaluation.

INTRODUCTION

Health informatics researchers have historically studied either the cognitive or the socio-technical effects of health information systems (HIS) upon health professionals (e.g. physicians, nurses and other allied health professionals) and their work [1]. This is especially apparent, when one reviews the literature surrounding computerized physician order entry (CPOE) and the error facilitating aspects of technology (i.e. technology-induced errors), where there has been a tendency to focus upon either the cognitive or the socio-technical implications of working with HIS and their effects upon health professional error rates [1]. In this paper the authors argue for an integrated perspective towards studying HISs: a cognitive-socio-technical perspective. An integrated cognitive-socio-technical approach involves examining HIS from a cognitive-socio-technical perspective rather than a: (1) cognitive or (2) socio-technical perspective alone. In this paper the authors argue that an integrated cognitive-socio-technical approach represents an extension of the existing cognitive and socio-technical theoretical approaches to the study of health informatics from a more holistic perspective. Such an advance would recognize the increasingly integrated nature of cognitive and socio-technical work that is embedded in HISs as technology evolves, improves, and becomes more ubiquitous and pervasive in healthcare over time [2]. The authors will use examples from the published literature on the error facilitating aspects of CPOE to illustrate the need to integrate these two approaches rather than study either alone.

REVIEW OF THE LITERATURE

Few researchers have attempted to integrate within their studies both the cognitive and the socio-technical implications of HIS and their associated computing devices (e.g. hand held devices and wireless carts) upon health professional work – i.e. a cognitive-socio-technical approach [3]. There are several reasons for moving towards a cognitive-socio-technical theoretical approach in health informatics research. Some of the reasons arise from the emergence and development of health informatics as a discipline over the past 30 years, and the evolution of cognitive and socio-technical research in health informatics over time [4, 34]. Other reasons include the maturation of health computing and health informatics specific research approaches and methods (involving the capture of richer and more comprehensive data sets) [5, 6]. In the next section of this paper the authors will briefly review the emergence of the cognitive and socio-technical literatures in health informatics and the implications of the maturation of health informatics research approaches for a unification of these two differing theoretical views as conceptual drivers of health informatics research.

Two Theoretical Views of Health Informatics: Cognitive and Socio-Technical

Two important and dominant theoretical views that have influenced health informatics research in recent years have been the cognitive and the socio-technical perspectives. Each of these theoretical perspectives has had a significant impact upon the evolution of health informatics as a discipline - differing from computer science, management science and the health sciences alone [7].

Evolution of Cognitive Theoretical Perspectives in Health Informatics

Cognitive theoretical perspectives in health informatics first emerged in the late 1990’s. Early pioneering work in this area conducted by Kushniruk and colleagues [8] and Patel and colleagues [9] found that an electronic medical record’s interface design could profoundly influence a physician’s information processing activities (i.e. cognitive activities including decision-making and reasoning). In this work the researchers found that physicians could become “screen driven” - following the organization of the electronic medical record’s interface rather than following academically developed reasoning processes learned during medical education and developed over time as medical expertise. This work was significant as the researchers found a relationship between the design of an electronic medical record interface and physician information processing activities or cognitive work. This work spurred on a number of studies that examined the implications of interface design upon cognition.

Patel and colleagues [9] designed a series of studies that examined the effects of interface design upon the knowledge, organization and reasoning strategies of physicians when working with computer based records. Patel et al.’s [9] work was significant. Patel found that electronic medical records or computer based medical records could “shape the way in which health care workers obtain, organize and reason with knowledge” [9 , p. 569] . For example, she found that physicians who moved from using paper patient records to electronic medical records and then proceeded to work with paper patient records continued to document their findings in the same way as they had in the electronic medical record environment they had used earlier. Physicians who had used the electronic medical record placed greater (p<0.05) emphasis on a patient’s main complaints and diagnoses while using the electronic record (as compared to the signs and symptoms of disease when using a paper patient record). Furthermore, she found that even after physicians returned to using a paper patient record after using an electronic medical record the effects of the electronic medical record continued to influence physician cognitive work (i.e. there were carry-over effects associated with electronic medical record use). In another study, Patel and colleagues [9] video recorded and analyzed doctor-patient-computer interactions. Analyses of data from these studies revealed physician interaction with the electronic medical record during doctor-patient interviews was influenced by the electronic medical record’s organization of information (i.e. physician-patient interviews followed the order of the text fields in the electronic medical record).

Later work, by other researchers continued to demonstrate the effects of HIS upon health professionals’ cognitive work. For example, Zhang et al. [10] developed a cognitive taxonomy for medical errors involving HIS. In a later work Beuscart-Zephir et al. [29] analyzed physician and nurse medication ordering processes using a cognitive approach. Today this work continues with researchers examining the effects of electronic health records upon other aspects of cognitive work or information processing activities such as health professionals information needs and information seeking behaviours involving this technology (two other activities that are a part of information processing or cognitive work) [11] and the relationship between cognitive work, medical errors, decision support systems, and physician order entry or prescribing systems [12-14].

Evolution of Sociotechnical Perspectives in Health Informatics

At the same time as cognitive researchers began to explore the effects of electronic medical records upon cognitive work in health professionals, socio-technical researchers started to investigate the implications of electronic medical records upon physicians and nurses activities in hospital settings [15, 16]. In the late 1990’s sociotechnical perspectives of HIS began to emerge in the health informatics literature [16]. Globally, HIS systems were being implemented. There were a number of examples of systems that were successfully implemented (e.g. Regenstrief Institute in Indiana, Intermountain Healthcare in Utah) [17]. At the same time, there emerged in the literature a new trend - HIS system implementation failures. HIS were not being used as technology designers had intended them to be used [16]. Some HIS were boycotted by health professionals [18]. In other cases, health professionals were only using a limited set of HIS system features and functions. Some publications also revealed health professionals were bypassing or ignoring HIS features and functions (e.g. alerts and reminders) in an effort to complete or conduct healthcare work [19]. Researchers and health care organizations began to review these occurrences of technology failure.

Further research revealed that suboptimal use of HIS by health professionals led to no or little improvement in traditional health professional work processes and patient outcomes [20, 21]. The trend continued to grow - health professionals were using HISs in ways that were not intended by the technology’s designers nor by the organization’s that procured, customized and implemented these systems [22]. Berg’s [16] seminal work and the publications of other socio-technical researchers suggested the origins of these implementation failures were socio-technical in nature. Poor fit between the health professional, the HIS and the organization where work was being performed (i.e. socio-technical system) led to adoption and appropriation failures involving the technology (i.e. unintended consequences) [16, 21, 23-26]. Socio-technical researchers found that systems lead to changes in organizational structures (e.g. management structures, power structures), organizational environments (e.g. increasing the number of organizational environments that health professionals must deal with) and altered the relationships between patients, health professionals and health professional teams who worked within these healthcare organizations [1, 27, 35].

Such changes have had a significant impact on the quality of health professionals’ work [16, 1]. Since the emergence of research examining the socio-technical aspects of HIS in the late 1990’s, there have been many researchers who have added to this body of knowledge. For example, Chaudhry’s [20] meta-analysis examining the effects of HIS upon the quality and safety of healthcare demonstrated that socio-technical fit may be key to achieving the health care outcomes that were initially promised by HIS systems designers. Chaudhry’s [20] findings suggest that many of the improvements in the quality of patient care that are associated with HISs have been documented for “home grown” systems (i.e. systems that were designed and developed by an organization for its own use). These systems were noted to have high socio-technical fit with the organization that designed them (i.e. the system was able to seamlessly support work). Chaudhry [20] notes these same findings were absent for commercially developed systems. Systems developed by vendors (i.e. commercial systems) are often developed in a model organization, purchased by another organization, customized to the local environment and then implemented [22, 37]. In such cases the purchasing organization adopted the processes and practices of the organization on which the design of the technology was modeled [37]. Therefore, fit may not have been ideal even after customization of the HIS to the new health care organization [27, 28]. Such poor fit may have led to implementation failures as exemplified by the bypassing of system functions, failure to use system functions, boycott of systems and systems being used in unintended ways [23,28]. In addition to this, poor fit may have led to changes in the way care was delivered or to no or few improvements in the quality of patient care. For example, research has documented the impact of systems such as CPOE upon the collaboration of physicians and nurses [29]. Beuscart-Zephir and colleagues [29] work illustrates this best. In her study she documents that the implementation of CPOE influenced the quality of the communication between physicians and nurses and may have impacted upon patient care. Other research, particularly research that has investigated the intended and unintended consequences arising from the use of CPOE continues to be published and continues to examine the sociotechnical intended and unintended implications of HIS upon health professional work [23, 29].

Integrating Cognitive and Socio-Technical Perspectives

Although research has clearly substantiated the need to evaluate the effects of HISs upon cognitive and socio-technical aspects of work, few studies have examined the integration of these two approaches. Increasingly, research is demonstrating that HISs can influence both the cognitive and socio-technical aspects of work concurrently. In addition to this with the move from traditional workstations to the use of ubiquitous computing devices (i.e. hand-held devices and wireless carts) there is preliminary research that suggests that software in conjunction with computing devices may also impact socio-technical aspects of work - how health professionals think, work, and interact with each other and patients in a more global health care context (from hospital to home) [2]. There is a need for further research to identify those instances of poor cognitive, socio-technical or cognitive-socio-technical fit that do not lead to health care improvements. Nohr and Boye [38] and Boye [2] illustrate this in their more recent works. The researchers identify that with the introduction of ubiquitous computing devices to health care, the everyday life of patients will be affected and decision support systems would need to be available at a patient’s fingertips (in the form of a ubiquitous computing device that would enable the patient to contact a physician or a nurse in real-time and receive advice anywhere and anyplace, from a hospital bed or in their home).

The complexities associated with this type of evaluation of a technology are significant as researchers will need to not only look at cognitive but socio-technical impacts of systems and tease out those that lead to intended and unintended consequences or technology implementation successes and failures. Such work (i.e. research involving constellations of HIS and computing devices) will determine when the HISs/device may facilitate medical errors (unintended consequences) or lead to improved patient health outcomes (intended consequences). For a researcher to be able to diagnose the causes of medical errors, there is a need for theoretical pluralism or an understanding of how the HIS and device influence cognitive and socio-technical aspects of work (i.e. cognitive-socio-technical fit) using an integrated view. To illustrate we provide an example involving a prescription writing system (i.e. CPOE) and a ubiquitous computing device in the next section of this paper. We also discuss the interaction of the two literatures in our discussion.

PHYSICIAN ORDER ENTRY AND TECHNOLOGYINDUCED ERROR

An Example of a Cognitive Approach to the Analysis of Technology-Induced Error in Order Entry

Kushniruk and colleagues [5, 30] have conducted a series of studies examining the cognitive impact of the introduction of a handheld prescription writing application (i.e. CPOE system) upon aspects of physician work. This application allowed for mobile entry of prescriptions and contained databases of information about thousands of drugs that can inform physicians of possible drug interactions. In this work a “think aloud” approach was adopted, as described in Kushniruk and Patel [5] where a fine-grained analysis of subjects’ (i.e. physicians) cognitive processing while entering prescriptions was undertaken. Screens from the device the physicians used (i.e. a Palm pilot) were video recorded along with all their verbalizations of their “thinking aloud”. Using a coding scheme that was designed to identify both medication errors (e.g. transcription errors, inappropriate dosage entered etc.) as well as usability problems [see 6] , the researchers were able to determine statistical relationships between particular usability problems and the occurrence of medication error (errors being classified as either cognitive “slips” if they were caught by the physician, or “mistakes” if they were entered into the database of a patient’s medications). The results of this study indicated that all medication errors were associated with at least one usability problem (e.g. display visibility problems, scrolling issues etc.) and that there was a strong statistical association between specific types of usability problems and medication error. The researchers application of a fine-grained analysis allowed for a tightly controlled experimental study of the conditions under which physicians might make errors that were “induced” or “facilitated” by the use of new CPOE systems (e.g. under certain circumstances physicians would not scroll through a list of dosages provided as they erroneously thought the top listed dosage was recommended). Think aloud data was coded illuminating what the subject was thinking while performing tasks. However, the context of use under which errors occurred in real settings (e.g. under what social circumstances) was not part of the study. Understanding the social aspects under which cognitive errors occur (e.g. the socio-technical aspects of use of the application) would be essential to complete our understanding of technology-induced error. Along these lines Kushniruk et al. [14] extended this work to include the study of physicians entering prescriptions while interviewing a “simulated patient” (i.e. a research collaborator playing the role of a patient), however further complementary work in naturalistic social settings is necessary to evaluate socio-technical fit.

An Example of a Socio-Technical Approach to the Analysis of Technology-Induced Error in Order Entry

Work by Koppel et al. [31] has also indicated that aspects of the design of user interfaces in CPOE can potentially facilitate medication error. Unlike the work by Kushniruk et al. [14], which focused on a fine grained analysis of cognitive aspects of human information processing (applying the “think aloud method” under simulated conditions with the objective of providing specific feedback for system refinements), Koppel et al. [31] conducted retrospective interviews and focus groups with health professionals after the introduction of a CPOE system as well as conducting observational analyses of health professionals using systems in-situ [31]. Thus this work considered the use of the system in the context of a complex organizational setting (a large hospital) and as such was able to generate recommendations regarding improving the integration of the system at the level of workflow. However, the methodological approach was not able to examine fine-grained aspects of health professionals’ cognitive processes as they interacted with the system to make medication decisions. Koppel’s work [31] has been highly influential and has informed health informatics by alerting us to potential unintended consequences at the level of system use in a large hospital setting by providing a rich real-world description of context to observe CPOE’s impact. However, a limitation of the approach is that it does not provide information about the cognitive processes that form the basis for how health professionals reason and make decisions at point of care while using CPOE. To the authors, this highlights the need for integration of theoretical frameworks that focus on the cognitive aspects of system use (such as that employed by Kushniruk, Patel and colleagues), with approaches that focus on analysis of systems in real-world organizational and socio-technical contexts (such as the work of Ash [23], Beuscart-Zephir [29], Berg [16], Koppel [31], and others).

Integrating the Cognitive and Socio-Technical Perspectives Through the Use of Clinical Simulations

In our most recent work we have developed clinical simulations (which involve realistic complex scenarios) that allow us to examine both the cognitive and socio-technical implications of the introduction of an HIS such as CPOE and medication administration systems [27, 36] within the same study. In this line of work, we straddle the distinction between laboratory-based artificial studies and naturalistic real-world studies [13, 28]. To do this we conduct high-fidelity simulations that take place in-situ in the organizations where the technology to be studied has or will be implemented. In one study that took place in a large hospital in Japan we conducted simulation studies of users of a new medication administration system that was to be implemented [3, 32, 36]. In doing this work we presented subjects (nurses and physicians) with realistic cases to respond to within a real organizational context (i.e. a patient room with all associated and integrated technologies in the hospital that the system is to be implemented in). Sixteen physicians and nurses were video and audio recorded as they interacted with a patient, the medication administration system, bar coding technology and other HISs. In addition, immediately after completing the medication administration tasks, subjects were probed about their thoughts in carrying out the tasks (using cued recall, where the experimenters replayed the video of their interactions and subjects were asked to comment on them). In this study the cued recall data was coded and provided insight into subjects’ thoughts. In addition video data of the subjects interacting with the HIS, device and patient was captured to observe the effects of the device and HIS upon health professional-patient workflows during medication administration [27]. It was found that the introduction of the system would dramatically change the nature of the health professional’s workflow and interaction with the healthcare professionals (making the workflow rigid and highly sequential) and that under situations involving emergency and stress the new system could lead to predictable error. This information was fed back to the implementation team and led to such modifications as inclusion of an emergency override, which would allow for more natural interactions among health professionals during emergency situations. As noted above, post task interviews were conducted upon completion of the medication administration to probe the health professionals about key decision making and reasoning steps in the process of medication administration to provide a cognitive perspective (i.e. allowing for analysis at the cognitive level). In addition, in the same study, detailed analysis of the workflow and interactions among health professionals and the patient were also carried out (allowing for analysis at the socio-technical level). Thus, the approach blurred the distinction between pure laboratory based study of cognitive aspects and naturalistic study of socio-technical issues. It essentially integrated cognitive and socio-technical approaches and perspectives to arrive at a practical analysis of the impact of the system that included both analysis at the cognitive level of individual interaction with the system under study as well as detailed analysis of the impact of the HIS on health professional workflow in real as well as realistic contexts of complex organizational use.

DISCUSSION – TOWARDS A COGNITIVE-SOCIOTECHNICAL PERSPECTIVE

For the most part, research that has focused upon the error facilitating aspects of technology has examined the role of HIS from either a cognitive or a socio-technical perspective. The researchers suggest the integration of these approaches would allow for a more comprehensive and integrated view of how HIS’s affect health professional work in organizational contexts and would account for interactions between cognitive and socio-technical work.

The authors have found that in order to really understand the impact of systems, the cognitive and socio-technical aspects of work cannot be treated separately. They are linked and are becoming increasingly more integrated as both HIS and device technologies (especially ubiquitous computing devices) are being used to replace physical work and are becoming increasingly integrated with cognitive work and embedded in the workflow of health care social systems (i.e. organizations such as hospitals, clinics, home care agencies, long term care facilities). We will now discuss an emergent model of cognitive-socio-technical fit that can be used to shed light on CPOE and its ability to facilitate medical errors (i.e. technology-induced errors) [14, 31] and to support this new theoretical view on health professional work in health informatics.

Fig. () illustrates our framework that considers user interaction with complex health information systems (such as CPOE and electronic health record systems) at multiple levels. At Level 1 in the figure is the level of the individual user interacting with a health information system – i.e. the “cognitive” level. At this level, the focus is on cognitive aspects (i.e. information seeking, individual reasoning and decision making) of the interaction of users (e.g. health professionals, patients and others) with the information technology. At this level human issues include the critical need for developing health information systems that are more useful and “usable” – i.e. systems that are efficient, effective and enjoyable to use [33]. The right hand side of Fig. () (at the bottom) indicates methods we have employed (and which were described in previous sections above) to examine Level 1 interactions. Study approaches include clinical simulations along with use of think aloud protocols and cued recall to assess the cognitive level of users’ interactions with health information systems.

At the second level (Level 2) depicted in Fig. () we begin to consider socio-technical issues in using and adopting healthcare information technology in terms of how such systems fit within the context of basic units of complex healthcare work activities and workflow. At this level systems can be considered in the context of how well they support complex human work activities, such as support of patient care by nurses or diagnosis of a patient’s illness by a physician. One of the major criticisms of current health information systems is a lack of appropriate integration of such technology within the routine work practices, decision making and reasoning processes of the users they are designed to support [6]. Further work in understanding the complex interaction among humans, computers and collaborative healthcare work activity are explored at this level. As can be seen from the figure, methods associated with examination of Level 2 interactions include clinical simulations (including video recording) and observational studies of real clinical interactions in naturalistic settings (extending simulation based recordings to real-world settings).

The third level (Level 3) in Fig. () depicts the organizational and social layer when considering healthcare information technologies and their use. At this level, application of knowledge from areas such as social psychology, organizational psychology and management science, as well as other related disciplines, are brought to bear on improving health information systems. This includes providing improved knowledge about the social context of use of such technology in complex social and organizational settings, identifying and explaining how healthcare professionals can work together to best support collaborative practice, as well as providing frameworks for modeling and evaluating the impact of health information technology. Such knowledge can be used to improve the design of health information systems by identifying trouble spots in organizational processes and structures, providing models for improved social and organizational structures and processes, and for supplying methods for improved design and evaluation of technologies to be deployed into organizational contexts.

In our work we have employed multiple approaches to the analysis of each of the above three levels of human interaction with health information systems. In addition, we have within our most recent work employed an approach where we have designed complementary targeted sub-studies that address two or more of the three levels indicated in Fig. (), modified and extended from Eason’s [27] levels of human-computer interaction). This has involved initial work starting with understanding the relationship between the technical, ergonomic, cognitive and social issues of implementing health information systems. For example, we are currently participating in a long-term evaluations of several major electronic health record systems implementations. Specifically we are conducting integrated examinations of cognitive, social and technical factors affecting adoption of each system. The research has initially involved study of individual users of systems by conducting individual usability testing sessions with users having recently undergone training in use of systems. This was extended to include in-depth analysis of the training processes and through complementary interview data we are also examining which factors are affecting adoption that cannot be explained from the initial cognitive analysis (conducted at Level 1 of Fig. (), which focused on collecting usability data on individual physicians interacting with the system in isolation to carry out data entry tasks). Our work with the system has indicated that there are issues at the level of user interaction of the system (i.e. physician use) while interviewing patients that need to be resolved (corresponding to Level 2 in Fig. () – user interacting with system and the environment to do basic work task) and consequently we are also conducting complementary clinical simulations to assess the impact of the system at this level. We will also be including naturalistic observation of use of the system in the clinical context at Level 3 (i.e. the level of multiple users interacting with each other to carry out multiple tasks as part of the organization). This approach involves integration of study data that sheds light on cognitive aspects of system use, integrated with data that can inform us of the impact of information systems at the group, social and organizational levels.

CONCLUSION

The purpose of this paper was to argue for an integration of cognitive and socio-technical approaches to assessing the impact of health information systems. Additionally, we have argued for the use of clinical simulations to evaluate the cognitive-socio-technical impacts of health information technology. Such evaluation can be conducted at various key points along the system development life cycle – from system prototyping, design, implementation, and customization (of commercial vendor products). In addition, an integrated ‘holistic’ approach where multiple methodological approaches are applied in order to achieve differing but complementary knowledge about user interactions is recommended. More specifically, the paper described how data gathered during clinical simulations (e.g. video, audio and documentation) can be analyzed to evaluate:

Cognitive aspects of work (i.e. information processing)

Social aspects of work (i.e. communication and coordination among health professionals and patients using technology)

Technical aspects (e.g. issues related to usability and workflow embedded in the technology and arising from the technology)

A growing body of work has appeared in the health informatics literature about both the cognitive aspects of user interactions with health information systems, and at a higher level the socio-technical aspects of these systems. Limitations of the approach include the need for development of new models and frameworks to integrate and guide studies involving cognitive and socio-technical aspects of health information systems. Also an integrated approach underlies the need for conducting more complex studies that include not only integration of multiple methods but also design of studies that address multiple levels, making this work challenging. Despite these issues, in this paper we have argued that the time is currently right for development of integrated theoretical and methodological approaches to better understand the impact of health information systems from a cognitive-socio-technical perspective.