Addressing the sociotechnical drivers of quality improvement: a case study of post-operative DVT prophylaxis computerised decision support.

BACKGROUND: Quality improvement (QI) initiatives characterised by iterative cycles of quantitative data analysis do not readily explain the organisational determinants of change. However, the integration of sociotechnical theory can inform more effective strategies. Our specific aims were to (1) describe a computerised decision support intervention intended to improve adherence with deep venous thrombosis (DVT) prophylaxis recommendations; and (2) show how sociotechnical theory expressed in 'Fit between Individuals, Task and Technology' framework (FITT) can identify and clarify the facilitators and barriers to QI work.
METHODS: A multidisciplinary team developed and implemented electronic menus with DVT prophylaxis recommendations. Stakeholders were interviewed and human factors were analysed to optimise integration. Menu exposure, order placement and clinical performance were measured. Vista tool extraction and chart review were used. Performance compliance pre-implementation was 77%.
RESULTS: There were 80-110 eligible cases per month. Initial menu use rate was 20%. After barriers were classified and addressed using the FITT framework, use improved 50% to 90%. Tasks, users and technology issues in the FITT model and their interfaces were identified and addressed. Workflow styles, concerns about validity of guidelines, cycle times and perceived ambiguity of risk were issues identified.
CONCLUSIONS: DVT prophylaxis in a surgical setting is fraught with socio-political agendas, cognitive dissonance and misaligned expectations. These must be sought and articulated if organisations are to respond to internal resistance to change. This case study demonstrates that QI teams using information technology must understand the clinical context, even in mature electronic health record environments, in order to implement sustainable systems.

Background

In modern healthcare's knowledge-intensive environment, information technology (IT) is frequently incorporated into quality improvement (QI) initiatives to narrow provider knowledge gaps and influence the trajectory of behaviour.1 It follows then that implementation specialists introducing new technology and computerised decision support (CDS) tend to use quality improvement methods popularised by Deming (ie, the Plan-Do-Study-Act (PDSA) cycle) and Donabedian (ie, Structure-Process-Outcome (SPO) framework).2–4 These models improve the rigour of an implementation by emphasising mechanisms of cause and effect and expressing performance outcomes quantitatively. However, QI initiatives are by their very nature, multi-dimensional and context dependent; the impact of technology on performance is often a function of the organisational culture as much as the technology or the systems-of-care.5 Unfortunately, most QI models incompletely capture complex, heterogeneous and culturally textured healthcare systems activities.6 7 In contrast, sociotechnical approaches drawn from the behavioural sciences explore the organisational and cultural interdependencies that influence technology adoption by incorporating facets of informatics, cognitive theory and organisational behaviour.6 8–14 Notably, Ammenwerth and colleagues have developed the ‘Fit between Individuals, Task and Technology’ framework (FITT), which expresses system change through the match between user group attributes, characteristics of the implemented technology, and affected tasks (figure 1).9 This framework may be used as a heuristic to inform more effective and durable quality interventions.15

Figure 1

The ‘Fit between Individuals, Task and Technology’ framework adapted from Ammenwerth et al.9 Success of quality improvement interventions is dependent upon the quality of fit between user, task and technology. The interfaces between each actor in the global activity system derive from several epistemological domains including cognitive sciences, social sciences and informatics.

The purpose of this manuscript is to show how qualitative assessment of organisational context can complement a QI implementation using information technology. Our specific aims are to (1) describe a CDS intervention intended to improve adherence with deep venous thrombosis (DVT) prophylaxis recommendations; (2) show how sociotechnical theory expressed in the FITT framework can help identify the facilitators and the barriers to adoption of new systems and (3) demonstrate how qualitative data categorised using the FITT model can complement the quantitative analysis characteristic of the PDSA model.

Problem statement

Venous thromboembolism is a rare but devastating post-operative complication affecting surgical patients and represents the most common preventable cause of in-hospital death.16 17 It is estimated that unprophylaxed patients undergoing major surgical procedures are at a 15–60% risk of developing a DVT.17 18 Several large randomised controlled trials have shown that pharmacologic prophylaxis with the routine administration of low-dose anticoagulants or lower extremity automated compression devices can significantly reduce DVT risk and prevent fatal pulmonary embolism (PE).16–19 For this reason, published guidelines endorse routine DVT prophylaxis.17 20 21 Likewise, the Veterans Affairs (VA) National Office of Quality and Performance has established performance standards requiring prophylaxis of most post-operative patients.22

Our VA facility had struggled to meet the national VA DVT prophylaxis performance target (92% of candidate patients) since establishment as a quality measure in 2007. Nationally conducted chart reviews showed that patients either did not receive prophylaxis or received non-standard prophylaxis 75% of the time. Consequently, hospital executive leadership and the surgical department jointly sponsored the development and implementation of health record CDS and an educational campaign to encourage surgeon compliance with DVT prophylaxis recommendations.

Methods

Setting and context

The study site is a 230-bed tertiary care teaching hospital serving United States veteran patients throughout the states of Oregon, Washington, Alaska and Idaho. The facility hospital and clinics use the VA computerised patient record system (CPRS), a HIMSS Stage 7 electronic record that supports computerised provider order entry.23 All patient data are stored in the VA health record database, the Veterans Integrated Systems Technology Architecture (VistA).

The Operative Care Division (OCD) is the administrative umbrella for university appointed staff in anaesthesia service, general surgery, and all surgical subspecialities. Residents rotate through OCD every 30–90 days and write the majority of electronic orders. Surgical quality is quantitatively assessed using a subset of nationally defined indicators. Facility, regional and national executives use the External Peer Review Program (EPRP) measures to compare facilities, grade performance and inform quality improvement initiatives.22

First improvement cycle: development of decision support

Numerous studies have shown that CDS can improve physician compliance with best practices, therapeutic guidelines and preferred formularies.24 Trials focussing specifically upon DVT prevention have shown that CDS can improve clinician adherence with recommended prophylaxis.25 26 Thus, our OCD sponsored development of a prophylaxis CDS ‘package’ to improve compliance with recommended DVT prophylaxis.

An interdisciplinary QI team consisting of a project manager, an internist, a pharmacist and three informaticians was charged with the development and implementation of new electronic order menus. Subject matter experts were recruited to provide clinical content and test prototype menus. The facility Pharmacy and Therapeutics committee provided oversight to ensure compliance with hospital prescribing policies and formulary use criteria.

The team used a combination of Plan-Do-Study-Act (PDSA) and user-centred design (UCD) processes throughout menu development.27 28 Both processes require multiple iterative cycles of incremental change informed by regular multi-modal evaluation (figure 2). The first step was assembly of a knowledge base and included a literature review of published clinical guidelines, collection of protocols from academic affiliates and discussions with subject matter experts. Next, the QI team used non-participant observation and think-aloud protocols to study the cognitive tasks, ordering patterns, and workflow processes of users. Finally, the team used all data to assemble flowcharts of clinical tasks and menu design requirements.29

Figure 2

Gantt chart illustrating deep venous thrombosis prophylaxis process improvement interventions according to Plan-Do-Study-Act (PDSA) cycles.

The team enlisted surgeons to test a series of menu mock-ups. Surgeon feedback was then used to build speciality-specific order menus. Prophylaxis options were organised according to patient risk categories (figure 3). The team developed unique orders with embedded meta-data ‘tags’ to facilitate order tracking. These orders appeared every time clinicians accessed post-operative order sets.

Figure 3

Sample order menu reflecting patient risk categories, suggested prophylaxis options and alternatives if contraindications to pharmacologic prophylaxis exist. CAD, coronary artery disease; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; INR, international normalized ratio; SCD, sequential compression devices; SQ, subcutaneous; TED, thromboembolic deterrent; tx, treatment; VTE, venous thromboembolism.

An educational campaign accompanied release of the new order menus. The QI team members gave presentations to the residents, pharmacists and nurses describing the purpose and design of the new menus. The surgical residents and staff attended morning reports dedicated to new menu content. The team posted informational flyers in all surgical staff and resident work areas and break rooms.

Second improvement cycle: analysis of human factors

After the first cycle, we gathered additional data to assess the organisational impact of the CDS implementation. The team used a variety of qualitative methods including non-participatory observation, semi-structured interviews with surgeon focus groups and chart reviews of the prophylaxis orders to understand sociotechnical barriers. At each step of collection, we categorised data according to the FITT model criteria (table 1). Each component of the activity system (ie, user, task and technology) was clearly defined and each interface between components was explored by drawing from several epistemological disciplines including the social and cognitive sciences.

Table 1

Sociotechnical issues identified by the team during CDS implementation and interventions taken to improve performance. Findings are categorised using the FITT framework (adapted from Ammenwerth et al8)

		Issues or barriers to CDS success	Interventions to improve performance
Actors in System	Task	Additional orders were needed to improve efficiency of order-entry time	Corollary orders created for each subspeciality to improve efficiency of transfer order-entry
	User	Heterogeneous cohort of surgical subspecialists Ordering providers rotate frequently	Custom order menus created for each speciality Training was embedded in resident orientation
	Technology	Orders needed to be tracked automatically No way to automatically capture contraindications	Unique prophylaxis order items built for tracking Created unique contraindication orders
Interfaces	Task Technology	Surgeons used unexpected order menu pathways Orders embedded in deep in sets were overlooked	Obsolete pathways removed from the order menus Prophylaxis orders placed at start of pathway
	User technology	Risk categories were difficult to apply to patients Decision support content difficult to interpret	Prophylaxis options organised by procedure Content updated according to user feedback
	User task	Guidelines discordant with local practice Providers unwilling to use enoxaparin or heparin	Clinical champions engaged to in-service trainees New warfarin order process developed with anticoagulation clinic

The team designed interventions to address each identified FITT barrier. We worked on menu usability by streamlining workflow pathways, rephrasing risk assessment criteria and reorganising resources. Addition of requested content such as new menus and corollary orders improved stakeholder buy-in. Guidelines were clarified and order set pathways were simplified. Because the second PDSA cycle coincided with the start of the academic year, we incorporated menu-training sessions into the new resident orientation curriculum.

Data analysis

Three metrics were established a priori to measure treatment effect: (1) the proportion of cases in which new order sets were displayed (ie, menu exposure); (2) the proportion of cases in which recommended prophylaxis was ordered using any process (ie, clinical performance) and (3) the proportion of cases in which prophylaxis orders were placed using the new menu (ie, menu use). Inclusion and exclusion criteria are listed in table 2. We used VistA database query tools to measure the frequency of menu exposure and the selection of prophylaxis orders. We assessed compliance with recommendations by manual chart review. The team used time series methods for data analysis.

Table 2

Definitions, inclusion criteria and exclusion criteria for study sample22

Data definition	Orders placed for prophylaxis: Subcutaneous low-dose heparin Low-dose enoxaparin Dose adjusted warfarin for DVT prophylaxis Sequential compression devices
Inclusion criteria	Hospitalised for at least 24 h Operative procedure performed by general surgery, gynaecology, neurosurgery, orthopaedics or urology Admitted between 3/1/2008 and 12/31/2009
Exclusion criteria	Surgery lasted less than 30 min Patient underwent laparoscopic procedure Patient already receiving full-dose anticoagulation

During the first improvement cycle, the team used several strategies to establish the internal and external validity of automated data collection. Our automated data collection method used the VistA database query utility to identify all enoxaparin, subcutaneous heparin and warfarin orders for patients meeting inclusion criteria. For patients with contraindications to anticoagulation, we queried charts for documentation of a contraindication and an order for compression devices. To establish the internal validity of our automated query, a clinician reviewed all surgeries meeting inclusion criteria during one week in April 2008 (N=30), searching for appropriate medication orders and documentation of drug administration or evidence of compression device application. The automated query accurately captured 100% of menu usage, medication orders and compression device orders, but missed some documentation of compression device application. In three instances (10% of cases reviewed), the documentation was captured in scanned ICU flowsheets, but not the chart notes. The automated database queries were incapable of extracting content from scanned documents. To establish external validity, the team compared local findings to facility data collected and scored by independent auditors (ie, national EPRP contractors). The compliance rate for 35 randomly selected charts was identical (compliance=77%).

To measure a potential relationship between use of the new menu and change in facility performance, we applied Pearson's correlation between the proportion of cases with recommended prophylaxis ordered and the use of the new order menu. All statistics were performed using SPSS version 14.0.

Results

Quantitative findings

There were 80–110 eligible surgical cases each month during the study period. General surgery, orthopaedics and urology had the highest case volume with approximately 30 cases per month per service. Neurosurgery averaged 10 cases per month and gynaecology completed one or two cases per month. The VA began measuring the frequency of DVT prophylaxis orders in FY2007.

After the first implementation cycle, data suggested that providers rarely saw the new menus; providers used embedded orders in only 10% to 20% of cases (figures 4 and 5). Chart review showed that surgeons used several patient-transfer order pathways that bypassed our new DVT prophylaxis menus. During the second cycle, reconfiguration of order menus appeared to improve order set accessibility; providers triggered the menu in over 90% of cases (figure 4). Average order use increased from 50% to 90% but adoption and use varied by speciality (figure 5). Services with the greatest case volume showed the greatest improvement; general surgery and neurosurgery had the largest absolute increase in prophylaxis compliance.

Figure 4

Proportion of deep venous thrombosis prophylaxis eligible surgical cases in which the provider triggered the new prophylaxis order menu. These cases may or may not have received prophylaxis orders from the new menu. New menu use increased across all specialities and reached a steady state averaging 90%. During the study interval, case load for general surgery was approximately 100 per quarter; gynaecology 2–7; neurosurgery 20–30; 100 for orthopaedics; and 89–90 for urology.

Figure 5

Proportion of deep venous thrombosis prophylaxis eligible surgical cases with prophylaxis orders selected from the new menu. There was a steep increase in prophylaxis order use from the new menu during the second Plan-Do-Study-Act cycle. During the study interval, case load for general surgery was approximately 100 per quarter, gynaecology 2–7, neurosurgery 20–30, 100 for orthopaedics, and 89–90 for urology.

Medical centre indicator performance also improved, exceeding the target by the end of the second PDSA cycle (figure 6). There was an overlap in the CIs for pre-measures and post-measures of EPRP performance with the exception of Q1, 2010. Performance trends appeared to correlate with new menu use (Pearson's correlation coefficient=0.832, p=0.010), although a substantial number of orders were selected from other menu paths. Use of menu options strongly predicted performance measure success; 36 of 38 randomly selected cases from the treatment cohort were compliant with the EPRP measures (PPV=95%) (table 3). The two noncompliant cases used mechanical prophylaxis when pharmacologic measures were indicated.

Figure 6

Proportion of surgical cases with prophylaxis orders selected from the new menu and facility performance on EPRP measures during the study time period. Error bars indicate 95% CI for EPRP sample evaluation. Facility performance improved and met the EPRP measures target after the second cycle of implementation. Improvement correlated with the proportion of orders selected from the new menu (Pearson's correlation coefficient=0.832; p=0.010). DVT, deep venous thrombosis; EPRP, External Peer Review Program.

Table 3

Review of 38 surgical cases with DVT prophylaxis ordered from the new menu. The results of the review indicated that all cases with DVT prophylaxis selected from the new menu met the recommended guideline

Speciality	# Cases	Principle procedure	DVT prophylaxis ordered
General surgery	12	Tracheal incision Pyloric reconstruction Colectomy Colostomy Suture, small intestine Late wound closure Gallbladder removal Exploratory laparotomy	Heparin 5000 units SQ Q8H and/or compression devices
Gynaecology	2	Oophorectomy Extensive hysterectomy	Heparin 5000 units SQ Q8H
Neurosurgery	3	Skull drainage Meningeal leak repair Wound repair	Heparin 5000 units SQ Q12H and/or compression devices
Orthopaedics	9	Revise/replace knee joint Total hip arthroplasty Drain thigh/knee lesion Total knee arthroplasty	Warfarin 5 mg PO QPM and/or compression devices
Urology	4	Radical prostectomy Open radical nephrectomy Cystectomy	Heparin 5000 units SQ Q8H and/or compression devices

Qualitative findings

Stakeholder interviews revealed several important technical and social barriers to CDS acceptance. We classified findings according to the components of the FITT framework (ie, user, technology, task) and the interfaces between components (table 1). The most important barriers identified included the presence of accessory or work-around pathways in CPRS, poor order set usability and misalignment between CDS recommendations and pre-existing surgical care patterns. Moreover, provider workflow was non-linear, and highly adaptive, affording ample opportunity to bypass our clinical pathways and recommended guidelines.

When clinical pathways were used, providers initially struggled with the order sets. Surgeons had increased order entry time and a decrease in productivity. Focus group respondents argued that patient risk stratification guidelines were frequently not prescriptive enough to effectively apply to individual patients at the point-of-care. Finally, many providers disagreed with the clinical guidelines, arguing that the risk of hemarthrosis outweighed the potential protection pharmacologic prophylaxis afforded against fatal PE.

Discussion

Our data suggest that surgeons variably used our DVT prophylaxis CDS, but that order use positively correlated with EPRP compliance. There appeared to be a gradual increase in order use over time; the trend lines show three general trajectories (figure 5), a slow adoption rate after the first PDSA cycle, a second more rapid adoption rate after the second PDSA cycle and a third gradual rise over the last two quarters. We believe that technical design changes (eg, removal of accessory order pathways) were responsible for the rapid increase in utilisation between the first and the second cycle. The team identified several technical problems with the CDS that disrupted workflow or limited usability. These issues were partially design related and partially related to the limitations of the legacy electronic record (eg, VistA/CPRS does not support complex branched logic). We suspect that the more gradual trajectories of performance improvement were a reflection of slower socio-organisational change including new learning, peer-to-peer influences and socio-political forces. Although we sought to accelerate this process using multi-modal educational campaigns, these strategies appeared to be only marginally effective. In circumstances where surgeons were unfamiliar with patient risk categories and prophylaxis options, teaching initiatives or menu revision proved useful. Conversely, surgeons with firmly established practice patterns or expressing concerns about anticoagulation risks strongly resisted change.

Although our intervention was successful, it illustrates the complex interplay between technical and human determinants. The quantitative PDSA cycle alone would not have elucidated many of the barriers that delayed CDS adoption. Its singular cause–effect analytics do not provide information to understand the inherently multivariate nature of implementation science. Conversely, organisational change theories popularised by Rogers propose explanations for how groups adopt innovations across large-scale organisations.30 Rogers diffusion of innovation theory describes several key organisational and technologic determinants of systemic change: (1) the characteristics of the technology, (2) users' readiness for change and (3) the organisational structure. When introducing new technology for the purposes of QI, it is important to explore entrenched stakeholder belief systems using exploratory methods and engage target users before applying incremental changes. By systematically studying the sociotechnical system, the implementation scientist can rationally choose interventions that accelerate improvement, limit resource expenditure and optimise the sociotechnical fit.

The FITT framework provides a coherent and adaptable heuristic to direct qualitative enquiry, identify barriers to implementation, and classify themes for the purposes of developing a coordinated response. Furthermore, the framework operationalises the diffusion of innovations theories popularised by Rogers in several discrete ways.30 The FITT framework (1) uses the components of user and task to help recognise and optimise the intrinsic characteristics of innovations that influence adoption (eg, usability, compatibility, relative advantage); (2) the framework emphasises the importance of the user as an agent of the social system; early adopters, innovation champions, opinion leaders and laggards all have an important influence upon the trajectory of adoption and (3) the framework normalises expectations for change in the context of QI; complex interventions that include a social, cultural, or behavioural component are likely to experience several stages of adoption and follow a non-linear pattern of diffusion. Although, we cite several technical changes that we hypothesise had a direct impact on the rate of menu use, it is equally likely that the sigmoidal pattern of uptake, in part, reflects the recruitment of different user types along the adoption continuum. Engaging service chiefs and senior residents was critical for identifying improvement opportunities, gaining political capital within the OCD and developing internal momentum to promote menu use.

In our example, the FITT framework provided a supplementary model, informing each component of the PDSA cycle. By striving to understand the barriers affecting activity system components and the interfaces between them, we were able to develop a plan that addressed user needs, implement an intervention that articulated with workflow, study the contextual determinants of performance, and act in alignment with stakeholder expectations. To our knowledge, this is the first report applying the FITT framework to a QI intervention and suitably captures the human-centred issues intrinsic to QI work. It enriched our QI methods by clarifying the basis of user resistance and highlighting the relationship between organisational attributes and outcomes. The framework draws strategies from the cognitive sciences, anthropology and applied informatics, which in turn, helped inform better overall design.

Despite the robustness of the FITT model to understand user acceptance of technology, it is not the only one reported in the health informatics literature.8 31–35 Most models are complementary, though few have been empirically tested in multiple settings and none have emerged as superior. Although the FITT model establishes an approach to assess the readiness of an organisation to adopt change and the specific facets deserving of study, it may not be suitable for all implementation efforts. For example, the framework may incompletely model the exchange of new information between unlike users in a heterogeneous clinician cohort. Also, the framework does not explicitly focus upon organisational behaviour and, therefore, may incompletely capture the cultural differences at the organisational, team or unit level.8 Finally, the model was based on analysis of the literature and aimed at new technology implementations. It may be less capable of detecting sociotechnical barriers in sites where clinicians use mature electronic record systems with embedded CDS. In our case study, it is notable that many barriers were not identified until the second QI cycle. Many sociotechnical issues were deeply entrenched in the activity system and only recognised in an applied setting possibly because (1) end users may not be positioned to articulate technology requirements, (2) cognitive dissonance and cultural resiliency are best observed in context, and (3) FITT dimensions are dynamic and tend to evolve in response to system change.7 9

There are several limitations to this study that deserve mention. First, time series methods lack randomised controls or blinding and are sensitive to bias. Second, the team only examined a single medical centre. Hence, our results may not be generalisable. Finally, the time and resource constraints typical of quality improvement work limited our ability to gather qualitative data. We did not digitally record and code clinical activity or user narratives. Instead, we relied upon ethnographic survey techniques and field notes.

There are also limitations beyond the parameters of this project that deserve mention. First, the model has not been well tested in diverse healthcare settings and, to our knowledge, has never been tested in a QI domain. Hence, it should be applied to other QI initiatives in a variety of healthcare contexts to ascertain its relevance to other CDS interventions and clinical sub-cultures. Second, the model is a theoretical representation of many co-existing processes and sub-cultures. Consequently, it may oversimplify or incompletely capture the complex healthcare activity system with sufficient granularity to model all the drivers of change. Finally, the model does not take into account the axis of time. As we stated before, the introduction of innovations into a system can have profound and complex social effects. Hence, it may be important to capture the organisational-state pre-implementation and track the evolution of state as work practices shift to reinforce the cultural norms threatened by new technology. For example, the framework did not explicitly articulate any transient socio-political phenomena, when, in fact, the facility did not have an identified section chief for Orthopaedics at the time of implementation. There may have been less faculty accountability and less stakeholder buy-in to the OCD strategic mission.

Conclusions

In summary, our clinical decision support system increased provider adherence to DVT prophylaxis guidelines and positively correlated with medical centre performance. This report shows one potential approach to synthesising the complementary disciplines of QI and sociotechnical theory. Although rationalist improvement methodologies appropriately emphasise a data-driven ethic, sociotechnical heuristics such as the FITT model invoke social science discourse to explain the ‘why’ as much as the ‘what’ or ‘how’ of system design. Hidden socio-political agendas, dissonant cognitive frameworks, and misalignment of expectations must be articulated if organisations are to identify and respond intelligently to internal resistance to change. Furthermore, this case study demonstrates that QI teams leveraging information technology must possess a nuanced understanding of context, even in mature electronic health record environments, if they expect to implement durable and sustainable change.