Surgical safety checklist audits may be misleading! Improving the implementation and adherence of the surgical safety checklist: a quality improvement project.

Despite good quality evidence for benefits with its use, challenges have been encountered in the correct and consistent implementation of the surgical safety checklist (SSC). Previous studies of the SSC have reported a discrepancy between what is documented and what is observed in real time. A baseline observational audit at our institution demonstrated compliance of only 3.5% despite a documented compliance of 100%. This project used quality improvement principles of identifying the problem and designing strategies to improve staff compliance with the SSC. These included changing the SSC from paper-based to a reusable laminated form, a broad multidisciplinary education and marketing campaign, targeted coaching and modifying the implementation in response to ongoing staff feedback. Five direct observational audits were undertaken over four Plan-Do-Study-Act cycles to capture real-time information on staff compliance. Two staff surveys were also undertaken. Compliance with the SSC improved from 3.5% to 63% during this study. Staff reported they felt the new process improved patient safety and that the new SSC was easily incorporated into their workflow. Improving compliance with the SSC requires deep engagement with and cooperation of surgical, anaesthesia and nursing teams and understanding of their work practices and culture. The prospective observational audit highlighted an initial 3.5% compliance rate compared with 100% based on an audit of the patient notes. Relying solely on a retrospective paper-based model can lead to hospitals being unaware of significant safety and quality issues. While in-person prospective observations are more time-consuming and resource-consuming than retrospective audits, this study highlights their potential utility to gain a clear picture of actual events. The significant variation between documented and observed data may have considerable implications for other retrospective studies which rely on human-entered data for their results. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Problem

Since its inception in 2008, WHO’s surgical safety checklist (SSC) has been adopted in almost 2000 operating theatre environments around the world.1 Adherence to the SSC has been associated with improved surgical mortality and morbidity globally.2 Despite validated evidence of benefits with its use, challenges have been encountered in the correct and consistent implementation of the checklist.3 4 A prospective observational audit at our institution, Flinders Medical Centre (FMC) found that the SSC process was conducted correctly and in its entirety in just 3.5% of surgical cases. In contrast, the document used to record the process in the patient record reported 100% compliance. The discrepancy between what occurred and what was recorded may impact patient care.

FMC is a large quaternary hospital in Adelaide, South Australia. As part of the Southern Adelaide Local Health Network (SALHN), FMC provides healthcare to southern metropolitan Adelaide as well as several state-wide clinical services. There are 13 operating theatres within the main operating theatre complex in which approximately 25 000 surgeries are performed annually across a range of surgical specialties.

Previous studies of the SSC have reported a discrepancy between what is documented and what is observed in real time.5 Notably, the lowest observed SSC completion rate reported in published literature is 10%, compared with 3.5% at FMC. Furthermore, the significant variation between documented and observed data may have considerable implications for retrospective studies which rely on human-entered data for their results. If relying solely on a retrospective review of the notes, a hospital would be unable to detect poor adherence thus preventing an impetus for improvement. This quality improvement (QI) project aimed to increase observed SSC compliance in operating theatres to 90% within six months, where compliance was defined as the operative team verbalising every item on the SSC. We predicted that increasing SSC compliance would help to enhance patient care and improve patient outcomes.

Background

Major perioperative complications occur worldwide at a rate of 3%–17%, and mortality for inpatient surgery is 0.4%–0.8%.2 6 Issues include the incorrect patient/procedure/surgical site, lack of equipment (anaesthesia and surgical), unanticipated blood loss, unsterile equipment and retained surgical equipment. In a large Australian study, the perioperative adverse event rate was found to be as high as 21.9% with a mortality rate of 4%.7

The 2008 WHO study trialled the SSC as part of their guidelines for Safe Surgery Saves Lives programme and reported reductions in patient mortality from 1.5% to 0.8% and patient complications from 11% to 7%.2 These findings were consistent across all sites, regardless of the size or wealth of the healthcare institution.3 Following the results of this study, locally adapted versions of the SSC have been implemented worldwide. Reports from institutions range from no significant reduction to as high as a 57% reduction in complications.8 9

Challenges and barriers to the successful implementation of the SSC into practice are well reported. Observational studies have noted a wide range of organisational, cultural and individual factors affecting compliance of the SSC including no perceived benefit, ambiguity and confusion over the questions, the checklist affecting workflow efficiencies, inappropriate timing of the checklist and poor communication.10–12 These studies concluded that checklist design should account for local organisational needs and the progress of implementation acceptance should be reviewed regularly for accountability.

Measurement

A baseline direct observational audit was undertaken for a variety of surgical specialties in an effort to gain an overview of the existing SSC process at our institution. Data were collected by 10 medical students and 6 junior doctors assigned to work and observe in each theatre. One of the researchers (BB) trained the auditors in a one-on-one environment prior to commencing data collection by reviewing the checklist and auditing tool and ensuring their understanding of the process. Discretion was advised to ensure staff working in the operating theatre remained blinded to the process. An electronic proforma accessed on a smartphone was used to assess whether each individual item of the SSC was verbalised and completed in the operating theatre. At the end of each case, the data collector observed the completed paper copy of the SSC and noted which information had been completed on the form and signed by one of the surgeons. Ongoing support for the clinical audit team was available in person and electronically from the project lead (BB).

Auditing of the SSC process was repeated over four further Plan–Do–Study–Act (PDSA) cycles throughout the project (see the Strategy section). Following a baseline compliance of 3.5% in January 2019, four further direct observational audits were undertaken in June 2019, August 2019, October 2019 and March 2020 for a total of five audits. Each audit collected data from 100 cases across a range of surgical specialties. The main outcome measure was percentage compliance with the items in the SSC. Run charts were created from the audit data for each individual stage of the SSC (preinduction, preincision, before unscrubbing) as well as for overall checklist compliance. Throughout this process, auditors also recorded near-miss events or safety issues that were identified during the SSC process.

Two online staff surveys were performed during the roll-out of the new process in June and September 2019 to assess staff members’ impressions of the effectiveness, safety and workflow integration of the SSC (online supplemental appendix A). The first survey consisted of one question to identify the respondent’s role, six closed-ended questions on a three-point Likert Scale (improved/not improved/unchanged) with an opportunity to comment and three open-ended questions. The second survey had additional questions to assess changes made to the SSC since the first survey, and it also focused on staff opinions regarding barriers to change and their suggestions on how to improve SSC compliance.

Design

QI process

SALHN has a 15-year history of implementing and sustaining QI initiatives via the Continuous Improvement Programme (CIP), an internally developed capability and support programme. The CIP uses methods derived from Intermountain Healthcare in Utah, USA,13–15 Lean Thinking,16 17 Redesigning Care17 and the Model for Improvement.18 SALHN also established an improvement faculty to coach and support staff, and to progressively improve the QI methodology. The education component comprises introductory off-site training sessions over 3.5 days, with presentations by members of the faculty. Day 1 topics include an overview of the CIP’s history and key objectives, the evolution of QI and the need for a standardised, patient-focused improvement framework. Days 2 and 3 involve small group work and an introduction to the diagnostic tools used in the CIP (breaking down the problem, process mapping, brainstorming, multivoting and Pareto charts). Other topics include creating targets and effective tools to improve flow, achieve standardisation and in-built quality.

The SALHN Continuous Improvement Framework involves an eight-step problem solving process: (1) define the problem, (2) breakdown the problem, (3) set a target/mission/aim statement, (4) root cause analysis, (5) interventions planning, (6) implementation, (7) evaluate/assess impact and (8) continuous improvement.19 These eight steps are designed to empower all staff to solve a range of simple and complex problems in a standard, systematic and patient-focused way. The framework involves ongoing, periodic changes in practice, underpinned by the ‘PDSA’ cycle.17 20 Distinct from conventional clinical trials where specific and controlled interventions are assessed after prolonged implementations, continuous QI creates a culture of real-time problem solving.21

Diagnostic process

After attending the CIP, the lead author assembled a multidisciplinary QI team of anaesthesia nurses, anaesthetists, surgical nurses, surgeons and administrators who were interested in improving the SSC. The QI team met regularly during the planning and implementation phases of this project. Major areas for change were identified by following a QI format, including brainstorming, creation of Ishikawa diagrams, multivoting and Pareto charts. A short presentation at the first meeting explored the background of the WHO SSC and the baseline direct observational audit results at FMC. Although attendees anticipated a low compliance rate, they were ‘genuinely shocked’ to discover that the rate was just 3.5%. They were further concerned to note the 100% completion of the paper form found in the patient notes. The QI team brainstormed reasons why checklist compliance was so low, which were then grouped into themes and arranged into an Ishikawa fishbone diagram (figure 1). The main themes were communicated to the QI team, after which they were asked to vote on the issues they felt were key to improving compliance. Three rounds of electronic voting led to consensus, and results were displayed in a Pareto chart (figure 2). The results of voting were presented to the QI team, being: (1) The form was complicated and had a poor layout; (2) There was no formal pause by staff to complete the checklist and (3) There was a lack of staff buy-in. Approaches to tackling these issues were then discussed and strategised.

Figure 1

Ishikawa fishbone diagram outlining the grouped themes brainstormed by the quality improvement team for reason why the surgical safety checklist was poor. EPAS, Electronic Patient Administration System.

Figure 2

Pareto chart showing the consensus following three rounds if electronic voting by the QI team. The top three issues identified were (1) The form was complicated and had a poor layout; (2) There was no formal pause by staff to complete the checklist and (3) There was a lack of stuff buy-in. QI, quality improvement.

Strategy

A key alteration in our SSC process was the creation of a new form, which involved modifications in format and structure. The form was changed from two stages (preincision and postoperative) to three stages (preinduction, preincision, before unscrubbing) to better align with the original WHO layout. The number of items on the form decreased from 21 to 19, and were presented in a more readable, user-friendly design (see online supplemental appendix B). Furthermore, instead of a hard copy being completed, signed by a surgeon and stored in the patient’s notes, we designed a reusable laminated form that was secured to a metal board and stored in each operating theatre. Each checklist stage was filled out in the operation theatre using a whiteboard marker, and wiped clean at the end of each case. The strategy behind this change was to promote a team discussion of safety issues, rather than the surgical doctor completing and signing a form for the notes. SSC completion was still recorded in the patient’s perioperative record, which was filed in the patient’s medical record, to provide evidence of completion for medical records, billings and auditors.

The launch of the new SSC process spanned several months, and included extensive staff-wide education programmes to increase their knowledge and buy-in. The new three-stage checklist had a stronger team focus and was implemented over four PDSA test cycles.

PDSA cycle 1 (January–June 2019): implementing the new SSC format

The initial intervention aimed to implement the novel SSC process (including the new form) with adequate preparation and education of all perioperative staff. Our multimodal approach included providing information in the form of teaser campaign posters, a video made by and with staff members, and educational brochures and documents distributed in the tearooms. Nurse educators developed an online education package and an online forum to improve access for staff working irregular shifts. The QI team presented the project to every surgical specialty at their weekly meeting, and arranged for emails to be sent to every staff member from their respective head of department to demonstrate buy-in from each department’s leaders. The launch day included pizza and snacks, on-the-ground coaching of theatre staff by QI team members and opportunities for staff to give informal feedback in person or on a whiteboard. The first formal staff survey was performed through an online Survey Monkey. The survey was loaded on all work computers, and a reminder pop-up message appeared when staff logged on to a work computer for the first time to encourage them to complete the survey. Providing staff with the opportunity to respond anonymously facilitated open communication with the QI team. An audit was completed at the end of this cycle, 1 month after the new SSC was launched.

PSDA cycle 2 (June–August 2019): modifying the SSC wording and process

The SSC form and process were altered to incorporate survey data and verbal staff feedback of issues they encountered in the initial trial period. This included the scrub nurses’ desire to be present for the first stage of the checklist which was not required by the WHO and initially not part of our new process. Some wording on the form was also adjusted following feedback and these changes were highlighted and fed back to staff as part of the launch of the second stage of the project. It was important for staff to feel that their feedback was being incorporated. The main focus now was on the execution and timing of the three stages of the SSC. A nurse became the initiator and team leader of the checklist, and all staff were instructed to pause at the time of each stage being completed to ensure everyone heard the information being shared. We continued to use a whiteboard to provide staff with an opportunity to anonymously write questions or concerns. The QI team regularly checked and responded to comments and questions on the whiteboard.

PDSA cycle 3 (August–October 2019): improving the ‘before unscrubbing’ stage

Cycle 3 focused on improving the compliance of the third stage of the checklist (before unscrubbing) which repeatedly had the poorest compliance. Stage 3 completion reminders were posted on staff computers and whiteboards, email reminders were distributed, and QI team members circulated in theatres. Having the visual reminders present and verbal reminders from the QI team helped to keep staff engaged in the SSC process.

PDSA cycle 4 (October 2019–March 2020): reinforcing the process

In the final cycle, staff were resurveyed about their experiences and attitudes toward the new SSC. Links to the second online survey were widely distributed and again available on all work computers. Throughout the project potential errors and near misses identified during the SSC (and therefore avoided as a result of the process) were documented and shared with staff. After months of supporting staff in the new SSC process, it was noted that while the new checklist was completed in its entirety with all team members participating most of the time, that some staff refused to participate. The organising team relied on heads of department to lead by example as well as lend support by emailing staff with the expectation to participate. The team also spoke in a one-on-one environment to individuals who were not engaging, to address their concerns and improve engagement.

Results

After the initial launch, overall SSC compliance improved from 3.5% to 63% and then remained relatively unchanged in the following three audits (figure 3A). While this is below the target set of 90% there has still been a statistically significant improvement overall.

Figure 3

(A) Overall success rate of compliance with the surgical safety checklist. A baseline audit demonstrated 3.5% overall compliance was found to be 63%. (B) Part 1 success rate of compliance with the surgical safety checklist. After four PDSA cycles, compliance was found to be 99%. Note that the new SSC changed to a three part checklist so no baseline data is available. (C) Part 2 success rate of compliance with the surgical safety checklist. After four PDSA cycles, compliance was found to be 74%. Note that the new SSC changed to a three-part checklist so no baseline data is available. (D) Part 3 success rate of compliance with the surgical safety checklist. After four PDSA cycles, compliance was found to be 69%. Note that the new SSC changed to a three-part checklist, so no baseline data are available. PDSA, Plan–Do–Study–Act; SSC, surgical safety checklist.

Subanalyses were performed for each stage of the SSC. Stage 1 of the checklist (preinduction) initially improved to 88% and ultimately 99% compliance on the most recent audit, surpassing our 90% target (figure 3B). Stage 2 of the checklist (preincision) improved to 91% but then declined to 74% (figure 3C). Stage 3 of the checklist (before unscrubbing) was the least consistently completed, with audit results ranging from 37% to 74% (figure 3D).

Secondary measures included staff acceptance of the new process and buy-in to the project. Two surveys were performed to gain insights into staff experiences of, and attitudes towards, the SSC. The first survey was performed 1 month after the roll-out and received a total of 103 responses, and the second survey performed four months later received 109 responses. Both surveys included a broad range of respondents including 22% anaesthesia nurses, 20% surgical nurses, 31% anaesthetists, 23% surgeons and 4% orderlies. This response profile was similar between both surveys. Regarding patient safety, 85% of respondents in the second survey felt the new SSC process had improved patient safety compared with 68% in the first survey. Staff reported improved workflow integration over time from 37% in the first survey to 72% in the second survey. Staff specifically commented that all teams were more involved in the process, and that verbalising team members’ names was useful (63% of respondents noted an improved knowledge of their team). The majority of staff are happier with the new process and believe that it is improving patient safety.

Near miss adverse events as reported by staff and observed directly by the research team were collected and documented throughout the project. While not exhaustive nor complete, 52 events as a representation were captured between June 2019 and March 2020. The main themes of these near miss events were incomplete surgical consents, operative sites not being marked or being marked incorrectly, equipment found to be unavailable or damaged and abnormal preoperative investigations of which some members of the team were previously unaware. In one case, a surgical specimen was nearly disposed of, but was instead sent to the laboratory after being identified in the third stage of the checklist.

Lessons and limitations

This project aimed to increase operating theatre staff compliance with the SSC. During the project, compliance increased from a baseline of 3.5% to 63%. Stage 3 of the checklist (before unscrubbing) was the least consistently completed, with audit results ranging from 37% to 74%.

We reflected on several lessons learnt in the roll-out of this SSC project. First, the formation of a strong QI team who shared key goals and a common view was critical in creating sustainable change. Recruiting ‘champions’ from each stakeholder group to join the QI team enabled us to meaningfully engage with each operating theatre staff group and build support and momentum for our project.

The operating theatres house several different surgical specialties with their own unique cultures and clinical situations. The QI team were tasked with designing one system that would work across a variety of scenarios. A key lesson was meeting with each specialty group early to identify and address their specific concerns. Members of the QI team met with each surgical and nursing department to learn about their unique requirements. This process highlighted potential issues for the roll-out and improved buy-in as staff felt listened to and involved from the outset.

A major change in this project was moving from a paper checklist completed and signed by a surgeon and stored in the patient notes to a reusable, laminated checklist which was mounted on a metal board and remained in the operating theatre. This process required an exigent shift in mindset for many staff. Crucially, it involved flattening the operating theatre hierarchy, with checklist ownership and responsibility moving from the individual surgeon’s remit to becoming a shared responsibility of the entire team. Staff were initially reluctant to lead the checklist and identified some cumbersome aspects that were subsequently addressed. However, with time, they have generally accepted the process and the checklist is now nurse-initiated. We feel this move in format was critical to improving the major problems initially identified by the QI team, particularly the lack of staff buy-in and lack of a formal pause. A systematic review12 had previously concluded that the format of the SSC did not significantly impact staff compliance with the checklist. Our findings were consistent with a more recent study, which showed increased staff engagement and compliance with a shift from a paper copy to a wall-mounted checklist.22

The project highlighted that resistance to change is a key barrier to implementation. Our health system has undergone significant change in the last few years and there was concern about ‘change fatigue’ affecting acceptance of the project. This was exemplified by some staff who were hesitant to change practice despite the evidence provided. Additionally, shift work presented barriers to reaching and educating all staff members prior to the initial roll-out. Using multiple mediums to reach staff at all times was invaluable to inform the largest number of people. In addition to in-person meetings held at different times during the day, online resources, both written and visual, were provided on staff forums which could be accessed at times of personal convenience. These resources were also printed and left in break rooms for staff to read without the need for internet access.

A starting point from our study was the finding of a major discrepancy between what is documented in patient notes and what was directly observed. Similar findings have been made in an Australian study on the SSC across 11 hospitals in which the directly observed checklist completion rate was 27%, compared with 86% completion in the medical record.5 Most quality and safety data in healthcare organisations rely on audits of medical records, however, in our facility, these data were found to be unreliable in relation to the SSC. Healthcare organisations may need to perform ‘spot audits’ of other clinical processes through direct observation rather than merely retrospectively reviewing notes. If hospitals do not ‘sense-check’ their audit data using prospective observation techniques, they may have a distorted view of the safety and quality of their services. Conversely, prospective observations are very time consuming and should be used appropriately. A strength of the study was using medical students and junior doctors who were familiar with the operations and processes of the operating theatres but were inconspicuous enough to maintain blinding of the auditing process.

Direct observation was a powerful tool which helped the team understand the gap between what was thought to be happening, and what was actually happening. Observation of current workplace conditions allowed the QI team to consider the whole context, including when the workplace culture had the potential to impact compliance with the process. From a sociotechnical perspective, an approach which integrates procedures with human factors is required to undertake tasks.17 23 Direct observation of current conditions is a key part of what process improvement experts describe as ‘learning to see’.17 24 This information can be helpful to potentially influence workplace culture, as it connected the QI team with staff and their stories of how the process was going and the resulting potential impact. Direct observation ensured information was fact based, challenged assumptions of what people thought the barriers were, picked up levels of engagement (or lack thereof), non-verbal cues and allowed those who did the work to adapt as they went. When people learn to see their own processes clearly enough to develop their own changes, the change is much more likely to be sustainable.17

Regarding limitations of our study, it was performed at a single hospital site and this may limit the generalisability of the results. However, our findings are consistent with other studies in the literature with similar barriers to change identified.10–12 25–27

The QI team worked hard to gather critical incident and near miss data which was averted by the SSC, however, when a member was not in the operating theatre, this process was reliant on self-reporting. This proved to be more limited than hoped, and with an unknown total number of events, it is difficult to draw conclusions about adverse events or near misses. This highlights the importance of in-person auditing, as relying on individuals to identify and report can be varied and unreliable compared with real-time data gathering.

Conclusion

Successful implementation of the new SSC process led to an observed improvement in compliance from 3.5% to 63%. This required coordination and cooperation of multiple surgical, anaesthesia and nursing teams to shift an ingrained workplace culture. Consistency during the initial phases, availability of the QI team during the roll-out, and ensuring staff felt involved in the change helped to improve staff buy-in and support.

Our prospective observational audit highlighted an initial 3.5% compliance rate compared with 100% based on an audit of the patient notes. Relying solely on a retrospective paper-based model can lead to hospitals being unaware of significant safety and quality issues. While in-person prospective observations are more time and resource-consuming than retrospective audits, this study highlights their potential utility to gain an accurate picture of the actual events in the operating theatre.