The effects of surgical checklists on morbidity and mortality: a pre- and post-intervention study.

BACKGROUND: Surgical checklists (SCs) have been developed to enhance teamwork and facilitate handovers, thereby improving the safety of surgical patients in health care organisations. The aim of this study was to determine whether the implementation of a 39-item SC reduced mortality and surgical adverse events (AEs) in patients undergoing inpatient surgery.
METHODS: A retrospective pre- and post-intervention study of two cohorts of surgical patients was conducted (n = 1602) in a tertiary teaching hospital. The patients' homogeneity was confirmed by studying 40 comorbidities, 13 analytical determinations and 14 patient- and intervention-related variables. A 39-item SC adapted from one by the World Health Organization was used. The primary endpoint was the occurrence of any AE, including death, within 30 days of the operation. Twenty-three types of AEs were analysed.
RESULTS: Following implementation of the checklist, the rate of AEs per 100 patients decreased from 31.5% to 26.5% (P = 0.39), the rate of infectious AEs decreased from 13.9 to 9.6 (P = 0.037) and non-infectious AEs decreased from 17.5 to 16.8 (P = 0.82). For non-elective patients, total AEs decreased from 60.4 to 37.0 (P = 0.017). The proportion of patients with one or more AE decreased from 18.1% to 16.2% (P = 0.35), and the death rate at 30 days decreased from 1.5% to 0.9% (P = 0.35).
CONCLUSION: The overall AE rate did not decrease significantly between the two periods. However, the rate of infectious AEs and overall AEs in patients with non-elective admissions had statistically significant reductions. Further research is needed to determine how and in which patients SC introduction can work successfully.

Editorial comment: what this article tells us

This article describes the work needed to implement the Safe Surgery Checklist in a big hospital. It confirms how difficult it is to document a convincing effect on the rate of adverse events and mortality after implementing the checklist in a modern hospital where the quality level is already high and where there are many confounding factors.

Surgical interventions have been used in health care for over a hundred years.1 An estimated 234 million operations are performed every year.2,3 However, surgical interventions are not completely risk-free. The mortality rates for surgical patients in developed countries range from 0.4% to 0.8%, with an adverse event (AE) rate of between 3% and 17%.3,4

In Spain, the incidence of patients with AEs directly related to hospital care is 8.4%.5 Surgical specialities had the highest number of AEs, particularly vascular surgery (16.1%), whereas medical specialities presented with the lowest rates (3.6%).5 Additionally, several studies3,4 have suggested that at least half of surgical AEs may be preventable.

In 2008, the World Health Organization (WHO) published a set of recommendations for improving the safety of surgical patients. These recommendations included a surgical checklist (SC) containing 19 items to be checked in the operating room (OR) at three different times.6

Since the publication of the WHO checklist, more than 1790 health-care centres worldwide have implemented this SC.*

As is the case for any health-care-related technology, and despite being based on a valid theoretical model, the actual utility of the SC must be proved on the basis of prevented morbidity and mortality, with consideration of the characteristics of the organisations in which it is implemented. Despite the increasing use of SCs and hospitals' efforts to implement them,7,8 very few studies to date have analysed their effectiveness in terms of AEs and mortality reductions in patients undergoing surgery.1,9–14

The main aim of this study was to determine whether the use of SCs reduced AEs and 30-day mortality rates in patients undergoing surgical intervention at a tertiary teaching hospital.

Methods

Study design

This study was conducted at the Hospital de Navarra, a tertiary teaching hospital in northern Spain with 500 beds and 10 ORs (9 for planned operations and 1 dedicated to emergency and urgent surgeries) that performs approximately 5300 surgical interventions a year.

The Navarra Ethics Research Committee (Pyto 55/2014) approved this study (Comité Ético De Investigación Clínica, Pabellón de Docencia. Irunlarrea, 3. 31008 Pamplona. Spain, Protocol number Pyto 55/2014, 30 April 2014).

A pre- and post-intervention study with two retrospective cohorts of surgical patients was conducted. The cohorts corresponded to two samples of patients who underwent surgery in 2008, before the introduction of the checklist, and in 2010, after the checklist was implemented throughout the hospital. A multi-stage method was used for sample extraction. An initial stratification was performed to ensure that the samples maintained the proportion of patients treated by each surgical department. Once the number of patients corresponding to each department was determined, the sample was extracted by simple randomisation.

The reference population comprised 10,121 patients who underwent surgery in 2008 and 2010. The sample included 1602 patients (801 for each study period). Given the variability of the AE rates before and after the introduction of the checklist among the sites in the seminal article published by Haynes et al.,1 the sample size was calculated using the average AE rate. The parameters used to calculate the sample size were a pre-intervention AE rate of 11%, a post-intervention rate of 7% and respective alpha and beta values of 0.05 and 0.8.

Inclusion criteria

All adults with a minimum hospital stay of 24 h who underwent surgery were included in the study.

Data collection

The information required for the patients who met the inclusion criteria was obtained from the clinical and administrative database (CADB), the haemotherapy database and medical records. Qualified personnel who were trained to perform this task extracted the information according to a standardised procedure. To avoid bias during data collection, the data managers were unaware of the working hypothesis.

Table 1 displays the variables used to describe the general characteristics of the patients. Overall, 40 comorbidities, 13 pre-operative analytical determinations, 23 surgical AEs in the 30 days following the intervention and intra-operative mortality and mortality at 30 days post-intervention were analysed following the definitions established by the American College of Surgeons National Quality Improvement Program (ACS NSQIP).15

Table 1

General characteristics of the patients studied.

	Pre-implementation	Post-implementation
General characteristics	(n = 801)	(n = 801)
Female, no. (%)	312 (38.9)	348 (43.4)
Age, year	61.2 ± 18,0	60.4 ± 18.4
Non-elective hospital admission, no. (%)	220 (27.5)	216 (27.0)
Non-elective intervention, no. (%)	138 (17.2)	147 (18.4)
Length of preop stay	1.65 (3.3)	1.65 (3.5)
Diagnoses (CADB)	4.3 (3.6)	3.9 (3.1)
Procedures (CADB)	2.3 (2.0)	2.2 (2.1)
Comorbidities, risk factor or analytical alterations	2.2 (1.76)	2.2 (1.60)
Diagnoses related group weight	2.7 (4.2)	2.7 (3.6)
Duration of the surgical intervention (min)	81.8 (71.3)	90.1 (77.8)
Anaesthesia type, no. (%)†
General	430 (53.7)	520 (64.9)
Spinal	279 (34.8)	199 (24.8)
Epidural	3 (0.4)	5 (0.6)
Others	89 (11.1)	77 (9.6)
Surgical wound classification, no. (%)
Clean	367 (45.8)	341 (42.6)
Clean-contaminated	325 (40.6)	359 (44.8)
Contaminated	35 (4.4)	29 (3.6)
Dirty	74 (9.2)	72 (9.0)
ASA classification, no. (%)
1	198 (26.0)	195 (25.4)
2	293 (38.4)	292 (38.1)
3	203 (26.6)	222 (28.9)
4	69 (9.0)	58 (7.6)
Surgical specialty, no. (%)
Ophthalmology	4 (0.5)	2 (0.2)
Otorhinolaryngology	20 (2.5)	19 (2.4)
General surgery	259 (32.3)	249 (31.1)
Traumatology	217 (27.1)	208 (26.0)
Urology	89 (11.1)	98 (12.2)
Neurosurgery	55 (6.9)	56 (7.0)
Cardiac surgery	65 (8.1)	70 (8.7)
Thoracic surgery	31 (3.9)	32 (4.0)
Vascular surgery	61 (7.6)	67 (8.4)

P < 0.05 in the comparison of pre-implementation vs. post-implementation phase. ASA, American Society of Anesthesiologists Physical Status; CADB, clinical and administrative database.

Intervention

In 2009, before the checklist was introduced, a pilot study was conducted in the Orthopaedics-Traumatology and Vascular Surgery Departments to evaluate the SC completion rate in 237 surgical interventions. The analysis of this pilot study led to some corrective actions, including a change in where pre-operative prophylactic antibiotherapy was administered (from the ward to the OR). Furthermore, 22 safe-surgery training sessions were held to raise awareness of the importance of completing the SC. A multidisciplinary team that included the medical heads of the Surgical and Anaesthesiology Departments, the head of the Surgical Nursing Department and staff members from the Preventive Medicine and Quality Control Department participated as lecturers in these sessions. The training sessions were evaluated using an anonymous questionnaire (n = 191, response rate 65%). Respondents gave the sessions a mean score of 7.2 (scale: 0–10) for both their overall satisfaction with them and the applicability of the material to their daily work.

In January 2010, the use of the SC was extended to all patients undergoing surgery at our hospital. The 39-item SC used (Table 2) is an adaptation of the SC issued by the WHO and was prepared by the Safe Surgery Committee at our hospital. The different elements were checked by the appropriate health-care professionals, who signed the corresponding section after completing the verification process. A paper format was used for the SC, and the Preventive Medicine and Quality Control Department assessed its completion.

Table 2

Checklist items.

Before the patient goes to the OR
Nurse reviews	The patient is informed of the question reiteration
	Identity
	Antibiotic prophylaxis
	Medical record including drug delivery sheet
	Cross-matching blood tests

ECG, electrocardiography; OR, operating room.

Outcomes

The primary end point was the occurrence of any AE, including death, within 30 days after the operation, even if the patient was no longer in the hospital at the time (Table 3). The definitions of AEs established by the ACS NSQIP were used.15

Table 3

Adverse events, number and percentage of adverse events per 100 patients studied before and after introduction of checklist.

Adverse events	Before		After		P value
	(n = 801)		(n = 801)
	No.	% (CI)	No.	% (CI)
Wound
Wound infection	57	7.1 (5.3 to 8.9)	48	6.0 (4.3 to 7.6)	0.419
Wound disruption	38	4.7 (3.3 to 6.2)	52	6.5 (4.8 to 8.2)	0.158
Lung
Pneumonia	22	2.8 (1.6 to 3.9)	11	1.4 (0.6 to 2.2)	0.077
Unplanned intubation	8	1.0 (0.3 to 1.7)	5	0.6 (0.1 to 1.2)	0.579
Pulmonary embolism	1	0.1 (−0.1 to 0.4)	0		1
Ventilator > 48 h	18	2.2 (1.2 to 3.3)	10	1.2 (0.5 to 2.0)	0.181
Kidney
Renal insufficiency	4	0.05 (0.0 to 0.1)	1	0.01 (−0.1 to 0.4)	0.374
Urinary tract infection	17	2.1 (1.1 to 3.1)	14	1.7 (0.8 to 2.7)	0.717
Nervous system
Stroke/CVA	3	0.04 (0.0 to 0.8)	3	0.04 (0.0 to 0.8)	1
Coma > 24 h	7	0.09 (0.2 to 0.15)	3	0.4 (0.0 to 0.8)	0.342
Peripheral nerve injury	0		3	0.4 (0.0 to 0.8)	0.250
Heart and vessels
Cardiac arrest requiring CPR	1	0.1 (− 0.1 to 0.4)	5	0.6 (0.1 to 1.2)	0.218
Myocardial infarction	0		1	0.1 (−0.1 to 0.4)	0.317
Bleeding transfusions > 4 units	12	1.5 (0.6 to 2.3)	14	1.7 (0.8 to 2.7)	0.844
Thrombophlebitis	4	0.5 (0.0 to 1.0)	3	0.4 (0.0 to 0.8)	1
Sepsis
Sepsis	16	2.0 (1.0 to 3.0)	4	0.5 (0.0 to 1.0)	0.011
Others
Reintervention	44	5.5 (3.9 to 7.1)	34	4.4 (0.2 to 1.5)	0.356

CI, confidence interval; CPR, cardiopulmonary resuscitation; CVA, cerebrovascular accident.

Information regarding retained foreign objects after surgery and wrong-site procedures was also collected. The AEs were classified into two groups: infectious and non-infectious. Surgical site infections, pneumonia, urinary tract infections and sepsis were considered infectious AEs, whereas the remainders were considered non-infectious AEs.

The proportion of patients with one or more complications and the number of AEs per 100 patients were reported.

For the 2010 cohort, the SC completion rate was evaluated in a consecutive sample of 452 interventions. Because written proof of how the different sections of the checklist evaluated was available, it was possible to evaluate both the percentage of surgical interventions with a checklist and the degree of completion of the checklist.

Confounders

The association between the checklist's implementation and AEs at 30 days, including mortality, may have been influenced by temporal changes in the baseline characteristics of the patient population. We considered gender, age, emergency status of surgery, type of anaesthesia, American Society of Anesthesiologists physical status and a set of comorbidities used by the ACS NSQIP15 as potential confounding factors in the analysis.

The values established by the Medical Lab Observatory were considered to be critical.16

Statistical analysis

Statistical analyses were performed using SPSS release 20 for Windows (IBM SPSS, Chicago, IL, USA). Pearson's χ2 tests or Student's t-tests were used to compare the baseline characteristics of patients before and after the checklist was implemented. Levene's test was used to check the homogeneity of the variances. A multivariate logistic regression analysis was used to adjust the association between SC implementation and outcomes for confounding factors. A P value of less than 0.05 was considered to indicate statistical significance.

Results

Proportion of patients with one or more AE

The proportion of patients with one or more AE was 18.1% during the baseline period and 16.2% after implementation of the checklist (P > 0.35) (Table 4). The odds of AEs were 14% higher for the patients in the baseline period than the period after the checklist was introduced, and the risk fraction due to the checklist was 12.3%. None of these values were statistically significant.

Table 4

Outcomes: number and percentage of patients with one or more adverse events or exitus before and after introduction of checklist.

	Before		After		P value	Percentage difference after/before	OR	Etiologic fraction in exposed (EFe/OR)
	No.	% (CI)	No.	% (CI)
Patients with adverse events
Total	145	18.1 (15.5–20.8)	130	16.2 (13.7–18.8)	0.35	−10.0.5	1.14 (0.9–1.8)	12.3 (−13.7 to 35.4)
Infectious events	88	11.0 (8.8–13.2)	71	8.9 (6.9–10.8)	0.18	−19.4	1.26 (0.9–1.8)	21.2 (−9.5 to 43.3)
Non-infectious events	93	11.6 (9.4–13.8)	93	11.6 (9.4–13.8)	1	0
Non-elective admission	71	31.8 (25.6–37.9)	44	20.4 (14.9–25.8)	0.006	−35.8	1.86 (1.2–2.9)	46.3 (17.0 to 65.3)
Elective admission	75	12.9 (10.2–15.6)	86	14.7 (11.8–17.6)	0.42	13.9	0.86 (0.6–1.2)	14.0 (20.0 to 38.4)
Exitus
Exitus day of surgery	0		0
Exitus within 30 days of surgery	12	1.5 (0.7–2.3)	7	0.9 (0.2–1.5)	0.356	−40.0	1.72 (0.7–4.4)	42.3 (−47.9 to 77.3)

CI, confidence interval; OR, operating room.

Although the proportion of patients with infectious AEs decreased from 11.0% in the baseline period to 8.9% after the checklist was implemented (P = 0.18), the proportion of patients with a non-infectious AE (11.6%) remained stable.

A total of 52 patients (6.5%) experienced more than one AE at baseline, whereas 48 (5.9%) did so after the checklist was implemented (P = 0.756).

The biggest decrease in the proportion of patients with AEs was observed for patients who were admitted on a non-elective basis (P = 0.006) (Table 4).

Number of AEs per 100 patients

The number of AEs per 100 patients was 31.5 at baseline and 26.5 after implementation of the checklist (P = 0.39) (Table 5).

Table 5

Outcomes: number and percentage of adverse events per 100 patients studied before and after introduction of checklist.

	Before		After		Percentage difference before/after	P value	Difference
	No.	% (CI)	No.	% (CI)			No.	% (CI)
Total	252	31.5 (25.3–37.6)	212	26.5 (21.5–31.5)	−15.9	0.39	40	4.9 (−2.9 to 12.8)
Infectious	112	13.9 (10.9–17.0)	77	9.6 (7.4–11.9)	−30.9	0.037	35	4.3 (0.6 to 8.1)
Non-infectious	140	17.5 (13.4–21.5)	135	16.8 (13.2–20.5)	−4.0	0.82	5	0.6 (−4.8 to 6.1)
Non-elective admission	133	60.4 (45.2–75.6)	80	37.0 (25.2–48.8)	−38.7	0.017	53	23.4 (4.2 to 42.6)
Elective admission	119	20.5 (14.5–26.4)	132	22.6 (17.3–27.8)	10.2	0.608	13	− 2.0 (−10.0 to 5.9)

CI, confidence interval.

Although the rate of infectious AEs dropped from 13.9 at baseline to 9.6 in the post-implementation period (P = 0.037), there was no significant change in the rate for non-infectious AEs.

A more detailed analysis of the reported AEs showed that only the rate of sepsis decreased significantly between the two periods (P = 0.011) (Table 3).

For non-elective patients, the AE rate was 60.4 at baseline and 37.0 after checklist implementation (P = 0.017). There were no significant changes in AE rates between the two periods of time for electively admitted patients.

Mortality

No patient died on the day of the intervention in either of the periods studied. Mortality after 30 days decreased from 1.5% in the baseline period to 0.9% after implementation of the checklist (P = 0.356) (Table 4).

Patient characteristics

The general patient characteristics did not differ between the pre- and post-implementation periods (Table 1) except for the type of anaesthesia used.

Only alcohol consumption (12.8% vs. 8.74%) and cerebrovascular disease with no neurological involvement (4.0% vs. 2.12%) decreased significantly after the SC was. Despite these differences, neither of these variables was related to a higher risk of suffering an AE in the multivariate logistic regression analysis.

Checklist verification

The percentage of patients who underwent a surgical intervention and had at least one SC item completed was 87% in 2010. For these patients 88% of the items were verified.

Discussion

AE

After the introduction of the SC, the percentage of patients with one or more AEs dropped from 18.1% to 16.2%, the post-operative AE rate per 100 patients dropped from 31.5 to 26.5, and the death rate went from 1.5% to 0.9%. These reductions did not reach statistical significance.

The results obtained differ from some articles published in the literature to date, which have demonstrated the effectiveness of the SC in preventing AEs and/or mortality in patients in a hospital setting.11

While the AE rate fell by 16% in our study, the AE rates fell by 36–65%1,12,14,17 in other hospitals, which are all statistically significant changes.

In contrast, mandatory adoption of SCs at hospitals in Ontario, Canada, did not produce significantly better outcomes.18

However, our study showed that after checklist implementation, the percentage of patients with non-elective admissions who experienced AEs fell by 39% and the AE rate fell by 36% (P < 0.05). The decrease in AEs in this specific patient population was also observed by other authors.13 These data support the importance of the SC in non-elective admissions, suggesting that the checklist effect might not be the same across patients or procedures with different risks.

Mortality

The mortality rates reported in previous studies showed statistically significant reductions after checklists were implemented.1,12,13,19 The mortality decrease in our study was very similar to the reported studies (40%), although the difference was not statistically significant, which is most likely due to a sample size effect. Bliss et al.17 found no statistically significant differences after implementation of an SC.

Baseline AE rates

The AE rates observed in our study are higher than those obtained in other studies,1,12–14,17 most likely due to the mix of patient cases (the hospital has Cardiothoracic, Vascular, Hepatobiliary, and Neurosurgery Departments but no Gynaecology/Obstetrics or Paediatric ones). Patients at tertiary teaching hospitals generally have a larger number of coexisting conditions and undergo more extensive procedures, increasing the likelihood of complications. The mix of cases could have an effect on the moderate reduction in AEs after the SC was introduced in our hospital.

Similarly, we analysed the 30-day AE and mortality rates, even for patients who had been discharged. The values reported would have been markedly lower if the AE and mortality rates had been studied only during hospital stays. In our study, 42% of the AEs occurred after patient discharge.

Methodology

The AE and comorbidity classification systems used in our study are the same as in the initial studies designed to measure SC effectiveness.1,13,17

Type of checklist and verification system

Our SC implementation strategies and the SC itself differ from those published previously, which could also help explain the different results obtained. For example, our SC has 39 items vs. the 19 on WHO's SC. Although some items on our SC can only be completed with the participation of various members of the surgical team, each professional involved in the surgical process was responsible for each item's verification. This requirement is a marked difference with respect to the SC proposed by the WHO, which recommends verification as a team. One of the advantages of our SC is that verification begins before the patient goes into the OR, thereby creating extra safety filters. Moreover, the SC used was the only one for which consensus was built among all of the professionals involved.

Implementation of the SC was highly recommended in our hospital, but not mandatory. There has recently been some controversy over the advantages of regulations to ensure compliance.18,20,21

Source of information

The fact that the information was collected retrospectively and at different periods of time could affect also the quality of the information collected. The source of information could also influence the results. We used clinical records, but other studies18 have used the CADB.

This study was conducted using data from 2008 and 2010. Data from 2009 were excluded because a pilot test was being performed in two surgical departments to study how to best introduce the SC throughout the hospital. During the pilot test, it was deemed impossible to comply with the SC without changing certain processes at the hospital. For example, in 2010, antibiotic prophylaxis was administered in the OR to ensure timely administration of it. These changes were considered to be part of the SC's implementation. Indeed, some authors12 have shown that the use of an SC optimises proper timing before an incision is made. However, initiatives involving more than one component make it more difficult to evaluate the SC's effectiveness. The 22 training sessions could have also influenced attitudes toward patient safety even before the intervention period began.

Study limitations

Design

A pre- and post-intervention study has some obvious limitations. However, the design of another type of study in which cases (patients with SCs) and controls (patients with no SCs) are assessed simultaneously would also present potential biases. For example, the introduction of an SC in some departments but not others would make a comparison of the results more difficult. The fact that some personnel (for example, anaesthesiologists) would treat cases and controls at the same time means that they could apply the SC to controls unconsciously.

During the period analysed, other quality-related factors might have been introduced, so it is not possible to conclude that the moderate improvement obtained was due exclusively to the SC. Furthermore, despite using a 39-item SC, there is no direct causal relationship between the SC and each of the AEs studied.

Confounders

Although all possible confounding factors have been considered by studying factors such as comorbidities and analytical alterations, there may nevertheless be some residual confounding factors that could explain the results obtained.

Sample size

The sample size was calculated based on the study performed by Haynes et al.1 However, the starting AE rate, and especially the lower than expected decrease after the SC was implemented, could have been related to the results not reaching statistical significance. The study may be underpowered for detecting clinically significant changes in overall AEs.

SC completion

The completion rate for the SC is another relevant factor. Indeed, to be effective, the SC completion rate must clearly exceed 80%.12 Other authors19 have demonstrated the effectiveness of an SC in reducing mortality only in patients for whom all of the SC items are verified, with no statistically significant differences observed for patients with partially completed SCs.

SC completion was measured in 2010 by analysing an SC sample. In contrast to other studies, there were no direct observers.1 The completion rates observed in our study (87% of interventions with checklists and 88% of items verified) did not differ markedly from those reported by authors such as de Vries et al.12 (80%), although the values reported in other studies ranged from 97.26%17 to 39%19 for patients with a completed SC.

The fact that the completion rate in our study was less than 100% could lead to an underestimation of the SC effect.

The implementation of an SC in any hospital is a major challenge. Indeed, numerous barriers to the effective implementation of a checklist have been identified.22 Emphasis has also been placed on the keys to success, such as the importance of leadership, adaptation of the SC to the local setting and the participation of professionals.23 In our hospital, health-care professionals and the management team led the project. Moreover, the professionals' perceptions regarding the utility of the SC and the main barriers to the SC implementation process have been evaluated. The health-care professionals considered the SC to be moderately useful (average 6.6; 1–10 scale), and 11.6% noted that actual errors had been avoided due to the SC.24

The impact of the SC can also be measured in terms of the health-care professionals' patient safety cultural change.25–27 Our study has concentrated exclusively on measuring final patient health outcomes. However, divulging the results is part of the strategy to improve the safety culture of our organisation. As a matter of fact, Bosk et al.28 stated ‘what a simple checklist can achieve is; on its own not much’. Safer care is achieved when organisations ‘summarise and simplify what to do; measure and provide feedback on outcomes; and improve culture by building expectations of performance standards into work processes’.28

In conclusion, our study showed that after the introduction of an SC, the proportion of patients with one or more AE, the number of AEs per 100 patients and the death rate decreased (11%, 16% and 40%, respectively), but these reductions did not reach statistical significance. Nonetheless, the rate of infectious AEs and the overall AEs in patients with non-elective procedures had statistically significant reductions. Further research is needed to determine the most effective way of introducing the SC and in which patients the SC can produce the highest reductions in post-operative AEs.