Compliance with the World Health Organization's surgical safety checklist and related postoperative outcomes: a nationwide survey among 172 health facilities in Ethiopia.

BACKGROUND: Ministry of Health (MOH) of Ethiopia adopted World Health Organization's evidence-proven surgical safety checklist (SSC) to reduce the occurrence of surgical complications, i.e., death, disability and prolong hospitalization. MOH commissioned this evaluation to learn about SSC completeness and compliance, and its effect on magnitude of surgical complications.
METHODS: Health institution-based cross-sectional study with retrospective surgical chart audit was used to evaluate SSC utilization in 172 public and private health facilities in Ethiopia, December 2020-May 2021. A total of 1720 major emergency and elective surgeries in 172 (140 public and 32 private) facilities were recruited for chart review by an experienced team of surgical clinicians. A pre-tested tool was used to abstract data from patient charts and national database. Analyzed descriptive, univariable and bivariable data using Stata version-15 statistical software.
RESULTS: In 172 public and private health facilities across Ethiopia, 1603 of 1720 (93.2%) patient charts were audited; representations of public and private facilities were 81.4% (n = 140) and 18.6% (n = 32), respectively. Of surgeries that utilized SSC (67.6%, 1083 of 1603), the proportion of SSC that were filled completely and correctly were 60.8% (659 of 1083). Surgeries compliant to SSC guide achieved a statistically significant reduction in perioperative mortality (P = 0.002) and anesthesia adverse events (P = 0.005), but not in Surgical Site Infection (P = 0.086). Non-compliant surgeries neither utilized SSC nor completed the SSC correctly, 58.9% (944 of 1603).
CONCLUSIONS: Surgeries that adhered to the SSC achieved a statistically significant reduction in perioperative complications, including mortality. Disappointingly, a significant number of surgeries (58.9%) failed to adhere to SSC, a missed opportunity for reducing complications.

Introduction

Globally, over 200 million surgeries are performed annually [1]. However, the growing number of surgical complications remain a public concern [2]. In Africa, the magnitude of perioperative mortality rate is high, 3.4–7% [3, 4]. Anderson et al. (2013) showed that adverse events occurred in 14.4% (IQR,12.5–20.1%) of 16,424 surgeries, 7.3% were surgical site infection [5]. In Ethiopia, a pooled prevalence of SSI estimated in a meta-analysis that involved 13,136 surgeries was 12.3% (95%CI:10.19–14.42) [6].

To ensure safety, Ministry of Health (MOH) of Ethiopia integrated the World Health Organization’s surgical safety checklist (SSC) into its surgical care strategy [7, 8]. White et al. (2021) showed the benefits of SSC in reducing surgical complications; mortality by 23% (RR 0.77; 95%CI 0.67–0.89) and risk of infections by 53% (RR 0.47; 95%CI 0.40–0.55) [9]. Another longitudinal study of SSC impact on clinical endpoints in 3733 surgeries found a significant reduction in perioperative death from baseline of 1.5 to 0.8% (P = 0.003), and other complications from baseline 11.0 to 7.0% (P < 0.001) [10, 11].

However, local evidence on SSC utilization and impact on surgical complications is lacking [12]. Hence, MOH commissioned this evaluation to generate evidence to shape Ethiopia’s surgical and anesthesia care program.

Methods

In 2016, MOH integrated SSC strategies into national surgical care program, and tracked SSC utilization through a district health information software-2 (DHIS2) in public facilities.

An institution-based cross-sectional study was used to retrospectively evaluate SSC utilization by reviewing surgical charts in 172 public and private health facilities in Ethiopia, December 2020–May 2021. Evaluation facilities were enrolled using a multi-stage stratified random sampling technique. A single population proportion formula was used to determine the minimum sample size; (n = z2pq/e2/1 + (z2pq/e2)*N) for a finite population with a 0.5 population proportions, 5% margin of error and 95% level of confidence. Of the total study sample of 203 (163 public and 40 private) facilities, 163 public surgical care facilities were selected from MOH’s health facilities registry (26 tertiary care specialized referral hospitals, 75 secondary care general hospitals, and 181 primary care facilities). The sample size for each stratum (level of care) was distributed using a proportion-to-size allocation method, size being the number of facilities in a specific stratum, i.e. specialized hospitals (nr = 15), general hospitals (ng = 43) and primary facilities (np = 105). Similarly, 88.9% (40 out of 45) private surgical care facilities were enrolled.

At facility-level, ten random charts were selected from surgical logbooks. All major surgeries performed within 90 days of evaluation were included while those performed beyond 90 days and minor surgeries were excluded. A total of 1720 surgical patients’ charts were audited by an experienced team of surgical clinician. A pre-tested data abstraction tool was used to collect information on SSC completeness, compliance, surgical complications (perioperative mortality, SSI, and anesthesia adverse event), and SSC completeness rate self-reported by facilities.

Statistical analysis

Data were cleaned on weekly basis, checked for completeness, correctness and consistency, and finally entered into a Redcap study database. Data exported to Stata version-15 software for computing descriptive statistics and compare categorical variables by chi-square test. Statistically significance declared at P < 0.05.

Ethical consideration

Ethical clearance was secured from the Armauer Hansen Research Institute (AHRI). MOH issued a letter of support to health bureaus and facilities.

Operational definition

Major surgery

An institution-based invasive intervention involving the incision, excision, manipulation, or suturing of tissue, usually requiring regional or general anesthesia or sedation.

Surgical safety checklist completeness

Assesses whether all entries of the checklist are completed and filled correctly as indicated in the SSC guide. Numerator: Count of surgical procedures that utilized SSC, and filled completely and correctly. Denominator: Total number of major surgeries performed within 90 days of chart audit.

Non-compliance rate

Assesses deviations from surgical safety guide. Numerator: Count of major surgeries that didn’t utilize the checklist and surgeries whose SSC was neither complete nor filled correctly. Denominator: Total number of major surgeries performed within 90 days of chart audit.

Results

Out of the sample 203 health facilities, 172 (84.7% response rate) participated in this evaluation; public and private health facilities accounted for 81.4% (n = 140) and 18.6% (n = 32) of the evaluation facilities, respectively. Of the total surgical patient charts audited, 93.2% (1603 of 1720), the proportion of surgeries that utilized SSC were 67.6% (1083 of 1603) (Fig. 1). SSC utilization rate was highest in public specialized hospitals (85%) and lowest in private health facilities (23.1%). An aggregate SSC completeness rate was 60.8% (659 of 1083) while the remaining (39.2%) were neither complete nor filled correctly (Table 1). SSC completeness rate self-reported to the DHIS2 by facilities was 81% while completeness rate computed by retrospective chart audit was 60.8%.

Fig. 1

Surgical Safety Checklist utilization during emergency and elective surgeries in public and private health facilities in Ethiopia, December 2020 to May 2021

Table 1

Surgical safety checklist completeness during an emergency and elective surgeries in public and private health facilities in Ethiopia, December 2020 to May 2021

Facilities evaluated by level of Health care		Number of surgical patient chart		Surgeries that utilized SSC		Completeness or correctness of SSC
Level of carea	Facilities. N(%)	Chart selected	Charts reviewed	Yes, N (%)	No, N (%)	Yes, N (%)	No, N (%)
Public Specialized Hospitals	16 (9.3%)	160	160 (100%)	136 (85%)	24 (15%)	80 (58.8%)	56 (41.2%)
Public General Hospitals	38 (22.1%)	380	370 (97.4%)	275 (74.3%)	95 (25.7%)	159 (57.8%)	116 (42.2%)
Public Primary Hospitals	86 (50%)	860	800 (93%)	609 (76.1%)	191 (23.9%)	382 (62.7%)	227 (37.3%)
Private Hospitals	32 (18.6%)	320	273 (85.3%)	63 (23.1%)	210 (76.9%)	38 (60.3%)	25 (39.7%)
Total	172 (100%)	1720	1603 (93.2%)	1083 (67.6%)	520 (32.4%)	659 (60.8%)	424 (39.2%)
1720 (100%)			1603 (100%)		1083 (100%)

aSpecialized hospital: tertiary level of care that serves 3.5 to 5.0 million people. General hospital: secondary level of care that serves 1 to 1.5 million people. Primary hospital: primary level of care that serves 60,000–100,000 people with an average inpatient capacity of 35 beds and has direct referral linkage with nearby primary care units (health centers and health posts). Health center (a.k.a. operation room blocks): primary level of care facility that typically has the capacity for providing emergency obstetric delivery services, and furnished by Ministry of Health to provide additional emergency and essential surgical care (primarily the Bellwether surgical procedures, i.e., Cesarean section, laparotomy, and open fracture management. (Ministry of Health. Health Sector Transformation Plan (2015/16–2019/20). Federal Democratic Republic of Ethiopia Ministry of Health. Addis Ababa; 2015)

The proportion of surgeries that neither complete nor filled the SSC correctly were 58.9% (944 of 1603). Non-compliance to SSC was most prevalent in public primary and private hospitals, 23.6 and 14.7%, respectively (Table 2). SSC adherence was achieved a significantly reduction in perioperative mortality (P = 0.002), and anesthesia adverse events (P = 0.005).

Table 2

Non-compliance to surgical safety checklist guide during major emergency and elective surgeries in public and private health facilities in Ethiopia, December 2020 to May 2021

Level of Health care	Number of surgical charts reviewed N(%)	Surgeries non-compliant to SSC guidea
		Didn’t utilize SSC. N(%)	Incomplete or incorrect SSC	Non-compliant, total
Public Specialized Hospitals	160 (100%)	24 (15%)	56 (41.2%)	80 (50.0%)
Public General Hospitals	370 (97.4%)	95 (25.7%)	116 (42.2%)	211 (13.2%)
Public Primary Hospitals	800 (93%)	191 (23.9%)	227 (37.3%)	418 (26.1%)
Private Hospitals	273 (85.3%)	210 (76.9%)	25 (39.7%)	235 (14.7%)
Total	1603 (93.2%)	520 (32.4% of 1603)	424 (39.2% of 1083)	944 (58.9%)

aNote the difference between the denominators for ‘Didn’t utilize SSC’ (32.4% of 1603 charts reviewed) and the denominator for ‘Incomplete or incorrect SSC’ (39.2% of 1083 SSC used)

Though the reasons for non-compliance varied by level of care, the most frequently mentioned reasons were lack of knowledge, shortage of time, and staff unwillingness. Staff unwillingness was the commonest reason in public health care units (41%) while shortage of time was the commonest among public specialized hospitals, 46.1% (Table 3).

Table 3

Reasons for non-compliant use of surgical safety checklist in public and private health facilities in Ethiopia in a 90-day interval of the study period, December 2020 to May 2021

Reasons for non-compliancea	Level of health care facilities
	Public Specialized Hospitals	Public Generalized Hospitals	Public Primary Hospitals	Private Hospitals
Shortage of time	6 (46.1%)	9 (31.0%)	9 (14.7%)	4 (14.3%)
Staff unwillingness	2 (15.4%)	2 (6.9%)	25 (41%)	7 (25%)
Lack of knowledge	3 (23.1%)	5 (17.2%)	13 (21.3%)	12 (42.9%)
Resource shortage	0	10 (34.5%)	9 (14.7%)	3 (10.7%)
Other	2 (15.4%)	3 (10.3%)	5 (8.2%)	2 (7.1%)

aMultiple reasons options were allowed

Adherence to SSC was associated with a statistically significantly reduction in perioperative mortality (P = 0.002), and anesthesia adverse events (P = 0.005). However, effect on SSI was not statistically significant (P = 0.086) (Tables 4, 5 and 6).

Table 4

Association between surgical safety checklist utilization and occurrence of surgical site infections in public and private health facilities in Ethiopia, December 2020 to May 2021

Key Performance Indicator (KPI)	Surgical Safety Checklist Utilization
		yes	no	P value
Surgical site infections	Yes	45 (27.3%)	15 (9.1%)	p = 0.086
	No	90 (54.6%)	15 (9.1%)

Table 5

Association between surgical safety checklist utilization and occurrence of perioperative mortality in public and private health facilities in Ethiopia, December 2020 to May 2021

Key Performance Indicator (KPI)	Surgical Safety Checklist Utilization
		yes	no	P value
Death (perioperative mortality)	Yes	23 (13.9%)	13 (7.9%)	p = 0.002
	No	112 (67.9%)	17 (10.3%)

Table 6

Association between surgical safety checklist utilization and occurrence of Anesthesia Adverse Events in public and private health facilities in Ethiopia, December 2020 to May 2021

Key Performance Indicator (KPI)	Surgical Safety Checklist Utilization
		yes	no	P value
Anesthesia adverse events	Yes	17 (10.3%)	10 (6.1%)	p = 0.005
	No	118 (71.5%)	20 (12.1%)

Discussion

Surgeries that were compliant to SSC guide achieved a statistically significant reduction in surgical complication, primarily perioperative mortality (P = 0.002), and anesthesia adverse events (P = 0.005). Though over a third (67.6%) of the surgeries utilized safety checklists, only 659 (60.8%) checklists were filled completely and correctly. The non-compliance rate was unacceptably high, 58.9%.

The SSC completeness and compliance rate in Ethiopia was lower compared to findings in other studies. In Tanzania, Hellar et al. (2018) observed a higher SSC utilization rate, 68.8–99.4%; primary health care units (PHCU) performed significantly better (99.4%, p < 0.05) than higher-level hospitals [13]. An observational study by Schwendimann et al. (2019) evaluated adherence to SSC protocol in the University Hospital of Basel, Switzerland, and showed that the adherence rate for three domains of surgery ranged between 96 and 100% in timeout domain and sign-in domains, respectively, and 22% in sign-out. While the adherence rates for the timeout and sign-in was higher, the unavailability of key OR team members at sign-out time was the most common reason for low rate of completeness or adherence during the sign-out phase [14]. Though the SSC utilization rate in LMICs is lower, < 30%, compared to high-income countries (88–89%), in Ethiopia, SSC was integrated to national safe surgery strategy only recently [9]. In our evaluation, PHCUs scored higher SSC completeness rate (health centers, 67.1%; primary hospitals, 62.1%) compared to tertiary- and secondary care hospitals, 58.8% and 57.8%, respectively. This favors an argument that safety improvement in resource-constrained settings is feasible.

High non-compliance rate in private health facilities (14.7%) could be partly explained by the absence of surgical care programs and tracking of surgical performance metrics in private facilities. Private facilities could enhance the quality and safety of surgical care by adopting best safety practices [15]. Al-Qahtani et al. (2017) published experience of private hospitals who managed to increase SSC compliance rate (96.5%) through training and mentoring of staffs [16].

The discrepancy between SSC completeness rate in the national DHIS2 database (81%) and the retrospective audit (60.8%) may result from data quality issues or fidelity of the different assessment methods applied to compute completeness rates; compared to surveys, facility self-reports usually overestimate performance. Giles et al. (2016) depicted the impact of applying different measurement methods for estimating SSC completeness rate, 86% versus 27% through chart audit and direct observation, respectively [17].

The effect of SSC utilization on surgical complications was remarkable; those surgeries that adhered to SSC guide achieved a statistically significant reduction in surgical complications, perioperative mortality (P = 0.002) and anesthesia adverse events (P = 0.005), a finding that agrees with other evidences on clinical endpoints of adherent use of SSC [11].

The absence of statistically significant change in SSI rate (P = 0.086) could be attributed to several factors: weak SSI tracking systems, suboptimum adherence to infection prevention and control protocols, absence of post-discharge community SSI tracking, among other factors. The African Surgical Outcomes Study (ASOS) Group’s cohort of 11,422 surgeries in 25 African countries revealed an alarmingly high postoperative complications rates, 18·2% (95%CI 17·4–18·9]), and infection was the most common complication (10·2%) [18].

Interestingly, the most frequently mentioned reasons for non-compliance were surgical team’s lack of knowledge, unwillingness and shortage of time, and these reasons do vary by facility type; for instance, shortage of time was commonest in private and specialized hospitals, 36.8 and 50%, respectively. Though unwillingness and shortage of time was reported as one of the reasons for non-compliance, Tan et al. (2021) presented a favorable attitude towards SSC use from an inquiry of 846 surgical team in 138 hospitals in China; only 12.7% reported that the checklist ‘took a long time to complete’ and the majority (78.8%) reported that SSC was ‘easy to use’. In this study, OR staffs believed that SSC improved the surgical safety (90.4%), team communication (85.6%) and reduced errors (89.5%) [19]. Surgical team’s competency, attitude and communication on safety increases the likelihood of adherence and clinical outcomes [20-22]. Treadwell et al. (2014) described adherence and team communication as predictors of positive outcomes of SSC [23].

On the other hand, SSC could be adapted to standardize surgical practice among surgical team and minimize occurrence of surgical errors. Naqvi et al. (2022) showed that intraoperative practices of spinal surgeons in United Kingdom varied greatly, and 47.5% (29/61) of surgeons had been involved in wrong level spinal surgery [24].

One strength of this evaluation is that it is the first nation-wide assessment of surgical safety practices in public and private facilities in Ethiopia. Surgical safety practices were audited by an experienced team of clinicians who deeply understand the surgical care processes, and generated a quality data that could be generalizable to other surgical care settings. As this evaluation is owned by MOH, the evidence would be applicable to shape the national surgical program. Nonetheless, this evaluation didn’t use a prospective direct observation method to validate the use of SSC during live surgery, and the current retrospective evaluation may overestimate completeness and compliance to the use of checklist in real-time.

Conclusions

Surgeries that adhered to the SSC guide achieved a statistically significant reduction in surgical complications, including perioperative mortality. While the aggregate adherence to SSC in public and private health facilities in Ethiopia was 60.8%, a significant number of surgeries (39.2%) missed an opportunity for reducing surgical complications. An overestimation of SSC utilization by self-report misleads decision-making.

Factors influencing compliance (lack of knowledge on how to complete safety checklist, team’s unwillingness and perceived shortage of time) need to be addressed through targeted training and coaching. Further research is warranted to better understand factors influencing adherence.

Leadership oversight and integration of SSC into existing perioperative risk assessment may increase likelihood of acceptance by surgical teams [9, 25, 26]. The evidence would inform Ethiopia’s safe surgery strategy.

Additional file 1.