How Much and What Local Adaptation Is Acceptable? A Comparison of 24 Surgical Safety Checklists in Switzerland.

OBJECTIVES: In 2009, the World Health Organization (WHO) published the WHO Surgical Safety Checklist, and 3 years later, the Swiss Patient Safety Foundation adapted it for Switzerland. Several meta-analyses and systematic reviews showed ambiguous results on the effectiveness of surgical checklists. Most of them assume that the study checklists are almost identical, but in fact they are quite heterogeneous due to adaptations to local settings. This study aims to investigate the extent to which the checklists currently used in Switzerland differ and to discuss the consequences of local adaptations.
METHODS: For the analysis, 24 checklists used in 18 Swiss hospitals are analyzed. First, general checklist characteristics are examined. Second, the checklist items are compared with the checklist items of the WHO and the Swiss Patient Safety Foundation.
RESULTS: The checklists contain a median of 34.5 items (range, 15-76). Compared with the checklists of WHO and Patient Safety Switzerland, which contain 12 and 21 process checks and 10 and 9 conversation prompts, respectively, the study checklists contain a median of 15.5 process checks (range, 3-25) and a median of 4 conversation prompts (range, 0-10).
CONCLUSIONS: There are major differences between the study checklists and the reference checklists that raise doubts about the comparability of checklists. More resources must be invested in proper checklist adaptions and better guidance on how to adapt safety tools such as the surgical safety checklist needed to local conditions. In any case, details of the checklists used need to be clearly described in studies on checklist effectiveness.

In 2009, the World Health Organization (WHO) published the WHO Surgical Safety Checklist aiming to reduce errors and adverse events in surgery.[1] The checklist was widely adopted throughout the world, including Switzerland.[2] In 2012, the Swiss Patient Safety Foundation adapted the WHO checklist for Switzerland and published a checklist in the 3 national languages, French, German, and Italian.[3,4]

In recent years, several meta-analyses and systematic reviews were published regarding the effectiveness of surgical checklists, often showing ambiguous results.[5-9] Most reviews and meta-analyses are based on the assumption that the checklists used in the studies are approximately identical. In fact, the individual studies are difficult to compare because in some cases, only individual checklist sections are examined or the checklist sections are carried out at different time points,[10-14] and many adaptations of most checklists are made to fit structural or local circumstances.[15] Thus, the intervention “checklist” itself is heterogeneous.

The checklists from the WHO and Patient Safety Switzerland consist of 3 sections that should be applied at different time points: The Sign In (SI) should be completed before induction of anesthesia; the Team Time Out (TTO), before skin incision; and the Sign Out (SO), before the patient or surgeon leaves the operating theater (OT). Both institutions recommend to adapt the checklists to the local setting and specific needs of the hospitals[3,16]; that is hospitals are completely free to adjust their checklist in terms of content, structure, sequence of items, wording, and so on. There is consensus in the literature that adaptation to the local context is part of translating evidence-based research into practice and that some flexibility to modification is needed.[15,17-20] However, although some adaptations are necessary (e.g., translation into local language), others might decrease the effectiveness.[20]

Solsky and colleagues[15] analyzed 155 English-speaking checklists on the extent and nature of checklist modification, revealing that all of them were modified. They compared the checklists with the WHO checklist focusing on the modification of single items (e.g., key detail alteration and minor language change) and structural aspects (e.g., presence of organization’s name or logo and background color). Most of the checklists contained more lines of text and items but also removed a considerable number of items of the WHO checklist, including items that aim to promote the exchange of critical information and should not be removed according to the implementation guidelines of the WHO.[16]

Although the study by Solsky et al[15] focused on the question to what extent hospitals adopted the checklist, the present study extends the analysis also to aspects of checklist application, such as time and definition of roles. The checklists are compared with the WHO checklist and the checklist of the Swiss Patient Safety Foundation. In addition to the analysis of the checklists themselves, the corresponding hospital guidelines are investigated.

METHODS

Setting

Data were collected as part of a national program initiated by the Swiss Patient Safety Foundation, aiming to measure and improve compliance with the surgical safety checklist in Switzerland. Hospitals from all 3 language regions (French, German, Italian) participate voluntarily in the program. At the beginning of the program, the 19 participating hospitals were asked to provide their surgical safety checklist(s) and guidelines as well as the year of checklist implementation. Because 2 hospitals are part of the same network and use the same checklist, we included only one of them in the analysis. Fourteen of these 18 hospitals submitted one checklist each; the remaining 4 hospitals submitted 2 to 4 checklists, resulting in a final sample of 24 checklists. The 4 hospitals that provided 2 or more checklists use different checklists for different types of anesthesia (e.g., local versus general), different types of intervention (e.g., surgical versus diagnostic intervention), or different disciplines (e.g., pediatric versus general surgery).

Coding

For document analysis of checklists, we developed a coding system. First, a binary code set containing all items of the checklists of the WHO and the Swiss Patient Safety Foundation was created. Item codes that belong to the same checklist part were grouped into higher-level codes, that is, SI, TTO, and SO. Additional codes that refer to general checklist information such as patient identity sticker or time of application were created. Based on this code set, all items of the 24 checklists were coded. If an item could not be assigned to any of the existing codes, a new code was created. If a checklist contained an additional checklist part, a new code (for the item) and additionally a new higher-level code (for the checklist part) were created.

Guidelines were coded as extensive instructions if they included detailed information on checklist purpose and the application of single checklist items as well as specifications regarding checklist coordination, time, and leadership. If guidelines contained general information about purpose and area of application but no specific information on the checklist application, they were classified as general directive. All coding steps were performed twice through the same person to ensure correct assignment.

Categorization Into Process Checks and Conversation Prompts

In its implementation guide to the surgical checklist, the WHO distinguishes between so-called process checks and conversation prompts. According to the WHO, “process checks remind team members to verify, perform, and discuss critical safety steps” and “conversation prompts remind team members to share and discuss critical information about the patient, risks, and surgical plans so that they are prepared to work together more effectively as a unit.”[16](p102) There is much overlap between these 2 definitions; both item types aim to promote the discussion of critical information and safety steps within the OT team. The difference between the 2 item types seems to be that the former has its focus on verifying information and the latter has its focus on sharing information that may not be known to all team members. In this article, we define the 2 item types as follows:

Process checks: an aspect relevant to patient safety is actively controlled and verbally confirmed, and the relevant information is exchanged within the team. The check is performed by comparing at least one further source of information, for example, comparing the patient record with the patient statement. At least 2 persons are involved in a process check. The aim of process checks is that the most critical and important steps relevant for patient safety are systematically checked; in case of a discrepancy, a stop is made and only continued as soon as this has been clarified.

Conversation prompts: information on the planned procedure, critical patient information, and potential risks, as well as questions and concerns are shared and discussed within the OT team. The aim of conversation prompts is that the OT team has a shared mental model, which improves communication within the interprofessional team and increases patient safety.

If one of the checklists analyzed contained an item from the reference checklists, we assessed whether this was designed as a process check or a conversation prompt. It is difficult to make this decision based solely on the checklist. Whenever possible, the guidelines of the respective checklist have been used for categorization. Where no guidelines available or the category remained unclear, we looked for indicators on the checklist itself. If it, for example, was explicitly mentioned that a comparison would be made with the patient wristband, the patient file, or the patient himself, we categorized the item as a check. If an item consisted of an open question or even contained a field in which to enter the information discussed during checklist application, we categorized the item as a conversation prompt. The software ATLAS.ti 8 (ATLAS.ti Scientific Software Development GmbH, Berlin, Germany) was used for coding the checklists.

Analysis

After coding was complete, gained data and additional information on the hospitals (e.g., type and number of beds) were entered in Stata 16 to perform descriptive statistics (StataCorp, College Station, Texas). In the first step, various characteristics of the checklists were evaluated, for example, number of checklist sections, and items in total and per checklist section. In the second step, we compared every single checklist and the corresponding guideline with the 2 reference checklists (WHO and Swiss Patient Safety Foundation). Further analyses included specifications for checklist application such as timing and checklist coordination.

RESULTS

Characteristics of the Hospitals

The 18 hospitals comprised different hospital types including 12 general hospitals (67%), 4 speciality hospitals (22%), 1 university hospital (6%), and 1 outpatient surgery center (6%). A broad variety of specialities was covered (e.g., pediatrics and ophthalmology). The hospitals differed regarding number of beds ranging from 0 to 979 (median, 102.5) and number of operating rooms ranging from 3 to 37 (median, 5). Ten hospitals were from the German-speaking part (56%), 4 from the Italian-speaking part (22%), 3 from the French-speaking part (17%), and 1 (5%) from a bilingual region (German and French-speaking). Checklist experience ranged across the hospitals from 1 to 11 years (median, 6 years).

Overall Checklist Characteristics

Table 1 gives an overview of the general characteristics of the 24 analyzed checklists. Most of the checklists are filled out on paper (88%), whereas the remaining 3 checklists are available as laminated version. For one checklist, the SI section is completed electronically. Almost all checklists (96%) require signatures on the checklist itself or at designated places in the patient file, and 83% have a field for a patient identity sticker. For 50% of the checklists, a general directive exists; one-third of the checklists are accompanied by extensive instructions.

TABLE 1

Characteristics of the Study Checklists (n = 24)

Characteristics	No. Checklists (%)
Checklist type
Form to fill in	21 (87.50)
Laminated version	3 (12.50)
Checklists that require signature(s)	23 (95.83)
Checklists that require patient identity sticker	20 (83.33)
Checklist language
French	3 (12.50)
German	16 (66.67)
Italian	5 (20.83)
Checklist guidelines
Not available	4 (16.67)
General directive	12 (50.00)
Extensive instructions	8 (33.33)
No. checklist items in total (nWHO = 22; nSPS = 30)	Median, 34.5; range, 15–76
SI (nWHO = 7; nSPS = 11)	Median, 9; range, 5–21
TTO (nWHO = 10; nSPS = 14)	Median, 9; range, 0–21
SO (nWHO = 5; nSPS = 5)	Median, 5; range, 0–13
Additional checklist sections	Median, 8; range, 0–47

WHO, number of items in the WHO checklist; SPS, number of items in the checklist of the Swiss Patient Safety Foundation.

The checklists contain a median of 4 checklist sections (range, 2–6) and a median of 34.5 items in total (range, 15–76). Median item number is 9 for both SI (range, 5–21) and TTO (range, 0–21); one checklist does not contain a TTO section. For SO, the median item number is 5 (range, 0–13); 2 checklists do not have an SO section.

Frequency and Type of Checklist Items

Two items that are included in almost all checklists in the SI section are identity (96%) and known allergies (92%), whereas the 2 items pulse oximeter (21%) and assignment to the correct operating room (29%) are least frequently mentioned in the checklists (cf. Table 2). In the TTO section, identity (96%) and procedure (92%) were the most frequently mentioned items, whereas the items team introduction and nursing team: sterility were only found in 25% and 21% of the checklists, respectively. With 83%, the item key concerns for recovery and postoperative management were the items most frequently found in the SO section, whereas the item equipment problems was only included in 29% of the checklists.

TABLE 2

Checklist Comparison Between Study Checklists and Original Checklists of the WHO and Swiss Patient Safety Foundation

Item		Checklist Swiss Patient Safety Foundation	Checklist WHO	Items Study Checklists (n = 24)	Items in Study Checklists Categorized as Process Checks	Items in Study Checklists Categorized as Conversation Prompts	Item Type in Study Checklists Corresponding to WHO Categorization, %
SI	(1) Identity	x	x*	23 (95.83%)	23WHO,SPS	0	95.83
	(2) Procedure	x	x*	18 (75.00%)	18WHO,SPS	0	75.00
	(3) Site	x	x*	17 (70.83%)	17WHO,SPS	0	70.83
	(4) Anesthetic plan	x		9 (37.50%)	9SPS	0
	(5) Consent	x	x*	14 (58.33%)	14WHO,SPS	0	58.33
	(6) Site mark	x	x	19 (79.17%)	19WHO,SPS	0	79.17
	(7) Implementation of anesthesia safety checks	x†	x	14 (58.33%)	14WHO,SPS	0	58.33
	(8) Pulse oximeter	x†	x	5 (20.83%)	5WHO,SPS	0	20.83%
	(9) Known allergies	x	x	22 (91.67%)	22WHO,SPS	0	91.67
	(10) Difficult airway or aspiration risk	x	x	12 (50.00%)	6	6WHO,SPS	25.00
	(11) Risk of >500 mL blood loss	x	x	11 (45.83%)	9SPS	2WHO	8.33
	(12) Assignment to the correct operating room	x		7 (29.17%)	7SPS	0
TTO	(13) Team introduction	x	x	6 (25.00%)	3	3WHO,SPS	12.50
	(14) Identity	x	x‡	23 (95.83%)	23WHO,SPS	0	95.83
	(15) Procedure	x	x‡	22 (91.67%)	21WHO,SPS	1	87.50
	(16) Site (mark)	x	x‡	21 (87.50%)	21WHO,SPS	0	87.50
	(17) Patient position	x		12 (50.00%)	12SPS	0
	(18) Antibiotic prophylaxis	x	x	20 (83.33%)	20WHO,SPS	0	83.33
	(19) Anesthetist: patient-specific concerns	x	x	18 (75.00%)	2	16WHO,SPS	66.67
	(20) Surgeon: critical or nonroutine steps	x	x	17 (70.83%)	4	13WHO,SPS	54.17
	(21) Surgeon: operative time	x	x	11 (45.83%)	5	6WHO,SPS	25.00
	(22) Surgeon: anticipated blood loss	x	x	9 (37.50%)	2	7WHO,SPS	29.17
	(23) Nursing team: sterility	x	x	5 (20.83%)	4SPS	1WHO	4.17
	(24) Nursing team: equipment issues or any concerns	x	x	19 (79.17%)	13	6WHO,SPS	25.00
	(25) Display of essential imaging	x	x	14 (58.33%)	14WHO,SPS	0	58.33
	(26) Implants	x		12 (50.00%)	12SPS	0
SO	(27) Name of the procedure	x	x	16 (66.67%)	11WHO,SPS	5	45.83
	(28) Completion of counts	x	x	18 (75.00%)	18WHO,SPS	0	75.00
	(29) Specimen labeling	x	x	18 (75.00%)	18WHO,SPS	0	75.00
	(30) Equipment problems	x	x	7 (29.17%)	2	5WHO,SPS	20.83
	(31) Key concerns for recovery and postoperative management	x	x	20 (83.33%)	11	9WHO,SPS	37.50

*These 4 aspects are combined in 1 item.

†These 2 aspects are combined in 1 item.

‡These 3 aspects are combined in 1 item.

WHO, categorization according to WHO; SPS, categorization according to Swiss Patient Safety Foundation.

In addition to the items of the 2 reference checklists, the SI section is most often supplemented by the item empty stomach (n = 10), the TTO section by the item allergies (n = 10), and the SO by an open item special incidents/uncertainties (n = 7).

Compared with the checklists of WHO and Patient Safety Switzerland, which contain 12 and 21 process checks and 10 and 9 conversation prompts, respectively, the study checklists contain a median of 15.5 process checks (range, 3–25) and a median of 4 conversation prompts (range, 0–10). Although for some items, especially in the SI part, there is consensus on whether the item is a process check or a conversation prompt (e.g., items 1–8), there are some items, especially in the TTO part, which are used as process checks and conversation prompts (e.g., items 19–24).

Checklist Application: Participants, Timing, and Checklist Coordination

The checklists and guidelines regarding application specifications for the 3 sections SI, TTO, and SO were further analyzed. Table 3 shows the results for time of application and profession of the checklist coordinator.

TABLE 3

Number of Checklists/Guidelines That Include Specifications for Checklist Application (n = 24)

	Time: In 21 Cases (87.50%), Information From Checklists/Guidelines Is Available for at Least One Checklist Section		Checklist Coordination: In 16 Cases (66.67%), Information From Checklists/Guidelines Is Available for at Least One Checklist Section
	When should the checklist be applied?		Who should lead the checklist application?
SI	Before induction of anaesthesiaWHO,SPS	18 (75.00%)	Anesthesia team	9 (37.50%)
	Other points in time	3 (12.50%)	Nurses/supportive staff	7 (29.17%)
	Not available	3 (12.50%)	Not available	8 (33.33%)
TTO	Before skin incisionWHO,SPS	19 (79.17%)	Nurses/supportive staff	12 (50.00%)
	Other points in time	2 (8.33%)	Surgeon	3 (12.50%)
	Not available	3 (12.50%)	Not available	9 (37.50%)
SO	Before patient leaves operating roomWHO	5 (20.83%)	Nurses/supportive staff	11 (45.83%)
	Before surgeon leaves operating roomSPS	6 (25.00%)	Anesthetist/surgeon	2 (8.33%)
	At the end of surgery (or similar points in time)	8 (33.33%)	Not available	11 (45.83%)
	Not available	5 (20.83%)

WHO, according to the checklist of the WHO; SPS, according to the checklist of the Swiss Patient Safety Foundation.

Time of Application

Most checklists or corresponding guidelines (88%) contain for at least one checklist section information on when a specific checklist section should be completed. Regarding the first 2 checklist sections, most checklists/guidelines mention the same point in time as the 2 reference checklists, that is, before induction of anesthesia for SI (75%) and before skin incision for TTO (79%). For the SO, 21% agree with the WHO recommendation (before patient leaves operating room) and 25% with the recommendation of the Swiss Patient Safety Foundation (before surgeon leaves operating room). At the end of surgery or similar points in time are mentioned in one-third of the checklists/guidelines.

Checklist Coordination

Two-thirds of the checklists/guidelines contain for at least one checklist section information on who should lead the checklist application. The coordination of the SI is assigned to the anesthesia team (38%) or nurses/supportive staff (29%), the coordination of the TTO to nurses/supportive staff (50%) or surgeon (13%), and the coordination of the SO to nurses/supportive staff (46%) or anesthetist/surgeon (8%).

Minimum Participants at Checklist Application

Nine checklists/guidelines (38%) contain information for at least one checklist section on who must be present when the checklist is applied, although there are large differences in the clarity of the information provided. Regarding SI, the overlap between the analyzed checklists and the 2 reference checklists is that the anesthesia team should be present, but some checklists/guidelines also require presence of the ward nurse, the nurse for surgical positioning, or other supportive staff. There is full agreement that the entire OT team should be part of the TTO, but the descriptions of who belongs to this team vary in detail. Although some checklists/guidelines state entire OT team, others name the single team members such as nurse, anesthetist and surgeon (WHO) or OT technical staff, anesthesia team, surgeon and further OT staff (Swiss Patient Safety Foundation). A similar picture emerges for the SO: the checklists/guidelines all mention that the OT team should be present, but the description ranges from OT team to nurse, anesthetist and surgeon (WHO) to OT technical staff, surgeon, and anesthetist (Swiss Patient Safety Foundation).

DISCUSSION

The analyzed checklists show some parallels regarding structure and basic items. Most checklists comprise the 3 parts SI, TTO, SO, and the total number of checklist items of the 3 checklist sections corresponds in most cases to the recommendations of the 2 reference checklists. However, there are also major differences that raise doubt about the quality and comparability of checklists, with good and less good examples of adaptations among the checklists analyzed.

The results show that essential items are often omitted. For example, according to WHO’s implementation guide, conversation prompts should never be removed.[16] In fact, many of these items are only included in 50% or less of the checklists (cf. Table 2). This is in line with the findings from Solsky et al,[15] who reported that items of the anticipated critical events section (items 19–24) were often removed.

The number of individual adaptations makes it difficult to compare the checklists and thus their effectiveness. Four hospitals even use 2 or more checklists themselves. For different types of interventions, it may be useful to develop a separate checklist for each type of intervention. For different types of anesthesia or surgical disciplines, one general checklist with optional items, which then only apply to certain cases or a simplified version for emergency cases as suggested by Seppey and colleagues,[21] is more useful with regard to usability, development, and training efforts.

It seems that many adaptations are not made systematically and by persons with checklist expertise, but rather based on personal and practical assumptions. The development and adaptation of checklists is a typical human factors expertise, but unfortunately, such experts are often not involved. Precise checklist application guidelines to avoid ambiguity and diffusion of responsibility are often missing. For example, although the timing for SI and TTO is rather clear, there are differences in the timing of the SO. However, this is important because it has an impact on who is present at the SO.[14] If the SO is defined as before surgeon leaves, it is clearly stated that the surgeon must be present during SO, which is essential to enable the discussion of the intervention and postoperative aspects.

Our results also highlight that there is no shared understanding about the actual character of the checklist. The question arises whether the checklist should be a process guide, a reminder, or a communication aid. For example, the item team introduction is only part of six checklists that is unfavorable, as this item is considered essential for team trust, good team communication, and an atmosphere that allows for speaking up.[22,23] On the other hand, seven checklists have an additional item in the SO section, which should encourage an open exchange within the team to discuss special incidents/uncertainties. According to Catchpole and Russ,[24] these items that aim to promote communication and teamwork are a special feature of health care checklists and not a traditional and indispensable part of classical checklists. Although they agree that these items can facilitate the change of communication patterns, they argue that good communication and teamwork is a prerequisite for authentic checklist completion. Consequently, the opportunity to raise concerns before the intervention would be much more important in terms of communication and teamwork than after the intervention.

Very often it seems as if the checklist coordination and documentation is being shifted to nurses and supportive staff. Sometimes the anesthesia team is also responsible, but very rarely the surgeon, although he or she is often the only constant in the OT team,[22] especially during long surgeries. It has been shown that medical doctors, especially surgeons, show resistance to checklist application.[10,11,14,21] The interactions within the OT team seem to be dominated by complex sociological and cultural challenges such as power distance and hierarchy rather than by the design and application of a checklist.[24] To emphasize the importance of the surgical checklist and to strengthen the responsibility of management, it might therefore make sense to transfer the lead of the checklist to the medical profession. This is supported by the findings from Russ et al,[13] who have already shown that the attention during TTO and SO is higher and more information is exchanged when the checks were led by the senior surgeon instead of the circulating nurse.

The previously described results are of relevance beyond checklist use because local adaptions are commonly recommended in implementation efforts, but the boundary requirements of these adaptions are rarely addressed.[20]

Limitations

Generalizability of this study is limited in that there was only one single coder. However, any uncertainties in the coding were discussed with 2 other project team members. The checklists analyzed originate exclusively from Switzerland, but because of the country’s multilingualism and the cultural differences between the 3 language regions, different languages and cultural conditions were nevertheless considered in the study. Because the German-language checklists are overrepresented, it was not possible to make statistically significant comparisons between the language regions.

No statements can be made regarding the effectiveness of the checklists analyzed, as the results are gained from document analysis and no clinical outcomes were included. However, recommendations are based on evidence-based theoretical assumptions and previous studies.

CONCLUSIONS

Not all adaptations made to fit local conditions and personal preferences add value. If the design of checklists is likely to have an impact on its effectiveness and checklist design varies largely, 2 major conclusions need to be drawn: first, more resources must be invested in proper checklist adaptions, and second, details of used checklists need to be clearly described in studies on checklist effectiveness.[17,20] To support this, better guidance on how to adapt safety tools such as the surgical safety checklist to local conditions is needed,[19] that is, better official guidelines and support by safety and human factors experts during adaptation processes. The more freedom there is for adaptation, the better and clearer guidelines are needed.