Barriers to staff adoption of a surgical safety checklist.

OBJECTIVE: Implementation of a surgical checklist depends on many organisational factors and on socio-cultural patterns. The objective of this study was to identify barriers to effective implementation of a surgical checklist and to develop a best use strategy.
SETTING: 18 cancer centres in France.
DESIGN: The authors first assessed use compliance and completeness rates of the surgical checklist on a random sample of 80 surgical procedures performed under general or loco-regional anaesthesia in each of the 18 centres. They then developed a typology of the organisational and cultural barriers to effective checklist implementation and defined each barrier's contents using data from collective and semi-structured individual interviews of key staff, the results of an email questionnaire sent to the 18 centres, and direct observations over 20 h in two centres.
RESULTS: The study consisted of 1440 surgical procedures, 1299 checklists, and 28 578 items. The mean compliance rate was 90.2% (0, 100). The mean completion rate was 61% (0, 84). 11 barriers to effective checklist implementation were identified. Their incidence varied widely across centres. The main barriers were duplication of items within existing checklists (16/18 centres), poor communication between surgeon and anaesthetist (10/18), time spent completing the checklist for no perceived benefit, and lack of understanding and timing of item checks (9/18), ambiguity (8/18), unaccounted risks (7/18) and a time-honoured hierarchy (6/18).
CONCLUSIONS: Several of the barriers to the successful implementation of the surgical checklist depended on organisational and cultural factors within each centre. The authors propose a strategy for change for checklist design, use and assessment, which could be used to construct a feedback loop for local team organisation and national initiatives.

Introduction

Since the development of checklists for use in operating rooms by the WHO in 2008, their use has become increasingly widespread and seems to be associated with a significant decrease in postoperative complications and mortality rates.1–4 Recently, however, questions have arisen about their ease of introduction into workflow patterns and their true impact on safety.5 6 These aspects require both high physician commitment and an ability to capture the work organisation.7 8 Thus, even if surgical checklists are widely used, proactive management of the organisational changes required is indispensable for use to be effective and enduring.

In January 2010, the French National Authority for Health (Haute Autorité de Santé, HAS), in collaboration with professional colleges, introduced a modified version of the WHO surgical checklist for which it provided training sessions, written materials and videos.9 This mandatory checklist has 22 items covering three timeframes: before anaesthesia (‘sign in’) (nine items), before skin incision (‘time out’) (eight items), and before leaving the operating room (‘sign out’) (five items) (figure 1). However, at a national meeting 6 months after the checklist was introduced, a representative of the French Society of Anaesthesia and Intensive Care declared that ‘bureaucratic use’ of the checklist was of no benefit and might even have a negative impact, and that health professionals should take organisational factors into account.

Figure 1

Surgical safety checklist published by the French National Authority for Health (HAS) (January 2010 version*). An updated version was published in January 2011.

The objective of this study was threefold: to assess the use compliance and completeness rates of surgical checklists in cancer centres; to identify and compare barriers to effective use in these centres; and to develop a strategy for effective use. A barrier was defined as a work situation causing problems involving organisational rules, human skills, cultural features and beliefs.

Methods

Setting

The study was initiated in October 2009 by the French National Federation of Cancer Centres (FNCLCC), in collaboration with a research team specialising in healthcare management (COMPAQH, Coordination for Measuring Performance and Assuring Quality of Hospitals, Institut Gustave Roussy). It included 18 (out of 20) centres which were implementing the mandatory checklist. These 18 centres performed 40 620 surgical procedures in 2009.

Measurement of use compliance and checklist completeness

The study included measurements of surgical procedures performed under general or loco-regional anaesthesia between 11 and 29 January 2010. Cases of topical anaesthesia, interventional radiology, gastro-intestinal endoscopy and central venous catheter implantation were excluded.

Checklists were kept as part of each patient's medical record. However, as medical records were not completely computerised, checklist data could not be extracted automatically. A random sample of 80 records from each centre was analysed. This sample size was also based on published studies.10 11 Individual scores were calculated for each sample and the 18 participating centres were ranked into three categories (top, middle and bottom). Definition of the categories was based on the degree to which the 95% CIs did or did not overlap the mean score. The method used partly features the Hospital Report Research Collaborative method. This method is fairer than simple ranking and accounts for uncertainty in the scores.12

The method used in this study had three categories and one CI—the 95% CI, whereas the Hospital Report Research Collaborative method had two CIs—90% and 99% CIs. For each centre, the compliance rate was the percentage of checklists found in the 80 patients' medical records. The mean compliance rate was calculated as the sum of the compliance rates for each centre divided by the number of centres.

For each centre, the completeness rate was the percentage of complete checklists found in the 80 patients' medical records. A complete checklist is a checklist in which all items have been ticked. The mean completeness rate was calculated as the sum of the completeness rates for each centre divided by the number of centres.

Analysis of barriers to effective checklist use

To identify barriers to effective checklist use and for good internal validity, data were collected in three ways (collective and individual interviews, an email questionnaire sent to surgical staff and direct observations), and the results triangulated. First, two researchers (AF and EM) collectively interviewed voluntary staff in 16 of the 18 centres about the benefits and drawbacks of checklist implementation during a 3 h session. Four surgeons, three anaesthetists, two nurses, six senior nurses and one quality manager took part in this collective interview. Then, eight other key surgical staff from different centres (two surgeons, three anaesthetists and three senior nurses) were scheduled for individual interviews using as substrate a semi-structured questionnaire (25 questions) based on the results of the collective interview. A ninth person, a surgeon, declined the interview because of a full agenda. All eight individual interviews were conducted by the same researcher (AF) to ensure that a similar emphasis was given to each question in each interview and that the meaning and importance ascribed to each barrier was explicit. The interviews lasted 30 min on average. Second, to capture a collective perspective, a contact person in each centre was sent an email questionnaire consisting of 25 questions plus five questions on further barriers brought up by the interviewees. This email questionnaire was piloted for clarity at the Institut Gustave Roussy and took about 10 min to complete. All centres returned the questionnaire. In most cases, it was completed by operating room staff but sometimes by quality department staff. Lastly, a researcher (AF or EM) made direct observations on all the barriers that had cropped up during the interviews and in the completed questionnaires. These observations were made over 20 h in two centres, one with a low (52%) and one with a high (84%) proportion of complete checklists.

Two researchers (AF and EM) processed and compared the data from the interviews, email questionnaire, and direct observations. Data were categorised as follows. Axial coding was used to relate responses and create a typology of the barriers hindering effective checklist use. A first typology was created after the collective interview. It was revised twice, first after addition of the data from the individual interviews and then after addition of the data from the questionnaires. Vertical coding was used to better define the content of each barrier. Dependency chains were sought to distinguish data related to missing checklist items and data related to barriers.13 The content of each barrier was defined after the interviews and fine-tuned after the return of the questionnaires and after making direct observations, using the revised typology. The typology and barrier contents were discussed and validated by the staff who had participated in the collective interview, on the basis of their working experience, during a second 3 h session during which discrepancies were resolved and minor adjustments were made.

Results

Compliance of use and checklist completeness

The analysis included 1440 surgical procedures (80 per centre). Overall, 1299 checklists were handed in and 28 578 items were analysed. The mean compliance rate was 90.2% (range 0–100) and the mean completeness rate was 61% (range 0–84) (figure 2). The 0% score was allocated to a centre unable to retrieve proof that the medical reports had been checked. The median compliance rate was 98.75 and the median completeness rate was 66.25. For the first quartile, the compliance rate was 92.50 and the completeness rate was 52.81. The values for the third quartile were 99.69 and 79.38, respectively. Most missing items (47%) occurred during ‘sign out’ (table 1).

Figure 2

Rates of compliance of use (A) and completeness (B) recorded for the surgical checklist.

Table 1

Frequency of missing items

Item	Number of times missing	%	Min (%)	Max (%)
Sign in
1. Patient identity	7	0.54	0	2.5
2. Procedure and operation site confirmed	12	0.93	0	3.75
3. Clinical and paraclinical information required available in operating room	74	5.71	0	52.5
4. Patient positioning	16	1.23	0	5
5. Surgical equipment required	34	2.62	0	13.75
6. Anaesthesia equipment required	38	2.93	0	12.5
7. Patient allergies	19	1.46	0	5
8. Risks of inhalation, difficulties of intubation or mask ventilation	32	2.47	0	6.25
9. Significant risk of bleeding	68	5.24	0	41.25
Total	300	22.59
Time in (cross-checking)		29.45
10. Patient identity correct	30	2.31	0	13.75
11. Planned procedure confirmed	35	2.70	0	13.75
12. Operation site identified	35	2.70	0	13.75
13. Patient position correct	37	2.85	0	13.75
14. Documents required available	65	5.01	0	32.5
15. Surgical information shared	67	5.17	0	18.75
16. Anaesthesia information shared	51	3.93	0	11.25
17. Antibiotic prophylaxis given	62	4.78	0	15
Total	382	22.59
Sign out (verbal confirmation)
18. Procedure recorded	97	7.48	0	23.75
19. Instrument, swab and needle counts correct	118	9.10	2	26.25
20. Specimens and samples correctly labelled	103	7.94	0	26.25
21. Problems with equipment and adverse events reported	143	11.03	0	33.75
22. Postoperative orders drawn up jointly by surgeon and anaesthetist	154	11.87	0	41.25
Total	615	47.42
Grand total	1297	100.00

Barrier typology

Eleven barriers to effective checklist implementation were identified. Their frequency, as judged from the answers to the email questionnaire, is given in decreasing rank order, with an example of content taken from the individual and collective interviews and/or from direct observations (table 2). Other elements of content support some key aspects of each barrier.

Table 2

Frequency of occurrence of 11 barriers with illustrative examples

Barrier	Centres (out of 18)	Illustrative examples
Duplication with existing checks	16	Checking patient identity, accounting for sponges and adverse event reporting
Poor communication between anaesthetist and surgeon	10	Did not always use the same document to record postoperative orders during ‘sign out’
Time consuming	9	Checklist too long to complete, especially when very busy (eg, emergency surgery, end of day)
Does not make sense	9	Staff in some operating rooms are not accustomed to count needles and this may not even be possible after disposal into appropriate containers during surgery to avoid injury
Inappropriate timing	9	Difficult to check sample labelling at the end of the procedure if the samples were sent to the pathology laboratory during surgery
Ambiguity	8	Did a ‘yes’ response for ‘allergies’ mean that the patient had an allergy or that the risk of allergy had been checked
Unaccounted risks	7	Checklist did not cover skin preparation and postoperative prevention of pain or vomiting
Oral confirmation of items	6	Reading out the entire list was found unnecessary
Identification of the role and responsibility of staff	6	Direct observation had difficulty in identifying the person implementing the checklist during emergency and/or short procedures as all staff were totally engrossed in their task
Patients' attitude to questions	5	Asking the patient his or her name three times over a very short time may cause alarm
Gaming	5	Ticking off unchecked items at the end of the day

The most common barrier (16 centres) was duplication with existing processes that already covered several of the items in the surgical checklist. For example, checking patient identity, accounting for sponges or adverse events reporting were already carried out before the use of the checklist, as procedures and documents existed for those examples. For adverse events reporting, an electronic system was also used in most of the centres. Therefore there was duplication in documents for reporting these events.

The next most common barrier (10 centres) was lack of communication between the surgeon and the anaesthetist at the end of the surgical procedure. The surgeon might leave the centre before ‘sign out’ and the anaesthetist might return during recovery after skin closure. This jeopardised the sharing of information on patient management.

Staff in nine centres found that the checklist took too long to complete as they already had a heavy workload, and did not perceive the added benefit. According to one surgeon, ‘the checklist could generate delays and conflicts’. High staff turnover, especially of nurses, in nine centres was also considered an obstacle to checklist implementation despite provision of training. In nine centres, some items did not make sense because they did not fit in with customary operating room practices (eg, needle counts after disposal) or because their timing was inappropriate.

The binary (yes/no) response system was ambiguous and confusing according to eight centres, that is, could a non-‘yes’ answer prevent moving on to the next question and postpone surgery? Seven centres reported risks that were not prevented by application of the checklist, even if it were fully completed (for example, thrombo-embolism, contamination and viral risk (HIV, HCV, HBV)). Staff in six centres had communication problems during checks. According to the nursing staff completing the checklist, the surgeons and anaesthetists who were ultimately responsible for the surgical procedure did not always listen to the items when they were read out. Nurses were therefore concerned about the legal implications of signing the checklist as they might be held accountable for errors. In five centres, staff thought that repeating questions that had already been answered several times might generate anxiety in patients about to undergo anaesthesia but this was not supported by direct observation. Five centres reported that items could be ticked off even when items were not checked because of time constraints. Items were ticked only to comply with the management audit; therefore, in this situation, checklists failed to improve patient safety.

However, during the 20 h of observation in two hospitals, many items were left unchecked or answers given by the wrong person but there were no instances of items being ticked off without actually checking the item at the beginning of the surgical procedure or at the end.

Variability between centres

Checklist use was compared by direct observation in two centres with similar case mixes but with different compliance rates (100% and 92.5%) and completeness rates (84% and 52%). In the centre with poor results, several staff members had not been involved in checklist implementation. Many items were not checked. The head surgeon in this centre had attended the training session but considered ‘administrative tools’ unsuited to assessing his field of work. He was therefore against using the checklist. His staff thought that the measurements that the anaesthetist, who was working on the centre's information systems, had made using the checklist were ‘boring’ and irrelevant to just-in-time management.

Discussion

Mean compliance with mandatory checklist use was high (90.2%) at introduction but far short of the desired 100% score. The mean percentage of complete checklists was relatively low (61%; range 0–84%), indicating that hospital staff encountered problems with implementation after checklist adoption. These results support those for the SURPASS checklist for which the completion rate was about 80% for most items but less than 30% for others.2

Eleven organisational barriers to implementation were identified and several of those already identified, such as poor communication between the anaesthetist and surgeon, lack of leadership, inappropriate timing for checking an item, time taken up by checklist completion, and difficulty in identifying the role and responsibility of each staff member, were confirmed.6–8 14 However, these results should not overly dampen enthusiasm and prevent use of the checklist as a ‘short, uncluttered by noncritical items, and carefully worded’ reminder of key actions.8

Two main reasons might explain the barriers identified. The first relates to work organisation. A checklist is often put across as a tool to enhance communication and as a reminder in stressful circumstances but, like other operational tools, it impacts on the organisation of work.8 15 Organisational changes are needed while implementing a surgical checklist in operating rooms, for example, harmonisation with existing adverse-event reporting systems and allocating the time needed to complete the checklist, communicate it, and check that it is complete. The second reason relates to professional relationships and cultural habits in clinical practice. Verbal communication between health professionals (surgeons, anaesthetists and nurses) has to be egalitarian for checklist use to be effective but the findings suggest that operating room staff practices are rooted in a time-honoured hierarchy, at least in France.

The number of unchecked items varied widely across centres, and was probably a reflection of the usual behaviour of the surgical team and of the ‘sense’ they made of the checklist.14 Operational tools uncoupled from operational activities because they lack ‘sense’ may fall short of the expected level of use or induce undesirable practices such as inhibition in care practice or gaming.16–19 Any strategy for organisational changes to be made on checklist introduction should therefore take into account the cultural maturity and history of operating room staff.

The findings of this study have practical implications for the design (areas of applicability), use (local contexts) and assessment of checklists by national health authorities. Checklist design should account for the organisational constraints to which the activity is subject. Clarity is also essential. HAS added the option ‘Not Applicable’ to the ‘Yes/No’ boxes of most checklist items in May 2010 and included all 11 barriers in their checklist's guideline to improve checklist use. As for all quality improvement measures, each centre should develop its own strategy for work organisation as indicated by the wide variation in checklist implementation across centres.20 For example, one cancer centre laid down five ways of improving surgeon–anaesthetist communication on peroperative risks, which was one of the most important barriers identified. In line with Parand et al's view that the priorities and concerns of surgeons are paramount, the surgeons and anaesthetists of this centre listed unexpected events, entered these events into a common thesaurus, set up an alert system for staff outside the operating room, made the presence of the anaesthetist compulsory at the end of each surgical procedure in order to share information on drug prescriptions, and asked for regular feedback on experience.7 At the close of the present study, the working group formed by the members of staff taking part in the collective interview developed a formal assessment plan with the support of the national Cancer Centre Federation. This plan included feedback from centres on how they dealt with the 11 barriers with a view to sharing of experiences, longitudinal follow-up of compliance and completeness, longitudinal measurement of a list of complications (to be drawn up in 2011) in order to include perioperative safety, and a yearly questionnaire assessing the cultural maturity of each centre.

This study has several limitations: The announcement by HAS that the checklist was mandatory may have resulted in high staff motivation and overestimation of checklist use (both compliance and completeness). Overestimation may also have arisen from the presence of an external observer (Hawthorne effect). However, completeness may have been underestimated as, despite training, staff were still relatively unfamiliar with the checklist. The staff members taking part in the collective interview were a source of data but were also involved in the validation of the analysis performed by the researchers. This may have led to some bias in interpreting the results, although discrepancies with the researchers' analysis were resolved by discussion and led to only minor adjustments. Lastly, situations such as interventional radiology and local anaesthesia were excluded. Whether the findings on surgery in cancer centres can be generalised to other types of surgery in other settings requires further analysis.

In conclusion, before considering the impact of a surgical checklist on safety, it is necessary to consider its adoption by staff and to highlight the barriers to effective use. Consideration should be given to tailoring checklists to services and areas with different needs (eg, the lab specimen labelling example) and/or education about items that are new but still considered important by the proponents of the checklist (eg, the needles examples). The strategy for change described in this study could be used to construct a feedback loop for local team organisation and national initiatives.