Improvement in staff compliance with a safety standard checklist in endoscopy in a tertiary centre.

National Health Service England published the National Safety Standards for Invasive Procedures (NatSSIP) in 2015. They mandated that individual trusts produce Local Safety Standards for Invasive Procedures (LocSSIPs), a set of safety standards drawn from the NatSSIP that apply to a particular clinical situation in a given department, for all invasive procedures. The project goal was to design and implement the LocSSIP within the endoscopy department. A draft LocSSIP was produced, and a pilot study conducted to gain initial feedback on its use. Version 1 of the checklist was produced and after approval, rolled out for use within the endoscopy department at 'time out' and 'sign out'. A scoring system was developed that allowed the quality of the performance of LocSSIPs to be assessed and recorded as a 'compliance score'. After 2 months, an independent observer spent a week assessing use of the checklist, recording completion and a compliance score. Analysis of this data led to a number of changes in performing the checklist, wider multidisciplinary team education and integration of the checklist into existing documentation, before reassessing at 12 months. In 2016, 'time out' checks were completed in 100% of cases, but full completion was only observed in 68%. 'Sign out' checks were completed in 91% of cases, with full completion in 71%. In 2017, 'time out' checks were completed in 100% of cases, with full completion in 85%. 'Sign out' checks were completed in 100% of cases, with full completion in 91%. The composite score for compliance in 2016 was 57% increasing to 90% in 2017. In conclusion, stronger departmental leadership, broadening education and integration of the checklist into routine documentation to reduce duplication led to significant improvements in compliance with use of the checklist. Ongoing education and assessment is imperative to ensure that compliance is maintained to ensure patient safety.

Problem

In 2015, National Health Service (NHS) England published the National Safety Standards for Invasive Procedures (NatSSIP)1 based on recommendations in a report from the Surgical Never Events Taskforce.2 This directed all healthcare providers that delivered NHS funded care to adhere to the safety standards contained within the NatSSIP when performing any invasive procedures on patients.

The NatSSIP provides a skeleton framework for any invasive procedure and as such some components of the NatSSIP are more or less applicable in different procedures or environments. The intention is for the NatSSIP to be modified to a Local Safety Standard for Invasive Procedures (LocSSIP). This LocSSIP is a set of safety standards drawn from the NatSSIP that apply to a particular clinical situation and are designed for a given hospital trust, taking into account human factors and working as teams.

University Hospitals Bristol NHS Foundation Trust (UHB) is a large teaching hospital. Endoscopy is performed in three sites across the trust: the Bristol Royal Infirmary, Bristol Children’s Hospital and South Bristol Community Hospital. Endoscopies are also performed outside of the endoscopy departments, mainly in operating theatres and the intensive care unit. Endoscopies are performed by a variety of clinicians including surgeons, gastroenterologists, respiratory physicians and nurse endoscopists. UHB offers both routine endoscopic services widely available in secondary care institutions and more advanced, interventional endoscopy available only in tertiary centres, such as endoscopic mucosal resection and radiofrequency ablation for dysplastic Barrett’s oesophagus and early oesophageal cancer.

The overall aim of this project was to develop and implement the LocSSIP in the endoscopy department at UHB. Our SMART aim and the focus of this project was to ensure completion of the standard safety checklist (SSC) in 100% of patients, as directed by NHS England,1 following development of the LocSSIP after a 12-month period. To ensure quality was maintained, our aim was to ensure compliance scores of >80% after 12 months.

As this was new guidance, no baseline data was available for comparison. Therefore, a plan was made to design and implement the LocSSIP over a 6-month period. Following introduction of the checklist, baseline assessment of compliance was made; with repeat measurement after change, cycles had been implemented.

Background

Endoscopy allows direct visualisation of the lumen of the gastrointestinal and respiratory tract for diagnostic and therapeutic purposes. As such, it is an invaluable tool for clinicians and patients. It is an invasive procedure, often carried out under conscious sedation or general anaesthesia and as a consequence carries with it significant risk.

The National Confidential Enquiry into Patient Outcome and Death report ‘Scoping our Practice’ investigated over 1800 deaths within 30 days of therapeutic endoscopy and highlighted failings in practice across the UK.3 It highlighted the shortcomings in procedure planning, safe administration of sedation and patient monitoring as opposed to technical competencies of the endoscopist. The report made 21 recommendations to improve patient safety. There are no large-scale evaluations available to assess whether these recommendations have been implemented and their effectiveness.

There is extremely limited published data specific to patient safety in endoscopy. One single centre study found that a median of one adverse event occurred in each endoscopic procedure (n=140) carried out over a 4-month period of observation.4 This study observed the whole patient pathway: preprocedure, intraprocedure and postprocedure.

In 2004, the National Patient Safety Agency (NPSA) published a report outlining the ‘seven steps to patients safety’ in response to the increasing complexity and risk associated with effective, modern healthcare.5 In 2009, a modified version of the WHO surgical safety checklist became a requirement within the NHS.6 The value of these developments was recognised and formed the basis of the NatSSIP published in 2015. It has been recognised that clinical errors cause significant morbidity, mortality and expense across the NHS, and the NatSSIP is an effort to ‘standardise, educate and harmonise’ safety standards across invasive procedures in the NHS.

Matharoo et al 7 published an excellent review into implementing SSCs in endoscopy.7 The paper identified enablers and barriers to implementation of an endoscopy checklist. Enablers included strong leadership skills from within the endoscopy team, alongside seeking support from ‘external’ respected sources and other departments familiar with surgical checklists. Barriers included professional hierarchy, lack of evidence for its use within endoscopy, viewing it as a ‘tick box’ exercise, cumbersome duplication of documentation and lack of team engagement.

Measurement

The WHO SSC has been in use in operating theatres for several years and is a well-embedded part of the culture in that environment. This checklist forms a good exemplar on which to base LocSSIP guidance for endoscopy, with similar end results and targets. Previous root cause analysis of two never events at UHB in 2014 had found 100% completion of the WHO SSC, but varying levels of attention from staff during its completion, identified as a possible contributor to the never events. In response to this, a ‘tick box’ paper SSC was replaced by a verbal checklist that follows visual prompts on a wall-mounted poster. NHS England recommended that organisations should develop scoring systems that allowed the quality of the performance of LocSSIPs to be assessed. A ‘compliance score’ was therefore developed and implemented to assess the quality and attention levels of team members after its implementation.

A ‘completion score’ documents that the SSC has been performed. Points are then deducted from this initial score for not fully completing the checklist and for incorrect timing of the checklist, forming a ‘full completion score’. Inattention is measured at time out, and categorised as mild (any team members completing a small task), moderate (talking to a colleague) or severe (not present). This inattention score is then deducted from the ‘completion score’ to give an overall ‘compliance score’.

Data were collected prospectively for ‘completion score’, ‘full completion score’ and the ‘compliance score’ to gain baseline data following design of the checklist and to compare outcomes following our interventions. To collect this data, we planned a period of independent observation within the endoscopy department at the Bristol Royal Infirmary over a 2-week period, 2 months after implementation of the safety checklist. A medical student observed endoscopies and recorded data for each case on a standardised data collection pro forma. Following implementation of change cycles, a further observational period was planned over 2 weeks, 12 months later. Feedback from staff directly involved with the SSC was sought to establish the likelihood of the observed outcomes being due to our interventions.

Design

The ‘perioperative never event working group’ is a trust wide body at UHB drawing members from a wide variety of specialties and professions. One of its roles is to facilitate the implementation of the NatSSIP across the trust. The team implementing the NatSSIP in endoscopy was members of this group and had access to support from administrative and managerial staff through the group.

The initial development process was to carefully examine the NatSSIP and to identify points that were or were not applicable to endoscopy. This allowed a draft LocSSIP to be written and circulated for review by the working group and lead nurse for endoscopy. Concurrently, draft documentation was produced to support initial implementation of the LocSSIP.

Draft LocSSIP guidance was based on the WHO surgical safety checklist. The five steps of surgical safety consist of a ‘team brief’ at the beginning of a list to highlight any foreseeable problems before the list begins. Each procedure has a ‘sign in’ when the patient arrives, ‘time out’ immediately prior to the procedure starting and a ‘sign out’ prior to the patient leaving the procedure room. Finally, an end of list debrief is important to raise any learning points.

A wall-mounted checklist was produced to provide a focus for all members of the clinical team and prompt of the information required for checklist completion. A pro forma was produced to be included in a patient’s notes to provide written documentation of completion of all stages of the checklist. Staff were given education to promote a ‘safety awareness’ culture within the department.

Strategy

Our SMART aim and the focus of this project was to ensure completion of the SSC in 100% of patients, as directed by NHS England, following development of the LocSSIP after a 12-month period. To ensure quality was maintained, our aim was to ensure compliance scores of >80% after 12 months. In order to do this, the project was broken into several phases.

Development

As discussed above, a draft of the LocSSIP was produced and circulated. Draft documentation was also developed to support the LocSSIP.

PDSA cycle 1: trial

This part of the process was designed to refine draft documentation and identify any obvious flaws in the early draft of the checklist.

To do this, the clinical lead attended an endoscopy list as an observer. Several variations of the checklist and documentation were trialled alongside the existing patient pathway. Between patients, comments were made by the endoscopy staff and further adaptations were made before the next patient’s procedure. The adapted format was then trialled on the next patient.

This was, effectively, multiple PDSA cycles carried out over the course of a few hours to develop the checklist and its supporting documents.

PDSA cycle 2: development

Initial feedback from staff after a pilot of the developed documentation showed that there was significant repetition in the ‘sign in’ and ‘time out’ components of the checklist. While this is essential in an operating theatre where the ‘sign in’ occurs before anaesthesia and the ‘time out’ immediately prior to the first incision. Different staff are present for the two stages of the procedure and there is a significant time lapse between arrival in the anaesthetic area and first incision. This is less applicable in endoscopy when the same staff are present for both steps and there is no significant delay between the steps. This format was therefore felt to be cumbersome and disruptive to the endoscopy team, introducing significant delays to the high volume endoscopy lists.

We hypothesised that the negative feedback would equate to poor compliance when the checklist was implemented. We decided, therefore, to combine the ‘sign in’ and ‘time out’ steps. This format had been previously proposed by the NPSA when introducing safety checklists into interventional radiology.8

PDSA 3: roll out

Version 1 of the checklist and documentation was produced and approved following changes made through PDSA 2.

A date was set to roll out the checklist, initially only at one of the endoscopy locations at UHB to enable full training of the team. In the weeks leading up to the start date, educational sessions were carried out with endoscopy staff.

The checklist was introduced and embedded for 2 months before collecting baseline data on completion and compliance. During this time, support was given to clinicians and support staff to maximise effectiveness.

Our initial assessment of the data showed that completion of the checklist was carried out in all cases at ‘time out’, but only 91% at ‘sign out’. Reasons for failure to conduct ‘sign out’ includes changeover in staff and where the procedure did not go to plan (eg, unsuccessful gastroscopy due to stricturing from previous surgery). In clinical situations that deviate from normal, the checklist potentially has an even more important role, and this was therefore, of some concern. Qualitative measurements showed high levels of inattention from members of the team during the ‘sign in’ and ‘sign out’ stages, leading to poor compliance measurements.

PDSA cycle 4: redesign and implementation

Poor compliance measurements after completion of PDSA 3 led the LocSSIP implementation team to undertake a period of reflection and gain feedback from all team members to try to understand why compliance was poor.

An initial assumption had been made that staff were familiar with the WHO surgical safety checklist, having used this in theatres for a number of years, and therefore introducing a similar checklist in endoscopy would be an extension of this process. However, a number of the endoscopy staff, both nursing and medical, had no prior theatre experience and therefore the process was entirely new to them. On reflection, the team felt that more time needed to be spent with individuals, particularly those unfamiliar with the checklist in other contexts, to explain the purpose and value of the checklist, and therefore improve compliance. Greater leadership within the endoscopy room and ownership of the checklist by one individual, who could be any member of the team, was also felt to have been useful.

Feedback from a number of team members was the high level of repetition between some of the information sought within the checklist and information documented elsewhere within the existing care pathway documentation. Analysis of results showed that when a question on the checklist had been answered in the pre-existing documentation, it was often glossed over or completely ignored.

In response to this, we felt that the best way to reduce repetition was to rewrite the patient care pathway documentation and thereby encourage all staff to engage more completely with the underlying principle of the LocSSIP: risk reduction. The care plan was rewritten, minimising duplication and placing the checklist at the core of the document. A full redesign of documentation required significant time and approval from a variety of committees across the trust. This phase therefore took around 6 months to complete, before the new integrated checklist/patient care pathway could be introduced.

When the revised documentation and version 2 of the checklist (online Supplementary appendix 1) had been prepared, staff were given further educational sessions, and the clinical leads within each specialty were approached in an effort to further support its implementation. Plans were made to perform observational reassessment of the data in July 2017, 12 months after baseline measurements had been made.

The number of adverse incidents reported in the department over the preceding 12 months was too small to allow statistical comparisons to be made to directly assess the efficacy of the checklist at reducing adverse incidents.

Results

Figure 1 summarises ‘time out’ and ‘sign out’ completion in the endoscopy department between assessments. It shows improvement in performance on all measures except for ‘time out’ completion, where full compliance had been achieved in the initial assessment.

Figure 1

Comparison of ‘time out’ and ‘sign out’ completion between the two audits.

Data from individual questions within the ‘time out’ and ‘sign out’ were also collected. Table 1 shows the proportion of times these questions were asked and answered during ‘time out’ between the two periods of assessment.

Table 1

Percentage of total ‘time outs’ that each checklist component was appropriatley completed

Criteria	2016 (%)	2017 (%)
Identity	100	100
Consent	93	100
Procedure	82	100
Labels	100	100
Allergies	84	96
Coagulopathy	80	89
Sedation plan	73	96
Monitoring	73	96
Equipment	73	93
Diathermy	73	89
Antibiotics	71	96
Glycaemic control	82	89

Table 2 shows the proportion of times individual questions were asked and answered during ‘sign out’ between the two periods of assessment.

Table 2

Percentage of total ‘sign outs’ that each checklist component was appropriatley completed

Criteria	2016 (%)	2017 (%)
Check patient details on report	76	89
Equipment problems	71	82
Follow-up arranged	73	96
Recovery concerns	76	85
Anticoagulation plan	76	85
Specimen labelling	91	89

The composite score for compliance was 57% in 2016 and 90% in 2017.

After the first audit, we identified that the lowest levels of compliance were seen in sections of the checklist where information was duplicated in the nursing care plan. Following our interventions, there were significant improvements made, particularly in levels of attention and full completion of the checklist.

The poor compliance data seen at baseline assessment was unforeseen at the outset of the project. The information sought during the ‘sign in’ and ‘time out’ was already being gathered within the nursing care plan prior to introduction of the formal checklist. It was assumed that this would mean a smooth integration of the checklist in the endoscopy rooms.

The significant time involved in integrating the checklist with the nursing care plan documentation was an unforeseen part of the project and meant that less time than anticipated had passed before reassessment of the data was conducted.

Lessons and limitations

A limitation of the study includes the ‘Hawthorne effect’, in which members of the team modify their behaviour due to their awareness of being observed. The results may not therefore be a true reflection of everyday practice.

Second, the study may be limited by the sample size. Baseline measurements observed 45 procedures, but on reassessment, this number was less at only 27 procedures during the 2 weeks of observation. If performing this study again, it would be beneficial to gather similar numbers at both baseline and following interventions to increase the validity of results. The 2-week period was needed to fit in with a special study module of the medical students involved as the independent observer and was therefore unable to be extended.

A representative sample was attempted by rotating between different endoscopy rooms, and therefore covering different procedures performed by different teams. However, use of the SSC in procedures such as bronchoscopy and endoscopic retrograde cholangiopancreatography were not observed. Thus, conclusions from this audit may not be reliably drawn across the whole of endoscopy.

This project identified from previous adverse events within UHB that simply assessing completion of a checklist is not enough and that introducing a measurement of quality is essential to impact on safety. This assessment of quality is a strength of the study. However, the study makes the assumption that improving compliance with the checklist will go on to reduce adverse incidents. Given that overall numbers of never events are very low, a much longer time frame is required to assess the impact of our changes on this and is outside the scope of the current project.

Conclusion

The aim of our project was to introduce a LocSSIP and safety checklist to the endoscopy department at UHB. Following its introduction, we aimed to achieve 100% completion of the SSC, with compliance levels >80% ensuring that this was performed in a high-quality manner, rather than simply a ‘tick box’ exercise. Compliance rates over this 12-month time frame rose to 90% which is very encouraging, though there were still some aspects of the checklist that were incomplete.

Feedback from all team members felt that the main contributions to the observed improvement were strong leadership within the endoscopy department, with a nominated individual taking ownership for the completion and being responsible to engage the whole multidisciplinary team at each ‘time out’ and ‘sign in’ checklist completion. Broadening education on why the checklist was so important has helped to drive a wider cultural change that safety is the shared responsibility of all individual team members.

Early on in the development of the project, it became clear that simply adding in a checklist into the existing procedures of the endoscopy unit would lead to poor compliance. We learnt that understanding the complete patient journey through the department and how the unit functioned meant that a tailored approach was required. While this was a slight departure from the standard checklists seen in other parts of the hospital, it proved to be far more effective.

There was some concern from staff that patients would become anxious with repeated asking of a number of questions. As part of the project, a small sample of eight patients were surveyed regarding their experience of being asked repeatedly to confirm details. In all cases, the patients answered ‘I would absolutely expect nothing less and I find it reassuring’ and this is likely to have contributed to their answers of ‘strongly agree’ to being made to feel that their safety was the highest priority.

This project was an essential piece of work following the mandate from NHS England to implement LocSSIPs for all invasive procedures at a national level. Use of the checklist is ongoing, and the trust is accountable for its use to Clinical Commissioning Groups and to the Care Quality Commission. High completion and compliance rates after 12 months is very encouraging, but maintaining these high scores is essential in the longer term to maintain safety. The development of LocSSIPs itself cannot guarantee the safety of patients, and it is essential that teams undergo regular, multidisciplinary training to promote teamwork and includes clinical human factor considerations.

While ensuring patient safety is the primary incentive for the LocSSIP, Never Events are an enormous cost to an organisation. Commissioners can withhold payment for the cost of the episode of care in which a Never Event has occurred and any subsequent costs involved in treating the consequences of a Never Event. In addition, significant time and resources will be used in undertaking root cause analyses, leading to higher costs for the trust.

The project has succeeded in its initial goals, but there remains room for ongoing change and improvement. Ongoing audit is essential, ideally with larger sample size to facilitate subgroup analysis and to highlight specific areas of weakness or specific procedures where compliance is poor. This should allow more targeted interventions to reduce variation in standards.

Our project demonstrates the initial development and implementation of LocSSIP within one department of our organisation. The scoring systems developed to assess quality of its implementation are transferrable to other departments implementing LocSSIPs, and it is hoped that some of our experiences may help other departments who are at differing stages of implementation.