Realistic evaluation of Situation Awareness for Everyone (SAFE) on paediatric wards: study protocol.

INTRODUCTION: Evidence suggests that health outcomes for hospitalised children in the UK are worse than other countries in Europe, with an estimated 1500 preventable deaths in hospital each year. It is presumed that some of these deaths are due to unanticipated deterioration, which could have been prevented by earlier intervention, for example, sepsis. The Situation Awareness For Everyone (SAFE) intervention aims to redirect the 'clinical gaze' to encompass a range of prospective indicators of risk or deterioration, including clinical indicators and staff concerns, so that professionals can review relevant information for any given situation. Implementing the routine use of huddles is central to increasing situation awareness in SAFE. METHODS AND ANALYSIS: In this article, we describe the realistic evaluation framework within which we are evaluating the SAFE programme. Multiple methods and data sources are used to help provide a comprehensive understanding of what mechanisms for change are triggered by an intervention and how they have an impact on the existing social processes sustaining the behaviour or circumstances that are being targeted for change. ETHICS AND DISSEMINATION: Ethics approval was obtained from London-Dulwich Research Ethics Committee (14/LO/0875). It is anticipated that the findings will enable us to understand what the important elements of SAFE and the huddle are, the processes by which they might be effective and-given the short timeframes of the project-initial effects of the intervention on outcomes. The present research will add to the extant literature by providing the first evidence of implementation of SAFE and huddles in paediatric wards in the UK. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

Large-scale multimethod, multi-informant evaluation.

Inclusion of children and parents’ own perspective of paediatric safety and experience of care.

Potential for variability in implementation across sites.

Heterogeneity in contexts for implementation makes interpretation of findings challenging.

Introduction

Evidence suggests that health outcomes for hospitalised children in the UK are worse than other countries in Europe, with an estimated 1500 preventable deaths in hospital each year.1 2 It is presumed that some of these deaths are due to unanticipated deterioration, which could have been prevented by earlier intervention (eg, sepsis). Inconsistent and often suboptimal standards of delivered care across hospitals have been proposed as a critical possible underlying cause.1 There are multiple and complex causes of inconsistent standards of care, including incorrect prioritisation of needs, diffusion of responsibility, delayed recognition or misdiagnosis of deterioration and poor communication between professionals and with patients.3 To provide safe and effective care, health professionals need to be able to proactively assess all of these relevant factors around the child, environment, parent and patient engagement, staff and the tasks required. Such an approach to healthcare delivery requires a dynamic system that places emphasis on proactive rather than reactive care with anticipation of what might happen and containment of possible risks to children's health and their care while in hospital. The Situation Awareness for Everyone (SAFE) intervention aims to achieve this proactive approach. In this article, we describe the realistic evaluation framework within which we are evaluating the SAFE programme. While we appreciate that this is not the only framework that acknowledges the role of context and mechanisms in understanding clinical outcomes4 it offers a coherent framework to draw on quantitative and qualitative data in considering ‘what works for whom in what circumstances’.

Situation awareness is defined as ‘the perception of elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future’.5 Originally a military concept, it has become a core aspect of safety in the aviation industry6 and has recently been translated into healthcare.7 8 In healthcare, situation awareness involves redirecting the ‘clinical gaze’9 to encompass a range of prospective indicators of risk or deterioration, including clinical indicators and staff and parent concerns, so that healthcare professionals can review relevant information and act on it as required. It takes the position that any person involved in the care of the patient could have a piece of information that is critical for understanding the future state of the child. This could include multiple perspectives from consultants (attendings), trainees (residents), nurses, allied health professionals and support staff in addition to patients and their families.

The mechanism SAFE employs to augment team situation awareness around the hospitalised child is the use of ‘huddles’. Huddles involve a suite of interventions to support a ward culture of proactive rather than reactive care. It provides a space for anyone who has any information on the child that may facilitate improving anticipatory awareness. The two core components of a huddle are: (1) a healthcare professional identifying patient risks using standardised tools (eg, a Paediatric Early Warning Scores (PEWS), a script to cover safety concerns) and (2) the ward team evaluating patients with identified risks in a huddle. In practice, the huddle is a rapid structured case review on the ward, which ensures exchanges focused on essential information. Early implementers of huddles7 8 propose three potential levels at which a huddle can occur:

The bedside: a nurse (or doctor) identifies patient risks using standardised tools and clinical judgement.

The ward: the ward team evaluates patients with identified risks in a unit huddle.

The hospital level: nurses from different wards meet with a safety/quality officer in an inpatient huddle to review any unresolved patient risks on their wards.

Evidence from Cincinnati Children's Hospital Medical Centre has shown that huddles are effective at reducing unrecognised clinical deterioration and serious safety events.8 The theory of change for huddles proposes that their effectiveness is achieved through increasing collaborative and efficient information exchanges, fostering a shared clinical view of the current health state of a patient, promoting increased situation awareness and therefore, opportunities to identify plans to minimise risk and prevent deterioration. Another study found that increasing situation awareness lead to ∼50% fewer unplanned transfers to higher levels of care (defined as ‘the transfer of patients from the acute care floor to the Paediatric Intensive Care Unit (PICU) where the patient received tracheal intubation, initiation of vasoactive medications for haemodynamic support or ≥3 fluid boluses in the first 60 min of PICU care or before arrival in the PICU”.7

The SAFE programme that we are evaluating in the UK is based on previous research.7 8 10–12 However, given evidence on the importance of tailoring implementation to local context,13 in the SAFE programme, teams first adapt the theory of developing situation awareness to their own environment and then develop their own approach to performing huddles to meet their local needs. The overarching aim of the evaluation of the SAFE programme in the UK is to understand how situation awareness is developed and whether it has an impact on outcomes for patients, parents and staff. Accordingly, the research question is: under what circumstances, by what means and in what ways might increasing situation awareness lead to improved safety, experience and other elements of quality for children under inpatient care? To answer this overarching question, five specific questions will be addressed:

What does increased situation awareness look like in different contexts?

Is situation awareness associated with improvements in safety outcomes and patient-reported experience of care?

What are the contextual factors within which an increase in situation awareness can lead to improved safety outcomes for children?

What mechanisms explain how improved situation awareness leads to improvements in safety outcomes?

What are the barriers and facilitators to implementation of SAFE in different settings?

Methods and analysis

Design

A realistic evaluation framework will be used, which is a theory-driven framework that aims to explore ‘what works, for whom, in what context and to what extent’.14 It focuses on the ways that the context (ie, settings) of a social intervention interacts with a mechanism of action (ie, underlying processes or structures) to produce outcomes. Multiple methods and data sources are used to help provide a comprehensive understanding of what mechanisms for change are triggered by an intervention and how they impact on the existing social processes sustaining the behaviour or circumstances that are being targeted for change. The evaluation of the SAFE programme will last 3 years. The quantitative arm of the evaluation will focus on outcome data from all participating wards; the qualitative arm of the evaluation will focus on a detailed process evaluation of the ward-level implementation of huddles in a subset of participating wards. Figure 1 shows the SAFE programme theory of change, which the evaluation aims to assess.

Figure 1

The SAFE programme theory of change.

Study design and participants

The SAFE quantitative outcome evaluation draws on data collected on all participating wards from the 12 sites implementing SAFE (six specialist children's hospitals and six district general (community) hospitals). All wards will be collecting safety outcome indicators (see table 1). Within each ward, a sample of around 30 staff members will be asked to complete safety culture surveys at 6-month intervals. The sample will be selected based on the staff lists provided by each ward to ensure representation of medical staff, nursing staff, healthcare assistants and allied health professionals. We estimate the overall number of surveys collected across wards and across time points to be ∼1000. As part of implementation, all parents of children staying on the participating wards (and children themselves where appropriate) are also being asked to complete experience of care surveys, which cover general experience of care and perceptions of safety throughout the implementation period of the SAFE programme (see table 1). Ethnographic field notes will also be taken as part of quarterly implementation site visits to all 12 participating hospitals.

Table 1

Domains, measures and data collection schedule

Domain	Source	Schedule
Implementation of SAFE	Huddle observations	Three time points: early, mid and late implementation
	Staff interviews	Three time points: early, mid and late implementation
Changes to safety culture	Staff safety climate survey	Three time points: early, mid and late implementation
	Staff interviews	Three time points: early, mid and late implementation
	Parent perceptions of safety questionnaire	Rolling basis, throughout the programme
Changes to situation awareness	Staff interviews	Three time points: early, mid and late implementation
Safety outcomes	Number of cardiac arrests on the ward (SCHs only)	Recorded daily, collated monthly
	Number of respiratory arrests on the ward	Recorded daily, collated monthly
Quality outcomes	Transfers to PICU	Recorded daily, collated monthly
	Transfers to higher levels of care	Recorded daily, collated monthly
Experience outcomes	Parent and child experience of care questionnaires	Rolling basis, throughout the programme
	Parent and child interviews	Rolling basis, throughout the programme

SCH, Specialist Children's Hospital.

The SAFE qualitative process evaluation draws on a subsample of two district general hospitals and two specialist children's hospitals. In order to maximise the study of context, these four sites will be sampled to allow variation in geographical region (north and south of the country), ward specialism (High Dependency Unit (HDU) or general paediatric inpatient) and ward size (small and large). The qualitative arm of the study is comprised of three components. First, interviews with five to six parents and children (aged 8 years or older, children younger than 8 will be excluded from the sample) from each of the four sites will be conducted. The parents (or young person, if they are aged 16 or older) will complete an expression of interest form on the ward at the point of their child's discharge, if they are interested in potentially taking part in an interview. The research team will collect the completed expression of interest forms from the sites and will then contact each parent to try and arrange their interview. This sample size is anticipated to be feasible for the research team in terms of their capacity, and also pragmatic in terms of potential difficulties in recruiting parents for interview after their child has been discharged from the ward.

Second, interviews will be conducted with staff, with 15–20 interviews conducted at each site at Time 1 (start of SAFE implementation); a smaller sample of 10–15 staff will be interviewed at each site at Time 2 (6 months after baseline) and Time 3 (12 months after baseline). These sample sizes are anticipated to be feasible for the staff members on the wards in terms of their capacity to take part in interviews (potentially across all three time points) during their shifts.

Third, at these three time points, huddles will be observed and audio recorded. A non-participant huddle Observation Tool (HOT) (J Edbrooke-Childs, J Hayes, E Sharples, et al. Development of the Huddle Observation Tool (HOT) for structured case management discussions to improve situation awareness in pediatric wards. in preparation) (see Measures) was designed to describe the core components of huddles in practice, including the structure of the huddle, the extent of disruptions to the huddle from the external environment, the collaborative culture of the huddle and whether there were clear opportunities to identify risks and discuss concrete plans to mitigate these risks.

Measures

Safety outcomes. To capture safety outcomes, an agreed suite or battery of indicators has been agreed across sites, which is shown in table 2. This suite was produced to allow different study sites to collect relevant data.

Table 2

Ward level outcome indicators and associated definitions

Ward-level outcome indicators	Definition
Cardiac arrests (SCHs only)	Any cardiac arrests occurring on ward defined as staff having to use chest compression or a defibrillator on a patient
Respiratory arrests	Any respiratory arrests occurring on ward defined as staff having to use a positive pressure ventilator on a patient
Unplanned transfers to a higher level of care	Any change in allocation (upwards)/escalation of care including: transfers to high dependency unit, any increase in level of observation and any increase in staffing ratios for that patient
Unplanned transfers to PICU	Includes all transfers: To PICU, except via surgery (elective) To the HDU or PICU where the patient received tracheal intubation, initiation of vasoactive medications for haemodynamic support or three fluid boluses in the first 60 min of PICU care or before arrival

Ward experience of care and safety. To measure ward experience of care and safety, the 10-item ward experience of care and safety questionnaire will be used, drawing on existing measures. The questionnaire captures two domains. First, experience of care received, drawing on a subset of five items from the Experience of Care Questionnaire15 (eg, ‘I feel that the people who have seen my child listened to me’). Although a child self-report version will be used, as the majority of children on participating wards are under the age of 8 (so not eligible to complete the questionnaire), it is likely that most respondents will be parents. Children and parents will respond to items on a three-point scale from 1 (certainly true) to 3 (not true). Second, parents’ experience of safety is assessed via the overall perception of safety subscale of the Hospital Survey on Patient Safety Culture comprising four items16 (eg, ‘The procedures and systems in this unit are good at preventing errors from happening’). Parents will respond to items on a five-point scale from 1 (strongly disagree) to 5 (strongly agree). Both measures have demonstrated reliability and validity in previous studies.15 17 18

Staff Safety culture. To measures staff safety attitudes, an adapted version of the Safety Climate Survey (SCS)19 will be used. The survey used in the present study includes 19 items from the original instrument20 with an additional two items from the Safety Attitudes Questionnaire.20 The SCS measures staff perceptions of safety in their wards and perceived management commitment to safety. Example items include ‘The senior leaders in my hospital listen to me and care about my concerns’ and ‘I am encouraged by my colleagues to report any safety concerns I may have’. The SCS is a widely used measure of safety climate recommended by the Institute for Healthcare Improvement and has demonstrated reliability and validity in previous studies.21

Huddle Observation Tool. This instrument has been developed specifically to capture the team processes that take place during huddles. It comprises four domains, each of which has one item rated on a five-point scale, from 1 (strongly disagree) to 5 (strongly agree), with free text response sections for notes: risk management (ie, ‘Were there opportunities to identify risks and come up with concrete plans for these risks?’), structure (ie, ‘Did the huddle have a clear structure?’), collaborative culture (ie, ‘Did everyone have the opportunity to contribute and were all points of view respected?’) and ‘environment’ (ie, ‘Was the huddle free from distractions?’). Huddles observed using the HOT will also be audio recorded. The HOT has being developed specifically for this evaluation and the data collected through completing the tool during research team site visits. Inter-rater reliability for the tool will be established as part of the evaluation.

Procedure and analytic strategy

Realistic evaluation involves the examination and testing of a theory of change.14 Our procedure for examining each of the four components of the SAFE programme theory of change—input, mechanisms, context and outcomes—is described below. The overarching analytic strategy will involve synthesis across the different strands of data collection to enable triangulation.22

Input. The SAFE evaluation aims to describe implementation at the ward level and how this has been experienced through quantitative and qualitative data capture. General characteristics of implementation will be recorded by each ward (eg, number of huddles per day, attendance of huddles) and reported using descriptive statistics. These data will be complemented with observational assessments captured using the HOT. Audio recordings from all Time 1 observations will be subjected to conversation analysis23 to capture early emerging practice in terms of how huddles take place, what core elements are included, how concerns are conveyed and who leads and steers discussions. The conversation analysis will be conducted to explore what huddle implementation looks like at each site, and to examine cross-site differences and similarities.

Qualitative interviews with ward staff will also provide further information about what implementation of SAFE looks like across a range of wards. All interviews will be transcribed verbatim. The framework approach24 will initially be employed to manage the qualitative interview data set. This will involve sorting extracts from the transcripts into a framework of predefined, ‘top-down’ categories (context, mechanisms, impact and barriers and facilitators). The emergence of ‘bottom-up’ themes within each of these predefined categories will then be explored using thematic analysis.25 Thus, overall, a combination of inductive and deductive analyses will be used. Our specific process will be as follows: two researchers will code the interview transcripts to all of the predefined categories in the top-down framework. One of the researchers will then conduct a thematic analysis of the content coded to a selected category (depending on the research question) and will develop a preliminary thematic framework (bottom-up themes) of the content coded within this category. The second researcher will then recode a subset of the content coded to this category to test the first researcher's preliminary thematic framework. The two researchers will discuss any discrepancies in coding and refine the thematic framework accordingly. In mid and late implementation, exploration of ‘input’ will consider (1) reach, as evidenced by observations of who is discussed in huddles and patients and parents’ awareness and experience of huddles; (2) adoption by ward staff on participating wards, as evidenced through records of huddle attendees from the HOT and staff reports of involvement in interviews; and (3) implementation consistency and adaptation, and the reliable maintenance of practices, based on huddles observations, field notes and staff-reported implementation.26

Mechanisms. The mechanisms being explored are changes in safety culture, and improved situational awareness, captured using the Safety Climate Survey19 (see Measures) and through interviews with ward staff. Descriptive and inferential statistics will be used to examine change over time within wards and differences between wards. Mediation tests may also be conducted, with staff safety culture mediating the potential relationship between implementation of SAFE and changes in safety outcomes. Additional factors of particular relevance, such as increased awareness and better anticipation of risk, will be captured using semistructured interviews with ward staff.

Context. The context includes the range of different conditions within each hospital or ward hypothesised to determine the extent to which input and mechanisms affect outcomes. Each site is different in the type of ward and the acuity of patients. Contextual data will primarily be derived from the routinely collected data from sites, for example, hospital and type, staffing and patient throughput. Staff interviews and interviews with parents and children, in combination with ethnographic field notes taken during implementation site visits, will provide further contextual information.

Outcomes. Change in patient safety outcomes, in terms of incidents of harm, is the high-level outcome of SAFE. Analysis of run charts spanning pre and postimplementation time periods will be carried out to identify any potential patterns (eg, shifts, trends and runs)27 in incidents of harm that might be associated with SAFE implementation. Analysis will be considered at individual ward level and as an aggregate across wards. Similarly, patient-reported and parent-reported measures will be used for group-level comparisons of perceived care between wards within time points, individual wards over time and aggregated across wards over time. Changes in these outcomes, identified based on ward-level run charts, will be triangulated with information about mechanisms of change drawn from quantitative and qualitative data to identify those that appear to be most strongly associated with improvements in safety outcomes.

The qualitative interviews with staff members will capture what the huddle looks like at each site, the impact of huddles and other safety improvement techniques associated with SAFE on the ward environment and on patient safety and barriers and facilitators to implementation and impact from staff members’ perspectives. The qualitative interviews with patients and parents will seek to capture their experiences of care and perceptions of safety on the wards, as well as their perceptions of and opinions on huddles.

Discussion, ethics and dissemination

In this article, we described the realistic evaluation framework within which we are evaluating the SAFE programme. Multiple methods and data sources are used to help provide a comprehensive understanding of what mechanisms for change are triggered by an intervention and how they have an impact on the existing social processes sustaining the behaviour or circumstances that are being targeted for change. The evaluation of the SAFE programme will last for 3 years, is based on previous research7 8 28 and in addition to the overall evaluation involving all participating wards, there is an in-depth qualitative study of a subsample of wards.

Approval was given by each hospital site's local Research and Development office. This ethical approval permits the secondary use of routine data already captured by participating sites (including respiratory arrests and transfers to PICU), data collected as part of the ongoing implementation (experience of care data and ‘improvement data’ collected by participating sites) and the collection of survey data from staff about the safety culture. Ethical approvals also permit interviews with parents, staff and children. Staff and parents are asked to provide written consent prior to involvement in interview. For children under 16 years of age to participate, parental consent and child assent must first be provided in writing.

Findings from the present research will be integrated through synthesis across the different strands of data collection to enable triangulation.22 It is anticipated that the findings will allow better understanding of the important elements of SAFE and the huddle, the processes by which they might be effective and initial effects of the intervention on outcomes. Key recommendations to be discussed will focus on lessons learnt that may affect changes to the current implementation of SAFE and huddles, suggestions for new sites wishing to implement these practices and how early adopters can embed and sustain changes.

Limitations should be considered when interpreting the present research and its future findings.29 A strength and limitation of realistic evaluation is the use of multiple data strands.14 Although this enables triangulation of evidence, and the use of ‘soft intelligence’,29 it is time and resource intensive for participants and researchers. Similarly, contextual differences of the participating wards and differences in tailoring and implementing SAFE and huddles may make it challenging to integrate findings across sites.30 Another limitation is the potential for bias the experience of care surveys; because parents and patients are completing these during contact with the ward, there is a possibility that participants may be less willing to be critical of the care received. In addition, the scale of the project is a great strength but it does mean that some compromise has to be reached between the breadth and depth of the qualitative research carried out. As such there will be limited scope to provide a detailed account of variation in implementation across different times of day and different days of the week. Finally, variation across sites is not only likely to affect implementation and outcomes, it is also likely to impact on mechanisms for data collection, data quality and data completeness.30

Notwithstanding these possible limitations, the present research will add to the extant literature by providing the first evidence of the impact of the intervention of huddles in the UK, a different cultural milieu to the published reports. The results of the evaluation of systematic implementation across different clinical settings and paediatric wards in the UK may contribute to evidence of how to improve the health outcomes for children in the UK and in all clinical settings worldwide.