Evaluation of a multiple-encounter in situ simulation for orientation of staff to a new paediatric emergency service: a single-group pretest/post-test study.

AIM: To assess the utility of a multiple-encounter in-situ (MEIS) simulation as an orientation tool for multidisciplinary staff prior to opening a new paediatric emergency service.
METHODS: A single-group pretest/post-test study was conducted. During the MEIS simulation, multidisciplinary staff with participant or observer roles managed eight children (mannequins) who attended triage with their parent/guardians (clinical facilitators) for a range of emergency presentations (structured scenarios designed to represent the expected range of presentations plus test various clinical pathways/systems). Participants were debriefed to explore clinical, systems and crisis-resource management issues. Participants also completed a pre-intervention and post-intervention questionnaire comprising statements about role confidence and orientation adequacy. Pre-test and post-test results were analysed using t-test and Wilcoxon signed rank test.
RESULTS: Eighty-nine staff participated in the MEIS simulation, with the majority completing the pre-simulation and post-simulation questionnaire. There was a significant improvement in post-intervention versus pre-intervention Likert scores for role confidence and orientation adequacy (p=0.001 and <0.001, respectively); effect sizes suggested the greatest impact was on orientation adequacy. Nearly all scenarios resulted in significant increases in participants' confidence levels.
CONCLUSIONS: The MEIS simulation was of utility in orientation of staff, at least with respect to self-reported role confidence and orientation adequacy. Its effectiveness in practice or compared with other orientation techniques was not assessed, but it did identify several flaws in planned systems allowing remediation prior to opening.

Introduction

Employee orientation is the process of introducing employees to a new work environment and/or new role. Done well, it can play a critical role in facilitating a successful transition with benefits for both staff and organisations. Its importance is such that a workplace orientation programme is a core requirement of the National Safety and Quality Health Service Standards in Australia.

Effective orientation is particularly pertinent in healthcare as the risk of medical error is high when staff are unfamiliar with their job requirements, associated policies and procedures plus the layout and organisational structure of their workplace prior to the commencing clinical duties.1 An environment such as an emergency department (ED), with its high acuity and unplanned patient load which operates continually, presents particular challenges in this respect. These are exacerbated by the requirement to function immediately when a new facility opens.

Workplace change also provokes anxiety in staff experiencing the change.2 High levels of anxiety and decreased self-confidence can themselves affect an employee’s ability to perform.3 Education and orientation of staff prior to commencement in a new or changing workplace can, however, mitigate some of these factors3–5 and help safeguard both staff and patients.

Despite the recognised importance of the orientation process, a recent review has concluded that a standardised framework for delivery of orientation is lacking and that an examination of models and theories that underpin the process is warranted.6 The format for optimal delivery is also not known6: Didactic lectures, peer-delivered case-based discussions, objective structured clinical examination baseline testing, concept mapping and web-based delivery are all possible modes of orientation with varying acceptability and effectiveness.

Some studies have attempted to identify recommendations about components of effective orientation, for example, content, procedures, delivery aspects, timing and social aspects.6–8 Inclusion of the latter is compatible with the TPI theory of orientation where the development of theoretical (T) and practical (P) skills are coupled with meeting the interactive (I) needs of the employees. Additionally, the findings of Ward7 with respect to the feedback from junior medical staff regarding their orientation experiences led him to propose the use of adult learning principles in the development of packages, thus at least basing them on a well-tested educational theory.

With budgetary and time constraints in acute medical care, the challenge for educators and administrators is to achieve effective and efficient orientation while delivering large amounts of information in palatable packages. Simulation offers a possible solution allowing experiential learning with retention of knowledge and skills in a limited time frame.9 It can facilitate communication (a common contributor to clinical errors) and enhance team skills such as role clarity and mutual respect.10 In situ simulation provides the high environmental fidelity necessary for complex team and systems training plus skills learnt are potentially directly transferable to the workplace.11 Using simulation to test a physical space can identify systems, design and equipment errors prior to providing clinical care in that area.12 Furthermore, a multiple-encounter simulation theoretically allows the testing of macrosystem issues surrounding high workload, changing environments and task fragmentation typical of a busy ED and referral service.13 Simulation-based orientation in the form of standardised patient encounters has been used in a few small studies of nurses and doctors entering their first clinical placement with reported success in familiarisation with their role and satisfaction of participants3 14 15 and in orientation of arrest teams to a new facility.16 With respect to road testing new critical care facilities prior to opening, it has been used in situ with individual scenarios to assess the operational readiness of the resuscitation area of a new ED13 and a more ambitious in situ multiple scenario simulation designed to test the macrosystems of new neonatal intensive care environment has been reported, although using existing staff and clinical pathways.17

We hypothesised therefore that a multidisciplinary multiple-encounter in-situ (MEIS) simulation of a busy opening day for a newly built children’s emergency service would be an efficient way of orienting large numbers of staff, both existing and newly recruited. Our primary objective was to evaluate whether the simulation would effectively orient staff to the new environment, new processes and patient flow pathways, and the new teams. Secondary objectives were to identify design flaws and systems issues prior to opening of the department, and to assess self-reported role confidence and anxiety of staff regarding this major workplace change.

Methods

A single-group pre-test/post-test study was conducted to evaluate the adequacy of a MEIS simulation in orienting new and existing multidisciplinary staff to a new children’s emergency and inpatient service. Institutional ethics review board approval was obtained prior to commencement (ref: HREC/12/QPCH/108); all participants provided written consent and data were collected confidentially.

Study setting

The study was performed prior to the opening of a dedicated children’s emergency and inpatient service at The Prince Charles Hospital, Brisbane. This hospital previously had an adult-only ED, followed by a mixed ED with limited paediatric presentations both in number and acuity and no inpatient paediatric facilities. On opening of the children’s service, annual presentations increased from 50 000 (of which 5000 were paediatric) to almost 65 000 (15 000 paediatric) within 12 months. In addition to the existing ED facilities, the purpose-built children’s service comprises a separate triage/waiting room, 12 acute treatment beds, 3 fast-track rooms, special treatment rooms and a new combined trauma/resuscitation bay. There is also a 20-bed combined emergency short stay unit/inpatient children’s ward, plus supporting services, including an adjoining satellite radiology with two new X-ray rooms, CT and ultrasound suites. Staff numbers were increased to meet the projected increase in service delivery resulting from the new department opening, with recruitment of medical, nursing, allied health and support staff.

Participants

A convenience sample of all staff that took part in the orientation exercise was invited to participate in the study. As hospital employees scheduled to commence in the new emergency service, they all had adequate qualifications and experience to meet the requirements of their role in the exercise, but were a mixture of existing staff and those specifically recruited for the new service. A control group was not used as orientation was considered to be too important to exclude any staff facing this major workplace change.

Intervention

The orientation MEIS simulation session was conducted in the newly built unit for a 4-hour period and repeated 1 week later in the month prior to the opening of the new service. Staff were purposefully rostered to one of the two sessions to ensure a good disciplinary mix. Participants in the first session were asked not to discuss the scenarios with their colleagues; however, no quarantine was possible. Although participants were aware that they were attending a simulation exercise, they had no prior exposure to the environment, and were given no information regarding the types of scenarios to which they would be exposed.

Patient scenarios were designed to represent the breadth of ED cases in order to test out and orientate staff to multiple elements of the new service. These included basic flow from triage through to discharge or admission, physical layout via transport of patients to acute treatment areas, radiology and to the ward plus the equipment and information systems. Other processes including the activation and functioning of resuscitation, trauma and medical emergency team (MET) plus consultation and referral pathways for the new department were incorporated into the scenario scripts. A detailed and realistic timeline of patient presentations was written and reviewed by multidisciplinary staff to ensure applicability for learners across the disciplines and adherence to the proposed educational objectives. Prior to the exercise, equipment was prepared including mock charts, labels for the patients and age-appropriate mannequins, equipment and props. Facilitators (staff role—playing as parents/guardians—one per child; and three exercise facilitators) were briefed about their individual scenario objectives, back-story and script plus potential issues and debriefing points relevant to their scenario.

Prior to commencement of the exercise (detailed in online supplementary appendix 1), participants were allocated non-randomised qualification-appropriate roles and labelled with their name and role plus whether they were to be participants or observers. The roles were allocated according to the number of staff that would be available on the opening day of the service to ensure realistic staffing for the exercise. The group received orientation to the new work processes and to the simulation process and then completed preintervention surveys. This was followed by a brief tour of the new facility which was interrupted near completion by a MET call to attend a fallen child outside the building. While the team assembled and dealt with this case, the first few ‘patients’ presented to triage. The exercise progressed through all eight patient scenarios in a time-appropriate manner culminating with a resuscitation team call in the final 15 min of the exercise. Senior staff that had been intimately involved in the development of the service were available throughout the exercise as area line managers and also as observers of the functioning of the new area. A group debrief followed by an individual scenario debrief was performed, using advocacy-inquiry methodology, with the involved participants in front of the whole group so that all had exposure to the learning points and feedback gleaned from the exercise. After the second exercise, the investigators met with senior staff to discuss issues exposed during the exercise that required attention prior to opening.

Measures

A quantitative questionnaire (see online supplementary appendix 2) was used to evaluate the effectiveness of the simulation exercise in improving self-reported confidence in a range of categories of projected patient presentations to the new service plus adequacy of orientation to the team, physical environment and patient flow processes. The pre-intervention questionnaire comprised 12 core statements, of which 8 related to confidence in aspects of role performance (sum score range 8–40) and 4 were concerned with perceptions of adequacy of orientation (sum score range 4–20). Each statement was scored using a 5-point Likert scale, ranging from ‘strongly disagree’ to ‘strongly agree’. There was also an additional statement about self-reported stress using a single statement with a 5-point Likert scale ranging from ‘very comfortable’ to ‘very anxious’. The same questionnaire was administered postsimulation debriefing with two further questions about perceived usefulness and benefit of the simulation exercise.

Data analysis

Data were entered into SPSS V.22 and checked with a 10% recheck (0.003% error rate). Sum scores were calculated for the 12-item scale, and each subscale. The preintervention sum scores were normally distributed and were therefore treated as parametric data for the purpose of further statistical analyses. Significance was set at p<0.05.

Results

Eighty-nine staff participated in the MEIS simulation over the 2 days with a reasonable spread and numbers of disciplines represented (table 1); the majority were paediatric staff (48%, n=43) and the largest professional group was nursing (45%, n=40). In terms of their simulation role, most were participants (57%, n=51), with the remainder (of those who identified their role) being either observers (16%, n=14) or facilitators (11%, n=10).

Table 1

Participants

Professional group	Week 1 (n)	Week 2 (n)
Administration officers	4	4
ED registrar	9	9
ED consultant	5	5
Paediatric consultant	4	2
Physiotherapist	2	2
Radiologist	2	0
RN adult ED	4	1
RN paediatric ED	6	12
RN ward	10	7
Social worker	0	1
Total	46	43

ED, emergency department; RN, registered nurse.

Based on the pre-intervention assessment, the 12-item Overall scale demonstrated very good internal consistency (Cronbach’s α 0.87), with the Confidence subscale performing similarly (Cronbach’s α 0.94), and the Orientation adequacy subscale performing acceptably well (Cronbach’s α 0.77). Similar levels of reliability were demonstrated using the post-intervention data (Cronbach’s α 0.85, 0.89 and 0.84, respectively).

For the purpose of analysis, the sum scores were treated as scale data. Paired sample t-tests were conducted to evaluate the impact of the simulation exercise on participants’ sum scores. Item scores were treated as ordinal data for the purpose of analysis. Wilcoxon’s signed rank test was used to compare differences in pre-intervention and post-intervention confidence and orientation adequacy item scores, and anxiety.

As shown in table 2, overall sum score, as well as sum scores for both confidence and orientation adequacy improved significantly, with moderate to large effect sizes. Most individual item scores also improved significantly, demonstrating medium effect sizes. However, two of the Confidence items did not reach statistical significance: item 1—managing children with minor complaints and item 6—managing children presenting with acute problems needing investigations such as X-ray and CT scan (p=0.052).

Table 2

Confidence and orientation adequacy scores preintervention and postintervention

Scale	Item	Pre-SIM mean (SD)	Post-SIM mean (SD)	n	z or t (df)	Significance p	Effect size
Confidence subscale	Minor: GP	4.02 (0.74)	4.16 (0.58)	75	−1.54	.124	r =0.18
	Minor: OP	3.71 (0.92)	4.09 (0.64)	76	−3.47	.001	r =0. 40
	Acute: SSU	3.71 (0.93)	4.06 (0.66)	77	−3.06	.002	r=0. 35
	Severe: ward	3.64 (0.97)	3.95 (0.77)	78	−2.60	.009	r=0. 29
	Severe: transfer	3.39 (1.01)	3.59 (0.87)	77	−2.20	.028	r=0. 25
	Acute: radiology	3.64 (0.82)	3.86 (0.72)	78	−1.95	.052	r=0. 22
	Resuscitation: retrieval	3.28 (0.99)	3.53 (0.94)	77	−2.12	.034	r=0. 24
	MET call	3.18 (1.01)	3.61 (0.77)	78	−3.19	.001	r=0. 36
	Sum score	28.8 (6.3)	30.8 (4.6)	73	−3.61 (72)	.001	Eta2=0.08
Orientation adequacy subscale	Physical	2.77 (1.23)	3.40 (0.96)	80	−3.84	<0.001	r=0.42
	Equipment	2.74 (2.46)	2.98 (1.04)	81	−4.36	<0.001	r=0.49
	Policy and procedures	2.47 (1.01)	2.82 (0.89)	82	−3.27	.001	r=0.36
	Team	2.95 (1.17)	3.33 (1.02)	82	−2.92	.004	r=0.32
	Sum score	10.6 (3.9)	12.5 (3.2)	78	−4.83 (77)	<0.001	Eta2=0.13
Overall score		39.1 (8.1)	43.2 (6.2)	71	−5.33 (70)	<0.001	Eta2=0.17

GP, general practitioner; MET, medical emergency team; OP, outpatient; SIM, simulation exercise; SSU, short stay unit.

Subgroup analysis showed some differences between groups (ED cf ward; nurses cf doctors cf others; participants cf observers cf facilitators), but these were of unknown clinical significance (results not shown). Of note, although participants showed greater improvement in scores, observers of the exercise, while not actually participating in the duties and debrief, still showed improvement: indeed change in role confidence scores showed no significant difference between the groups. Anxiety was not affected (z=−0.32, p=0.749). The majority of participants either agreed or agreed strongly that the exercise was useful for their orientation (87%, n=73/84, median score 4) and felt that others would benefit from it (93%, n=77/83, med 4).

Discussion

The results of the study suggest that the MEIS simulation of a busy opening day conveyed a large amount of information to staff in an acceptable manner and had some utility as an orientation tool in a number of different ways.

There were increases in self-reported orientation adequacy to the new facility and its processes and team and in role confidence for the majority of the scenarios. The two items where an improvement did not occur were relatively straightforward; one of them (item 1) demonstrated the highest pre-intervention mean score of all items, and therefore had the least room for improvement.

Similar effects on role confidence have been reported elsewhere in the limited literature regarding simulation as an orientation tool. For example, Dearmon et al 3 reported significant increases in knowledge and reduced stress and increased self-confidence in 50 graduate nurses who took part in an orientation simulation in a simulation laboratory prior to their first clinical placement. There is also a report of graduates requiring half the standard orientation time if involved in off-site 10-day simulation training skills course.12 Our study adds to this evidence base in that it occurred in situ, that is, in the actual work environment itself; in addition, it was also a multiple encounter simulation.

Of note, the increase in role confidence occurred whether the staff were playing an active role in the simulation and debrief or were simply observers to the process, suggesting that at least some of the benefit could be gained passively. This aligns with a previous study of laboratory simulations designed to test outcome with respect to knowledge, skills and self-reported confidence for a range of paediatric emergency scenarios, where each individual participated in a small number of scenarios but observed multiple others.18 This is of interest as it potentially allows larger groups to be orientated than there are individual roles within a planned simulation exercise.

Role confidence may be regarded as a surrogate measure of level of anxiety and skill level.4 Increased levels of anxiety can adversely affect task performance and negative performance adversely affects confidence setting up a loop that may further impair ability to perform.3 As major workplace change has also been shown to cause increased stress and poorer sleep in staff,2 reducing this stress through improving role confidence via this sort of orientation simulation exercise can potentially help safe guard the health of both staff and patients.

Interestingly, despite the other benefits, self-reported anxiety levels were in fact unchanged pre-intervention and post-intervention; other studies that have used simulation as an orientation strategy however report the opposite.3 It may be that the survey wording affected the answering of this question (see online supplementary appendix 2). An alternative explanation may be that the participants experienced a transition from fears of the unknown prior to the exposure to the simulation exercise to new, but now known, fears. Although this might not decrease their anxiety, it would at least potentially allow the fear to be addressed prior to service opening, for example, by further education or orientation. This explanation is plausible in that the exercise was still strongly rated to be beneficial to participant and the participants recommended its use in other settings.

Importantly, similar to the microsystems testing of a resuscitation area of a new ED13 and the macrosystems testing of a new neonatal unit17 previously described, the study described herein allowed a range of issues not otherwise identified in the planning process to be revealed via the simulation and debrief. Problems included physical barriers in triage making it difficult to see incoming patients, problems with the buzzer alert system and also with team composition of resuscitation response teams, missing planned equipment in the MET bag and issues surrounding ward transfer processes plus staffing during surge periods. A number of these were able to be remedied prior to opening through post-exercise discussions with the management team. This is a potential advantage compared with other methods of orientation, the benefit of which is self-evident with respect to risk management. Indeed, the utility of in situ simulation for detection of latent safety threats is not restricted to new departments.19

Kobayashi et al 13 also recommended expanding such orientation simulations to a multiple-encounter simulation and hypothesised this would teach compensatory behaviours within a highly changing emergency environment.20 We have shown in our study that this type of MEIS simulation is possible and is of utility in improving role confidence and orientation for staff working in this environment. Its ability to teach compensatory behaviours was not measured, nor was it designed to achieve this but anecdotally the newly formed treating teams adapted to working within the new environment with new processes and effectively managed the patients presented via the scenarios. The use of MEIS simulation allowed testing of multiple processes in the one exercise and it is our contention that it is the only realistic way to test an environment such as an ED for latent macrosystem safety issues.

Limitations

Despite the self-reported benefits of the exercise and its compressed time requirements for participants, it was resource intensive for the facilitators. The key author of the scenario dedicated in excess of a 100 hours to the development, dissemination and delivery of the exercise while associated instructors dedicated approximately 50 pooled hours for scenario instruction, equipment preparation and exercise delivery.

The design of this study was also limited by a number of factors. The intervention did not cover all requirements of orientation6; new staff did however receive standard hospital orientation in addition. The test subjects were a purposive sample of staff available on the exercise days plus subjects had varying levels of other orientation to the department itself prior to the exercise (details not recorded). There was also a mixture of existing staff and new staff with resulting differences regarding both history of working with each other and paediatric experience. In addition, each participant would have had a unique experience of the scenario depending on background and role allocated. While the facilitators had the same script in weeks 1 and 2, instructor practice in both scenario delivery and debriefing may have affected educational delivery between the 2 weeks. All of these factors mean that confounders could not be entirely eliminated. There was also no control group of any kind (either no orientation or orientation of a different format).

The current study could have been improved by more formal recording of qualitative feedback (from debrief and surveys to identify themes, elucidate latent safety threats and potential causes of suboptimal care). Although 5-point Likert scales are widely used, the questionnaire used in this study was designed specifically for this exercise and hence has not been validated. Although the scenarios used have face validity, their educational value with respect to actual practice has not been demonstrated. There was also no formal assessment of the participants’ knowledge/performance pre-exercise, during or post-exercise. Statistical significance may not therefore translate into improved knowledge/performance never mind patient outcomes.

Additional studies with stricter control of these confounders and with a control group plus using validated scenarios and more formal assessment of staff performance (similar to work done by Adler et al for training purpose21) and/or combining other outcome measures as described in recent review22 is now required to strengthen the evidence in this area. Indeed, comparison with other methods of orientation is also warranted, including cost analysis before any formal recommendations for its routine inclusion in healthcare orientation can be justified.

Conclusions

Following the MEIS simulation exercise, self-reported adequacy of orientation to the new service was increased. Although anxiety had not actually decreased, staff confidence in performing their role was improved and they rated the educational intervention as beneficial. Within the limitations of the methodology, given the large numbers of staff and the volume of material covered, it demonstrated some utility as an orientation tool. In addition, some systems and facility errors were identified and remedied prior to the service opening as a result of the simulation.

We would recommend that clinicians introducing new staff to an existing service and/or designing a new service consider in situ simulation as part of the orientation programme. Further research is however needed to compare this approach with standard orientation and evaluate its adequacy, acceptance and cost-effectiveness in both new facility and general orientations.