Changes in performance during repeated in-situ simulations with different cases.

Introduction: The aim was to describe changes in the performance of clinical actions, during repeated in-situ simulations with different cases, by teams of healthcare professionals with different experiences of the systematic clinical observation of deteriorating patients, after an introduction to the Airways, Breathing, Circulation, Disability, Environment/Exposure (ABCDE) approach.
Methods: A descriptive observational study was conducted of repeated in-situ simulations using a patient simulator (SimMan 3G), carried out by teams in a public nursing home (NH, least experienced), an out-of-hours general practice (OOH-GP) service and a hospital emergency department (ED, most experienced). The cases had similar clinical presentations but different underlying diagnoses unknown to the teams. Four blinded clinical experts independently assessed the simulations on the basis of transcripts, providing comments, an overall score and scores for the clinical actions.
Results: The assessors commented on the overall lack of a systematic ABCDE approach in the NH and OOH-GP in all simulations, while the comments for the ED concerned the choice of treatment. Across the teams, the overall score was highest in the first simulation and second highest in the third simulation. The team in the NH received low overall scores for all simulations, but the last simulation received markedly better scores on the clinical actions. The teams in the OOH-GP and ED had no such clear pattern in the scores for clinical actions and thus no indications of improvement with repeated simulations.
Conclusion: The observation in this study was that the overall assessment by the blinded assessors showed no consistent improvement in clinical actions from repeated in-situ simulations, and the teams did not seem to adhere to the ABCDE approach throughout the simulations. This indicates that the teams were not able to apply their newly acquired experiences of using the ABCDE approach from one case to another, different case. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

There is some evidence for the effectiveness of repetitive in-situ simulations of the same clinical case in high-risk situations, but no studies have looked at repeated in-situ simulations focussing on the ABCDE (Airways, Breathing, Circulation, Disability, Environment/Exposure) approach using different cases.

Doing repeated in-situ simulations does not consistently change performance of clinical actions.

It is likely that practicing a generic skill with different cases increases the cognitive load to such an extent that teams are unable to apply their newly acquired experience from one case to another, different case.

Introduction

The early detection and treatment of patients with a rapidly deteriorating acute condition is a skill needed by professionals in different parts of the healthcare services1 because it can prevent hospitalisation or death.2 This is especially so for elderly people susceptible to acute illnesses3 and studies have indicated that clinical deterioration needs to be identified earlier.4 Skills in systematic clinical observation are needed in all parts of the services, from nursing homes to emergency departments.5 6 However, there seems to be a lack of competence among healthcare workers in this regard.2 7

One approach to systematic clinical observations that is generic and can be used in different settings is the Airways, Breathing, Circulation, Disability, Environment/Exposure (ABCDE) approach.8 This approach is used to triage patients in an acute disease phase and for early recognition of deterioration.9 10 As with any skill,11–13 practicing it several times is necessary to learn and improve competence.

One solution for repeated practice among healthcare professionals in different settings is to provide simulation training.14 15 This includes in-situ simulation, which is simulation conducted in the practice field in order to resemble real-life situations.16 This makes it possible to practice the same skill several times adapted to the work situation by, for example, changing the cases used in the simulation to mirror the real-life need to be able to conduct a generic skill across different situations. We have only identified a few publications on the outcome of repeated in-situ simulations for health professionals and they have been conducted in rare, high-risk situations using the same clinical case.12 17 18 We have not found publications describing changes in performance of repeated in-situ simulations focussing on the ABCDE approach with different clinical cases. Such studies are needed to learn more about what goes on during repeated in-situ simulations aimed at improving competence in systematic clinical observation.19

The aim was therefore to describe changes in the performance of clinical actions, during repeated in-situ simulations with different cases, by teams of healthcare professionals with different experiences of systematic clinical observation of deteriorating patients, after an introduction to the ABCDE approach.

Methods

This was a descriptive observational study with blinded expert assessment of clinical actions in three consecutive in-situ simulation performances, 1 week apart, by three teams of healthcare professionals working in three different healthcare units. The data were collected in February 2017 in a Norwegian city with 50 000 inhabitants.

Participants

The inclusion criteria were health professionals working in a nursing home, an out-of-hours general practice (OOH-GP) service and a hospital emergency department to ensure variation in the experiences of systematic clinical observation of deteriorating patients. To be included, the unit had to be in the city, handling older patients with emerging acute conditions and able to allocate time and resources to participate in the in-situ simulations. There were no exclusion criteria.

There is only one OOH-GP service and one hospital emergency department in this city, and they volunteered to participate. To recruit a nursing home, the municipality’s healthcare department was contacted, and a nursing home was then selected in a meeting with its managers. The leaders of the participating units recruited the health professionals.

Introduction and simulation

The teams participated in one introduction session, which was held 3 weeks before the first of the three repeated in-situ simulations, which were carried out 1 week apart.

The same introduction was provided separately for all participants by an experienced simulation facilitator (author RB). The introduction focussed on getting to know the simulator manikin (SimMan 3G, Laerdal Medical, Norway), understanding the ABCDE approach for clinical observations and conducting an in-situ simulation (table 1).

Table 1

Description of the introduction, in which all the teams participated before the in-situ simulations. The introduction lasted 2 hours

Duration	Learning goals	Activity
20 min	Familiarity with the patient simulator (SimMan3G).	Presentation of and interaction with the simulator manikin and the functions relevant for ABCDE observation.
20 min	Understanding of the ABCDE approach for systematic clinical observation.	Lecture on how to observe a patient systematically using the ABCDE approach. Written handouts including an overview of the ABCDE method were provided.
20 min	Knowledge about simulation as a method.	Lecture on simulation with presentation of the phases—briefing, simulation and debriefing—explaining the method and purpose of each phase.
60 min	Experience in performing in-situ simulation.	Conducting a simulation with briefing, simulation and debriefing phases. The clinical case involved heart failure and participants were instructed to focus on the ABCDE approach.

ABCDE, Airways, Breathing, Circulation, Disability, Environment/Exposure.

To reflect real-life situations, three different cases were used with similar clinical presentations but with different underlying diagnoses, based on suggestions from the teams (see Results section). All teams simulated the same clinical case in each round of the simulations and were not informed about the underlying diagnoses. The cases were further developed by the facilitator (author RB), who cooperated with the unit managers, all of whom were advanced nurses, on how to make the clinical presentation, including progression of deterioration, relevant for each unit. The cases were then validated by an experienced cardiologist.

During the simulation, the facilitator (author RB) administered the use of the manikin, including being the voice of the patient, expressing signs of distress according to the development of the case, asking questions and responding to questions from the team to increase the realism. The simulator manikin was transported in and out of the units by an ambulance team who acted as though the simulation was a real-life situation to add authenticity.

Each simulation included a briefing phase, a simulation phase and a debriefing phase. In the 10 min briefing phase, the facilitator systematically went through the simulator’s features using the ABCDE approach. The participants were invited to ask questions and to try out the simulator manikin’s features during the briefing phase.

The simulation phase in the nursing home started with the nurse on night watch informing the day team about a new ‘patient’. The ‘patient’ was registered in the electronic health record system. In the emergency department and in the OOH-GP service, the simulation phase started with the handover from the ambulance team, who reported the patient’s case history and clinical status. During the simulation phase, the teams operated on their own, using the available equipment at the unit. The simulation phase ended when the facilitator considered the simulation situation to be under control. In the nursing home, this was when the ambulance team arrived, and in the OOH-GP service and the emergency department, when the ‘patient’ was stabilised or revived after cardiopulmonary resuscitation (CPR).

The 30 min debriefing phase focussed on systematic observations using the ABCDE approach. The specific clinical actions performed during the simulation were also discussed, but infrequently.

Data collection

The in-situ simulations were video recorded to identify the clinical actions performed by the teams. The videos were transcribed into what was said and done and then broken down into their smallest parts while still containing enough information to be meaningful (table 2).

Table 2

Excerpt of transcript from a simulation in the OOH-GP service. The clinical actions are clearly marked in the transcripts

Line number	Activity
57	Nurse 1 approaches the patient and informs them that they are to receive morphine 5 mg.
58	The physician looks at Nurse 1, who injects the morphine.
Clinical action 1	5 mg of morphine is given intravenous
59	Nurse 2 says, ‘Should I turn on some oxygen?’
60	The physician answers, ‘and so some oxygen yes’.
61	Nurse 2: ‘How much will you give him then?’
62	The physician: ‘Why don’t we try a couple of litres first, then we can see.’
63	Nurse 2 gives 2 litres of oxygen
64	on a double oxygen catheter.
Clinical action 2	Two litres of oxygen are given on a double oxygen catheter.

OOH-GP, out-of-hours general practice.

All clinical actions, as highlighted in table 2, were identified using the following self-developed definition: ‘A concrete action taken to stabilise, ease or take over the patient’s vital functions, relieve physical discomfort and pain attached to the condition and initiate treatment measures’. Authors HB and AS used this definition to independently identify the clinical actions in all the transcripts, discussing the few differences between their identifications until achieving consensus.

Assessment of outcomes

Four clinical experts assessed the outcomes. These assessors were selected on the basis that they did not work together, they were not involved in the project and they had experience of both simulation and emergency medicine. To recruit them, the project team contacted people they knew to meet these criteria. The assessors consisted of three postgraduate intensive care nurses and one medical doctor (one female and three males). One of the assessors had a PhD degree. One worked in a hospital emergency department, one in a university medical education facility, one in a university hospital and one in a simulation company.

The assessors were blinded to the specific learning objective, the diagnoses, the teams, the units, the order of the simulations and each other’s assessments. This was achieved by removing references to teams, units and diagnoses and providing the transcripts in a random order. The assessors used only the transcripts as the basis for their assessments.

The outcome measures were developed by HB and AS and included an overall score for each simulation, a comment on the whole simulation and an assessment of each of the clinical actions (table 3). The performance of the clinical actions was chosen because they represented the outcome of the clinical observations; there is little use in making correct observations if they do not prompt correct actions. This approach is in line with the demand for studies measuring an intervention’s impact on perceived or real knowledge or performance.14 20 Thus, the underlying assumption was that using the ABCDE approach would help improve the clinicians’ clinical actions.

Table 3

Questions for the assessors to answer for overall assessment of each complete simulation and each clinical action

Area	Question	Scale
Overall assessment score for each complete simulation	What is your overall assessment of the treatment of the patient in this simulation?	1–10 scale: worst possible to best possible
Written overall assessment	What is your short overall assessment of the treatment of the patient in this simulation?	Free text
Assessment of each of the clinical actions	Was this the correct choice of clinical action based on the preceding situation?	Yes/no/unknown
	Was the correct method of administration chosen?	Yes/no/unknown
	Was the clinical action in accordance with the preceding observations?	Yes/no/unknown
	What is your assessment of the choice of the clinical action?	1–10 scale: worst possible to best possible
	What is your assessment of the method(s) of administration?	1–10 scale: worst possible to best possible

Analysis

The written comments to the open-ended question were categorised according to the topic concerned. To conduct the analysis, authors HB and AS read all the comments. HB made suggestions for categories, which were reviewed and discussed with AS. The assessors could not comment on the changes in performance because they were blinded to team and order, so changes in performance were identified by analysing the assessors’ comments from one simulation to another. However, it was not possible to identify any meaningful differences between the simulations within each team. The analysis therefore focussed on which topics the assessors commented on for each team to identify the overall pattern.

The total of all scores in the analyses of the clinical actions comprised the scores from the assessors for each observation (292 scores: 73 clinical actions x 4 assessors). The quantitative outcomes are presented with means and SD. To evaluate the assessors’ agreement, the interclass correlation coefficient was calculated using the average Cronbach’s alpha for the three continuous outcomes: (a) clinical action, (b) administration and (c) overall score. The assessments for each clinical action were analysed using the Kruskal-Wallis test, with comparisons of the outcomes for each simulation within each team. Due to the descriptive nature of this study, statistical tests are not presented for each analysis. Statistical analyses were carried out using SPSS V.25.0 (Armonk, New York: IBM Corp).

Results

Three teams from three units participated (table 4). Each team was composed of three healthcare professionals. One nurse at the nursing home and one nurse in the emergency department had to be replaced in one of the simulation sessions, and one physician could not participate in one simulation session at the emergency department. None of the participants were familiar with simulations using an advanced simulator, but some had experienced simulations as part of their educational training. One simulation in the OOH-GP and two in the emergency department ended in situations requiring CPR.

Table 4

Description of the clinical values and patient and team characteristics for each simulation in each unit

	Nursing home			Out-of-hours GP service			Emergency department
	Pulmonary emboli	Sepsis	Heart failure	Pulmonary emboli	Sepsis	Heart failure	Pulmonary emboli	Sepsis	Heart failure
Clinical values (range):
Systolic blood pressure	120–81	120–81	114–78	120–90	115–90	110–0	117–90	120–78	109–0
Diastolic blood pressure	80–40	80–32	73–35	80–50	78–50	70–0	76–40	80–30	66–0
Pulse (heartbeat)	80–108	98–112	56–32	81–85	81–85	52–0	81–67	98–47	48–0
Saturation (SO2)	98–85	97–86	95–80	81–85	81–85	52–0	91–0	91–50	93–0
Respiration	16–33	23–33	23–11	16–33	23–32	22–0	24–0	23–8	28–0
Temperature	37.2	38.0	37.4	37.5	38.0	37.2	37.0	38.0	37.0
Patient characteristics
Gender	M	M	M	M	M	M	M	M	M
Age	75	77	77	75	77	77	75	77	77
Name	Jan	Nils	Per	Jan	Nils	Per	Jan	Nils	Per
Team members
Number of registered nurses	2	2	2*	2	2	2	2	2*	2
Number of enrolled nurses	1	1	1	0	0	0	0	0	0
Number of physicians	0	0	0	1	1	1	0	1	1

*One nurse was replaced with another nurse from the unit.

GP, general practice; SO2, oxygen saturation.

Analysis of assessors’ comments

For the nursing home, the assessors mainly commented on the absence of a systematic ABCDE approach. Typically, it was commented that the health professionals started to perform an observation in one area but did not follow the order systematically as the case developed. It was noted that this absence of a systematic approach led to wrong decisions and inefficiency, and that the participants spent too much time not initiating clinical actions.

‘They start well with ABC, but do not follow-up. Did not finish with one letter before continuing’ (Expert 1, commenting on the overall assessment in the nursing home).

For the OOH-GP service, the assessors also commented on the lack of a systematic ABCDE approach; however, there were more comments on the choice of actions and lack of observations and whether they should have administered other medicines.

‘(They should have) evaluated blood pressure before giving such a high dose of morphine’ (Expert 4, commenting on missing observations in the OOH-GP service).

For the emergency department, there were fewer comments on the lack of a systematic clinical approach compared with the other units. The written assessors’ comments were mostly related to the chosen treatment.

‘(They) gave Plavix orally when the patient was nauseous and might have thrown up, instead of giving Afipran first, and then Plavix. The dose (for those aged) >75 years is 75 mg (300 mg was given)’ (Expert 1, commenting on missing actions in the emergency department).

Outcomes

The interclass correlation coefficient for agreement between the assessors was 0.640 for the clinical action scores, 0.614 for the administration scores (both of which indicate moderate reliability) and 0.803 for the overall scores, indicating good reliability.21

In the nursing home, the number of clinical actions conducted by the team was similar across all simulations, with an average of 3.6 (table 5). They spent 15 min on average on each simulation, with a two-times-2 min reduction in duration from the first to the last. The overall score for each simulation was generally low, with the highest score for the first simulation. The last simulation (heart failure) received markedly better scores than did the first two simulations on correct clinical actions and administration, and the accordance between these. The scores for clinical action and administration were rather low for the first two simulations but high and close to the level of the teams in the OOH-GP and emergency department for the last simulation.

Table 5

Assessment of the simulation as a whole, the performance of clinical actions and the methods of administration of the three simulations in each of the teams in the participating units. The p values for the test of difference within each team varied between 0.148 and 0.649 (details not shown). Numbers are percentages (SD percentage points) or average scores (SD)

Unit and case	Number of clinical actions	Time used to complete simulation (minutes)	Average overall score (SD) (range 1–10)	% of correct clinical actions (SD)	% of correct administration(SD)	% of correct accordance between observation and clinical action (SD)	Average score for clinical actions(SD) (range 1–10)	Average score for administration(SD) (range 1–10)
Nursing home
Pulmonary emboli	4	17	4.8 (1.3)	60% (13)	35% (35)	75% (29)	4.9 (1.1)	5.5 (1.7)
Sepsis	4	15	3.3 (1.0)	50% (46)	63% (32)	63% (48)	4.2 (2.3)	4.3 (2.6)
Heart failure	3	13	4.5 (2.1)	83% (14)	92% (14)	92% (14)	6.1 (0.3)	7.7 (0.4)
Out-of-hours GP service
Pulmonary emboli	5	11	7.8 (2.6)	95% (11)	100% (0)	95% (11)	7.8 (0.7)	8.0 (0.9)
Sepsis	4	12	4.8 (1.7)	69% (38)	88% (25)	77% (31)	6.0 (2.5)	6.5 (2.3)
Heart failure	14	12	5.5 (1.3)	89% (19)	84% (21)	89% (16)	7.4 (1.9)	7.7 (1.6)
Emergency department
Pulmonary emboli	13	16	7.0 (1.2)	85% (24)	87% (28)	90% (16)	7.7 (1.8)	7.7 (2.0)
Sepsis	7	14	6.3 (1.0)	76% (18)	85% (21)	100% (0)	7.0 (1.2)	7.1 (1.3)
Heart failure	19	20	6.3 (0.5)	88% (18)	91% (21)	90% (18)	7.7 (1.6)	8.2 (1.5)

GP, general practice.

The team in the OOH-GP unit performed 7.6 clinical actions on average, but with marked variation, the last simulation having approximately three times as many clinical actions as the first two because of the need for CPR (table 5). Nonetheless, the time spent on the three simulations was almost identical, between 11 and 12 min (table 4). They obtained their best scores, which were also high, on all outcomes, including the overall score, in the first simulation (pulmonary emboli). They obtained the lowest score for the second simulation, obtaining scores for the last simulation that were closer to the first simulation, except for the overall score (7.8 on first simulation vs 5.5 on the last).

In the emergency department, the team performed 13 clinical actions on average in each simulation, with the highest number for the two cases in which they had to perform CPR. There was thus also some variation in the time taken. The overall score was somewhat higher for the first simulation, but in general they had a consistent high proportion of correct clinical actions and methods of administration and a consistently high score on all simulations. However, there was no clear pattern of one simulation being better than the others.

For all teams, the overall score was highest in the first simulation (pulmonary emboli), while both clinical actions and administration received the highest score in the last simulation (heart failure), although the score was not markedly better than that for the first simulation for the OOH-GP and emergency department. In addition, all teams received the lowest score for the second simulation (sepsis) across all outcomes, the only exception being accordance between clinical action and administration for the team in the emergency department, which was slightly higher in the second simulation.

It was observed that the staff became more observant of shortcomings in their competencies and routines. The nursing home prepared an emergency cart after the first simulation. Both the nursing home and the OOH-GP service prepared a form after the first simulation that summarised the main points of conducting a systematic observation based on the ABCDE approach.

Furthermore, it was observed that the teams focussed on identifying the underlying diagnosis more than on the ABCDE approach. One observed consequence of this was that one of the teams performed clinical actions targeting cardio/pulmonary problems in the sepsis case instead of making systematic observations that could have helped them identify the correct actions on the basis of observed clinical values and conditions.

Discussion

The assessors commented on the overall lack of a systematic ABCDE approach in the nursing home and the OOH-GP in all simulations, while the comments for the emergency department concerned the choice of treatment. Across the teams, the overall score was highest for the first simulation and second highest for the third simulation. The team in the nursing home obtained low overall scores for all simulations, but obtained markedly better scores for the clinical actions in the final simulation. However, the teams in the OOH-GP and emergency department had no such clear pattern in their scores for clinical actions and thus there were no consistent indications of improvement during the repeated simulations.

Strengths and limitations

The use of blinded external assessors constitutes the main strength of this study; most studies report on students’ or health professionals’ self-reported learning experiences,22 23 and studies have compared self-reporting and objective measurements without finding any correlation between these.24 25

The major weaknesses were the number of teams, the relatively low number of observations for each team, the absence of a control group and the lack of prior validation of the questions that formed the basis of the assessments. In two of the simulations, participants were replaced by colleagues and this may have affected the results. However, the assessments of those two simulations were similar to the others.

The use of transcripts as the basis for the assessments standardised the data provided to the assessors and prevented them from identifying the teams they were assessing in each simulation. While watching videos of the simulations might have given the assessors a better understanding of what was done, it is not likely that doing so would have increased the difference between each simulation.

Discussion of findings

The main observation in this study was that repeated in-situ simulations with different cases focussing on the ABCDE approach did not show a consistent improvement in the teams’ performances of clinical actions in any substantial way. However, the team with the least experience of systematic clinical observation of deteriorating patients (nursing home) was observed to derive the most benefit from the repeated in-situ simulation. The reason for the lack of change in performance for all teams may be ascribed to several reasons, besides the methodological limitations mentioned above. One explanation may be that repeated in-situ simulations do not consistently improve the performance of clinical actions. However, this is contradicted by the improvement seen in the team in the nursing home for the specific outcomes, findings from other studies12 17 18 and common wisdom that repetitions lead to improvement. Therefore, we do not find it likely that the lack of improvement in this study was due to a general lack of effect of repeated in-situ simulations.

A more likely explanation is that we used different cases from one simulation to the next, which could have removed or diluted the effect of constant repetition. Using different cases with different underlying diagnoses, although with similar clinical presentations, meant that the clinical actions needed to change from one simulation to the other. It was also unsystematically observed that the teams were more concerned about the diagnosis than about adhering to the ABCDE approach. If this dilutes the effect of repetition, it could mean that doing the same type of simulation, but on a new case, makes it difficult to apply a newly acquired skill. This could be caused by cognitive overload, whereby learning is not achieved due to the number of things happening, leading to an overburdening of the working memory.26 The likely explanation would then be that practicing the ABCDE approach in new situations (simulations and unknown cases) resulted in cognitive overload. One observation supporting this is that the experts stated that the teams did not follow through on the systematic ABCDE approach throughout each simulation. One solution to this may be to organise the learning tasks from the simple to the more complex, moving from low-fidelity to high-fidelity environments24 and thereby supporting the build-up of complexity to reduce the initial load and improve competence more gradually.27 28 In practice, this could mean more ABCDE training on the same case in a less demanding environment before increasing variability and contextual interference.

The results may also indicate that the teaching and information applicable for using the ABCDE approach was not good enough, at least not to penetrate into the simulation. A randomised controlled trial on the learning effect of using simulations versus didactic methods to teach the ABCDE approach found that simulations were a more effective strategy for learning assessment skills.25 Other studies have found an effect of interventions for learning the ABCDE approach performed in lower-fidelity environments.29 30 Nevertheless, even if more specific ABCDE training in low-fidelity environments was used (ie, not in a full simulation), and this improved the participants’ procedural skills, the low-fidelity environment would not represent the situation faced when a real patient needs to be observed.

Conclusion

In this study, the overall evaluation conducted by the blinded assessors found no improvement from repeated in-situ simulations, which did not consistently improve performance of clinical actions, even though the team with the least experience of systematic clinical observation of deteriorating patients scored best on their last simulation. The teams did not appear to follow the ABCDE approach but focussed on the underlying diagnoses. This indicates that the teams were unable to apply their newly acquired experience of using the ABCDE approach from one case to another, different case.