Visual Attention of Anesthesia Providers in Simulated Anesthesia Emergencies Using Conventional Number-Based and Avatar-Based Patient Monitoring: Prospective Eye-Tracking Study.

BACKGROUND: Inadequate situational awareness accounts for two-thirds of preventable complications in anesthesia. An essential tool for situational awareness in the perioperative setting is the patient monitor. However, the conventional monitor has several weaknesses. Avatar-based patient monitoring may address these shortcomings and promote situation awareness, a prerequisite for good decision making.
OBJECTIVE: The spatial distribution of visual attention is a fundamental process for achieving adequate situation awareness and thus a potential quantifiable surrogate for situation awareness. Moreover, measuring visual attention with a head-mounted eye-tracker may provide insights into usage and acceptance of the new avatar-based patient monitoring modality.
METHODS: This prospective eye-tracking study compared anesthesia providers' visual attention on conventional and avatar-based patient monitors during simulated critical anesthesia events. We defined visual attention, measured as fixation count and dwell time, as our primary outcome. We correlated visual attention with the potential confounders: performance in managing simulated critical anesthesia events (task performance), work experience, and profession. We used mixed linear models to analyze the results.
RESULTS: Fifty-two teams performed 156 simulations. After a manual quality check of the eye-tracking footage, we excluded 57 simulations due to technical problems and quality issues. Participants had a median of 198 (IQR 92.5-317.5) fixations on the patient monitor with a median dwell time of 30.2 (IQR 14.9-51.3) seconds. We found no significant difference in participants' visual attention when using avatar-based patient monitoring or conventional patient monitoring. However, we found that with each percentage point of better task performance, the number of fixations decreased by about 1.39 (coefficient -1.39; 95% CI -2.44 to -0.34; P=.02), and the dwell time diminished by 0.23 seconds (coefficient -0.23; 95% CI: -0.4 to -0.06; P=.01).
CONCLUSIONS: Using eye tracking, we found no significant difference in visual attention when anesthesia providers used avatar-based monitoring or conventional patient monitoring in simulated critical anesthesia events. However, we identified visual attention in conjunction with task performance as a surrogate for situational awareness. ©Arsène Ljubenovic, Sadiq Said, Julia Braun, Bastian Grande, Michaela Kolbe, Donat R Spahn, Christoph B Nöthiger, David W Tscholl, Tadzio R Roche. Originally published in JMIR Serious Games (https://games.jmir.org), 22.03.2022.

Introduction

Continuous patient monitoring in anesthesia is well established in today’s operating theaters and described by the World Health Organization as essential to achieving safe surgical conditions [1]. However, although patient monitoring is a crucial tool for situation awareness in the perioperative setting [2-4], it has several shortcomings, mainly related to the number- and waveform-based monitoring, which may not fully exploit the capabilities of human sensory perception [5-7]. Information overload and alarm fatigue adversely affect care providers’ situation awareness [6-9], potentially leading to critical errors during anesthesia [10].

To enhance situation awareness, new approaches are explored [8,11-15], one being avatar-based patient monitoring [16]. Based on the conventional numerical monitoring data, we created the visual-patient-avatar, representing an animated virtual model of the monitored patient. The avatar abstracts the information and enables health care professionals to assess the patient's condition globally and detect subtle but consequential changes [17]. Computer-based studies found that health care professionals retrieved more vital signs with increased diagnostic confidence and reduced perceived workload when using avatar-based instead of conventional monitoring [17-19]. In addition, a high-fidelity simulation study found that the technology helped anesthesia teams to diagnose what was wrong with the patient more quickly [20]. Moreover, the technology received positive feedback from health care professionals and was rated as easy to learn and use [21,22].

Exploring how health care providers visually interact with new medical devices may provide insights into their usage and acceptance and identify areas for improvement. Furthermore, the spatial distribution of visual attention is a fundamental process for achieving adequate situation awareness [23] and thus a potential quantifiable surrogate for situation awareness [24]. A powerful tool for objectifying the visual attention between users and their environment is eye tracking [25].

Using realistically simulated critical anesthesia events, this study used eye tracking to investigate whether avatar-based patient monitoring influences anesthesia providers’ visual attention on the patient monitor. Based on the accelerated and simplified information transfer found in the previous studies [17-19,26,27], we hypothesized that the anesthesia provider's visual attention on the patient monitor would decrease when using avatar-based patient monitoring. Furthermore, we tested how several potential confounders such as work experience, profession (ie, physician or nurse anesthetist), or task performance in managing the simulated critical events influenced visual attention on the patient monitor.

Methods

Ethics Approval

The Cantonal Ethics Committee of Zurich, Switzerland, reviewed the study protocol and issued a declaration of no objection (Business Management System for Ethics Committees Number Req-2020-00059). We collected all data under written informed consent by all participants.

Study Design

This investigator-initiated, prospective, randomized, eye-tracking study investigated anesthesia providers’ visual attention on the patient monitor during simulated critical anesthesia events. We used three different patient monitor screen modalities (Figure 1). We collected all eye-tracking data during a simulation study conducted in May 2020 at the University Hospital Zurich in Switzerland [20].

Figure 1

Different screen modalities used in the simulation study. a) Conventional number- and waveform-based monitoring. b) only avatar-based monitoring. c) split-screen monitoring, displaying both modalities side-by-side. White boxes indicate our area of interest on the patient monitor used for post hoc semi-automated video analysis.

Study Procedure

We included 52 anesthesia teams consisting of a nurse anesthetist and an anesthesiologist. In randomized order, each team simulated the scenario of severe bronchospasm, myocardial infarction, and malignant hyperthermia once. Each simulated scenario lasted 10 minutes. For each scenario, the teams used a different screen modality: only conventional patient monitoring (ie, number- and waveform-based monitoring), only the avatar-based patient monitoring (visual-patient-avatar), or split-screen monitoring consisting of both modalities shown side-by-side simultaneously (Figure 1). In a randomized order, we chose one of the team members to be the team leader. During the scenarios, mentioned leader (ie, either the nurse anesthetist or the physician) wore a mobile eye-tracking device (Pupil Invisible; Pupil Labs, GmbH) while the team managed the critical incident together. We used Research Randomizer Version 4.0 [28] to randomize the order of the scenarios, respective team leaders, and screen modality.

Simulation Environment

We conducted the study in a backup operating room with an analogous setup as the study center’s active operating rooms. To enhance the simulation fidelity, we used real medications and airway management tools in addition to a state-of-the-art, full human patient simulator (HAL S301; Gaumard Scientific Company, Inc). We used a Philips InteliVue MX500 (Koninklijke Philips NV) patient monitor to closely resemble the study center’s MX550 and MX800 monitors. In the simulation environment, we tagged the patient monitor as our area of interest using pre-generated QR codes. These QR codes enable the Pupil Player software to automatically detect mentioned areas of interest, a prerequisite for automated processing. In Multimedia Appendix 1, we present a video sequence showing an anesthesia team managing a critical anesthesia event, providing a good overview of the simulation environment, the patient monitor used, and the eye-tracking footage.

Data Collection, Video Analysis, and Data Exclusion

Before starting each scenario, we calibrated the eye-tracking device to the participant. We recorded the subject's field of view as a video feed with Pupil Invisible, a mobile eye-tracking device similar in shape and size to regular glasses. The device was connected to a mobile phone that participants carried in their pockets, which served as a power source and storage unit. After each recording, we uploaded the eye-tracking data to a research server and made backup copies on physical hard drives.

For the analysis, we first manually checked the eye-tracking data's quality. Then, for the post hoc semi-automated video analysis, we used Pupil Labs proprietary software Pupil Player on an Acer Aspire V15 Nitro laptop (Acer Inc). Within Pupil Player, we delimited the patient monitor as our area of interest, using the surface tracker plugin. Using the fixations detector plugin and cropping the videos to the start and end of the 10-minute simulation (designated by a bell signal), we were then able to automatically export all fixations and their durations as Microsoft Excel spreadsheets (version 16.58; Microsoft Corporation). During post hoc editing, all recordings were manually stopped five times per recording to ensure the accuracy of the boundaries of the areas of interest. Multimedia Appendix 2 shows an example sequence of the analyzed eye-tracking data in Pupil Player.

Outcome Measures

As our primary outcome, we defined visual attention as fixation count and dwell time. As a fixation, we counted every instance where the subject’s gaze rested in one single location within the area of interest for more than 100ms. The dwell-time corresponds to the cumulative time in seconds that the participant’s gaze was focused on the predefined area of interest. In addition to visual attention, we collected potentially influencing variables such as screen type (only conventional, only avatar or split-screen monitoring), scenario (bronchospasm, myocardial infarction, or malignant hyperthermia), sequence of the scenarios, profession (nurse anesthetist or anesthesiologist), and work experience (in years). Furthermore, we looked at the relation of our primary outcome and the participant’s task performance. The task performance was based on the time required for participants to perform critical diagnostic and therapeutic tasks during the scenarios [20]. An example of such a therapeutic task in the malignant hyperthermia scenario is stopping the trigger or administrating dantrolene.

Statistical Analysis

In this exploratory study design, a power calculation was not performed. For descriptive statistics, we report means with standard deviation and medians with IQRs for continuous data and numbers and percentages for categorical data. We used mixed linear models to analyze fixations count and dwell time. In the models, we included the potentially influential variables task performance, screen type, scenario, sequence of the scenarios, profession, and work experience as covariates. We used R version 4.0.5 (R Foundation for Statistical Computing,) to analyze all data and used Prism 9 (GraphPad Software Inc) to create all figures. We considered a P-value of less than .05 to determine statistical significance.

Results

In May 2020, we recruited 52 teams performing 156 simulations. We excluded 12 teams as the eye-tracking setup was revised because the laminated QR codes used reflected and were not detected by the eye-tracking software. In addition, we had to exclude another team because the video footage was incompletely recorded. Finally, after a manual quality check of the data, we excluded 18 scenarios due to inaccuracies in eye-tracking calibration (eg, alternate blinking or wearing of prescription glasses). This left us with 99 ten-minute video sequences for the eye-tracking video analysis. Table 1 provides additional study and participant characteristics. Figure 2 shows the study setup and the exclusion criteria of the video footage in more detail, and Figure 3 summarizes our results.

Table 1

Study and participant characteristics.

Study characteristics				Values
	Number of simulations conducted, N			156
	Number of eye tracking analyzed, N			99
		Screen modalities, n (%)
		Only conventional monitoring	37 (37%)
		Only visual-patient-avatar	33 (33%)
		Split-screen monitoring	29 (30%)
	Scenarios, n (%)
		Severe Bronchospasm	30 (30%)
		Myocardial infarction	36 (36%)
		Malignant hyperthermia	33 (33%)
Participant characteristics, N				39
	Team leader
		Nurse anesthetist	16 (41%)
		Anesthesiologist	23 (59%)
	Experience team leader (years), median (IQR)			4 (1.5-8)

Figure 2

Study setup and exclusion criteria. We analyzed 99 ten-minute scenarios performed by 39 anesthesia teams. Exclusion of 12 teams because the laminated QR codes used reflected and were not detected by the eye-tracking software; exclusion of 1 team because the video footage was incomplete; exclusion after the manual quality check of 18 scenarios due to inaccuracies in eye-tracking calibration (e.g., alternate blinking or wearing of prescription glasses).

Figure 3

Results for adjusted, mixed linear models a) for fixation counts and b) for dwell time. Both models take the potential influential variables screen type (conventional, only avatar or split-screen monitoring), performance in managing the simulated critical anesthesia events (task performance, in percent), profession (nurse anesthetist or anesthesiologist), work experience (in years), the simulated scenario (bronchospasm, myocardial infarction, or malignant hyperthermia) and the sequence of simulation into account.

Visual Attention on the Patient Monitor

Overall, participants had a median (IQR) of 198 (93-318) fixations on the monitor with a median (IQR) dwell time of 30.2 (14.9-51.3) seconds. This means that participants spent around 5% of their visual attention on the patient monitor screen during the 10-minute simulation.

Visual Attention and Screen Modality

Comparing the fixations on the two avatar-based screen modalities with the fixations to conventional monitoring, the mixed linear model yielded a coefficient of 4.89 fixations (95% CI –59.57 to 69.35; P=.89) for only avatar-based monitoring and a coefficient of -4.33 fixations (95%CI: –74.55 to 65.90; P=.91) for the split-screen modality. This means that participants using only avatar-based monitoring in the simulated scenarios had about five fixations more, and participants using the split-screen modality had about four fixations less than when they used conventional monitoring. However, those results were not statistically significant.

Comparing the dwell time on the avatar-based screen modalities with the dwell time on the conventional monitor using an adjusted, mixed linear model, we found a coefficient of 1.56 seconds (95% CI –8.98 to 12.09; P=.78) for only avatar-based monitoring and a coefficient of –0.02 seconds (95% CI –11.50 to 11.46; P=1.00) for the split-screen modality. This means that participants using only avatar-based monitoring in the simulated scenarios looked around 1.6 seconds longer on the monitor screen. Participants using the split-screen spent as much time on the patient monitor as those using conventional monitoring. However, those results were also not statistically significant.

Visual Attention and Task Performance

Regarding task performance, the mixed linear models showed that the fixation count and the dwell time decreased with better task performance regardless of the screen modality used (fixation: coefficient –1.39; 95% CI –2.44 to –0.34; P=.02 and dwell time: coefficient –0.23; 95%CI –0.4 to –0.06; P=.01). Thus, with each percentage point of better task performance, the number of fixations decreased by about 1.39, and the dwell time diminished by 0.23 seconds.

Visual Attention and Other Potentially Influencing Factors

Regarding the potentially influential variables, profession, work experience, and sequence of the scenarios, the mixed linear models yielded no evidence for differences in fixations and dwell time (Figure 3). However, the scenarios themselves differed with respect to fixation and dwell time (Figure 3). Participants had significantly more fixations (coefficient 141.97; 95% CI 75.62 to 208.32; P<.001) and a higher dwell time (coefficient 23.23; 95%CI 12.48 to 34.08; P<.001) on the patient monitor in the myocardial infarction scenario.

Per-Screen Analysis For Split-Screen Modality

Further, we found that for the conventional half of the split-screen modality, participants had 158 (IQR 63-226) fixations and a dwell time of 24.3 (IQR 10.0-36.8) seconds. For the avatar-based half of the split-screen modality, participants had 44 (IQR 28-84) fixations and a dwell time of 6.8 (IQR 4.3-13.3) seconds. Using a Mann-Whitney test to compare both halves of the split-screen, we found that subjects had significantly fewer fixations (P<.001) and significantly less dwell time (P=.001) on the avatar part of the patient monitor.

Discussion

Overview

This study investigated whether avatar-based patient monitoring influences the visual attention of anesthesia providers on the patient monitor. We assessed 99 eye-tracking videos of anesthesia personnel managing simulated critical anesthesia events. We found no significant difference in visual attention when anesthesia providers used avatar-based or conventional patient monitoring in simulated critical anesthesia events using adjusted, mixed linear models.

Anesthesia personnel devoted about 5% of their time to the patient monitor. These results are consistent with findings under real-life conditions [29]. However, other simulation studies reported higher percentages of dwell time on the patient monitor [29,30]. The high fidelity of our simulations may explain these differences. We used an in-situ simulation design and enhanced the simulation’s fidelity further by using real medications and airway management tools in addition to a state-of-the-art, full human patient simulator. Thus, it is conceivable that a very realistic simulation is more likely to align participant behavior with outcomes under real-life conditions than a simulation with lower fidelity.

We hypothesized that the anesthesia provider's visual attention on the patient monitor would decrease with avatar-based patient monitoring due to accelerated and simplified information transfer found in the previous studies [17-19,26,27]. The basic idea behind this hypothesis is that the qualitative visualization of the patient avatar may lead to a quicker overview of the patient's situation [27,31]. In addition, avatar-based patient monitoring highlights pathophysiological changes, eliminating the time-consuming task of creating a mental model from the various numerical values of the conventional patient monitor [5,6,32]. This may speed up the perception of the situation, lead to fewer fixations and less dwell time on irrelevant vital signs and consequently lead to less visual attention on the patient monitor. In other words, the anesthesia provider knows where to look and can therefore perceive the necessary information with less visual attention. However, this eye-tracking study did not confirm our hypothesis. We found no significant difference in participants' visual attention when using avatar-based monitoring compared to conventional patient monitoring. Unfamiliarity with the novel, avatar-based technology may have masked its potential benefits and may serve as a possible explanation for the finding as all participants used the visual-patient-avatar for the first time. Benefits and acceptance of a new technology depend heavily on users' exposure [33].

However, we found significant differences between the two halves of the split-screen modality. Participants had significantly more visual attention on the conventional part than on the avatar part of the split-screen monitor. This may indicate an interaction effect after all. Perhaps the avatar drew their attention to something (eg, a vital sign outside the norm) that they checked on the conventional screen. Because the qualitative visualization provided by the avatar is intuitive and quickly understood [27,31], participants paid relatively little visual attention to the avatar. To verify the qualitative input, participants had to extract information from the various numbers and waveforms on the conventional screen, a time-consuming task [5,6,32].

Qualitative data collection on how participants used the patient monitors might have helped clarify the result concerning the split-screen modality. Mixed methods are often more powerful than purely quantitative data analysis for such complex human factors work [34].

Visual Attention, Task Performance, and Situation Awareness

We found that an increase in the anesthesia team’s task performance was associated with decreased visual attention. The correlation between the two parameters supports the notion that visual attention and task performance act as indirect indicators of situation awareness [4,30,35-38]. The distribution of visual attention determines what is in the perceptual field and, therefore, contributes to the sensory input, an essential aspect of perception (ie, situation awareness level I) [32]. Good clinical performance in managing the simulated critical anesthesia events comes at the end of good decision-making [38,39], which requires a sufficient understanding (ie, situation awareness level II) and a projection of the situation's near future (ie, situation awareness level III) [32]. A combination of eye tracking and performance measures simultaneously determined what information had been seen and to what degree this information had been perceived and comprehended by the anesthesia provider, giving us a good idea about all three levels of situation awareness achieved (Figure 4).

Figure 4

Illustration of situation awareness in the context of health care. (Adapted from Schulz, C.M. et al., Situation Awareness in Anesthesia: Concept and Research. Anesthesiology 2013; 118:729–742 and Endsley, M.R., Towards a theory of situation awareness in dynamic systems. Hum Factors 1995; 37:32–64) The framework illustrates that adequate situation awareness is a prerequisite for informed decision-making. The acquisition of situational awareness starts with the perception of sensory inputs (mainly visual and auditive). The inputs must be understood, and based on that understanding, a projection must be made on the present and future of the situation. Now good decision-making can occur, leading to good task performance in the clinical context. Individual, task, and environmental factors may influence all levels of situation awareness. As a situation changes over time, a continuous reevaluation is obligatory to maintain adequate situation awareness.

This study may contribute exciting aspects to the current debate on how to best measure situation awareness [4,30,40-42]. Questionnaires to be used during simulations were proposed and validated as direct measurement tools for situation awareness [4,40]. These direct measurement methods require pausing the simulation to answer the questionnaire before resuming the task again [40]. Evidently, this instrument for assessing situation awareness has limited application in clinical reality, as there is no time to stop the treatment of a critical patient to interview the treating physician. For this reason, we propose the combination of the indirect measurement parameters visual attention and task performance as a surrogate when measurements of situation awareness in clinical praxis are wanted.

The three simulated emergencies received different amounts of visual attention. In the myocardial infarction scenario, participants looked at the patient monitor more frequently and had a higher dwell time (Figure 3). In this simulation, there was the additional option of displaying a 12-lead electrocardiogram on the patient monitor, necessary to diagnose myocardial infarction. Although we manually cropped the 12-lead electrocardiogram sequence because we did not analyze it in the context of conventional patient monitoring or avatar-based patient monitoring, participants may have searched for a relatively long time on the monitor to activate the 12-lead electrocardiogram function. This circumstance may explain the significantly increased visual attention on the patient monitor in this scenario.

Furthermore, we found a tendency (not significant) that anesthesia nurses had more fixations and a higher dwell time on the patient monitor (Figure 3). Anesthesia nurses perform important preparatory tasks but generally spend less time with direct anesthesia management. This circumstance may result in anesthesia nurses paying slightly more visual attention to the patient monitor to gain the same level of situational awareness as an anesthesiologist whose main task is anesthesia management.

Strengths and Limitations

Our study had several limitations. First, we had to exclude more than one-third of all simulations from our analysis due to technical issues or poor data quality of the eye-tracking footage. Although we used one of the latest mobile eye-tracking devices on the market, we faced several challenges while recording the data: calibrating the glasses for ocular pathologies (eg, alternating squinting or wearing prescription glasses), battery-life issues, or the device slipping of the participants face during physical tasks (eg, manual resuscitation). This shows that despite the massive development of eye-tracking hardware and software in recent years, the technology is still error-prone. Second, all study participants were unfamiliar with avatar-based patient monitoring. Therefore, the results may vary as anesthesia personnel becomes accustomed to this new technology. Third, the median work experience of our participants is relatively low at four years. Finally, we conducted this study in a university hospital in Europe. Therefore, results may differ under other conditions and in other parts of the world.

The study had several strengths. First, we reduced selection bias by balanced participant selection and consistent randomization of the scenario sequence, team leader, and screen-modality. Second, a rigorous manual quality check of the eye-tracking recordings ensured excellent data quality, allowing us to replace error-prone manual eye-tracking analysis with automated analysis. Third, we attempted to circumvent the bias of authenticity inherent in all simulation-based studies [43] through our in-situ study design and our efforts to represent clinical reality as accurately as possible in this high-fidelity simulation study. Finally, we used eye-tracking hardware that was no more distracting than wearing regular glasses to produce high-quality video footage.

Conclusions

We found no significant difference in visual attention when anesthesiologists used the novel avatar-based or the conventional patient monitoring in simulated critical anesthesia events. However, when using the split-screen displaying the conventional monitoring alongside the avatar-based monitoring, significantly less visual attention was paid to the avatar side of the screen. This may indicate an interaction effect. Perhaps the avatar drew the participant's attention to a vital sign outside the norm that they checked on the conventional screen. Because the qualitative visualization provided by the avatar is intuitive and quickly understood, participants paid relatively little visual attention to the avatar. To verify the qualitative input on the conventional monitor screen seems to have taken more time. In addition, we identified visual attention in conjunction with task performance as a valuable surrogate for situational awareness as it covers all three levels of situational awareness.