Feasibility and Utility of an Eye-Tracking Device for Assessing Teachers of Invasive Bedside Procedures.

Patient-related complications from invasive bedside procedures (IBPs) are attributed to the experience and proficiency of the operator. Furthermore, IBP complications by trainees may be due to practice variability and competency among IBP teachers. The use of gaze metrics technology to better understand the behaviors of IBP teachers may aid in the creation of faculty development checklists and, ultimately, reduce procedural complications. Prior research on gaze patterns has focused on the individual performing the procedure, but the goal of this pilot study was to assess gaze behaviors of supervising teachers of IBPs, which is a paradigm shift within procedural education. In this study, pulmonary and critical care medicine fellows placed a central venous catheter on a simulated task trainer as pulmonary and critical care medicine faculty supervised while wearing an eye-tracking device. Both quantitative and qualitative data were obtained. Gaze analysis was divided into 2 areas of interest (ultrasonography and procedure site) and 3 procedural tasks (venous puncture, dilation, and flushing the line). Study findings included the following: (1) calibration was easy and took seconds to complete, (2) the device is relatively comfortable and did not interfere with tasks, (3) a trend toward a higher fixation frequency and dwell time on the ultrasound images during the puncture segment, and (4) variations in fixation frequency on the ultrasound images among supervising IBP teachers. This study documents the feasibility of the eye-tracking device for assessing behaviors of supervisory IBP teachers. There may be a signal suggesting differences in gaze patterns among supervisory teachers, which warrants further study.

Preventable medical errors and serious complications still occur frequently in modern medicine. Procedural complications are the second most common cause of adverse events in hospitalized patients, and technical errors are the most common cause of procedure-related mistakes. In the intensive care unit (ICU), invasive bedside procedures (IBPs) are unpredictable and often occur urgently. Complications from IBPs are thought to be avoidable and largely attributable to the operator’s experience and proficiency.1, 2, 3

At academic centers, IBPs are frequently performed by trainees who are supervised by faculty physicians. Various studies have found that supervision improves proficiency and safety and reduces preventable complications. Effective IBP teachers have a recognized skill set. However, despite the presence of supervising teachers, complications persist. One reason for this problem may be related to variations in procedural practice and skill level among IBP teachers.

Objective data could elucidate the distinctive perspectives and behaviors of effective IBP teachers in the ICU. One method for obtaining this data is through the use of eye-tracking devices, which, in other studies, have been associated with effortful thought processing.7, 8, 9, 10, 11 Eye-tracking involves analyzing eye movements and the behaviors of the pupils using infrared cameras. Gaze metrics provide valid, reliable, and objective indices to assess the technical performance of various professionals,12, 13, 14 including physicians. Gaze metrics have been used to quantify surgical skills, to study differences between novice and experts surgeons,, and to analyze surgeons’ scanning behaviors., To date, the use of eye-tracking devices within the medical field has predominantly focused on the individual performing the procedure, namely, the proceduralist. Thus, studying the gaze metrics of supervising teachers is a paradigm shift within procedural education. The goal of this pilot study was to understand the feasibility and utility of an eye-tracking device as a means for assessing the behaviors of IBP teachers.

Methods

Study Setting and Participants

Volunteer pulmonary and critical care medicine (PCCM) fellows and faculty were solicited via email as part of an educational quality improvement initiative. This study was deemed exempt by our institutional review board. Verbal consent was still obtained for all participants. This pilot study was conducted in the Multidisciplinary Simulation Center at Mayo Clinic in Rochester, Minnesota. During the scenarios, fellows were asked to place a central venous catheter (CVC) on a task trainer using ultrasound guidance. Location of the CVC was the right internal jugular vein because it is the most frequent site used within our practice. The simulation room was constructed similar to what is commonly encountered in our medical ICU, including the CVC kit, sterile fields, ultrasonographic equipment, and cardiac monitor. During the procedures, faculty physicians wore the eye-tracking device while supervising the fellows.

Eye-Tracking Device

The Pro Glasses 2 (Tobii AB) eye-tracking device was used for this study. This device has a temporal resolution of 100 Hz, allowing for 100 gaze samples to be collected every second. The glasses consist of a 1920 × 1080 high-definition scene camera that records the environment with a first-person view and 2 infrared cameras aimed at each eye for binocular tracking. Near-infrared light illuminates the eyes and creates reflection patterns on the cornea and pupil, and the cameras capture images of the eyes and reflection patterns. Point of gaze is represented by a circular cursor and is overlaid on the video when replayed (Figure). Data were downloaded from the SD card onto a secured computer and analyzed with Tobii Pro Lab software.

Figure

Tobii Pro Lab software analysis. A, View of recording with the creation of the area of interest (ultrasonography and procedure site). B, View of recording with red circle as cursor demonstrating point of gaze of the supervising teacher.

Pilot Outcomes

The primary outcomes of this pilot study were to test the feasibility and utility of the eye-tracking device, and both quantitative and qualitative data were obtained.

Feasibility Assessment

We conducted a calibration assessment with 10 random participants to assess the ease of calibration (calibration success and number of attempts) and comfort of the device itself (Likert scale of 1-10, with 1 being uncomfortable and 10 being comfortable). Only participants who wore contact lenses or did not wear glasses were assessed for calibration. Qualitative data included feedback from participants via a post-pilot survey that was distributed immediately after the simulation session and during the calibration assessment.

Gaze Metric Outcomes

The simulated CVC procedure was divided into 3 segments for gaze metric analysis: (1) venous puncture (use of an introducer needle to gain access to the internal jugular vein), (2) dilation of the vein (use of the dilator to allow easy insertion of the CVC at the internal jugular vein), and (3) flushing the CVC lines (aspiration of venous blood and clearing the CVC lines to ensure adequate venous flow). The procedure was broken down into these segments given the potential for human error and risk of complications such as inadvertent arterial puncture during attempted venous puncture, pneumothorax during dilation of the vein, and air embolism during flushing of CVC lines, thus highlighting the importance of appropriate supervision. Other critical portions of the procedure, such as local analgesia and confirmation of venous entry, could not be assessed easily given the limitations of the task trainer used.

The primary outcomes were measured on the basis of visual areas of interest (AOIs), which were divided into the procedural site and ultrasonography. The primary outcomes measured included dwell time, fixation frequency, and visit frequency—these measurements were chosen given their correlations to a participant’s attention within a specified AOI. Dwell time was the total time (in seconds) that the participant fixated their gaze within a specific AOI. A longer dwell time within an AOI suggests an area of focus. Fixation frequency is the number of fixations within an AOI. Higher values for fixation frequency indicate more focused attention. Visit frequency is how often an AOI is entered and left. This value highlights areas that repeatedly attract a participant’s attention—lower numbers reflect increased interest and attention. Our perception is directed by alternating through sequences of fixations and saccades. Fixations occur when the eye stops scanning and focuses within an AOI, allowing time to process the image at hand. Conversely, saccades are movements that rapidly move the fovea from one point of interest to another, which results in poor image quality; therefore, most of the cognitive processing occurs during fixations.

Statistical Analyses

Although this pilot study was not powered a priori for the detection of significant effects, exploratory analyses were performed to investigate the presence of trends to guide future study. To explore whether trends in gaze metrics differ across the segments, a 1-way analysis of variance was performed for each gaze metric outcome (independent variable) among the 3 segments (dependent variables) of the CVC procedure across all participants. Only one data collection was performed for each participant, so comparisons in the gaze metrics across the experience levels of the participants was done by comparing the magnitude of the individual metric values. The threshold for statistical significance was set at α=.05.

Results

Participants

Three PCCM fellows and 3 faculty physicians participated in the pilot session. The fellows included 2 first-year fellows and 1 second-year fellow. The 3 faculty physicians were PCCM trained and were divided into 3 categories on the basis of years beyond fellowship training: novice (2 years), intermediate (4 years), and experienced (12 years).

For the calibration assessment, we assessed a total of 10 participants—5 women and 5 men. All 10 participants were successfully calibrated with 90% calibration within the first attempt. The calibration took less than 15 seconds. The median comfort level was 8 among the participants, 60% of whom (6 of 10) did not wear glasses or contacts at baseline. Some of the comments from participants included feeling mildly dizzy (n=1), discomfort from the earpiece with the cord connection (n=1), and the small lens affecting peripheral vision (n=2).

The faculty reported that the device was easy to wear and did not affect their ability to supervise the procedure. Feedback regarding the device included the need to be mindful about looking down with the head, not just with the eyes, and that the cord connection was slightly bothersome. The trainees noted that their performance was not affected by the faculty wearing the device. One of the trainees actually did not realize the device was being worn by the faculty member. The trainees thought that the scenario was realistic, although the noise level was quieter in the simulation center than often encountered in the ICU.

Across the segments of the CVC procedure, the puncture segment had a higher visit frequency on ultrasonographic AOI compared with the dilation and flushing of the CVC (P=.02), and there were trends toward both a higher fixation frequency (P=.05) and dwell time (P=.09) on ultrasound images during the puncture segment. As seen in Table 1, during the task of venous puncture, the fixation frequency and visit frequency were higher in the more experienced supervisory teacher for ultrasonography as the AOI. The dwell time was longer in those with less experience. During dilation, the more experienced teacher had higher fixation frequency and visit frequency at the procedure site than those with less experience (Table 2). The least experienced teacher had higher fixation frequency and visit frequency during the flushing of the CVC (Table 2).

Table 1

Gaze Metric Analysis During Venous Puncture on Simulated Central Venous Catheter Placement

Supervising teacher categorya	Dwell time/%,b procedure site	Dwell time/%,b ultra-sonography	Fixation frequency,c procedure site	Fixation frequency,c ultra-sonography	Visit frequency,d procedure site	Visit frequency,d ultra-sonography
Novice	414/4.4	507/5.39	10	4	5	3
Intermediate	215/0.64	1737/5.18	12	7	12	6
Experienced	831/1.77	337/0.71	8	17	8	10

Years beyond fellowship training: novice = 2; intermediate = 4; experienced = 12.

Dwell time = total fixation duration (seconds)/% of total duration of the task.

Fixation frequency = No. of fixations within an area of interest.

Visit frequency = how often an area of interest was entered and left.

Table 2

Gaze Metric Analysis During Dilation and Flushing Lines on Simulated Central Venous Catheter Placement

Supervising teacher categorya	Dwell time/%,b procedure site	Fixation frequency,c procedure site	Visit frequency,d procedure site
Dilation
Novice	344/0.71	12	2
Intermediate	894/3.09	9	8
Experienced	559/0.91	25	22
Flushing lines
Novice	252/1.12	30	14
Intermediate	195/1.63	2	2
Experienced	869/3.72	6	5

Years beyond fellowship training: novice = 2; intermediate = 4; experienced = 12.

Dwell time = total fixation duration (seconds)/% of total duration of the task.

Fixation frequency = No. of fixations within an area of interest.

Visit frequency = how often an area of interest was entered and left.

Discussion

To our knowledge, this is the first study to assess the feasibility and utility of eye-tracking devices specifically among supervising IBP teachers. We found that the device was easy to use, relatively comfortable, and user friendly. Based on our limited sample size, there appear to be differences in gaze metrics among supervising IBP teachers that are worth further investigation.

Differences in gaze patterns among novice and expert surgeons performing laparoscopic procedures have been reported., Experts often have longer fixations and focus their gaze on surgical objects. Conversely, novices trace the movements of their instruments. The gaze patterns of experts correlate with faster task completion, fewer movements, and shorter tool paths. Specifically within CVC placement, experts fixated more on the ultrasonography whereas novices spent time tracking the needle and ultrasound probe. Contrary to the current literature on gaze patterns in procedural education, this study focused on supervising the IBP teacher. We found that the use of the eye-tracking device for assessment of gaze metrics among supervising teachers is feasible and likely has some utility given the potential signal suggesting differences in gaze patterns among IBP teachers.

Because this was a pilot study, there are several limitations. First, we had a small sample size. Nonetheless, we were able to identify a possible signal that warrants further evaluation. Second, the classification of novice vs expert was based on years beyond fellowship training, implying that years of experience relates to expertise. Future studies could incorporate more specific criteria in the identification of an expert IBP teacher, such as number of supervised procedures, teacher awards, and fellow-selected experts. Third, the gaze analysis focused on 3 specific tasks during the simulated CVC placement. A larger study would need to evaluate all aspects of the procedure with more AOIs because there may be more differences found when the whole procedure is analyzed. Lastly, the PCCM fellows were of varying experience level and procedural competency, which could affect the gaze metrics of the supervisory IBP teacher. Future studies would need to control for the degree of competency for performing the procedure.

Currently, there are no best practice standards on how to supervise IBPs in the ICU. However, with the help of understanding gaze metrics, checklists could be developed to ensure appropriate competence of the supervisory IBP teacher. The use of gaze patterns could help novice teachers develop patterns similar to those of experts. Prior studies have found that training novices to have gaze patterns similar to those of experts resulted in faster task completion times, improved skill retention, and transfer to more complex tasks., We postulate that similar improvements in supervision and teacher competency could be achieved with training of gaze patterns.

Conclusion

Based on this pilot study, the eye-tracking device was, overall, easy to calibrate and did not affect supervision of the simulated CVC procedure. In addition, there appears to be a signal suggesting gaze differences in supervising IBP teachers. We believe the feasibility and utility of the eye-tracking device is appropriate and warrants future larger studies to evaluate for differences in expert and novice IBP teachers.