Assessing attentive monitoring levels in dynamic environments through visual neuro-assisted approach.

Objective: This work aims to establish a framework in measuring the various attentional levels of the human operator in a real-time animated environment through a visual neuro-assisted approach. Background: With the increasing trend of automation and remote operations, understanding human-machine interaction in dynamic environments can greatly aid to improve performance, promote operational efficiency and safety. Method: Two independent 1-hour experiments were conducted on twenty participants where eye-tracking metrics and neuro activities from electroencephalogram (EEG) were recorded. The experiments required participants to exhibit attentive behaviour in one set and inattentive in the other. Two segments ("increasing flight numbers" and "relatively constant flight numbers") were also extracted to study the participants' visual behavioral differences in relation to aircraft numbers.
Results: For the two experimental studies, those in the attentive behavioral study show incidences of higher fixation count, fixation duration, number of aircraft spotted, and landing fixations whereas those in inattentive behavior study reveal higher zero-fixation frame count. In experiments involving 'increasing flight numbers', a higher percentage of aircraft were spotted as compared to those with 'constant flight numbers' in both the groups. Three parameters (number of aircraft spotted, and landing fixations and zero-fixation frame count) are newly established. As radar monitoring is a brain engagement activity, positive EEG data were registered in all the participants. A newly Task Engagement Index (TEI) was also formulated to predict different attentional levels.
Conclusion: Results provide a refined quantifiable tool to differentiate between attentive and inattentive monitoring behavior in a real-time dynamic environment, which can be applied across various sectors. Recommendation: With the quantitative TEI established, this paves the way for future studies into attentional levels by regions, time based, as well as eye signature studies in relation to visual task engagement and management and determining expertise levels to be explored. Factors relating to fatigue could also be investigated using the TEI approach proposed.

Introduction

With the increasing trend of automation, understanding human-machine interaction can greatly aid to improve performance, promote operational efficiency and safety (Janssen et al., 2019). Better comprehension of human characteristics and limitations can also facilitate a smoother integration and interaction between the human operator and machine systems (Jamieson and Vicente, 2005; Sheridan & Parasuraman, 2005; Wickens and Hollands, 2000). Attention, a form of cognitive resources, seeks to filter and select channels of information from the environment to be processed. Attentional processes are often not automatic, and the cognitive resources available are limited. A more difficult a task assigned such as texting while driving, or being distracted while carrying out monitoring tasks might constrain performance of other tasks due to these limits. In numerous attention-relevant situations, the human operator is also faced with a wide range of dynamic sources of information (Wickens, 2021). Therefore, studying attention, particularly attentive and inattentive monitoring behavior in a dynamic environment can be safety critical and aid in performance enhancement when undertaking tasks. It can also provide valuable measured insights into situational awareness (SA).

Situational Awareness (SA) is defined as “the perception of the elements within the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1995, pp. 36). SA becomes more crucial when it serves as decision evaluation aids for operators in dynamic or animated application areas (Lanini-Maggi et al., 2021). Examples include autonomous driving, surveillance, clinical and medical monitoring, and aviation, where operational controllers are called to execute safety critical decisions.

The ability to decode monitoring behavior in dynamic display environments can reinforce and improve the understanding of human performance operating in complex systems (Parasuraman et al., 2008). An improved understanding of different levels of monitoring behavior or different levels of task engagement in an animated display environment can also improve the effectiveness of human-machine interaction (HMI) (Nalepka et al., 2019), thereby allowing better design of automated systems (Jamieson and Vicente, 2005; (Sheridan and Parasuraman, 2005)).

Various existing methods of studying monitoring behavior in dynamic environment were available. These were mainly through questionnaires (Rafaeli and Tractinsky, 1989), studying mouse-clicks, solving paper-based and computerized interactive problems (Kefalidou, 2017). Monitoring behavior or engagement with tasks on animated display could also be quantified through capturing the operator's physiological signals via electrocardiogram (ECG) (Zhao et al., 2012), Functional Magnetic Resonance Imaging (fMRI) (Wang et al., 2020), Electroencephalogram (EEG) (Li et al., 2018; Yuvaraj et al., 2017, 2020; Zhao et al., 2012) and eye-tracking devices (Batty, 2020; Brand et al., 2020; Rudi et al., 2020; Wee et al., 2017a, Wee et al., 2017b). For studies in relation to Air Traffic Monitoring context, extant literature indicated that visual search sequences and ocular parameters can reflect cognitive complexity and attention, hence eye-tracking method was chosen as the primary approach in conducting attentional studies for the experiments conducted in this paper (Marchitto et al., 2016; McClung and Kang et al., 2016). EEG was also found to provide insights into brain activities when conducting attentional studies (Aricò et al., 2016; Li et al., 2018; Wee et al., 2017a, Wee et al., 2017b; Yuvaraj et al., 2017). Furthermore, extensive research of eye-tracking and EEG were deployed in various individual surveillance fields such as aviation (Taukari et al., 2010; Nguyen et al., 2019), driving (Underwood et al., 2003; Schwehr and Willert, 2017), pilot assessment (Muehlethaler and Knecht, 2016), and healthcare (Harezlak and Kasprowski, 2017).

Lanini-Maggi et al. (2021) suggested that effective monitoring and tracking of objects in dynamic and animated displays is also a problem faced by many, such as that of visualization scientists, the aviation sector, and the public. Furthermore, with recent adoption of Multiple Remote Tower Operations (MRTO) which allowed aerodrome Air Traffic Control operations to be performed at a virtual Tower or remote location by a single ATCo on two or more different airports that are separated geographically. This makes attentiveness and attention distribution of ATCo, one of the key human-computer interaction issues of study in the formulation of multi monitoring task models (Li et al., 2018). Wang et al. (2021) also documented that quantifiable measures on eye movement activities have not been fully explored, particularly in the context of air traffic management.

This work seeks to develop and formulate a visual neuro-assisted approach allowing objective quantitative measurement of varying attentional states of a human operator in an animated environment, for the context of this paper, through radar display monitoring. The formulated index should allow monitoring behaviors to be quantified with a good degree of accuracy. Eye-tracking is utilized as the main method because extant literature suggests that visual scan patterns and ocular metrics could reflect cognitive complexity and attention in the Air Traffic Control context as discussed earlier (Marchitto et al., 2016; McClung and Kang et al., 2016). ATCos also usually make use of visual cues to derive pertinent information to aid in their decision making (Imbert et al., 2014). Furthermore, in the Salience, Effort, Expectancy and Value (SEEV) model (Wickens, 2021), visual parameters are used to predict the stochastic scan behavior of individuals based on their area of interests (AOIs), highlighting the effectiveness of eye-tracking in studying attention.

For the proposed visual neuro assisted approach, a newly established Task Engagement Index (TEI) quantifiable tool is used to differentiate between attentive monitoring and inattentive monitoring behavior of operators in a real-time dynamic environment. TEI consists of two main portions namely General Attentive Monitoring (GAM), which studies the general area of focus; and Object of Interest Focus Area (OIFA), which focuses on precise targeted objects when determining attention levels. TEI can be applied across various sectors and industries and is based on a set of five visual parameters, which will be discussed under the ‘Methods’ section. EEG readings will only be used as a binary indication (0 or 1) to confirm human cognitive engagement (brain signal activities) is present during the studies, a design consideration during remote operations.

Methods

Participants

Twenty participants were randomly selected from the engineering students in Nanyang Technological University (NTU) based on sampling selection from a list of students who responded to the advertisement posted in the university. Participants include both undergraduate and PhD students (6 females, 14 males, age range = 21–30, Mean = 24.2, S.D. = ± 2.6 years). Training and guidance was then given to the participants prior to the experiment explaining the basic radar information, allowing participants to have a basic understanding of what was being monitored. As such, simple random sampling method was employed. All participants satisfy the minimum inclusion requirements of normal or corrected-to-normal vision, with basic command of English for communication and comprehension. All participants have provided written informed consent prior to the experiment and the research is ethically approved by the NTU-Institutional Review Board Research Integrity and Ethics Office following all established ethical guidelines and regulations (approval no. IRB-2020-05-021-01).

Experiment design

Figure 1 shows a schematic diagram of the experimental setup for both sets of attentive and inattentive experiments while Figure 2 illustrates the actual experiment setup. The 3 main equipment used to collect data for the experiments were: 1) a 2048-pixel by 2048-pixel NLR's Air Traffic Management Real-time SIMulator (NARSIM) display interface developed by the Netherlands Aerospace Centre, 2) a Tobii X2-30 remote eye tracker, and 3) an Emotiv EPOC + EEG recording device (Figure 2). Real-time simulated NARSIM display containing aircraft information (Call Sign, Type, Altitude, Heading, Groundspeed) (Figure 3) were shown to the participants throughout the experiment with a positional refresh rate of 9.8 seconds which corresponds to the time taken for one radar revolution.

Figure 1

Diagram of experimental design.

Figure 2

Experimental set-up with participant. Note. (a) NARSIM interface, (b) Tobii X2-30 Remote Eye-Tracker, (c) Emotiv EEG.

Figure 3

Flight label (Left) and track (Right) on the radar display.

Visual physiological signals were obtained through the Tobii eye tracker. The Tobii X2-30 screen-based eye-tracker (Tobii, 2021) was positioned 55 cm (1.8 ft) from the participants' eyes, and 18 cm (0.59 ft) from the radar monitor (Figure 2) with a capture rate of 30 Hz to record eye movements. Based on this configuration, the accuracy and precision of the Tobii X2-30 were assessed at 3.5° and 1.1° respectively for head position within this defined 3D boundary with a validity of 93.7% (Holmqvist et al., 2012). The eye-tracker's angle was also measured to be 30°, with a 18 cm (7.08 inches) offset away from the radar screen. The velocity threshold was found to be 105°/s, while the fixation radius is 35 pixels (Holmqvist et al., 2012; Wee et al., 2019). The Emotiv neuroheadset measured the participants' neural (EEG) signals during monitoring with a sampling rate of 128 Hz (Figure 2). Positive neural signals were ensured throughout the entire duration of the experiment, with positive connectivity periodically checked. Figure 4 showed the EEG connection under the presence of human brain activities with the left showing good 100% connectivity and the right showing bad or partial connectivity of 41% at positions AF3, AF4, F8, F3, F4, FC5, FC6, T7, T8, P7, P8, O1, and O2, based on the international 10–20 system with reference to the ear (Emotiv, 2018; Kotowski et al., 2018; Yuvaraj et al., 2017). Figure 5 shows no EEG connectivity detected in the absence of human brain activities. In this study, EEG connectivity were present showing the presence of a human subject throughout the experiment duration. Further treatment, analysis, and validation of PSD data for monitoring engagement are not within the scope of the paper.

Figure 4

(Left):Good brain contact quality, (Right): Bad Connectivity or Partial Brain Contact Quality.

Figure 5

Absence of brain activity or human subject.

Procedure

Each participant completed two independent experiments, namely attentive monitoring, and inattentive monitoring with a 15-minute break in between the experiments (See Figure 6). During attentive monitoring experiment, participants were instructed to give due attention and concentration to the unfolding air traffic simulation without having to explicitly complete any task or answer any question. This forms the attentive baseline. For inattentive monitoring experiment, participants were either looking partially away from the screen or engaging in other activities (usage of mobile devices, resting, engaging in conversation). Two 5-minute segments (31 frames each) from the same test period of the participants were then extracted for data analysis, with the first segment displaying increasing flight numbers (12–31) per frame and the second segment displaying a relatively constant flight numbers (30–35) per frame. This is done to study the visual behavioral differences of participants with varying aircraft numbers. In total, there are 1402–1923 flights for each set of attentive and inattentive experiment per participant throughout the 62 extracted frame images. With traffic safety in mind, short time segments were extracted for investigation to assess the detection of visual differences between the participants should a loss of attentiveness or situational awareness (SA) during that time duration. It must be said that the approach could also cater to longer time duration studies.

Figure 6

Experiment flow.

Data processing and parameter identification

In capturing monitoring behavior in ATC context (Wee et al., 2017a, Wee et al., 2017b; Wee et al., 2019), eye-tracking was used as the main method in conducting attentional studies for the experiments in this paper. Eye tracking represents the most direct and accurate attentiveness response measure on task monitoring (Wee et al., 2020). Eye fixations derived from the eye data are based on the velocity threshold fixation identification (I-VT) algorithm (Wee et al., 2017a, Wee et al., 2017b; Wee, 2020), which classifies the eye movements according to the velocity of the directional shifts of the eye. Eye-tracking metrics are computed every 9.8 seconds. A post-processing dynamic data alignment and timestamp synchronization model which aligned and synchronized the timestamp of the eye data from the I-VT algorithm and dynamic radar display data from the NARSIM output data developed by Wee et al. (2019) was used to record and process these data into a synchronous ‘frame’ (See Figure 7). In total, 366 frames of eye-fixation counts, and fixation duration were collected in a 1-h session.

Figure 7

Sample Frame Image captured via dynamic data alignment and timestamp synchronization model.

A newly established set of five eye-tracking parameters was used to analyze the general monitoring tasks. (These parameters were defined from Figure 7). A parametric student's t-test is then performed to assess the statistical significance between the data sets. All parameters are reported as average values with standard deviations as error bars. Statistical significance is defined by p < 0.05. As visual tracking parameters may be affected by aircraft numbers that changes over the experiment duration, the total number of fixation counts (Fix-C), fixation durations (Fix-D) and the number of aircraft spotted (Fix-AS) were normalized with respect to the total number of aircrafts, hereafter as AN, in the frame.

Eye-fixation count over aircraft number (Fix-C/AN)

Eye-fixation count, hereafter as Fix-C, refers to the number of fixation points (grey boxes) on the radar captured by the eye-tracker in each individual frame (Figure 7), as defined in (1). It is measured through the Tobii X2-30 screen-based Eye-Tracker and mapped onto the fixation frame via the I-VT Algorithm. Fix-C is found to be a suitable measure in assessing the participant's ability to notify visual cues (Wee et al., 2020).

Eye-fixation duration over aircraft number (Fix-D/AN)

Fixation duration, hereafter as Fix-D, refers to the length of time in milliseconds (ms) that the eye is fixated on the screen, as defined in (2). It provides a gauge of the amount of time that the participants spent fixating on the radar display. The shade of the grey boxes in Fix-D denotes the variation in duration (lighter grey = lower duration, darker grey = higher duration) (see Figure 7). A longer fixation duration will suggest that more time is needed to extract information from the radar display.

Number of aircraft spotted over number of aircraft (Fix-AS/AN)

Number of aircraft spotted, hereafter as Fix-AS, refers to the number of aircraft identified when a fixation landed on it, as defined in (3). To account for data duplicity in counting, only one Fix-AS count will be assigned even when multiple fixations landed on the same aircraft. Fix-AS is used as a direct measure of the participant's attention or task engagement level (see Figure 7).

Number of landing fixations on aircraft over number of fixation count (Fix-Land/Fix-C)

Landing fixations on aircraft, hereafter as Fix-Land (see Figure 7), is defined as the eye fixations landed on either the flight (indicated by the blue circle linking to label of track), its label (indicated by the square box) or both. For this parameter, a ratio of the total Fix-Land to total Fix-C (Fix-Land/FC) over a specified number of time frames will be computed, as seen in (4).

Zero-fixation frame count over number of frames (Fix-Zero/FN)

Zero fixation frame count, hereafter as Fix-Zero, provides an alternative way to measure attentiveness. It is defined as the number of “null” fixations recorded per frame and was deemed to be independent of AN. A Fix-Zero data is recorded when there were absence of Fix-Cs (Grey boxes) in any particular frames generated. To determine this parameter, Fix-Zero is normalized with respect to the number of total specified frame images, hereafter as FN, as seen in (5).

Figure 7 shows a frame image with the following parameter numerals:

Fix-C/AN = , Fix-D/AN = , Fix-AS/AN = , Fix-Land/FC = , Fix-Zero/FN = .

Results

Visual parameters on attentive and inattentive task engagement

Table 2 presents two sets of experimental results obtained for attentive (A) and inattentive (N) behavior of participants whereas Table 3 illustrates the Pearson's correlation value between the attentive and inattentive participants for all five parameters. Observations and analysis will be divided in the following two main sections: (a) general visual parameter relationships to task attentiveness, and (b) establishment of a TEI on attentive engagement.

Table 2

Correlation of attentive and inattentive overall Data.

Table 3

Percentage (%) of aircraft spotted for ‘Increasing flight number’ versus ‘Relatively constant flight number’ segment.

General visual parameter relationship to attentiveness

From Tables 1 and 2, four key observations can be derived. First, there exists significant arithmetic mean differences between the sets of attentive and inattentive tasks in each of the five parameters examined. By first examining (1), Fix-C/AN (0.23) of attentive monitoring is 2.3 times that of the inattentive ones (0.10). This suggests that the participants from the attentive session on average fixate more than those from the inattentive experiment. The difference in visual parameters between attentive and inattentive participants' data also reinforced extant literature's idea in determining eye-tracking method to be suitable in capturing monitoring behavior (Marchitto et al., 2016; Wee et al., 2017a, Wee et al., 2017b; Wee et al., 2019). Through eye-tracking method, a clear distinction could be seen between attentive and inattentive participants, and this distinction would be used in the later section to formulate a Task Engagement Index to identify the varying attentional states of human operators in an animated environment.

Table 1

Comparison between Attentive and Inattentive overall visual parameters.

Second, for (2), Fix-D/AN values in the attentive session (214.30) are 2.49 times on average more than those in the inattentive session (86.05). A longer duration suggests more gaze time spent in monitoring traffic movements on the radar display rather than looking away from it.

Third, for Fix-AS/AN, defined in (3), participants in the attentive study (0.08) are 4 times on average spotted more aircraft than the inattentive ones (0.02). Results indicate that the participants in the attentive session are more active in observing the aircraft locations and flight information on display than the inattentive ones. Because attention of participant for inattentive experiment sessions were distributed between distractions and the actual task itself, results seem to suggest that attentiveness is indeed an issue, especially during multi-monitoring task models as proposed by Li et al. (2018). This could be a safety concern, and a quantifiable approach in assessing attention distribution in ATCos could help to better understand and identify this problem.

Fourth, on Fix-Land/Fix-C, as seen in (4), higher percentage of 43% (0.43) is registered on average in the attentive participants compared to 22% (0.22) of the inattentive ones (Figure 8). The Fix-Land can either be on flight location, its label, or both. Besides this, the findings take on greater significance when they are also able to differentiate attentive and inattentive ones even when some have displayed “vacant gaze” behavior where they simply stared on the screen without performing many actions. Such behavior is termed as "look but not see" phenomenon as reported and deemed difficult to resolve in the research studies by Nguyen et al. (2019) and Smolensky (1993).

Figure 8

Percentage (%) of Total landing fixation over Fixation Count (Fix-Land/FC).

Figure 9 shows the Fix-Zero results for both the attentive and inattentive experiments. Results indicate a notable preponderance of the number of Fix-Zero in the Inattentive experiment compared to the Attentive experiment for 95% of the participants (p = 8.90 × 10−5). For Fix-Zero/FN, as seen in (5), the average values for attentive and inattentive experiments are 0.06 and 0.47 respectively. A lesser 0.06 Zero/FN score in the attentive session means that a participant spends less “time away” from the display. The “time away” can be looking away or taking an “eye break” from the screen display. It is observed on average, one Fix-Zero is registered at intervals of 16–17 frames in the attentive group, as compared to every 2 frames in the inattentive ones. This may mean that an inattentive participant is more likely to lose task focus due to distractions or fatigue.

Figure 9

Total zero-fixation frame count (fix-zero).

Finally, in assessing correlation values between attentive and inattentive data (Table 3), all the five parameters of Fix-C/AN, Fix-Land/FC Fix-D/AN, Fix-Zero/FN and Fix-AS/AN showed weak positive correlations ranging from 0.01 to 0.37 between these attentive and inattentive groups.

In summary, a comparison between Attentive and Inattentive experiment data sets suggests that when participants are asked to engage in radar monitoring, there are significantly differences between them. For Fix-C/AN, on average, attentive participants register 2.3 times higher than inattentive ones, 2.5 times higher in Fix-D/AN, 4 times higher on Fix-AS/AN, 95%, 2 times for Fix-Land/FC and 7.8 fewer times in Fix-Zero/FN than the inattentive ones. Nevertheless, the data also suggest that the attention levels displayed by the participants were wide ranging despite carrying out the same attentive monitoring tasks. This will enable one to better comprehend the nature of task at hand. Being quantitative in nature, this approach also enables one to readily set the attentiveness target values required by the participant to undertake a particular monitoring task.

The relationship between AN on screen and the percentage (%) of aircraft spotted is shown in Figure 10. The detail of the parameters can be found in Table 3. Two similar frame segments are extracted from both the Attentive and Inattentive study groups; Set A consists of 31 frames showing increasing number of aircraft on screen, while Set B shows 31 frames that displayed relatively constant number of aircraft. The total AN ranges from 441 – 875 in Set A and 983–1048 in Set B. The Attentive group shows a significantly higher percentage of aircraft spotted across both sets as compared to the Inattentive group. In the Attentive group, participants also display a higher average percentage of aircraft spotted in Set A (9.66%) than Set B (7.78%). A similar trend is also observed in the Inattentive group where higher average percentage of aircraft spotted is found in Set A (3.30%) than in Set B (1.73%). This higher average percentage of aircraft spotted can be attributed to ease of focus when fewer aircraft are displayed (Set A).

Figure 10

Percentage (%) of aircraft spotted for ‘Increasing flight number’ versus ‘Relatively constant flight number’ segment. Note. Attentive data set is represented by the two box plots to the left; Inattentive data set is represented by the two box plots to the right.

Establishment of TEI for assessing attentive levels

This section proposes a TEI system that is used to measure the attentive levels of participants based on the five visual parameters. In this paper, task engagement refers to the attentive monitoring of the presented air traffic scenarios on the NARSIM radar display. Participants are assumed to be statistically independent, random, and identically distributed.

The scoring system is generated based on both data sets of attentiveness (n = 40). The aim is to see whether the system can identify the attentive grouping and corresponding attentiveness level in each participant correctly. Each participant is scored based on two categories of attentiveness namely: 1) General Attention Monitoring (hereafter as GAM) comprising Fix-C/AN, Fix-D/AN and Fix-Zero/FN, and 2) Object Interest Focused Attention (hereafter as OIFA) consisting of Fix-AS/AN and Fix-Land/Fix-C. For the OIFA set, its scores are multiplied by a factor of 1.5 to bring weightage parity between the two groups of score measured to provide an unbiased measurement of parameters.

The correlation coefficient (Table 4) also suggests that negative correlation between the parameters whenever there was Fix-Zero/FN parameters present. This indicates that Fix-Zero could be a strong indicator of inattention. To assess the dependency between these parameters, independent t-tests are performed between the pairs. The results are shown in Table 5.

Table 4

Correlation of parameters.

Table 5

P-values of independent parameter pairing Analysis.

Under the law of large numbers, the data tends to the mean value, and a quartile classification approach was used to classify the data for this proposed approach. A box plot was also utilized for the TEI because due to the overlap between Attentive and Inattentive dataset in attentiveness. In the absence of a quantitative measure, participant level of attentiveness are either grouped as Attentive or Inattentive and not on a graduated scale. Each parameter, x, in a participant session is given a score of 0–3 according to a set of parameters, within a range of 0–100 percentiles. Pending on the desirability trend where 0 means least desired and 3 most desired, a scoring system is introduced as follows (Table 6):

Table 6

Parameter value range and score.

As there are five independent parameters, each parameter in the GAM set will have a maximum score of 3, whilst in the OIFA set, each parameter will have a maximum score of 4.5. The maximum overall attentive score for each participant can be 18 and the least being 0. Based on the scoring system, the TEI scores for the attentive and inattentive participant sessions can be found in Table 7 and Figure 11.

Table 7

Scoring System for Attentive and Inattentive Data Sets (Based on quantile).

Figure 11

Box plot of Attentive & Inattentive Task Engagement Index (TEI) (Based on quantile).

For the TEI system, four score bands are used to measure the overall attentive levels of the participants as in Table 7 and Figure 11. From the results, the following observations can be made.

First, all TEI scores of the participants in the Attentive study group (M = 13.1, SD = 2.94) register significantly higher scores than the corresponding Inattentive study group (M = 4.63, SD = 3.89). The Attentive group records a mean score of 13.1 indicating “Attentive Engagement”, whereas the Inattentive group has a mean score of 4.63 which falls in the “Inattentive Engagement” band.

Second, when the TEI scores are banded into four quartiles, Table 8 shows the following:

Table 8

TEI scoring System.

For the attentive group, 9 (45%) of the participants were in the Attentive engagement band (9 ≤ x < 13.5), 10 (50%) of the participants were highly attentive (13.5 ≤ x ≤ 18), with only 1 (5%) participant registering 5.5 score of Low Engagement (4.5 ≤ x < 9) band. In the inattentive group, the scores were more diverse where 11 (55%) participants were in the “Highly Inattentive” band (0 ≤ x < 4.5); 6 (30%) - “Inattentive” band; 3 (15%) - “Attentive” band and 0 were in the “Highly Attentive” band. This disparity can be attributed to the varying inattentive activities carried out by participants during the experiment. The findings highlighted the capability of the TEI to provide some level of granularity on attentiveness of the participants, thereby offering a new way to explicitly evaluate the various monitoring activities (tasks) that the participants were engaged in over a stipulated time. Nevertheless, it can differentiate and correlate well in terms of the attentive level exhibited by the two sets of attentive and inattentive studies.

Discussion

Through the newly established set of visual parameters established, results from the studies for the Attentive and Inattentive group were able to be differentiated well into four levels of attention. The lower percentage of aircraft spotted when aircraft numbers increased could also attribute to difficulty in focusing attention owing to higher traffic density over a fixed time span. This problem was noted in the earlier introduction section as one of the main concerns for Multiple Remote Tower Operations (MRTO) and human-machine collaboration in remote settings (Li et al., 2018). Despite the availability of commonly used existing models in situational awareness studies such as Situation Awareness Rating Technique (SART), and Situational Awareness Global Assessment Technique (SAGAT), the evaluation by such approaches tend to be qualitative and subjective as well as interruptive to the monitoring processes via freezing of displays for queries denoted by Endsley (1995). Besides, timely real-time assessment of a select situation is not possible. Questionnaires were also employed in such models where response biasness was reported to be one of the major drawbacks where guessing is prominent (Epperson and Peck, 1977, Tellis and Chandrasekaran, 2010). A combination study with physiological parameters such as that of visual gaze data could reinforce and quantify findings in such models.

The proposed visual neuro assisted approach via employing a targeted and quantifiable Task Engagement Index (TEI) system can therefore add value to existing literature and models by providing greater objective insight into assessing and quantifying visual attention data in dynamic environments such as that of Air Traffic Control. Results from the studies and the formulated TEI also suggest that the attentive and inattentive monitoring behaviours can be identified and differentiated. The approach can aid in addressing a fundamental challenge in air traffic monitoring and management highlighted by Suarez et al. (2014) of the need to develop tools that can measure the complexity inside sectors of the airspace in short intervals as well as the accuracy of the prediction of these values.

The ATM system is also changing; for example, the transformation from radar control to trajectory-based operations with the reconstruction of the roles, responsibilities, and requirements for both humans and automation (Thomas et al., 2014; Wang et al., 2021). In the future, monitoring, managing and ensuring the automation functions properly could become a key activity. This is for when a failure or degradation of performance occurs, quick resume of system control by controllers would be critical (Thomas et al., 2014; Hasse et al., 2012). Studying air traffic controllers' eye movements is of relevance in both the ATM domain and other domains due to the unique nature of their work. Controllers' decision-making processes could be better discerned if their eye movements are properly understood. This would aid to provide insights into their thought processes (Wang et al., 2021). Eye-movement is also deemed as the natural indicator of information seeking by the brain (Wang et al., 2021), and many studies were carried out on the topic of task-directed information seeking through eye movements (Willems et al., 1999; Gottlieb et al., 2013; Wang et al., 2021; Marchitto et al., 2016; McClung and Kang et al., 2016). However, despite the existing measures being proposed to capture human's ability to process information in a complex environment, this effectiveness has yet to be fully explored in air traffic management field (Wang et al., 2021). The proposed approach using TEI could contribute towards enhancing the quality of controller training and to further understand the information-seeking mechanisms humans use when executing complex tasks.

Limitations

Despite remote eye tracking offering a good quantitative means in assessing one's attention level, it does have its limitations. The eye-tracking metrics can indicate what objects on the display the individuals have sights on, they may not necessarily translate to what they perceive. Thoughts are also unrecordable through eye-tracking. In this work, EEG data was only used to indicate the presence and absence of the participants throughout the duration of the experiment, however further treatment and analysis of the EEG data in relation to levels of attention were not explored. Further exploration of EEG combined with visual data could be a future area of exploration which could provide interesting insights into monitoring behavioural activities.

The study used for TEI formulation proposed in this contribution is also limited to 40 experiment sessions. To test the validity of the TEI, a setup described by Wee et al. (2019) was replicated using TEI as an attentional quantification tool. Data were extracted from 83 1-h experiment sessions using a real-time NLR ATM Research SIMulator (NARSIM) on participants with either ATC experience or ATC training, aged between 24 and 41 years (Mean = 29, S.D = 5.34). Air traffic scenarios were run in a fashion to mimic actual radar conditions, which was displayed on a 2K radar screen of 1:1 aspect ratio (of dimension 50cm, or 20 inches), and refreshed every 9.8 seconds. Pseudo-pilots were also deployed to follow the same scenario-script so that events in the same scenario maintain the same time-congruence for every participant. Results of the data sets were presented in Figure 12.

Figure 12

Task engagement index (TEI) of ATCOs with attentive and inattentive baseline. Note. ATCos = air traffic control officers.

From Figure 12, data showed that all 83 experiment sessions (21 novices, mean = 14.9; 19 intermediates, mean = 16.3; 43 expert ATCOs experiment sessions, mean = 15.5) were on the high end of the attentive band when carrying out attentive monitoring tasks. Results from the above data displayed promising reliability of the TEI, noting this set of data involves highly monitoring engagement activities. This paper also studied mainly the difference between attentive and inattentive monitoring behavior, factors relating to decision making was also not explored. The quantitative TEI proposed in this paper could however be used for time based regional attention analysis, formulation of eye signature studies in relation to human behavior on task activities, fatigue studies and determination of expertise levels.

Conclusion

This paper proposes a real-time visual neuro assisted approach employing a TEI system to quantify varying levels of attention by participants engaged air traffic monitoring tasks in a dynamic changing animated environment. Good predictability of up to 95% can be achieved in identifying those in the Attentive group, and 85% for the inattentive group and Inattentive study groups. A new set of visual parameters measuring attentiveness are also established, providing achievable measurements of varying attention levels. Significant differences, at a 95% confidence level, between Attentive and Inattentive monitoring participant scores are observed. A notable preponderance of Fix-C/AN (85%), Fix-D/AN (95%), Fix-AS/AN (100%) and Fix-Land/AN (95% more) are observed in the Attentive group when compared with the Inattentive group. Fix-Zero counts are also 95% fewer in the Attentive group than the Inattentive group. A higher Fix-AS/AN in the ‘increasing flight’ segment (Set A) is registered compared to ‘relative constant flight’ segment (Set B) in both attentive and inattentive groups. EEG data is also used as a qualifier for attentiveness, with positive brain activity response recorded throughout the radar monitoring activities. The findings can determine the behavioral differences in varying attentional states displayed by the participants while undertaking various ATC monitoring tasks. This research can therefore aid to better measure and integrate human-machine teaming responses in real-time environments with animated display systems. The TEI formulated through the GAM and OIFA scores also allows for a refined and quantitative measurement of attentiveness in real-time dynamic environments and identify potential weaknesses and address them accordingly.

Declarations

Author contribution statement

Sun Woh Lye: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Yu Fei Li: Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Yuvaraj Rajamanickam: Performed the experiments; Contributed reagents, materials, analysis tools or data.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.