Pilots' mental workload prediction based on timeline analysis.

BACKGROUND: The aircraft cockpit is a highly intensive human-computer interaction system, and its design directly affects flight safety.
OBJECTIVE: To optimize the display interface design in complex flight tasks, the present study aimed to propose a dynamic conceptual framework and a timeline task analysis method for the quantization of the dynamic time effect of mental workload and the influencing factors of task types in the mental workload prediction model.
METHODS: The multi-factor mental workload prediction model based on attention resource allocation was integrated to establish the dynamic prediction model of mental workload. The ergonomics simulation experiment was carried out by recording the data on the performance of embedded subtasks, National Aeronautics and Space Administration-Task Load Index (NASA-TLX) subjective evaluation, and eye tracking.
RESULTS: The results indicated that the prediction model had a good prediction accuracy and effectiveness under different simulated interfaces and complex tasks, and the real-time monitoring of pilots' mental workload state was realized.
CONCLUSION: In conclusion, the prediction model and the experimental method could be applied to avoid the overload of the pilot throughout the flight phase by optimizing the display interface and adjusting the flight task.

Introduction

Pilots often need to process large amounts of information in a relatively short time and make quick response decisions to deal with possible urgent airspace situations [1, 2]. Moray et al. proposed that rationally optimizing the operator’s mental workload distribution could effectively reduce human error and improve system reliability and operator comfort [3, 4]. Therefore, the study of pilots’ mental workload prediction can maximally prevent flight accidents by effectively adjusting the flight task and the pilots’ mental state in time [5, 6].

At present, the quantitative modeling research of mental workload is still in its infancy [7, 8]. First, most models are descriptive [9, 10], and the quantitative modeling of mental workload is mostly based on post-mortem measurement of mental workload [11, 12]. Second, still many imperfections exist in the quantitative prediction of mental workload. Bin and Salvendy [13, 14] proposed a dynamic time conceptual model for mental workload prediction, including instantaneous mental workload (IMW), average mental workload (AMW), cumulative mental workload (CMW), maximum mental workload, and total mental workload (TMW), independent of a particular field of mission scenario. Laughery et al. [15, 16] conducted a task analysis method to calculate mental workload by decomposing complex tasks into multiple simple behavioral elements and connecting the elements hierarchically and logically. However, the massive task was difficult to analyze in detail. According to the single-channel theory, Siegel and Wolf [3] introduced the timeline analysis and prediction (TLAP) method in which the time workload equaled time required/time available (TR/TA). However, these models do not consider the influencing factors of mental workload in the aviation field, nor do they take into account the impact of different task types on mental workload. Xiao et al. [17] comprehensively adopted factors such as information amount, time pressure, visual coding, and attention resource allocation to establish a multi-factor mental workload prediction model. However, the model does not consider the conflicts within task types and dynamic time effect.

Considering the aforementioned problems, the present study aimed to optimize the display interface design under different flight tasks. On account of the dynamic concept of mental workload, the timeline task analysis method was used to decompose complex tasks into several task units and simple behaviors that changed dynamically with time and employed the McCracken-Aldrich scale [5] to divide and assign each simple behavior. A dynamic prediction model of mental workload based on timeline analysis was established according to the mental workload prediction based on attentional resource allocation and information processing [17]. The model was verified by carrying out an ergonomic experiment on the civil cockpit simulator. A correlation analysis was promoted between model prediction and experimental results, concerning comprehensive metrics including the performance of embedded subtasks, National Aeronautics and Space Administration-Task Load Index (NASA-TLX) subjective evaluation, eye tracking, and other technical indicators. The significant main effects and correlations indicated that the model has a good validity and availability for different interfaces and complex tasks with dynamic time variation.

Model development

Timeline task analysis

In practice, tasks, especially high-mobility military tasks, are complex and usually consist of one or more simple tasks [1]. Meanwhile, the flight information required by the pilot is different under different flight phases. As a result, to accurately determine the mental workload state of the pilot over time, the timeline analysis method is used to decompose the complex tasks and derive the time-variant basic task units and the time attributes of each task unit, including start time , end time , and duration (where refers to the -th task unit).

Task type correlation coefficient solution

Since different task types have different effects on mental workload, they need to be quantified according to the internal cognitive mechanism of task types [18]. Each task unit contains one or several specific behaviors [15]. The behavior of task segmentation is executed by different parts and actions of the operator [16]. The McCracken-Aldrich Mental Stress Assessment Scale [5] divides behaviors into four channels: visual, auditory, cognitive, and motor. Each channel is further subdivided into different scales of complexity. According to the multi-resource theory [3], behaviors that are not in the same channel can be executed simultaneously. The task type workload value of the -th decomposition behavior can be quantified. Then, the task type workload value is the summary value of decomposed behaviors of the -th task unit.

The task type workload value calculated using the McCracken-Aldrich scale is considered to be overly heavy if more than 8 [5]. Therefore, the task type correlation coefficient of a task unit is calculated as follows, where is the total number of task units decomposed by the -th task. When , the mental workload caused by the -th task is considered overwhelmingly high.

Multi-factor mental workload prediction

The multi-factor mental workload prediction based on attention resource allocation [17] is triggered by two decisive factors: external task demand and internal resource allocation of mental workload, integrating the information amount , time pressure , visual coding , and attention resource allocation .

The average amount of information of different information appearing over time is calculated as follows, wherein the information occurrence probability of the -th areas of interest (AOI) represents a coefficient related to the information number and information value.

During the information perception, the attentional allocation factor is used to indicate the intrinsic mental resources that the operator needs to consume due to the information of a certain AOI. The attention resource occupied by AOIs is ; then is the ratio of the attention resource to the total attention resource for the -th AOI.

After the information is captured by the visual system and transmitted to the mental procession by the nervous system, the attention resources allocated to the -th AOI are as follows, where each of these information has a certain probability of being noticed in one gaze. Information prominence is quantified by the applicability of the highlighted format. indicates the effort that the vision system needs to pay to acquire the -th AOI, and is the information importance attribute associated with the assignment task.

[17] is the visual coding comprehensive performance eigenvalue, with . is the weight coefficient corresponding to the -th visual coding in the -th AOI, which is obtained by the G1 method through expert evaluation. is the comprehensive performance eigenvalue of the -th visual coding obtained by meta-analysis, and is the fuzzy weighted average operator.

Related experimental results [2, 5, 17] show that the influence of visual coding on mental workload is less significant than that of information amount and time pressure. Therefore, the square root of visual coding is applied as the coordinated quantitative result in the model.

Considering the dynamic time effect of the mental workload, the present study improved the method of quantifying time pressure [17]. According to the TLAP, the time pressure affecting the mental workload is highly correlated with TR/TA on the timeline process. Therefore, the adapted calculation of the time pressure is as follows, where is the available time of the -th AOI, given by the system for the operator to perform a task, and is the required time of the -th AOI, needed by the operator.

can be represented by the time required by the operator from information presentation to perception, understanding, and reaction. According to the “behavior time prediction” method, the time from perception to reaction includes the reaction time and the exercise time . can be subdivided into simple reaction time and selective time. The simple reaction time of different sensory channels can refer to empirical data [3]. Hick-Hyman Law [11] can be employed in calculating the selective time. is the information quantity constant, with the value 50–157 ms, and the average is 92 ms. is the average information amount of the stimulation signal in the -th AOI. The exercise time is calculated by Fitts’ Law [7], where is the motion control constant, with the value 70–120 ms and the average 100 ms; is the motion difficulty index, where is the target distance and is the width of the target object.

Altogether, the multi-factor mental workload prediction value of the -th task unit decomposed by task 1 is obtained:

Dynamic time effect calculation of mental workload

To monitor the pilots’ mental workload state in real time, the IMW, CMW, TMW, and AMW are established under the dynamic conceptual framework of mental workload [13, 14].

IMW is defined as the instantaneous amount of mental workload at a given time. Combined with and , the IMW (in bit) for the -th task unit decomposed by task can be obtained:

The TMW (in bit s) is defined as the total accumulation of mental workload from the start time to the end time. In the formula, represents the initial time, and T represents the termination time. Mathematical expressions of other mental workloads can be obtained from the mathematical expressions of IMW.

The AMW (in bit) is defined as the average intensity of mental workload throughout the time course.

Experimental method

Participants

The study was performed on 21 participants (19 males and 2 females), with an average age of 24.5 years (N 21, Mage 24.5, SD 1.0). They had an aeronautical knowledge background, right hand, normal or corrected vision, and no color blindness. The participants were required to participate in flight simulation training until they were fully familiar with the flight operations.

Study design

The study was based on the actual flight tasks, using the display interface (3) flight task (4) two-factor repeated measurement of the interior design of the test. As shown in Fig. 1, the display interface included three levels: the virtual main flight display (PFD) of B737, A320, and ARJ21, with proper simplification and abstraction. The flight phase included the takeoff phase and three cruise phases (90 turn, straight, and 180 turn). To avoid the practice and fatigue effects, the experimental sequence was designed in Latin.

Figure 1.

Simulated interface of B737, A320, and ARJ21, cruising flight phase (left) and simulation (right).

Materials

The experiment was carried out in a Boeing 737–800 flight simulator, with real-time performance and eye movement indicators monitored. Within 15 min after the end of each experiment, the participants were asked to finish the subjective evaluation of different display interfaces and flight phases according to their real feelings.

Since the flight task performance was difficult to measure directly, the reaction time and the accuracy of the embedded subtask were used as performance indicators, which were automatically recorded by MATLAB programming through keyboard operation. The Swedish noncontact infrared eye tracking system Smart Eye Pro 4.5 was used to record the eye movement data, with an accuracy of better than 1 and a 60 Hz sampling rate. National Aeronautics and Space Administration-Task Load Index (NASA-TLX) [5] software was used for subjective evaluation.

Experiment task

This experiment applied the embedded subtask method proposed by Wickens et al. [3], requiring the participant to monitor and identify the flight information on the PFD and navigation display (ND) using the remaining capabilities while ensuring that the manual control task (main task) was completed, including airspeed, altitude, pitch, and heading. During the experiment, the display interface was frozen and covered at any time, and the subject answered questions about the current value or change characteristics of the flight information within the specified time.

Manual control flight task is a simulated dynamic flight task for takeoff and cruise under a specified interface. The participants need to complete the takeoff operation by controlling the landing gear, throttle, flaps, and so on. The simulated cruise tasks consist of three phases. The participants are asked to control the steering wheel so that the aircraft can fly along the prescribed flight path and keep the airspeed, altitude, and attitude within the normal range.

Results

Theoretical predictions

First, the takeoff phase lasts 109 s. The operations using accumulated long-term memory related to operation steps through experience and learning are divided into 11 task units. Second, the cruise phase lasts for 240 s and is decomposed into a single task unit. PFD and ND are divided into five AOIs. Then, the IMW value is calculated, as shown in Fig. 2. The TMW and AMW are shown in Table 1.

Figure 2.

IMW of the three display interfaces during the takeoff phase.

Table 1

Mental workload prediction and descriptive statistics (mean SD)

		TMW /bit × s	AMW /bit	Correct rate /1	Reaction time /s	Subjective evaluation /point	Blink frequency /Hz	Average gaze time /s
B737	1	590.92	5.42	0.40 ± 0.18	4.98 ± 1.71	66.47 ± 12.67	0.22 ± 0.14	0.20 ± 0.15
	2	876.91	3.65	0.60 ± 0.21	4.44 ± 1.35	67.91 ± 7.89	0.15 ± 0.12	0.17 ± 0.14
	3	362.07	1.51	0.67 ± 0.16	4.44 ± 1.30	54.97 ± 11.31	0.15 ± 0.12	0.18 ± 0.15
	4	1010.02	4.21	0.61 ± 0.15	4.47 ± 1.36	70.72 ± 8.48	0.11 ± 0.09	0.15 ± 0.12
A320	1	552.63	5.07	0.47 ± 0.16	5.80 ± 2.14	67.08 ± 13.37	0.23 ± 0.12	0.17 ± 0.12
	2	876.60	3.65	0.55 ± 0.20	4.54 ± 1.08	67.50 ± 7.78	0.17 ± 0.11	0.15 ± 0.10
	3	362.86	1.51	0.67 ± 0.16	4.74 ± 1.95	52.75 ± 11.08	0.16 ± 0.11	0.20 ± 0.19
	4	997.89	4.16	0.63 ± 0.20	4.54 ± 1.43	72.20 ± 7.98	0.12 ± 0.09	0.14 ± 0.12
ARJ21	1	606.10	5.56	0.44 ± 0.17	5.19 ± 1.36	67.25 ± 11.78	0.22 ± 0.11	0.24 ± 0.20
	2	905.02	3.77	0.52 ± 0.21	4.69 ± 1.63	70.18 ± 8.45	0.14 ± 0.11	0.17 ± 0.12
	3	370.46	1.54	0.63 ± 0.21	4.33 ± 1.78	54.60 ± 10.29	0.15 ± 0.11	0.22 ± 0.19
	4	1022.92	4.26	0.57 ± 0.20	4.26 ± 0.94	72.46 ± 7.81	0.13 ± 0.10	0.18 ± 0.14

Note: MW conditions are 1 (takeoff), 2 (90 turn), 3 (straight), and 4 (180 turn).

Experiment result

The repeated-measures variance analysis (RMANOVA) of the two-factor design showed that the main effect of the display interface was only significant for the average gaze time ( 0.005). The main effects of the flight phase on all indicators were significant ( 0.005). The interaction effect between the two factors was not significant.

The multiple comparisons after single-factor test showed that the performance, subjective evaluation, and blink frequency were not significant among the three interfaces, and the gaze time was partially significant ( 0.042, 0.004). Meanwhile, performance, subjective evaluation, and blink frequency were significant between the takeoff and straight phase ( 0.001, 0.013, 0.001, and 0.001), and only the blink frequency was significant between 90 turn and 180 turn ( 0.001).

As shown in Fig. 3, the Pearson correlation analysis between the prediction and experimental results indicated a significant positive correlation between TMW and subjective evaluation ( 0.9, 0.001) and a significant negative correlation with mean gaze time (0.681, 0.015). The AMW significantly positively correlated with the subjective evaluation ( 0.809, 0.001) and reaction time ( 0.537, 0.072) and significantly negatively correlated with the correct rate (0.839, 0.001). The blink frequency was not related to the TMW and the AMW.

Figure 3.

Distribution of gaze points (left) and correlation between mental workload prediction and some experimental values (right).

Discussion and conclusion

The ergonomics simulation experiment verified the sensitivity of each measure indicator to the mental workload under different display interfaces and flight tasks, so the performance of embedded subtask, National Aeronautics and Space Administration-Task Load Index (NASA-TLX) subjective evaluation, and eye tracking could construct a comprehensive evaluation system of pilots’ mental workload in actual flight environment. The low correct rate, long reaction time, high subjective evaluation, low blink frequency, and short average gaze time could mean the overload of the pilots which should be avoided.

Since the main effect of NASA-TLX is significant in different flight phases, and the correlation between the results of NASA-TLX and the TMW compared with the AMW is better, the TMW can be measured by the subjective evaluation of NASA-TLX, which is consistent with the results of dynamic mental workload framework [13, 14]. The main effects of performance in different flight phases are also significant. AMW negatively correlates with accuracy and positively correlates with reaction time, showing that the performance of embedded subtasks is reliable to AMW. Consequently, the research framework with AMW, TMW. and IMW is more accurate than that with a single workload.

On one hand, studies have proved that gaze time not only is a sensitive indicator of task difficulty but also can be used to judge the readability of display interface [2, 5, 8, 10]. The main effect of the gaze time is significant under the display interface and in the flight phase. Furthermore, a significant positive correlation exists with the theoretical prediction under the display interface, while a significant negative correlation is present in the flight phase. This may be due to the fact that, to achieve better performance, participants allocate more mental resources to complete the embedded subtask under the condition of the lower workload of the main flight task, thus showing the opposite trend of the gaze time. On the other hand, related studies have explained that an increase in the perceived workload can lead to a decrease in the blink frequency, and an increase in cognitive workload can lead to a faster blink frequency [2, 5, 8, 17]. The main effect of the blink frequency in the flight phase is significant: takeoff straight 90 turn 180 turn. The possible reason why the theoretical prediction does not correlate with the blink frequency is that during the takeoff, the mental workload is mainly the cognitive workload caused by decoding memory, while during the cruise, the mental workload is mainly due to the perception workload caused by tracking flight information. With the increase in the turning angle, the perception workload increases and the blink frequency decreases, which is consistent with the theoretical prediction and verifies the prediction model.

The significant main effects and correlations in the flight phase indicate that the prediction model has good validity and availability for complex tasks with dynamic time variation. The display interface is only significant at the gaze time because the three interfaces are from active models that meet airworthiness standards and do not significantly increase pilots’ mental workload. Nevertheless, the trend is basically the same under each metric: the ARJ21 interface has the highest subjective evaluation, the lowest correct rate, the longest gaze time, and the lowest blink frequency. The difference between B737 and A320 is small, which is consistent with the theoretical prediction.

Compared to most of the existing mental workload quantitative models, the present study proposed a dynamic prediction model of mental workload based on timeline analysis. Through the timeline task analysis, complex flight tasks were decomposed into several simple task units. The McCracken-Aldrich scale was used to assign the task units to four channels, and the impact of task type and complexity on mental workload was comprehensively analyzed and quantified. Benefiting from the improved time pressure quantification method and the adjusted proportion of visual coding in the model, the accuracy of multi-factor mental workload prediction improved. Besides, the dynamic time conceptual framework of mental workload was introduced to realize real-time monitoring, analysis, and prediction of pilots’ mental workload status. As a result, the model could be applied to reduce aviation accidents by predicting overload of the pilots and proposing real-time and effective solutions throughout the flight phase. Meanwhile, without considering resource interference between multitasking, the model had limited accuracy in the multi-task environment. Follow-up studies may further improve the mental workload prediction model.