Real-time eye tracking for the assessment of driver fatigue.

Eye-tracking is an important approach to collect evidence regarding some participants' driving fatigue. In this contribution, the authors present a non-intrusive system for evaluating driver fatigue by tracking eye movement behaviours. A real-time eye-tracker was used to monitor participants' eye state for collecting eye-movement data. These data are useful to get insights into assessing participants' fatigue state during monotonous driving. Ten healthy subjects performed continuous simulated driving for 1-2 h with eye state monitoring on a driving simulator in this study, and these measured features of the fixation time and the pupil area were recorded via using eye movement tracking device. For achieving a good cost-performance ratio and fast computation time, the fuzzy K-nearest neighbour was employed to evaluate and analyse the influence of different participants on the variations in the fixation duration and pupil area of drivers. The findings of this study indicated that there are significant differences in domain value distribution of the pupil area under the condition with normal and fatigue driving state. Result also suggests that the recognition accuracy by jackknife validation reaches to about 89% in average, implying that show a significant potential of real-time applicability of the proposed approach and is capable of detecting driver fatigue.

Introduction

Detection of driver fatigue using electronic and information technology is receiving more and more attention in the driving safety assistance system [1, 2]. Driver fatigue is one of the most possible factors for traffic accidents due to the fact that it affects driver's ability to make decision, slow down reaction time and decrease driver's attention [3]. Some studies show that almost 15–20% of all fatal traffic accidents are related to driver fatigue and recent statistics estimate that annually 1200 deaths and 76,000 injuries can be attributed to fatigue-related crashes [4, 5], so that it is crucial necessary to promote the technologies for detecting or preventing driver fatigue.

The emergence of artificial intelligence and the rapid development of electronic and information technology provide more opportunity to detect driver fatigue. Fatigue usually divided into three different types: sensory fatigue, muscle fatigue and cognitive fatigue. It should be pointed out that there is no a precise and scientific definition for fatigue, and thus there is not any quantitative criterion to measure it. However, there are mainly three techniques for monitoring driver fatigue [6, 7]: vehicle driving parameters such as the speed, lane tracking, the steering wheel rotation; driver physiological parameters such as electrocardiogram, electrooculogram, electromyogram and electroencephalogram (EEG); driver behavior characteristics such as eye movement and facial expression. Furthermore, as one of the most vital features of face characteristics, eye movement can play a salient role in expressing the driver's physical and mental situation for a long-duration driving [8]. Particularly, there are some evaluation criteria such as PERCLOS, GAZE, AECS for revealing a relation between the fatigue and eye movement [9-11]. PERCLOS and AECS do not affect normal driving by its non-touch monitoring but its real-time performance of image processing is difficult. In gaze measurement, fixation time and pupil size are often used for object evaluation of driving fatigue [12, 13].

Numerous researches involve eye movement such as biological recognition, intelligent image processing and driver fatigue detection. Drivers hardly keep alertness when they are fatigue, and this is accompanied by some stable measurable changes in eye detection and eye tracking. Eye movements such as blinking, eye fixation and pupil size during driver fatigue can be utilised in the driver fatigue detection system [14, 15].

In this work, we focused on measuring the average fixation time of the driver's AOI and the pupil area of eyes, because this vision-based method is not intrusive and will not cause aversion to drivers, and it gives fast and accurate results by fuzzy K-nearest neighbour (FKNN) classification algorithm. Section 2 summarises the design and implementation of the system experiment, Section 3 demonstrates the methodology of the system, Section 4 discusses the results and discussion. The work's conclusions are described in Section 5.

System experiment

We proposed a novel automatic system based on the eye movement behaviours for driver fatigue detection including five main aspects (Fig. 1), where concerning the visual strategy studies on driving. The good sensitivity, real-time assessment and user friendliness of eye tracking and eye movement analysis can satisfy the importance of driver fatigue detection.

Fig. 1

System overview

In system experiment, ten young healthy men, range 19–24 years (mean = 22.4), participated in highway-driving simulator experiment. They were in good health and had normal sleep time. All experimental procedures were performed using a static driving simulator (the ZY-31D car driving simulator, produced by Peking ZhongYu Co., Ltd). In the experiment, subjects were seated on a soft chair without armrests in a quiet shielded room, watching a screen, and making appropriate responses according to the test scale and the indication screen. Then the driver's eye movement data was collected directly by the D-Lab surveillance system (Fig. 2). The D-Lab analysis system shows the real-time driving behavior and different state data and can also calculate the pupil size, visual trajectory, fixation time, pupil position and visual field information shown as in the lower left of Fig. 2 and captured by a head mounted device. The sampling frequency of D-Lab system is 25 Hz.

Fig. 2

Usage of D-Lab surveillance system

In order to get the driver into a state of fatigue, driving environment selected for this study was a highway with medium traffic density and the driving experiment was started at 2 p.m. On the highway, the environment is monotonous, boring, and required to maintain a driving speed between 60 and 80 km/h. Before the experiment, they practiced the driving task for 5 min to become acquainted with the experimental procedures and purposes. Five minutes later, eye movement data of the subject was begun to collect. Acquisition time is 20 min. The 20 min recording data was labelled as the subject's normal state. Then every 15 min, the participant was asked to evaluate their fatigue status according to the Chalder Fatigue Scale and Li's Subjective Fatigue Scale [16, 17]. According to these two scales, once the participant was in state of fatigue, the experimenter began to collect the subjects’ eye movement data and label as the fatigue data. Acquisition time is also 20 min. When the fatigue state data is collected, the whole experiment is over.

Methodology

During a visual task, eye movement behavior can be classified according to their characteristics [18]. We hereby describe the main eye movement feature parameters including the average fixation time and the pupil area, which accurately captured and operated by the D-Lab system (shown as in the lower right of Fig. 2). The detailed process of surveillance and information acquisition is as follows. The participant sat in a relatively spacious and enclosed room, which the window closed with curtains and the fluorescent lamp was also closed, in order to avoid the uncertainty of the pupil area caused by different illumination. The subject sat in the driver's seat for a high-speed driving simulation, which wore a head-mounted eye-tracking device (Dikablis glasses shown in the upper left corner of Fig. 2). This eye tracker has two cameras, namely A and B, the camera A can collect video and images within 270° in the line of sight in front of the subjects and the camera B is responsible for monitoring the subject's left eye movement. The camera can place as the red zone in the upper right corner of Fig. 2: hung on the auricle, balanced the nose and forehead with C and D, and then adjusted the focus independently via rotating two cameras. In the driving process, data such as the fixation time and pupil size changes are transmitted to the D-Lab system.

Fixations are the stationary states of the eyes during which a user is looking at a specific location in the visual scene. The duration of fixation is the time to stay focused on a single viewpoint. The length of the duration reflects the degree of difficulty in capturing and processing traffic information. At the same time, it also reflects the degree of interest of the driver in the field of vision. In addition, it can indirectly reflect whether the driver concentrated on driving and whether the driver was in fatigue state. In many studies, the average fixation times was used as evaluation index of mental fatigue. In this paper, the front area of the driver's simulation car is set to AOI. The D-Lab system was used to record the driver's fixation time in the AOI during the subject was driving in normal and fatigue state. Therefore, the average fixation time (T) in the AOI is defined as Fig. 3 shows the comparison of the fixation time in the AOI of No. 1 subject in the state of normal and fatigue, which demonstrating that there is significant difference with the fixation time due to the fact that the fixation time of fatigue state is basically longer than that of normal state. According to (1), the average fixation time of driver in the AOI under the condition of normal and fatigue state, respectively, was 359.001 and 426.993. It is also obvious that the average fixation time of fatigue state in the AOI was greater than that of normal state. Based on the t-test statistics of the fixation time data in two states, the value of p is 0.0015, <0.01, which show that there are obvious differences for average fixation time in two states. Therefore, it could be effective to use the average fixation time to distinguish driving states between normal and fatigue.

Fig. 3

Comparison of the fixation time of No. 1 subject in AOI area under the condition of normal and fatigue state

The pupil is a retractable circular hole in the centre of eye iris. The correlation between pupil diameter and mental fatigue was high [19]. Therefore, pupil diameter which can sensitively reflect the degree of mental fatigue is an effective index to detect driver fatigue. In this paper, the pupil area in driver's driving situation is used as a measure of pupil size, which collected the height and width of the driver's pupil for every 40 ms by the D-Lab system. The pupil area can be conveniently calculated by the formula, S = πab, which a and b represent the half of the height and width of the pupil size, respectively. Shown in Fig. 4, it is a diagram for describing the different pupil size of two subjects under the condition of normal and fatigue state. It is obviously observed that the pupil sizes of different subjects have major influence on the driver fatigue detection based on the pupil area. Thus, we should adopt a suitable strategy to avoid the difference of domain value distribution in the pupil area for each subject.

Fig. 4

Comparison diagram of the difference of pupil area under the condition of normal and fatigue state

a For normal state from No. 1 subject

b For normal state from No. 2 subject

c For fatigue state from No. 1 subject

d For fatigue state from No. 2 subject

Through statistical analysis, it can be found that the pupil size of all subjects was between 0 and 5000 px. Based on the t-test statistics of the pupil area data in two states, the value of p is 0.00001, <0.001, which show that there are significant differences for pupil area in two states. Fig. 5 shows the change of pupil area under the condition of normal and fatigue. It can be seen that concentration area of the pupil area data for normal and fatigue state is different. In order to reduce the difference of domain value distribution in the pupil area under the condition of normal and fatigue driving state, firstly the pupil area is divided into five regions as [0-1000], [1000-1500], [1500-1800], [1800-2000] and [2000–∞]. Then the probability of pupil area appearing in these five areas was calculated. The probability (p(i)) of each driver in these five regions is calculated as where the numerator indicates the number of samples in the corresponding region and the denominator indicates the total of samples number.

Fig. 5

Comparison of the pupil area of No. 1 subject under the condition of normal and fatigue state

Fig. 6 shows a comparison of the probability of pupil area appearing in the five regions for No. 1 subject under the condition of normal and fatigue state. It is obvious that the probability value of pupil area in the five regions is quite different. The majority of pupil area values in normal state were [1000-1500], [1500-1800] and [1800-2000], but the majority of pupil area values of fatigue state were [1800-2000] and [2000–∞]. Therefore, it is also effective to distinguish the normal and fatigue state by the probability of pupil area in five regions.

Fig. 6

Comparison of the probability of pupil area in the five regions of No. 1 subject under the condition of normal and fatigue state

In the whole process of detecting driver fatigue via eye movement behavior, the steps of eye movement data processing are shown in Fig. 7. In this work, owing to its good performance and ease of use, the FKNN classification method was used to train the training set and the optimal K value was calculated [20]. According to the optimal K value, the test set was classified. In this paper, the 80 groups from the 100 groups were randomly selected as the training set and the other 20 groups of data as the test set to test the FKNN classifier.

Fig. 7

Flowchart to show the operation process of eye-based fatigue detection system

Results and discussion

The testing dataset was input into the FKNN classifier, and had a comparison by the average fixation time as a single feature, the pupil area as a single feature, and their combination as fusion feature. The comparison classification results are shown in Fig. 8.

Fig. 8

Comparison classification average accuracy of using single feature and combined feature (*p < 0.05)

As can be seen from Fig. 8, the initial value of K is 1, and with the value of K increasing to 4, the single feature and the combined feature classification accuracy are generally improved. When the value of K is 3, the accuracy of each single feature and the combined feature were the highest, which are, respectively, 77.68, 79.82 and 88.75%. However, along with the K value continues to increase, the accuracy of using the single feature or combined feature all decreased. In addition, the accuracy of using combined feature almost achieved the best results except when the value of K is 2 or 5.

The jackknife cross-validation approach is known to be an almost unbiased estimation algorithm, which is often used to examine a classifier for its effectiveness in practical application [21]. Then the jackknife cross validation was used for evaluating the classification accuracy, the mean values of test results are shown in Table 1. Furthermore, in recent years, several teams worked on this problem via using facial features to study driving fatigue detection, a detailed review see [22]. For example, LV proposed an image processing method for fatigue recognition based on adaptive locality preserving projections and obtained a classification accuracy of 93.8% by FKNN of using adaptive neighbourhood selection strategy [23]. Lin studied a calculation method for driver fatigue detection based on machine vision to calculate the value of PERCLOS to determine the eye closure proportion in a given time and achieved the average precision of 84% by the adaboost cascade classifier [24]. Choi developed a gaze zone detection system of using deep learning techniques for recognising driver's gaze zone and the correct rate reached to 95% in average [25]. It is observed that image processing techniques by watching driver's facial expressions have reached quite a high level of sophistication, especially used of deep learning techniques [26]. In other applications, Yan studied the influence of light zones on the dynamic visual characteristics and information perception of drivers when they pass through the extra-long tunnels on highways, which recorded fixation duration and pupil area for predicting and analysing the influence by a back-propagation artificial neural network [27]. Ohn-Bar proposed a new in-vehicle, vision-based gesture recognition system for a customised multi-modal interface by image processing techniques [28]. In addition, the researches on the usage of eye movement behavior made a lot of research results shown in Table 2 show that the FKNN classifier used in this paper was superior to the adaboost algorithm and Harr algorithm, but is slightly close to the SVM algorithm, which implying that our result obtained a good performance.

Table 1

Test results given by jackknife cross validation

Subject no.	Accuracy, %
1	88.56
2	88.91
3	88.49
4	89.01
5	88.52
6	88.32
7	87.82
8	88.86
9	88.70
10	88.59
mean ± variance	88.75 ± 0.116

Table 2

Performance comparison of the previous works

Author	Classifier	Accuracy, %
Punitha [29]	SVM	93.50
Mbouna [30]	SVM	87.58
Hemadri [31]	Harr	80.00
Fan [32]	Adaboost	80.19
Mandal [33]	SVM	85.02
this paper	FKNN	88.75

Conclusion

In this paper, the driver's eye movement feature is collected by the simulation driving experiment and the D-Lab surveillance system, and the fatigue driving model based on FKNN algorithm is built to detect driver fatigue. The conclusions are as follows: Of course, there are some deficiencies in this conclusion. Since the experiment was carried out under the simulated driving environment and limited by the number of test samples and other experimental conditions, there is a certain deviation between the test results and the actual situation. In addition, because the sectional area of pupil area is estimated according to the distribution features of the sample area data, there is a certain degree of inaccuracy. Although this work has achieved a better performance, however, comparing to the EEG-based driver fatigue detection system [34-37], the detection performance is still weaker.

(i) The average fixation time of AOI and the pupil area can be used as an effective indicator of driver fatigue detection.

(ii) Owing to the difference of domain value distribution in the pupil area under the condition of normal and fatigue driving state, it has the influence of individual pupil sizes on the results. In this paper, the different pupil sizes were divided into five regions and the probability of pupil area appearing in the five regions was calculated.

(iii) For achieving a good cost-performance ratio and fast computation time, FKNN algorithm was used to detect driver fatigue and it hit a higher detection result.

In future work, we will further increase the accuracy of segmented regions by increasing the number of samples and the improved classification model will also be used in the later development of the driver fatigue detection system.

It is necessary to detect the fatigue state of the driver in time. Since the driving fatigue can affect the driver's attention, perception, thinking, judgement, decision and movement etc. Continue to drive after fatigue, the driver will feel sleep lethargy, weakness, inattention, and decreased ability to judge, even a trance or transient memory loss and so on. Therefore, the driver should rest and stop driving when tired.

Conflict of interest

None declared. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.