Visualization and quantification of eye tracking data for the evaluation of oculomotor function.

Oculomotor dysfunction may originate from physical, physiological or psychological causes and may be a marker for schizophrenia or other disorders. Observational tests for oculomotor dysfunction are easy to administer, but are subjective and transient, and it is difficult to quantify deviations. To date, video-based eye tracking systems have not provided a contextual overview of gaze data that integrates the eye video recording with the stimulus and gaze data together with quantitative feedback of metrics in relation to typical values. A system was developed with an interactive timeline to allow the analyst to scroll through a recording frame-by-frame while comparing data from three different sources. The visual and integrated nature of the analysis allows localisation and quantification of saccadic under- and overshoots as well as determination of the frequency and amplitude of catch-up and anticipatory saccades. Clinicians will be able to apply their expertise to diagnose disorders based on abnormal patterns in the gaze plots. They can use the line charts to quantify deviations from benchmark values for reaction time, saccadic accuracy and smooth pursuit gain. A clinician can refer to the eye video at any time to confirm that observed deviations originated from gaze behaviour and not from systemic errors.

Introduction

Evaluation of oculomotor function and subsequent identification of oculomotor dysfunction may serve different purposes. In the developing child, adequate oculomotor functioning is necessary to support the development of other basic functions such as eye-hand coordination, as well as the development of functional skills such as reading and writing (Schneck, 2010). The assessment of potential contributing factors, such as oculomotor dysfunction, to problematic development of basic functions and functional skills, is vital to the development of therapeutic interventions (Schneck, 2010).

The evaluation of oculomotor functioning can also assist towards the diagnosis and monitoring of neurodegenerative disorders (Antoniades and Kennard, 2015). Although eye movements are not routinely assessed for the initial diagnosis of neuro-degenerative disorders, recordings of eye movements may provide supportive data regarding disease severity, progression of the condition and regression of functional abilities in the patient. In addition to tracking unique manifestations of neurodegenerative conditions in patients, recordings of eye movements may also be used to evaluate the effectiveness of proposed neuroprotective and neuro-restorative therapies (Anderson and MacAskill, 2013).

The purpose of this paper is to introduce a novel computer-based eye tracking system for the evaluation of oculomotor functioning, particularly saccades and smooth pursuits, through the integrated visualisation of eye movements and quantification of eye tracking data. Even though evaluation and testing procedures are standardised, results from clinicians are variable and the same results are interpreted differently by different clinicians. By tracking the eye movements on camera and having the “luxury” to review eye movements repeatedly, better diagnoses and conclusions may be made.

We do not propose a new visualisation technique, but rather a new combination of existing techniques to provide a multidimensional and integrated view of the same data. To date, remote video-based eye tracking systems have not provided a contextual overview of gaze data that integrates the eye video recording with the stimulus and gaze data. The contribution of this paper lies in the introduction and description of an integrated system to allow clinicians and researchers to obtain multiple views of an eye movement recording. This will allow detailed observations and analyses of oculomotor function and abnormalities, for example abnormal visual reaction times, over and undershoots, and smooth pursuit impairments. The system addresses shortcomings in current video-based eye tracking systems as well as challenges with psychometric and observational tests of oculomotor function.

This integrated system will be of interest to clinicians such as neurologists, optometrists and occupational therapists to evaluate oculomotor function as part of diagnostic, monitoring and therapeutic procedures. Researchers who are interested in eye movements and eye movement visualisation techniques will also benefit from the system. Although at this stage the system does not yet provide normative data, it allows future studies to develop norms for comparison of quantitative measurements.

The paper is organised as follows: The literature review covers two main areas, namely

an overview of eye movements that are relevant to this paper, with a focus on saccades and smooth pursuits, and

current practices and challenges in the evaluation of oculomotor function.

The literature review is followed by a detailed description of the proposed system for the integrated visualisation of eye movements and quantification of eye tracking data. Four main areas are addressed in the description of the system, namely

the hardware and software;

a description of specific tasks to generate saccade and smooth pursuit data;

the proposed visualisations of saccadic and smooth pursuit data and how they are integrated and presented in context with the gaze target, as well as the various possibilities for customisation, filtering, smoothing and corrections;

the proposed method for quantification of eye tracking data in statistical graphs and how they should be used to quantify aspects of gaze behaviour and identify abnormal gaze behaviour.

The paper is concluded with a short summary and reiteration of the application value of the system for clinicians and researchers.

Background

Research on eye movements go back several years and there has long been agreement that the most common type of eye movement is a rapid jerk (referred to as a saccade), followed by a period of 0.1–0.3 s during which the eye is relatively stationary (referred to as a fixation) (Duke-Elder, 1932; Carmichael and Dearborn, 1948; Barlow, 1952). The eyes are, however, never completely still and tremor (a.k.a. as optokinetic nystagmus), micro-saccades and drift occur during fixations (Engbert and Kliegl, 2003). These fixational eye movements overcome neural adaptation and prevent visual fading (Martinez-Conde and Macknik, 2008).

A detailed description of eye movement classes are beyond the scope of this literature review. The proposed visual tracking system is mainly designed to investigate saccades and smooth pursuits, and therefore, emphasis will be placed on the review of these two groups of eye movements.

Saccades

Saccadic eye movements are ballistic and change the point of fixation rapidly (Sunderland, 2001). Saccades range in amplitude from the small movements made while reading, for example, to the much larger movements made while gazing around a room. Saccades can be elicited voluntarily, but also occur reflexively whenever the eyes are open. Neither voluntary effort, nor practice, will alter the velocity of saccadic movements (Becker and Fuchs, 1969).

Different types of saccades can be distinguished, for example memory-guided saccades, anti-saccades and visually guided saccades (Anderson and MacAskill, 2013; Holmqvist et al., 2012, p 305; Anderson and MacAskill, 2013). The focus in this paper will be on normal visually guided saccades.

Human saccades can be described in terms of their duration, magnitude and peak velocity. It takes about 200 ms for the eyes to respond after onset of a target (Engel et al., 2000; Sunderland, 2001). During this delay, the path that the eyes must move to position the fovea on the target is computed and the extra-ocular muscles are activated to move the eyes towards the target (Sunderland, 2001). Since the saccade-generating system cannot respond to changes in the target position during a saccade, saccades are said to be ballistic (Sunderland, 2001; Becker and Fuchs, 1969). If the target moves again during this time, the saccade will miss the target, and a second saccade must be made to correct the error.

The length of a saccade, referred to as its amplitude, is usually measured in degrees or minutes of arc. The saccade gain is the ratio of the actual saccade amplitude divided by the desired saccade amplitude. Gains of <1 indicate that the saccade was too small or hypometric; gains of >1 indicate that the saccade was too large or hypermetric. Three types of visually guided saccades can be distinguished, namely catch-up saccades (towards target from behind), anticipatory saccades (gaze moving ahead of target towards anticipated target position) and over- and undershoots (towards target, but missing the target slightly, followed by a correcting saccade) (Van Gelder et al., 1995).

Abnormal saccadic behaviour has been documented well in literature (Dowiasch et al., 2015). Causes may include increasing age, alcohol consumption, intravenous Valium, lack of natural sleep and drowsy states of the diurnal cycle, a depressed or dysfunctional oculomotor system (Becker and Fuchs, 1969). Several neurodegenerative conditions that may affect saccadic movements have been investigated, including Lewy body Parkinsonian diseases, Huntington disease and dementia disorders (Anderson & MacAskill, 2013, 2016) and autism spectrum disorder (ASD) (Benson et al., 2009; Zalla et al., 2016).

Smooth pursuit

Dodge, in 1903 (cited in Robinson, 1965), was among the first to draw a distinction between saccadic and smooth pursuit human eye movements. Saccades are primarily directed at stationary targets whereas smooth pursuit movements are used to track moving targets (Orban de Xivry and Lefevre, 2007). The principal objective of a smooth pursuit is to maintain smooth eye velocity close to object velocity, thus minimising retinal image motion (the so-called retinal slip) and maintaining visual acuity (Barnes, 2008). This skill is especially important in most sports as it allows participants to catch, hit or kick a moving ball, for example.

Smooth-pursuit eye movements are continuous, slow rotations of the eyes (Spering and Gegenfurtner, 2008) that are used to stabilise the image of a moving object of interest on the fovea (Their and Ilg, 2005; Nagel et al., 2012; Lisberger et al., 1987). Minor eye movements, such as tremors and micro-saccades, occur also during smooth pursuit although they do not necessarily show in low-resolution eye-tracking data (Santini et al., 2016).

Smooth pursuit movements are voluntary in the sense that the observer can choose to track a moving stimulus or not (Sunderland, 2001). However, conscious attention is needed to maintain accurate smooth pursuit (Hutton and Tegally, 2005; Madelain et al., 2005).

For normal subjects, smooth pursuit gain is expressed as the ratio of smooth eye movement velocity to the velocity of a foveal target (Sharpe, 2008). If the gain is less than 1, gaze will fall behind the target to create a retinal slip that will have to be reduced by one or more “catch-up” saccades (Van Gelder et al., 1995). According to Meyer et al. (1985), normal subjects can follow a target with a gain of 90% up to a target velocity of 100 deg/s.

Because of the inherent delay of visual processing and oculomotor reaction, positional errors arise when a moving stimulus changes its trajectory unexpectedly as the change in smooth eye velocity cannot be instantaneous (Orban de Xivry and Lefevre, 2007).

Smooth pursuit eye movement can respond at the onset of target movement or unexpected change in direction after about 100–150 ms (Bahill and McDonald, 1983; Engel et al., 2000; Orban de Xivry and Lefevre, 2007). Becker and Fuchs (1985) found that smooth pursuit velocity decelerates rapidly about 200 ms after a target had disappeared and stabilises at a gain between 0.4 and 0.6 if subjects attempted to continue tracking the invisible target (Bahill and McDonald, 1983; Soechting et al., 2010; Ross and Santos, 2014). Previous experience and learning can increase the efficiency of smooth pursuit (Von Hofsten and Rosander, 1997; Zambrano et al., 2010). When a target moves in an unpredictable fashion, smooth pursuit and so-called catch-up saccades are used to avoid large position error and prevent the eye gaze from lagging behind the target (De Brouwer et al., 2001; Orban de Xivry and Lefevre, 2007).

A normal change in the environment may cause a change in the performance of a smooth pursuit related task, such as attention or background changes behind the target (Van Donkelaar and Drew, 2002; Souto and Kerzel, 2011). Smooth pursuit performance is also affected by target position (Pola and Wyatt, 2001), target velocity (Kowler and McKee, 1987; Meyer et al., 1985), target visibility (Becker and Fuchs, 1985; Pola and Wyatt, 1997), target direction (Engel et al., 2000) and predictability of target direction (Soechting et al., 2010).

Abnormal smooth pursuit movements may be the result of increasing age (Morrow and Sharpe, 1993; Dowiasch et al., 2015) or various disorders and dysfunctions, ranging from low to high severity across a spectrum of conditions. Smooth pursuit impairment and dysfunction can, for example, be linked to mental illnesses such as schizophrenia (Levin et al., 1988; Holzman et al., 1976; Holzman and Levy, 1977; O'Driscoll and Callahan, 2008), physical anhedonia and perceptual aberrations (Simons and Katkin, 1985; O'Driscoll et al., 1998; Gooding et al., 2000), Alzheimer's disease (Fletcher and Sharpe, 1988; Antoniades and Kennard, 2015), Huntington's disease (Antoniades & Kennard), attention-deficit hyperactivity disorder (ADHD) (Fried et al., 2014) and autism spectrum disorders (Wilkes et al., 2015; Johnson et al., 2016).

Current practices and challenges in the evaluation of oculomotor function during saccades and smooth pursuit

Different approaches for the evaluation of eye movements may be used with different objectives. The current methods used to diagnose OMD include psychometric (e.g. the King-Devick, and DEM tests), observational (e.g. NSUCO), and computer-based tests (Iyer et al., 2010). According to Iyer et al. (2010), psychometric tests are easily administered, have acceptable norms and are reasonably objective, but are loaded with cognitive factors that may skew the results. These tests are also difficult to administer to younger children and patients with cognitive deficits (Iyer et al., 2010).

With observational tests, such as the North-eastern State University College of Optometry oculomotor test (NSUCO) (Maples and Ficklin, 1990) and broad H-test (Carlson, 2004), clinicians hold an object in front of a patient with the instruction to follow the object with the eyes while the clinician inspects the patient's eye movements subjectively as the object moves. While observational tests have the advantage of not being cognitively loaded, they are subjective and depend on the skills of the examiner (Iyer et al., 2010). Furthermore, observations are transient and it is difficult to quantify deviations. If the clinician misses a small jerk of the pupil, the moment is lost. Even though internationally standardised observational tests are widely used amongst clinicians, their test-retest reliability and inter-rater reliability is questionable.

The above-mentioned limitations of psychometric and observational tests necessitate the development of an objective evaluation system for the evaluation of eye movements. Video-based eye tracking has been used in the past to detect and evaluate ocular function (Ivanov et al., 2016; Chouinard et al., 2017; Kasneci et al., 2017), but some manufacturers do not make the eye video available to users and clinicians have to rely on the processed gaze data for conclusions. Many systems make use of a head-mounted eye tracker, which is not desirable when working with populations such as young children or individuals with cognitive impairments as they often do not tolerate the head-mounted eye tracker well. We are not aware of any remote desktop eye tracking application that allows clinicians to apply their specialist knowledge to examine the eye movements post-hoc and analyse the gaze data in context with the original stimulus.

Instrumentation

Video-based eye tracking devices record gaze points of a patient1 (a.k.a. samples) at a regular interval or framerate. The framerate of the eye camera determines the interval between samples. Depending on the framerate, a large amount of data can be captured in a short period. Software was developed to analyse the recorded data and provide a multidimensional and integrated view of visual behaviour – facilitating identification of abnormalities or pathologies for a specific individual.

A self-built eye tracker with two infrared illuminators, 480 mm apart and the UI-1550LE camera from IDS Imaging (https://en.ids-imaging.com) was used to capture gaze data (Fig. 1). The camera has a daylight cut filter (allows only infrared), a 1600×1200 sensor with pixel size 2.8 μm and a native framerate of 18.3 fps. Software was developed using C# with .Net 4.5 along with the camera manufacturer's software development kit (SDK) to control the camera settings and process the eye video. The camera allows the selection of a smaller area of its sensor to be communicated through USB 2, which allows a higher framerate. The examples that are shown in this paper were recorded with a framerate of 200 Hz (5 ms intervals between samples). This is necessary to register saccades (typically 500°/s) and measure the latency between onset of a stimulus and visionary reaction accurately (typically 200 ms). Data was recorded with a desktop computer with an i7 processor and 16 GB of memory, running Windows 10.

Fig. 1

Eye tracker hardware consisting of two infrared illuminators and camera.

Although the above hardware configuration is not freely available, the focus of this paper is on the method and not as much on the specific hardware that was used. The method can be incorporated into any software system that analyses eye-tracking data.

Every frame that is captured by the eye camera was analysed and the centres of the pupils and the corneal reflections (glints) were identified. The position of the pupils change with respect to the glints as the eyes rotate and a regression-based approach was followed to map the pupil-glint vector to a point of regard in display coordinates. The regression coefficients are determined through a calibration process. Details of the calibration and gaze mapping are beyond the scope of this paper.

To reduce clutter and simplify analysis and plotting of gaze data in a spatio-temporal fashion, gaze data samples are often classified, through the use of some or other algorithm (referred to as a filter), as belonging to either fixations or saccades (Spakov and Miniotas, 2007). Algorithms exist also to detect smooth pursuit (Larsson, 2010).

The ability of humans to make sense of raw gaze data is considered paramount to that of algorithms. For shorter recordings and with good visualisation techniques, it is needless to spend time with an error-prone algorithmic classification. See the ease with which under- and overshoots can be identified in Fig. 2, for example.

Fig. 2

Gaze plot for a patient showing the raw data samples of eyes combined for a saccades test using targets at random positions. The encircled clusters indicate the occurrence of under- and overshoots.

As was the case for the hardware configuration, the specific software that was used is not the primary focus of this paper. The contribution of this paper lies with the proposed integrated method of analysis as a principle and not with the specific implementation thereof.

Experimental

Patients can be instructed to follow a visual target on the computer display that either appears at specific positions for 1.5 seconds at a time (saccades test) (Clips 1a and 2a) or follows a predefined trajectory at a velocity that could be specified between 1 deg/s and 40 deg/s (smooth pursuit test) (Clips 3a, 4a and 5a).

The reader may refer to the accompanying video clips for a clearer understanding of the various tasks and visualisations. Please note, however, that the screen recording software placed an extra burden on the computer resources with the result that target movements and visualisations do not appear as smooth as they normally are.

All recordings that were made as part of this study was carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). All patients provided consent that their gaze may be recorded and recordings were made in accordance with the approval obtained from the ethical committee of the Faculty of Health Sciences of the University of the Free State.

For a saccades test, the clinician can choose to display the targets at random positions on the display or to use the so-called NSUCO (Northeastern State University College of Optometry) oculomotor tests (Maples and Ficklin, 1990) as adapted for use on a computer display. Fig. 2 shows a gaze plot of the two eyes combined for the saccades test using targets at random positions (see also Clip 1b). Note also the easy visual localisation of under- and overshoots. Please note that Figs. 2, 3, 4, 7, and 8 are printed smaller to preserve space, but the original images filled about 90% of a 22″ display, i.e. subtending a visual angle of about 40°.

Fig. 3

Gaze plot for a patient showing the raw data samples of the left eye (green), the right eye (red) and the eyes combined (blue) for a smooth pursuit test with a bouncing ball. The target trajectory is shown in black. The periodic vertical deviations are the result of blinks.

Fig. 4

Gaze plot for a patient showing the raw data samples of the left (green) and right (red) eyes for the NSUCO smooth pursuit test. The annotations indicate the direction and sequence of target movements.

Fig. 7

The same data as in Fig. 3 after localised positional adjustments and filtering of blinks.

Fig. 8

The same data as in Fig. 7 after smoothing with a window size of 31 samples.

For the normal NSUCO saccades test, the examiner holds two targets at a distance of no more than 40 cm in front of the patient, and 20 cm apart (gaze angle about 28°). The patient is requested to look from one target to the other on command while the examiner observes the saccadic eye movements and rates the patient according to a rating scale in four categories: head movement, body movement, ability and accuracy. After every trial, the targets are rotated clockwise so that vertical, horizontal and diagonal saccades can be examined. Five round trips (or ten saccades from the one target to the other) have to be made. For the current system, this test is adapted by displaying the targets at 34 cm apart to retain the prescribed gaze angle at a gaze distance of 700 mm. One blue and one grey target is displayed and the patient is requested to look at only the blue target as it switches position with the grey target (Clip 2a).

For a smooth pursuit test, the clinician can choose between several possibilities. The target can move at various angles, for example horizontally, vertically or diagonally across the display and bounce back from the edges (hence the name “bouncing ball”) (cf Clips 3a and 3b). The target trajectory can also follow the lines of a pattern (referred to as a broad-H), which will test upward, downward and horizontal pursuits (cf Clips 4a and 4b). Fig. 3 shows a gaze plot of a smooth pursuit test with a bouncing ball. Note the periodic vertical deviations that are the result of blinks.

For the normal NSUCO smooth pursuit test, the examiner holds a single target at a distance of not more than 40 cm from the patient and moves it in a circle with a diameter of 20 cm. For the adapted test on a computer display, the radius was set at 17 cm to retain the prescribed gaze angle at a gaze distance of 700 mm (Clips 5a and 5b). Two rotations are made clockwise and two rotations counter-clockwise with a horizontal sweep along the midline when changing direction. Fig. 4 shows the gaze data of the left and right eyes for a NSUCO smooth pursuit task. The radius of the clockwise and counter-clockwise circles are different to allow for easier separation of the gaze data during replay and visual inspection.

Analysis

Visualisation techniques

Visualisation of gaze data allow researchers to analyse recorded eye tracking data in an exploratory and qualitative way (Blascheck et al., 2017). Visualisation techniques help to understand spatiotemporal aspects of eye tracking data and complex relationships within the data, and can serve as an indicator of the mental workload of a participant (Di Nocera et al., 2016).

The type of visualisation depends very much on the specific task that a participant must perform and the research question that is asked. There is no single “all-in-one” visualisation to solve all possible analysis tasks (Kurzhals et al., 2017). Gaze plots (or scan paths) (cf. Dolezalova and Popelka, 2016) and attention maps (or heat maps) (cf. Stellmach et al., 2010) include qualitative information about visual perception of the underlying stimulus.

Statistical graphics (Blascheck et al., 2017) can be used to present quantitative data while preserving the advantages of visualisations. They can be presented in-context, i.e. the stimulus and visualisation can be linked and synchronised with each other, and different plots (e.g. left and right eye data, or gaze and target position) can be superimposed on top of one another for easier comparison (cf. Hain, 2012).

Visualisations of saccadic and smooth pursuit data

This paper argues that an integrated visual presentation of the original data as a gaze plot that is linked to the eye video, statistical graphics and target, will allow the analyst to identify gaze data samples as belonging to one of the four classes of data (fixations, saccades, smooth pursuit and noise) and read off quantitative metrics as they apply at specific moments in time.

An interactive facility with a slider on a timeline is provided to allow the analyst to scroll through a recording timewise and compare data from three different sources at the same moment in time (cf Clips 2c and 5c). By inspecting the overall gaze plot, the clinician can quickly scroll to the applicable frame of the eye video and inspect it for possible abnormalities. The granularity of the scrolling depends on the framerate of the camera, as the analyst is able to scroll frame by frame through the data.

In this way, the analyst can, for example, infer whether a blink is responsible for a sudden outlier in the spatial representation of gaze data or an unexpected peak in the graph of eye velocity against time. In other words, it can be inferred that data that appears to be part of a saccade is in fact not a saccade.

Gaze plots

For the gaze plots (cf. Fig. 2), the analyst has the option to display data from the left eye, the right eye and/or the average of both eyes. This is valuable as it often happens that data from the two eyes are separated, even for a brief period, which could be symptomatic of pathological issues. The analyst also has the option to show links between the sample points. In order to infer the temporal order of samples, links are shown in all gaze plots in this paper. Fig. 5 shows the control panel for gaze plot visualisations with the various options for customisation that exist.

Fig. 5

Control panel for gaze plot visualisations.

Eye video

The eye video (Fig. 6, Clips 1b, 2b, 2c, 3b, 4b and 5b) provides a frame-by-frame image of the patient's eyes. The frame changes as the patient scrolls through the timeline with synchronised changes in the gaze plot and graphs. The eye video provides answers for irregularities in the data, e.g. blinks, look away, head movement out of the box, sharp gaze angles, extra reflections, drooping eyelids, etc. In other words, if a graph shows an unexpected peak, the analyst can scroll to that position on the timeline and inspect the eye image at that moment.

Fig. 6

A frame from the eye video that is linked with the gaze plot of raw data samples. The frame changes as the analyst scrolls through the timeline. The crosshairs show the reported centres of the pupils and corneal reflections.

Statistical graphs

The analyst can choose between several statistical graphs to display quantitative information about gaze position, direction and velocity of saccades, gain with respect to the target as well as accuracy and precision of fixations, against time. Using the timeline, the analyst could position the slider on a specific event and inspect the gaze plot and eye video at the same moment. These graphs are discussed in more detail in Section 5 below.

Corrections for inaccurate or noisy gaze data

Corrections for inaccurate recordings

The difference between a target and the reported point of regard (POR) may be expressed either in terms of the distance (measured in degrees of gaze angle) between the target and the POR at a specific moment in time, or in terms of the time that it will take the POR to reach the current position of the stimulus (measured in milliseconds). Unfortunately, it is not as simple as a mere comparison of position at a specific timestamp, since inherent system inaccuracy or calibration errors would affect the distance between the target and the POR. Therefore, an experiment that involves smooth pursuit should be designed in such a way that the target direction changes from time to time and the trend should be examined over a period of time rather than at a few specific moments in time.

A visual inspection of this trend could also indicate whether offsets are caused by patient errors or system inaccuracies. In Fig. 3, it is clear that for the largest part of the target trajectory, the gaze data follows the path with reasonable accuracy. In the bottom-left corner, however, there are clear signs of systematic error, i.e. there is a trend of offsets in some regions of the trajectory. The system allows for global or localised corrections by dragging the gaze data within a specified radius (that can be adjusted by the analyst) in a specific direction so that the data is aligned with the stimulus path (Fig. 7, Clip 3b). Note that these corrections only apply to position with respect to the target. Velocities are calculated for point-to-point sample data and are not affected as consecutive points are corrected with the same amount and in the same direction.

Filtering and smoothing

Filtering of gaze data can be done to exclude samples with point-to-point distances that are larger than a specific threshold. The threshold can be adjusted by the analyst. In Fig. 7 above, a threshold of 1.0° was used to filter out the periodic vertical deviations that are the result of blinks.

The gaze plots can also be smoothed through median filtering that uses a sliding window of a specified number of samples. Fig. 8 shows the same data as in Fig. 7 with the data smoothed with a window size of 31 samples. Although smoothing may hide many important events in the gaze data, it serves well to cater for system instability.

Quantification of eye tracking data through graphs

Graphs for the saccade tests

Fig. 9 shows a line chart that are used to visualise point-to-point velocity for the saccades test with targets at random positions. The vertical red line indicates the timestamp at which the eye video and gaze plot are synchronised. This line moves over the graphs as the mouse cursor moves (Clips 2c and 5c).

Fig. 9

Line chart of the saccade velocity for the saccades test with targets at random positions.

The onset of targets are indicated with black diamonds and the delay between target onset and the saccade towards the target is shown for every target. The average reaction time of this participant was 267.7 ms over 9 targets. Note the annotations that indicate an initial long saccade, which lands somewhat short of the target and is followed by a shorter correcting saccade.

Using the dynamic and interactive nature of the line charts, the analyst can mark the exact time of onset and completion of saccades and fixations, which can be used to determine the durations of these events.

Graphs for the smooth pursuit tests

In this system, two steps were taken to correct for the inherent system noise that go with remote video-based eye tracking:

Gaze velocity was resolved into two perpendicular components – one along the direction of target movement and one perpendicular (normal) to it. Only the parallel component should be compared with target velocity.

Gaze velocity was calculated as an average over time. The time interval can be changed by the analyst.

Fig. 10 shows line charts for the direction of target movement, position as normalised X and Y values of the display dimensions and the two components of gaze velocity superimposed on the target velocity. Once again, the vertical red line indicates the timestamp where the eye video and gaze plot are paused.

Fig. 10

Line charts for two components of gaze velocity superimposed on the target velocity (top) and gain (bottom).

Catch-up saccades can easily be identified where the blue plot is above the target velocity (gain > 1). Anticipatory saccades can have either positive or negative gain. When a target is approaching a fixed object, e.g. the edge of the display, it may jump ahead (gain > 1) and then slow down (gain < 1) to allow the target to catch up. If the patient expects the target to bump against the edge and return, the gaze can turn short (lazy eye) with the result of a temporary lead on the target, which can also be corrected by slowing down the gaze while the target finishes the turn.

Fig. 11 shows line charts for the relative position of the POR with respect to the target. If the POR is behind the target, which is the normal behaviour, the positional as well as the time differences are negative. If the POR is ahead of the target, as is possible for anticipated gaze, the positional as well as the time differences are positive.

Fig. 11

Graphs of distance (top) and time (bottom) between the POR and target. If the POR is ahead of the stimulus, the positional and time differences are positive. If the POR is lagging behind the stimulus, these values are negative.

Related work

Comparison with a manual system

Although the focus of this paper is on the tool itself and not as much on experimental results, a study was done to compare the results of eye movement tests done with this computer-based eye tracking system with the current gold standard manual observations (Schutte et al., 2018). Thirty-three healthy young adults (ages 21–24) were recruited to participate in this comparative study. Three manual tests were performed: saccadic movements and smooth pursuits using the double-O (NSUCO, Maples and Ficklin, 1990) and double-H (Elliot, 2014) methods. The same measurements were then recorded using the computer-based eye tracking system. Findings of the manual tests and computer-based tests were then compared for similarity.

In summary, the results revealed that for the saccadic movements, similar results were obtained for the majority of the patients, with the computer-based results showing a greater sensitivity to pick up over- and undershoots that were missed during manual testing. For the smooth pursuit measurements (double-O and double-H tests) the similarity was high. All statistical comparisons showed no significant differences (p > 0.05) between manual and computer-based testing, indicating good comparability of the findings between the two systems. However, it is evident in the results that there is a tendency for the computer-based system to pick up minor deflections in eye movements that was missed with the naked eye during manual testing. Consequently, the results underscore the potential of this system to enhance the accuracy of the assessment and analysis of eye movements in the clinical setting.

Limitations

We are not aware of any commercially available system that offers computer-based eye movement tests and presents the results in the form of an eye video, gaze plot and statistical graphs immediately after a recording in an integrated and synchronous manner. There is no reason why the proposed approach towards diagnosis and examination of results cannot be implemented by commercial software developers, but, alas, clinicians such as optometrists still have to rely on their skills and experience to interpret observations.

Patients with visual or cognitive impairments that prevent them from focusing cannot be tracked with this system. For example, a person with strabismus or severe nystagmus is unable to focus on a stimulus, and gaze plots or statistical graphs might not be available. What is possible, however, is that the eye videos of such participants can still be recorded and the eye video will always be available for post-hoc analysis, future reference or consultation with a colleague.

The system could be expanded to include other visualisation techniques such as heatmaps, bee swarms, convex hulls around fixations, comparisons of scan paths, statistics on areas of interest, etc., but the system in its current state is deemed sufficient for the purpose of clinical interpretation.

Conclusions

The main contribution of this paper is the on-screen presentation of stimuli (as opposed to a clinician asking a patient to focus on a physical object) and the multi-modal presentation of eye tracking results in an integrated and synchronised manner (as opposed to a clinician observing and subjectively interpreting transient eye movements).

Normally, when using observational tests to evaluate oculomotor function, clinicians hold an object in front of a patient with the instruction to follow the object with the eyes while the patient's eye movements are inspected visually. This procedure is subjective and will have a great inter-examiner variability due to variability in the way that stimuli are presented, and inter-examiner variance in interpretation of findings. Small movements made by the eye(s) may also be missed due to the transient nature of eye movements and the natural inability of observers to pick up small deflections in eye movements.

In the past, video-based eye tracking has been used for the same purpose (Kooiker et al., 2016), but as far as we are aware, synchronised and integrated visualisation of (i) the eye video, (ii) scan plot and (iii) statistical graphs have not been used to analyse saccadic and smooth pursuit dysfunction.

Saccadic and smooth pursuit performance may be affected by physical, physiological and psychological aspects and abnormal performance in this regard may serve as indicator of pathological issues. Clinicians would, however, prefer to apply their specialist knowledge to inspect a recording of the original clinical observations, rather than relying solely on error-prone systems to do the interpretations on their behalf.

With the use of a high quality, programmable, off-the-shelf camera to construct an eye tracker capable of recording data at a framerate in excess of 200 Hz, a system was built that included several tests with which saccade and smooth pursuit performance can be evaluated. Adaptations of the standard NSUCO tests for saccades and smooth pursuit are also included.

The recorded data can afterwards be analysed through replay of the eye video, a gaze plot of samples overlaid on the target trajectory and line charts that provide quantitative feedback on saccade and smooth pursuit performance. Instead of aggregating the samples into fixations and saccades, it was argued that presentation of the gaze plots with the original data is justified because of the relative short recordings and because humans would find it easy to identify clusters of raw data visually.

Blinks or other unwanted data could be filtered out while signs of micro-saccades, nystagmus and system instability could be ironed out through smoothing, if necessary. Systematic errors could be handled through global or localised corrections by dragging all samples within a specified radius to match the target trajectory.

The visual and integrated nature of the analysis allows localisation and quantification of saccadic under- and overshoots as well as determination of the frequency and amplitude of catch-up and anticipatory saccades. Furthermore, the statistical graphs provide exact feedback on reaction time, saccade durations and smooth pursuit gain.

By inspecting the overall gaze plot or one of the statistical graphs, the analyst can quickly scroll to a connected frame of the eye video and inspect it for possible abnormalities. In this way, the clinician has the best of three worlds: the gaze plot provides an overview of the recording, the statistical graphs provide processed quantitative data and the eye video provides visual confirmation of the trends that are observed.

In conclusion, it is argued in this paper that an integrated visual presentation of the original data as a gaze plot that is linked to the eye video, statistical graphics and target, will allow the analyst to identify gaze data samples as belonging to one of the four classes of data (fixations, saccades, smooth pursuit and noise) and read off quantitative metrics as they apply at specific moments in time. The overall average accuracy and precision measurements can aid in future standardisation of reference values. Based on abnormal patterns in the gaze plots, a clinician will be able to identify oculomotor dysfunction that may be related to neurodegenerative, neurological, pervasive developmental and other disorders. The line charts can be used to quantify deviations from benchmark values in literature for reaction time, saccadic accuracy and smooth pursuit gain. The clinician can at any time refer to the eye video to confirm that observed deviations originated from gaze behaviour.

Declarations

Author contribution statement

Pieter Blignaut: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Marsha Oberholzer, Elize Janse Van Rensburg: Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This work was supported by the special project for inter-disciplinary research at the University of the Free State.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.