Controlling an effector with eye movements: The effect of entangled sensory and motor responsibilities.

Restoring arm and hand function has been indicated by individuals with tetraplegia as one of the most important factors for regaining independence. The overall goal of our research is to develop assistive technologies that allow individuals with tetraplegia to control functional reaching movements. This study served as an initial step toward our overall goal by assessing the feasibility of using eye movements to control the motion of an effector in an experimental environment. We aimed to understand how additional motor requirements placed on the eyes affected eye-hand coordination during functional reaching. We were particularly interested in how eye fixation error was affected when the sensory and motor functions of the eyes were entangled due to the additional motor responsibility. We recorded participants' eye and hand movements while they reached for targets on a monitor. We presented a cursor at the participant's point of gaze position which can be thought of as being similar to the control of an assistive robot arm. To measure eye fixation error, we used an offline filter to extract eye fixations from the raw eye movement data. We compared the fixations to the locations of the targets presented on the monitor. The results show that not only are humans able to use eye movements to direct the cursor to a desired location (1.04 ± 0.15 cm), but they can do so with error similar to that of the hand (0.84 ± 0.05 cm). In other words, despite the additional motor responsibility placed on the eyes during direct eye-movement control of an effector, the ability to coordinate functional reaching movements was unaffected. The outcomes of this study support the efficacy of using the eyes as a direct command input for controlling movement.

Introduction

There are approximately 294,000 individuals living with a spinal cord injury (SCI) in the United states. Of these individuals, less than 1% experience neurological recovery and almost 60% of injuries result in tetraplegia, or paralysis of all four limbs [1]. For these individuals, restoring arm and hand function has been indicated as the most important factor for maintaining independence [2]. Assistive technologies are a promising solution for regaining lost upper limb function.

One type of assistive technology are robotic arms, which are often fixed to an individual’s wheelchair to reach and grasp everyday objects [3-5]. Powered arm supports are similar, however these devices are attached to the limb and can be controlled to move the individual’s arm to aid in tasks of everyday living [6]. Another type of assistive technology, wearable robotic exoskeletons, have actuators aligned with the individual’s joints to aid in functional movement [7, 8]. Finally, neuroprostheses restore function to an individual’s limb by electrically stimulating the muscles through Functional Electrical Stimulation (FES) [9]. These assistive technologies have shown promise in restoring reaching function to individuals with SCI. However, a reliable method for sending motor commands from the user to the assistive device through a human machine interface (HMI) remains elusive.

Individuals with C4-level spinal cord injuries and above are paralyzed from the neck down, meaning the only available command signals that can be used as inputs to the HMI are located above the neck and shoulders. Previous studies have used surface electromyography (EMG) to trigger actions from the assistive device in response to neck muscle movements [10, 11]. Additional interfaces include sip-and-puff systems [12], tooth clicking [13], magnetic tongue switches [14], and speech recognition [15]. While effective for simpler tasks, these command signals generally lack the throughput and sensitivity necessary for complex motor tasks such as unconstrained reaching [16].

The richest source of command signals has its origin in the brain. Electroencephalography (EEG) involves placing electrodes on the surface of the scalp to record brain activity. Recording signals through the scalp and cranium leads to low spatial resolution and a low signal to noise ratio [17, 18], severely limiting the number of commands that can be decoded for controlling complex reaching motions. Conversely, electrocorticography (ECoG) maintains a higher spatial resolution and a higher signal to noise ratio due to the implantation of electrodes directly on the surface of the brain. Recent studies have used ECoG to record command signals for controlling reaching with an assistive device [19, 20]. While promising, ECoG involves highly invasive surgical procedures, requires extensive recovery times, and is accompanied by high clinical and post-clinical costs [21].

In contrast, eye movements are naturally available, directly observable, and highly correlated with upper-extremity motor function [22-24], and as such are candidate input sources for controlling reaching. However, eye movements contain rapid changes in acceleration, and humans do not maintain focus on one object of interest for the duration of a task [25, 26]. Therefore, it may be difficult for people to control the position of an assistive device directly. Some studies have supplemented eye-tracking with other sensors to reduce the cognitive burden on the user. Recent studies used computer vision and object recognition to aid in target selection [27-29], as well as finite state machines to assist in intention prediction [30]. Combining inputs in this way is one approach that has shown promise in controlling assistive devices. In the previously mentioned studies, eye tracking was used primarily to select targets that were then reached for or acquired by the assistive device through perception and planning. In the current work, we are interested in investigating using eye movements to directly inform effector movement as an alternative approach for controlling assistive devices.

Because eye movements contain rapid changes in acceleration and are not always directed smoothly toward a target, it is necessary to apply filters that extract important features which characterize the movement. These filters are based on the natural behavior of the eyes. In order to view an object with a high level of detail, the eye must be rotated such that the desired object can be viewed by the fovea, the area of the eye with the highest visual acuity. These rotations are executed through quick, ballistic movements called saccades. The periods of time between successive saccades are called fixations, which indicate where a person is directing their focus at a specific moment in time [31]. During natural functional reaching, the eyes generally fixate around the desired object, but they do not necessarily maintain constant focus on the target throughout the task [25, 26]. Positional errors in reaching are corrected through visual feedback from the eyes and proprioceptive feedback from the arm and hand [32]. For individuals with an SCI, proprioceptive feedback from the arm and hand is unavailable.

When controlling an assistive device with eye movements, the device serves as a replacement for the function of the arm and hand. In this case, the eyes must now provide the command input that determines the position of the end effector of the assistive device as well as observe errors in the end effector position. In this sense, the eyes serve as both an actuator to guide the effector as well as a sensor to observe positional errors. This leads to a situation in which the responsibilities of the eyes are entangled, meaning, the execution of the eyes’ sensory and motor roles affect each other. For example, in order for the eyes to observe the position of the device’s end effector, the eyes must rotate, which then moves the end effector further. If there is a large error between the end effector position and where the human is looking, a positive feedback loop can be generated, which causes the end effector to drift. The same phenomenon can occur when using eye movements to control a mouse cursor on a monitor [33]. The goal of this study was to determine if humans can use eye movements to control a cursor to target locations despite the entangled sensory and motor responsibilities that this method introduces. To our knowledge, the effect which the additional motor responsibility on the eyes and the resulting entangled sensory and motor functions have on humans’ ability to control an effector with eye movements remains unknown. However, there is some evidence to suggest that presenting visual feedback to the user improves performance and user experience [33, 34].

We undertook this study to assess the feasibility of using eye movements to directly control an effector. Specifically, we addressed the following research questions: First, how accurately can humans direct an effector to a desired location with their eye movements when the eyes’ sensory and motor responsibilities are entangled? Second, how does the eye fixation error compare to hand endpoint error when there is no eye-function entanglement? The results of this study will inform continued research on direct eye-movement control of assistive technologies.

Materials and methods

In this study, we aimed to determine if humans can direct an effector to desired target locations using eye movements. More specifically, we aimed to understand the effect of the eyes’ entangled sensory and motor responsibilities on eye fixation error during functional reaching. We collected eye and hand movement data while participants performed reaches during different experimental conditions and compared the eye fixation error between them. Conditions varied by whether the participant was instructed to reach for, or just look at targets displayed on a monitor, as well as the inclusion or omission of a cursor. The cursor provided feedback for the eye tracker’s estimation of the participant’s point-of-gaze.

Participant information

A total of 7 individuals (5 male, 2 female) participated in this study. They ranged between the ages of 23 and 30. None of the participants had any neurological or visual disorders, and all participants had normal or corrected-to-normal vision. Informed written consent was obtained for each participant according to the testing protocols approved by the Institutional Review Board at Cleveland State University (IRB-FY2019–93). Participants were informed of the experimental protocol and agreed to all testing in writing. Participants were also informed and provided written consent for their data to be de-identified and used in scientific publications and research. Prior to this study, none of the participants were familiar with eye-tracking systems nor were they expert users. Each testing session lasted approximately one hour, and participation was voluntary.

Experimental conditions

Each participant performed the experiment under four conditions, in a randomized order:

Eye-Alone

Eye-Hand

Eye-Alone with Cursor

Eye-Hand with Cursor

In all of the conditions, participants were instructed to select targets displayed on the monitor by moving an effector to each target. For this study, we refer to the effector as the element being moved to the target, and we refer to the actuator as the motor command that moves the effector. Errors in the effector position are observed by the sensor. For example, in the case of controlling an assistive-robotic arm with eye movements, the effector is the end effector (usually a gripper) of the robot, the actuator is the muscles of the eyes, and the sensor is the visual feedback received by the eyes. Of course, the motors of the robot are what physically move the robot’s end effector, but in this study, we use the term “actuator” to refer to the command input that determines the desired location of the effector. Likewise, the robot may contain internal sensors such as potentiometers and accelerometers, but in this study, we use the term “sensor” to refer to the mechanism by which the human observes positional errors in the effector. While these definitions may be oversimplified, defining terminology in this way should help to clarify the differences between experimental conditions (Fig 1).

Fig 1

Experimental conditions.

Summary of important characteristics for each experimental condition. Conditions were characterized by different combinations of goal, effector, actuator, and sensor. Each participant performed all experimental conditions in a randomized order.

In the ‘Eye-Alone’ condition, participants were instructed to look from the center of the starting target to the center of the task target (see Visual stimuli), and to maintain focus on the task target while it was displayed. No special instructions were given about how to move their eyes. In this condition, the position of the point-of-gaze was the effector and the muscles of the eyes were the actuator. When foveating a target, the change in the visual field as well as eye-muscle proprioception provide feedback to help direct the point-of-gaze to the desired location. Therefore, we simply label the eyes as the sensor in this condition. This condition was meant to provide insight into how humans foveate objects without a movement response from an assistive device.

The ‘Eye-Hand’ condition was similar to the ‘Eye-Alone’ condition, except that the hand component was introduced. Participants were instructed to move their index finger from the center of the starting target to the center of the task target and to maintain hand position on the target while it was displayed. No special instructions were given about how to move their eyes or hand. In this condition, the position of the finger was the effector and the muscles of the arm and shoulder were the actuator. Positional errors in the effector position were observed both through visual feedback by the eyes and through proprioceptive feedback by the muscles of the hand, arm, and shoulder. This condition was meant to provide insight into how humans look at and reach for objects without a movement response from an assistive device.

In the ‘Eye-Alone with Cursor’ condition, a cross-hair cursor was displayed at the eye tracker’s estimation for the participant’s point-of-gaze. In effect, the cursor provided visual feedback for where the participant was looking on the monitor and could be thought of as being similar to the control of the endpoint of an assistive device. Participants were instructed to move the cursor from the center of the starting target to the center of the task target, and to maintain cursor position on the task target while it was displayed. No special instructions were given about how to move their eyes or hand. In this condition, the cursor was the effector. The eyes served as both the actuator to move the effector as well as the sensor to observe errors in the effector position, thus the sensory function and concurrent motor responsibility of the eyes were entangled. In the context of an assistive system, this condition was meant to represent the case of direct gaze control of an assistive device.

The ‘Eye-Hand with Cursor’ condition was similar to the ‘Eye-Hand’ condition, except the cursor was displayed at the participant’s point-of-gaze for the duration of the condition. Participants were instructed to move both the cursor and their index finger from the center of the starting target to the center of the task target, and to maintain both cursor and hand position on the center of the target while it was displayed. No special instructions were given about how to move their eyes or hand. In this condition, the existing proprioceptive feedback from the arm and hand was not altered, but the eye’s source of feedback was augmented by the inclusion of the cursor. Here, the cursor and the hand were both effectors, with their corresponding actuators and sensors being the eyes and the muscles of the arm and hand, respectively. In the context of an assistive system, this condition might represent teleoperation, where the human reaches for objects remotely, and triggers a movement response from an assistive device.

Experimental setup and data acquisition

For the duration of the experiment, the participant was seated behind a table. On the table was a 55” Samsung monitor (60 Hz refresh rate), within arm’s reach of the participant (Fig 2). Targets were displayed on the monitor and the participant was instructed to look at and reach for the targets (see Visual stimuli). A 6” tall foam arm rest was placed on the table between the participant and the monitor. The participant was permitted to rest their arm on the arm rest between reaches to reduce the effects of fatigue during data collection. Participants were positioned in front of the monitor such that they could comfortably reach all of the target locations, and their forward gaze fell approximately on the center of the screen. All experiment logic was programmed in the MATLAB Simulink environment (Mathworks; Natick, MA) and executed on a microLab box real-time target computer (dSPACE; Paderborn, Germany). Eye movements were recorded with the ETL-600 head-mounted eye-tracking system (iSCAN; Woburn, MA) with a sampling frequency of 240 Hz. This system consisted of two infrared cameras that monitored the position of the pupils. It utilized a 6 dof magnetic head position sensor to allow free head movement. The point-of-gaze was computed through iSCAN’s 5-point calibration procedure and returned as x and y screen coordinates. When collecting eye-tracking data, the head may be constrained to prevent head motion. However, this study was designed to investigate whether eye-movements are a good candidate input signal for informing reaching motions in the context of gaze-controlled assistive devices. Under those circumstances, the head would not be constrained. Therefore we wanted to allow free head motion during data collection. Additionally, in this study, we were only interested in the participant’s point-of-gaze on the monitor in relation to the targets. Further, the eye-tracking system accounts for variations in the head orientation in the computation of the point-of-gaze. Finger position was recorded using the Optotrak 3D Investigator (Northern Digital; Waterloo, Canada), sampled at 1000 Hz and accurate to 0.4 mm. A marker was affixed to a stylus and attached to the participant’s index finger to monitor hand movement. The coordinate systems of the monitor and the Optotrak were aligned for ease of data analysis. Signals from the eye-tracker and the motion capture system were sent to the microLab box, which facilitated clock synchronization, experimental logic, and data logging. Raw experimental data can be found in the attached S1 File.

Fig 2

Experimental setup.

Each participant was seated at a table within reach of a monitor, which displayed task targets. Eye movements were recorded with a commercial head-mounted eye tracker. A motion capture marker was affixed to the participant’s finger to record hand movements. The participant was permitted to use the arm rest between reaches to reduce fatigue.

Visual stimuli

Each participant was instructed to perform a series of reaches, in which a starting target was displayed in the bottom center of the screen, and 6 task targets were displayed separately, in a randomized order (Fig 3). During initial tests of the experimental protocol, we found that if the starting target was displayed in the center of the screen, the participant’s arm would rapidly fatigue. Thus, we moved the starting target to the bottom of the screen and included an arm rest to reduce arm fatigue. All visual stimuli were displayed on the monitor using the OpenGL graphics framework. White targets on a black background were selected to reduce eye strain. Target stimuli were presented as a combination bulls-eye/cross-hair with a 2.5 cm diameter, which has been shown to attract participant attention and reduce gaze dispersion [35].

Fig 3

Visual stimuli target array.

Distances from the green starting target centers to the center of each white task target were as follows: 1) 24.2 cm, 2) 12.1 cm, 3) 12.1 cm, 4) 24.2 cm, 5) 13.0 cm, and 6) 26.0 cm. Here, the starting target is shown in green, however during the experiment, all targets were white. This figure depicts the locations that the targets would appear on the monitor, as well as an example position of the cross-hair cursor. During the experiment, only one target would be present on the screen at a time.

At the beginning of an experimental condition, only the starting target was displayed. After a delay, one of the task targets appeared and the participant looked at and/or reached for (depending on the experimental condition) the task target. After another delay, the task target disappeared and the participant returned their gaze and/or their hand back to the starting target. Delays were chosen randomly between 2 and 5.5 s to prevent participants from anticipating target movement. Looking and/or reaching from the starting target to the task target and back to the starting target was considered one trial. Participants performed approximately 8 trials per target location, depending on the randomization, for each of the four experimental conditions.

Analysis

To extract saccades (eye movements) and fixations (stable periods between saccades) from the raw eye data, we applied an offline fixation filter based on the methods by Olsson [36]. The first step was to interpolate any missing data due to blinks or dropped frames. When a participant blinked, the system returned zeros for the point-of-gaze coordinates and those values were flagged. The second step was to define a sliding window size within which large changes of the signal mean were detected. Typical fixations last between 150 and 600ms [31], rarely lasting less than 100ms [37]. The sliding window should be large enough to adequately detect changes in the signal mean, but not too long to span multiple fixations. Thus, we defined our sliding window as 80ms. The third step was to find the ‘peaks’ within the sliding window. For this step, we used the MATLAB function findchangepts(), which employs a parametric global method for determining changes in the signal mean. The fourth step was to remove peaks that occur too close together in the time domain. If more than one peak occurred within the sliding window, only the peak with the highest magnitude was recorded. The final step was to estimate the spatial positions of the fixations using the median and merge fixations that occurred within a specified radius. In this case, we used 1.1cm as the merge threshold. Once fixations were detected, we computed the eye fixation error as the Euclidean distance between the center of the task target and the extracted eye fixation position, and then computed the mean eye fixation error across all targets for each participant within a given condition. We then took the average across all participant means as the group-mean eye fixation error for that condition. The normality assumption was verified for within-participant as well as group eye fixation errors. For clarification, the computation of the eye fixation error was identical in the ‘Cursor’ and ‘No Cursor’ conditions. The only difference between conditions was that the cursor was displayed to provide augmented visual feedback to the participants.

In addition to detecting fixations from eye movement data, we also extracted the hand movement endpoints from each hand trajectory, which estimate the point in space where the movement is terminated. Specifically, the endpoint was identified as the point in the movement trajectory where movement velocity dropped below a threshold of 10 cm/s [38]. If there were multiple instances where the hand movement velocity crossed the threshold within a one second time frame, then the endpoint was based on an average of those positions. We computed the hand endpoint error as the Euclidean distance between the center of the task target and the hand movement endpoint, and found the mean hand endpoint error across all targets for each participant within a given condition. We then took the average across all participant means as the group-mean hand endpoint error for that condition. The normality assumption was verified for within-participant as well as group hand endpoint errors.

We compared the group-mean eye fixation errors between experimental conditions to understand how eye fixations were affected by the entangled sensory and motor responsibilities of the eyes depending on the condition. Statistical analyses were performed using a two-way, effector by cursor repeated-measures ANOVA with the following factors:

Effector: Eye-Alone/Eye-Hand

Feedback: No Cursor/Cursor

In the current study, we were interested in how the eye fixation error when the visual feedback was augmented by the inclusion of the cursor compared to the hand endpoint error when the eyes’ normal feedback modality was present. To do this, we compared the group-mean eye fixation error in the ‘Eye-Alone with Cursor’ condition to the group-mean hand endpoint error in the ‘Eye-Hand’ condition. This can be thought of as similar to comparing the control of an assistive device with eye movements to natural reaching, albeit with much less complexity. Comparing these two conditions can give us an indication of the effectiveness of replacing lost arm and hand function with a gaze-controlled assistive device. Statistical analyses were performed using a paired-samples equivalence test. Statistical analyses were performed using both SPSS statistical analysis software (IBM; Armonk, NY) and Minitab statistical software (Minitab; State College, PA).

Results

Fixation estimation and hand movement detection

Using an offline fixation filter, we computed the fixation positions from the raw eye movement data. Similarly, using a velocity threshold, we identified the hand movement endpoints. The typical temporal sequence of events consisted of the appearance of the task target, followed by the eye fixation, and finally the hand reaching the target location (Fig 4). This is consistent with observations made in other studies [38, 39].

Fig 4

Temporal sequence of events.

One trial is defined as movement from the starting target, to the task target, and back to the starting target. Just the movement in the horizontal direction from the starting target to the task target is shown here for one representative trial. A typical trial sequence consisted first of the task target appearance, followed by the eye fixation, followed by the hand endpoint.

Raw eye movement data contains high signal variability due to the rapid, saccadic nature of the eye movements combined with the fact that the eyes do not necessarily focus on the object of interest throughout the duration of reaching tasks [22–24, 31, 36]. The plot in Fig 5A illustrates the variable nature of the eye movement trajectories, yet fixations are accurately estimated. In contrast, the hand movement trajectories are smoother as the hand cannot accelerate as quickly as the eyes (Fig 5B). Another possible reason for this difference in signal variability is the difference in sampling rates between the eye-tracker and the motion capture system. However, we have shown that an offline filter can be used to accurately estimate fixation positions in eye movement data despite the inherent signal variability. Raw subject data can be found in S1 File.

Fig 5

Eye fixations and hand endpoints.

Representative participant data collection session for the ‘Eye-Hand with Cursor’ condition. Forty trials in the session are shown here. A) Raw eye movement data with eye fixations (blue X’s) for all trials. B) Raw hand data with marked hand positions at the end of the hand movement (red X’s) for all trials.

Eye fixation and hand endpoint error

As described in Analysis, we performed a two-way effector (‘Eye-Alone’, ‘Eye-Hand’) by cursor (‘Cursor’, ‘No Cursor’) repeated-measures ANOVA. The eye fixation and hand endpoint group-mean error values in each experimental condition are shown in Fig 6. The first result we report is the effect of the inclusion of the hand in the task on the eye fixation error. While descriptive statistics revealed that participants’ mean eye fixation error was slightly lower when using their eyes alone (1.71 ± 0.35 cm) compared to when movements were made with both their eyes and hand (1.84 ± 0.32 cm), the results of the ANOVA revealed that there was no significant main effect of the type of effector on eye fixation error (F(1,6) = 1.814, p >.05, ). In other words, whether or not the participant reached with their hand or just looked at targets did not affect their eye fixation error. For clarity, these values come from averaging the ‘Eye-Alone’ conditions (‘Eye-Alone’ and ‘Eye-Alone with Cursor’) and the ‘Eye-Hand’ conditions (‘Eye-Hand’ and Eye-Hand with Cursor), respectively.

Fig 6

Eye fixation and hand endpoint error.

Group-mean eye fixation error and hand endpoint error for each condition including standard error of the mean.

The second result we report is the effect of the inclusion of the cursor in the task on the eye fixation error. According to the ANOVA, there was a significant main effect of the cursor on participants’ eye fixation error (F(1,6) = 11.983, p < .05, ) such that participants’ mean eye fixation errors were lower when the cursor was displayed (1.06 ± 0.17 cm) compared to when the cursor was omitted (2.49 ± 0.53 cm). In other words, when the cursor was presented, the error between the effector position and the target was reduced compared to when the cursor was omitted. For clarity, these values come from averaging the ‘with cursor’ conditions (‘Eye-Alone with Cursor’ and ‘Eye-Hand with Cursor) and the ‘without cursor’ conditions (‘Eye-Alone’ and ‘Eye-Hand’), respectively.

The final result we report is the equivalence between the eye fixation error in the ‘Eye-Alone with Cursor’ condition and the hand endpoint error in the ‘Eye-Hand’ condition. We used a paired-samples equivalence test to compare the group-means from those conditions. We defined the equivalence interval for the difference in means to be ± 1cm. This equivalence interval corresponded with the diameter of the stylus attached to the finger. Because the trial instructions were simply to touch the target center with the tip of the stylus, we assumed the error of the reach could not be less than the diameter of the stylus, as reflected by the equivalence interval. While the mean hand endpoint error in the ‘Eye-Hand’ condition (0.84 ± 0.05 cm) was slightly lower than the mean eye fixation error in the ‘Eye-Alone with Cursor’ condition (1.04 ± 0.15 cm), the results of the equivalence test revealed the means were equivalent within ± 1cm (t(6) = 7.1, p < .05 and t(6) = -4.71, p < .05). In other words, participants were able to direct an eye-driven cursor to the target location with error like that of the hand during reaching with no cursor presented.

In summary, the overall quality of participants’ eye-movement performance was similar whether or not concurrent hand reaches were made, while the addition of an eye-driven cursor resulted in a lower mean effector error across all participants. Moreover, participants’ lowered eye fixation error in the ‘Eye-Alone with Cursor’ condition was comparable to the hand endpoint error in the ‘Eye-Hand’ condition (Fig 6).

Discussion

Previous studies have investigated how the functions of the eyes and hand interact to perform reaching tasks [22–26, 38–44]. In the current study, we aimed to understand if humans are able to perform similar reaching tasks when the normal sensory responsibility of the eyes is entangled with an atypical motor responsibility. We induced this situation by introducing an additional effector (cursor) that the participants were required to move to a target with their eye movements. We measured the performance of participants’ reaches using various offline methods as described in Analysis. The main finding of the current study is that not only are humans able to use eye movements to direct a cursor to a desired location, but they can do so with error similar to that of the hand during reaching without augmented visual feedback. The outcome of this study suggests that humans are able to control effector movement while simultaneously processing visual feedback for the effector position, and that gaze-control may be an effective replacement modality for lost arm and hand function. These results highlight the potential for eye-controlled motion of assistive technologies.

Fixation estimation

Through the use of an offline fixation filter, we computed the participants’ fixation positions from their eye movement trajectory data. Despite the observed high signal variability, we were able to estimate fixation positions (Fig 5A). In the ‘Eye-Alone’ condition, the participants’ existing feedback mechanisms were not augmented, and thus the mean eye fixation error from this condition (2.38 ± 0.22 cm) is a good indication of the fixation filter’s performance under normal conditions.

Directing the cursor with eye movements

As discussed in (Eye fixation and hand endpoint error), a repeated-measures ANOVA was used to elucidate the effects of both the use of the hand and the inclusion or omission of the cursor on participants’ ability to look at and/or reach for targets. The ANOVA revealed that when participants were instructed to select targets with their hand, there was no effect on their eye fixation error compared to when they were instructed to select targets with their eyes. This is an encouraging finding, particularly for individuals with a spinal cord injury, because it suggests that humans’ ability to foveate objects during a reaching task should not be hindered by the removal of the arm and hand. In addition, the ANOVA revealed that the inclusion of the cursor significantly decreased participants’ eye fixation error. This finding may suggest that providing point-of-gaze feedback improves the ability of humans to foveate objects. However, it is important to remember that the eye fixation error is a measure of the difference between the cursor and the desired target, where the cursor appears at the location that the eye-tracking system estimates the user to be focusing their gaze. This means that in some cases, the participant may be looking at the center of the desired target, but the point-of-gaze position contains slight errors, resulting in a shift in the cursor position. In the conditions where the cursor was not displayed, the participant had no knowledge of any potential discrepancy between where they were looking and the position of the cursor. Therefore, the decrease in eye fixation error from the ‘without cursor’ to the ‘with cursor’ conditions in Fig 6 can be understood as the participants’ ability to correct for errors in the position of the effector. This finding is encouraging because it suggests that if there are slight errors present in the position of the end effector of an assistive device comparable in magnitude to those experienced in this study, humans may be able to correct these errors using eye movements.

The results here seem to suggest that in an experimental environment, humans can accurately direct an effector to a desired location with their eye movements, despite both the presence of small errors in effector position as well as the additional actuation responsibility placed on the eyes. This, however, may be dependent upon the magnitude of the eye-tracking error. The research question regarding the comparison to hand performance without augmented visual feedback is answered through an analysis of the eye fixation error in the ‘Eye-Alone with Cursor’ condition and the hand endpoint error in the ‘Eye-Hand’ condition (Fig 6). The paired-samples equivalence test revealed that the means were equivalent within ± 1cm. This means that when the cursor feedback was presented, participants directed the cursor to the target location with error similar to their hand during reaching when the cursor was omitted. This finding is encouraging because it suggests that gaze-control may be an effective replacement modality for lost arm and hand function.

Limitations, implications and future directions

There are a number of limitations to the current study design that are worth discussing. The first limitation is the small sample size. We performed a power and sample size test and found that with the observed difference in means, we had an observed power of 75% and 83% for the ANOVA and equivalence tests, respectively. To achieve a power level of at least 90% in both measures, we would need a sample size of at least 11 participants. Another potential source of error were inconsistent number of reaches per target. On occasion, the participant would fail to notice the appearance of a target, particularly the outer targets, and would not look at or reach for the target. We believe this is due to the fact that participants relied on their peripheral vision to alert them when a target had appeared. This resulted in some targets being missed and an inconsistent number of data points per target. Finally, there may have been slight differences in the distance participants were from the monitor. This may have contributed to differences in eye fixation error. However, each participant performed all of the experimental conditions in the same position. Therefore, between-participant differences may be present, but within subject differences should be relatively small. Future studies may want to consider controlling for this potential source of error.

The goal of this study was not to solve all of the issues facing real-time, gaze-controlled assistive devices, but rather, to investigate the effects of augmented visual feedback from introducing an additional effector and the resulting entangled sensory and motor responsibilities placed on the eyes. It may be important, however, to discuss the potential challenges to controlling real assistive technologies with eye movements in light of the findings of this study. We have shown that humans are able to correct for errors in effector position with their eye movements. However, this study was conducted in a controlled, laboratory environment in a two-dimensional plane. In real-world, unconstrained reaching with an assistive device, the errors present in the end effector position may be higher in magnitude than those experienced in this study. The error will likely be compounded by additional sources of error such as the estimation of the depth component of the user’s point-of-gaze, as well as end-effector positional error present in the assistive device itself. Additionally, controlling the endpoint of an assistive robot arm will introduce noise and movement delays that are not present in the cursor displayed on the monitor in the present experiment. It is currently unclear if humans will be able to correct for these potentially greater sources of error, but the results of this study show the potential for the correction of positional errors with small magnitude.

Another common challenge associated with direct gaze-control of movement is the ‘Midas Touch’ problem [44]. This refers to a system’s inability to differentiate between eye movements intended to initiate movement and those intended simply for perception. When controlling a physical assistive device, the Midas Touch problem must be addressed. Common solutions are to introduce an additional modality such as a blink, a speech command, or a dwell time. In the current work, we focused only on the eye-movements directly responsible for effector control. The results of this study suggest that humans are able to direct an effector to a desired target location and can cope with the Midas Touch problem if their focus is directed on the task at hand, albeit for a limited amount of time.

In the current study, we have shown that humans are able to use their eye movements to direct an effector to a desired target location despite the additional motor responsibility placed on the eyes. Moreover, the current results were based on data collected during a single experimental session, which lasted about an hour. Thus, with minimal task experience and training, the addition of point-of-gaze positional feedback resulted in performance gains similar to the performance of the hand during reaching with unaltered visual feedback. Although many challenges still remain, the results of this study suggest that eye movements may be used to directly control an effector to desired locations, and validates continued research toward extending these principles to controlling physical assistive technologies.

Participant raw rata.

This zip file contains the raw gaze and hand endpoint positional data for each participant and each experimental condition.

(ZIP)

Click here for additional data file.

6 Oct 2021

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Markus Lappe

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please amend your Methods section to state that participants provided informed consent, and please state the type of consent (i.e., written, verbal, etc.).

3. Thank you for stating the following financial disclosure:

“Schearer, E.M. (PI) received the grant award entitled “Controlling Functional Reaching with Eye and Head Movements of People with High Cervical Spinal Cord Injuries.” from the Ohio Department of Higher Education. The award effective dates are 6/1/2020 – 5/31/2022. Grant numbers are not assigned by the Ohio Department of Higher Education. More information can be found here:

https://icorpsohio.org/apply/

https://www.ohiohighered.org/grants-rfps”

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors report an experiment on the usability of eye movements to control the motion of an effector, i.e. a tool that is moved toward a target object.

The background is that people with movement impairments, for example as a result of a spinal cord injury, lose movement control over their limbs. Restoring hand and arm function is essential for people to be able to perform everyday tasks and regain independence. This can be achieved, for example, with the help of robotic arm prostheses. One of the ways to control these aids can be eye movements, as they are closely connected to our actions. For example, people usually direct their gaze to the objects they want to grasp next.

In this study, the authors investigated whether additional motor demands on the eyes, which normally serve to assess the accuracy of e.g. hand movements, affect eye-hand coordination. The authors investigated whether, when controlling an effector by means of eye movements, the combination of the usual sensory and the additional motor demands leads to a decrease in fixation accuracy. The main question of the study was whether, when controlling a cursor with the eyes, i.e. combining motor and sensory demands on the eye, fixation accuracy is lower than the accuracy with which people reach for objects by hand under natural conditions.

Answering this question is relevant when evaluating if eye movements are a suitable means to control assistive devices. The present study is thus timely and important. The finding that the participants in this study were capable of directing an effector to a target object via eye movements as accurately as they reached for a target object with their hand highlights that the visual system can handle the combined motor and sensory demands. This study is thus an encouraging starting point for further research on the role of eye movements in the control of prostheses and other assistive devices. Although in my opinion the experimental design is appropriate for the research question and the conclusions drawn are sound, I nevertheless have come critical comments, especially on the statistical analysis.

Statistical Analysis:

First, I noticed that comparisons between conditions and factor levels are initially made on the basis of descriptive values. It is not clear to me why no statistical test is carried out before conclusions are drawn about possible differences between conditions or different factor levels (p. 7, ll. 260-278), especially since not all of those assumptions are subsequently validated by the ANOVA (e.g. p. 7, ll. 260-264; pp. 7-8, ll. 275-279). Moreover, later on, when the results of the ANOVA are reported, the descriptive values described in the beginning of the results section are repeated (e.g. p.7, l. 262 and p. 8, l. 286). This seems redundant.

Second, I stumbled over the phrase “to verify these interactions” (p. 7, ll. 274) for two reasons. First, the previously reported descriptive values were not indicative of an interaction. Second, no interaction is verified in the following. Instead, only the main effects of the ANOVA are reported while the interaction of cursor and effector is not mentioned at all.

Third, the main result of the study, namely that “participants were able to direct an eye-driven cursor to the target location with accuracy like that of the hand during reaching with no cursor presented” is based on a non-significant t-test. It is important to distinguish between a non-significant finding, implying that there is no substantial difference in hand-reaching and fixation error, and actual evidence of equal error magnitude. Especially given the small sample size of this study, I would be cautious about interpreting a non-significant result. Since the finding that there is no significant difference in accuracy is the central finding of the study, I would suggest performing an additional Bayesian t-test on the data that can actually provide evidence for H0 and thus equal accuracy/error magnitude. If it proves difficult to determine a prior, the test could still be run over a range of prior widths. In addition, I suggest the authors acknowledge the small sample size in the section “Implications and future directions” where they discuss the limitations of their current work.

Fourth, the authors have specified the degrees of freedom of the t-test as 7 (p. 8, l. 295). Since the sample consists of 7 participants, there seems to be an error here.

Fifth, during reading I noticed that it was easy to get confused with the terms fixation accuracy, fixation error, hand endpoint error, hand movement endpoint error, hand error and hand-reaching error. On p. 8 ll. 292-293 the authors wrote that they compared the hand endpoint error with the mean eye fixation accuracy while they probably meant that they compared the fixation error with the hand endpoint error. I would recommend using a more consistent terminology. This would further improve the readability and clarity.

Figures:

Figure caption 1: Even though it is already mentioned in the text, perhaps specify that the different conditions are characterized by a different combination of goal, effector, actuator and sensor.

Figure caption 3: perhaps point out in the caption that not all white targets in the Figure 3 were present on the screen at the same time. Otherwise this figure can easily be misinterpreted at first glance.

Figure caption 4: in the caption it says “representative participant trial for the Eye-Hand with Cursor condition” while all trials (=reaches) are depicted in the figure. That does not make sense to me.

Figure 4: it seems that there was an unequal number of fixations per each fixation target? Is that a result of randomization?

Reviewer #2: In the present study the authors evaluated the possibility of controlling an effector using eye movements. In particular, authors compared the accuracy of fixating a target in the presence and absence of an augmenting point-of-fixation feedback (cursor). Furthermore, the accuracy of the effector (cursor) end position was compared with that when a hand reach is initiated.

Overall, the manuscript is written in a rather clear manner in understandable language. However, there are several methodological issues that are in question and need to be improved from my point of view. In the following, I will comment on each of the sections separately.

Introduction

The motivation is described rather well. However, the section would benefit from a clearer description of how the effector is defined. Perhaps, authors can add a clarifying schematic or a diagram where the effector, actuator and sensor are indicated. In particular, how the 3 terms are defined in the context of an assistive system.

Furthermore, in the Discussion when comparing the results in the conditions without cursor and with cursor, authors interpret the better accuracy in case the cursor is present as a potential advantage of the gaze-point visual feedback. While I think it is a valid interpretation, I recommend dedicating few sentences with relevant references into the Introduction where the role of the gaze-point visual feedback is addressed.

In lines 52-63, authors mention “filtered unwanted eye movements…”. I recommend improving the phrasing here. From my understanding, in a reaching task there are different eye movements involved: fixations and saccades. In the present study you focus on fixations and the typical areas of interest fixated in a reaching task. Generally, saccades are also an important part of performing a reaching task, however, here what you do is applying a fixation detection algorithm.

Line 7: I believe the term “extremities” is constrained to hands and feet. As authors also include arm injuries, a broader term “limbs” would be more appropriate.

Line 99: ranged from -> ranged between

Materials and Methods

Participants information

The sample size in this study, 7 participants, is rather small. Was there an effect size analysis done before the study? While I understand that in the current global health situation it can be hard to recruit participants, I find it important to recognize this issue in the manuscript. Furthermore, please indicate whether participants were naïve or trained eye tracker users – from my experience it significantly affects the eye movement data.

Although I had to re-read the section few times, I believe the structure is logical. Also, if authors add a small diagram with effector, actuator, and sensor definition mentioned before, I think it would be easier to follow. I recommend, though, to align the order of the condition description with Fig.1: Either start the conditions description from “Eye-Alone” condition, or shuffle the conditions blocks in the figure such that it starts in the left upper corner with “Eye-Hand” condition. You can also add the listing numbers of the conditions to the figure to help the reader to follow.

From my perspective, it would also be helpful to add a sentence of reasoning for each condition. Specifically, compliment the conditions description with interpretation of each condition in the context of an assistive system. I could deduce the idea of comparing the cursor with the robotic arm only from the discussion, but not earlier.

Experimental setup and data acquisition

Generally, the main measured parameter in this study, the eye accuracy, is affected by the eye tracker accuracy. One major issue that I see in this study implementation is that there is no information about the distance between the eyes and the screen. Was it fixed? When it comes to accuracy measure in the context of eye trackers, it is typically given in degrees of visual angle (deg va), not meters. In other words, a fixed accuracy in deg va will result in a varied accuracy in meters depending of the distance from the screen. Did authors control for this during the experiment? What was the variability of the eye-screen distance during the experiment. I believe this is an important aspect for this study approach and it should be mentioned in the manuscript. In general, the manuscript would benefit from information regarding the variability of head rotation. Such, when it comes to fixation/saccade detection in case of free head movement, one common challenge is vestibulo-ocular reflex (eyes compensate the head movement in order to stabilize the image). Was it an issue in your data?

Also, was the height of the chair, table or screen adjusted for each participant such the forward gaze point would fall in the center of the screen? Please indicate this in the methods.

Analysis

If I understood correctly from the discussion, in the conditions where the cursor was present, the accuracy is defined as the error between the center of the target and the tip of the cursor. Is that right? Please indicate explicitly how you define accuracy for the cursor-present conditions as it comes in question whether you compare the same metric across conditions when comparing eye-only and the eye-cursor conditions.

Lines 200-201: The duration of the saccades is very much dependent on the task at hand and depending on the distance to the target quite often is longer (around 250 ms). But more importantly, I missed some more details on the fixation detection algorithm. I understand, the authors used velocity threshold algorithm for fixation identification. Which velocity threshold you applied? Which minimum duration is for it to be a fixation? Did you merge fixations that are very close in time into one larger fixation? How the fixation position is defined – is it a centroid of all raw gaze positions belonging to the fixation? Full understanding of the underlying parameters is important as it directly affects the results of the study, therefore, I recommend adding a more detailed description to the manuscript. Perhaps, this reference can be useful: (Salvucci & Goldberg, 2000)

Was the eye movement and hand movement data recording controlled from one device? Or did you have to additionally synchronize the clocks of both?

Results

Line 253-254: another reason is also a much higher sampling rate of the hand tracking device compared to the eye tracker

The figure 4A is very overloaded and therefore not very informative. Perhaps, authors could consider displaying only a small part of one trial. It is often helpful to plot the gaze data as a scatter plot, not a line plot. To indicate the order of the gaze positions a color coding can be used where a color bar would indicate time.

Line 262: I believe instead of “Hand-Alone” you meant “Eye-Hand”

Discussion

I rather enjoyed reading this section, it is easy to follow and I think the interpretation of the results is supported by the data. Authors nicely indicated the limitations of the study. The section can be complimented with other issues mentioned above (e.g. sample size).

Salvucci, D. D., & Goldberg, J. H. (2000). Identifying fixations and saccades in eye-tracking protocols. Proceedings of the Eye Tracking Research and Applications Symposium 2000, 71–78. https://doi.org/10.1145/355017.355028

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

18 Nov 2021

All of these responses are included in the attached file "Response to Reviewers". These responses below summarize the main points.

Journal Requirements:

1) We reviewed the PLOS ONE style requirements and made a few changes, namely author affiliations and file naming.

2) We added a sentence in the Methods section that states that participants provided written informed consent. (p. 3, ll. 106-107)

3) We added the suggested sentence to the Cover Letter.

4) We added an in-text citation referring to the Supporting information in the Results section (p. 8, ll. 323). A caption for the Supporting Information was included in the previously submitted manuscript.

Reviewer 1:

1) Our approach here was to initially assess the data and provide a general description of the pattern of the results, and then follow up with the statistical results. However, we believe the reviewer is correct in pointing out the redundancies this approach introduces. We re-worked the section in question to eliminate the redundant report of the descriptive values and to suggest conclusions only after the statistical analysis had been reported (p. 8, ll. 324 – p. 10, ll. 382).

2) The reviewer makes a good point in that we are investigating the main effects here, not interaction terms. We addressed this comment by re-working the Results section as described in the previous comment response (p. 8, ll. 324 – p. 10, ll. 382).

3) We agree with the reviewer here that a non-significant t-test result does not allow us to accept the null hypothesis. Because we are trying to determine equivalence, rather than the difference between the samples, we should instead use an equivalence test with paired data.

Here, we are comparing the mean eye fixation error in the ‘Eye-Alone with Cursor’ condition to the mean hand endpoint error in the ‘Eye-Hand’ condition. We want to see if the mean eye fixation error is equivalent to the mean hand endpoint error within an equivalence interval. If the difference between the mean eye fixation error and the mean hand endpoint error is within ±1cm of the mean hand endpoint error, we consider this to be equivalent. The equivalence test showed that the confidence interval was within the equivalence interval (p < .05), therefore we can claim that the means are equivalent. We have described all of this and updated the manuscript accordingly (p. 8, ll. 303-305 and p. 9, ll. 364-376).

4) Thanks to the reviewer for pointing out the typo. This has been corrected in the manuscript in the reworked section as part of the previous response (p. 9, ll. 364-376).

5) The reviewer makes a great point here about consistent terminology. Throughout the paper, we eliminate all references to “accuracy.” We instead only use the terms “Eye fixation error” and “Hand endpoint error”. Eye fixation error refers to the Euclidean distance between the center of the target and the participant’s point of gaze. Hand endpoint error refers to the Euclidean distance between the center of the target and the position of the motion capture marker at the end of the reaching movement. Additionally, this comment got us thinking about other terminology inconsistencies. We now only refer to the target which begins a reach as the “starting target”. The target that then appears on the screen that the participant looks at or reaches for is referred to as the “task target”. The starting target always appears in the same location (bottom-center of the screen). The task target appears randomly in one of the 6 determined positions. These terms are updated throughout the manuscript.

6) This is a good suggestion. We updated the figure caption in the manuscript.

7) The reviewer makes a good point here. We updated the figure caption to reflect the suggested change.

8) This is a good catch. In the manuscript, we defined one reach (movement from starting target, to task target, back to starting target) as one trial. We should have said “session” instead of “trial” in the figure caption. This has been updated in the manuscript. We also added a figure showing the initial movement in the horizontal direction of a typical trial. It highlights both the definition of a trial and the typical temporal sequence of events.

9) For this particular participant, it is true that there are fewer fixations in some of the targets. There are a few reasons for this. First, the starting target was displayed in every trial. Therefore, effectively there should be 6 times as many X’s within the starting target as there are in the task targets. Second, on occasion, the participant did not notice the appearance of a task target and did not make a movement toward it. We think this is due to the fact that participants relied on their peripheral vision to detect the appearance of the task target. Therefore, it makes sense that the outer targets, or targets more in the participants’ periphery, were the ones missed more often. Also, because the timing and locations of the targets were randomized, it was impossible for the participants to predict where and when the next target would appear. This also probably contributed to a few missed targets. These missed trials were not included in the results. We added details regarding these points to the manuscript (p. 11, ll. 450-455). We also changed the figure (now Figure 5) so that the starting target was a different color than the task targets. This should help to make the distinction between targets clearer.

Reviewer 2:

1) The reviewer makes a good suggestion here as to how to clarify the terminology. We modified the section in the Introduction to better describe the goal of the study (p. 3, ll. 79-80). We also added an example of how the 3 terms are defined in the context on an assistive system, as per the reviewer’s suggestions. We added this example in the Experimental Conditions sub-section of the Materials and Methods section (p. 4, ll. 123-131).

2) This is a good suggestion. We added some information into the Introduction in response to this comment (p. 3, ll. 84-85).

3) We agree with the reviewer that the phrasing here is somewhat clunky. It is true that eye movements are characterized by both saccades and fixations, however, for this study, we were only interested in fixations, and not the eye movements leading up to them. The passage in question has been rephrased in the manuscript (p. 2, ll. 52-56).

4) This is a good catch. We have updated this in the manuscript (p. 1, ll. 7).

5) Updated (p. 3, ll. 104-105).

6) We agree that the sample size might be smaller than desired. We performed a power and sample size test. For the ANOVA comparison, to detect a difference of 1.3cm (the average difference between pairs of conditions), with a power of at least 90%, we would need 11 participants. Our current observed power for that measure was 75%. Similarly, for the equivalence test (see response to Reviewer 1, Comment 3), to detect a difference of 0.2cm (on average what we observed), with a power of at least 90%, we would need 8 participants. Our current observed power for that measure was 83%. So, while our power for each measure of this study was not bad, more subjects would have been better. This limitation has been included in the manuscript (p. 11, ll. 446-450).

We did not explicitly ask if the participants were familiar with other eye-tracking devices, however, none of the participants had used this system before and were all given the same instructions on how to use it. This has been added in the manuscript (p. 3, ll. 108-110).

7) We believe the reviewer makes a good suggestion here regarding the organization of this section. We ultimately decided to go with describing the experimental conditions in the following order: 1) Eye-Alone, 2) Eye-Hand, 3) Eye-Alone with Cursor, 4) Eye-Hand with Cursor (p. 4, ll. 115-116). To the figure (Fig 1), we added condition numbers which correspond to the order in which the conditions are listed in this section.

8) This is a good suggestion. We added a sentence or two to each condition providing some perspective on how each condition might be observed in the context of an assistive system (p. 4, ll. 151-153; p. 5, ll. 162-164; p. 5, ll. 175-176; p. 5, ll. 186-189).

9) The reviewer makes some good points regarding the factors which can lead to inaccuracies in eye-tracking. We do our best here to explain our thought process when designing this experiment with the primary goal being to understand if eye-movements are a good candidate input signal for controlling reaching motions with an assistive device.

We had each participant sit in a chair facing the monitor. We positioned the chair a distance away from the monitor such that the participant could reach all of the target locations comfortably. The chair was not moved for the entirety of the testing session (all 4 experimental conditions). During data collection, we did not stabilize the head position. For this study, we were interested in the participant’s point-of-gaze on the screen. Our eye-tracker was a head-mounted system which contains a magnetic sensor for accounting for head orientation in point-of-gaze computation. In this case, we wanted the setup to resemble how an individual might control an assistive device with such a system, with the only constraint being the targets displayed in one plane (the monitor). Therefore, each participant should be able to maintain a steady point-of-gaze despite slight differences in head rotation. However, this is a potential source of error and has been added to the manuscript.

With the setup we used, we believe that the distance between the participant and the monitor was fairly constant, within each data collection session. It is true that there may exist between-subject differences in this distance, but this was not measured in the present study. All of the above points have been addressed and added to the manuscript (p. 6, ll. 207-215 and p. 11, ll. 455-460).

10) The chair and table height were such that the participant’s forward gaze point fell approximately in line with the center of the screen, however this was not strictly controlled for. This element of the experimental setup was the same for each participant. This was clarified in the manuscript (p. 5, ll. 197-199).

11) Yes, this is correct. The only difference between the ‘Cursor’ and ‘No Cursor’ conditions is that the cursor is displayed so that the participant can see it. Thus, the difference is in providing visual feedback to the user. The computation of the fixation error is the same, regardless of whether the cursor is displayed. This has been made more explicit in the manuscript (p. 7, ll. 271-274).

Additional details about the cursor could have been included. We did not include a figure showing what specific form of visual feedback was presented to the user. This was an oversight on our part and has been addressed by adding a depiction of the cursor in the Target Array figure (Fig 3) and accompanying updated figure caption.

12) The fixation filter methods were based primarily on the methods outlined in the cited paper (Olsson, 2007). The steps were as follows:

1. Interpolate any missing data due to blinks or dropped frames

2. Compute the difference in the means between sliding windows. The typical length of fixations is 150ms – 600ms (Duchowski, 2017) (and rarely fewer than 100ms, according to the reviewer’s suggested reference (Salvucci & Goldberg, 2000)). Using these values and visually inspecting the data, we set our window size to 80ms. The window should be defined such that it is long enough to detect adequate changes in the signal, but not too long that it spans multiple fixations.

3. Find the “peaks” within the difference vector. For that step, we used the MATLAB command findChangePoints() instead of the method described by Olsson.

4. Remove peaks that are too close together in the time domain. Because multiple saccades can occur within the specified averaging window, we only kept the peak with the highest magnitude within the averaging window.

5. The last step was to compute the spatial distance between fixations using the median and merge fixations which occur within a specified spatial radius (1.1cm in this case).

These steps are outlined in detail in the cited reference (Olsson, 2007), however we added more detail to the manuscript regarding the modifications we made and parameter values we used. We believe the reviewer makes a good point here that while we cite this paper in the manuscript, more details should be added in the text. This has been addressed in the manuscript (p. 7, ll. 245-265).

Olsson P. Real-time and offline filters for eye tracking; 2007.

Duchowski AT. Eye tracking methodology: Theory and practice; 2017.

13) The eye data was recorded using the iSCAN etl-600 device and the hand data was recorded using the Optotrack motion capture system. These two signals were sent to a dSPACE real-time target computer which facilitated clock synchronization, experimental logic, and data logging. This was made clearer in the manuscript (p. 6, ll. 219-221).

14) This is a good point. We have added this to the manuscript (p. 8, ll. 319-321).

15) The primary purpose of this figure (now Fig 5) was to show how variable human eye movements can be compared to their hand movements, as well as visually demonstrate the performance of the fixation filter. However, we agree with the reviewer here that the figure is cluttered. We changed the figure in question to show the data from a different participant whose raw eye data was not as variable. We also did not plot as many trials to reduce the visual clutter a bit. In addition, we changed the color of the starting target to make the distinction between targets clearer.

To better show the temporal sequence of events, we added another figure (now Fig 4) showing the target, eye, and hand data in the horizontal direction for the first half of one representative trial. The purpose of this figure is to make clearer the definition of a trial (as per a comment from reviewer #1), as well as to highlight the temporal sequence of events. We believe the modification of the figure in question (Fig 5), and the addition of the one-dimensional single trial figure (Fig 4) will help to address both reviewers’ concerns and clarify the content for the audience.

16) This is a good catch. The type has been corrected in the manuscript.

17) We appreciate the kind words. We have taken to heart the comments and suggestions above and have re-worked the Discussion section to better reflect the changes we made in response to your previous suggestions and those of Reviewer 1.

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

2 Dec 2021

22 Dec 2021

1. Most of my comments have been adequately addressed by the authors.

Regarding the equivalence test replacing the t-test, I miss the justification for the equivalence limits chosen. 1 cm might be a reasonable choice, but especially if the equivalence bounds are not set before the initial data analysis, the rationale behind the choice should be briefly explained in the manuscript.

This is a good suggestion. We added a brief explanation regarding the choice of 1cm for the equivalence interval. To measure the hand motion, we placed a motion capture marker on a stylus, which was attached to the participant’s finger. During experimental conditions including hand motion, the participants were instructed to touch the tip of the stylus to the center of the target. The tip of the stylus had a diameter of 1cm. Therefore, if the eye fixation error was within 1cm of the hand error, we considered this to be equivalent, as reflected by the 1cm equivalence interval. We addressed this response in the manuscript (p. 9, ll. 332 - 335).

2. P 11, ll. 436-437: typo: “despite” is repeated

Good catch. This has been fixed (p. 10, ll. 396).

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

20 Jan 2022

Controlling an Effector with Eye Movements: The Effect of Entangled Sensory and Motor Responsibilities

PONE-D-21-26505R2

Dear Dr. Schultz,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Markus Lappe

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

25 Jan 2022

PONE-D-21-26505R2

Controlling an effector with eye movements: The effect of entangled sensory and motor responsibilities

Dear Dr. Schultz:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Markus Lappe

Academic Editor

PLOS ONE