Seeing the Forrest through the trees: Oculomotor metrics are linked to heart rate.

Fluctuations in a person's arousal accompany mental states such as drowsiness, mental effort, or motivation, and have a profound effect on task performance. Here, we investigated the link between two central instances affected by arousal levels, heart rate and eye movements. In contrast to heart rate, eye movements can be inferred remotely and unobtrusively, and there is evidence that oculomotor metrics (i.e., fixations and saccades) are indicators for aspects of arousal going hand in hand with changes in mental effort, motivation, or task type. Gaze data and heart rate of 14 participants during film viewing were used in Random Forest models, the results of which show that blink rate and duration, and the movement aspect of oculomotor metrics (i.e., velocities and amplitudes) link to heart rate-more so than the amount or duration of fixations and saccades. We discuss that eye movements are not only linked to heart rate, but they may both be similarly influenced by the common underlying arousal system. These findings provide new pathways for the remote measurement of arousal, and its link to psychophysiological features.

Introduction

Remotely and unobtrusively detecting fluctuations in arousal is of wide interest to researchers in fields such as human-computer interaction, psychology, and ergonomics. This interest is due to the fact that changes in arousal are not only related to physical exertion, but also to psychological concepts for which arousal is often assessed as an objective approximation, such as the degree of excitedness, drowsiness, or mental effort during a given task. Given that arousal levels are related to task performance following an inverted U-shape function [1, 2], they have a profound effect on task performance–for instance on various critical tasks, arousal may affect the safety of operators and other people who rely on those operators [3]. Although fluctuations in arousal can be detected from various objective sources such as electroencephalography (EEG), functional Magnetic Resonance Imaging (fMRI), heart rate, or skin conductance [4], these methods require direct physical interaction with measurement devices or can be quite obtrusive. Only few parameters can be assessed remotely, such as oculomotor metrics obtained via video-based eye-tracking.

In the current study we investigate how well heart rate–one of the best investigated central indicators of arousal–can be predicted from remotely accessible oculomotor metrics as alternative peripheral indicators of arousal. A link between these indicators is plausible given the extensive support for correlations between oculomotor metrics and various psychological concepts, such as mental effort. For instance, it has been shown that the degree of pupil dilation can provide an accurate indication of participants’ mental effort in both controlled and naturalistic viewing tasks [5, 6]. Furthermore, it has been shown that the peak velocity of saccades decreases as mental effort increases [7, 8] and increases as motivation increases [9]. Similarly, mental effort has been shown to covary with heart rate and with several derivatives of heart rate measures [10]. Additionally, it has been shown that changes in arousal are paired with an altered rate of eyeblinks [11, 12], and that spontaneous eyeblinks occur in tandem with an increase in heart rate variability [13].

While oculomotor measures are fairly robust, they can be influenced by the environmental circumstances under which they were obtained. For instance, pupil dilation is impacted by the luminance of the scene that is being watched, and microsaccades and peak velocities of saccades can only be reliably measured by expensive high-speed, low-noise trackers. Additionally, among eye tracking scientists there is no unified concept of how fixations and saccades should be defined–and thus the application of differing fixation- and saccade detection techniques may result in differing outcomes, even if they are applied to the same dataset [14]. As such, incorporating several metrics which can be independently extracted (e.g., pupil size, oculomotor movement, blinks) would improve robustness of the model, as it reduces dependence on one single extraction technique. This also applies to cases in which pupil dilation measurements are unreliable or missing, or the eye tracker’s sampling rate is too low to extract peak saccade velocities.

The benefits of relating oculomotor metrics to heart rate are two-fold. Firstly, scrutinizing the links between oculomotor metrics and heart rate can foster our theoretical understanding of common underlying mechanisms and thereby our definition of arousal. Secondly, ever since the seminal works of Buswell [15] and Yarbus [16], we have been aware that eye movements are foremost driven by task type (top-down) and visual saliency (bottom-up). Later on, it has been shown more reliably that task type–such as search or free-viewing–influences gaze behaviour [17-19], and that oculomotor metrics besides pupil dilation [20] and peak saccade velocity (e.g., saccade amplitude, fixation duration) can provide sufficient information for machine learning algorithms to predict task type at above chance level [21, 22]. In this manuscript, we describe that, besides top-down and bottom-up mechanisms, arousal–estimated by the link to heart rate–also contributes to eye movements.

To this end, we use data from the studyforrest dataset [23]. This dataset contains eye tracking and pulse oximetry measurements from participants while they watched the 1994 motion picture Forrest Gump. We investigate whether oculomotor metrics can provide sufficient information for regression models and machine learning models to accurately predict high or low heart rates of participants in this naturalistic viewing task. Furthermore, we investigate how strongly each oculomotor feature contributes to the correct prediction of heart rate, thereby providing insight into how specific aspects of oculomotor movement is driven by a common measurement of arousal, such as heart rate.

Methods

All analyses were performed with Python 3.8.10, using SciPy version 1.6.2 and scikit-learn version 0.24.2 [24, 25]. All code and outcomes can be retrieved from https://osf.io/skcd8/.

Raw data

Eye tracking data and pulse oximetry data were obtained from the studyforrest dataset, which contains data of fourteen participants that were measured while being presented with the 2-hour film Forrest Gump [23, 26]. The raw eye tracking data was measured with an Eyelink 1000 at a frequency of 1 kHz and pulse oximetry measurement was applied to record heart rate data at an effective frequency of 100 Hz. A full description of the recordings and anomalies can be found in [23] and at https://studyforrest.org.

Oculomotor feature detection

Fixations and saccades were extracted based on the algorithm proposed in [27], which operationalizes fixations and saccades as phases of slow and fast eye movements, respectively. Firstly, the raw 1 kHz x and y gaze signals were smoothed by applying a Savitzky-Golay filter. We then applied an adaptive velocity threshold algorithm to this smoothed signal, thereby obtaining candidate fixation phases, with everything in between being candidate saccade phases. Thereafter, we applied two basic merging criteria. Firstly, saccade candidates with amplitude < 1.0° were removed, thereby merging neighbouring fixation candidates. Subsequently, all fixation candidates with duration < 60ms were removed. This procedure successfully removes large differences in oculomotor event classification between different algorithms [28]. Gaze amplitudes and velocities in pixels were converted to degrees of visual angle by multiplying their values by 0.0186 [23]. Lastly, blinks were detected by finding periods in which no pupil data was measured. All events which lasted less than 30ms, or more than 3 seconds, were removed. These thresholds were set so that neither brief nor longer periods of data loss would be incorrectly detected as blinks.

Data pre-processing

After extraction of oculomotor metrics, data of each participant was split into 240 chunks of 30 seconds each. However, the last chunk was often shorter than 30 seconds, and some chunks had too much data loss. As such, these chunks were discarded, resulting in 3327 data points. Then, heart rate detection was performed over the raw pulse oximetry signal within these chunks, using HeartPy [29]. Thirty seconds were selected as chunk size because it provides a balance between sufficient data per chunk (> 20 fixations on average, and sufficient time for accurate heart rate detection), and a sufficient number of chunks for machine learning purposes.

For each chunk, twelve features were extracted: (1, 2, 3) the duration of each fixation, saccade, and blink event; (4, 5) the amplitude of each fixation and saccade event; (6, 7) the peak velocity of each fixation and saccade event; (8, 9) the mean velocity of each fixation and saccade event; and (10, 11, 12) the count of fixation, saccade, and blink events in that chunk.

We took a two-fold approach to testing whether oculomotor metrics can be sufficient predictors of heart rate. Firstly, we posed that the prediction of heart rate could be considered a regression problem, in which we aimed to predict heart rate on a continuous scale. Secondly, we posed that the prediction of heart rate could also be considered a binary classification problem (above some threshold or below some threshold). This approach can be useful when the aim is to only predict whether someone is either excessively or insufficiently aroused.

To prepare our dependent variable for binary classification, the heart rate of each chunk was expressed as a z-score; the number of standard deviations from the median heart rate of that respective participant over the full film. Each z-score was then converted to a binary variable–namely low if z < -.5, and high if z > .5. All other chunks were considered neutral and discarded. Since the distributions of heart rate were often skewed, and due to slightly differing amounts of data loss, our binarization did not result in equally large samples of high and low labels. As a result, 513 chunks were below the threshold, and 607 chunks were above the threshold. A total of 1120 data points remained after binarizing the heart rate data. Distributions of each feature, split per label, are reported in Fig 1.

Fig 1

Distributions (kernel density estimation) for each of the twelve features, per label (high or low heart rate).

The distributions are computed over all chunks for all participants, thus 1120 data points per feature (513 low, 607 high). Orange and blue values indicate median and standard deviation of each of the high and low heart rate distributions, respectively.

Feature pre-processing

As is common in machine learning pipelines, our classifier required an equally long set of features per chunk of data, and the described feature set did not comply with this requirement. For example, if 30 saccades were made within one chunk, and 40 saccades were made in another chunk, the peak saccade velocity variable would contain 30 and 40 values for each of those chunks, respectively. Therefore, our data needed to be aggregated. Three methods were explored, as outlined in the next subsections.

Averaging

Within each chunk, the average of each of the twelve features was computed, providing one value per feature for each chunk. This approach provides the most intuitive insight into the amount of information contained within each feature, which in turn contributes towards correct classification.

Feature explosion

It could be argued that simply calculating the mean value over features would discard relevant information, since, for instance, the mean saccade velocity across chunks may be equal, but the variance across chunks could be different. Similar to the approach of Kootstra et al. (2020), a set of 13 statistical descriptors (e.g., mean, variance, uniformity) was employed to describe the distribution of each of the features 1–9 within each chunk (see S1 Table for a full list of the statistical descriptors). Through this method, the dataset was thus ‘exploded’ and contained 3 count features + (9 features × 13 descriptors) = 119 features.

Feature explosion and dimensionality reduction

To aid interpretation of these 119 features, each of the oculomotor metrics was to be described in at most two variables. To this end, each of the nine exploded features was reduced from a description of dimensionality 13 to a description of dimensionality 2 by taking the two components with the highest explained variance from Principal Component Analysis (PCA). This resulted in a set of 3 count features + (9 features × 2 descriptors) = 21 features. On average, the first two components taken from PCA provided an explained variance of 98.98% for the nine features.

Regression pipeline

We fitted a multiple linear regression with heart rate per chunk as the dependent variable, and either of the features obtained by the methods outlined above as independent variables. In addition, a similar but polynomial regression was fitted, to identify possible non-linear links. All regression models were fit to the train set and R2 was evaluated on the test set.

Machine learning pipeline

Logistic Regression, K-Nearest Neighbours and Random Forest Classifier were used to predict high versus low heart rate from oculomotor metrics. Each type of model was run independently 50 times, with a new 80/20% stratified train/test split for each run, and with the default set of parameters as provided by scikit-learn. On average over those 50 runs, and across the three different pre-processing approaches, the Random Forest classifier performed best of the three models, and thus this model was selected for further optimization (see Table 2).

Table 2

AUCs of the model pre-selection process (averaged over 50 independent model runs).

	Logistic Regression	K-Nearest Neighbours	Random Forest	Random Forest + optimizationa
Averaging	.622	.617	.696	.698 (.04)
Explosion	.590	.588	.660	.664 (.05)
Explosion + reduction	.614	.585	.666	.678 (.04)

aThe average (SD) outcome on the test set over 50 runs of the optimized model is reported.

Subsequently, hyperparameter optimization of the Random Forest classifier was implemented over the number of trees (range 10–200; step size 1) and the maximum depth per tree (range 1–30 + unlimited depth). All other hyperparameters were kept as default. We then constructed 500 candidate combinations of hyperparameters by randomly sampling from their specified distributions. Each candidate combination was assigned the same 80% training set and was evaluated on that set using 5-fold stratified cross-validation and Area Under the Curve (AUC) as performance metric. An AUC of 0.50 constitutes classification at chance level and 1.0 constitutes complete accuracy. The model and parameter combination that led to the best cross-validation result was then tested on the 20% holdout set. To compensate for randomness effects in the sampling of the training- and test sets, and in the sampling of hyperparameters, this search process was repeated 50 times and means and standard deviations are reported.

Finally, the contributions of all features towards correct classification were extracted from the best-performing model using permutation importance [30]. For each feature, a one-sample t-test was performed to test whether that feature’s importance differed significantly from the overall mean (higher importance is better; t-test α = .05).

Results

Regression

R2 for regression models ranged between < .01 and .30 (see Table 1 for full results), indicating that oculomotor metrics provide limited information towards prediction of heart rate as a continuous variable.

Table 1

Outcomes (R2) of the linear regression models, per pre-processing approach.

Features were either derived directly from the pre-processing approach, or with added second-degree polynomials for each feature. Models were fit to the 80% train set and evaluated on the 20% test set.

	R2	R2 (with 2nd degree polynomials)
Averaging	.18	.30
Explosion	.21	<. 01
Explosion + reduction	.17	.12

Classification

Overall, the averaging pre-processing approach provided the best performance at classifying whether a participant had a high- versus low heart rate within a chunk (AUC = .696). The model pre-selection results and the results of optimization are reported in Table 2.

The best-performing model performed consistently above chance and achieved an average AUC of .703 (SD = .02) on the cross-validation sets, and an average AUC of .698 (SD = .04) on the test sets over 50 independent runs. An overview of the best models and the runner-up models is reported in Table 3.

Table 3

AUCs and parameters of the best-performing models and runner-up models resulting from hyperparameter search (on the averaging pre-processing approach).

	Model rank 1	Model rank 2	Model rank 3
Cross-validation performance (AUC)	.703	.701	.700
20% holdout set performance (AUC)	.698	-	-
Average number of trees	126.5	135.0	127.2
Average maximum depth per tree	20.2a	19.3a	19.5a

Model ranks were defined based on cross-validated classification performance. All values are averages over 50 runs. In each run, only the best model was tested against the test set.

aIncludes at least one model where the maximum depth was unlimited

The extraction of feature importance’s revealed blink rate, duration, and features associated with oculomotor movement to be most predictive of heart rate ([fixation and saccadic] median velocity, saccadic peak velocity; Fig 2). All other features were found to contribute worse-than-average towards classification.

Fig 2

Mean (± 95% CI) feature importance’s as extracted from the best-performing model of each of the 50 runs.

Higher values imply a higher degree of information within the variable. The vertical dashed line represents the overall mean of all importance values. The asterisks represent where feature importance’s differed significantly from the overall mean.

Discussion

In the current study, we investigated how well oculomotor metrics may predict heart rate and which of these features drive this prediction predominantly. To this end, we used a public dataset of participants whose physiological data were obtained while watching the 1994 Forrest Gump motion picture. Although oculomotor metrics provided limited predictive value for linear and polynomial regressions (up to R2 of .30), a Random Forest model could predict high- versus low heart rate consistently at above-chance level. In this model, the features which contributed most strongly towards correct classification were blink rate, blink duration, and the median velocity within fixations and saccades, and the saccadic peak velocity.

Interestingly, each of the features that contributed most strongly pertains to either information regarding blinks, or regarding oculomotor movement (velocities and amplitudes), and not so much to durations or counts of fixations and saccades. The importance of blink rate and blink duration provides support for the suggested link between an altered rate of eyeblinks and changes in arousal [11, 12] and changes in heart rate metrics [13]. At first sight, the relative importance of fixation velocity might be surprising, since fixations are spatially stable. However, differences in fixation velocities may be the result of physiological drift or microsaccades, sometimes referred to as fixational drift or fixational eye movements [31]. The occurrence of microsaccades has been found to be positively coupled to heartbeat, and may thus explain the amount of information captured in the fixation velocity variable [32]. The peak and median velocity of saccades are fourth and fifth in the list of informative features, which aligns with earlier literature which suggested that saccadic peak velocity indicates mental effort [7, 8] and motivation [9]–two cognitive processes closely linked to modulations in arousal.

Feature importance, however, does not indicate specifically which aspect of a distribution provides the most information towards correct classification. This makes it difficult to speculate about the direction of the effect of the included features, further complicated by inconsistencies in the literature. For instance, microsaccades occur more frequently with high mental effort in some tasks, but not in others [33, 34], suggesting that the modulation of eye movement and heart rate by the arousal system is highly task-dependent. This is further evidenced by the fact that we find increased saccadic- and fixational velocities in high heart rate periods, whereas it is usually found that saccadic and fixational velocity are negatively correlated with arousal [7, 33]. While, except within velocity, no consistently different medians within features were found between low- and high heart rate periods, it is remarkable that standard deviations were consistently equal or higher when heart rate was low, as compared to when it was high (with the exception of median saccade velocity). High arousal levels could be associated with a reduction in variability in oculomotor behaviour, as is the case with heart rate [35].

Based on these findings, we speculate that heart rate is not only linked to fixational eye movements [32], but to oculomotor movements in general. This link might come into place due to changes in the common underlying arousal system, or merely as an effect of changes in blood pressure during the heartbeat cycle. Our findings therefore suggest that a substantial portion of oculomotor behaviour is linked to heart rate, and not only by top-down goals of the beholder [22], or bottom-up visual features of the scene [36], as is commonly assumed. To this end, other physiological indicators could be compared to oculomotor metrics in their ability to predict heart rate. Because there is no unified definition of arousal, investigating the links between the aforementioned indicators would allow to isolate more specific subcomponents of arousal, and improve our definition of the term.

Speculating about neural underpinnings for a link between the oculomotor features described here and heart rate, we see a potential role for the locus coeruleus, a sympathetic center in the brain that acts antagonistically to parasympathetic activation associated with heart rate variability [37]. The noradrenergic locus coeruleus affects oculomotor behavior mainly via its inputs to the superior colliculus that is crucial in bringing about several oculomotor behaviours [38]. Note that locus coeruleus-centered and superior colliculus-centered circuits have been associated with differential attentional functions at the level of the brain stem, including alerting and orienting [38].

Another putative candidate might be the hypothalamus [13] (though bidirectionally linked to the locus coeruleus [38]) which modulates activity in the autonomous nervous system. Its link to the basal ganglia (and changes in the dopamine system) might explain the relation between blinks and heart rate, as changes in dopamine levels in the basal ganglia are monitored with changes in blink rate [13]. Although a relation between heart rate and oculomotor features and these two brain regions seems plausible, it is important to stress that this currently mere speculation and should be the subject of future research.

The current study is limited in its comprehensiveness of oculomotor features. For instance, pupil dilation has been shown to encode aspects of arousal [5, 6]. However, these measurements are distorted when gaze position changes [39] and could therefore not be reliably measured. Another step could be to link eye movements to on-screen movements in order to obtain smooth pursuits. This might be meaningful, as deviations in smooth pursuit trajectories have been found to be indicative of mental effort–and thus by proxy arousal [40]. Currently, smooth pursuits are likely to be captured within fixations and saccades at the high- and low ends of their respective velocity distributions. Lastly, as a next step, microsaccades could be investigated in detail [41, 42]. This would require robust detection algorithms that work without static scenes and with monocular data.

Different parameter sets and pre-processing approaches all lead to similar model performances, as shown in Tables 1–3. For example, the pre-processing approaches of averaging and feature explosion without reduction led to very similar outcomes in classification accuracy. However, the averaging approach required less processing time and can be interpreted more intuitively. Furthermore, classification accuracy could be improved by using more complex models, but again at the cost of interpretability.

The current study does not directly provide a method for the real-time prediction of heart rate from oculomotor metrics, since the proposed Random Forest classification pipeline requires that a baseline heart rate is established from which to derive low or high heart rate labels as the dependent variable. However, future research may attempt to establish a baseline heart rate measurement before the start of a given task and subsequently investigate whether the prediction of heart rate can be conducted in real-time.

Conclusion

In conclusion, oculomotor metrics obtained during a naturalistic viewing task contain sufficient information to predict high versus low heart rates above chance during that same task. These findings not only establish oculomotor metrics as unobtrusively measurable predictors of heart rate, but open new pathways for investigation of the link between oculomotor metrics and various indicators of arousal.

List of statistical descriptors.

(DOCX)

Click here for additional data file.

23 May 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Enkelejda Kasneci, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“This work was supported by ERC [ERC-CoG-863732].”

We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“This work was supported by ERC [ERC-CoG-863732], https://erc.europa.eu/, awarded to SVdS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors investigated the connection between faxation and saccade metric and changes in heart rate during film viewing. Based on the gaze data, they trained random forest models (both for regression and classification) in order to predict participants‘ heart rate during specific segments oft he movie.

There are a lot of aspects of this study that I like. They use a public dataset and provide their analysis scripts, so reproducability should be possible. I also think that the topic of oculomotor events and physiological arousal is very interesting and rarely investigated. I commend the authors for the work they have put into this manuscript!

The study design an methodology seem fine for the most part and the language is easy to understand. However, there a several concerns that need to be addressed before I can fully approve oft he manuscript as it is presented here.

The major concern is potentially confounding factors that are not looked into: namely blinks and smooth pursuits. Blinks are disregarded even though they could impact eye movements or may even add an additional puzzle piece fort he link that the authors are investigating. Nakano and Kuriyama (2017) showed that spontaneous blinks (that often occur at attentional breakpoints, which would be very fitting fort he movie context) are associated with increased heart rate, so this avenue seems like a good addition tot he feature portfolio employed by the authors. Blinks could also distort gaze metrics if they occur to frequently, so at least stating how they are addressed would improve my confidence in the presented results.

Smooth pursuits are likely relevant for gaze analysis of a movie as slowly moving targets or cameras moving in relation to actors or objects occur rather often. Smooth pursuits could especially skew fixation metrics like dispersion, velocity, or amplitude if not treated accordingly. Similar tot he issue with blinks, this should be addressed in some way.

Finally, in their discussion the authors mention microsaccades as a potential explanation for their high importance of fixation-related metrics. I would like to see this thought explored in more detail! The sampling frequency oft he eye tracker is 1000Hz, so reliably investigating microsaccades should be possible. This could really help to shed more light on the link between eye movements and heart rate. For datasets with lower sampling frequencies where microsaccades cannot be determined reliably, the authors‘ currently suggested metrics would then work as a substitute.

Minor issues:

- In the context of pupil diameter, the authors state that „… among eye tracking scientists there is no unified concept of how fixations and saccades should be defined – and thus the application of differing fixation- and saccade detection techniques may result in differing outcomes, even if they are applied to the same dataset (Hessels et al., 2018).“ The same argument likely applies to the metrics used by the authors as velocity and aplitude of fixations and saccades are a major feature in their approach.

- It may be tough to disentagle eye movement characteristics that are caused by physiology and arousal from those that appear stimulus driven which opens the door for many confounding factors.

- The imbalance on the two classes is not addressed, eventhough it is only slightly out of balance.

- How much variance was preserved by the PCA? This may help to judge the feature explosion and reduction approach.

- Was the z-normalization as a preprocessing step for forming two classes performed on a participant level or globally with the dataset as a whole?

- The figures do not seem to scale well. The authors may need to redo them as vectored graphics to help with readability.

Reviewer #2: In this study, the authors took data from 14 participants viewing films and compared oculomotor metrics to heart rate, querying whether they would be linked in a way where noninvasive oculomotor monitoring could predict heart rate.

In terms of analytics, the authors found that heart rate had to be split into high vs low, rather than as a continuous variable. This limits the predictiveness of the oculomotor metrics, as noted clearly by the authors. They found that 4 metrics: fixational and saccadic velocities, saccade peak velocity, and saccade amplitude were the best features for a random forest model to categorize each chunk of the movie watching as high or low heart rate better than chance.

This is a simple and elegant study. It is an initial proof of concept study (my description rather than the authors), towards the stated goal of using oculomotor metrics to predict heart rate in real time. The authors clearly note that the current method cannot be used in real time due to needing a baseline heart rate for the task at hand, but future work could improve classification accuracy or determine if a pre-task baseline can be used for real time prediction.

My concerns with the manuscript are based upon the short discussion. There are a couple of areas where the discussion could put the results in more context for the benefit of the readers.

The manuscript discusses the 4 features as feeding into the random forest model, and then some interpretations about why for each feature. However, the discussion does not clearly state the differences in light of low/high heart rate. For example, fixation velocity is appropriately described as potentially reflecting microsaccades, where microsaccade rate can vary by arousal or complexity, but which way? Alertness can improve fixational stability when focused on a difficult task, but arousal can increase exploratory gaze behavior. How is the metric of fixation velocity related to low vs high heart rate chunks, in this task of movie watching? The same lack of explanation occurs for the other 3 featured metrics. Or is it a given pattern/combination? While each metric's distribution is depicted in Figure 1, qualitatively there's a more visible difference between high/low heart rates for counts than for median velocities, yet the analytics showed median velocities over counts.

It would be useful for information about how the metrics (as a pattern, or individually) are related to the two heart rate categories to better relate the oculomotor system to heart rate, as movie watching and heart rate is related to limbic responses rather than the references in the discussion relating arousal to task difficulty and other achievement-style contexts. As the authors note, it may be due to a common underlying process. Underlying limbic system mechanisms could have a different effect on the oculomotor system than from say ascending reticular activating system or prefrontal-mediated executive functions such as attentional and inhibitory control.

And that is a second area to potentially added to the discussion - if there's a common mechanism, what would that putative mechanism be? Any known connectivity to the oculomotor system? Are these differences arising from oculomotor nuclei in the brainstem, subcortical areas, prefrontal? The results are clear, but the implications or interpretations that could link them more broadly are missing.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Tobias Appel

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

1 Jul 2022

We kindly thank both of the reviewers for their positive and useful feedback, and hope that the implemented changes and responses address all points adequately. As part of the revision, we have reanalyzed the data and completely revised the results, along with both figures and all tables.

Reviewer #1

The authors investigated the connection between faxation and saccade metric and changes in heart rate during film viewing. Based on the gaze data, they trained random forest models (both for regression and classification) in order to predict participants‘ heart rate during specific segments oft he movie.

There are a lot of aspects of this study that I like. They use a public dataset and provide their analysis scripts, so reproducability should be possible. I also think that the topic of oculomotor events and physiological arousal is very interesting and rarely investigated. I commend the authors for the work they have put into this manuscript!

The study design an methodology seem fine for the most part and the language is easy to understand. However, there a several concerns that need to be addressed before I can fully approve oft he manuscript as it is presented here.

1. The major concern is potentially confounding factors that are not looked into: namely blinks and smooth pursuits. Blinks are disregarded even though they could impact eye movements or may even add an additional puzzle piece fort he link that the authors are investigating. Nakano and Kuriyama (2017) showed that spontaneous blinks (that often occur at attentional breakpoints, which would be very fitting fort he movie context) are associated with increased heart rate, so this avenue seems like a good addition tot he feature portfolio employed by the authors. Blinks could also distort gaze metrics if they occur to frequently, so at least stating how they are addressed would improve my confidence in the presented results.

Thank you for this excellent remark. We have now included blink rates and blink duration in our detection algorithm, and as features in our models.

Whereas our best model previously achieved an AUC of .646 on the test set, it now classifies with an AUC of .698, meaning that blinks indeed add meaningful information to our model.

We have addressed this in various places in the manuscript, including completely revising Figures 1 and 2, Tables 1, 2, 3 and the results section. The most notable changes are as follows:

[lines 53-56] “Additionally, it has been shown that changes in arousal are paired with an altered rate of eyeblinks (e.g., Maffei & Angrilli, 2019; Wood & Hassett, 1983), and that spontaneous eyeblinks occur in tandem with an increase in heart rate variability (Nakano & Kuriyama, 2017).”

[lines 117-120] “Lastly, blinks were detected by finding periods in which no pupil data was measured. All events which lasted less than 30ms, or more than 3 seconds, were removed. These thresholds were set so that neither brief nor longer periods of data loss would be incorrectly detected as blinks.”

[lines 249-252] “The importance of blink rate and blink duration provides support for the suggested link between an altered rate of eyeblinks and changes in arousal (Maffei & Angrilli, 2019; Wood & Hassett, 1983) and changes in heart rate metrics (Nakano & Kuriyama, 2017).”

2. Smooth pursuits are likely relevant for gaze analysis of a movie as slowly moving targets or cameras moving in relation to actors or objects occur rather often. Smooth pursuits could especially skew fixation metrics like dispersion, velocity, or amplitude if not treated accordingly. Similar tot he issue with blinks, this should be addressed in some way.

We agree that smooth pursuits could be a relevant metric to include. However, we have found smooth pursuits to be difficult to reliably extract as an independent metric from this dataset. Namely, we would need to know whether there are pursuable objects on the screen, and where they are, at any given moment – information that is currently not available in this public data set.

A second option is to set a filter (e.g., fixational events with velocity > 2 degrees per second) and label those events as smooth pursuits. Smooth pursuit events are currently likely to be captured within both fixations and saccades at the high- and low ends of the velocity distributions respectively. Including smooth pursuits like this would therefore be a relatively coarse approach that will likely not only include smooth pursuits, but also other types of oculomotor behavior. To ensure interpretability we thus prefer to not include smooth pursuits as a separate feature in the final analysis. Yet, we fully agree with the reviewer here and discuss the potential of smooth pursuits as further predictor as follows:

[lines 307-312] “Another step could be to link eye movements to on-screen movements in order to obtain smooth pursuits. This might be meaningful, as deviations in smooth pursuit trajectories have been found to be indicative of mental effort – and thus by proxy arousal (Kosch et al., 2018). Currently, smooth pursuits are likely to be captured within fixations and saccades at the high- and low ends of their respective velocity distributions.”

3. Finally, in their discussion the authors mention microsaccades as a potential explanation for their high importance of fixation-related metrics. I would like to see this thought explored in more detail! The sampling frequency of the eye tracker is 1000Hz, so reliably investigating microsaccades should be possible. This could really help to shed more light on the link between eye movements and heart rate. For datasets with lower sampling frequencies where microsaccades cannot be determined reliably, the authors‘ currently suggested metrics would then work as a substitute.

Microsaccades are certainly informative (particularly regarding the link to heart rate), but are hardly retrievable during free viewing with sufficient confidence. We agree that sampling rate and accuracy of the tracker would be sufficient to track microsaccades – but still, a reliable estimation of microsaccades is questionable (see explanation above). For one, Engbert & Kliegl (2003) describe microsaccade detection for binocular data, and while it is also possible with monocular data (like in the dataset used here), this is more prone to noise. Furthermore, classical studies using microsaccades strictly ensure fixation, which is impossible to do during free viewing and would almost certainly drive-up noise even further. We now address this interesting further step by discussing:

[lines 312-314] “Lastly, as a next step, microsaccades could be investigated in detail (Duchowski et al., 2020; Engbert & Kliegl, 2003). This would require robust detection algorithms that work without static scenes and with monocular data.”

4. Minor issues:

a. In the context of pupil diameter, the authors state that „… among eye tracking scientists there is no unified concept of how fixations and saccades should be defined – and thus the application of differing fixation- and saccade detection techniques may result in differing outcomes, even if they are applied to the same dataset (Hessels et al., 2018).“ The same argument likely applies to the metrics used by the authors as velocity and aplitude of fixations and saccades are a major feature in their approach.

We agree with the reviewer that this is an important issue and now addressed it more clearly. Namely, we applied merging criteria which have been demonstrated to reduce such event-detection induced effects to a minimum. We now write:

[lines 111-115] “Thereafter, we applied two basic merging criteria. Firstly, saccade candidates with amplitude < 1.0° were removed, thereby merging neighbouring fixation candidates. Subsequently, all fixation candidates with duration < 60ms were removed. This procedure successfully removes large differences in oculomotor event classification between different algorithms (Hooge et al., 2022).”

Furthermore, our main intention in the quoted paragraph was to emphasize that including additional features – or a combination of several features – would eventually make models more robust to this variance. We have now accentuated this in the manuscript:

[lines 64-68] “As such, incorporating several metrics which can be independently extracted (e.g., pupil size, oculomotor movement, blinks) would improve robustness of the model, as it reduces dependence on one single extraction technique. This also applies to cases in which pupil dilation measurements are unreliable or missing, or the eye tracker’s sampling rate is too low to extract peak saccade velocities.”

b. It may be tough to disentagle eye movement characteristics that are caused by physiology and arousal from those that appear stimulus driven which opens the door for many confounding factors.

We fully agree with the reviewer here – this endeavour will be an ongoing one for the field to which we try to contribute a first step. We see the combination of several indicators of arousal (here heart rate) with oculomotor metrics as a promising step in this direction, yet these results only yield correlational not causal evidence. We address this by writing in the discussion:

[lines 280-284] “Our findings therefore suggest that a substantial portion of oculomotor behaviour is linked to heart rate, and not only by top-down goals of the beholder (e.g., Kootstra et al., 2020), or bottom-up visual features of the scene (e.g., Itti & Koch, 2000), as is commonly assumed. To this end, other physiological indicators could be compared to oculomotor metrics in their ability to predict heart rate.”

c. The imbalance on the two classes is not addressed, eventhough it is only slightly out of balance.

We have clarified this in the manuscript:

[lines 144-147] “All other chunks were considered neutral and discarded. Since the distributions of heart rate were often skewed, and due to slightly differing amounts of data loss, our binarization did not result in equally large samples of high and low labels.”

d. How much variance was preserved by the PCA? This may help to judge the feature explosion and reduction approach.

Taking the first two components from PCA for each of the features provided an overall average of 98.98% explained variance. We here report the full table, but only include the overall average in the manuscript for brevity.

Movement type Feature Explained variance (%)

Blink Duration 91,93

Fixation Amplitude 99,88

Fixation Duration 99,55

Fixation Median velocity > 99.9

Fixation Peak velocity > 99.9

Saccade Amplitude > 99.9

Saccade Duration 99,43

Saccade Median velocity > 99.9

Saccade Peak velocity > 99.9

Overall 98,98%

Adjusted in the manuscript:

[lines 181-183] “On average, the first two components taken from PCA provided an explained variance of 98.98% for the nine features.”

e. Was the z-normalization as a preprocessing step for forming two classes performed on a participant level or globally with the dataset as a whole?

z-normalization of heart rate was performed per participant. We now more clearly state:

[lines 141-143] “To prepare our dependent variable for binary classification, the heart rate of each chunk was expressed as a z-score; the number of standard deviations from the median heart rate of that respective participant over the full film.”

f. The figures do not seem to scale well. The authors may need to redo them as vectored graphics to help with readability.

Thank you for noticing this. We have changed the figure format.

Reviewer #2

In this study, the authors took data from 14 participants viewing films and compared oculomotor metrics to heart rate, querying whether they would be linked in a way where noninvasive oculomotor monitoring could predict heart rate.

In terms of analytics, the authors found that heart rate had to be split into high vs low, rather than as a continuous variable. This limits the predictiveness of the oculomotor metrics, as noted clearly by the authors. They found that 4 metrics: fixational and saccadic velocities, saccade peak velocity, and saccade amplitude were the best features for a random forest model to categorize each chunk of the movie watching as high or low heart rate better than chance.

This is a simple and elegant study. It is an initial proof of concept study (my description rather than the authors), towards the stated goal of using oculomotor metrics to predict heart rate in real time. The authors clearly note that the current method cannot be used in real time due to needing a baseline heart rate for the task at hand, but future work could improve classification accuracy or determine if a pre-task baseline can be used for real time prediction.

My concerns with the manuscript are based upon the short discussion. There are a couple of areas where the discussion could put the results in more context for the benefit of the readers.

1. The manuscript discusses the 4 features as feeding into the random forest model, and then some interpretations about why for each feature. However, the discussion does not clearly state the differences in light of low/high heart rate. For example, fixation velocity is appropriately described as potentially reflecting microsaccades, where microsaccade rate can vary by arousal or complexity, but which way? Alertness can improve fixational stability when focused on a difficult task, but arousal can increase exploratory gaze behavior. How is the metric of fixation velocity related to low vs high heart rate chunks, in this task of movie watching? The same lack of explanation occurs for the other 3 featured metrics. Or is it a given pattern/combination? While each metric's distribution is depicted in Figure 1, qualitatively there's a more visible difference between high/low heart rates for counts than for median velocities, yet the analytics showed median velocities over counts.

Thank you for raising this important theoretical question. Indeed, it is useful to know how specifically any of our given features link to heart rate. Of course, a combination of features might explain more, by interactive effects, than individual features. We now explain why feature importances contain limited information about directionality, and that these do not allow conclusions about patterns within features. We relate our findings to inconsistencies within literature, as pointed out by the reviewer. We have also included medians and standard deviations to the distributions in Figure 1. As can be seen, medians are often very similar across high- and low heart rates, regardless of whether these features contribute strongly to the models. Yet, standard deviations are consistently equal or higher when heart rate is low, compared to when it is high (with the exception of median saccade velocity), which indicates that high arousal levels could be associated with reduced variability in oculomotor metrics.

We now address this by writing:

[lines 262-276]: “Feature importance, however, does not indicate specifically which aspect of a distribution provides the most information towards correct classification. This makes it difficult to speculate about the direction of the effect of the included features, further complicated by inconsistencies in the literature. For instance, microsaccades occur more frequently with high mental effort in some tasks, but not in others (Pastukhov & Braun, 2010; Siegenthaler et al., 2014), suggesting that the modulation of eye movement and heart rate by the arousal system is highly task-dependent. This is further evidenced by the fact that we find increased saccadic- and fixational velocities in high heart rate periods, whereas it is usually found that saccadic and fixational velocity are negatively correlated with arousal (Di Stasi et al., 2013; Siegenthaler et al., 2014). While, except within velocity, no consistently different medians within features were found between low- and high heart rate periods, it is remarkable that standard deviations were consistently equal or higher when heart rate was low, as compared to when it was high (with the exception of median saccade velocity). High arousal levels could be associated with a reduction in variability in oculomotor behaviour, as is the case with heart rate (Kazmi et al., 2016).”

2. It would be useful for information about how the metrics (as a pattern, or individually) are related to the two heart rate categories to better relate the oculomotor system to heart rate, as movie watching and heart rate is related to limbic responses rather than the references in the discussion relating arousal to task difficulty and other achievement-style contexts. As the authors note, it may be due to a common underlying process. Underlying limbic system mechanisms could have a different effect on the oculomotor system than from say ascending reticular activating system or prefrontal-mediated executive functions such as attentional and inhibitory control.

And that is a second area to potentially added to the discussion - if there's a common mechanism, what would that putative mechanism be? Any known connectivity to the oculomotor system? Are these differences arising from oculomotor nuclei in the brainstem, subcortical areas, prefrontal? The results are clear, but the implications or interpretations that could link them more broadly are missing.

These are interesting questions and agree that a discussion on putative mechanisms is interesting. We believe that hypothalamus and locus coeruleus are most likely underlying centres, yet this is not necessarily exclusive and cannot be answered exhaustively with the current data. We now mention the following in the discussion:

[lines 288-303] “Speculating about neural underpinnings for a link between the oculomotor features described here and heart rate, we see a potential role for the locus coeruleus, a sympathetic center in the brain that acts antagonistically to parasympathetic activation associated with heart rate variability (Mather et al., 2017). The noradrenergic locus coeruleus affects oculomotor behavior mainly via its inputs to the superior colliculus that is crucial in bringing about several oculomotor behaviours (Strauch et al., 2022). Note that locus coeruleus-centered and superior colliculus-centered circuits have been associated with differential attentional functions at the level of the brain stem, including alerting and orienting (Strauch et al., 2022). Another putative candidate might be the hypothalamus (Nakano & Kuriyama, 2017; though bidirectionally linked to the locus coeruleus, Strauch et al., 2022) which modulates activity in the autonomous nervous system. Its link to the basal ganglia (and changes in the dopamine system) might explain the relation between blinks and heart rate, as changes in dopamine levels in the basal ganglia are monitored with changes in blink rate (Nakano & Kuriyama, 2017). Although a relation between heart rate and oculomotor features and these two brain regions seem plausible, it is important to stress that this currently mere speculation and should be the subject of future research.”

Submitted filename: Response to reviewers.docx

Click here for additional data file.

19 Jul 2022

Seeing the Forrest through the trees: Oculomotor metrics are linked to heart rate

PONE-D-22-08472R1

Dear Dr. Hoogerbrugge,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Enkelejda Kasneci, Ph.D.

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I want to thank the authors for their in depth rebuttal letter and their revision adressing all my concerns! Great work!

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Tobias Appel

Reviewer #2: Yes: Mazyar Fallah

**********

25 Jul 2022

PONE-D-22-08472R1

Seeing the Forrest through the trees: Oculomotor metrics are linked to heart rate

Dear Dr. Hoogerbrugge:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Dr. Enkelejda Kasneci

Academic Editor

PLOS ONE