Seeing through the forest: The gaze path to purchase.

Eye tracking studies have analyzed the relationship between visual attention to point of purchase marketing elements (price, signage, etc.) and purchase intention. Our study is the first to investigate the relationship between the gaze sequence in which consumers view a display (including gaze aversion away from products) and the influence of consumer (top down) characteristics on product choice. We conducted an in-lab 3 (display size: large, moderate, small) X 2 (price: sale, non-sale) within-subject experiment with 92 persons. After viewing the displays, subjects completed an online survey to provide demographic data, self-reported and actual product knowledge, and past purchase information. We employed a random forest machine learning approach via R software to analyze all possible three-unit subsequences of gaze fixations. Models comparing multiclass F1-macro score and F1-micro score of product choice were analyzed. Gaze sequence models that included gaze aversion more accurately predicted product choice in a lab setting for more complex displays. Inclusion of consumer characteristics generally improved model predictive F1-macro and F1-micro scores for less complex displays with fewer plant sizes Consumer attributes that helped improve model prediction performance were product expertise, ethnicity, and previous plant purchases.

Introduction

Most consumer product choices (>90%) are made at the point of purchase (POP) [1]. Retailers have a variety of tools (signs, labels, display fixtures, merchandise arrangement, etc.) for organizing the store environment to capture consumer attention and motivate consumer choice. Consumer packaged goods companies invested approximately $29.7 billion in shopper marketing in 2016, accounting for more than 13% of their marketing budget [2]. Marketing organizations perceive this investment to be effective, as indicated by 73% of surveyed executives who rated in-store marketing as very or quite useful [3], making the retail space a critical location for purchase decisions.

When shopping, consumers identify visual cues then cognitively process those cues to make a purchase decision [4, 5]. Visual attention, which is requisite for purchase, is the process by which stimuli are selected and integrated for cognitive function allocated to the task at hand. Portable and affordable eye tracking (ET) equipment has created a reliable, objective mechanism to study visual cue selection leading to purchase. Past ET research has shown that single cues such as signage [6, 7], pricing [8], packaging [9], brand [10, 11], and product display [12] influence visual attention and purchase intention. Yet, consumers often ignore some visual cues. For example, Hendrickson and Ailawadi [13] observed that in a store setting, shoppers do not look up unless they are “way finding” and store signs are noticed only by about 3% of shoppers. While visual time in product selection has been investigated, the order in which consumers access visual cues (e.g., gaze sequences) has not been widely studied.

Gaze sequence, or the progression of visual cues to which consumers attend, is not random, but task specific [14]. Some ET research has studied the areas at which consumers look prior to product choice, but a breakdown of the gaze sequence merits more investigation. In an in-store setting, we assume that when given a shopping task consumer search is purposeful [14] and that there might be gaze sequences that are predictive of choice. That is, not just what a consumer views or how often an item is viewed, but the order in which components of a display are viewed may be informative of consumer choice. Additionally, by solely focusing on consumer visual attention to products and POP communication tools, previous studies have not investigated the possible relationship between looking away from product visual cues and purchase choice. The gaze aversion literature [15-17] suggests that humans may sometimes need to look away from a main visual target to facilitate information processing. Since the range of product choice and the number of visual cues at the POP can be overwhelming at times, it seems plausible to conjecture that a similar mechanism may affect consumers who seek to alleviate their mental discomfort of “choice overload” [18]. Thus, the purpose of our study is to investigate whether a relationship exists between subcomponents of consumer gaze sequence and product choice for a minimally packaged, unbranded product, and to understand whether looking away, which we term Look Away to Decide (LATD), plays a role in a consumer’s gaze sequence on their path to product choice.

Study significance and literature gaps

The majority of ET studies focus on branded and/or fast-moving packaged consumer goods displayed on shelves [10, 19, 20], with products in identical packaging size and product quality assessed by reading labels or examining packaging. Our study focuses on non-branded, similar but non-uniform (non-identical) products (live plants) where quality assessment may require product scrutiny. In the next section, we review studies that have focused on aspects of gaze sequence, such as gaze aversion and central gaze bias. It is important to note, however, that few of the studies investigated the gaze paths in which visual cues were accessed and no studies investigated whether that relationship predicted product choice. Our study integrates extant literature findings and expands them by documenting the patterns in the cue selection process that leads to a product choice. In our analysis of visual behavior, we add a new and heretofore unexamined viewing behavior, LATD. The findings of our study have both theoretical and practical implications. From a theoretical standpoint, linking features within consumer gaze sequence to product choice aids in our understanding of cognitive processing of visual cues. Commonalities and differences in gaze sequences can lend insight into product choice by pinpointing cues that are selected (vs. ignored). For managers, the ability to pinpoint one or more gaze paths that lead to purchase intention will enable retailers to reimagine the environment in which merchandise is presented, (e.g., signage/pricing placement and display spacing). Our goal is to better understand subcomponents of the gaze sequence that are associated with product selection by consumers.

Review of literature

Decision-making cues

Eye-tracking has been utilized in hypothetical consumer studies related to product choice, largely through laboratory investigations of attendance and non-attendance to package, label, or menu cues. For example, Ballco et al [21] demonstrated nutritional claims were influential in a hypothetical discrete choice experiment. Tórtora and colleagues [22] found nutritional warnings attracted attention and required less time and fewer fixations to process compared to facts on a front panel. Additionally, Otterbring and Shams [23] used eye-tracking while showing a video to subjects in a lab setting to find that prior exposure to a seemingly healthy consumer produced greater visual attention towards products perceived to be healthy and eliciting healthier cereal choices. That study extended prior work showing student subjects who viewed unhealthy body types depicted on a menu spent more visual time viewing healthy food options [24].

Visual attention time correlates positively with purchase, such that increased fixation time on certain product attributes indicates an increased purchase likelihood [25-29]. Gidlöf et al. [27] reported, “the very act of looking longer or repeatedly at a package, for any reason, makes it more likely the product will be bought.” Similarly, Scarpi et al. [20] found that fixation time for prototypical products increased purchase intention. Khachatryan et al. [30] found subjects who scored high on the buying impulsiveness scale fixated less on POP information and more on the product (live plants). The studies cited above focus on how consumers view a display (e.g., time fixating on the product, specific attributes, or display features), rather than the sequence in which these cues are viewed.

Top-down and bottom-up cues

Attention to visual cues implies an awareness of the stimuli in the conscious mind, which is driven by both top-down and bottom-up processes [14, 31]. In the visual cognition literature, bottom-up processes refer to stimulus driven attention [32] or low-level stimulus features that capture attention. Research on bottom up processes finds that visual attention is driven by salient stimuli features (e.g. contrast, color) and is mostly involuntary and reactive [33]. Top down processes refer to the conscious experience of visual stimuli [34], are context dependent [35] with visual attention being allocated based on the nature of the task [33, 35]. That is, attention is drawn to relevant areas based on the task at hand (e.g. selecting a blue flower). The visual cognition literature suggests that in the absence of a task, bottom-up processes influence visual attention, but once the task tells us what to look for, it changes the way we look at stimuli [36]. In other words, in the absence of a task, visual attention might be driven by the most salient feature of the stimuli, i.e. driven by a bottom-up process, but once a task is assigned visual attention will be directed toward those features that fulfill the task.

Wedel and Pieters [14] apply the findings of visual cognition literature to visual attention to marketing. They posit that (marketing) stimuli salience and informativeness act in combination to affect attention. Aligned with the visual cognition literature, Wedel and Pieters [14] propose that bottom-up processes operate when consumers attend to salient features such as color or contrast and this attention is mainly involuntary. Top-down processes, as identified by Wedel and Pieters [14] refer to perceptions of marketing stimuli that could be influenced by consumer demographics, brand familiarity, expertise, involvement and attitudes or goals. Unlike tasks assigned in the visual cognition literature which have a correct/incorrect outcome [33, 35] marketing study tasks are usually open ended (which brand would you buy) with no right or wrong answer. Outcomes are determined by consumer factors. For this study, we are interested in analyzing which, if any, consumer characteristics influence gaze pattern for plant displays.

Additionally, prior studies have shown that consumers who are more knowledgeable (have greater expertise) about a product make purchase decisions differently than less knowledgeable consumers [37]. Expertise arises from training, practice, or time spent learning about a particular topic [38, 39]. Alba and Hutchinson [40] and Shanteau [41] reported that consumers with high product expertise were more selective of the information they examined prior to making a choice since they had a better understanding of what product attributes should be examined. Expertise has been linked to visual attention as well [42].

Gaze sequence

Several ET studies provide researchers with a starting point for understanding the role of gaze paths in purchase intention and product choice. In an early ET study conducted in a retail context, Russo and Leclerc [43] characterized four stages of a consumer choice process. They documented a sequence of visual cue selection, followed by deliberate and effortful brand elaboration, then by final fixations and post-announcement (choice) fixations. Clement [19] expanded the findings of Russo and Leclerc [43] by using gaze paths to identify several decision phases: pre-attention, succeeded attention, tipping point, physical action, semantic information process, and post-purchase phases. Ҫöltekin, et al. [44] reported that geography experts (vs. novices) had shorter, and thus more efficient, gaze sequences when completing the task of finding a point on a map. Josephson and Holmes [45] had mixed results in their effort to identify gaze patterns when subjects viewed a webpage. Drusch et al. [46] also investigated scanpaths for webpages and identified two groups, from 91 subjects, with similar scanpaths using Hausdorff distance metrics. The empirical contribution of these studies is the documentation of a multi-stage gaze or search strategy that precedes purchase decision; however, none of these studies specified particular gaze sequences that guided this process and led to product choice.

An ET study of online information search patterns demonstrated that customers limited their attention to approximately three attributes for one product [47]. A key finding was the identification of a gaze sequence in which a preferred alternative was used as the basis, or anchor, for comparing other alternatives. Shi et al.’s [47] model contributes to the understanding of gaze paths or patterns by illustrating how this phenomenon is empirically linked to a preferred alternative, with that alternative becoming the basis for comparison of other alternatives. Thus, early in the search, process consumers appear to identify an initial (product) candidate and make iterative comparisons back to that primary choice. This would make the first visual point of contact as the one that establishes the bar or standard for others to meet or exceed. Onuma et al. [48] further showed that the second look at a chosen item was longer than the first look, which they posit was evidence for encoding information sequentially with a comparison phase in choice. Rebollar et al. [49] identified a gaze sequence (referred to as scanpath by the authors) of chocolate bar attributes as focusing initially on the bar name (the most prominent feature), followed by the brand. However, these studies neither compare sequences of fixations beyond the first few nor do they examine how fixation patterns can be informative of purchase decision.

Literature linking gaze sequence to consumer information processing and decision-making is scant and disparate. The published literature involving product choice utilizing ET technology includes studies mainly focused on first, last, central, or important areas of interest (AOIs) rather than gaze sequences. Atalay et al. [50] demonstrated the central gaze effect where branded products in the center of a vertical display had greater visual activity and were more likely chosen. Armel et al. [51] showed that the likelihood of selecting the product on the left (in a binary choice scenario) increased with longer gaze time. Reutskaja et al. [52] conducted a series of simulated choice studies on a computer screen and incorporated eye-tracking. They showed that as the number of product options (snack items) increased, consumer slightly increased the number of alternatives evaluated. Furthermore, products positioned centrally in the vertical display had more first fixations especially when the number of choices in a set increased, consistent with Atalay et al. [50]. Bialokva et al. [53] conducted lab and in store studies for product choice in a supermarket. They found that product brand and flavor were the key drivers of visual interest. Huddleston et al. [54] identified five studies that used packaged goods to link visual measures to actual product choice in the retail environment [7, 9, 19, 43, 55], but none involved gaze sequence analysis.

Related to the Shi et al. [47] study are two ET studies that indicate a central gaze bias tendency and identification of different viewing behaviors for horizontal versus vertical displays. Atalay et al.’s [50] studies investigated how horizontal location on a webpage affects choice, finding that brands in the center were more frequently chosen, received more eye fixations, and were looked at longer. Additionally, consumers were more likely to fixate on the centrally located brand in the last few seconds of gaze duration; these central fixations increased the likelihood of choosing a brand located in the center. They concluded that a centrally located option in a product display is preferred and garners more attention. However, the central position did not have an impact on the inferences that consumers made about the fictitious brands (e.g., amount of market share) or memory of which brand they chose [50]. This suggests that a centrally located position does not influence consumer product judgments. Comparing horizontal versus vertical display types, Deng et al. [56] discovered that consumers purchased higher quantities and greater varieties from horizontal displays; horizontal displays increased processing fluency; thus consumers perceived that horizontal assortments had a higher level of variety; and when faced with time constraints, consumers spent more time processing horizontal displays and fixated on a larger number of items. For a menu selection, related somewhat to product choice, Yang [57] showed that the scanpath of subjects reading a restaurant menu consisted of two vertical (top to bottom) scans; this finding was counter to the almost circular viewing path the restaurant industry hypothesized.

Lahey and Oxley [58] acknowledged the great utility that visual metrics have in behavioral economic studies and called specifically for analyzing the scan path or gaze sequence. Still, within the realm of product choice, scant work has been published implementing gaze sequence analysis. Thus, there is a need for more investigation in the gaze sequence related to product choice. Based on these findings, central position and horizontal displays appear to enjoy an advantage in stimulating consumer choice. Our study, while informed by the central gaze theorem, differs from previous work in that we focus on horizontal versus vertical displays and, rather than using identical branded packages, our stimuli are unbranded plants that vary slightly in appearance.

Thus, we pose our first two research questions:

RQ1: Are consumer gaze sequence paths predictive of product choice?

RQ2: Do consumer characteristics (top-down factors) enhance the predictive accuracy of gaze paths?

Gaze aversion (look away to decide)

When faced with a plethora of visual cues (e.g., signs, labels, product assortment), consumers must somehow visually select and then cognitively process these cues to arrive at a decision. Glenberg et al. [17] opined, “when engaged in difficult cognitive activity, we close our eyes or look at the sky to suppress the environment’s control over cognition (p. 651).” Psychology literature suggests that humans are predisposed to avert their gaze directed at others (gaze aversion) when faced with difficult decisions or complex mental tasks as an attempt to facilitate cognition. Several researchers [15-17] suggested that in face-to-face interactions, people needed to avert their gaze when answering difficult questions. Gaze aversion helps individuals avoid processing unnecessary visual cues and, consequently, enhances their cognitive control and improves their performance when answering questions [16, 17]. One study [59] pinpointed an adverse effect of gaze aversion when they found that interview answers were less reliable in conversational interviews when there was gaze aversion.

The gaze aversion literature has focused on face-to-face human interaction, not consumer viewing patterns in the context of product choice. Nonetheless, we believe that these findings can be extrapolated to a shopping environment because, similar to face-to-face interactions, consumers in a shopping context are inundated with an excess of visual cues and may need “visual space” to cognitively process this information. In a shopping environment, there are often many extraneous visual cues that do not aid in decision making, and it is possible that consumers may engage in LATD to facilitate their decision-making process.

The authors conceptualize LATD as the direction of consumers’ central gaze (gaze fixations) toward visual cues away from the choice task at hand. Therefore, LATD is not related to peripheral vision. Gaze centrality is the area to which the eyes are fixed and indicate selective attention directed to a visual cue [50, 60]. On the other hand, peripheral vision is the visual area lying outside the central gaze, or narrow foveal angle, with a limited perception of shapes and colors [61, 62]. Scant literature about peripheral vision and product choice provide evidence that consumers can engage their vision peripherally to eliminate non-relevant visual cues from active consideration [61]. The opposite happens when consumers engage in LATD, they redirect their central gaze either to visually process non-relevant cues instead, such as the store ceiling, the floor, or a product display background to reduce or eliminate stimulus input for the moment.

Thus, we pose our third research question:

RQ3: Does LATD improve predictive accuracy in a gaze path sequence to product choice?

Methods

Open science statement

Data, analytic code, stimuli, survey instrument, and supplemental information (Appendix A depicting how 3-mers were developed) can be found online in the Open Science Framework (OSF) website: https://osf.io/hykwu/?view_only=5e36084775af40218c6a1617b3337e3b

Stimuli

The study protocol and questionnaire were approved by the Michigan State University Institutional Review Board (17–458) which included approval and use of a written consent form. We conducted a 3 (display size) X 2 (price) within-participants experimental design in May 2017 (examples of the display designs are depicted in Figs 1–3). The six table-top displays showed three varieties of flowering plants [large (24 plants), moderate (12 plants), small (6 plants)] at two price levels (sale, non-sale). The sign displayed either the sale (low) price ($1.99) or the non-sale (high) price ($3.99) using black Calibri font (size 80 for non-price wording and 40 for price information). Text and spacing were identical across all displays. Displays were isolated from others with black cloth that extended fully around the sides of each display. Plants occupied approximately 0.12 m2, 0.25 m2, and 0.45 m2 of display space for the 6, 12, and 24-plant designs, respectively. Each display provided about 0.46 m2 of display area. All participants were exposed to a total of six plant display conditions, and subjects could view only one display at a time and began the study at successively different displays to reduce order bias. Each subject proceeded from one display to the next unaccompanied using a progressive start to the first display with no other participants viewing displays. Two researchers were in the same room, but out of the visual line of sight while subjects participated in the study.

Fig 1

Example of six-plant display with low price sign.

Fig 3

Example of 24-plant display with low price sign.

Sample and protocol

In the lab at a midwestern university, subjects were asked to view each of the six displays. Their task was to the one product (plant) from each display that they would be most likely to purchase and verbally indicate how likely they would be to purchase the plant using the 11-point Juster scale [63]. Alternatively, subjects could indicate they would not buy any of the plants in a display, reflecting a 0 on the Juster scale. After viewing all plant displays, subjects completed an online survey that measured their level of both self-reported and actual product expertise (following [42]) as well as their purchase history in the six months prior to the study. To measure purchase history, participants answered the following question: “How much did you spend (in total) on gardening supplies and plants (excluding mechanical equipment like mowers and tillers)?” Response options included “$0”, “$1 to $24”, “$25 to $49”, and then increased in $50 increments up to “$500 or more”. The survey also collected their demographic information.

Subjects [n = 92, mean (standard deviation) age = 35.3 (12.9) years; females, 69.6%; identified as white, 68.5%; identified as Asian, 17.4%; identified as African American, 4.4%; and, mean income = $73,294 ($50,412)] were recruited through an online panel maintained by researchers (panel comprised of both students and non-students) at a large Midwestern university and paid $25 for their participation. After completing the informed consent process, subjects were fitted with Tobii Pro Glasses 2 eye tracking glasses (with a sampling frequency of 50 Hz, 20 ms sampling interval, and binocular accuracy of 0.62°) and given three practice rounds with chocolate bars to familiarize them with the product choice task and Juster scale. They were instructed first to point to the product (candy bar or plant) they were most likely to buy, and then to verbally state their likelihood of buying that item using whole numbers from 0 to 10 [63] with a score of 0 indicating that the subject chose to not make a purchase. The displays were set in a random order in the lab, but subjects viewed the randomized displays in sequential order, with each subject starting in a successive sequence. Table 1 shows a two-way ANOVA with repeated measures to evaluate the effect of different display sizes (small, moderate and large) with various prices (sale vs not-sale) on the likelihood to buy. Two main effects (display size and price) are significant, which indicates the display sizes and price both affect consumers’ likelihood to buy. There is also a statistically significant interaction between display sizes and price on the likelihood to buy, F-value = 7.524, p<0.0001. Fig 4 indicates that participants’ buying likelihood on average are greater in a display under the sale price. Besides, the sale price, displays with a large (24 plants) or small (6 plants) number of product options tend to have greater mean likelihood to buy, compared to the moderate display size.

Table 1

Two-way repeated measures ANOVA on likelihood to buy.

Factor	Df	F-value	p-value
Display_size	2	9.551	0.000114*
Price	1	15.63	0.000152*
Display_size * Price	2	7.524	0.000725*

Fig 4

Average likelihood to buy score with 95% confidence intervals over three display sizes (small, moderate and large) and price (sale and not-sale).

The black square represents “not-sale” group, and the red square represents “sale” group.

Data handling and analysis

We digitally created areas of interest (AOIs) around each plant using Tobii Pro Lab software (version 1.73) and assigned those AOIs a letter (A–F in the six-plant display, A–L in the 12-plant display, and A–X in the 24-plant display), with the informational sign denoted as “Y” and gazes off the sign or plant (gaze aversion) denoted as “Z”. The raw gaze data for each participant, available in 20 ms intervals, was exported, then reduced to just the gaze changes between AOIs. Thus, for each subject at each display, we created a gaze sequence file showing the progression of gaze events throughout the display, ending when the subject made a choice. For example, the gaze sequence with looking-away fixation of one participant viewing the six-plant display was “ZZYZYCECACACZYBYZZBYB” until verbal choice. To interpret this gaze sequence, the subject looked away (Z) six times, looked at the sign (Y) four times and looked at plant C four times, plant E once and plant B three times. Similarly, the gaze sequence without looking-away fixation would be “YYCECACACYBYBYB”.

A custom Python program was prepared to parse the gaze sequence files. For each gaze sequence, all possible subsets of consecutive sequential fixations (called k-mers, where k is the number of sequential fixations) were identified and subjected to further analyses. For example, when k is equal to 3 (3-mer), relative frequencies of consecutive subsequences with length 3 (ZZY, ZYZ, YZY, ZYC, YCE, CEC, ECA, etc. for the first gaze sequence example above) are extracted as features contained in each gaze sequence with look-away fixations (see Appendix A for a detailed example). The use of k-mers to break down the gaze fixation sequences is inspired by k-mer analysis of nucleic acid sequences in bioinformatics [64, 65]. Single fixations or sequences of two fixations (2-mers) hold very little information about participants’ sequential gaze. However, as the number of fixations in a sequence increased to 4-mers and above, the proportion of participants sharing an identical gaze sequence decreased drastically, thus causing an analytical problem of finding relevant patterns in the data. This dilemma is also known as sparse feature matrix [66]. For that reason, we examined gaze patterns as sets of 3-mer fixations. In this study, 3-mers retain quantitative data regarding the total number of fixations on each object in the display, but they also preserve information about the order of fixations between the objects in the display. Furthermore, these 3-mers intuitively often contain gaze patterns employed by the test subject. In the gaze sequence above, CAC, ACA, YBY and BYB suggest a comparative thought process by the subject. We found 3-mers to be both interpretable as well as sufficiently abundant for the analyses in this project. The next step was to investigate whether gaze 3-mers would be predictive of product choice.

Two machine learning classifiers, Random forests (RF) and Support Vector Machine (SVM) were used to predict final plant purchase choice (outcome variable) using participants’ gaze patterns (the relative frequency of 3-mers) and survey data. The RF classifier is an ensemble of multiple decision tree models that select the best set of predictors to partition the data into smaller sets with high within-group homogeneity in respect to the dependent variable [66]. The SVM is another popular machine learning tool for classification tasks, which constructs a set of hyperplanes in a high-dimensional space to categorize various groups. We selected RF and SVM machine learning approaches because they are both well suited for classifying high-dimensional data sets such as the present one, which would be impractical in the context of traditional linear models due to parameter estimation problems. In this study, we conducted the RF classification via the randomForest R package [67] and SVM classifier via the R package.

The RF implementation followed Sundararajan et al.’s [68] approach. First, we used a 4:1 split ratio to randomly divided the data into two non-overlapping sets: (1) a training set and (2) an external testing set. Using the training set only, we employed a fivefold cross validation to identify an optimal model. The randomForest options ntree (5, 10, 15, 25, 50, 75, 100, 250, 500) and nodesize (0.5, 1, 2) were varied, and the best combination of ntree and nodesize values, as determined by the average classification performance for 5-fold internal test sets within the training set, were retained to construct the final model [66, 69]. All other randomForest parameters were set to their default options. The final model was then applied to the external testing set [70]. Meanwhile, we applied similar procedures for SVM classifier with a radial basis kernel. Two hyperparameters within the radial basis kernel, the gamma option (0.1, 0.3, 0.5, 0.7, 0.9, 1, 2, 10) and cost option (0.01, 0.1, 1, 5) were tuned.

The best average classification performance was indicated by: (1) Confusion entropy (CEN), which is a misclassification measure ranging from 0 (optimal classification performance) to 1 (the lowest classification performance); (2) Overall predictive accuracy (OA), which is the ratio between the number of correctly-predicted cases relative to the total number of cases; (3) Average predictive accuracy (AA), which is an auxiliary measure of accuracy that evaluates the average model performance in respect to each category of the dependent variable (see [71, 72]) for a review of the performance indicators); (4) F1-macro score, which is a F1 score (the harmonic mean of the precision and recall) with averaging precision and recall of each individual class; and (5) F1-micro score, which is calculated from individual true positives, true negatives, false positives and false negatives. In the multi-class classification task, F1-micro is also the classifier’s overall accuracy. Reported results are based on the external testing set, a standard practice to increase the generalizability of research findings [70].

Our final models focused on predicting consumer choice for plants selected by at least 10 consumers (top two plants in six-plant displays and top three plants in the 12- and 24-plant displays) and for the no-plant choice. Consequently, the effective sample size ranged from 43 to 92 across displays. Specifically, displays 1–6 had sample sizes with 92, 76, 54, 77, 43 and 56 for classification tasks, respectively. Because eye-tracking technology captures a tremendous amount of data per participant, the effective sample size range was appropriate for analytical purposes with the lower bound (43) being slightly below the average sample size (46.59) of 75 other similar eye tracking studies reported in two meta-analytical articles [73, 74]. The decision to model the most commonly selected plants and the no-plant choice is logically supported by our intention to understand the most prevailing consumer choices as opposed to focusing on rarely selected plants; because this focus helps develop parsimonious RF classifiers; and because our use of independent data subsets for the RF classifier optimization [68] means that a sparse outcome would be undesirable as it could yield an over-specified model with low external validity.

We collected consumer attributes in an online survey following display viewing and product choice. Thus, we constructed four models with various features: Model 1 considers frequencies of consecutive 3-mers without looking away fixations from the gaze sequence (denoted as “3-mer-without-LATD”). Model 3 considers frequencies of consecutive 3mers with looking away fixations (denoted as “3-mer-with-LATD”). Model 2 and Model 4 included consumer attributes from the online survey in addition to the features used in Model 1 and Model 3, respectively. However, not all the features may have been relevant for predicting product choice. Given each model setting, variable selection was performed before modeling by using ANOVA F-test for every numerical feature, in order to extract those with variations among various final purchasing decision/plant. Feature reduction helped to remove irrelevant features, reduce the dimensionality of feature space and reduce over fitting of the trained model.

Results

We examined the most common gaze sequences (3-mers) by plant display. Fig 5 displays the average frequency in which the top ten 3-mers occurred across the participants’ gaze sequences for one display (6 plants, sale price, as shown in Fig 1). This information shows that, irrespective of the number of plants in the display or the plant price, looking away from the products (denoted by the letter Z) is extremely frequent. For instance, for the display with 6 plants, sale price, the 3-mer ZYZ (i.e., looking away, looking at the price signage, and looking away again) occurred 1.72 times for each participant overall, while it occurred 1.8 times for the non-buyers and 1.7 times for subjects who elected to purchase a plant from that display. The percentage of look-away fixations was statistically above expectation ().

Fig 5

Average frequency for top ten 3-mers in display with six plants at the low price.

Next, we evaluated the predictive performance of the eight final RF and SVM classifiers. Only three variables from the consumer attributes contributed meaningfully to the models’ predictive accuracy: expertise, ethnicity (68.5%), and the prior purchase of annual plants (58.7%). The zero-order correlations between three consumer attributes are shown in

First, we examined the performance of model 1. For instance, for a 6-plant, sale-price display, using 3-mers alone resulted in a low entropy (CEN) of 0.3 with 72% accurate predictions (OA), which is ~2.2 times more accurate than randomly guessing product choice (prediction baseline = 1/6). The average accuracy (AA) of model predictions by choice (top two plants and no-plant choice) was 44%. Overall, the predictive performance of all models could be improved for at least one of the metrics by including consumer attributes.

Moreover, we compared the final RF classifier performance when gaze sequences and LATD were employed with and without consumer attributes (models 3 and 4). Table 4 shows the CEN and overall and average accuracies for each model by display type. While CEN was identical for one model (6-plant, non-sale display), it was lower (representing less error) for four of the five remaining models, indicating that the addition of consumer attributes (model 4) enhanced the model’s performance. For two of the six models, overall accuracy was identical with and without the addition of consumer attributes. However, overall accuracy was improved with the addition of consumer attributes for three of the models while it was lower for the model predicting choice in the largest display with a non-sale price. Average accuracy was similar for one model (six-plant, non-sale display) but higher for three of the remaining models (see Table 5). Thus, the preponderance of evidence leads us to conclude that model performance is enhanced by the inclusion of consumer attributes (model 4). The model associated with the sale price and large display has the highest predictive performance across all models when consumer attributes are included, resulting in a very low CEN of 0.08, and 75% of accurately predicted product choices (overall and by choice category), which is approximately three times better relative to the random prediction for a 24-plant display (4%).

Table 4

Comparison of predictive accuracy across all four random forest models.

Display Number	Number of Plants	Price	Confusion Entropy (CEN)				Overall Accuracy (OA)				Average Accuracy (AA)
			Model 1	Model 2	Model 3	Model 4	Model 1	Model 2	Model 3	Model 4	Model 1	Model 2	Model 3	Model 4
1	6	Low	0.3	0.38	0.29	0.27	0.72	0.67	0.78	0.72	0.44	0.42	0.56	0.44
2	6	High	0.64	0.63	0.64	0.63	0.53	0.53	0.53	0.53	0.55	0.57	0.55	0.57
3	12	Low	0.43	0.43	0.25	0.36	0.55	0.55	0.73	0.55	0.54	0.58	0.71	0.54
4	12	High	0.42	0.4	0.36	0.35	0.47	0.53	0.53	0.60	0.35	0.4	0.43	0.53
5	24	Low	0.46	0.39	0.39	0.08	0.38	0.25	0.38	0.75	0.38	0.25	0.38	0.75
6	24	High	0.33	0.28	0.54	0.45	0.55	0.73	0.45	0.36	0.54	0.71	0.46	0.38

Model 1: 3mer-without-LATD;

Model 2: 3mer-without-LATD + consumer attributes;

Model 3: 3mer-with-LATD;

Model 4: 3mer-with-LATD + consumer attributes.

Note: The numerically superior result in each model (lower CEN, higher OA, and higher AA) is displayed in bold.

Table 5

Comparison of predictive accuracy across for random forest and SVM models.

Classifier	Number of Plants	Price	F1-macro score (f1_macro)				F1-micro score (Overall Accuracy) (f1_micro/OA)
			Model 1	Model 2	Model 3	Model 4	Model 1	Model 2	Model 3	Model 4
RF	6	Low	0.25	0.32	0.24	0.30	0.72	0.78	0.67	0.72
SVM	6	Low	0.20	0.20	0.20	0.24	0.67	0.67	0.67	0.67
RF	6	High	0.28	0.28	0.28	0.28	0.53	0.53	0.53	0.53
SVM	6	High	0.16	0.16	0.16	0.16	0.47	0.47	0.47	0.47
RF	12	Low	0.29	0.36	0.28	0.29	0.55	0.73	0.55	0.55
SVM	12	Low	0.17	0.40	0.34	0.34	0.36	0.73	0.64	0.64
RF	12	High	0.18	0.26	0.20	0.30	0.47	0.53	0.53	0.60
SVM	12	High	0.10	0.10	0.10	0.10	0.33	0.33	0.33	0.33
RF	24	Low	0.16	0.19	0.11	0.39	0.38	0.38	0.25	0.75
SVM	24	Low	0.18	0.28	0.40	0.38	0.38	0.50	0.75	0.75
RF	24	High	0.33	0.25	0.39	0.19	0.55	0.45	0.73	0.36
SVM	24	High	0.31	0.08	0.14	0.04	0.55	0.18	0.27	0.09

Model predictors:

Model 1: 3mer-without-LATD;

Model 2: 3mer-without-LATD + consumer attributes;

Model 3: 3mer-with-LATD;

Model 4: 3mer-with-LATD + consumer attributes.

Note: The numerically superior result in each display under either F1-macro or F1-micro score. The higher F1-macro score and the higher F1-micro score (is equivalent to overall accuracy in classification tasks) are displayed in bold.

Lastly, we compared the final resulting RF classifier performance when gaze sequences and consumer attributes are employed with and without LATD information (models 2 and 4). Table 6 shows the results of the model performance with and without LATD gazes. Generally, CEN was lower or slightly lower with LATDs for five of six displays. The exception was for the moderate-sized 12-plant display with a sale price. Overall accuracy was greater or equal when including LATD for three of the six models, and average accuracy had a similar result. Taking all the parameters into account, lower error and similar or higher accuracy when LATD events were included were achieved for five of the six displays with the LATD information (model 4) compared to without (model 2). Our highest predictive accuracy was for the smallest display with a low price (78%).

Table 6

Models with the best performance.

Number of Plants	Price	% of LATD fixation on average among participants	Model 1	Model 2	Model 3	Model 4
			3mer-without-LATD	3mer-without-LATD + Consumer attributes	3mer-with-LATD	3mer-with-LATD + Consumer attributes
6	Low	38%		■ ●
6	High	40%	■ ●	■ ●	■ ●	■ ●
12	Low	35%		■ ●
12	High	33%				■ ●
24	Low	31%			■ ●	●
24	High	33%			■ ●

The square (■) represents the model with the best F1-macro score among 8 model settings (4 models * 2 machine learning method). The circle (●) represents the model with the best F1-micro score (is equivalent to overall accuracy in this classification task) among 8 model settings.

Discussion and conclusion

Our first research question centered on whether features contained within gaze sequences are predictive of product choice. These results provide evidence that breaking down a consumer’s gaze path into short segments (3-mers) of visual fixations can help us to predict product choice. Chance of randomly guessing would be 33% for 6 plant displays (due to 3-class classification), and 25% for all other displays (due to 4-class classification). Our models indicate that consumers’ gaze sequence alone could accurately predict choice from 38% (large display, sale price) to 72% (small display, non-sale price) of consumers, which represents an improvement ranging from four to about nine times relative to the prediction baseline. Russo and Leclerc [43] and Clement [19] documented distinct consumer fixation patterns across the multiple phases of in-store purchase decision process. Our findings advance that knowledge by providing an indication that consumers’ fixation sequences are predictive of choice. However, our results indicate that the predictive power associated with consumer gaze sequence can be improved by the addition of consumer attributes or LATD information in various settings.

Our second research question focused on whether consumer characteristics help to improve the predictive accuracy of gaze paths in product choice. While gaze sequence alone predicts purchase intention with high accuracy (up to 78% overall accuracy in the sale price, 6-plant display), we were able to improve the overall accuracy for simpler displays when consumer attributes (top-down factors) were included. Three top-down factors (expertise, ethnicity and prior plant purchase) enhanced predictive accuracy. Purchase and plant-related activities in the U.S. historically has been an activity participated in largely by more affluent white consumers. This is evidenced in nearly three decades of research conducted by the National Gardening Association (see https://gardenresearch.com/) also documented in Dennis and Behe [75]. However, more recent NGA studies show the gap between white and persons from other ethnic backgrounds closing as participation among non-white consumers increases.

These results also aligns with the findings of Joo et al. [42] that expertise influenced time of visual attention to both intrinsic and extrinsic cues. This effect was evident especially for the larger, non-sale priced display, perhaps suggesting that more cognitive resources are needed when faced with more financially risky decisions (i.e., higher price). Similar to the present study, other physiological measures have been related to purchase choice [68, 76]. Ravaja et al. [76] observed a price effect with more electroencephalogram (EEG) readings, which measure brain activity, for lower-priced products which may have been an indication of further consideration; while there was less brain activity for higher-priced products which may have been dismissive as a potential choice, indicating that brain activity is influenced by product price. Sundararajan et al. [68] used EEG readings to predict choice of 181 students in 10 dichotomous food choices with an average peak accuracy of 84%. Our study used visual attention as the physiological measure. When comparing our results to the aforementioned studies, in a purchase decision task with more choices (24 here versus Sundararajan et al.’s [68] two choice study), our predictive accuracy improved when using gaze patterns in combination with consumer attributes.

In answer to research question 3 as to whether gaze aversion plays a role in product choice, we found that (1) look-aways (LATD) are included in ~70% of the 10 most frequent gaze sequence 3-mers, and that (2) the inclusion of LATD reduced error in 5 of 6 models and produced a somewhat higher predictive accuracy. Therefore, it seems that looking away from product displays is not only a frequent behavior at the POP but is also a meaningful predictor of consumer choice. Other studies acknowledge the importance of looking at the product [48] and the time spent viewing the product [25-29]. Here, we interpret the abundance of look-aways in the gaze paths to suggest that the look-away is also important for cognition and allows reflection, which helps confirm a product choice. The predictive value of including LATDs was greater for displays with more plants and higher prices. This suggests that having a sale price or lower price option does not require as much cognitive effort and that the look-away gazes for these lower priced options do not appear to help finalize the decision to the same extent as higher-priced or bigger displays. This also confirms the importance of bottom-up factors in the choice process.

For the first time to our knowledge, the need to look away to inform the purchase choice (LATD) is documented. Computer data must be processed in order to create output. Here, we see the visual behavioral corresponding to when possible cognition is likely occurring in the look-away gazes and their inclusion in modeling choice has been shown to improve predictive accuracy in product choice. One managerial implication of our findings is that in creating a merchandise display, retailers should add neutral space where consumers' visual attention can focus while higher cognitive processes are involved in the decision-making process.

There are several limitations to laboratory studies including the present study. First, subjects were asked to imagine a purchase decision. Conducting the study in situ introduces the real distractions and motivations for shopping, which would enhance the external validity of a study. Furthermore, the type of product (live plants) may not appeal to all subjects who agreed to participate in the study. Additional studies with related packaged goods (e.g. fertilizer, pest controls, etc.) could provide additional insight with regard to visual assessment of alternatives prior to choice or selection.

This exploratory study invites many avenues for future research. Only one product type (live plants) was investigated; thus, future studies should bring in other product types and include branded merchandise and consumer packaged goods. Future studies should explore the effect of larger and more elaborate displays on gaze sequence and LATD. Additionally, more extensive research is needed to understand the differential functions and the interplay of centrally (LATD) and peripherally [61] gazing at non-relevant visual cues.

13 Jul 2020

PONE-D-20-14107

Seeing through the forest: The gaze path to purchase

PLOS ONE

Dear Dr. Behe,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Reviewers provided a deep and detailed evaluation of the paper. They highlight several critical points and offer some useful comments that might help improving it. In particular, you need to be more convincing about the reliability of the analysis, the applied methods and their choice, and the data quality.

Please submit your revised manuscript by Aug 27 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Claudio Soregaroli

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please modify the title to ensure that it is meeting PLOS’ guidelines (https://journals.plos.org/plosone/s/submission-guidelines#loc-title). In particular, the title should be "specific, descriptive, concise, and comprehensible to readers outside the field" and in this case it is not informative and specific about your study's scope and methodology

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is my review of the manuscript, entitled “Seeing through the forest: The gaze path to purchase.” The authors present an eye-tracking study on 92 participants examining, in a 3 (display size) x 2 (price) within-subjects design, the link between the gaze sequence in which individuals viewed a display and their hypothetical choice of an unbranded plant product. While I am not able to evaluate the authors’ machine learning approach, my expertise lies in eye-tracking research, when applied to business settings. In that sense, I enjoyed reading the present manuscript, and I think the authors have a well-crafted initial submission. Therefore, based on my feedback provided below, my reviewer recommendation is to invite the authors for a minor revision. I start by delineating the larger topical concerns and areas for improvements, after which I continue with some minor suggestions to improve the readability and contribution of this piece.

MAJOR POINTS:

• In connection to the authors’ theorizing and discussion on gaze aversion, I was missing some notes on whether and how this construct is related to, or clearly distinct from peripheral vision, which plays an important role in consumer choice situations. For example, how does the gaze aversion conceptualization differ from the account proposed by Wästlund et al. (2018; references provided in the end of this review)? This is important given that the authors write at several places that consumers often ignore some visual cues.

• While eye-tracking studies in lab settings have their merits, they lack a certain degree of ecological validity. This should be acknowledged in the limitations section of the authors’ manuscript.

• I found the authors’ exposition to be a bit front-heavy, particularly the review of literature section, which I think can be written more succinctly.

• When it comes to the authors’ contribution and significance statements, I think they should make more of an effort in order to not over-sell their findings. For example, they note that “few of the studies investigated the gaze paths in which visual cues were accessed and no studies investigated whether that relationship predicted purchase” (p. 4, last paragraph); that their “goal is to better understand the sequence of cues that help get products off the shelf and into the cart” (p. 5, first paragraph); and imply that they look at actual product choices and purchase decisions. Strictly speaking, however, all these claims are inaccurate, as the authors are solely examining hypothetical choice in a rather artificial lab setting (see my ecological validity point above). Please be precise.

• I think the authors should report a 3 (display) x 2 (price) within-subjects ANOVA on the 11-point likelihood of buying scale, both as a validity check and as a way to potentially boost their managerial implications. Certainly, it would be interesting for a retail owner to know if there is a main effect of display size, such that, for instance, people indicate greater buying likelihood in a display with a larger (vs. smaller) number of product options. Similarly, it would be interesting, as a validity check, to see if participants’ buying likelihood were greater in the presence (vs. absence) of price discounts (with corresponding effect sizes), and to explore whether display size interacted with price to influence participants’ buying likelihood. Please implement these analyses or provide very solid arguments as for why such details are not included (arguments of the type “fell beyond the purpose of our study” is not enough in my view, but I will not hijack the story).

• The authors provide arguments supporting their findings on product expertise and previous annual purchases of plants. However, virtually no arguments at all appear for the finding that race (white) helped improve the model’s predictive accuracy, which left me with the question “why?” Please elaborate.

MINOR POINTS:

• Please specify the design (within-subjects) in the abstract.

• Please correct the claimed non-student sample in the abstract, as the main text in the manuscript seems to suggest that some students were also included in the study.

• For the sales figures presented on p. 2 (final paragraph), the authors may want to have a more recent reference than 2016. Relatedly, they may want to add the fact that retail sales from physical stores still account for 90-95% of total retail sales (e.g., Otterbring & Lu, 2018).

• At p. 4 (first paragraph), I got confused, as the authors start by stating that the number of visual cues at the POP (point-of-purchase) can be OVERWHELMING at times, after which they state that their main goal is to study the relationship between gaze sequences and product choice for MINIMALLY packaged, UNBRANDED products.

• When reviewing visual attention and purchase likelihood, the authors may want to add some recent eye-tracking references using hypothetical choice as the focal dependent variable (e.g., Ballco et al., 2019; Otterbring et al., 2020; Otterbring & Shams, 2019; Tórtora et al., 2019).

• At p. 6 (first paragraph), the authors indicate some inconsistencies in findings, but they do not specify how these inconsistencies are manifested. Please clarify.

• The real, imagined, or implied presence of others tend to influence individuals’ visual attention (e.g., Lie & Yu, 2017; Milani et al., 2019; Wong & Stephen, 2019). Therefore, I think further methodological details are needed in the manuscript with respect to whether an experimenter was present in the same room as the participant(s) during the study and how many individuals that attended the eye-tracking sessions simultaneously.

• Did all the signs displayed in the study use the same color scheme on and around the letters?

• Please clearly indicate the sampling frequency and accuracy of the eye-tracking device used.

• At p. 16 (last paragraph), the authors state that CEN was identical for one model and lower for four of the five remaining models after adding the consumer attributes, after which they state that overall accuracy was identical with and without the addition of consumer data for two of the six models. This seems contradictory – was it one or two models that were not improved after addition of the consumer attributes? Please clarify.

• Please present the zero-order correlations between the included consumer attributes: expertise, being (vs. not being) white, and prior plant purchase.

In sum, I think the authors have an interesting study and my evaluation as a reviewer is that they may have the chance of getting their work published, in case they meticulously address all points raised above. I wish them good luck in their future work on this topic and I hope that my comments were helpful.

References

Ballco, P., de-Magistris, T., & Caputo, V. (2019). Consumer preferences for nutritional claims: An exploration of attention and choice based on an eye-tracking choice experiment. Food Research International, 116, 37-48.

Liu, N., & Yu, R. (2017). Influence of social presence on eye movements in visual search tasks. Ergonomics, 60(12), 1667-1681.

Milani, S., Brotto, L. A., & Kingstone, A. (2019). “I can see you”: The impact of implied social presence on visual attention to erotic and neutral stimuli in men and women. The Canadian Journal of Human Sexuality, 28(2), 105-119.

Otterbring, T., Gidlöf, K., Rolschau, K., & Shams, P. (2020). Cereal Deal: How the Physical Appearance of Others Affects Attention to Healthy Foods. Perspectives on Behavior Science, 1-18.

Otterbring, T., & Lu, C. (2018). Clothes, condoms, and customer satisfaction: The effect of employee mere presence on customer satisfaction depends on the shopping situation. Psychology & Marketing, 35(6), 454-462.

Otterbring, T., & Shams, P. (2019). Mirror, mirror, on the menu: Visual reminders of overweight stimulate healthier meal choices. Journal of Retailing and Consumer Services, 47, 177-183.

Wästlund, E., Shams, P., & Otterbring, T. (2018). Unsold is unseen… or is it? Examining the role of peripheral vision in the consumer choice process using eye-tracking methodology. Appetite, 120, 49-56.

Tórtora, G., Machín, L., & Ares, G. (2019). Influence of nutritional warnings and other label features on consumers' choice: Results from an eye-tracking study. Food Research International, 119, 605-611.

Wong, H. K., & Stephen, I. D. (2019). Eye tracker as an implied social presence: awareness of being eye-tracked induces social-norm-based looking behaviour. Journal of Eye Movement Research, 12(2), 1-17.

Reviewer #2: This study investigates the relation between overt attention and purchase choices (i.e., buying a plant) in a real-world scenario using 3D Tobii glasses. Participants’ gaze positions are monitored while asked to choose one of plant out of different many, and each is tested in six different conditions, which result from a manipulation of set size (6, 12, 24 plants), and the price band (low, high). From the eye-movement responses, they seem to consider the frequency of the first three fixations, and together with survey data of their consumer preferences, train a set of different random forest classifiers aimed at predicting which plant will be chosen. The classifiers seem to display a good accuracy for the small set-size (72%) and their performance improve when trained with survey data and a feature of the eye-movement defined as Look Away to Decide (LATD), which represents fixations outside of any Area of Interest.

It is interesting to read about the application of eye-tracking technology in different disciplines, and I truly appreciate the effort of building computational models based on eye-movement responses to make predictions in real-life contexts. The authors have also provided the data and scripts to run the analyses which is a nice and honourable aim. However, overall, the study falls short on its methodology, theoretical underpinnings and implications for it to make a substantial contribution.

In what follows, I highlight my major concerns and a few minor other comments that the authors may want to take on board to improve their study.

Major comments:

(a) There seems to be confusion about what the theoretical contributions of this work are and especially what their novelty should be. Since the seminal work of Yarbus, 1967, it is widely known that eye-movement responses can successfully predict the task participants perform. The notion of task can be broadly construed to encompass several things among which also the target object that people may be attentively selecting. It is not surprising then, that when considering the first three fixations only (more later about why this may be a methodologically problematic choice) it may be enough to predict what object is receiving most attention, and so correlate with the final choice. So, I find this result, per-se, not novel nor striking. What’s also odd is that the authors claim that looking away from the object is a good predictor of choice, that’s counterintuitive as usually fixation time correlates with the amount of attention that a target got. Furthermore, even in the context of consumer psychology, there are other studies, some of which mentioned by the authors, showing already systematic relations between gaze behaviour and choice.

(b) The other novelty seems to be that the inclusion of LATD and survey data improves the prediction accuracy of the models. However, the predictions of the models presented in Table 1 seem to be erratic across display types, and do not follow any clear theoretical logic as to why that’s the case. In fact, the best accuracy seems to be Model 3 for a display of 6 plants at low price (called J, 78%), which includes consumer attributes. At the same time, we also see that a display with 24 plants and a high price (Q) reaches an accuracy of 73% with Model 2, which is the one with LATD features but not consumer attributes. If both consumer attributes and LATD are included as training features (Model 4), we see an incredible drop in performance for the display Q (36%), while for the display P, also 24 plants and low-price, there is a much better accuracy (75%). The lack of consistency across displays, or between training data and achieved model performance casts some important doubts about the reliability of these classification results, so undermining any theoretical statements related to it.

(c) LATD feature is tentatively argued to be reflecting gaze aversion. But, instead, it could simply relate to data quality and nothing else. In fact, it does reflect fixations falling outside all possible area of interests. I would be really surprised that participants actively engaged in a choice task would spend time looking at no informative areas of the display. Thus, it would be important to quantify how many fixations fall outside the area of interests, whether the percentage of such fixations varies (or it is stable) across participants and displays, etc. In practice, shed better light to the nature of such out of range fixations. In general, there is very little detail about eye-tracking data quality, which is much needed. Also, the argument of gaze aversion does not really fly, in my humble opinion, as the literature this hypothesis refers to involves human-human interaction, while here, it is a very different setting with no-animate agents taking parts beyond the participants.

(c) The study needs to be much better contextualised within the visual cognition literature rather than only using references to consumer psychology. There are few claims or arguments about bottom-up vs. top-down processes, for example, that seem misleading, inaccurate or poorly referenced. I have added a few works on the topic at the end of this review, so that they may give the authors a starting lead to relevant literature. Line 106:108 “Attention drawn to a sudden loud noise, flashing lights, or an attractive person is the result of a bottom-up process, while finding a friend’s face in a crowd is the result of a top-down process.” There should be a mention of what drives bottom-up processes, e.g., low-level visual saliency, and clarify what is meant by top-down. Is it the role of the task, e.g., visual search as opposed to free-viewing? Why is looking for faces top-down, because of the stimuli (i.e., faces) or because of the task? Also, it is not clear how the current study taps into such a dichotomy. In fact, as far as I can see, the experimental design does not systematically manipulate any of the two types of mechanisms. There is a manipulation of set size, referred by the authors as displays, and price, but none of the two is systematically related to any of the results presented. As said in my previous context, the results from the modelling do not show any systematicity with respects to these two independent variables.

(d) There is also a long-standing literature about eye-movement scan-patterns, starting from the seminal paper by Noton & Stark 1971, and about modelling, or measuring similarity, between them. The authors seem not to be fully aware about this bulk of literature, and I think instead that such literature should be better discussed. Also, their analysis of scan-patterns sounds a bit unusual. For what I have understood, from all possible fixations that each participant made during a trial, only the first three were kept, and they used a rather unconventional naming for it (i.e., 3-mers). There is a weak attempt to explain why that was done but it did not fully convince me. I think the authors should show what happened when the scan-patterns were fully considered, or at least show, what happens to the modelling results as they increase the number of data-points from 3 to N. More importantly, from their analysis I gathered that the sequential aspect of scan-patterns was not considered. It seems that the random forest classifiers were in fact trained using relative frequencies of fixating at the different AOI, if that’s the case, then I do not see how this data can be defined as a scan-pattern. Modelling a scan-pattern implies preserving its sequential aspect, and utilising methods that can maintain it untouched, e.g., Hidden Markov Models. They should perhaps refocus their analyses in terms of extracting eye-movement features, and that they decided to consider the first three fixations as those one carrying important information about the task. I also wonder whether they may look for other characteristics of the oculo-motor behaviour, it may well be that there are other types of features, object-based, best fit to their classification task.

(e) The justification of using Random Forest instead of any other methods seems a bit weak. The authors say, Line: 289-291: “We selected the RF machine learning approach because of its many properties for classifying a high-dimensional data set such as the present one, which would be impractical in the context of traditional linear models due to parameter estimation problems.” I am not sure why, for example, Support-Vector machine would not work for their purpose. SVM can be trained on a technically infinite high-dimensional feature space to predict a certain outcome variable. As the features their random forest classifiers were trained on are relative frequencies of looks to AOI and survey data, I guess that SVM would also here. I think, they should better motivate their choice, and show that their method of classification, i.e., random forest, would really give them a better F-score than some other method. By cross validating their predictions using different methods may help them also realise what may be generating variability across displays.

(f) Survey data is used to train the random forests, but it is never explained at length what is the data consisting of, how is it treated, and inputted. For example, I take that each participant contributed to a single data point per questionnaire, but there are many more eye-movement data points per display. It is crucial that the treatment of the data is much better elucidated, otherwise the reader cannot really follow and appreciate what was done in the modelling.

(g) The task itself is severely underspecified. I am still not sure, even after reading the paper a few times, whether participants had always to make a choice to buy something or not. If not, it is not clear how many times they did not make any choice, and how were these trials treated. In general, it is never clear how many trials were used to train the classifiers. Even if the data is provided, there is the need for greater details about the treatment of the data, and better descriptive statistics.

Minor comments:

- It may be useful to give an example of the Area of Interests. There seems to be a lot of overlap between the plants in the display, so I wonder how boundaries are defined. Also, the plants further away in the display may receive less fixations just because of their relative position.

- Improve the quality of the only Figure, which is rather low. Also, it is not clear what this Figure is depicting. I would suggest to add more Figures to aid the understanding of the readers.

- Line 283:284 “The next step was to investigate whether gaze 3-mers would be predictive of product choice. For that purpose, we computed, for each participant, the relative frequency in which each 3-mer occurred in the first 100 gaze events in their gaze sequence.” Not sure I understand this passage. I got that 3-mers are the first 3 fixations in a sequence, how can they be 100 gaze events? How did they derive this number? Are 3-mers, three area of interest? If so, how were these regions chosen if the display contains many more (i.e., a minimum of 7 and a maximum of 25 including out of range fixations)?

- Line 256:257 The authors mention a chocolate bars practice task, but it is never clear why was this done, and what did it entail.

- Line 258:259. The participants have to verbally state how much from 0 to 10 would they like to buy a product. But, was not it a buy vs. no-buy binomial outcome variable that’s modelled? If it is a continuous variable, what are the random forest predicting? A preference score? If so, how is prediction accuracy then measured?

- Line 231:232 “Plants occupied approximately 0.12 m2, 0.25 m2, and 0.45 m2 of display space for the 6, 12, and 24-plant designs, respectively” The authors should provide measurements in terms of degree of visual angle, as the size of objects is relative to the distance of the observer.

Line 202: There seems to be a typo.

Line 309:312 “Our final models focused on predicting consumer choice for plants selected by at least 10 consumers (two plants in six-plant displays and three plants in the 12- and 24-plant displays) and for the no-plant choice.” Obscure passage. This sounds like data dredging. It is unclear how many final data points contributed to the modelling.

Line 323:324 “To determine which attributes had a strong impact on the plant choice, variable selection was performed using a series of F-tests.” The authors now mention feature selection, but it was never clear how many features were originally selected, and what were F-tests based on.

Table 1: There is what seems an arbitrary list of letters to identify the displays. Not sure what a reader could make of it.

Line 364: There is a mention of baseline, but it is not clear what the baseline is, and how it is computed.

Line 400:403 There is a mention to EEG research but it feels quite tangential and off-topic.

References

Coco, M. I., & Keller, F. (2014). Classification of visual and linguistic tasks using eye-movement features. Journal of Vision, 14(3), 11. https://doi.org/10.1167/14.3.11

Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transaction on Pattern Analysis and Machine Learning, 20(11), 1254–1259. https://doi.org/10.1109/TPAMI.2012.125

Noton, D., & Stark, L. (1971). Scanpaths in eye movements during pattern perception. Science, 171(3968), 308–311. https://doi.org/10.1126/science.171.3968.308

Rothkopf, C. A., Ballard, D. H., & Hayhoe, M. M. (2007). Task and context determine where you look. Journal of Vision, 1410, 1–20.

Simola, J., Salojärvi, J., & Kojo, I. (2008). Using hidden Markov model to uncover processing states from eye movements in information search tasks. Cognitive Systems Research, 9(4), 237–251. https://doi.org/10.1016/j.cogsys.2008.01.002

Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136. https://doi.org/10.1016/0010-0285(80)90005-5

Yarbus, A. L. (1967). Eye Movements and Vision. Plenum Press.

Wolfe, J. M., & Utochkin, I. S. (2019). What is a preattentive feature? Current Opinion in Psychology, 29, 19–26. https://doi.org/10.1016/j.copsyc.2018.11.005

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

26 Aug 2020

We have responded to all reviewer comments in the file labeled "Response to reviewers". Thank you.

Submitted filename: Response to Reviewers-PONE-D-20-14107R1.docx

Click here for additional data file.

22 Sep 2020

Seeing through the forest: The gaze path to purchase

PONE-D-20-14107R1

Dear Dr. Behe,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Claudio Soregaroli

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a responsive revision. I have no further comments and I recommend acceptance of the manuscript.

Reviewer #2: The authors have done a very thorough and careful job at addressing my major concerns. I feel that the addition of the new SVM analysis, and especially the clarification of the k-mers fixation sequences as rendered the paper much clearer, hence strengthening its theoretical contributions.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Moreno I. Coco

25 Sep 2020

PONE-D-20-14107R1

Seeing through the forest: The gaze path to purchase

Dear Dr. Behe:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Claudio Soregaroli

Academic Editor

PLOS ONE

Table 2

Two-way repeated measures ANOVA on percentage of looking-away fixations.

Factor	Df	F-value	p-value
Display_size	2	30.75	3.13e-12*
Price	1	0.948	0.333
Display_size * Price	2	2.641	0.074

Table 3

Correlation between the included consumer attributes.

	Expertise	Ethnicity	Prior Plant Purchase
Expertise	1.00	0.35	0.47
Ethnicity		1.00	0.38
Prior Plant Purchase			1.00