Dynamic weighting of feature dimensions in visual search: behavioral and psychophysiological evidence.

Dimension-based accounts of visual search and selection have significantly contributed to the understanding of the cognitive mechanisms of attention. Extensions of the original approach assuming the existence of dimension-based feature contrast saliency signals that govern the allocation of focal attention have recently been employed to explain the spatial and temporal dynamics of the relative strengths of saliency representations. Here we review behavioral and neurophysiological findings providing evidence for the dynamic trial-by-trial weighting of feature dimensions in a variety of visual search tasks. The examination of the effects of feature and dimension-based inter-trial transitions in feature detection tasks shows that search performance is affected by the change of target-defining dimensions, but not features. The use of the redundant-signals paradigm shows that feature contrast saliency signals are integrated at a pre-selective processing stage. The comparison of feature detection and compound search tasks suggests that the relative significance of dimension-dependent and dimension-independent saliency representations is task-contingent. Empirical findings that explain reduced dimension-based effects in compound search tasks are discussed. Psychophysiological evidence is presented that confirms the assumption that the locus of the effects of feature dimension changes is perceptual pre-selective rather than post-selective response-based. Behavioral and psychophysiological results are considered within in the framework of the dimension weighting account of selective visual attention.

Introduction

The selection of sensory data that are relevant for the control of behavior and thinking is a key cognitive ability and models of selection constitute a core component of cognitive theories. The notion that the neural activations representing a limited set of visual features in the brain are modulated by a process prioritizing a circumscribed part of the visual field has turned out to be a powerful account of selective attention. In her seminal Feature Integration Theory (FIT), Treisman (1988; Treisman and Gelade, 1980) proposed that visual objects are selected by a mechanism involving two consecutive processing stages. On an initial stage, representations of visual attributes or features are created. Features of a set of independent visual dimensions color, orientation, motion, size, etc. (referred to, by Wolfe and Horowitz, 2004, as attention-guiding attributes) are coded independently of each other in a topographic fashion. The generation of coherent representations of objects requires the allocation of the focus of attention to a specific spatial location in order to integrate visual features coded at the same location of the separate dimensional representations. Focal attention binds feature representations into object files that are compared to object descriptions stored in long-term memory for object recognition and identification. Feature binding and comparison occurs in a serial fashion, that is, object files are processed in turn.

Going beyond FIT, in the Guided Search (GS) model of selection, Wolfe (1994; see also Wolfe et al., 1989) proposes that entry-level feature coding feeds into the generation of a saliency representation of the visual scene. The saliency representation is conceived of as a stimulus-based (bottom-up) map of activations (peaks and troughs), which reflect the conspicuity, or differentness, of each feature at a given spatial location relative to their surroundings. The focus of attention is guided by the strength of the saliency activation, with the highest peak being attended first, the next peak second, etc. Visual information within the focus of attention is gated through to mechanisms of object recognition and response selection. Visual information within the focus of attention is made available to a processing stage of limited capacity that essentially processes one object at a time.

According to Treisman and Gelade (1980, p. 98), “features are registered early, automatically, and in parallel across the visual field.” Similarly, Wolfe (1994, p. 202) states that GS “distinguishes between a pre-attentive, massively parallel stage” of feature processing which is followed by “a limited-capacity stage that performs other, more complex operations.” In both models, assuming a dichotomy between parallel and serial processing phases, a capacity-unlimited stage of parallel feature processing prepares the scene for capacity-limited serial processing.

In the present paper we review recent behavioral and electrophysiological evidence demonstrating that dimension-based mechanisms mediating the generation of saliency signals, by drawing on limited processing resources, selectively modulate the relative strength of dimension-based saliency representations; consequently, ensuing selection processes are also modulated. The Section “Priming in Pop-Out Search” briefly introduces findings showing that search for a conspicuous target is expedited when the feature or dimension that defines the target is repeated across consecutive experimental trials. In the Section “Dimension Weighting” the Dimension Weighting account (DWA, Müller et al., 1995, 2003), a model accounting for the dynamic modulation of search performance is presented and empirical evidence in its support is discussed. The Section “Locus of Dimension-Based Modulations: ERP Evidence” considers the debate on whether dimension-based weighting occurs at perception-based processing stages prior to selection by focal attention or at response-based stages following selection. The argument here draws on the patterns of event-related electro-cortical potentials that, in previous electrophysiological studies, have been shown to reflect stimulus-based and response-based processing. Overall, behavioral and electrophysiological findings provide converging evidence in support of the assumption of a pre-selective locus of dimension-based dynamics as proposed by the DWA.

Priming in Pop-Out Search

Search for a stimulus that differs from distractors by one salient feature is efficient and independent of the number of distractor items present in a search array. For example, a red vertical bar presented among green vertical bars is detected seemingly effortlessly. Analysis of the search reaction time (RT) function relating the time it takes to discern the presence vs. the absence of an odd-one-out item in the array to the number of distractor items (set size) shows that RTs do not increase with increasing set size: Independently of how many green bars there are in the display, the red bar is detected efficiently – phenomenally, it “pops out” of the display. Interestingly, it has been shown that increasing the number of distractors can expedite search. Bravo and Nakayama (1992) had participants decide on the shape of a diamond target; in detail, they had to indicate whether the diamond’s left or right edge was cut off. The target was the only red diamond among green distractor diamonds or the only green among red diamonds. When the assignment of target and distractor colors changed randomly across experimental trials, the increase of the number of distractors resulted in expedited RTs. The finding is consistent with the concept of a bottom-up saliency representation. Adding distractors increases the strength of the feature-based saliency activation. Note that the shape judgment, a discrimination task, requires the selection of the target by focal attention; other tasks in the study of Bravo and Nakayama (1992) involving the mere detection of the presence of a feature that do not require focal attention were not expedited when set size was increased (see also, Turatto et al., 2010). The decision that no target is present is made almost as rapidly as a target-present decision, that is, there is hardly a difference between target-present and target-absent RTs. The phenomenon of pop-out constitutes the core evidence for the assumption that feature extraction and processing are automatic in nature (e.g., Treisman, 1988).

The automaticity assumption was called into question by Maljkovic and Nakayama (1994, 1996). They showed that the RT in a task requiring the detection and processing of a target item that differed from distractors by its color was affected by whether the color of the target item (and associated with it, that of the distractor items) was repeated or changed across consecutive trials. In more detail, search items were red or green diamonds with the target item being the uniquely colored item. Repeating the target- (and distractor-) defining color resulted in a RT benefit on the current trial compared to the preceding trial(s), whereas changing the target color (i.e., swapping it with that of the distractors) across trials incurred an RT cost. Maljkovic and Nakayama’s (1994, 1996) findings demonstrate that, at variance with the automaticity assumption, processing of information within a visual dimension (color) is subject to modulation by variations of the stimulus. Given the additional finding that RTs were apparently not modulated by pre-knowledge of the upcoming target feature, Maljkovic and Nakayama (1994, 1996) interpreted their findings in terms of the concept of implicit memory of the target-defining feature yielding “visual priming”; as a result, repetition of the target-defining feature yields faster processing on the current, compared to the preceding, trials (but see also, Leonard and Egeth, 2008).

Dimension-based modulations in feature search

Müller et al. (1995) investigated the effects of presenting observers with pop-out targets defined, within a given block of trials, either within a single dimension (i.e., orientation) or varying across dimensions (i.e., orientation, color, size). Distractors in all conditions were small vertical gray bars. Targets were right-tilted, left-tilted, or horizontal bars in the within-dimension condition, and right-tilted, black, or large bars in the cross-dimension condition. Analyzing search RTs to the right-tilted (orientation) target presented in both conditions revealed the RTs in the cross-dimension condition to be significantly slower than in the within-dimension condition (see also Treisman, 1988). That is, variability of the target-defining dimension within a block of trials incurs a substantial RT cost relative to the target features varying within a constant dimension.

Based on these findings of cross-dimension RT costs, Found and Müller (1996) used inter-trial analyses in a feature pop-out task to further examine whether it is the target dimension or feature on the preceding trial n − 1 that critically influences search performance on the current trial n. In their Experiment 1, distractor items were green vertical bars; targets were either red or blue vertical (color targets) or green left-tilted or right-tilted bars (orientation targets). Observers’ task was to indicate, as quickly and accurately as possible, whether or not an odd-one-out item was present in the search array (see Figure 2, left-hand panel, for an example of a search display). Two types of change were compared to repetitions of the target feature: feature-based (within dimension) and dimension-based (across dimension). Results showed that changing the target-defining dimension across trials (e.g., color on trial n − 1 → orientation on trial n) incurred a RT cost relative to a repetition, across consecutive trials, of the target-defining dimension (e.g., orientation on trial n − 1 → orientation on trial n). By contrast, changing the target-defining feature across trials (e.g., left-tilted on trial n − 1 → right-tilted on trial n) did not incur any cost compared to repeating the feature (e.g., right-tilted on trial n − 1 → right-tilted on trial n), neither for the color nor the orientation dimension.

Figure 2

Examples of search displays used in the empirical studies discussed in the present review. Left-hand panel: Singleton feature search. The target item differs from the distractors by its color or orientation (shown) in non-redundant target trials or by color and orientation in redundant target trials; distractors all have the same color and orientation. Right-hand panel: Compound search task. The to-be-detected target differs from distractors by its color (shown) or orientation; the response is defined by the position of the gap cut into the target item at the top or bottom part. Distractors all have the same color and orientation. Cuts are located in the upper and lower parts, respectively, in half the items. Spatial locations of all the search items are randomly shifted relative to center of the cell of a virtual grid underlying the spatial arrangement of the search display. Note that the number and spatial arrangement of the display items varied according to the research issue under investigation. Note that the present example displays are not drawn to scale.

Significant RT costs on dimension change relative to repetition trials were also observed in a task in which the response required knowledge of the target feature (e.g., red, left-tilted: response 1 vs. blue, right-tilted: response 2). Separate analyses of orientation and color trials showed that there were no additional costs associated with changes of orientation features while there was some evidence that feature changes incurred a cost in the color dimension. There were additional costs associated with the requirement to access the level of feature representations (see also Müller et al., 2004).

The absence of feature-based inter-trial transition effects in color and orientation trials of Experiment 1 and the orientation trials of Experiment 2 in the Found and Müller (1996) study is at variance with the findings of significant feature change vs. repetition effects in a number of studies. As stated above, Maljkovic and Nakayama (1994) observed benefits of feature repetition as opposed to feature changes within the color dimension, which they interpret in terms of bottom-up priming. More recently, Leonard and Egeth (2008) also found stimulus-driven RT benefits of feature repetitions within the color dimension. Furthermore, they also showed that foreknowledge of the target feature expedites search RTs, providing evidence for independent mechanisms of bottom-up and top-down enhancement of individual features in search tasks. The contrasting results give rise to the question of how the divide can be resolved theoretically. The color dimension is at the core of the divergent findings. Accordingly, Found and Müller (1996) suggest that the color dimension might be conceived of as comprising a series of sub-dimensions coding colors such as red, green, and blue. They base their proposal on theories of color perception assuming the perceived colors are the result of a comparison process, which in turn could be likened to the computation of saliency signals. Importantly, provided the proposal is accepted, the mechanisms of the DWA would apply within the color sub-dimensions. Further research is required to provide empirical support for the hypothesis.

The results of dimension-based effects in feature detection tasks require a modification of models of visual search to reflect the dynamics at the level of dimension-based processing. The model proposed by Müller et al. (1995, 2003; Found and Müller, 1996) is discussed in the following section (see Figure 1 for an illustration of the functional architecture of the DWA).

Figure 1

Illustration of functional architecture proposed by the DWA (Müller et al., . The search display contains a target differing from distractors by color. Target and distractor features are registered in dimensionally organized feature maps. Dimension-specific saliency maps are computed separately for each dimension. The dimension-specific saliency signals are then integrated, in a weighted fashion, into an overall-saliency map, which supports both detection responses and the allocation of focal attention. Saliency signals are computed and integrated separately for each stimulus location.

Dimension Weighting

Müller et al. (1995; Found and Müller, 1996) interpreted their findings in terms of a DWA of search. Similar to GS (Wolfe, 1994), the DWA assumes that saliency activity integrated across separate dimensions (i.e., a supra-dimensional saliency map) signals the presence of a target in the search array and guides the allocation of focal attention. In contrast to GS, the DWA assumes that the dimension in which the target is defined is tracked by a mechanism that weights dimension-based feature contrast signals by allocating a limited-capacity resource (attentional weight). As the total amount of weight is limited, an increase of weight assigned to one dimensional module (e.g., orientation) entails a reduction of the weight assigned to other modules (e.g., color, etc.). The distribution of attentional weights established on a given trial episode (i.e., a number of sequential trials) persists across trials, at the least the weight established on a particular trial persists into the next trial episode. Shifting attentional weight between dimensions is achieved in a time-consuming process. Therefore, if the target on the current trial n is defined in the same dimension as that on the preceding trial n − 1, no weight shift is required and RTs are faster compared to a trial sequence that, following a change of the target-defining dimension, requires attentional weight to be shifted. The finding that a change of the target-defining feature across trials does not affect RTs relative to a repetition of the feature suggests that attentional weighting modulates the output of dimension-based saliency computations. Stated differently, weighting occurs prior to the integration of the dimension-based saliency activations by the overall-saliency representation. In sum, the results so far suggest that the strengths of dimension-based saliency activations are modulated by dynamic shifts of processing resources (attentional weights) to potential target-defining dimensions.

In the remainder, the research conducted to empirically examine the characteristics and predictions of the DWA is discussed. One key issue refers to the question whether the allocation of processing resources may be controlled or modulated by top-down foreknowledge of the dimension on which the target is defined in the upcoming experimental trial. Note that, in contrast to features, dimensions are abstract entities: Whereas features are used to describe objects such as for example “a red apple,” statements such as “a colored apple” do not convey any useful information. Both FIT (Treisman and Gelade, 1980; Treisman, 1988) and GS (Wolfe, 1994) propose that top-down modulation is at the level of the representation of concrete features; the question whether dimension-based processing mechanisms are susceptible to top-down modulation is in part motivated by the nature of these feature-based processes.

Top-down modulation of attentional weight

Dimensional weighting (e.g., as demonstrated by Found and Müller, 1996) seems to be largely driven by stimulus characteristics, and weight shifting does not require any conscious effort on the part of the observer (e.g., Müller et al., 2004). But does (semantic) pre-knowledge of the dimension of the upcoming target influence search performance? Müller et al. (2003) investigated the issue by presenting participants, before the onset of the display in a pop-out search task, with a symbolic cue. The cue indicated, with high validity, the dimension in which the target on the imminent trial was defined. In more detail, in their Experiment 1, targets were defined in either the color or the orientation dimension. A symbolic cue, the word “color” or “orientation,” was presented at the start of each trial to pre-cue the likely target dimension, with a cue validity of 80%. As an example, following the pre-cue “color,” that target was color-defined in 80% of the trials (valid trial) and orientation-defined in 20% of the trials (invalid trial). RTs to search displays preceded by valid and invalid dimension cues were compared to a baseline condition in which the cue word “neutral” was presented. Results showed that RTs were significantly faster on valid relative to neutral trials, and significantly slower on invalid relative to neutral trials. To more closely examine whether the effect indeed reflects weighting at the level of dimensions, rather than features, in another experiment, cue words (“red,” “blue,” “left,” “right”) indicating the exact target feature were presented prior to search display onset. For example, the cue “red” meant that the target was defined by the feature red in 79% of (valid) trials and the features blue, left-tilt, or right-tilt on each 7% of the (invalid) trials. The DWA assumes that cues, independently of whether they indicate a specific feature (e.g., red) or a dimension (e.g., color), primarily bias the weighting to favor any signals defined in the encompassing dimension (i.e., in the example, color) and only secondarily to a specific feature value within this dimension (e.g., red, blue). In line with this prediction, an invalid feature (e.g., blue) within the cued dimension (i.e., color), following the cue “red” produced shorter RTs compared to targets defined in the invalid (orientation) dimension.

These results show that dimensional weighting is susceptible to (top-down) modulation on the basis of pre-knowledge of the upcoming target, at least to some extent. Converging evidence for this position was provided by a comparison (in Experiment 1) of the inter-trial (dimension repetition/change) effects on valid and invalid trials relative to those on neutral trials. As expected, dimension-based effects were observed in the neutral condition: RTs were significantly slower on dimension change than on repetition trials (feature change did not affect RTs). The size of this effect was significantly reduced, compared to the neutral condition, on both valid and invalid trials. Thus, a cue that, for example, validly predicted the (changed) target dimension on a dimension change trial significantly reduced the RT cost that is typically incurred by the change (feature changes were not affected). A similar reduction was evident on invalid compared to neutral trials; this time, however, the reduction was due to a relative slowing of RTs on dimension repetition trials. In other words, the RT slowing occurred when the cue was misleading, incorrectly predicting that the target would be defined in a changed dimension, when it was actually defined in the same dimension as on the preceding trial. While this pattern provides strong evidence that top-down (knowledge-based) weighting processes can modulate dimensional weight settings established (bottom-up) in response to the stimulation, the top-down influence is limited: dimension change/repetition effects remained evident even when the pre-cue was 100% valid.

The findings of Müller et al. (2003; see also Zehetleitner et al., 2011) cuing study show that dimension-based processing modules can be modulated by semantic foreknowledge of the target item in a dynamic (trial-by-trial) fashion. A related question is whether a top-down attentional set affects dimension-based processing for an extended time range such as the duration of an experimental block or an entire experiment. The question is of theoretical interest because continuous weighting of a relevant dimension relative to irrelevant dimensions might constitute a strategy to avoid capture by stimuli defined on an irrelevant dimension (cf. Bacon and Egeth, 1994). The issue was investigated in a series of three studies that are discussed in the following section.

Longer-term effects of dimensional attentional set on search performance were demonstrated in a series of studies concerned with the phenomenon of “attention capture.” Theeuwes (1992) had shown that the presence of a salient singleton color distractor affects RTs if observers search for a (somewhat less salient) singleton form target; distractors that are less salient (e.g., form) than the target (e.g., color) do not affect RTs. Theeuwes (1992) argued that the more salient stimulus automatically captures attention and that, when the distractor is more salient than the target, RTs to the target are increased by capture preventing the direct allocation of focal attention to the target. In Theeuwes’ (1992) experimental setup, observers were instructed to respond to the form singleton (in which case the color singleton is the distractor) or to the color singleton (in which case the form singleton is the distractor) and to ignore the respective other singleton. Vis-à-vis the DWA, the question arises why observers appear unable to down-modulate the weight of the “salient” irrelevant dimension so as to eliminate, or at the least reduce, the adverse effect of a singleton defined in this dimension on performance. Müller et al. (2009) addressed this question in a study in which they systematically varied the proportion of distractors presented in a given block of trials. Observers’ task was to identify a form target while ignoring a (more salient) color distractor. Within a given block, color distractors were present in 0, 20, 50, 80, or 100% of the trials; further, in the very first trial block, distractor proportion was either 0 or 100%, providing observers with either no practice or with extensive practice to deal with distractors. The results showed that distractor interference is differentially modulated by both the initial condition (0 vs. 100% distractor trials) and the likelihood of a distractor occurring. Generally, distractor interference was high for observers who had never encountered a distractor before (0% initial condition) and when distractor probability was low; by contrast, interference was substantially reduced, if not completely eliminated, when distractor probability was high, in particular for observers who had experienced ample initial exposure to distractors. Müller et al. (2009) argued that the amount of distractor interference observed in the different conditions is explained by a strategy of (top-down) suppressing the distractor dimension. How efficiently this strategy can be or is applied depends on experience with distractors generally (prior practice) and the incentive to consistently operate this (effortful) strategy (which is high when distractor probability is high). Consistent with the latter, when the incentive is low, the first distractor after a no-distractor trial causes maximum interference, because the occurrence of the first distractor leads to the recruitment of top-down control. Interference is greatly reduced on the next trial if it also contains a distractor. Taken together, these findings suggest there does exist a mechanism capable of (actively) down-modulating the effects of salient stimuli defined in an irrelevant dimension (see also Geyer et al., 2008).

Sayim et al. (2010) recently showed that the ratio of distractor trials also affects the proportion of saccades directed to an additional onset distractor presented at a location not occupied by a distractor. The observers of Sayim et al. were instructed to execute a saccade to the one of six disks (presented on the circumference of an imaginary circle) that changed its color while ignoring the onset of a salient distractor disk. Distractors were presented on 0, 20, 50, 80, or 100% of trials. Results show that the proportion of saccades going to distractors significantly decreased as distractor proportion increased (from about 2/3 of all saccades in the 20% to about 1/5 of saccades in the 80% distractor condition). This finding could be taken to suggest that modulation may occur at early processing stages before the generation of cortical representations. The block-wise variation of the likelihood of distractor ratios used by Sayim et al. (2010) limits the generality of the finding somewhat because potential effects of different inter-trial contingencies might affect performance. Recently, Moher et al. (2011) demonstrated that the adjustment of the oculomotor system in response to foreknowledge occurs very rapidly on a trial-by-trial basis. In detail, Moher et al. had participants execute a saccade to a shape target while, in a proportion of trials, a color distractor appeared. At the start of each trial, a cue reliably indicated the likelihood (low: 10%, high: 90%) of a distractor. The results show that on high-likelihood trials, costs associated with the presentation of a distractor were significantly reduced compared to low-likelihood-trials. In sum, Moher et al.’s findings can be taken to suggest that the top-down modulation is able to respond to current conditions in a highly flexible manner.

Locus of dimension-based effects: Behavioral evidence

The following section is dedicated to the question whether dimension repetitions and changes across trials affect selection at the level of response selection mechanisms. The issue came up because some researchers (e.g., Maljkovic and Nakayama, 1994, 1996 in their shape judgment studies or Theeuwes, 1991, 1992 in his capture studies) used compound search tasks and some argued that the dimension-based effects arise at the level of response selection (e.g., Mortier et al., 2005; Theeuwes et al., 2006). Whereas in simple feature (pop-out) search tasks observers simply discern target presence vs. absence, in compound tasks the detection-critical and response-critical features of the target are dissociated; that is, detection and response features may vary independently across trials (see Figure 2, right-hand panel, for an example of a compound search display). As in feature search tasks, the target differs from the distractors by a salient feature, for example, its color or orientation, allowing for efficient target detection. An additional, response-relevant feature, for example, the tilt of a line presented within a square or a gap near the top or bottom end of a bar, defines the required response (e.g., left-/right-hand key press for left-/right-tilt or upper/lower gap). Note that, as a rule, a target is present on each trial in compound tasks. Although changes of the target-defining dimension were also shown to affect response decision making in compound search tasks, the dimension-based inter-trial effects are greatly reduced in compound search relative to feature search tasks (Krummenacher et al., 2002b; Theeuwes et al., 2006). Theeuwes et al. (2006) had taken the reduction to suggest that the locus of inter-trial effects is post-selective, at the level of response selection processes – rather than pre-attentive, at the level of saliency signal computation, as proposed by the DWA (Müller et al., 1995, 2003). However, there exist more parsimonious alternative explanations consistent with the DWA; these are discussed below.

In the study of Krummenacher et al. (2002b), observers were presented with an array of colored squares; the target differed from distractors either by its color (e.g., being the only red among green items) or orientation (e.g., being the only square rotated by 45° relative to the orientation of the distractors). The response was defined by the orientation of a line presented within the target item (lines were also presented within the non-target items): a left-/right-pointing line required a left-/right-hand key press. While dimension-based inter-trial effects were reduced in this task, there was also an asymmetry between color and orientation trials: dimensional effects were observed for color-defined targets, but not for orientation-defined targets. The authors argued that this pattern was owing to the requirement to switch attention from the target- to the response-defining dimension: as the response dimension was invariably orientation, observers would always need to weight the response-defining (orientation) dimension on any given trial n, irrespective of the dimension of the target-defining feature. Therefore, in the subsequent trial (n + 1), weight would be on the orientation dimension and the dimension-based processing stage would be biased toward processing orientation-defined targets. Accordingly, dimension change effects from color-defined to orientation-defined targets tended to be small while they were large when the target was color-defined requiring the shift of attentional weight from the orientation to the color dimension.

Another important aspect contributing to the reduced magnitude of cost/benefits of dimension-based changes/repetitions in compound relative to feature detection tasks was revealed by an analysis of the change of the target-defining feature contingent on the change of the response-defining feature (see, e.g., Müller and Krummenacher, 2006). Trial sequences involving changes were compared to repetition sequences. The results revealed an interaction between the two factors: there were costs associated with a change relative to a repetition of the target dimension from trial n − 1 to trial n, but only when the response feature remained the same; when the response feature changed, dimension change costs were abolished. This pattern explains the general reduction of dimension-based inter-trial effects in compound relative to feature search tasks. Although the two types of change, of the target- and response-defining features, are statistically unrelated, the processing system behaves as if they were correlated. The target-defining attribute (color, orientation) is available to the system before the post-selective stage of (focal-attentional) processing at which the response-defining attribute (orientation) is extracted. If the target attribute on the current trial n is the same as on the preceding trial n − 1, the system behaves as if it assumed that the response-defining attribute would also remain the same – thence, an unchanged response is facilitated, and a cost is incurred when the response attribute changes. If the target attribute changes, any response priors are discarded and processing starts from scratch, that is, processing on the current trial n is unaffected by processing on the preceding trial n − 1. In their explanation of the contingencies between target-defining and response-defining features, Müller and Krummenacher (2006), referring to a similar line of reasoning by Kingstone (1992) (see also Hillstrom, 2000), argue that the linkage between selection-relevant and response-relevant features might arise because it is easier to abandon both of these parameters than to abandon one of them while maintaining the other one.

A further aspect that contributes to the reduction of dimension-based inter-trial effects in compound as opposed to feature search relates to the nature of the processing required to solve the tasks. Krummenacher et al. (2009) compared RTs and inter-trial effects in feature and compound search tasks. In the feature search condition, observers indicated whether a color (red, blue) or orientation (left-tilted, right-tilted) target bar was present in an array of green vertical distractor bars; in the compound condition, the target also differed from distractors by its color or orientation, but the response was defined by whether a gap was positioned near the top or the bottom of the target bar (see Figure 2). The results showed that inter-trial effects in the feature detection condition were dimensional in nature and largely uninfluenced by cross-dimensional variations of the spatial location of the target in the search array. Conversely, in the compound search condition, the effect of dimension changes was markedly reduced while there was a significant effect of the distance between the locations of the target on the previous and on the current trial: RTs to the target on trial n increased linearly with increasing distance from the target on trial n − 1. Krummenacher et al. (2009) interpreted this finding as suggesting that dimension change/repetition effects are modulated by the amount of attentional focusing required by the task, with space-based attention altering the integration of dimension-specific feature contrast signals at the level of the overall-saliency map: When the task does not, or only minimally, require focal-attentional processing of the target (as in simple detection tasks), dimension-based inter-trial effects can operate across the whole search array; by contrast, when focal-attentional analysis of the target is required (as in compound search tasks), dimension-based effects become confined to a narrow region around the target location (the focus of attention).

Krummenacher et al. (2010) went on to show that dimension-specific and feature-specific effects of the target definition on the preceding trial on search RTs to the target on the current trial constitute a marker of the level (depth) at which the target was processed. Processing at the level of dimensional saliency signals entails dimension-based, but not feature-based, inter-trial effects, whereas processing at the feature level gives rise to feature-based inter-trial effects (in addition to dimension-based effects). Krummenacher et al. (2010) presented observers with search displays containing either a color (red or blue vertical bar) or orientation (green left-tilted or right-tilted bar) target embedded in an array of homogeneous distractors (green vertical bars) or a single target or distractor item that was presented in isolation. When search arrays were presented, repetition or change of the exact feature defining the target on the preceding trial n − 1 would not affect the RT on the current trial n because participants responded on the basis of the overall-saliency activation signaling the presence of a target. By contrast, when the item was presented in isolation a feature change incurred a RT cost because observes processed items at the feature level in order to assign the item to the target or non-target category.

In summary, the studies discussed above show that the magnitude of the dimension-based costs associated with cross-trial alterations of the target is modulated by task demands. Compound search tasks as compared to feature detection tasks give rise to effects that are related to response processing such as the requirement to shift weight to the response-relevant feature or the abandonment of settings from the previous trial if the target dimension changes. Further, compound searches require focalization of attention as reflected in space-based inter-trial effects. Feature-based effects are observed when processing of individual features is required to perform the tasks. While some of these effects can be explained by feature-based attention (see, e.g., Egeth et al., 1984; Bichot et al., 2005; Maunsell and Treue, 2006), the DWA is able to theoretically integrate the full picture of findings obtained in the above studies.

Feature integration

Analyses of inter-trial effects revealed benefits and costs associated with repetitions and changes of dimensions across trials in feature search tasks that can be explained by the DWA (Müller et al., 1995, 2003). However, as the vast majority of objects in the real world are defined by multiple rather than single feature differences, inter-trial effects give rise to the question whether the integration of features also draws on the processing resources involved in feature detection.

Krummenacher et al. (2001, 2002a) addressed the issue by examining the mechanisms underlying the integration of dimension-based saliency signals into the overall-saliency representation guiding focal attention, adapting the so-called redundant-signals paradigm (see, e.g., Mordkoff and Yantis, 1991) to feature search. In their adapted paradigm, search performance for feature targets differing from distractors in one dimension (e.g., color or orientation: a red vertical bar or a green right-tilted bar presented in an array of green vertical bars) is compared to performance in search for targets differing from distractors in two dimensions (e.g., color and orientation: a red right-tilted bar among green vertical bars). In principle, integration of dimension-based saliency signals into the overall-saliency representation may be serial, parallel, or parallel co-active in nature. Analyses of mean RTs and RT distributions can be employed to differentiate between the three possible integration mechanisms. On a serial account, dimension-based signals are integrated in a sequential fashion, resulting in increased RTs for targets differing from distractors in two dimensions compared to just one dimension. On parallel accounts, mean RTs are determined by the faster of the two single-definition RTs. If the response is based on a parallel-race between signals, overall mean RTs to targets defined in two dimensions are expected be somewhat faster than the overall mean RTs to targets defined in a single dimension, because the response is always elicited by the faster of the two signals. Stated differently, slow RTs in the on average faster dimension are compensated for by fast RTs in the on average slower dimension. Parallel races produce a RT benefit or RT redundancy gain that is referred to as “statistical facilitation” (Raab, 1962). However, the amount of mean RT redundancy gain compatible with the assumption of a parallel race of signals is bound by properties of the (probability) distributions of RTs to single-dimension targets. Miller (1982) proposed a distribution-based procedure to test whether RT gains are compatible with the assumption of a parallel race, the “race model inequality” (RMI): P(RT < t|C&O) ≤ P(RT < t|C + P(RT < t|O). The RMI states that the probability P of having responded to a target defined in two dimensions (e.g., color C and orientation O) at a given point in time t after display onset [P(RT < t|C&O)] must be smaller or equal to the probability of having responded to a color target at time t after onset [P(RT < t|C)] plus the probability of having responded to an orientation target at time t after onset [P(RT < t|O)]. Violations of the RMI then provide evidence for parallel co-active processing of signals; that is, signals are integrated into a common representation before activating subsequent processes such as response selection and execution.

Krummenacher et al. (2002a) had their participants respond to feature targets defined in the color, the orientation, or both the color and orientation dimensions. Distractors were always green vertical bars; the target could be a red or blue vertical bar (one-dimension color target), a 45° left or right-tilted bar (one-dimension orientation target) or a bar that was red and left-tilted, red and right-titled, blue and left-tilted, or blue and right-tilted (two-dimension color and orientation target). The results showed that RTs to targets defined in two dimensions were significantly faster than RTs to targets defined in one dimension [RTs to two-dimension targets were also significantly faster than RTs to targets of the faster of the two single-dimensions (color)]. Importantly, the distribution-based analyses showed violations of Miller’s (1982) RMI that provides evidence for parallel co-active integration of dimension-based saliency signals into the overall-saliency representation.

Krummenacher et al. (2002a) further investigated whether multiple feature signals are integrated if they are presented at spatially different locations. Observers were presented with single (e.g., a red vertical bar or a green left-tilted bar) and dual (e.g., a blue vertical and a green right-tilted target) feature targets among green vertical distractors. Search RTs of four dual target distance conditions were compared to RTs of single target conditions. Different-dimension (color and orientation) and same-dimension (color and color; orientation and orientation) dual signals were presented either at (horizontally, vertically, or diagonally) neighboring display locations (distance d1), locations separated by one distractor item in between two targets (d2), or separated by two distractor items (d3). Additionally, different-dimension dual signals were presented at the same (d0) location (i.e., a single red and left-titled target, as in the paradigm described in the previous paragraph). Results showed that (i) RTs to single targets defined in two dimensions (d0) and RTs to dual targets (d1 to d3) were significantly faster than RTs to single targets defined in one-dimension only; (ii) RTs to dual same-dimension targets were slower than RTs to dual different-dimension targets, and both were slower than RTs to single two-dimension targets. Testing for violations of Miller’s (1982) RMI showed parallel co-active processing of dimension-based signals in the single and dual different-dimension conditions, however, in the latter condition violations were observed only when the spatial distance between the two (different-dimension) target signals was small (d0 and d1). By contrast, same-dimension dual targets, though producing significantly faster mean RTs than single targets, are processed in parallel-race mode. Consequently, the finding suggests that co-active signal integration requires the presence of saliency activation in more than one dimension.

Detailed analyses of RMI violations demonstrate that co-active integration of dimension-based saliency is modulated by the topographic location of dimensional saliency signals. Presentation of dual targets at neighboring locations (d1) produces mean RT gains together with violations of Miller’s (1982) RMI, a result that provides evidence for parallel co-active signal processing; presentation of dual targets with one or two distractor items in between (d2, d3) only yields mean RT gains without RMI violations, suggesting a parallel race of signals. The pattern of co-active vs. parallel processing is explained by the assumption that saliency activations of dual targets presented at the same (d0) or neighboring (d1) locations are integrated (summed) into a single overall-saliency representation while saliency activations of dual targets at larger distances generate separate representations at the level of the overall-saliency representation. Further, dual targets defined on the same dimension, while producing RT gains, do not violate Miller’s (1982) RMI; that is, dual signals defined on the same dimension are processed in parallel-race mode.

In sum, processing of single and dual targets defined on the same or different dimensions each generate specific RT patterns that in turn can be explained by the spatial distribution of saliency activation. Results showed that in trials with multiple target signals, mean search RTs to both different- and same-dimension redundant targets were significantly faster than mean RTs to single targets (compared to the faster of the two single dimensions). Analyses of RT distributions using Miller’s (1982) RMI revealed that different-dimension dual target trials are processed in co-active activation mode, same-dimension dual targets, though significantly faster than single targets, are processed in a parallel race.

Krummenacher et al. (2002a) had argued that redundancy gains arise at the pre-selective processing level at which dimension-based bottom-up saliency activations are integrated into an overall-saliency representation before focal attention is allocated. However, some researchers claimed that co-active integration is post-selective, that is, that redundancy gains arise the response selection or execution stage. Krummenacher et al. (2002a) addressed the issue by examining whether RTs to redundantly defined targets are expedited if observers have pre-knowledge of the location where the target is highly likely to appear relative to when there is no advance knowledge; in other words, they investigated whether feature integration depends on the previous allocation of focal attention or whether integration occurs pre-attentively, before the allocation of focal attention. The authors used a symbolic pre-cue – an arrow presented prior to the onset of each search display – to indicate, with a validity of 80%, the display quadrant in which the target item would appear. The arrow instructed observers to shift the focus of attention to the indicated quadrant. The results showed that the pattern of RT gains was unaffected by whether the cue validly or invalidly indicated the quadrant containing the target. However, overall RTs in valid-cue trials were significantly faster than RTs in invalid-cue trials. Krummenacher et al. (2002a) interpreted the finding to suggest that the mechanism underlying dimension-based signal integration is not affected by whether it takes place under focal attention or not.

Overall, the results strongly suggest that dimension-based saliency signals are integrated before focal attention is allocated to a particular location of the search display. In addition, the findings show that (saliency) information from the slower of the two dimensions (in single dimension trials) is able to further increase the processing speed of the faster of the two dimensions, when two signals are presented. This finding provides converging support for the assumption that dimension-based signals are integrated into a common activation, in a co-active fashion, before the attentional focus is allocated to a display location (i.e., independent of focal attention; but see Cohen and Feintuch, 2002; Feintuch and Cohen, 2002).

Even though the behavioral results discussed above provided strong evidence in support of the assumption that redundancy gains arise at a pre-selective processing stage, the use of event-related potentials (ERPs) of the electroencephalogram (EEG) is an approach that is ideally suited to decide between the perceptual pre-selective and post-selective response-based accounts. ERPs measure changes in the electro-cortical potential associated with specific stimulus or response events.

Locus of Dimension-Based Modulations: ERP Evidence

The DWA (Müller et al., 1995, 2003) assumes that the allocation of limited attentional resources (i.e., dimensional weighting) occurs at an early pre-selective (or perceptual) stage of processing. The assumption of pre-selective weighting has been challenged by researchers postulating that the dimensional inter-trial change and repetition effects arise at the stage of response preparation and/or execution. Cohen and colleagues (e.g., Cohen and Feintuch, 2002) proposed a Dimension Action (DA) model that, akin to GS (Wolfe, 1994) or the DWA, assumes dimension-based processing modules. In contrast to GS and DW, the DA model claims that each dimensional module has its own response selection devices. Dimension-based response selection is operated by a winner-takes-all process that excludes all but one of the mutually inhibitory dimensional activations with the winner activating the response execution stage. In an empirical investigation of the DA model, Cohen and Shoup (1997) used a variant of the flanker paradigm (e.g., Eriksen and Eriksen, 1974) in which the target and distractor stimuli activate multiple dimension-based response units (associated with different effectors) in parallel. In detail, Cohen and Shoup (1997) presented observers with a central target stimulus that was flanked by two distractor stimuli. In an exemplary experiment, the target required a left-hand response when it was a right-tilted red bar or a left-tilted green bar and it required a right-hand response when it was a right-tilted green or a left-tilted red bar. Flanking stimuli could be compatible or incompatible with the central stimulus. Cohen and Shoup (1997) argue that competition between responses elicited by the irrelevant incompatible flanking and the relevant central stimulus is resolved by allocating focal spatial attention to the task-relevant central stimulus. According to the DA model, RT redundancy gains observed in conditions in which relevant and irrelevant stimuli activate the same response, compared to the conditions in which relevant and irrelevant stimuli activate different responses, occur at a post-selective stage. In order to contribute to the resolution of the debate and to produce evidence in favor of the DWA or the DA model, the pattern of ERP signatures associated with perceptual-selective as opposed to response-related processes were examined in a series of studies the main findings of which are presented in the following.

Gramann et al. (2007) examined the (non-lateralized) N2 component of the ERP following dimension changes vs. dimension repetitions in search for color and orientation feature targets to identify brain-electric correlates of the weight shifts associated with dimension changes. The anterior N2 had been associated with the detection of pop-out targets in visual search (Luck and Hillyard, 1994) and Gramann et al. hypothesized that dimension changes may be reflected by N2 modulations. The results showed that the N2 amplitude at anterior electrode sites was enhanced in dimension change compared to dimension repetition trials in tasks that required the mere detection of a target or the identification of the target-defining dimension. Gramann et al. (2007) argue that the N2 modulation mirrors the detection of a dimension change and the initiation of the attentional weight shift. Their interpretation relies on the results of imaging studies by Pollmann and colleagues (Pollmann et al., 2000, 2006; Weidner et al., 2002; Pollmann, 2004) that revealed a fronto-posterior network of cortical areas involved in dimension-based weight shifting. Although modulations were also observed in other ERP components (P3, slow wave), the authors concluded that results are consistent with the assumption that dimension change costs are caused by perception-related processes.

Töllner et al. (2008) analyzed the modulation of ERPs that are directly linked to perceptual and response-related processes, namely the (lateralized) N2pc and the lateralized readiness potential (LRP) in a compound search task that allows independent changes or repetitions of the search-relevant (color, form) and response-relevant features (line orientation). The N2pc is an enhanced negative-going deflection over posterior visual brain areas of the hemisphere contralateral to the location of an attended stimulus with a maximum in the N2 time range. The N2pc has been interpreted as reflecting the allocation of focal attention on the basis of perceptual stimulus attributes (Luck and Hillyard, 1994; Eimer, 1996) and it can be taken to mark the end of pre-attentive sensory coding; therefore, N2pc latencies indicate the speed of pre-attentive processing. The LRP can be tied to the time of the stimulus onset or the response. The stimulus-locked LRP reflects processing prior to the activation of a specific response while response-locked LRP mirrors the time taken by the production of the response. If dimension-based processes are pre-selective in nature, the N2pc component is modulated, if they are post-selective the LRP component is modulated by dimension change vs. dimension repetition across consecutive trials.

Observers in Töllner et al.’s (2008) study searched for color (e.g., red among green) or form (e.g., square among circle) targets; the response was determined by the orientation (horizontal, vertical) of lines running through the objects. The results showed that, irrespective of the (motor) response, dimension changes were reflected in shortened latencies and enhanced amplitudes of the (stimulus-locked) N2pc component. Analyses of the response-locked LRP revealed that, irrespective of dimension changes, motor response changes were accompanied by enhanced amplitudes of the LRP. The response-locked LRP is generally interpreted as mirroring preparation or activation of a motor response (e.g., Smulders and Miller, 2011) and, on account of the way it is computed, is agreed to cancel out any perceptual and/or cognitive processes. In sum, the ERP results demonstrate that dimension changes are reflected in an early N2pc modulation while response changes are mirrored in a later LRP modulation. This finding provides further evidence for a pre-selective mechanism of dimension-based effects. However, as the LRP results show, a later, response-based contribution to the effect cannot be ruled out.

Recently, Töllner et al. (2011), also in an ERP study, used the redundant-signals paradigm to obtain further evidence to establish the locus, perceptual vs. response-related, of dimension-based effects. The approach was mainly based on Krummenacher et al.’s (2001, 2002a) findings, discussed above, suggesting that redundant dimensional signals are integrated before the activation of response processes. Feintuch and Cohen (2002), challenging the pre-selective interpretation by Krummenacher and colleagues, claimed that redundancy gains arise at the stage of response rather than perceptual processing. Using a feature identification task in which participants were required to respond if one of three possible color or/and one of three possible orientations was present, Feintuch and Cohen (2002) observed redundancy gains only if focal attention was directed to two visual objects both of which were defined by a target feature (Recall, however, that Krummenacher et al., 2002a – at variance with the finding of Feintuch and Cohen, 2002, and also Miller et al., 2009 – had shown that integration of dimensional signals in search for redundantly defined targets is independent of the allocation of focal attention.) Therefore, Töllner et al. (2011) examined ERP components to provide further evidence for the assumption that dimension-based signals are integrated before the response processing stage. The authors again focused on the N2pc and LRP components asserting that N2pc modulations reflect pure processing at the perceptual level and LRP modulations pure processing at the motor level. Analysis of the ERPs demonstrates that both amplitude and latency of the N2pc to redundantly defined targets differ significantly from the N2pc wave of color and orientation targets. No difference between waveforms whatsoever was observed in the response-locked LRP. Therefore, the ERP study by Töllner et al. (2011) provides clear evidence for an early locus of dimension-based redundancy gains.

The visual perceptual locus of redundancy gains was confirmed in an ERP study by Grubert et al. (2011) using a similar experimental paradigm as Töllner et al. (2011). Importantly, in addition to replicating Töllner et al.’s findings, Grubert et al. further showed that the occurrence of behavioral and ERP redundancy gains is modulated by top-down task set. Grubert et al. (2011), in a variant of the redundant-signals paradigm, instructed participants to respond only if the odd item in the display was defined on one of the two possible target-defining dimensions (color or orientation) while an odd item defined on the respective other dimension could be ignored and did not require a response. Target items differed from distractors on the relevant dimension or (“redundantly”) on both the relevant and the irrelevant dimensions; that is, while the relevant (i.e., target) dimension required a response, the irrelevant (i.e., non-target) dimension did not require a response. The ERP results show reliable N2pc components to odd items defined on the target and the non-target dimension; importantly, however, the amplitudes of the N2pc to (odd-one-out) items defined on the non-target dimension were significantly reduced compared to items defined on the target dimension and to items defined on the target and the non-target dimensions. Grubert et al. (2011) argue that the finding can be taken to suggest that the stimulus-based saliency signal that is generated at the perceptual processing level is top-down modulated by the task set.

As the psychophysiological studies discussed above employed the lateralized N2pc component of the EEG it could be argued that the dimension-based effects revealed in the search experiments are contaminated by aspects of space-based processing. Zhang and Luck (2009) produced evidence that can be taken to disprove this objection. They showed that feature-based attention can influence feed-forward sensory activity, as reflected by the P1 wave of the EEG. Similarly, Gramann et al. (2010) demonstrated dimension-based modulations of the P1 component. In Zhang and Luck’s (2009) experiment, a continuous stream of intermixed red and green dots was presented in one (e.g., the left) visual half-field. Observers attended to either the red or the green dots in that half-field to detect occasional luminance decrements in the attended color. To test the assumption that feature-based attention can influence feed-forward sensory processing and that the modulation is independent of spatial attention, red-only or green-only probe arrays were presented in the other (e.g., the right) half-field. If the task-irrelevant probe arrays of the unattended half-field elicit a larger P1 wave when they appeared in the attended, as opposed to the unattended, color, the P1 modulation would constitute convincing evidence for a (space-independent) attentional influence on feed-forward sensory processing. Results show that after about 100 ms post-stimulus amplitudes of the P1 component over the cortex contralateral to the probe were significantly larger for probes presented in the attended (rather than the unattended) color. That is, visual signals can be modulated at processing stages that are activated briefly after stimulus onset by a non-spatial mechanism.

Concluding Remarks

In summary, the behavioral and electrophysiological results discussed in the present review suggest that the processing of dimension- or feature-based information is dynamically modulated by a mainly stimulus-driven process of dimension weighting. The mechanism underlying dimension weighting is located at a pre-selective, perceptual processing stage, and dimension weighting can be modulated by symbolic knowledge such as cues and task set.

In terms of the brain systems that might be involved in the dimension-based modulations, a number of potential candidate structures were proposed in the literature. In an account that integrates extensive psychophysical and neurobiological evidence, Fecteau and Munoz (2006) proposed a distinction between saliency and priority representations. Saliency refers to bottom-up processing of search items by spatially selective neurons that do not encode particular visual characteristics. Priority combines saliency with the relevance of a particular stimulus with respect to the observer’s goals. Fecteau and Munoz (2006) argue the saliency and priority are represented by the structures of the oculomotor system with the superior colliculi and the frontal eye field constituting the core components. Bisley and Goldberg (2010) suggest that the parietal cortex, in particular the lateral intraparietal area (LIP), combining bottom-up sensory and top-down intentional information, represents saliency (see also, Gottlieb et al., 1998). Similarly to Fecteau and Munoz (2006), Serences and Yantis (2006) reject the concept of a single saliency representation made by psychological models in favor of the assumption of multiple representations associated with the structures of the oculomotor system.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.