A Facilitatory Effect of Perceptual Incongruity on Target-Source Matching in Pictorial Metaphors of Chinese Advertising: EEG Evidence.

Using evoked response potentials, we investigated the implicit detection of incongruity during target-source matching in pictorial metaphors of Chinese advertising. Participants saw an image of a product (the target in a visual metaphorical relationship), and then made a same-different judgment in response to a second image (the source in a visual metaphorical relationship) which was (in)congruous to the first image in terms of shape and/or function. We collected behavioral (button-press reaction time and accuracy), and neural (N270, delta and theta band activity) measures. The time-frequency analysis showed faster processing of incongruous visual information. Moreover, shape and conceptual incongruity were associated with increased N270 amplitude as well as delta (1-3 Hz) and theta (4-8 Hz) band power. Noticeably, compared with conceptual incongruity, shape incongruity evoked a larger N270 amplitude and stronger delta and theta band oscillation. In addition, the average topographical analysis revealed a frontal and central distribution of the power activity. The analysis of attitudes towards the advertising metaphor pictures also proved the supportive role played by incongruity. In conclusion, incongruity facilitates target-source matching in pictorial metaphors of Chinese advertising. The findings obtained from the study are important to metaphor designs of advertising pictures. Copyright:

Introduction

Pictorial Metaphor and Incongruity

Pictorial metaphor, defined as depicting an object in terms of a different kind of object to which it bears a resemblance in form and function (e.g., flight path depicted as highway, (Dent-Read et al., 1994)), involves seeking similarities between targets and sources (Forceville, 1994). However, in the context of advertising, the image of the target product (e.g., a laptop) is designed to be unexpectedly incongruous to the source item (example.g., a towel), creating a novelty to capture consumers’ scarce attention, which is termed artful deviation by McQuarrie and Mick (1996).

(In)congruity in advertising is a (mis)match between an element of the advertising stimulus (e.g., product, brand, endorser, music, or any execution element in an ad) and the existing set of semantic meanings that one holds about the stimulus. For example, in the pictorial metaphor in which a laptop is represented on a hanger where a towel is generally supposed to appear, incongruity occurs in what participants stereotypically perceive as an electronic product. The degree of (mis)match between the representation (e.g., attributes) of an object (e.g., product) and the related set of semantic meanings determines the intensity of incongruity. This can be measured by the similarity between the two stimuli and its influence on cognitive response (e.g., Meyers-Levy & Tybout, 1989; Peracchio & Tybout, 1996; Stayman & Smith, 1992; Sujan, 1985). It is noted that incongruity in advertising pictorial metaphors does not always facilitate cognitive processing. On one hand, Kaplan (1992), in a conceptual analysis of form and content in visual metaphors, found that incongruity can capture attention. Mick (1992) also observed that visual metaphors are more frequently recalled than nonmetaphors. On the other hand, far too complex an incongruity may leave one unable to understand the intention of an advertisement (McQuarrie & Mick, 1992; Tourangeau & Sternberg, 1981) because the positive effect of incongruity lies in the audience’s successful resolution of it. There have been a number of postulations to explain the contrasted results, such as the influence of individuals’ schema (Mandler, 1981). The curvilinear model (McQuarrie & Mick, 1996) predicts the incongruity effectiveness on the basis of an interaction between visual complexity and message effect. The tipping point effect (van Mulken et al., 2014) suggests that in ratings of liking a product, moderate complexity of incongruity in pictorial metaphors is comprehended and appreciated the most. The studies were performed to assess the effectiveness of incongruity application because it is necessary to identify when and how incongruity is detected and processed, and the nature of the response it is likely to evoke.

Cognitive incongruity has been assessed within many dimensions, such as color (Wang et al., 2004), number magnitude (Li et al., 2003), and spatial matching (Mao & Wang, 2008; Yang & Wang, 2002). The most important processing that moderates the cognitive outcome in these studies is matching comparison, operationalized by the degree to which the compared dimensions are incongruous to each other. The electrophysiological findings typically relating to the incongruity of the processing of visual metaphor are those by Scannella et al. (2016), who claimed that N270 is sensitive to the conflict strength. In the context of advertising, Szubielska et al. (2012) found that the amplitudes of information conflicting components (N270, P300, and N400) were sensitive to incongruity between the product category and the previously presented brand. Ma et al. (2016) examined the N300 for the crossmodal incongruity in the processing of product pictures and their verbal names. However, the studies of cross-category conflict strength marked by perceptual and conceptual representations remain less explored, except those performed by van Weelden et al. (2011) and Hu et al. (2014). They consistently found, in their behavioral experiments of button-press response, that shape facilitates metaphorical processing in the visual modality. However, paradigms using evoked response potentials (ERPs) remain less often employed to further explore the specific neural mechanism underlying the processing of incongruity based on perceptual and conceptual features. This is of great significance to the working-memory study of visual stimuli in advertising because the typical observation time for print advertising is two seconds in an editorial context (Pieters & Wedel, 2004). Establishing the neural changes of a brief exposure to visual advertising stimuli can contribute to the development of both cognitive studies and marketing research of pictorial metaphors.

Metaphor comprehension is an analogical mapping process that involves selection of specific representations of the referent (also known as source domain) in the metaphorical relationship (Gernsbacher & Robertson, 1999), including perceptual (e.g., shape) and conceptual (e.g., function) representations. From a psycholinguistic point of view, perceptual representation expresses semantic meaning, indicating an object’s physical state as well as a relationship with other contextual elements (Levinson & Levinson, 2003). Research indicates that perceptual representation plays a more salient role in picture processing at the very early stage and conceptual representation—at a later stage (Langley et al., 2008; Snodgrass et al., 1996). Loftus and Ginn (1984), in a picture-memory paradigm using noise masks (defined by luminance and attention demand) that either immediately followed the offset of a 50 ms-duration picture or 300 ms after offset, found that perceptual and conceptual processes are two qualitatively different psychological processes operated at two consecutive time epochs, with an interval of roughly 300 ms dividing them. One of the perceptual representations, shape, is documented to prime object recognition (Langley, 1996). In contrast, conceptual primes influence a later stage of processing, after object recognition (Loftus & Ginn, 1984). According to Keil (1992), after shape-based object recognition, an item is likely to be assigned to a corresponding conceptual category based on its function. More than lists of perceptual features, these function categories are based on physical interaction (experience) with the object. It is claimed that two similarly-shaped objects tend to be categorized to the same conceptual domain (Glenberg, 1997; Ortiz, 2010; Rosch et al., 1976; Zwaan et al., 2002; van Weelden et al., 2011). This view is supported by Ortiz’s study (2010), where she examined the relationship between shape and function in pictorial advertisements. In this regard, perceptual representation has a significant influence on cognitive processing, compelling for its early processing and facilitating role.

Incongruity and Evoked Potentials

The mental processing of visual information incongruity can be detected by evoked potentials, such as N2, N270, P300, and N400. In consideration of the differences of paradigm and modality in previous studies, we decided to focus on the N270, based on a careful comparison of these components. Firstly, N2 is elicited in stimulus feature-relevant (matched) conditions (Smid et al., 1999). However, the present study employed a same-different judgement (SDJ) experiment. Secondly, N400 is mostly studied in verbal contexts with an incongruous ending (Kutas & Hillyard, 1980), using semantic, phonological, or orthographic priming paradigms (Kuperman et al., 1995; Rugg, 1985). Thirdly, N300 is image-specific and elicited in semantically incongruous conditions such as between-category mismatches (Federmeier & Kutas, 2002; Hamm et al., 2002; Franklin et al., 2007) and discourse-level mismatches (West & Holcomb, 2002). The N300 amplitude is reportedly larger in processing metaphorical than literal sentence (Balconi & Amenta, 2010), but we assume that in the pictorial modality, the involvement of the visual pathway will facilitate the processing of metaphorical comprehension (Cao et al., 2018). As N270 can be evoked in different stimulus conflicting conditions, it is reasonable to propose that early N270 would reflect the mental conflict when the brain accesses incongruous information, perceptual and/or conceptual.

In particular, N270, a negative event-related potential (ERP) component with peak latency of about 270 ms, was elicited on the human scalp when the second stimulus (S2) of a pair showed any different attributes from the first one (S1) in a visual S1-S2 matching task (Wang et al., 2000; Cui et al., 2000; Kong et al., 2000; Zhang et al., 2001; Yang & Wang, 2002). According to Wang et al. (2000), N270 can be evoked not only in physical feature discrimination tasks but also by the conflict of mental mismatching. Some may argue that this N270 effect is similar to mismatch negarivity (MMN). As far as the current study is concerned, visual N270 and visual MMN are not related. N270 is activated frontally when observed for the analysis of attention and semantic processing, while, in general, visual MMN is elicited temporooccipitally. Thus, visual N200 and visual MMN involve two different processing mechanisms.

Note that there are two common methods for the SDJ paradigm, with two stimuli presented simultaneously or sequentially. Due to the temporal difference of perceptual and conceptual processing and the memory function of pictorial advertising stimuli, sequential presentation is used in the present study. No template mismatch effect could occur, as the perceptual and conceptual features were the two interactive and both task-relevant dimensions in the experiment.

Incongruity and Time-Frequency

In addition, different frequency bands activity underlying evoked potentials can indicate some processing that the ERP may not fully display. The current study also examined neural oscillations (delta and theta) for their significant role in human cognition (Siegel et al., 2012). Delta oscillatory responses, defined as EEG oscillatory responses in the 0.5–3.5 Hz frequency range, also gain importance in studies in which a cognitive load is presented. This cognitive load, in turn, is correlated with attention, perception, learning, and memory. Delta oscillatory responses are usually increased in the frontal and central locations during cognitive load (Baslar & Stampfer, 1985; Güntekin & Başar, 2016; Stampfer & Baslar, 1985). Moreover, theta oscillations (4–8 Hz) have been consistently documented to play a major role in visual working memory functions (Liang et al., 2017; Liang et al., 2012; Sauseng et al., 2010). Generally, theta power increases in response to semantic or syntactic violations and is mainly distributed in frontal areas of the brain (Bastiaansen et al., 2002; Davidson & Indefrey, 2007; Hald et al., 2006).

According to the above, the current study, following the experimental procedure introduced by van Weelden et al. (2011), aims to provide electrophysiological evidence of time-domain and timefrequency measures, underlying incongruity detection that occurs to the target-source matching in the visual metaphors. To effectively reveal the contributions of time-frequency components to the evoked potentials, the time-principal component analysis (t-PCA) method was applied for time-domain analysis and Morlet Wavelet Transform method for time-frequency analysis (Zhang et al., 2017).

The EEG data were recorded while participants viewed a target product picture (target domain), followed by an item picture (source domain) which was (in)congruous with the product picture. To evaluate the responsiveness of the indicators of information conflict to the incongruity salience, a 2 × 2 (Shape [congruous, incongruous] × Concept [congruous, incongruous]) experiment was designed for S2 in the SDJ task. It is reasonable to make the following hypotheses for the processing of visual advertising metaphors based on incongruity intensity: (a) the reaction time (RT) will be shorter for the perceptually incongruous than congruous picture pairs; (b) the incongruity caused by shape differences and conceptual differences will be detected by an enhanced N270 and an increased delta and theta band power; and (c) the shape incongruity will be stronger compared to conceptual incongruity, and therefore, lead to a larger N270 effect and stronger delta and theta power.

Methods

Subjects

Twenty-seven healthy graduates (18 women; age 19-29 years; Mage = 23.81 years) from Dalian University of Technology participated in the experiment. All subjects except one were right-handed and had normal or corrected-to-normal vision and were without history of psychiatric or neurological disorders, as established by self-report. The EEG data of four subjects were excluded because they retained fewer test trials after artifact removal. This study was carried out in accordance with the Declaration of Helsinki, and written informed consent was secured from all participants. The protocol was approved by the Biological and Medical Ethics Committee of the Dalian University of Technology.

Experiment Stimuli

Picture stimuli were presented at the center of screen with a white background, a viewing distance of 90 cm on a 17 in. CRT monitor (1044 pixels × 900 pixels, 85 Hz refresh-rate, 5.31 ° × 5.31 ° of visual angle). All the stimuli were realistic pictures and outlines were retained by Adobe Photoshop 13.0. Colors were removed because of the metaphorical implications colors might have. Finally, picture stimuli were all in black-and-white color, presented against a black/white background. The S1 of a stimulus pair was always a picture of target product (for example, a laptop), followed by the S2, a picture of regular item as a referent extracted from a metaphorical relationship (for example, a towel). The stimulus pairs were divided into five conditions: (a) S2 was exactly the same as S1 (the same); (b) S2 was congruous to S1 both in shape and concept (S+C+); (c) S2 was incongruous in shape but congruous in concept (S-C+); (d) S2 was congruous in shape but incongruous in concept (S+C−); (e) S2 was incongruous to S1 both in shape and concept (S-C−). We used 18 experimental picture sets, each containing one target product object. These picture pairs resulted in 90 experimental trials. We added 54 filler trials (with the target and the paired counterparts being the same) to counterbalance the probability of YES and NO answers in the sequential SDJ task.

The picture stimuli were selected based on the following procedure: 232 advertisement pictures with two metaphorically related items visually represented were initially collected from China’s markets of various media. As familiarity has a modulating influence on metaphor process ing (Schmidt & Seger, 2009), a preliminary test with a five-point Likert scale ranging from 1 (the least familiar) to 5 (the most familiar) was carried out among 30 graduates in Dalian University of Technology, China, who did not participate in the later ERP experiment.

Stimulus examples.

Eighteen least familiar pictures (about 8% of the initial stimuli) were retained, for which the targets and sources were identified and extracted, covering the top sellable product categories (cars, electronic appliances, personal care, food and beverage). Source object in each picture was categorized into one of the four conditions (S+C+, S+C−, S-C+, S-C−) based on ratings made by 11 postgraduates. For the three uncovered conditions, extra items referring to the same product category in metaphors were attempted for matching based on the degree to which they resemble to the target picture in shape and function. Then, manipulations of object shape and function were respectively pretested by subjective ratings among 40 undergraduates and 37 undergraduates in Dalian University of Technology, China, who did not participate in the later ERP experiment (p < .01).

Procedure

Each trial of stimulus pictures and detailed timing were presented as follows (see Figure 2.): fixation cross onset for 1 s , S1 picture onset for 200 ms, 50 ms mask and 1 s blank white screen. After that, the S2 picture appeared and remained on the monitor until response or for 1500 ms. Prior to beginning the experiment, participants were instructed to judge whether the two pictures were the same using their right index finger to press J on the keyboard for yes and left index finger to press F for no, as quickly and accurately as possible.

Firstly, participants practiced 16 trials repeatedly for a minimum of 70% accuracy rate before taking the experiment. There were 720 trials in the experiment, 90 trials for each condition. To balance yes and no responses, 54 extra pictures were used and evenly allocated in the same condition. For these stimuli, S1 pictures and S2 pictures were the same, leading to yes responses. As such, there were 360 trials for both yes/no responses, a balanced probability to avoid response bias. In consideration of the recall pictures in all five conditions being the same with probe picture as the only variation, the extra trials data were not analyzed. Every 48 trials, there was a break of 20 s, after which participants could decide whether to resume the experiment or to continue the break.

In order to further differentiate the effect of perceptual and conceptual processing in terms of the cognitive outcome, a questionnaire was designed and delivered to the participants, which contained 20 sets of the experimental pictures in four conditions: S+C+ (25%), S-C+ (25%), S+C− (25%), and S-C− (25%). A sentence was presented on top of the pictures, indicating “This is an advertisement about XXX.” Then following the questionnaire by van Mulken et al. (2014), four questions were created to measure the attitudes towards the advertising metaphor picture. The answers used a 7-point Likert scale, ranging from 1 (strongly disagree) to 7 (strongly agree). The four rating statements included: 1. I have a good feeling about this advertisement; 2. I will consider buying this product; 3. I will recommend this product to my friends; 4. This product may be good for me.

The experimental procedure of the trial.

Recording and Processing of EEG Data

Continuous EEG signals were recorded by a 64-channel amplifier (500 Hz sampling rate, ANT Neuro EEGO Inc., Germany) via an elastic cap, containing 64 unshielded and sintered Ag-AgCl electrodes, with all of the electrodes laid out according to the International 10–20 electrode system. Data were re-referenced offline to the average of the mastoids. For monitoring ocular movements and eye blinks, electroculogram (EOG) recordings were taken from four electrodes placed lateral to each eye, as well as above and below the right eye. Electrode impedance was kept below 10 kΩ. EEG preprocessing was conducted in Matlab using custom functions, built-in functions, and the EEGLAB toolbox (http://sccn.ucsd.edu/eeglab). The analysis included the use of a 50 Hz notch filter to remove line noise, as well as IIR filter-based filtering (0.5 Hz high-pass filter, 30 Hz low-pass filter), segmentation of the filtered continuous EEG into single trials (each trial was extracted offline from 200 ms pre-stimulus onset to 800 ms post-stimulus onset), baseline correction using the 200 ms preceding cue onset, and artifact rejection of amplitudes exceeding ± 100 μV. Blink artifacts were attenuated by conducting a temporal ICA decomposition on the individual recordings (extended Infomax algorithm in EEGLAB). Trials were superposition-averaged under the same stimulus of the same subjects. Artifactual components were identified by a visual inspection of the component time-course, topographic distribution, and frequency spectrum, and were removed from the back reconstructed time-course (Bridwell et al., 2018; Delorme & Makeig, 2004).

Wavelet filtering was then applied to filter the averaged data (lv = 9; wname = ['rbio6.8']; kp = [2 3 4 5 6], Cong et al., 2010; 2015). The data were processed by wavelet filtering and the components of interest were separated by t-PCA before being projected back. Twenty components were retained (total variance explained was 97.41%), and the first and twelfth components were selected from the 20 components for back-projection according to the properties of interest component in time and space domains, so that the total averaged waveform and brain topographic maps were obtained (Dien, 2010; 2012; Dien et al., 2005).

We focused on the dynamic oscillation of frontal delta and theta after S2 was presented. A complex Morlet continuous wavelet transform (CMCWT, Demiralp et al., 2001; Kronland-Martinet et al., 1987; Tallon-Baudry et al., 1996) was used for time-frequency analysis of the averaged ERP data in the MATLAB toolbox (Bridwell et al., 2018). The parameters of central frequency and bandwidth were both 1, with exploring frequencies ranging from 1 to 30 Hz in steps of 0.3 Hz. EEG and behavioral data were analyzed with repeated measurements of variance by statistical analysis, and the Greenhouse-Geisser correction was performed when necessary.

Combined with the previous studies and present data, Fz and FCz electrodes were selected for the analysis of evoked delta oscillation (1–3 Hz); Cz and CPz for the analysis of evoked theta oscillation (4–8 Hz) between 200 and 500 ms. Fz and FCz were selected for the time-domain analysis. One-way repeated measure analysis of variance (ANOVA) was used to analyze behavioral data and delta and theta oscillations, with shape features (similar versus dissimilar) and conceptual features (similar versus dissimilar) as within-subject factors.

Results

Behavioral Results

To compare the influence on the judgment by shape and concept, a one-way repeated measure ANOVA was conducted for accuracy (ACC) and reaction time (RT) respectively in five conditions (the same, S+C+, S-C+, S+C-, S-C-).

As can be seen from Figure 3 (right panel), the main effect of ACC was significant, F(4, 23) = 19.79, p < 0.001, MSE = 0.002, ηp2 = .473. The one-way repeated measure ANOVA showed the same versus S+C+ (p = .017), the same versus S-C+ (p < .01), the same versus S+C− (p = .01), the same versus S-C− (p = .01), S+C+ versus S-C+ (p = .001), S+C+ versus S+C− (p = .011) and S+C+ versus S-C− (p = .003). This may reflect that it is easier to recognize the mismatched conditions than the matched conditions in the cognitive comparison stage.

In terms of RT, as can be seen from Figure 3 (left panel), there was a main effect across conditions, F(4,23) = 5.26, p = .008, MSE = 803.94, ηp2 = .20. The one-way repeated measure ANOVA showed statistically significant effects of S+C+ versus the same (p = .037), S+C+ versus S-C+ (p = .027), S+C+ versus S+C− (p = .021), S+C+ versus S-C− (p < .001). It means that S2 and S1 produce conflicts in participants’ behavior when they are congruous both in shape and concept.

Electrophysiological Data Acquisition and Analysis

Time-domain analysis. The waves of time-domain N270 are shown in Figure 4, with the topographical distribution shown in Figure 5. The time window was set in the 200-500 ms range because it temporally followed the perceptual processing of the target stimuli and fell within the average range of participants’ RT, which was assumed to indicate the comparison phase of visual working memory when the conflicting processing between incongruous pictures occurred. It can be seen from Figure 4 that the main effect of experimental condition was significant, p < .001, F(4, 23) = 41.273, MSE = 8.375, ηp2 = .614), indicating that, compared with other conditions, the condition when the pictures had dissimilar shapes elicited the largest amplitude over the Fz and FCz, while the same condition elicited the smallest amplitude.

For the ERP amplitude, repeated measures ANOVAs showed a significant effect between the same condition versus S+C+ condition (p < .001), the same condition versus S-C+ condition (p < .001), the same condition versus S+C− condition (p < .001), the same condition versus S-C− condition (p < .001), S+C+ condition versus S-C+ condition (p < .001), S+C+ condition versus S-C− condition (p = .025), S+C+ condition versus S+C− condition (p = .009), and S-C+ condition versus S-C− condition (p = .031). There was no significant difference between S-C+ condition versus S+C− condition (p = .056) and between S+C- condition versus S-C− condition (p = 1).

Behavioral results.

Time-domain evoked potential of incongruous minus congruous at Fz (upper panel) and FCz (lower panel).

Time-frequency analysis. Multivariate repeated measures ANOVAs were computed on evoked delta and theta oscillation. Based on the topographical distribution, evoked band power was analyzed at the Fz and FCz, Cz, and CPz electrodes, respectively (see Figure 6). The topographical distribution of the evoked theta oscillation (4-8 Hz at 200-400 ms) and delta oscillation (1–3 Hz at 300–500 ms) is shown in Figure 7.

For the time-frequency analysis, repeated measures ANOVAs found a main effect of experimental conditions in the time window of 300-500 ms, p = .017; F(4, 23) = 3.997, MSE = 10686.19, ηp2 = .133, with S+C+ versus S-C+ (p = .002), S+C+ versus S+C− (p = .04), and S+C+ versus S-C− (p = .004). The pairwise comparison found the main effect of experimental condition in the time window of 200-400 ms, p = .002, F(4, 23) = 6.699, MSE = 5233.192; ηp2 = .205), S+C+ versus S-C+, (p = .001), S+C+ versus S-C− (p = .018), and S-C+ versus S+C− (p = .014). When shaped differently, the S2 elicited stronger delta and theta. However, it did not reveal any significant effect between the same versus S+C+ (p = 1), the same versus S-C+ (p = .302), the same versus S+C− (p = 1), S+C+ versus S+C− (p = .735), S-C+ versus S-C− (p = 1), and S+C- versus S-C− (p = 0.122).

Thus, the present study found that, at the behavioral level, the accuracy in the same condition was significantly lower and the RT of S+C+ condition was significantly slower. At the neural level, the N270 amplitude and evoked delta and theta power significantly increased for incongruous picture pairs compared to congruous ones.

Topographical distribution of N270 component in the five experimental conditions: The same = S2 was exactly the same as S1; S+C+ = S2 was congruous to S1 both in shape and concept; S-C+ = S2 was incongruous in shape but congruous in concept; S+C− = S2 was congruous in shape but incongruous in concept; S-C− = S2 was incongruous to S1 both in shape and concept.

Time-frequency representations of evoked delta and theta oscillations at four electrodes in two time windows. Panel A: Fz electrode in the time window of 200-400 ms (4-8 Hz); Panel B: FCz electrode in the time window of 200-400 ms (4-8 Hz); Panel C: Cz electrode in the time-window of 300-500 ms (1-3 Hz); Panel D: CPz electrode in the time-window of 300-500 ms (1-3 Hz).

Topographical distribution of the evoked delta and theta oscillations at two time windows. Panel A: Topographical distribution of the evoked theta oscillations at 200-400 ms (4-8Hz); Panel B: Topographical distribution of the evoked delta oscillations at 300-500 ms (1 -3Hz).

Mean value of advertisement attitudes under the four experimental conditions.

Advertisment-attitude analysis. The advertising metaphor pictures in the S+C+ condition rank the top in response to the four questions, however, the previous neuroscience experiment has proved that the cognitive effect of the S+C+ condition was lower than any other experimental condition. Therefore, the analysis of the questionnaire focuses on the advertisement attitude under the interactive conditions of perceptual and conceptual incongruity. It can be seen that, in the case of S-C+, the average score was higher than any other conditions (see Figures 5 and 8). It can be concluded that the advertising attitude will be better when the shape of the target product and the source domain object is incongruous but the concept is congruous. At the same time, metaphor advertising pictures which are incongruous both in shape and concept are the least popular. This is consistent with many existing research results. The inconsistency of shape and concept will lead to overly difficult cross-domain mapping between the target product and the source object in cognitive processing. Thus, it can be seen that only when shape or concept is incongruous can a better advertising attitude be produced, which should be the ideal condition for moderate incongruity to play a role.

Discussion

In the current study, we investigated the effect of perceptually and conceptually incongruous visual information on the target-source matching of the pictorial advertising metaphors by using a SDJ paradigm. A one-way repeated measure ANOVA for a 2 × 2 ([Shape: similar, dissimilar] × [Concept: similar, dissimilar]) experimental condition was conducted on ACC, RT, N270, and delta and theta band activity.

Analyzing behavioral (ACC and RT) performance, we found that the detection of target-source congruous matching was slower than for incongruous matching. We speculate that for pairs of metaphorically congruous items, feature-by-feature matching generally requires increased cognitive effort. Importantly, however, the behavioral data did not show a significant difference in responses among the other four conditions in which the two pictures were perceptually and/or conceptually different. We attribute this to the possibility that the behavioral and cognitive conflict processes do not share common resources.

Analysis of the N270 evoked potential and related time domains and time frequencies revealed that perceptual incongruity facilitates the metaphorical processing of visual modality. Consistent with the findings from previous studies, the present study showed that N270 was elicited after the onset of a stimulus when its attributes were incongruous to the preceding one. In the present study, the N270 amplitude was larger for the cases where S2 was incongruous either in shape, in concept, or both, compared to where it was the same as S1. A more negative waveform of N270 for S2 was also found when S2 was incongruous in shape or incongruous both in shape and concept compared to when S2 was congruous to S1 both in shape and concept. Similar findings were obtained by Wang et al. (1999), Kong et al. (2000) and Cui et al. (2000) who reported that in their match-to-sample tasks, the mid-frontally distributed N270 was elicited and was more negative in non-matched tasks concerning object shape. The current study again lends support to a N270 effect for mismatches between S2 and S1 in shape (Cui et al., 2000; Wang et al., 2004; Zhang et al., 2001), suggesting that the N270 represents neural activity relating to incongruity detection in the presented information (Kuperman et al., 1995; Rugg, 1985).

What is worth noting is that when the two pictures were congruous in shape but incongruous in concept (S+C−), N270 amplitude was larger, supporting that N270 indexes both perceptual and semantic conflict processing. More importantly, however, relative to the condition of shape and content incongruity (S-C−), the condition of shape incongruity and concept congruity (S-C+) was found to elicit a larger amplitude of N270. We believe that salient shape differences between the target and the source may interfere with the semantic processing of conceptual differences. Participants may identify the two pictures as incongruous based on their different shape and stop further processing of the discrepancy of functional compatibility. This may be explained by the response competition model by Eriksen and O’Hara (1982), which illustrates that response activation starts as soon as information associated with the responses has become available and before comparison of the objects is completed. On the other hand, sequential difference between perceptual and conceptual processing (Loftus & Ginn, 1984) suggests that the incongruity processing to visual input of a source item in a metaphorical relationship is temporally independent of the conceptual difference. It is possibly reasonable to claim that for objects coming from the same and/or different conceptual categories, shape dissimilarity facilitates finding conceptual differences at the electrophysiological level, as indicated by “shape is function” (Ortiz, 2010), in much the same way that shape similarity facilitates finding conceptual correspondences between objects (van Weelden et al., 2011).

In addition, the examination of N270 waveform showed a larger amplitude when the two sequential stimuli were of incongruous shapes than when they were incongruous concepts. The difference between the extent of detection of mismatch in the shape and concept dimensions might result from differential speed of information processing on the two dimensions. This, in turn, implies that processing conceptual information requires more time compared with shape processing.

This assumption was verified in the time-frequency analysis. For the time window of 200-400 ms (4-8 Hz), which roughly spans the time frame of the perceptual processing, the role that shape plays in identifying incongruous information was observable. In this early time window, a significantly different theta activity was identified with picture pairs when S2 was incongruous in shape, whether congruous in concept or not, compared with pairs when S2 was congruous both in shape and in concept. Meanwhile, a significant discrepancy occurred when S2, which was incongruous in shape but congruous in concept, was compared to S2 which was congruous in shape but incongruous in concept.

The role that concept plays in processing incongruous information was narrowed down to the late time window of 300-500 ms (1-3 Hz). Relative to S2 when it is congruous both in shape and in concept, S2, when it is congruous in shape but incongruous in concept and when it is incongruous both in shape and in concept, elicited a stronger delta band activity. We postulate that this reflects a later and slower detection of incongruous conceptual features concerning visual metaphors. Surprisingly, a prolonged difference of delta activity was still found when the picture pair with S2 incongruous in shape but congruous in concept was compared with the picture pair with S2 congruous both in shape and in concept. In all, these findings support the proposition that frontal delta and theta activity reflect visual working memory-related processing and detection of the mismatched information in visual working memory. Many neurons in subregions of the frontal and parietal cortex show elevated responses during tasks requiring the active storage of feature- or stimulus-specific visual information (e.g., Bisley & Pasternak, 2000; Mendoza-Halliday et al., 2014; Miller & Tippett, 1996). These results indicate that frontal delta and theta activities observed in the delayed matching task are not the neural index of external behavioral conflicts, but rather reflect the endogenous detection of mismatch processing (Liang et al., 2012).

The findings obtained in the current study are significant for the design of visual advertising metaphors. The study identified and verified indicators of incongruous information on the neural level, the frontally distributed N270 as well as delta and theta. They can be used to assess the effectiveness of artful deviation of perceptual and conceptual representations, as the identified brain region and activity strength can reflect whether advertisement readers detect the mismatched features in an implicit manner. As shown in the study, the most effective way to lead readers to detect the mismatch is to create a shape incongruity between the source and the target, which will facilitate the processing of visual metaphors. The examination of advertisement attitude also supported the findings.

Conclusion

Incongruity plays an important role in the target-source matching of pictorial advertising metaphors. By observing N270 as well as the delta and theta band, the current study revealed the facilitating effect that perceptual incongruity has on the detection of the target-source discrepancy in pictorial metaphors. Future studies should investigate how incongruity is processed at a later time so that a full panorama of the processing mechanism of metaphorical incongruity in advertising context can be revealed.