Investigating vision in schizophrenia through responses to humorous stimuli.

The visual environment of humans contains abundant ambiguity and fragmentary information. Therefore, an early step of vision must disambiguate the incessant stream of information. Humorous stimuli produce a situation that is strikingly analogous to this process: Funniness is associated with the incongruity contained in a joke, pun, or cartoon. Like in vision in general, appreciating a visual pun as funny necessitates disambiguation of incongruous information. Therefore, perceived funniness of visual puns was implemented to study visual perception in a sample of 36 schizophrenia patients and 56 healthy control participants. We found that both visual incongruity and Theory of Mind (ToM) content of the puns were associated with increased experienced funniness. This was significantly less so in participants with schizophrenia, consistent with the gestalt hypothesis of schizophrenia, which would predict compromised perceptual organization in patients. The association of incongruity with funniness was not mediated by known predictors of humor appreciation, such as affective state, depression, or extraversion. Patients with higher excitement symptoms and, at a trend level, reduced cognitive symptoms, reported lower funniness experiences. An open question remained whether patients showed this deficiency of visual incongruity detection independent of their ToM deficiency. Humorous stimuli may be viewed as a convenient method to study perceptual processes, but also fundamental questions of higher-level cognition.

Introduction

The visual environment of humans is replete with ambiguity and fragmentary information. Yet, when we look and perceive we do not see microscopic bits in a stream of visual information that we would then attempt to aggregate in larger ensembles; in spite of the complexity of the information flow we always perceive coherent macroscopic objects and scenes that have temporal stability. This phenomenon became the cornerstone of gestalt theory (Köhler, 1920) in psychology: At a fundamental stage of visual perceptual processing, holistic disambiguation and pattern formation of the visual input take place. This early stage of processing commonly occurs preattentively, without the perceiver's conscious allocation of attention.

Previous research has indicated that in schizophrenia spectrum disorder this 'gestalt stage' of perception may be compromised (Silverstein and Uhlhaas, 2004). Problems in gestalt perception were found to be associated with the degree of cognitive symptoms of schizophrenia patients as well as in persons at risk for schizophrenia (Uhlhaas and Silverstein, 2005). Perceptual paradigms such as motion-induced blindness and apparent motion, which implement gestalt coordination, have been used to investigate how patients with schizophrenia may have altered visual perception (Tschacher et al., 2006, Tschacher et al., 2008). Their findings were that the perception of ambiguous stimuli depended on the current psychopathological state of patients; there were no clear signs of a general gestalt deficiency or of a general gestalt excess.

In the present study, we adopted a novel, and probably unconventional path to investigate the early Gestalt stage in vision: by analyzing humorous stimuli. According to psychological theories of humor appreciation, funniness is associated with the incongruity contained in a joke, visual pun, or cartoon, and the viewer's subsequent incongruity resolution (Hempelmann and Samson, 2007, Hempelmann and Samson, 2008, Ruch, 2007, Suls, 1972). Humor is thus based on conflict and incongruity of stimuli; perception of this incongruity together with a quick process of incongruity resolution is experienced as funny. Humor relies on an early event in perception, which may not require explicit cognitive processing — if one has to explain a joke, it is usually no longer funny. Therefore, our basic idea was to use humor appreciation, operationalized by experienced funniness, as a measure to analyze early perceptual processes in participants. This idea is based on the assumption that humor processing requires gestalt-like perceptual coordination.

Humor has, of course, several more facets as it is mediated by many aesthetic forms, such as jokes, gags, irony, sarcasm, black humor, situational humor, etc. Different faculties of mental functioning work cooperatively in humor perception and humor appreciation: Affect and emotion are essential, as humor is in first line joyful (expressed by laughing), yet often blended with other emotion dimensions, especially surprise and aggression (like in Schadenfreude humor; Ferguson and Ford, 2008 for a review). Humor is a perceptual ability, as you have to 'get the joke' by picking up the incongruity that is present in all humorous stimuli. And humor is social because it often takes two persons, the joke teller and the listener, and sometimes more (such as the third persons both are making fun of). To get the joke in social humor, the perceiver must apply Theory of Mind (ToM), i.e. be able to switch between the perspectives of the protagonists in the joke. And finally there is a strong cultural aspect to humor, which may transport, for instance, political, national, or gender stereotypes.

Humor can be expressed in different sensory modalities, and it is predominantly studied in linguistic form, in the analysis of told jokes. Humorous material based on visual stimuli, our focus in the present study, may come in different types, the genres of cartoons, comics, and visual puns. A cartoon is a drawing, sometimes accompanied by a verbal caption or speech balloon. Comics are series of cartoons that convey a narrative. Visual puns are drawings without any text where one visual element evokes two meanings — similarly to verbal puns, in which one word evokes two meanings (Hempelmann and Samson, 2007). Comparing visual to linguistically based humorous stimuli, the difference is that the latter unfold linearly in time; the joke teller can control when ambiguous information is presented. In the pictorial array of a visual pun, the sequence of processing is up to the perceiver.

We used visual puns exclusively in the present study to put the focus on incongruity in vision. The resolution of incongruity is experienced as positive affect (Hempelmann and Samson, 2008). In visual puns, the incongruity cannot arise from the relation between the iconic stimulus and the verbal caption; therefore it arises from one visual element signifying two or more meanings or cognitive schemata at the same time. Thus, perception of funniness in visual puns is a marker of a perceiver's ability to perceive the conflictual content in ambiguous pictures, and resolve the ensuing incongruity. In terms of gestalt psychology, to get the joke one has to be able to construct the two or more conflicting gestalts contained in a single ambiguous visual display.

Research on humor and schizophrenia has reported reduced appreciation in patients with schizophrenia; this may be true for several reasons. The dysfunctional perception of incongruity, our present concern, is to our knowledge a novel and specific explanation in the field of schizophrenia research. Possible other explanations were given such as altered affect and emotionality: Patients often suffer from symptoms such as anhedonia and depression and may therefore enjoy humorous stimuli less (Falkenberg et al., 2007); alternatively, as patients commonly have reduced quality of life, they may have less positive state affect. Patients' neurocognitive impairment and problems with attention may entail reduced humor appreciation (Bozikas et al., 2007) and humor recognition (Tsoi et al., 2008). In addition, recent research has focused on mentalizing and ToM of patients, and as it is an established finding that patients with schizophrenia have compromised mentalization abilities (Brüne et al., 2007), this may be causing their reduced appreciation of ToM-related humor (Corcoran et al., 1997).

The present study therefore explored the relation between incongruity in visual stimuli and the humorous responses of participants. First, our general prediction was that patients show reduced appreciation of incongruity-based visual puns, measured by experienced funniness. Second, as ToM-related stimuli are commonly funnier than other puns (Samson and Hegenloh, 2010), we hypothesized that the funniness of puns is connected to ToM-related content of visual puns, and that patients appreciate ToM-related humor less than other participants. A third goal was to explore potential further predictors of funniness such as demographic variables, personality traits, and affective state in an attempt to control for confounders of the assumed association of incongruity and funniness. Fourth, in the schizophrenia subgroup we explored the association of funniness with psychopathological states. Based on previous research, it was hypothesized that cognitive symptoms may be related to decreased funniness experiences in response to visual puns.

Methods and materials

Sample

Ninety-two adult participants (53 male, 39 female; mean age 37.11 years, range 18–65 years) took part in this study. The study group consisted of 36 patients (21 male, 15 female; mean age 37.7 years) all fulfilling ICD-10 criteria for a diagnosis of schizophrenia. Exclusion criteria were substance or alcohol abuse and acute psychosis. A healthy control group had 56 participants (32 male, 24 female; mean age 36.7 years). Groups were not statistically different with respect to age (t(90) = 0.38, p = .70) or sex (chi2(1) = 0.01, p = .91). Patients, however, had a reduced level of education (chi2(4) = 22.9, p < .0001). The study was approved by the cantonal ethics committee on the basis of written informed consent.

Stimuli

A set of 50 visual puns was presented in random order on a portable computer (after the initial 33 participants, the stimulus set was reduced to 40 puns). The visual puns portrayed varying levels of incongruity and more or less Theory of Mind (ToM) was assumed to be necessary to understand the joke in the visual pun. Examples for visual puns are shown in Figure 1. After each presentation, the participant was prompted to rate funniness, on a scale ranging from 1 to 6 ("How funny do you think this picture is?"), and whether the participant understood the point of the pun. Independent of data acquisition, each stimulus was assessed by investigators (ACS, ES, WT) as to how much incongruity it contained (0 = little incongruity, 1 = some incongruity, 2 = much incongruity) and whether mentalizing ability (i.e., ToM) was needed for understanding the joke (0 = no ToM needed, 1 = some ToM needed, 2 = ToM needed). In previous studies, visual puns were used to assess humor that did not require ToM to understand the jokes; these puns were compared to cartoons where one character in the cartoon had a false belief (so-called ToM-cartoons, see Samson et al., 2008). Visual puns in the present study differed in the degree to which ToM was necessary for understanding the joke.

Figure 1

Examples of visual puns used in the current study. From left to right, pun 76 (figure-ground reversal picture, some incongruity; no ToM needed; mean funniness rating 2.4), pun 111 (much incongruity; some ToM needed; mean funniness rating 4.1). Copyrights: pun 76, detail of an M.C. Escher painting; pun 111, Oswald Huber © (cf. Samson et al., 2008).

Questionnaires

PANSS: All patients participated in standardized clinical interviews (Positive and Negative Syndrome Scale PANSS, Kay et al., 1987) to assess the level of schizophrenia symptoms at the time of testing. Trained psychologists performed the interviews. The model of Lindenmayer et al. (1995) was used to cluster PANSS psychopathology into five factors: positive symptoms, negative symptoms, excitement, depression, and cognitive symptoms of schizophrenia. The average symptom level of study patients was low. The mean factor scores were 1.68 (SD = 0.68) for positive symptoms, 2.22 (SD = 0.80) for negative symptoms, 1.96 (SD = 0.62) for excitement, 2.24 (SD = 0.66) for depression, and 1.39 (SD = 0.43) for cognitive symptoms.

NEO-FFI: The Five Factor Personality Inventory (NEO-FFI, Borkenau and Ostendorf, 1993) is a multidimensional questionnaire for the self-assessment of personality dimensions (the “Big Five” are neuroticism, extraversion, openness for new experience, agreeableness, conscientiousness). The five dimensions are measured by 60 items with five-point scales.

MWT: The Mehrfachwahl-Wortschatz-Intelligenztest (Multiple-choice vocabulary intelligence test, Lehrl, 1995) is a brief test to assess the passive vocabulary of a person (as an aspect of general intelligence). The test consists of 37 items. Each item offers a row of five words, of which only one is a valid expression in German. Participants have to mark the valid word in each item.

PANAS: Prior to presentation of the humorous stimuli, all participants rated their own momentary affective state using the Positive and Negative Affect Scale, PANAS (Krohne et al., 1996). It consists of twenty emotion adjectives (e.g., 'active', 'interested', 'upset', 'afraid') that are rated on five-point scales ranging from 'very slightly or not at all' to 'extremely'. The twenty items load on two factors, positive affect and negative affect.

MASC: The MASC (Movie for the Assessment of Social Cognition) is a test to assess ToM in a movie depicting various social interactions (Fleck, 2007). The movie is interrupted several times for a multiple-choice display offering options on the thoughts or motifs of the actors at that point in the movie. The correct option can be inferred if the participant uses perspective taking in 'mind-reading' the social cognition of the actors. The participant's sum of all correct responses provides a measure of this participant's mentalizing ability.

Statistical analysis

To assess each hypothesis, we performed a hierarchical modeling procedure in which we considered different explanatory variables as predictors of experienced funniness, the dependent variable. To do this we used a step-up procedure, considering each explanatory variable in turn to explore different combinations of predictors (i.e., models). The software package used was JMP 11 (SAS Institute Inc., Cary, NC). The complete dataset of all assessments of puns viewed by N = 92 participants (patients and controls) comprised close to 4000 observations. This dataset contained statistically dependent data because the puns were repeatedly presented to participants. We applied mixed-effects analysis to explain the variance of the dependent variable ‘Experienced funniness’ by various fixed effects (i.e., predictors). In all models, 'Participant' and 'Visual pun' were entered as random effects, which defined the dependency structure inherent to this hierarchical dataset, and the variance component attributed to each random effect is given in each model. We evaluated the models in terms of the Akaike information criterion (AIC) to select the best model.

We fitted one mixed-effects analysis, containing several models, for each of the four hypotheses or goals of this investigation.

Results

Concerning the first hypothesis, the mixed-effects analysis showed that patients' and healthy participants' responses to the visual puns were not statistically different (Model 1.2 in Table 1). A strong effect was found for the association between experienced funniness and the incongruity contained in a pun (Model 1.3). An additional significant effect resulted for the interaction effect: The incongruity–funniness link was significantly weaker in schizophrenia patients (Model 1.4). Hence the first hypothesis was supported that schizophrenia patients responded less to the incongruity incorporated in the puns, although patients did not show significantly reduced appreciation of visual puns.

Table 1

Mixed effects models of N = 92 participants each viewing between 40 and 50 visual puns. For each model, fixed effects estimates, random effects estimates, whole model variance r2 and AIC are listed (top to bottom). Dependent variable, funniness of a pun.

	Model 1.1 (n = 3969)	Model 1.2 (n = 3969)	Model 1.3 (n = 3969)	Model 1.4 (n = 3969)	Model 1.5 (n = 1398)
Fixed Effects
Group [Sz]		t = − 0.99	t = − 1.01	t = − 0.96	t = 0.86
Incongruity			t = 5.56⁎⁎⁎⁎	t = 4.97⁎⁎⁎⁎	t = 3.14⁎⁎
Incongruity × Group [Sz]				t = − 4.86⁎⁎⁎⁎	t = 0.49
Random Effects
Participant (% variance)	22.43	22.43	24.43	24.58	39.24
Visual pun (% variance)	21.14	21.15	14.05	13.98	6.38
r2 (% variance)	44.59	44.59	44.59	44.92	50.55
AIC	12785.1	12789.3	12769.4	12753.1	4306.7

Note. [Sz] = in categorical variables, the schizophrenia group = 1 and control group = 0. AIC = Akaike's Information Criterion. AIC minimum printed in boldface.

p < .01,

p < .0001.

Regarding the second hypothesis, we replicated that patients with schizophrenia have problems of social cognition. In the present sample, the mean number of correct ToM responses in the MASC test was 16.6 (SD = 2.5) in healthy participants, but 11.18 (SD = 4.22) in schizophrenia patients, a significant difference (t(88) = 7.63, p < .0001). ToM measured by the MASC, however, was not a significant predictor of the funniness responses (Model 2.4 in Table 2). At the same time, the ToM content of the visual puns (assessed previously by the investigators) was strongly connected with funniness (Model 2.2), and was significantly less appreciated by the patient subgroup (Model 2.3). In an attempt to compare hypotheses 1 and 2, we re-tested Model 1.4 (Table 1) on the condition that only puns without ToM-content were used. The resulting Model 1.5 listed in Table 1 no longer had a significant interaction effect, while it still showed that incongruity was associated with funniness.

Table 2

Mixed effects models of N = 92 participants each viewing between 40 and 50 visual puns. For each model, fixed effects estimates, random effects estimates, whole model variance r2 and AIC are listed (top to bottom). Dependent variable, funniness of a pun.

	Model 2.1 (n = 3969)	Model 2.2 (n = 3969)	Model 2.3 (n = 3969)	Model 2.4 (n = 3919)
Fixed Effects
Group [Sz]	t = − 0.99	t = − 1.01	t = − 0.95	t = − 1.47
ToM pun		t = 6.28⁎⁎⁎⁎	t = 5.56⁎⁎⁎⁎	t = 5.85⁎⁎⁎⁎
ToM pun × Group [Sz]			t = − 5.97⁎⁎⁎⁎	t = − 5.47⁎⁎⁎⁎
MASC				t = − 1.13
Random Effects
Participant (% variance)	22.43	24.80	25.02	25.05
Visual pun (% variance)	21.15	12.73	12.59	12.85
r2 (% variance)	44.59	44.58	45.08	45.43
AIC	12789.3	12764.8	12736.9	12593.8

Note. [Sz] = in categorical variables, the schizophrenia group = 1 and control group = 0. AIC = Akaike's Information Criterion. AIC minimum printed in boldface. ToM pun = ToM content of a pun. MASC = Movie for the Assessment of Social Cognition.

p < .0001.

Third, we modeled the impact of demographic variables and personality on humor appreciation. We found that male participants reported higher levels of funniness (Model 3.2 in Table 3). Extraverted participants had reduced responses (Model 3.4), and funniness was positively related with a positive affective state prior to the testing (Model 3.5). Humor appreciation was unrelated to IQ, education level, and age. When we added the three significant predictors in a post-hoc test to the predictor of Model 1.4 (Table 1), no change of significance of the fixed effects of Model 1.4 resulted. Thus, the results on hypothesis 1 were stable in the presence of further predictors.

Table 3

Mixed effects models of N = 92 participants each viewing between 40 and 50 visual puns. For each model, fixed effects estimates, random effects estimates, whole model variance r2 and AIC are listed (top to bottom). Dependent variable, funniness of a pun.

	Model 3.1 (n = 3969)	Model 3.2 (n = 3969)	Model 3.3 (n = 3929)	Model 3.4 (n = 3929)	Model 3.5 (n = 3729)	Model 3.6 (n = 3729)	Model 3.7 (n = 3689)
Fixed Effects
Age	t = 0.82
Sex [male]		t = 2.16⁎	t = 2.13⁎	t = 2.50⁎	t = 1.96	t = 1.96	t = 1.78
Education level			t = 0.50
NEO-neuroticism				t = − 0.68
NEO-extraversion				t = − 2.23⁎	t = − 2.31⁎	t = − 2.25⁎	t = − 2.32⁎
NEO-openness				t = − 0.83
NEO-agreeableness				t = 1.02
NEO-conscientiousness				t = 1.44
PANAS positive					t = 2.51⁎	t = 1.89	t = 2.45⁎
PANAS negative						t = − 0.33
MWT IQ							t = 0.03
Random Effects
Participant (% variance)	22.5	21.71	22.09	19.48	18.69	18.87	18.94
Visual pun (% variance)	21.12	21.33	21.76	22.85	23.06	23.01	22.97
r2 (% variance)	44.59	44.59	44.26	44.63	44.63	44.63	44.62
AIC	12794.7	12785.8	12609.7	11969.2	11956.9	11959.4	11842.7

Note. AIC = Akaike's Information Criterion. AIC minimum printed in boldface. NEO = Five Factor Personality Inventory. PANAS = Positive and Negative Affect Scale MWT = Multiple-choice vocabulary intelligence test.

p < .05.

The fourth hypothesis was focused on the psychopathology measured by PANSS interviews of the patient subgroup (Table 4). The (negative) impact of cognitive symptoms on experienced funniness was significant only at a trend level (Model 4.6 in Table 4). Excitement symptoms were significantly predictive of funniness (Model 4.4).

Table 4

Mixed effects models of N = 36 patients each viewing between 40 and 50 visual puns. For each model, fixed effects estimates, random effects estimates, whole model variance r2 and AIC are listed (top to bottom). Dependent variable, funniness of a pun.

	Model 4.1 (n = 1460)	Model 4.2 (n = 1460)	Model 4.3 (n = 1460)	Model 4.4 (n = 1460)	Model 4.5 (n = 1460)	Model 4.6 (n = 1460)	Model 4.7 (n = 1460)
Fixed Effects
PANSS-negative		t = − 1.23
PANSS-positive			t = 0.96
PANSS-excitement				t = 2.47⁎			t = 2.43⁎
PANSS-depression					t = 1.47
PANSS-cognitive						t = − 1.74	t = − 1.71
Random Effects
Participant (% variance)	24.24	23.96	24.3	21.69	23.60	23.14	18.94
Visual pun (% variance)	16.53	16.59	16.51	17.09	16.68	16.80	22.97
r2 (% variance)	42.74	42.74	42.74	42.73	42.74	42.74	44.62
AIC	4811.3	4813.6	4813.9	4809.0	4812.6	4811.0	4808.9

Note. AIC = Akaike's Information Criterion. AIC minimum printed in boldface. PANSS = Positive and Negative Syndrome Scale.

p < .05.

Discussion

This study focused on the idea that appreciation of humor may offer insights into visual perception of schizophrenia patients. Psychological theories of humor assume that the incongruity of stimuli is the generator of the funniness response. Therefore, we operationalized incongruity detection and incongruity resolution using participants' experienced funniness as they viewed visual puns that had no linguistic captions but one visual element that evoked two meanings simultaneously. The data clearly supported the assumption that incongruity of puns scaled with funniness in all participants, and that this was significantly less so in schizophrenia patients. This finding is consistent with the 'gestalt hypothesis' of schizophrenia and with past findings of perceptual organization impairments in vision in people with schizophrenia, which have shown that patients encounter difficulties at the early coordination stage of visual perception. The present result of patients' attenuated incongruity–funniness linkage was not mediated by potential confounders (affective state, depression, extraversion, or sex), which are commonly associated with funniness responses.

We assessed an alternative to this visual-impairment hypothesis: Visual humor is known to be especially pronounced in puns that have more Theory of Mind (ToM) content. Schizophrenia patients have, like individuals with Autism Spectrum Disorder (ASD), reduced ToM abilities and hence problems in social cognition. Samson and Hegenloh (2010) found that individuals with high-functioning ASD did not differ from a control group in their humor appreciation of visual puns, yet had difficulty appreciating ToM cartoons (humorous stimuli that clearly required ToM to be understood); we replicated both findings with schizophrenia participants (see Model 2.3 in Table 2) using visual puns with varying levels of ToM.

Thus, did participants with schizophrenia show a deficiency to perceive the incongruity, or a deficiency to understand ToM in visual puns? In other words, did we detect deficiencies of visual perception or of higher-level cognition? We interpreted our data in the following way: Incongruity humor and ToM-based humor are not mutually exclusive sources of funniness. ToM-based visual puns use social cognition to generate (extra) incongruity in a visual scene. Our analyses showed that schizophrenia patients had both problems, detecting visual incongruity per se (Hypothesis 1), and, in addition, detecting ToM content that can entail 'social incongruity' in visual puns (Hypothesis 2). This interpretation is corroborated by the fact that the participants' ability to apply ToM, measured directly by the MASC test, was not a predictor of funniness in our data.

A limitation of the study is nevertheless that ToM and incongruity were not disentangled in the stimulus material and in the setup of the study. To do this, an experimental design should be considered. On the basis of the available data, the re-analysis shown in Model 1.5 (Table 1) may indicate that in the absence of ToM-related content of puns, patients' responses to visual incongruity were no longer different from controls' responses. This would cast doubt on our accepting Hypothesis 1. Yet it may likewise indicate that the subset of puns used for the re-analysis was insufficient to tap patients' specific visual perception. The main limitation of the study design is therefore that it did not allow ruling out that higher-level conceptual processes were (also) at work in incongruity detection. It would have helped to have an additional measure of perceptual organization among the tasks to directly explore the linkage of humor responses with gestalt processing. The weighing of hypothesis 1 (humor in schizophrenia is reduced because of deficient early vision) against hypothesis 2 (humor in schizophrenia is reduced because of ToM deficiency or higher-level cognitive deficits) is a task of future research.

In conclusion, we believe that testing humorous stimuli is a promising, and participant-friendly (!), method for the study of visual perception. Humor stimuli generate a state of tension or dissonance through incongruity, which may be regarded a laboratory equivalent of everyday perceptual demands, such as perceiving novelty and surprise, picking up of deviations from expectation, and ignoring irrelevant information. A goal of future research may lie in distinguishing the relative contributions of basic perceptual, higher-level conceptual, and linguistic cognitive processes in dealing with visual incongruity.

In terms of systems theory (Haken, 2000), the detection of incongruity in a stimulus represents a momentary increase of system complexity, which then suddenly gives way to new self-organization of the perceptual system: in all humorous material, this is enacted by the resolution of incongruity that constitutes the core of the funniness experience. The incongruity resolution process, understood as a sudden reduction of incongruity tension or, more generally, of a gradient, obeys general laws and is therefore of fundamental interest. Tschacher and Haken (2007) and Friston and Stephan (2007) have linked gradient reduction with the intentionality and adaptivity of complex systems such as the brain. Seen from the structural perspective of systems theory, this may reflect analogous processes acting at various levels of psychotic phenomena and symptoms — gestalt perception problems, disorders of intentionality, disorganized thought, and delusions; these apparently different symptoms may thus have their common root in dysfunctional gradient reduction. In this sense, humor research also allows investigating fundamental processes of vision and cognition that can be concisely described by an encompassing structural theory of complex systems.