Editors' Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science.

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain-general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Introduction

Drawn sequences of images are all around us, from comics and picture books, to instruction manuals, storyboards, and infographics. They are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and sequential images extend across human cultures and time periods (Petersen, 2011), making them a fundamental and universal part of human expression. Despite this, the proportion of studies examining visual narratives in Cognitive Science is staggeringly small compared to the wealth of research on language and text‐based narratives (Magliano, Higgs, & Clinton, 2019; Magliano, Loschky, Clinton, & Larson, 2013). Here, we examine why Cognitive Science may have overlooked this fundamental aspect of human meaning‐making, how visual narratives have typically appeared in studies of cognition, and why researchers should turn toward them with greater focus. We provide a framework that describes the underlying structures involved in drawn sequential narratives. In discussing this framework, we hope to pose interesting questions that engender interest in the study of psychology of visual narratives.

What are visual narratives?

Visual narratives are sequences of images created with meaningful intent, typically illustrating a continuous event sequence, particularly to tell a story. They may be considered a subset of meaningful sequences of images, which in less narrative forms also include instruction manuals and signage. Drawn visual narratives typically appear in comics, picture stories, and storyboards, and have many historical precedents (Petersen, 2011). Dynamic visual narratives appear as film, which uses actual percepts instead of drawings (animation aside), though they are often preceded in production by drawn narratives in the form of storyboards.

Why might visual narratives be interesting to those who study the mind? Consider first Fig. 1, a wordless comic from JA! by Ángela Cuéllar and Jonás Aguilar. In the first panel of this sequence, a man and his pets, a dog and a cat, enter a pet shop. In panel 2, the cat looks at a fish in a bowl, ominously peering into it in panel 3. In the fourth panel, the man then wonders what happened to his dog, only in panel 5 for the dog to be revealed as stuffed into the fish bowl (presumably by the cat).

Figure 1

An example of visual narrative from JA! by Ángela Cuéllar and Jonás Aguilar (© 2016; https://revista-exegesis.com/2016/01/ja/).

We will analyze this sequence in more detail below, but let us first point out several important aspects about how a mind might construe the meaning described above. First, a comprehender must recognize that the lines and shapes convey meanings, despite being drawings with a high degree of abstraction unlike natural percepts. Second, within these drawings, some elements do not look like natural percepts, and require conventional knowledge, such as the balloon indicating speech (which contains no text). Third, a comprehender must recognize that there are five distinct sub‐images (panels) within this broader spatial array, and that these segmented units must be viewed in a particular order. Fourth, understanding of these sequential images requires linking the meaning across images—knowing that the elements in one image (such as a cat) are the same entities in other images, and that there are changes in states between them (i.e., “referential continuity”), along with relations in narrative time, space, and causality (e.g., Zwaan & Radvansky, 1998). Fifth, in this sequential understanding, we must recognize that, although we see some events, there are other meanings that we do not see, and that we must infer, such as the cat stuffing the dog into the fishbowl (Graesser, Singer, & Trabasso, 1994). Sixth, we understand that characters have intentions that may motivate their actions (Trabasso, van den Broek, & Suh, 1989), and our expectations might be confouded for narrative effect (such as the possible expectation that the cat wants to eat the fish at panel 3, only to turn out that it wants to stuff the dog into the fishbowl). Seventh, the presentation of this sequential meaning follows particular choices for when we see the things that we do, in what pacing, and with what framing.

None of these aspects of comprehension are trivial, and they relate to many established structures already studied in Cognitive Science: object and scene perception, spatial cognition, event cognition, sequence processing, theory of mind, inferencing, discourse processing, and many other fundamental aspects of cognition. If this example was a block of written language, the importance of studying its cognitive underpinnings would be taken for granted, and that doing so would be fundamental for assessing the mind more generally. Yet, for some reason as a drawn narrative it does not intrinsically carry such priorities. The question is: why?

Why aren't visual narratives studied?

Given the above, why might visual narratives not be studied with the type of seriousness afforded other types of human expression and communication? We see several potential factors involved. First, sequential images are frequently used in tasks and stimuli to investigate other aspects of cognition (e.g., Baron‐Cohen, Leslie, & Frith, 1986; Boroditsky, Gaby, & Levinson, 2008); this means researchers are certainly aware of visual narratives as potentially effective communicative tools, but they just have not consolidated around studying them in and of themselves.

Second, the use of visual narratives as stimuli often presumes that their understanding is fairly transparent. This may relate to phenomenologically based beliefs that drawings are not internally complex or simply map directly into spatial or event knowledge, despite evidence to the contrary (Cohn, 2012; Willats, 2005; Wilson, 2016). While there may be cross‐domain processes that support comprehension across narrative experiences (Magliano et al., 2013), the conventions of the visual modality likely lead to non‐trivial dimensions for how these mechanisms operate and the awareness of them (Cohn, 2013b; Magliano et al., 2019).

Third, though attitudes have been changing in recent years, visual narratives have historically been afforded low esteem in culture, in the form of comics or illustrated picture books, for example, compared to language or film. Drawn visual narratives have often been associated with younger readers, again, likely tied to their presumed transparency. Nevertheless, while attitudes have been changing over the past decades in the United States, children have never been the sole audience for visual narratives (Duncan, Smith, & Levitz, 2015), and such stereotypes are not as strong in other countries (Schodt, 1983). If something is perceived as simplistic, with low cultural value, why should cognitive scientists study them seriously? Below, we address these myths of the simplicity of visual narrative comprehension. We then detail why addressing them is important for the use of sequential images in experimentation, and finally make the case for the benefits of studying visual narratives in Cognitive Science.

Myths of visual narrative

We contend that at least three myths about visual narratives have contributed to the lack of a formal field of study on them. The first is the myth of transparency. There are many assumptions about the universal transparency of sequential image comprehension. Sequential images are thought to be understandable by everyone—including young children—with little learning or decoding (e.g., McCloud, 1993). This assumed transparency likely arises from beliefs that they require basic perceptual or event cognition alone. The logic follows something like this: Sequential images visually depict objects and events, and basic perceptual processing allows everyone to understand events (Gibson, 2014); thus, everyone should understand sequential images. This thinking has likely motivated the frequent use of sequential images in psychological experiments, IQ tests and clinical assessments, and educational, humanitarian, and/or anthropological research, especially with non‐literate and native populations (see below).

Despite these presumptions, sequential image understanding requires exposure and expertise, as suggested by a scattered literature of cross‐cultural, developmental, and cognitive research. First, various cross‐cultural studies have found that individuals from indigenous communities have difficulty construing even basic aspects of sequential images. These responses often manifest at a basic level, where individuals do not construe that the characters in one image are the same as those in subsequent images—that is, referential continuity. Such findings appear with individuals in Nepal (Fussell & Haaland, 1978), Papua New Guinea (Bishop, 1977; Cook, 1980), and various populations in Africa (Byram & Garforth, 1980; Duncan, Gourlay, & Hudson, 1973; Liddell, 1996), among others. The consistent trend in these studies is that sequential images were not understood , but rather were interpreted as a series of isolated images each depicting their own scene, with no binding continuity. Higher level deficits (i.e., inference processes) have also been observed by rural populations in Turkey for film (Ildirar & Schwan, 2015). These individuals who do not construe sequential information typically come from rural communities. They also have minimal exposure to visual narratives (comics, illustrated books, films), which often correlates with low literacy or education (Cook, 1980; Le Guen & Pool Balam, 2012). Overall, though, the findings are consistent: Sequential image understanding requires exposure and practice with visual narratives (for review, see Cohn, in press).

Second, developmental research has suggested a trajectory for when children begin to comprehend a sequence of images as a sequence. This work has suggested that children at or below the age of 4 have difficulty recognizing the referential continuity of repeated characters as indexing the same entities (Bornens, 1990; Trabasso & Nickels, 1992). Recognition of this continuity begins between 4 and 5, typically with full understanding around age 6 (Bornens, 1990). This age range is also when children begin to become more proficient at picture arrangement tasks, where they organize randomly ordered images into a coherent sequence (Fivush & Mandler, 1985; Friedman, 1990; Weist, Lyytinen, Wysocka, & Atanassova, 1997), and when they are able to describe a coherent sequence of narrative events from sequenced images (Trabasso & Nickels, 1992). Also, around age 5 they begin to recognize and infer omitted content from sequences (Brown & French, 1976; Kunen, Chabaud, & Dean, 1987; Schmidt & Paris, 1978), though both picture arrangement and inference continue developing into later years, and are modulated both by age and by experience with comics (Nakazawa, 2016). Altogether, this research suggests that children develop an understanding of image sequences across a developmental trajectory conditioned by their exposure to visual narratives (for review, see Cohn, in press).

While lack of comprehension due to exposure may seem like an extreme case, effects of expertise also appear between “fluent” comic readers (Lee & Armour, 2016; Zhao & Mahrt, 2018). Metrics of self‐assessed comic reading/drawing proficiency have been shown to be consistent predictors of individual differences in studies of visual narratives assessed by reaction times (Cohn, Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012), self‐paced viewing times (Cohn & Wittenberg, 2015), segmentation choices (Cohn & Bender, 2017), comprehension ratings (Cohn, Murthy, & Foulsham, 2016; Cohn & Wittenberg, 2015), accuracy judgements (Hagmann & Cohn, 2016), and amplitude differences of brainwaves (Cohn & Kutas, 2015, 2017; Cohn et al., 2012). In addition, processing may not be modulated just by general expertise with visual narratives, but also by the patterns found in specific visual narrative systems. Recent analysis of the neurocognitive processes of a particular sequential pattern was found to be modulated by frequency of reading comics that prevalently use that pattern (Cohn & Kutas, 2017). Thus, even among experienced readers of visual narratives, expertise may modulate processing.

Overall, this work suggests that sequential image comprehension requires a “fluency” that is acquired from exposure and practice with visual narratives across a developmental trajectory. These findings go against the myth of transparency that sequential image understanding comes “for free” with perception and/or event cognition. It is also noteworthy that the works documenting these findings belong to scattered disciplines, with little integration, much less placed in the context of the broader study of the mind. If we want to truly understand these forms—and their connection to other aspects of cognition like language, spatial cognition, or event understanding—they must find a place within Cognitive Science.

The second myth is that of universal comprehension processes. Some have argued that many of the cognitive processes that support meaning‐making in text should also support the processing of visual narratives (Baggett, 1979; Gernsbacher, 1990; Magliano et al., 2013). As such, there may not be a perceived need to study the psychology of visual narratives, given a rich history of research on narrative discourse (see McNamara & Magliano, 2009 for an extensive review). However, comprehension proficiency between text and visual narratives is weakly correlated in children (Pezdek, Lehrer, & Simon, 1984). Moreover, only a few studies have directly explored the extent to which processes are common across textual, filmic, or drawn narratives (Baggett, 1979; Coderre et al., 2018; Magliano, Kopp, McNerney, Radvansky, & Zacks, 2012; Robertson, 2000; West, 1998). Finally, there are reasons to also assume that visual narratives require unique processes (Cohn, 2013b; Magliano et al., 2019).

Consider bridging inferences, of which theories of comprehension universally assume are important for comprehension (McNamara & Magliano, 2009). Bridging inferences establish how two or more narrative events are connected. They involve a range of inferences, from anaphor resolution to causal processing (i.e., establishing that two narrative events are causally connected). A few recent studies have demonstrated that bridging inference is important for the comprehension of visual narratives (Cohn & Kutas, 2015; Cohn & Wittenberg, 2015; Hutson, Smith, Magliano, & Loschky, 2017; Magliano, Kopp, Higgs, & Rapp, 2017; Magliano, Larson, Higgs, & Loschky, 2015), which is consistent with the assumption of universal comprehension processes. However, bridging inferences in visual narratives involve attentional selection and visual search of images—aspects of scene perception that cannot occur in text (Hutson et al., 2017). While there may be universal cognitive processes (e.g., bridging inferences) across narrative media, modality‐specific processes may support that meaning making (Cohn, 2013b; Loughlin, Grossnickle, Dinsmore, & Alexander, 2015). As such, strong adherence to the myth of universal cognitive comprehension processes has potentially prevented important insights into how these processes are achieved across media. The time is ripe for a systematic study of the psychology of visual narratives.

A final myth is that visual narratives are neutral experimental materials, which is arguably a consequence of the first two myths. Because of the presumptions of transparency, experiments often rely on drawings and sequential images as experimental stimuli under the assumption that they will require no expertise to decode. Researchers in psychology use sequential images in experimental tasks to study event cognition (Tinaz, Schendan, Schon, & Stern, 2006), temporal cognition (Boroditsky et al., 2008), discourse (Gernsbacher, 1985), theory of mind (Baron‐Cohen et al., 1986; Sivaratnam, Cornish, Gray, Howlin, & Rinehart, 2012), and social intelligence (Campbell & McCord, 1996), not to mention as elicitation tools for studying language (Berman & Slobin, 1994; San Roque et al., 2012), among others. They have also become a staple of general intelligence (IQ) tests (WAIS‐IQ, WISC), and clinical assessments (Kaufman & Lichtenberger, 2006; Ramos & Die, 1986).

Consider also the widespread use of visual narratives as stimuli in developmental research, as in tasks for Theory of Mind (Baron‐Cohen et al., 1986; Sivaratnam et al., 2012). Sequential images are often used to study the developmental trajectory of ToM in children because they are perceived as easily controllable and transparent (and appealing) to children. However, as discussed above, the age of onset for sequential image comprehension appears to be around 4–5 years old. Thus, assessing the development of ToM may be confounded by the concurrent development of “visual narrative fluency,” which is not measured. This confound may be a challenge to numerous domains that use visual narratives, including temporal cognition (Ingber & Eden, 2011; Weist, 2009), narrative cognition (Burris & Brown, 2014), and sequential reasoning (Zampini et al., 2017).

Another prevalent use of visual narratives comes in the Picture Arrangement Task (PAT), where participants arrange unordered images into a sequence, which is then scored against an expected, target sequence. The PAT has long been a key part of general intelligence (IQ) tests (WAIS‐IQ, WISC) and clinical assessments (Kaufman & Lichtenberger, 2006; Ramos & Die, 1986) like brain damage (Breiger, 1956; Huber & Gleber, 1982) and other clinical diagnoses (Beatty, Jocic, & Monson, 1993; Beatty & Monson, 1994). Nevertheless, questions persist about what these tasks index (Lipsitz, Dworkin, & Erlenmeyer‐Kimling, 1993; Ramos & Die, 1986; Tulsky & Price, 2003), perhaps because such studies never include measures of visual narrative reading experience, despite robust findings that PAT proficiency differs across age and experience with visual narratives (Fivush & Mandler, 1985; Friedman, 1990; Nakazawa, 2016; Weist et al., 1997). In addition, the PAT would deem any “unexpected” sequence order as incorrect, even though reordering can yield multiple well‐formed sequences when accounting for the structural constraints of visual narratives (Cohn, 2014).

Thus, despite the use of visual narratives as stimuli and in tasks across Cognitive Science, rarely do these studies account for aspects of the participants (expertise) and stimuli (stimulus structure). If sequential images are to be used in such experimental contexts, then it behooves us to examine just how they are understood, and how that understanding interacts with language and other cognitive systems.

Aspects of visual narratives

A growing number of studies have begun to examine the actual properties of sequential image understanding (Cohn, 2013b; Loschky, Magliano, Larson, & Smith, 2019; Nakazawa, 2016). This work has pointed to a complex structure that affords questions about visual narrative processing as a unique focus of scientific inquiry, and connected to fundamental issues to the broader study of cognition.

Sequential image structure is complex

Despite the stereotype that visual narratives are simple, they actually involve numerous levels of structure which interface with commonly studied aspects of cognition. A model of visual narrative structure is provided in Fig. 1, which divides visual narrative systems across a modality of expression (graphics), organized using combinatorial systems (grammar), which express conceptual information (meaning), and does so across both the unit and sequence levels. While these structures can be described abstractly, here we return to our example from Fig. 1 across its component structures. We will start by discussing aspects of the modality at both the unit (graphic structure) and sequence (layout) levels. Then we will discuss the combinatorial systems that organize form and meaning for both units (morphological structure) and sequences (narrative structure), before addressing aspects of the meaning itself with conceptual/event structures. Finally, we discuss the connection of visual narratives with text in multimodal relationships. Each section concludes with suggestions for possible, future research questions (Fig. 2).

Figure 2

A model of the structures involved in visual narratives.

Graphic structure

At the most surface level, comprehending a visual sequence requires decoding the —the system organizing the physical manifestation of the graphics of the representation. Such a level would be analogous to phonological structure in spoken or signed languages, organizing the modality of expression itself. The systematized aspects of graphics—lines, shapes, junctions—map to the meanings expressed in a visual image. To understand Fig. 1, a reader must understand that various shapes correspond to expressed meanings, both referential entities (a man, cat, dog, etc.), locations (inside and outside of a pet store), and events (walking, looking, questioning). Processing this information is supported by perceptual and attentional selection (Loschky et al., 2019; Magliano et al., 2013).

Yet, as discussed above, while basic perceptual processes operate in visual narrative understanding, they must interact with specialized knowledge that is unique to the domain, and may be culturally specific (Cohn, 2013b). Graphic schemas entrenched in long‐term memory can be considered the “visual vocabulary” of drawn representations, and range in size from the small subparts of images like hands, eyes, and head‐shape, to larger composites like whole figures and even scenes (Cohn, 2013b). Learned patterns also exist for abstract relations of “morphology” like speech balloons and thought bubbles, or the motion lines that illustrate the paths of moving objects (discussed further below). All of these patterned representations must be stored in the minds of artists who create them, and, while iconicity may enable broader comprehension, frequent exposure may in turn be encoded preferentially for comprehenders of particular visual lexicons (Nakazawa, 2016).

While understanding these iconic images, in a range of styles (Kendall, Raffaelli, Kingstone, & Todd, 2016), may seem simple to people surrounded by graphic images, understanding and producing graphic representations is fairly complex. First, with regard to production, increasing evidence points to drawing development being guided by the imitation of graphic schemas (Huntsinger, Jose, Krieg, & Luo, 2011; Okada & Ishibashi, 2017; Wilson, 1988). If learners seek to acquire a visual vocabulary of graphic schemas, it would explain why learners in many cultures appear to reach the apex of a critical learning period for drawing around puberty, while no such apex is apparent in cultures with exposure and practice with rich visual vocabularies (Cohn, 2012; Wilson, 2016). Thus, people who “can’t draw” may not have sufficiently learned a visual vocabulary.

Second, in terms of comprehension, cross‐cultural findings have suggested that drawings of different styles are not always comprehended in the ways conforming to Western expectations (Fussell & Haaland, 1978; Wilkins, 1997). For example, Wilkins (1997) recounts showing Australian Aboriginals a drawing of what we would likely construe as a horse at a lateral side‐view, running with poofs of smoke kicked up around its hooves. However, many of his Aboriginal respondents reported that the horse was lying down or dead, because their indigenous visual narrative system maintains a fixed aerial viewpoint. With a fixed aerial view, a “lateral view” of a horse was viewed as if looking down on it, and thus it was lying down or dead. Such findings again imply a non‐trivial role of exposure and expertise to the understanding of graphic systems.

Finally, recent advances in machine learning have increasingly been able to recognize elements from visual narratives like faces, objects, textures, balloons, and other visual features (Nguyen, Rigaud, & Burie, 2017; Rigaud, Guérin, Karatzas, Burie, & Ogier, 2015; Saito et al., 2015). While these efforts show the growing effectiveness of machine learning for recognizing targeted aspects of visual narratives, such methods often underperform both human testers and computational methods trained on naturalistic photographs (Khetarpal & Jain, 2016; Takayama, Johan, & Nishita, 2012). However, significant advances are being made with such computational methods, which are rapidly becoming more reliable (see Laubrock & Dunst, 2019).

Such results raise various questions for numerous aspects of Cognitive Science. How to do people learn to draw? What is the role of exposure to particular graphic systems on comprehension? How do drawn percepts differ from naturalistic percepts? How might models of computer vision adjust to the complexity of visual narrative representations?

Layout

The physical properties of images also extend beyond the internal relations in a unit, but also to the of juxtaposed images, be it spatial or temporal. In a spatial layout, a comprehender must navigate across juxtaposed images in order to comprehend the content. Layouts have a range of complexity. In illustrated picture books, single images are often placed on each page or page‐spread. Fig. 1 uses a fairly simple layout, a left‐to‐right and down “Z‐path” that mimics the reading order of Western writing systems (Cohn, 2013a). Other comics use more complex and varying features of layout (Bateman, Veloso, Wildfeuer, Cheung, & Guo, 2016; Cohn, 2013a), involving vertical columns, staggering of panels, inset panels placed inside of other panels, and other decorative features. Negotiating the reading‐path across these features involves constraints that go beyond those in writing systems (Bateman, Beckmann, & Varela, 2018; Cohn, 2013a), and are certainly not learned through interactions in the real world.

Some work in the study of temporal cognition has speculated that the layout of narrative images can reveal biases about time‐space metaphors (Boroditsky et al., 2008). However, this conflates the understanding of layout and the content of images, as well as disregards that visual narratives have unique constraints guiding layouts, independent of writing and content (Cohn, 2013a). Though there are relationships between the layout of visual narratives and writing systems, the constraints operating on visual narrative layouts may be among the more modality‐specific features of visual narratives.

The study of layout opens a host of interesting questions across fields: What are the constraints on navigating page layouts, and how do they differ across cultures, genres, and so on? What are the principles by which layouts connect to their content? What is the relationship of navigating a visual narrative layout and writing systems? To what extent does navigation of visual narrative layouts rely on experience with visual narratives in general, or specific types (such as culture‐specific comic conventions)? How does the structure and navigation of layout in visual narratives relate to that in other media?

Morphological structure

In language research, morphemes refer to the smallest units of meaning instantiated in a form, and morphological structure refers to the combinatorial properties governing the units of a language. Similarly, visual narrative systems have been shown to have repeated, combinatorial features similar to the morphological knowledge that supports language (Cohn, 2012, 2013b; Forceville, 2011; Wilson & Wilson, 1977). Such a constitutes both the schemas involved in basic, iconic drawings, which render images that typically appear like basic percepts and scenes, and more symbolic representations like speech balloons, motion lines, or lightbulbs above heads. These latter visual morphemes depart from perceptual resemblance, using combinatorial structures to attach conventionalized visual signs to other “stems.” For example, in Fig. 1 the fourth panel shows the man “speaking” with a speech balloon (here albeit with imagistic contents). This “carrier” attaches to the stem of the speaker, and forms like these have been likened to bound morphemes—affixes—in verbal languages (Cohn, 2013b; Forceville, 2011). Similar affixation occurs with signs like hearts or stars floating above characters’ heads, motion lines attaching to moving objects, or substitutions of hearts or stars for characters’ eyes (Cohn, 2013b; Forceville, 2011). In many cases, this affixation uses hierarchic embedding: Gears may float above a head to show thinking, but so too might motion lines surround the gears to show they are spinning. Thus, an affix (motion lines) attaches to an affix (gears) which attaches to a stem of a face.

All of these elements use combinatorial principles that are highly constrained and conventionalized, and their understanding is modulated by comic reading expertise (Cohn et al., 2016; Forceville, 2011; Newton, 1985). Often, these conventions are not recognizable without knowing their culture‐specific origins, such as nosebleeds depicting lust or a bubble out of the nose as indicating sleep, which come from Japanese manga (Cohn, 2013b), and now subsequently appear in the emoji used in messaging applications. While many visual morphemes use purely conventional meanings, others draw from metaphoric origins (Cohn, 2013b; Forceville, 2011; Szawerna, 2017). For example, gears spinning above a head evoke the idea of the mind as a machine, while steam out of the ears evokes the head as an overflowing pressurized container (Forceville, 2005).

To what extent is visual morphology constrained by spatial and conceptual relationships with other elements (e.g., faces), and to what extent does context modulate their understanding? Is their comprehension modulated by frequency of exposure, and/or culture‐specific familiarity? What are the combinatorial mechanisms that allow us to create composite meanings in a single visual representation, and how quickly is the understanding of new visual signs acquired? To what extent do comprehenders need to compute the metaphors that many of these forms evoke, or are they entrenched in memory? Do cultures use conceptual metaphors in visual morphology that are consistent with metaphors in their languages?

Narrative structure

Though we extract meaning from a visual narrative, as will be discussed below, that meaning is organized by patterns in the sequence itself—its . For example, Fig. 1 follows a fairly standard narrative arc, with the opening panels establishing the broader context and initiating the primary events, before resolving the sequence at the final image. Interestingly, this sequence lacks a depicted climax, the events of which are instead left inferred (i.e., the dog being stuffed into the fishbowl). Inferences like this are interesting for mental model construction, as discussed below, yet they also arise with intent: The author chose to not show the key information of the sequence. This means that such an omission has a structural purpose, not arising merely by happenstance, and such structure can thus be characterized in addition to the meaning‐making.

This narrative structure provides constraints on the ordering and organization of information in a visual discourse. The schematic properties of such narrative structures may be modality independent, but growing evidence suggests that drawn narratives may use patterning unique to the medium (Cohn, 2013b, 2013c; Cohn & Kutas, 2017). Because it provides a combinatorial system that mediates the expression of meaning in visual sequences, recent work has argued that the narrative structure of sequential images follows constraints similar to that of a grammatical structure in language (Cohn, 2013c). Functionally, like syntax in language, this narrative structure in visual narratives can be viewed as processing instructions to the cognitive systems responsible for building situation models (Christiansen & Chater, 2016; Givón, 1993). Structurally, also like syntax, narrative imbues units (images) with categorical roles in relation to a sequencing schema, which organizes them into a recursive, hierarchic structure that can be further altered using modifiers (Cohn, 2013b, 2013c). Such a “narrative grammar” expands on earlier discourse theories of “story grammars” (Mandler & Johnson, 1977) which were characterized by older Chomskyan models of syntax (Chomsky, 1965), with ambiguous relations between syntax and semantics (Black & Wilensky, 1979). This newer approach is consistent with contemporary models of construction grammar that motivates sequencing through stored schemas rather than procedural rules (Culicover & Jackendoff, 2005; Goldberg, 1995), with a clear separation between narrative and semantic processing (Cohn, Jackendoff, Holcomb, & Kuperberg, 2014; Cohn et al., 2012). Growing behavioral and neurocognitive research has supported the basic constructs of this narrative grammar, like the presence of narrative categories (Cohn, 2014; Magliano et al., 2017), constituent structure (Cohn & Bender, 2017; Cohn et al., 2014), and entrenched narrative patterns beyond just the canonical arc (Cohn & Kutas, 2017).

Experimental research measuring the electrophysiology of the brain has bolstered the claims of similarities between the narrative structure operating in sequential images and the syntactic structure in sentence processing (for review see Cohn, 2019b). Manipulation of the narrative grammar evokes neural responses consistent with combinatorial structure and/or structural predictions (anterior negativities, often left lateralized) and with processes associated with syntactic revision (P600s) (Cohn et al., 2014; Cohn & Kutas, 2015, 2017). These comparisons suggest similarities in the processing both across domains (visual images, speech/text), and across levels of information structure (sentence level, narrative/discourse level), perhaps pointing toward more abstract aspects of sequence processing in general (Christiansen, Conway, & Onnis, 2011; Patel, 2003).

Recent efforts using computational methods have also begun analyzing visual narrative sequencing (see Laubrock & Dunst, 2019). Some marginal results have been found by convolutional neural networks in recognizing sequencing structure in four‐panel comic strips (Ueno & Isahara, 2017; Ueno, Mori, Suenaga, & Isahara, 2016). More extensive analyses have attempted to use deep neural architectures to predict aspects of narrative and inference in a corpus of over a million panels from almost 4,000 classic American superhero comics (Iyyer et al., 2017). In all cases, the computational methods greatly underperformed human assessments, again suggesting that visual narrative structure—even in short four‐panel strips—has complexity beyond what computational models currently allow.

This research overall motivates us to ask, what are the representations and structures that guide visual narrative sequencing? To what extent does visual narrative share structural principles and processing mechanisms with those found in other domains (e.g., sentence processing)? Do narrative schemas manifest in similar ways across modalities? How does narrative structure interact with meaning‐making and situation model construction?

Conceptual/Event structures

As an expressive and communicative modality, visual narratives convey meanings, both within their units and across sequences. Individual images convey referential information about entities and locations, along with basic information about the events they undertake (i.e., their actions and experiences). These events extend further across sequences, where they are connected into broader event structures. Similarly, narrative text convey events within and across sentences and as such, readers need to compute the meaning of sentences and establish their semantic relationship to the prior discourse (Graesser, Millis, & Zwaan, 1997). Understanding narrative, whether verbal or visual, requires one to build situation models that reflect an underlying structure of the depicted events which unfold in narrative time and space (van Dijk & Kintsch, 1983; Gernsbacher, 1990; Zwaan & Radvansky, 1998).

The literature on narrative text processing has stressed the construction of a situation model of the growing understanding of a narrative episode, that is, characters performing goal‐directed actions in a narrative time and space (Zwaan & Radvansky, 1998). Situation models represent the agents involved in the narrative, the events of the narrative, the spatial temporal locations where these events occur, and how these events are related, in space, time, and causality (Zwaan & Radvansky, 1998). Building situation models requires that understanders monitor continuity in these dimensions, and update their representations when discontinuities are experienced (Gernsbacher, 1990; Zwaan & Radvansky, 1998). As a reader progresses through a visual sequence, changes in dimensions of characters, spatial location, and events trigger the need to update a situation model with new information (Cohn & Kutas, 2015; Loschky et al., 2019; Magliano et al., 2012; Radvansky & Zacks, 2014; Zwaan & Radvansky, 1998).

Certainly, perceptual changes across drawn sequential images play a role in signaling that updating is required (Loschky et al., 2019; Magliano et al., 2012; Radvansky & Zacks, 2014), but remarkably little research has explored how this might be the case (Loschky et al., 2019). Yet, as discussed above, even connecting basic referential information across images requires a “fluency” in sequential image processing (Byram & Garforth, 1980; H. F. Duncan et al., 1973; Ildirar & Schwan, 2015; Liddell, 1996), meaning that perceptual processing alone cannot account for tracking information across images.

Visual narratives also involve meaning‐making processes beyond monitoring continuities in situational dimensions (Magliano et al., 2013), such as various types of inferencing (McNamara & Magliano, 2009) and conceptual metaphors (Forceville, 2016). Consider the bridging inference used in Fig. 1 to understand the primary events of the sequence. In panels 2 and 3, the cat peers into a fishbowl, potentially implying that it wants to eat the fish. Yet, in panel 5, we see the dog stuffed into the fishbowl. The primary event is not depicted—the inferred actions of the cat stuffing the dog into the bowl, thereby changing our conception of why the cat looked in the bowl in the first place (with ill intent toward the dog, not the fish). Understanding this comic arguably requires inference for how the dog got into the fishbowl, so that coherence can be established between the final and preceding panels (Cohn & Kutas, 2015; Hutson, Magliano, & Loschky, 2018; Loughlin et al., 2015; Magliano et al., 2017).

As discussed above, inferences like this require that a reader reconcile sequencing that otherwise makes little sense in its surface form. As in the study of discourse (McNamara & Magliano, 2009), inference has been a primary focus in visual narratives since early in their study (McCloud, 1993). Readers are indeed sensitive to omitted information, and such costs are evident in both behavioral and neurocognitive measures (Cohn & Kutas, 2015; Cohn & Wittenberg, 2015; Magliano et al., 2017; Magliano et al., 2015). Such work again suggests connections across modalities, as similar brain responses are implicated to inferences in visual as in verbal narratives (Cohn & Kutas, 2015, 2017), as are similar memory systems (Magliano et al., 2015). Nevertheless, modality‐specific processes may also operate while facilitating inference in visual narratives. For example, Hutson et al. (2018) showed that, while constructing bridging inferences in visual narratives, readers engage in attentional selection not afforded by other narrative media, such as text or film. These findings raise questions about the domain‐specificity and generality of mental model construction in visual and verbal modalities (Cohn, 2013b, 2019; Magliano et al., 2019; Magliano et al., 2013). To what extent does visual narrative use modality‐specific processing, and to what extent are they shared with other modalities?

Multimodality

Finally, in addition to structures of sequential images in isolation, they also unite with other domains in multimodal interactions. While considerable research has examined multimedia expository information (Mayer, 2009), surprisingly little cognitive research has been done on multimodal visual narratives. In film, both the visual and linguistic (and musical) streams of information support situation model construction (Magliano, Dijkstra, & Zwaan, 1996), while static, drawn visual narratives typically integrate text and images. This raises interesting interactions between multiple streams of information that construct a holistic meaning (Cohn, 2016).

On the front end of processing (extracting information from and directing attention to panels/pictures), comprehending such interactions between text and images thus places demands on the attentional and perceptual systems (Magliano et al., 2013; Mayer, 2009), which can affect how readers negotiate between these information sources (Kirtley, Murray, Vaughan, & Tatler, 2018; Laubrock, Hohenstein, & Kümmerer, 2018). On the back end, further cognitive mechanisms must integrate the meaning and narrative structures of (at least) two streams of information (Cohn, 2016; Magliano et al., 2013; Mayer, 2009). This opens up questions about how much and at what level each modality is processed separately or integrated. Prior work has indeed shown that visual sequences can modulate the comprehension of written and spoken language (Manfredi, Cohn, De Araújo Andreoli, & Boggio, 2018; Manfredi, Cohn, & Kutas, 2017), just as sentence contexts can modulate the meaning of images (Ganis, Kutas, & Sereno, 1996; Weissman & Tanner, 2018).

An inherent challenge of studying the multimodality of visual narratives is the complex relationship between linguistic and visual content (Cohn, 2016). Consider the strips in Fig. 3a and b, both from Journal Comics by Drew Weing, which use captions above fairly illustrative images. Both strips use a “balanced” combination of text and image where both modalities contribute substantially to the multimodal whole. Yet, when the text is omitted, it should be apparent that they differ in their sequential contributions: Fig. 3a/c has a dedicated sequence to the images alone (a temporal order), while Fig. 3b/d does not (a semantic field about smells). One could thus readily rearrange the images in Fig. 3d, but rearrangement in Fig. 3c would change the overall gist. Thus, in building a situation model from both visual and verbal content, readers must negotiate varying types of combinatorial relations between images as well as between modalities.

Figure 3

Comic strips (a and b) from Journal Comic (© Drew Weing) which balance the meaning between text and images, but differ in terms of their sequencing: a/c uses a dedicated order (temporal) while b/d uses a semantic field about smells, which could be rearranged.

Exploring these connections between verbal and visual streams of information opens up interfaces with established studies of multimodality, such as co‐speech gesture (Goldin‐Meadow, 2003; McNeill, 1992)—does multimodal processing take similar characteristics across different integrated modes? How are these information sources processed and represented to convey meaning? How does the nature of different types of multimodal relationships affect processing, representation, and durability in a mental model? Such questions can well interface with education research, which has already explored issues of multimodality (Mayer, 2009), and where visual narratives have growing advocacy (e.g., Hosler & Boomer, 2011; Wong, Miao, Cheng, & Yip, 2017). Finally, given that corpus research on comics has shown that multimodal interactions can change over time in systematic ways (Cohn, Taylor, & Pederson, 2017), and processing differs based on cross‐cultural narrative patterns (Cohn & Kutas, 2017), might readers habituate to processing multimodality in particular ways based on exposure?

Sequential image processing is not uniform

In line with visual narratives’ representational levels discussed above, the cognition operating on sequential image processing is multilayered, and not uniform. Studies using event‐related brain potentials (ERPs) have directly measured the neural activity of participants comprehending sequential images (for review see Cohn, 2019b). Here, a complex interplay arises between various cognitive mechanisms, such as semantic access indexed by N400 effects (Coderre et al., 2018; Cohn et al., 2012; West & Holcomb, 2002), or integration and updating processes indexed by P600 effects (Cohn et al., 2014; Cohn & Kutas, 2015; Cohn & Maher, 2015). Other effects have been implicated related to combinatorial processing and/or working memory, such as anterior negativities (Cohn et al., 2014; Cohn & Kutas, 2015, 2017), and sensitivity to frequency effects suggested by frontal positivities (Cohn & Maher, 2015; Manfredi et al., 2017). These findings suggest that visual narratives are not simply reducible to one type of processing (ex. updating or inference), but, like most complex cognitive phenomena, evoke multiple interacting mechanisms.

In addition, these neural responses ostensibly appear to overlap with sentence processing and music (Kaan, 2007; Patel, 2003)—outside the context of narratives—thereby suggesting that visual narratives tap into fairly general cognitive mechanisms. Similar overlap in cognitive processing is also implicated for memory systems and segmental processes using behavioral methods (Magliano et al., 2012; Magliano et al., 2015). As noted above, only a few studies have directly compared the cognitive processing of visual narratives (films or drawn narratives) to text‐based versions, while attempting to control for narrative content (i.e., attempting to present the same events). These studies have typically explored claims that aspects of comprehension are similar across media, and indeed consistencies do appear in the processing of different versions of a story (Baggett, 1979; Coderre et al., 2018; Magliano, Clinton, O’Brien, & Rapp, 2018; Magliano et al., 2012).

Given that visual narratives have often been used for diagnosis and treatment, additional clues about their processing come from various clinical populations. Damage to the frontal lobe has long been shown to impair visual narrative sequencing (e.g., McFie & Piercy, 1952), as has selective damage to both the left and right hemispheres (Bihrle, Brownell, Powelson, & Gardner, 1986; Fucetola, Connor, Strube, & Corbetta, 2009; Huber & Gleber, 1982; Marini, Carlomagno, Caltagirone, & Nocentini, 2005; Tinaz et al., 2006). Recent ERP work has shown attenuation for processing semantic incongruities (N400) in both verbal and visual narratives for individuals with Autism Spectrum Disorder (ASD) compared to neurotypical controls (Coderre et al., 2018). Additional deficits in comprehension and sequencing of visual narratives have also been shown by individuals with Developmental Language Disorder, formerly known as Specific Language Impairment (Bishop & Donlan, 2005; Nenadović, Stokić, Vuković, Đoković, & Subotić, 2014), a diagnosis assigned to children with delayed language development, particularly of syntax, who still maintain proficiency at non‐verbal IQ tests. Such results further imply connections between the neural architecture of language and visual narratives, and call into question presumptions that all visual materials may be easier to process than verbal ones for various clinical populations (Coderre, 2019).

Overall, this work suggests that the processing of visual narratives may overlap with that of other media (e.g., Gernsbacher, 1990), and may overlap with different levels or types of expressive systems, not just narratives (e.g., sentence processing, music). However, this research by no means can be characterized yet as systematic in exploring the various sources of complexity described above. It does not preclude the possibility that important differences could be attributed to specialized knowledge, processes, or affordances that are unique to the visual modality (Cohn, 2013b; Magliano et al., 2019). Such work thus raises important questions balancing these concerns: What are the brain regions and mechanisms involved in sequential image understanding? How much overlap is there with other, domain‐general mechanisms? To what degree does visual narrative processing balance those that are domain‐general and those that are domain‐specific?

Sequential image comprehension is culturally variable

Contrary to the myth of transparency, visual narrative systems differ in many ways, even when setting aside the modality differences of filmic and drawn narratives. For example, comics and illustrated picture stories vary in many conventions (layout, morphology, narrative structure), despite both conveying meaning across sequential images. Within works labeled as “comics” around the world, systematic variation arises across most all levels of structure. Corpus analyses have suggested cultures’ comics differ across narrative patterns (Cohn, 2019a), page layout (Cohn, Axnér, Diercks, Yeh, & Pederson, 2019), motion events (Cohn, Wong, Pederson, & Taylor, 2017), and others. In addition, visual narrative systems have been observed to change over time in dimensions like layout, narrative, and multimodality (Bateman, Veloso, & Lau, 2019; Cohn, Taylor, et al., 2017; Pederson & Cohn, 2016). Other systems vary in ways significantly different from those used in comics, like the visual narratives drawn in the sand by Central Australian Aboriginals (Green, 2014; Wilkins, 1997). Overall, such observations parallel with language: While there may be abstract similarities and typological constructs underlying visual narratives of the world, they manifest in distinct and patterned systems used by a particular population.

While the iconicity of images aids in cross‐cultural understanding, fluency is modulated by knowledge of specific visual narrative systems, and knowledge of one visual narrative system does not grant full fluency in others. For example, the aforementioned sand narratives used by Central Australian Aboriginals use highly conventionalized representations that are likely opaque for those unfamiliar with it (Green, 2014; Wilkins, 1997), while some individuals from this community also had trouble understanding the sequencing of Western comics (Wilkins, 1997). In addition, a study among self‐described “comic readers” found that neural responses to a pattern found more prevalently in Japanese manga is modulated by readership of those works (Cohn & Kutas, 2017).

Such findings raise questions about the nature of cross‐cultural variability of visual narratives and the fluency associated with them: Do visual narratives across cultures and history share “universal” features and or underlying typological regularities? To what degree is visual narrative comprehension contingent on familiarity with visual narrative systems in general or with specific systems given a particular genre, culture, and/or historical context? To what degree does the typological structure of a visual narrative system overlap with the structures in other expressive capacities, like verbal or signed languages? How does cross‐cultural diversity relate to the identity of their producers and the sociocultural treatment of different types of images?

Sequential image comprehension requires a fluency, learned across a developmental trajectory

Finally, in discussing myths, we made the case that being a fluent reader of drawn sequential narratives requires exposure. In many cultures, children’s first exposure to reading is through multimodal visual narratives, and various works have suggested that visual narratives might be beneficial for literacy, language learning, and inferences (Kendeou et al., 2019). Yet, as discussed above, understanding of visual narratives themselves does not come “for free” with visual perception and event cognition, and requires a fluency developed with exposure and practice. While foundational research has suggested that understanding sequential images moves through identifiable steps in development, we still know very little about this process in detail: How much exposure and practice are necessary? What are the stages of visual narrative understanding? How does this development balance general cognitive principles and culturally specific patterns?

In addition, there is often an asymmetry between visual narrative comprehension and production—far more people can comprehend sequential images than can create them (Wilson, 2016). Important work has shown that exposure to visual narratives—and especially copying existing systems—is crucial for developing proficiency in creating visual narratives (Stoermer, 2009; Wilson, 1988; Wilson & Wilson, 1977). Yet little is understood about the trajectory of this learning, how it lines up with the development of comprehension, and the effects of such proficiency on other systems, like literacy and intelligence.

All of these issues also raise questions about the development of visual narrative comprehension relative to other domains, like the concurrent development of Theory of Mind and language. To what degree does the development of these domains line up with each other? To what degree is visual narrative understanding (in)dependent of other aspects of cognition?

Conclusion

Throughout, we have emphasized that visual narratives reflect fundamental aspects of human meaning‐making, with complexity beyond what is often presumed. This complexity deserves a focus of its own within the broader study of human cognition, and should not be stifled by cultural myths of its transparency or universality. At the same time, though, the complexity of visual narratives affords fundamental questions across Cognitive Science. Scholars in a variety of sub‐disciplines of Cognitive Science (event cognition, scene perception, psycholinguistics, clinical psychology, literacy, narrative comprehension, communications, cultural psychology, and more) have begun developing viable programs of research on this topic. We are at a crucial juncture where these efforts could either converge into a new and emerging sub‐field of Cognitive Science, or these efforts could simply continue as fractionated research across several sub‐disciplines.

How do we prevent such balkanization? One solution is to establish a common understanding of the problem we’re trying to understand. We believe the framework presented here provides one approach by giving access points for identifying problems that researchers can address. As discussed above, the framework outlines that visual narratives express meaning through a graphic modality, which is organized using combinatorial structure that operates at both the unit and sequence level. This complex framework manifests in culturally different ways, is multimodal in nature, is learned across a developmental trajectory, and recruits non‐uniform processing mechanisms. In the context of this framework, we posed questions about these issues that we find interesting and worth pursuing, but by no means should be considered exhaustive.

Another potential solution is proposing theories that afford intersections between different areas of Cognitive Science. We believe our proposed framework makes the case that this is warranted. Several emerging theories of visual narrative processing range in the extent to which they accomplish this agenda, spanning fields of psycholinguistics and cognitive neuroscience (Cohn, 2013b, 2019b), scene perception and event cognition (Loschky et al., 2019), attention (Smith, 2012), developmental psychology (Nakazawa, 2016), multimodal discourse (Bateman & Wildfeuer, 2014), and computer science (Augereau, Iwata, & Kise, 2018), among others. However, we have intentionally not discussed these theories herein. Rather, we wish to inspire scholars to test these models and/or develop their own, but hope that said theories are sensitive of the need to cut across different areas of research on cognitive processing. In doing so, we believe that we will not only learn about how visual narratives are processed, but can also learn about how the brain and mind are coordinated to make meaning across domains.