Hierarchical processing in music, language, and action: Lashley revisited.

Sixty years ago, Karl Lashley suggested that complex action sequences, from simple motor acts to language and music, are a fundamental but neglected aspect of neural function. Lashley demonstrated the inadequacy of then-standard models of associative chaining, positing a more flexible and generalized "syntax of action" necessary to encompass key aspects of language and music. He suggested that hierarchy in language and music builds upon a more basic sequential action system, and provided several concrete hypotheses about the nature of this system. Here, we review a diverse set of modern data concerning musical, linguistic, and other action processing, finding them largely consistent with an updated neuroanatomical version of Lashley's hypotheses. In particular, the lateral premotor cortex, including Broca's area, plays important roles in hierarchical processing in language, music, and at least some action sequences. Although the precise computational function of the lateral prefrontal regions in action syntax remains debated, Lashley's notion-that this cortical region implements a working-memory buffer or stack scannable by posterior and subcortical brain regions-is consistent with considerable experimental data.

Introduction

In 1951, Karl Lashley1 suggested that the human capacity for serial ordering of action, whether of language or music or simply making coffee, presents central problems for our understanding of brain function. Lashley observed that then-prevailing models of action based on simple stimulus–response chaining had a fundamental flaw: although such serial models can capture the “one thing after another” characteristic of routine action, they fail to recognize the central importance of sustained goals and subgoals in more complex action. In complex actions, overall goals (e.g., making coffee) must persist over time while subgoals are initiated and completed (e.g., grinding beans, heating water, and adding cream) or interruptions occur (e.g., a phone call). Several aspects of such action sequences are inconsistent with a chaining model. Most obvious are errors of omission, where some step is accidentally left out (e.g., starting the coffee machine without first adding coffee). In chaining models, each completed action serves as the stimulus for the next one, and omission of any step would cause the sequence to grind to a halt. This and other phenomena, argued Lashley, demonstrate the need for hierarchical models of action planning—a requirement widely acknowledged today.2

Lashley further observed that core aspects of spoken language and music share this need for hierarchical structure, at multiple levels. For example, when pronouncing the words tire and right, the same basic vocal gestures appear but in different orders. How could a chaining system cope with such flexible rearrangement of basic actions into multiple larger composites? Phrasal syntax makes the point even more clearly. When reading the sentence, “The boy who patted the dog chased the girl,” an English speaker knows that the boy, and not the dog, chased the girl, despite the fact that this sentence contains the sequence “the dog chased the girl.” Linguistic syntax illustrates that serial order alone provides an inadequate basis for language processing, and that processes involving longer temporal scales must persist during the processing or production of shorter actions or simple chains. Such observations were foundational in early generative linguistics, which recognized the central role of phrase structure, rather than word order, in syntax.3,4

Lashley also suggested that these similarities in language, music, and other complex actions are not coincidental, instead hypothesizing that the hierarchical nature of music and language was inherited from more basic features of motor planning and action.

In this review, we reassess Lashley's observations and hypotheses from a modern perspective, concluding that his ideas were remarkably insightful and consistent with considerable new behavioral and neuroscientific data. We explore Lashley's idea that hierarchical structuring of temporal sequences is a key capability underlying human music and language, but one that has deeper phylogenetic roots in the domain of action. Unlike Lashley, who eschewed localizationist arguments, we make full use of brain imaging and lesion data as supporting evidence, with a particular focus on the role of Broca's area in processing hierarchical structures in the time domain.

Specifically, we will consider the following issues: (1) whether processing of sequential hierarchy is a well-defined specific capability, independent of other hierarchical processing (e.g., visuospatial) but applicable to both music and language; (2) to what extent the mechanisms subserving linguistic and musical syntax inherit properties from those typifying a syntax of action more generally; and (3) whether humans are unusual in the degree to which hierarchical abilities are developed, and if so, which neural mechanisms support this proclivity.

We evaluate these questions using behavioral, neuroanatomical, and brain-imaging data that go well beyond the data available in Lashley's day. We conclude that these data converge upon the notion of a distinct neural substrate for hierarchical sequence processing, unusually well developed in humans relative to other primates, and reflected in specific expansions of prefrontal brain regions including Broca's area, as well as a considerable remodeling of connections between prefrontal and posterior regions. Although the degree to which music and language inherit key properties from action syntax remains controversial, available data are consistent with overlap in these systems from both neural and computational viewpoints. More precise computational models and experimental designs will be required to further evaluate this hypothesis, which is quite different from currently popular hypotheses based upon mirror neurons and embodiment. Thus, 60 years on, Lashley's insights remain valid and intriguing and can inform current neuroscientific debates, and thus deserve renewed attention.

Defining temporal hierarchy: hierarchical sequences and hierarchical sets

Terms like syntax and hierarchy are notoriously ambiguous, so our first task in evaluating Lashley's hypotheses is to distinguish possible interpretations of the term hierarchy, which has many meanings in neuroscience. Especially in older literature, hierarchy connotes little more than the fact that cortical regions are above or below one another in processing terms, such that bottom–up processing proceeds from regions lower to regions higher in the visual hierarchy. A central characteristic of the visual system is that neurons in cortical regions closer to retinal input, such as V1, have smaller receptive fields than those in higher regions like V2. Our focus goes beyond this basic and widespread form of hierarchical layout of cortical regions themselves.

Here, we adopt a more specific and precise notion of hierarchy to denote a tree-like organization, where higher levels incorporate multiple lower levels in structural representations and/or processing. This notion is wholly independent of how the cortical regions that process such stimuli are arranged, entailing no specific commitments about neuroanatomical localization.

To be more exact we utilize the following mathematically-based definitions5,6 throughout this paper: a set is an unordered collection of distinct, unique objects; while a sequence is a collection of objects, perhaps including duplicates, ordered by some rule.

Although the set {a, b, c} is identical to the set {b, c, a}, the sequences [a b c] and [b c a] are distinct and different. Furthermore, because sequences but not sets can contain duplicate items, sequences are not, strictly speaking, a type of set. These distinctions are crucial in distinguishing Lashley's temporally ordered sequences, where order matters, from static hierarchy, in which order is typically irrelevant.

Hierarchy denotes a set or sequence of elements connected in the form of a rooted tree (a connected acyclic graph, in which one element is singled out as the root element). Hierarchies thus possess the following key properties: (1) all elements are combined into one structure (connectedness); (2) one element (the root) is superior to all others; and (3) no element is superior to itself (that is, there are no cycles, direct or indirect).

These definitions allow us to clarify a core distinction between temporally ordered and unordered hierarchies. Examples of hierarchical sets include visuospatial part–whole hierarchies, such as the schema of a face, in which certain components contain others without any necessary sequential ordering of components at each level. A face incorporates the set {eyes, nose, mouth}, but there is no need to consider one component to come first. Such hierarchies involve the minimal structure required to consider a representation hierarchical, namely that the superior/inferior relation between levels is specified. But at any given level, the elements form an unordered set.

In hierarchical sequences, in contrast, sequential order matters. The simple two-syllable utterances “callow” and “low-cal” contain the same set of syllables, but arranged in different orders, and they represent different concepts. Similarly, in music each arrangement of a set of notes constitutes a different melody. Finally, in action sequences, order is often important: if we grind the coffee beans at the end of the action sequence, after brewing is completed, poor coffee results. Thus temporal hierarchies incorporate an additional ordering component, where at least some elements at any given level represent a sequence rather than a set.

Lashley considered hierarchical structure to be an obvious component of language, music, and action. In each domain, some superordinate structure (his “determining tendency”) is required to lend coherence to subordinate components (speech sounds, notes, or simple actions). Although these higher level elements may differ considerably between domains (e.g., meaning for language versus an overall goal for action), Lashley's focus was what all three domains have in common: the need to impose the correct temporal ordering on the subelements. He cites, as a key example, language, where the elements of meaning are often cotemporal sets, but different languages have different ordering rules by which the same meaning must be expressed in a sequence: different “schemas of order.” Typing or speech errors (e.g., spoonerisms) or children's games like pig Latin provide evidence that the temporal ordering component is both independent of semantics, and a general characteristic of many “generalized schemata of action” that apply across the three domains. Thus, it is not the general problem of hierarchy but the specific issue of hierarchical sequencing that Lashley considered the essential problem of serial order, and that will be our focus in this review.

Lashley observed that we can often convert between spatial and temporal hierarchies. For instance, to reverse a piano melody, he stated, “I can only do it by visualizing the music spatially and then reading it backward” (p. 129). Similarly, when translating between languages with different word orders, such as English and German, a fluent bilingual can extract the meaning from one language, and convert it to a properly sequenced sentence in the other, freely reordering the individual words as needed. It is therefore possible that the distinction between sequential and nonordered hierarchy is of little importance at a fundamental neural or computational level. This is the first issue that we address using modern imaging data.

Sequential hierarchical processing is not colocalized with visual hierarchy systems

As noted above, there is often an increase in spatial receptive field size in cortical regions more distant from sensory input; this involves a widespread neural form of spatial hierarchy, where higher regions integrate information from multiple lower regions.7 In this functional sense, hierarchical processing is a widespread property of brain systems.8–16

In some cases, such patterns of neuroanatomical layout may reflect a functional form of cognitive hierarchy. For instance, in the motor domain, posterior frontal lobe regions encode basic movements (extending the arm), whereas anterior regions fire more selectively to particular contexts in which these movements are performed (e.g., making coffee and shaking hands).10 In the auditory cortex, core areas respond to simple acoustic characteristics, whereas anterior regions along the supratemporal plane display selective firing to more complex sound categories.12 In the visual ventral stream, anterior regions in the temporal cortex support categorical decisions of greater abstraction (e.g., animal versus nonanimal) than posterior regions (e.g., blue versus green).14 Given these pervasive patterns of brain organization, is there evidence for more abstract and modality independent hierarchical structuring in the brain?

Starting with hierarchical sets, many recent studies explore the brain regions involved in representing and processing hierarchical structures in both spatial and social domains.17,18 Representation of hierarchical structures in the social domain, which allows the evaluation of dominance relationships, appear to be encoded in the hippocampus.18 However, this structure is also active in encoding hierarchical ranks in nonsocial domains.18 In the visuospatial domain, for example, correct integration of landmarks within context frames recruits the parahippocampus,8 and also requires the integrity of the medial temporal lobe (MTL) and the retrosplenial cortex.17

Interestingly, this same MTL system may also encode high order hierarchical associations in the motor and linguistic domains.19–22 It has been proposed that these medial structures are important for binding items within contexts,23,24 even for offline imagined scenes,25 as well as for episodic memory, which involves binding different perceptual and memory features into unified representations.26,27

In addition to these brain regions, a quite different set of activations is typically observed in temporal hierarchical processing, regardless of whether these involve musical, linguistic, or other tasks. A consistent finding within the neurolinguistic literature is that sequentially structured hierarchical processing activates the posterior prefrontal cortex, centering on Broca's area (a term we will use hereafter to denote both Brodmann's areas [BA] 44 and 45, typically in the left hemisphere, but not neighboring areas28). Increased Broca's activation typically correlates with increased demands on working memory, for example, when processing long-distance dependencies, where early information must remain active for proper interpretation of later elements. Ongoing controversy concerns the degree to which this active maintenance of past information is specific to hierarchical sequences per se22,29,30 or is found for nonhierarchical sequences as well,31,32 but it is clear that increasingly complex hierarchical structures lead to greater Broca's activation across multiple languages and laboratories.

Furthermore, in musical syntax, hierarchically dependent processing can stretch over long time spans (long-distance dependency).33 Neuroimaging studies on musical syntax (as probed with sequential harmonic or melodic structures) also consistently reveal activation in BA 44/45, typically in both the right and left hemispheres, but sometimes biased to the right side.34–39 Evidence that these prefrontal activations are not restricted to meaningful language or music comes from artificial grammar research using nonsense syllables (spoken) or words (written) arranged according to particular rules. Again, irrespective of the input domain, both auditory and visual patterns, and even nonlinguistic visual symbols,40 elicit Broca's activation when they are sequentially presented. On the basis of these data, further explored below, we conclude that, despite MTL overlaps, the neural mechanisms for processing hierarchical sequences are not fully colocalized with those that process hierarchical sets. Despite formal similarities in these abstract hierarchical structures, these data argue against their identity. We now explore the functions of the prefrontal regions preferentially involved in hierarchical sequence processing.

Prefrontal cortex and Broca's area play central roles in processing hierarchical syntax

We first attempt to characterize the capacity for processing sequential hierarchies more precisely, to better understand its neural basis. Although Lashley was a confirmed agnostic regarding neural localization, there is a long tradition of associating the frontal lobes, specifically the prefrontal cortex (the portion of the frontal lobe anterior to the primary motor strip and the premotor cortex), with the organization of complex plans in general, and hierarchical sequences in particular, in multiple cognitive domains. The notion that the prefrontal cortex is specifically involved in action planning and executive control has been advanced notably by Luria,41 Shallice,42 and Baddeley,43 among many others. Clinical data leave little doubt that pathology in the frontal lobe can lead to disorganized action.44 We do not review this extensive literature here, but simply take it for granted that the frontal lobes play important roles in planning, working memory, and executive control. Our goal will be to specify this contribution more precisely, paying particular attention to the role of the inferior frontal gyrus (IFG) and Broca's area.

We note first that virtually any motor behavior can be conceptualized as hierarchical. A monkey eating peanuts involves a set of actions (grasp, ingest, chew) each of which can be further broken down (grasp: extend arm, open fingers, close fingers, retract arm). Thus, any action sequence that involves repeated subparts displays a simple form of hierarchy, pervasive to all motor action in all species.45 Such hierarchies have a fixed and limited depth, and were not Lashley's focus (because they can be accounted for by chaining). To clarify what he was getting at, we start with the clear case of language, and use this to guide our further inquiry through music, back to action.

Broca's area (BA 44 and 45 or, roughly, the pars opercularis and the pars triangularis of the IFG) is one of the most intensively studied brain regions, and its importance for speech production requires little emphasis.46–48 Since the 1970s, this region has also been recognized in clinical studies to play a special role in syntax, both production and perception.49 Thus, a broad consensus exists that this region of the IFG plays an important role in language processing.30,50–52

An elegant study by Pallier et al.53 explored the specificity of IFG involvement in hierarchical syntax by crossing two manipulations. In the first, they parametrically varied the chunk size of 12-word sentences, spanning from scrambled words (where there are no chunks, giving a constituent size of one) to full sentences (constituent size = 12). Intermediate strings had constituent sizes of two, three, four, and six words (an example with four-word chunks would be “mayor of the city/he hates this color/they read their names”). The prediction was that activation in a region specifically encoding constituent structure would increase with greater constituent size. Such a correlation was indeed found, including both IFG structures and temporal lobe areas, especially along the superior temporal sulcus.

To get at syntax specificity, a second manipulation involved substituting all content words with nonsense words to create “jabberwocky” constituents of varying sizes.53 For example, the four-word jabberwocky could read “tuyor of the roty/he futes this dator/they gead their wames.” This parametric manipulation removes the effect of semantic integration, but again elicited specific correlated activation in the IFG (centered on BA 45). The clever design controls for both semantics and word transition probability (because the nonsense words, by definition, have zero probability), and confirmed a specific role for Broca's area in processing hierarchical constituent structure in language.

Broca's region is involved in processing musical harmonic syntax

Regarding specificity of function, however, there has been considerable debate about the degree to which the function of Broca's area is specific to syntax, or whether it performs one or more computational functions that are shared by multiple aspects of language, including phonology or semantics,50 or even more broad cognition including cognitive control, action planning, and music.35,54–56

The best characterized, and to us most convincing, nonlinguistic function of the IFG is in music processing and memory.57–60 There is a long tradition, based on dissociations in lesion patients, of considering music and language to be instantiated in distinct brain regions.61 Initial evidence for a shared role for the IFG in musical processing, particularly processing harmonic syntax, came from event-related potential (ERP) and magnetoencephalography (MEG) studies.36,37,62 Furthermore, there are multiple studies indicating overlap of both cognitive and neural resources used in music-syntactic and language-syntactic processing based on interactions between the two domains.63–65 These studies showed consistent similarities in early, anteriorly located reactions to anomalous items in linguistic and musical stimuli, and the MEG study by Maess et al.37 concluded simply that “musical syntax is processed in Broca's area.”

These results, from normal subjects, stand in contrast to the long clinical tradition identifying patients with damage to music and not language, or vice versa.61 Recent commentators57,58 have reconciled this apparent discrepancy by noting that although certain abstract operations may be shared by music and language, the units over which these functions operate are clearly quite different (phonemes and words for language, notes and chords for music). Furthermore, there are fundamental differences among different components of music. For example, a key component of musical rhythm is isochronicity—the tendency of key elements to be equally spaced in time.66 This is not an aspect of rhythm shared by normal speech or action, although other aspects like metrical structure may represent hierarchical sequencing.67 Given these basic differences, we do not expect complete overlap of musical and linguistic processing, either neurally or behaviorally. Rather, specific parallels in the manipulations applied to musical and linguistic stimuli (e.g., violations of hierarchical structural expectations33) are needed to elicit parallel neural effects.

Despite the broad consensus concerning a role for the IFG in these aspects of music processing, both ERP and MEG studies have difficulties with localization, and these early findings have not gone unchallenged by more recent functional magnetic resonance imaging (fMRI) studies.68,69 A particular issue concerns group analysis of different subjects. It is possible that each individual in a study could activate distinct but neighboring regions for linguistic versus musical syntax. Because of individual variability in precise localization, averaging all these individually distinct activations would give a false impression of thorough overlap in the two domains.

A recent study probed this issue by first identifying language regions in individual subjects, and then examining activations elicited in those regions by various other tasks involving working memory, cognitive control, and music.70 This study used a sentence-reading task, contrasted with reading nonsense words, to delineate the language regions of interest (ROIs). Although the study found considerable specificity to language in IFG activations, a musical task also preferentially activated their language-based ROIs, in particular overlapping with activations found in the left posterior IFG for both the sentence-reading task and a verbal working-memory task. Although the authors concluded from their results that specific language areas exist, not activated by most other tasks, this is an overly general conclusion applicable only to their specific musical and linguistic tasks. It is also important to recognize that the Patel or Koelsch shared-resource models57,58 do not predict exact overlap for these two different tasks (reading visually presented sentences versus listening to melodies) given the different input modalities. Thus, the significant language-area activations that were seen for the musical task support Lashley's contention that these two domains share important processing subsystems.

A second source of apparently contradictory evidence involves rhythm perception. A series of recent imaging studies concerning rhythm perception paint a consistent picture:71–73 rhythmic stimuli tend to activate motor areas (such as the supplementary motor area, other dorsal prefrontal regions, or even the basal ganglia) involved in sequencing,74,75 and do not activate the IFG. But this work focuses on isochronous rhythms, and likely reflects the tendency of such rhythms to induce a desire to move to the regular beat—the so-called “groove” of the music.76,77 Precisely because this aspect of music is not shared with language, we do not consider such data to contradict Lashley's hypothesis or the well-established notion that the IFG plays a role in processing other components of musical syntax.

In summary, a considerable body of neuroimaging data, from many laboratories, is consistent with the hypothesis that the prefrontal cortex, and the IFG in particular, play important parallel roles in processing specific shared aspects of language and music. These data are consistent with both Lashley's early ideas and Patel and Koelsch's more specific hypotheses about the nature of the shared processing resources.

The role of frontocortical mirror neurons in understanding action

The third component of Lashley's triad is goal-directed action. There has been an explosion of interest recently in the role of the frontal lobes, and the IFG, in action understanding. This work typically makes no reference to Lashley's ideas, but was instead spurred by the discovery of mirror neurons in the macaque. Mirror neurons are neurons in motor and premotor cortex that fire, not only when a monkey executes an action itself, but also when it sees another agent perform the same action. The dual nature of the metaphoric mirror—in both production and perception—has led to revived interest in the mostly discredited motor theory of speech perception, which posits that speech perception is mediated by virtue of a resonance with the motor programs that generate the sounds (vocal gestures).78 By this hypothesis, the objects of speech perception are articulatory motor events, rather than acoustic or auditory events (for a critique, see Ref. 79).

Recent speculation concerning mirror neurons has extended the reach of the motor theory beyond its original domain of speech perception to language more generally. The region of macaque cortex where mirror neurons are found—F5—neighbors or overlaps the regions constituting the human Broca's area (BA 44/45). Many studies show activation of motor and premotor regions during language production and comprehension tasks,80,81 thought to support the notion that mirror neurons in this region play a central role in the parity of production and perception that typifies all aspects of language.82–84 This idea has been used to support gestural hypotheses for language evolution, whereby initial stages of linguistic syntax and semantics evolved in the context of a gestural protolanguage, rather than a vocal system.83,85–87

A detailed critique of mirror neurons or their possible importance in language evolution is not our purpose here (but see Refs. 88–92). However, it is crucial to distinguish Lashley's hypothesis about action syntax from those linked to mirror neurons or embodiment. Lashley's model concerns a general and abstract notion of sequential hierarchy, and is only tied to motor actions insofar as these demand planning and control of a particular type. Lashley's model has nothing to do with grasping or recognizing graspable objects, and indeed involves structuring entities for which no such low-level action schemata are available. When applied to the role of the IFG, this leads to different empirical predictions from those of mirror-system hypotheses.

Action syntax: beyond mirror neurons

Because localization is central to mirror-based arguments, precision regarding specific regions of the frontal cortex is key to the following discussion. The cytoarchitecture and connectivity of neighboring frontal regions changes drastically over small distances. These properties range from agranular cortex in motor and premotor regions (BA 4 and 6), to dysgranular BA 44, to granular BA 45 and anterior frontal cortex. The motor cortex generates strong descending motor pathways (e.g., the corticospinal tract), but BA 44/45 are both truly prefrontal, and clearly functionally distinct from motor regions. With broad or imprecise spatial localization, or multisubject averaging of activations, it is easy to confuse (or conflate) activations in one of these regions with that of the others.

Starting with motor and premotor activations, brain-imaging studies reveal clear and consistent activations for certain classes of words or images connoting either direct actions (“throw” or “kick”) or tools providing affordances for action (“hammers,” “pliers”). Such motor-charged stimuli elicit (pre)motor activation; other more visually charged stimuli (e.g., animals) do not.80,91 Recent studies35 showing motor activation for particular components of language provide no special evidence for motor (or mirror neuron) involvement with language more broadly, because many linguistically structured concepts (color, clouds, beauty) have no motor component. Second, those few mirror neurons that have been directly documented in the human brain (during single-unit recording in 21 preoperative epilepsy patients) were found in the medial frontal (supplementary motor area) and the temporal (parahippocampal) cortex;93 it is therefore possible that mirror neurons are much more widespread in human brains than in macaques. Simple observation of IFG activation in the human brain therefore does not specifically implicate mirror neurons, and more direct and precise tests are needed to distinguish between general frontal involvement and mirror neuron involvement.

A key testable prediction of mirror neuron-based hypotheses follows from the fact that, by definition, mirror neurons only exist for motor actions the subject can actually perform itself. To the extent that mirror neurons are required to understand or process some observed action, this predicts that unfamiliar or unproduceable actions should not excite the mirror system, and should be uninterpretable (or at least difficult to process). This is rather difficult to test in the domain of language, because all normal humans are both producers and perceivers of language. Nonetheless, in brain-damaged patients with lesions in proposed sites of mirror neurons (the IFG and parietal regions), speech perception is not disturbed as predicted.94 This study only found impaired speech perception when lesions involved auditory regions in the temporal lobe, inconsistent with motor theories of speech perception and of the need for mirror neurons to support accurate speech perception.

Additional data do not support the prediction that motor familiarity is required to elicit appropriate processing of speech stimuli. An infant optical imaging study95 found frontal activation to speech-based patterns in newborns, despite their lack of experience in producing speech motor patterns themselves. Similarly, an artificial grammar-learning study by Bahlmann et al. employed abstract visual patterns, sequentially presented, which had no motor (or verbal) component whatsoever. Items were arranged into either context-free patterns favoring hierarchy or finite-state patterns favoring pure sequence. The authors found activation of BA 44 only for the first case, supporting a role for this region in sequential hierarchy in the absence of motor-action understanding.40

Perhaps the most direct test of the mirroring prediction is possible with music, where many humans are devoted music listeners but lack the motor capability to play this music themselves. Mirror neuron enthusiasts often cite studies indicating motor involvement and Broca's activation during music perception by expert musicians on their own instruments.35 But the crucial test is actually with nonmusicians, where the data are clear: Broca's activation is found during music perception, and musical syntax tasks in particular, even in nonmusicians, whose understanding of instrumental music presumably has little or no motor component.37,96 Furthermore, although many findings are consistent with the general notion that music perception relies partially on motor and action-related circuitry,60,72,75 most of the motor areas implicated are neither part of the putative mirror neuron system, nor specifically related to music production (e.g., singing or playing).

The dangers of rough localization and characterization are illustrated in an intriguing paper by Fazio et al.,54 which examined the abilities of patients with frontal aphasia to correctly understand event sequences. Previous work81 showed a particular deficit for prefrontally damaged patients for interpreting action. The Fazio study expanded this by first showing participants short video sequences of human actions or physical events and then presenting still images extracted from the videos on a touch screen. The patients’ task was to properly sequence these images by touching them (requiring no overt language). The six patients in this study showed a general deficit relative to healthy controls, but were more impaired for sequencing human actions (e.g., a man grasping a bottle) than physical events (e.g., a bicycle falling down). Superimposing the MRI-determined lesion locations of patients upon one another revealed that the only region consistently damaged in all patients was centered on BA 44. However, all but one of these patients also had much more extensive damage, often stretching back into the parietal lobe. Thus, although intriguing, these results do not support the strong inferences some draw about a specific role of Broca's area in this task,35 which involves action sequencing but not hierarchical sequencing per se.

Broca's area and hierarchical action planning

Much more convincing evidence for a specific role for Broca's area in hierarchical action, beyond music and language, comes from an elegant fMRI study by Koechlin and Jubault,97 which introduced a task designed to discriminate simple sequencing from hierarchical sequencing. This work built upon the computationally explicit model of hierarchical action planning of Dehaene and Changeux,98 which recognizes that at least three levels of nesting are needed to clearly discriminate simple gestures, motor sequences, and hierarchical plans. This model is based on the “Tower of London” task introduced by Shallice,42 which involves moving colored beads on three posts from some starting state to an end state designated with a target image. This task can be smoothly varied in complexity, from an easy variant where the beads are simply moved directly to the goal, to challenging end states that require five or more moves, some of which require moving beads away from their final goal state.99 Shallice found that patients with left prefrontal damage have difficulties specifically with these more complex problems,42 and brain-imaging results are broadly consistent with prefrontal involvement in this task.100 Dehaene's and Changeux's model specifies three levels of action: simple motor gestures (e.g., grasping a bead), operations (e.g., moving a bead from one post to another), and plans (nested sequences of operations to reach some specified goal state). Only the planning level provides clear evidence of hierarchical (nonchained) sequencing, particularly when it involves intermediate operations leading away from the goal. When the computational model was “lesioned” by removing planning units, these more difficult problems were specifically impaired, just as in Shallice's prefrontal patients.

Koechlin and Jubault96 designed an analogous task. Repeated button presses (motor condition) or simple, overlearned left—right sequences are produced in the scanner, under different stimulus conditions. In the simple chunk condition, overlearned sequences alternated with the motor control sequencing task. But in the superordinate condition, chunk sequences were arranged into higher level orders. All sequences were highly overlearned and automatized, so no learning occurred during scanning. Although simple motor acts elicited motor and premotor activity, higher order actions elicited more anterior activations within Broca's area, with transitions between chunks favoring BA 44, and termination of superordinate chunks activating BA 45. These data provide the clearest evidence to date of a specific role for Broca's area in planning hierarchically structured action.

We conclude that neuroimaging data support Lashley's hypothesis that music, language, and some types of action planning share a common substrate, and that Broca's area plays a key role in this distributed system. Although none of these data exclude a role for mirror neurons in action understanding in music or language, they argue against a necessity for motor knowledge to either understand sensory input or to elicit specific activity in the IFG. In contrast, these results remain consistent with Lashley's more abstract proposals, on the basis of computational similarities underlying planning of complex action in all three domains. The burning question thus remains concerning what, precisely, this supramodal system is doing at a computational level.

Broca's area as Lashley's “scannable buffer”

Lashley emphasized that the core requirement of sequential hierarchy is that higher level processes and goals can be placed on hold during the execution of lower level processes. To accomplish such persistence, he suggested, certain neural circuits must act as a buffer scannable by other circuits. In computational terms, sequential hierarchies require the equivalent of a register (holding one element), a queue (multiple freely accessible elements) or a stack (holding multiple elements, but with restricted last-in-first-out access), and require that these forms of persistent memory be available to other ongoing computational processes. Psychologically, these are all different forms of working memory that provide temporary storage of intermediate results (analogous to a scratchpad or blackboard). Although a reflex chain of actions, where each action triggers the next, does not require such persistent intermediate storage, both processing and production of hierarchical sequences do. Furthermore, the more complex and multileveled the hierarchy, the greater the capacity of the intermediate storage mechanism must be.

From a computational viewpoint, there is no doubt that effective, succinct computation requires such additional persistent storage. The most common means of achieving it in computer science is via a stack. For example, during the execution of complex computer programs, each call to a subroutine places a pointer to the current routine (and often intermediate results) onto the stack before branching to execute the subroutine. Once the subroutine completes its computation, typically returning some required result, execution of the original calling function can resume where it left off, by popping its pointer back off the stack. This can be embedded multiple times (with each level of nesting requiring additional stack space), and clearly the storage capacity of the stack limits the possible extent of embedding.

Both this storage capacity and restrictions on access influence the types of computations that are possible. Theoretical computer scientists have mapped out, in considerable detail, the computational limitations of registers versus stacks versus queues in abstract models of computation called automata. For instance, a finite-state automaton (FSA) is a system where discrete computational states are linked with no persistent memory. When such a system is augmented with additional working memory structures, new automata classes result. A push-down automaton (PDA), for example, is an FSA with an additional stack provided, whereas a Turing machine is an FSA with an endless queue (unrestricted order of access). Automata are models of processing, but important theorems link each of these types to specific classes of string sets (“languages”) that can be generated or recognized by the corresponding type of automaton. Unsurprisingly, the less restricted the additional memory store of the automaton, the broader and less restricted the type of string sets it can handle. These automaton/language pairs can thus be arranged into a classification system, termed the formal language hierarchy or extended Chomsky hierarchy:101 a fundamental theoretical concept in computer science.102–104

One appealing version of such stack-based computing would involve domain-specific templates and filler information, retrieved and maintained by a multidomain processor. For example the sentence “John likes apples” would involve retrieving the template “X likes Y,” along with the specific elements “John” and “apples” (which might just as well be “Mary” and “chocolate”). The role of the scannable buffer here is simply to maintain the current template-element bindings during the processing of further perceptual information or motor planning. Such a mechanism could be used to deal with many types of templates, independent of format (e.g., verbal, melodic, and motor), and thus would be available for multiple domains.

Although the distinctions between registers, stacks, and queues are a major focus in computer science, from a psychological or neural point of view the relevance of such abstractions is less obvious. Lashley's critical point was that hierarchical sequence processing requires some form of scannable intermediate storage, and limitations on this capacity obviously restrict the capacity of a system to handle hierarchical structures. Formally, the crucial distinction is between FSA-only systems (which can generate the so-called regular languages and have only a very limited and inflexible capacity to process hierarchy) and the augmented automata above these (e.g., PDAs or Turing machines), which correspond to the supraregular languages.

A foundational result in computational linguistics is that natural languages require processing resources over and above those of an FSA, and are thus supraregular.105,106 Considerable debate has surrounded the question of how far beyond regular the natural languages go; current consensus is that languages go a bit beyond the capacities of a PDA and belong to the class termed mildly context-sensitive grammars,107,108 requiring multiple stacks, or even a stack of stacks to be computed. With music, the situation is less clear, but some theorists suggest that music requires computational supraregular resources including at least one stack109—the so-called context-free grammars.

Marrying these computational considerations with our prior conclusions on the basis of neural localization leads to the following hypothesis. Cortical resources in the IFG, including at least BA 44 and BA 45, implement a storage buffer scannable by other cortical and subcortical circuits subserving sequential behavior. This buffer is required to implement supraregular hierarchical sequence processing, and its processing load increases with the depth and complexity of the hierarchy being processed.

This hypothesis, which has been entertained in related forms by multiple researchers,52,97,110 extends Lashley's speculations by positing a discrete neural locus implementing his scannable buffer and by specifying its supraregular nature. Direct evidence that processing load increases with depth is provided by parametric studies,53,111,112 and the localization aspect to the IFG is consistent with a wealth of neuroimaging data. This hypothesis does not require that the IFG is the only site of a scannable neural buffer (similar buffers may be implemented, for example, in hippocampus or basal ganglia circuitry). Nor does this hypothesis suggest that the stack or buffer represents all of the relevant data or computations: the idea is that it can activate and link representations generated in other brain regions (akin in computer terms to storing a pointer, rather than all of the data). Nonetheless, the involvement of Broca's region in hierarchical sequencing in music and language, and in some complex action planning, seems abundantly clear. Koelsch has termed this idea the syntactic equivalence hypothesis.113

This hypothesis does not follow directly from the fact that cortical systems for both perception and action are laid out hierarchically. For if each level of a processing hierarchy mapped directly onto a neural hierarchy, implemented on the cortical surface, there would be strict limitations on the possible depth of hierarchical structures. For example, the data on hierarchical action planning above suggested a direct mapping from first-level motor gestures in posterior (motor and premotor) cortex and third-level superordinate planning to anterior (BA 45) regions. But prefrontal cortex cannot be extended forever, and a literal mapping would suggest a strict limit on the level of embedding possible in language, music, or action. Such a limit is not evident in action (even making coffee can be construed as involving four levels, and if interrupted by a phone call, five), nor in music or language. The well-known restriction of embedding to three levels in language, for example, applies only to a particular syntactic structure—center embedding—not to syntax or language as a whole.114,115

Having thus extended Lashley's hypothesis to a specific neural circuit, we turn to Lashley's final hypothesis: that the capacity to process musical and linguistic hierarchical sequences is unusually developed in humans, but builds upon and extends preexisting action sequencing mechanisms present in animals. Addressing this issue demands an examination of comparative data.

The phylogenetic roots of hierarchical sequencing: comparative data

Many comparative studies have addressed animals’ ability to produce and perceive hierarchical structure,116 mostly in the auditory domain but also exploring visual capabilities.45,117 Much of the recent animal work has adopted the computational principles outlined above, contrasting finite-state grammars with context-free grammars to see which (if any) animal species can go beyond the regular grammar level.

This comparative work built upon human experimental psychology research begun by George Miller in the late 1950s, in one of the first computerized testing laboratories at Harvard.118 Rule-learning problems were generated and tested at different computational levels. The results led Miller to put forth the following hypothesis, which we call the supraregular hypothesis: adult humans have a proclivity to induce rule systems at context-free (or higher) levels, even when the data do not require such systems. That is, even stimulus sets that could be captured by regular grammars tend to be captured using supraregular grammars.

Miller's hypothesis suggests not only that humans are able to go beyond regular grammars, but also that human observers are biased to view sequences as hierarchically structured, even when the generating algorithm is a serial, finite-state system. Because all stimulus sets generated by finite-state grammars can also be parsed by context-free grammar, this may be a cognitive example of the aphorism “to a man with a hammer everything looks like a nail.” We might think of this bias as a form of dendrophilia—an inordinate fondness for tree structures—in our species.119

Both hierarchy and supraregularity are clear and well defined mathematically. Unfortunately, it is not trivial to adjudicate between finite-state and context-free requirements on the basis of string processing alone.120 Structure is an invisible (mental) construct, and can only be empirically probed indirectly (e.g., by observing acceptance and rejection of well-chosen string sets,121 by adding prosodic cues at or between putative phrase boundaries,122 or by observing reactions to clicks played at or between putative boundaries123). Such indirectness renders empirical testing of these notions challenging, but by no means impossible.

Returning to animal research, existing studies paint a relatively consistent picture. Many species, from rats to songbirds to baboons, have well-developed sequencing capabilities that can be characterized as finite state systems. Such systems, for example, appear adequate to capture the syntactic structure seen in birdsong116 or the recognition of simple long-distance dependency in squirrel monkeys.124 In contrast, the capacity of animals to induce supraregular grammars remains contentious. Despite numerous examples of successful induction of regular grammars, only a few species have been claimed to go beyond this,116 and both the data and the methods in these studies have been sharply questioned.125–127 Perhaps the most convincing attempt to show supraregular abilities in a nonhuman species came from Gentner et al. in their operant conditioning work128 with starlings (Sturnus vulgaris), songbirds with complex song and extended vocal learning in both females and males.129 Gentner's et al. used a supraregular grammar denoted AnBn, which means that the number of A units must be precisely matched by the number of B units (in this case the A and B elements were two types of starling song motifs, termed warbles and rattles, presented acoustically). This grammar is provably supraregular, and allows multiple possible structural interpretations ranging from count-and-compare, to center-embedded, to cross-serial structure.121 After very extensive training (between 9400 and 56,200 trials per bird, with a mean of 30,000 trials), three of four starlings tested with various probe strings showed evidence of having acquired the supraregular language AnBn. This suggested that, although quite difficult, it was at least possible for some birds to acquire a supraregular rule. However, a follow-up study in zebra finches125 showed that such levels of performance could be explained by simpler rules and averaging together of the performance of different birds.

On the basis of these and other results (reviewed in Ref. 116), we can tentatively formulate a comparative extension of Miller's human-oriented supraregular hypothesis—the supraregular exceptionality hypothesis, which posits that, in contrast to humans, nonhuman animals display a particular difficulty inducing hierarchical patterns from strings of data, especially any structures requiring computational resources above the finite-state level.

Evidence that an organism fails on supraregular grammars (e.g., AnBn, mirror grammars, or copy grammars101), although succeeding on regular grammars (e.g., (AB)n, AB*A, or sequential transition–probability grammars130,131) constitutes support for this hypothesis. Unlike Miller's human-specific supraregular hypothesis, which has abundant support from human studies,40,118,132–134 our supraregular exceptionality hypothesis, which concerns nonhuman animals, remains tentative for several reasons. First, nearly every comparative study has used a different species, and only a handful of nonhuman species have been tested to date. This renders broad generalizations about animals impossible. Even statements about one species remain tentative: multiple empirical issues need to be considered in pattern-learning experiments that make it difficult for any single study to control for all interpretations. Finally, very few grammars have been tested (indeed, all animal-based claims for supraregularity are based on a single grammar AnBn), which renders generalizations to broad formal classes impossible. Although humans succeed on multiple supraregular grammars (including copy and mirror grammars), to our knowledge no one has tested such grammars in nonhumans. Firm conclusions will require more research, on different species and different grammars, and replications across laboratories.

Nonetheless, these comparative data provide initial support for Lashley's notion that a hierarchical syntax of action is particularly well developed in our species. It seems too early to draw conclusions concerning the degree to which this proclivity builds upon preexisting supraregular abilities in animals.135–137 Certainly, we have no evidence of action syntax in animals even approaching the complexity of making a pot of coffee: the closest would be certain instances of tool use in some chimpanzee populations.138 Before returning to the possible neural basis for expanded hierarchical abilities in humans, we need to briefly forestall misunderstanding by noting an area of pervasive confusion in the recent comparative literature.

Recursion and supraregularity: an unfortunate confusion

For unclear reasons much of the comparative literature just discussed conflates several distinct issues, mixing up multiple computational abilities as if they were one. Our concern in this review is the processing of hierarchical sequences, defined as sequences with a tree-formed structure in which element order matters. Such structures are best processed by supraregular systems. Unfortunately, much of the last decade's research based on formal language theory has conflated hierarchy and supraregularity, in these precise senses, with various notions of recursion. Recursion refers to that broad class of computational processes in which a function calls itself (in computer science), or to those structures where the same hierarchical structure is repeated at multiple levels of the hierarchy (“self-embedding”). An example of a recursive structure is a fractal, where the same structure repeats itself ad infinitum at all hierarchical levels.139

Crucially, although all recursive trees are hierarchical, all hierarchies are not recursive. For many types of hierarchy discussed in this review, such as motor actions embedded within plans, it is unclear what self-embedding would even mean. Although a context-free grammar can be implemented recursively, for finite sets it need not be, and regular grammars can also be implemented recursively. The well-defined supraregular divide, between finite-state automata and those supraregular automata that have some additional scannable memory, is thus orthogonal to the issue of recursion. This issue has now been repeatedly discussed in the literature,121,140–144 so we will not belabor it here. We can only hope that the unfortunate current tendency to substitute the ambiguous high-profile buzzword recursive for distinct (and more empirically tractable) concepts like supraregular or hierarchical will eventually run its course.

Evolutionary expansion of Broca's region in humans

We end with a brief review of recent comparative anatomical data pointing to a general expansion of prefrontal connectivity in our species, along with a more specific and pronounced increase in the size of Broca's area and changes in its pattern of connections.

The human brain is greatly expanded in size relative to other primates, including our nearest relatives the great apes (chimpanzees, gorillas, and orangutans).145 However, ours are not the largest brains in the animal kingdom: the brains of elephants and toothed whales like dolphins or orcas are larger.146 Nor do human brains occupy the greatest relative proportion of total body weight: that distinction goes to small mammals like shrews.147,148 It is only when brain size is considered relative to what would be predicted, given body size (the so-called encephalization quotient), that human brains reign supreme.149 In these terms, although great apes have relatively large brains for mammals, humans are still quite exceptional, with a roughly threefold increase in total brain size relative to chimpanzees (or early fossil hominids like Australopithecus150).

Recent comparative neuroanatomical work allows us to go beyond these long-known facts to inquire whether human brain expansion affected specific brain regions and circuits. Although human frontal regions, like the cortex overall, are much larger than in other primates, whether this increase is disproportionate to overall size increase remains contentious. One of the first studies to use MRI in living primates to address this issue151 compared the volume of the frontal lobe in four great ape species with humans, along with lesser ape (gibbon) and monkey (Macaca and Cebus) brains. They concluded that great apes in general have an expanded frontal cortex relative to gibbons or monkeys, but frontal lobe volume was not disproportionately different from humans relative to total brain volume. This is consistent with many findings that apes are cognitively superior to monkeys.152 However, this study looked at the entire frontal lobe, as demarcated by the central sulcus (thus including both motor and anterior areas).

To examine the relative size of the prefrontal cortex specifically, Schoeneman et al. used MRIs of 11 primate species to examine the brain volume anterior to the corpus callosum.153 A separate analysis of gray and white matter found a striking dissociation between these two components: human relative gray volumes differed significantly from only a few species, whereas white volumes were greater than all but two species. A subsequent analysis of gray:white size ratios indicates that any human expansion of the prefrontal cortex relative to total brain volume is almost entirely the result of an increase in white matter. This suggests that, in addition to their raw increase in absolute size, human prefrontal regions have become disproportionately connected to the rest of cortex. Whether human frontal lobes (gray or white) are significantly and disproportionately larger than predicted depends on the comparison group (apes or monkeys). But differences from nonape primates are established, whereas differences from apes remain controversial.154–157 Furthermore, absolute frontal volume increase is undisputed, and relative white matter increases seem relatively clear.

To what extent do these increases in size and connectivity apply equally to all prefrontal regions? This question is challenging to answer, because it requires that multiple brains are analyzed using detailed cytoarchitectonic methods. In apes, such analysis has to date been performed only for prefrontal areas BA 10 (frontal pole), 44, and 45 (along with V1 and BA 13, part of the insula). Comparisons between the brains of 12 chimpanzees158 with previous work in humans159 revealed a striking finding: areas 44 and 45 are the most greatly expanded cortical areas yet identified in humans. Left areas 44 and 45 are six times larger in humans than in chimpanzees,158 disproportionate to the roughly threefold increase in total brain size, or the 4.5-fold increase in frontal cortex in total. This important study shows that Broca's area in particular has expanded disproportionately since our divergence from chimpanzees (roughly 6 million years ago).

Regarding connectivity, a recent study exploited the technique of diffusion tensor imaging (DTI), which uses MRI to estimate, in intact brains, axonal connectivity between brain regions.160 Applying DTI to human, macaque, and chimpanzee brains, Rilling et al.161 found significant changes in the pattern of connectivity between Broca's area and posterior brain regions. Humans, and to some degree chimpanzees, showed connectivity between prefrontal and temporal regions via a dorsal pathway through the parietal cortex, but only in humans did this pathway (the arcuate fasiculus) have a strong uninterrupted connection to the posterior temporal cortex. In contrast, rhesus macaques showed a very weak dorsal pathway, whereas ventral connections between the temporal cortex and frontal regions via the insula dominated. These results again converge on the conclusion that it is not simply the size of prefrontal regions that has increased in humans; the pattern of connectivity has changed as well.162

Conclusion: Broca meets Lashley

Combining the behaviorally anomalous status of human music and language relative to animal communication systems with the comparative anatomical data and imaging data reviewed above, we begin to see the outlines of a neurally and biologically grounded hypothesis concerning the undeniable differences between humans and other animals that does justice to the very deep cognitive and neural foundations that we share with other vertebrates (including other primates). For each of Lashley's three questions, laid out in the introduction, current data support a tentative positive answer: processing hierarchical sequences appears to be a well-defined and neutrally localizable function, shared by musical and linguistic syntax (Q1), humans are unusually well developed in this ability (Q3), and it is at least plausible to suggest that this ability may inherit key components from some type of action syntax that predated the evolution of human music and language (Q2).

From a modern perspective, Lashley's action syntax can be localized to a set of widespread brain circuits that have, as a key hub, prefrontal regions centered on Broca's region. The prefrontal regions play a primitive role in the hierarchical planning and sequencing of action, a function presumably shared with other primates, and particularly with chimpanzees, whose consistent and intelligent use of tools has no peer among other nonhuman primates.163–165 However, the apparent remit of this type of hierarchical planning increased greatly during human evolution, to include both perception and production of all types of hierarchical sequences. The most prominent additions to this Broca-centered action sequencing capacity were, by hypothesis, those two great human achievements: music and language.

From an evolutionary viewpoint, this idea provides for a certain continuity between humans and other primates (particularly chimpanzees) while acknowledging the drastic increases in certain cognitive capacities that have occurred since our divergence with chimpanzees. They also make sense of the fact that although prefrontal cortex and Broca's area are not novel brain regions, they have expanded disproportionately and have changed their patterns of connectivity to other brain regions during recent human evolution. In proposing this, we are fully aware that Broca's region is by no means the sole seat of language or music. Both capacities rely on a far-flung network of both cortical and subcortical brain regions (all of them, again, shared with other primates). But Lashley's ideas bring into focus the precise cognitive and computational changes that might underlie our expanded capacities for planning, action, language, music, and thought itself, in the context of a wide array of shared capacities. This is, by hypothesis, our broadly developed ability and indeed proclivity to structure both action and perception hierarchically. The clear distinctions between currently popular mirror neuron-based hypotheses and the more abstract computational hypothesis of Lashley have the potential to drive more refined experimental procedures and to provide further empirical evidence relevant to understanding Broca's region and its function. We conclude that, although Lashley has been rarely cited in the large number of studies implicating Broca's region in action, music, and language, his ideas have withstood the passage of time well, and deserve renewed attention.