Recurrent dynamics in the cerebral cortex: Integration of sensory evidence with stored knowledge.

Current concepts of sensory processing in the cerebral cortex emphasize serial extraction and recombination of features in hierarchically structured feed-forward networks in order to capture the relations among the components of perceptual objects. These concepts are implemented in convolutional deep learning networks and have been validated by the astounding similarities between the functional properties of artificial systems and their natural counterparts. However, cortical architectures also display an abundance of recurrent coupling within and between the layers of the processing hierarchy. This massive recurrence gives rise to highly complex dynamics whose putative function is poorly understood. Here a concept is proposed that assigns specific functions to the dynamics of cortical networks and combines, in a unifying approach, the respective advantages of recurrent and feed-forward processing. It is proposed that the priors about regularities of the world are stored in the weight distributions of feed-forward and recurrent connections and that the high-dimensional, dynamic space provided by recurrent interactions is exploited for computations. These comprise the ultrafast matching of sensory evidence with the priors covertly represented in the correlation structure of spontaneous activity and the context-dependent grouping of feature constellations characterizing natural objects. The concept posits that information is encoded not only in the discharge frequency of neurons but also in the precise timing relations among the discharges. Results of experiments designed to test the predictions derived from this concept support the hypothesis that cerebral cortex exploits the high-dimensional recurrent dynamics for computations serving predictive coding.

Humans shift their gaze, on average, three times a second to take snapshots of their environment in order to collect the evidence required for a seamless reconstruction of the embedding visual world. This implies that the visual system is capable of accomplishing the following operations within fractions of a second: 1) extract from the two-dimensional and continuous distribution of electromagnetic waves on the retina the features that have proven useful during evolution for the analysis of visual scenes; 2) examine the relations among these features and compare them with stored information about meaningful constellations; 3) group features whose constellations are identified as meaningful—a function addressed as feature binding; 4) segregate such bound entities from each other and from the embedding background—a function addressed as scene segmentation or perceptual grouping; 5) identify the grouped feature constellations by matching them with stored knowledge about perceptual objects; 6) analyze the spatial and/or semantic relations among the recognized objects; and 7) store the content of these snapshots for later combination with previous and future snapshots in order to generate the percept of a coherent visual scene.

All of these functions require comparison of incoming sensory signals with stored information about regularities of the visual world—a function addressed as “unconscious inference” by von Helmholtz (1) and as predictive coding in the recent literature (2–7). This notion raises several challenging questions. Where and how is the internal model of the visual world stored? How is it possible to retrieve, within a fraction of a second, the specific priors required for the interpretation of a particular snapshot and to compare the ever-changing sensory evidence with stored knowledge? How is the factually infinite space of possible relations among features explored in order to segregate relevant from irrelevant relations? How are meaningful relations encoded and made amenable to further processing?

In the following sections, a concept will be presented that could account for some of these functions. It combines two complementary processing strategies: first, the analysis of relations in hierarchically structured feed-forward architectures and, second, the dynamic and context-dependent encoding of relations in recurrent networks. Subsequently, experimental evidence related to this concept will be reviewed. The data are taken mainly from anatomical and electrophysiological investigations of the mammalian visual cortex and include results of recent experiments designed to test the predictions derived from the proposed concept.

Two Complementary Strategies for the Analysis and Encoding of Relations

The virtually infinite variety of perceptual objects results from variable combinations of a relatively small set of elementary features, just like the 26 letters of the Latin alphabet suffice to compose western literature. Therefore, cognitive systems need effective strategies to identify these features and to encode the relations among them.

Encoding of Relations in Feed-Forward Networks.

One common strategy for the encoding of relations is based on the generation of conjunction-specific neurons in hierarchically structured feed-forward networks. Neurons tuned to elementary features distribute their responses through divergent and convergent connections to neurons of the respective next layer. By appropriate recombination of these connections and adjustment of their gain, the nodes of higher layers become tuned to represent specific feature constellations. In the mammalian visual cortex, for example, orientation-selective edge detectors receive convergent input from retinal ganglion cells with colinearly aligned receptive fields (RFs) (8–10), and thereby capture the prevalence of elongated contours in natural environments (11). In the mammalian visual cortex, this strategy is iterated over numerous processing stages (12). By varying the degree of overlap and the gain of the connections, neurons ultimately become selectively responsive to the highly complex constellations of features characterizing natural objects (13–17). Because these conjunction-specific nodes always signal the same constellations of features, this strategy of analyzing and representing relations is addressed as the labeled line code.

At low levels of the visual processing hierarchy, axonal convergence is mainly determined by genetic programs and leads to Gabor-like RFs. These capture elementary features of natural scenes and accord with the coding principles of independence and sparsity (18, 19). However, during early postnatal development, the feed-forward connections are highly susceptible to experience-dependent selection following a Hebbian modification rule (20, 21). Axons conveying correlated activity stabilize selectively on common target nodes. Due to this essentially self-supervised pruning process, nodes become tuned to respond preferentially to frequently occurring constellations of features (20, 22–24). Consequently, the responses of conjunction-specific neurons capture the statistical regularities of the visual environment (25). Some of these adaptive processes are carried over into adulthood and support perceptual learning (26). At lower levels, these learning processes serve the fine-tuning of feature selectivity (27–29) and, at higher levels, the generation of nodes responding to complex constellations of features (30). This strategy for the encoding of relations is also the hallmark of artificial neuronal networks, designed for the recognition and classification of patterns (31–33). The recent and highly successful developments in machine learning known as “deep learning convolutional networks” (34–36) capitalize on the scaling of this principle in large multilayer architectures and on the additional implementation of supervised learning algorithms (37).

However, labeled line coding has limitations. One is the high hardware cost. To achieve invariance, to cope with the nested relations characterizing natural scenes, and to encode context, an astronomical number of conjunction-specific nodes would be required—a problem addressed as combinatorial explosion. Another shortcoming of feed-forward networks is their lack of a temporal dimension. This makes it difficult for them to cope with temporal sequences and to establish relations among temporally noncontiguous events. To cope with this problem, modules have been added that have memory functions such as long short-term memory (35, 38, 39).

Dynamic Encoding of Relations in Recurrent Networks.

A complementary strategy to capture relations among components relies on dynamic combinatorial codes, similar to those used by natural languages. This strategy was proposed by Donald Hebb (40) more than half a century ago. It posits that relations among components should be encoded by transiently binding feature-selective neurons into functionally coherent assemblies, the Hebbian assembly. Through cooperative interactions, these neurons would collectively signal the presence of a particular constellation of components. The formation of such transient assemblies requires self-organized cooperativity among network nodes and is therefore difficult to implement in feed-forward architectures. By contrast, the required interactions can be realized elegantly in recurrent network architectures. In theory, recurrent networks can be rolled out and simulated by feed-forward networks. However, this is only possible if the reciprocal interactions can be discretized, which may not be the case in natural systems. But, even then, a very large number of layers would be needed to represent the results of the recursive updating steps. For natural systems, this is not an option, because of the immense hardware costs and the constraints on processing speed.

Recurrent architectures are ubiquitous in nervous systems and particularly evolved in structures of vertebrate brains like the cerebral cortex (41, 42). According to Hebb´s proposal, the recurrent, reciprocal connections must be endowed with correlation-sensitive synaptic plasticity mechanisms (Hebbian synapses) in order to stabilize, preferentially, assemblies representing frequent constellations of features. With exposure to natural visual scenes, nodes tuned to frequently cooccurring features would become coupled more strongly and therefore engage more readily in cooperative interactions when activated by the respective feature constellation. In this coding scheme, conjunctions of features are represented by groups of temporarily cooperating nodes rather than by individual conjunction-specific neurons.

The ensemble coding strategy has a number of advantages that ideally complement those of feed-forward processing. One is the combinatorial nature of the code which economizes on hardware. Nodes can be flexibly and dynamically recombined to capture different constellations of features. Moreover, this strategy expands the storage space for the internal model because information about the statistical contingencies of features in natural scenes is now encoded not only in the topology and synaptic weights of feed-forward connections but also in those of the recurrent connections. Quantitative data on the cortical connectome indicate that these recurrent connections outnumber, by far, the feed-forward connections (41, 42).

Another advantage of assembly coding is that it can exploit not only discharge rate but also spike timing to convey information. This expands coding space and accelerates processing speed. Donald Hebb (40) had initially proposed that nodes forming a cooperating assembly should be distinguished by joint increases of discharge (firing) rate. And there is evidence that recurrent interactions can enhance discharge frequency (43). However, the notion that cooperating neurons should be distinguished solely by joint increases in discharge rate has been challenged (for a review, see ref. 44). First, different simultaneously active and spatially intermingled assemblies are difficult to distinguish from one another if all neurons participating in assemblies simply discharge more vigorously—a complication addressed as the “superposition problem” (45, 46−47). Second, increases in discharge rate are an ambiguous signature for a relational code because discharge rates also reflect stimulus energy and/or matches between stimulus and RF properties. Third, distinguishing cooperating neurons on the mere basis of enhanced discharge rates slows down processing speed because discharge rates of cortical neurons are low and can carry only a little information when integrated over short intervals. The latter problem could, in principle, be solved by configuring nodes as clusters of cells with similar feature selectivity and by averaging across their activity. However, this strategy is costly in terms of hardware and energy. In addition, it is hampered by the fact that fluctuations of cortical activity are correlated (“noise correlation”). Thus, averaging amplifies not only the signal but also the noise (43, 48).

The Binding by Synchrony Hypothesis

In the 1980s, a serendipitous observation in the visual cortex of awake kittens led to a discovery that offered solutions to the problems associated with rate-coded assemblies. Neurons in the visual cortex activated by continuous contours were found to synchronize their spike discharges with millisecond precision even if located in functional columns segregated by up to 7 mm (49, 50). Because synchronization enhances the impact of discharges in downstream targets (51–53), it was proposed that the joint increase in salience of precisely synchronized discharges rather than joint rate increases would identify the transiently cooperating nodes of an assembly. This temporal code would substantially reduce the superposition problem because coincidence-sensitive downstream neurons can distinguish between different synchronous events with high temporal resolution (refs. 44, 46, 47, and 54; for reviews, see ref. 55). For the same reason, encoding relations by coincident firing rather than joint rate increases would allow for much faster detection of cooperating nodes (56) and also render the signature of relatedness independent of rate fluctuations.

Synchrony and Oscillations.

Early studies on spike synchronization in the visual cortex used very simple stimuli such as moving light bars and gratings. For reasons discussed later, these stimuli, in particular the gratings, not only induce synchronous firing but also cause an oscillatory modulation of population activity in the gamma frequency range. Because increases in oscillatory power and spike synchronization covary in these special conditions, the important distinction between oscillatory patterning and spike synchronization got blurred, and the two phenomena were often regarded as equivalent (57). Oscillations are particularly easy to detect in signals that reflect the summed activity of large neuron populations such as are obtained with field potential, electrocorticogram, EEG, and MEG recordings. Although these signals reflect mainly synaptic currents, the power of narrow-band oscillations and the coherence of oscillations recorded from different sites are commonly considered as measures of spike synchronization. Consequently, band-passed oscillations and their coherence rather than the timing relations among discharges of distributed neurons became the target variable in numerous studies investigating response synchronization (for review of the extensive literature on oscillations, see refs. 58–62). Yet, what matters for information processing are not the oscillations but the rate and the precise timing relations of discharges. As detailed below, oscillatory mechanisms do play an important role in the temporal patterning and coordination of discharges (63–68), but oscillations, per se, likely convey no information.

The Relation between Gestalt Rules and Response Synchronization.

Analysis of the synchronization phenomena observed in the visual cortex revealed that neurons tuned to features that have a high probability to cooccur in natural environments transiently synchronize their discharges with millisecond precision when coactivated by such related features. This was the case for continuous contours, colinearly aligned contour segments, contours sharing the same orientation, and contours moving with the same speed in the same direction (55). These feature constellations correspond to the basic Gestalt criteria for perceptual grouping, suggesting that synchronization probability reflects the statistical regularities of natural environments and serves feature binding (for a review, see ref. 44). Interestingly, this grouping by synchronization turned out to be dynamic and context sensitive: Typically, when a single elongated moving contour is presented, all nodes activated by this contour synchronize their discharges (50, 69). However, when two contours with different orientations overlap in space and move in different directions, the activated neurons split up into two groups (70–72). The neurons within each group discharge in synchrony, but there is no correlation at millisecond time scales among the discharges of neurons belonging to different groups. This self-organized grouping depends on the preferences of the neurons for the two contours. Those neurons join the same synchronization group that shares preferences for the same contour. Thus, the synchronized firing of neurons is context sensitive and signals whether the neurons are activated by a single “object” or two different “objects.” As discharge rates of all neurons were elevated to the same extent, this segregation of neurons into two distinct groups would not have been detectable by decoding only rate responses. Simulations of recurrent networks with enhanced coupling of nodes tuned to groupable features have reproduced such context-dependent synchronization phenomena (73–75). The results of these and related studies eventually led to the proposal that transient and context-sensitive synchronization of discharges could serve as a mechanism for perceptual grouping and inspired the “binding by synchrony” (BBS) hypothesis (76).

Counterarguments.

In subsequent years, the BBS hypothesis has been challenged (77 and 78). As the early findings suggested a close association between oscillations and synchrony (see above), most arguments focused on the coherence of sustained oscillations rather than on the synchrony of discharges. Because oscillations are only sustained and stable in frequency under special stimulation conditions (79), vary in frequency as a function of stimulus energy and visual field eccentricity (77), and, if synchronized globally, limit the capacity of information transfer, oscillation-based synchrony was considered an epiphenomenon and irrelevant for information processing (for a review, see ref. 80). However, more-recent investigations in awake animals have revealed that synchronization of discharges, while often associated with oscillations, is not dependent on the occurrence of sustained oscillatory activity. Typically, episodes of synchronized firing are transient, occur in short bouts, follow at irregular intervals, and, if associated with oscillations, the oscillations tend to persist only over a few cycles (81–84). Similar results have been obtained in simulations of recurrent networks. Depending on the dynamic state of the network, synchronous events can occur in conjunction with brief bouts of oscillatory episodes but sometimes also in the absence of detectable oscillations (85, 86). Further evidence for the fast emergence and dissolution of stimulus-specific correlation structures in population responses of cortical neurons is discussed below when reviewing the effect of stimuli on cortical dynamics in awake monkeys.

Another argument against the BBS hypothesis was derived from the finding that, in area V1, synchronization did not comprise the entire population of spatially distributed neurons activated by a perceptual object (87). This argument assumes that the formation of object-specific Hebbian assemblies is achieved already at low levels of the visual processing stream. However, since neurons in the primary visual cortex represent only very elementary features, the grouping operations performed at this level can achieve only the low-level binding functions required for scene segmentation and perceptual grouping. Evidence actually suggests that correlation-based binding of attributes of perceptual objects does occur, but only at higher levels of the ventral visual processing stream, where individual neurons are already selective for object-specific feature constellations (88).

The Anatomical Substrate of Synchronization.

The anatomical substrate for the transient, feature-specific synchronization of neurons in the primary visual cortex is, in all likelihood, the network of intracortical recurrent connections that extend tangentially to the laminae, are particularly abundant in supragranular layers, and link, preferentially, nodes responding to features which frequently cooccur in natural scenes (86, 89–96). Direct evidence for the synchronizing effect of these reciprocal cortico–cortical connections has been obtained by severing callosal connections. These connections share numerous similarities with the tangential intraareal connections (94, 97, 98) and likely serve the same functions, albeit across the two visual hemifields. Severing the callosal connections disrupts feature-specific synchronization between neurons that are located in different hemispheres and respond to contours crossing the vertical meridian (99). Another noteworthy result of this experiment was that interhemispheric synchronization of spike discharges had occurred with zero phase lag despite the considerable conduction delays imposed by the callosal fibers. This proves that reciprocal interactions can support zero phase lag synchronization despite coupling delays, a phenomenon that has also been confirmed by simulation studies (100–104) and is of great relevance for the encoding of relations by coincident firing.

Together, these observations suggest that the cooperative and virtually simultaneous interactions among neurons of recurrent networks can be exploited to encode relations among features, using temporal coordination of spiking activity (spike timing) as a tag of relatedness—a tag that permits faster and less ambivalent encoding and readout of relations than joint rate increases.

Rate and Temporal Codes Convey Complementary Information

Combination of Feed-Forward and Recurrent Processing.

The coexistence of feed-forward and recurrent processing architectures in the brain raises the question of how these two processing strategies are combined with one another. The scheme depicted in Fig. 1 illustrates a possible scenario that exploits the respective advantages of the two processing strategies.

Fig. 1.

A cluttered scene, consisting of oriented contours (bottom), leads to activation of orientation-selective cells (blue circles) in input layer IV of V1. Because all contours have the same contrast and shape, the layer IV cells with corresponding RFs are assumed to be activated equally and to drive pyramidal target cells (1–5) in layers II/III via ascending feed-forward connections (blue arrows). These pyramidal cells are reciprocally coupled through excitatory collaterals (red arrows) of axons (blue and orange arrows) projecting to layer IV target cells (a–e) of the next processing stage (V2). Pyramidal cells 3 and 4 are coupled more strongly than their neighbors because they have a shared preference for colinearly aligned contours of the same orientation (indicated by thicker recurrent collaterals). This enhanced coupling is due to prior exposure to elongated contrast borders that are frequent in natural environments (for details, see The Binding by Synchrony Hypothesis). Because of strong coupling, cooperativity between neurons 3 and 4 is enhanced, and their responses become more coherent (indicated by orange color of feed-forward axons). This, in turn, leads to enhanced responses in target cells c and d, because cortical cells are driven more effectively by synchronous than temporally dispersed excitatory postsynaptic potentials (EPSPs) (51–53, 228). In addition, the discharges of neurons c and d are likely to be more synchronized, as well. In this way, recurrent coupling that captures the statistical regularities of natural environments can be exploited to detect relations among features, to convert these relations into temporal relations among discharges that, in turn, impact the spike responses of input cells at the next processing stage.

The example depicted in Fig. 1 illustrates how prior “knowledge” about environmental regularities, stored in the coupling weights of the recurrent network, can support the grouping of features by modulating the temporal coherence of responses to thereby evoke different rate responses at the input stage of a downstream area. In this example, all stimuli are assumed to have the same salience, and all feed-forward connections are assumed to have the same gain. Consequently, all nodes of the recurrent network, whose feature selectivity corresponds to matching features in the scene, receive the same excitatory drive. However, the grouping operations accomplished by the recurrent network enhance the temporal coherence of responses to “related” stimuli, in this case, colinearly aligned contours—and this leads to increased firing of the coincidence-sensitive target cells in the downstream area that receive synchronized input.

This simple grouping operation can, of course, also be accomplished in a strictly feed-forward architecture. However, the hardware costs are substantial because all possible combinations of colinear contours would have to be represented by individual conjunction-specific neurons. This combinatorial explosion can be avoided by dynamic binding, because individual neurons can be associated flexibly to signal different feature constellations. Moreover, delegating grouping operations to the dynamic interactions among nodes provides the additional option to render local grouping operations dependent on global context. In recurrent networks, local perturbations propagate across the whole network, often giving rise to traveling waves and avalanches that mediate interactions over large distances (105–109). Hence, local grouping (synchronization) probabilities depend not only on the coupling strength of the directly connected nodes but also on the dynamic state of the whole network. As this state reflects the match between sensory evidence and the entirety of the weight distributions of the recurrent connections, the numerous relations among contours of cluttered scenes can be evaluated in parallel to disambiguate local grouping options. Experimental evidence demonstrating the power of this highly parallel computation of grouping probabilities is discussed below.

Rate and Temporal Codes Convey Complementary Information.

The proposed combination of feed-forward and recurrent processing also assigns complementary roles to changes in discharge rate (rate code) and precise spike timing (temporal code), respectively. Firing rate and synchronization can vary independently (110–114), and hence the two variables can, in principle, be used in parallel to convey different information. Two recent studies in awake monkeys (111, 115) reported that stimuli whose features match predictions provided by context mainly enhance synchrony, whereas nonmatching stimuli strongly increase discharge rate. In these studies, the feature space was color rather than orientation. Monkeys were shown large monochromatic surfaces, and these induced strikingly strong, oscillatory synchronization among the discharges of all simultaneously recorded, responsive neurons in visual area V1. In contrast, increases in discharge rate were moderate. Here the feature, in this case, the color, present in the RF centers of the responding neurons was fully predicted by the embedding context. If, however, the stimulus in the RF center was segregated from the surround by a gray ring or if the center stimulus had a different color, the respective neuron responded with a strong increase in discharge rate but no longer fired in synchrony with cells responding to the embedding background. In agreement with the proposal of Vinck and Bosman (116), this indicates that nodes activated by stimuli that match the ”predictions” provided by context engage in cooperative interactions and identify themselves through synchronization as coding for mutually predictive features. By contrast, cells activated by a nonpredicted feature signal this “prediction error” by a strong increase in discharge rate and do not synchronize with the neurons responding to the nonmatching surround. This agrees with the BBS hypothesis and, as depicted in Fig. 1, assigns complementary roles to increases of discharge rate and synchrony. The synchronously discharging neurons signal feature constellations that match priors while vigorously discharging neurons, whose spikes are uncorrelated signal prediction errors. Whether these complementary signals are relayed through different channels across downstream areas awaits further exploration.

So far, I have only discussed how recurrent networks can support dynamic grouping of features by establishing temporal correlations among distributed responses. However, the high-dimensional dynamics unfolding in such networks offer many more and fascinating options for computations. In the following paragraphs, I shall, therefore, discuss some of these computational capacities, derive predictions, and review evidence, partly from experiments designed to test these predictions.

Computations Exploiting the Dynamics of Recurrent Networks

The complex dynamics generated by recurrent networks are exploited by a subclass of AI strategies addressed as “reservoir computing” whose instantiations have become known as “echo state” or “liquid” computing (117–122). Although these artificial networks lack many of the essential features of their biological counterparts, they have greatly inspired the exploration of computational capacities provided by recurrent processing in the cerebral cortex (123–128). The basic operation consists of transforming low-dimensional input patterns into high-dimensional dynamic states (dimensionality expansion) and using these high-dimensional patterns as input to classifiers. By this transformation, stimuli that overlap in low-dimensional space become linearly separable.

While sharing this essential capacity, natural recurrent networks such as are realized, for example, in the superficial layers (II and III) of the mammalian primary visual cortex differ in essential aspects from their artificial counterparts. These differences are summarized in Fig. 2. 1) The network nodes are feature selective, most of them showing shared and graded selectivity for different feature domains [“mixed selectivity” (129)]. This mixed selectivity probably arises from convergence of heterogeneous feed-forward connections that become stabilized selectively during development because they respond to frequently cooccurring features. Here, only the orientation of visual contours is considered. 2) The nodes have a propensity to oscillate because a fraction of cells have pacemaker properties (130, 131) and because local inhibitory feedback circuits (60, 61, 132) support rhythmic discharge patterns. 3) The topology and gain of the recurrent connections are anisotropic because these connections exhibit spatial gradients and are susceptible to experience-dependent modifications (see above). Hence the coupling of nodes reflects the probabilities with which the nodes have been coherently activated in the past and captures the statistical regularities of the visual world. And the mixed feature selectivity of the nodes further increases the dimensionality of the representational space (see above and ref. 133).

Fig. 2.

Simplified graph of recurrent connections among functional columns in layers II/III of the primary visual cortex. Nodes with similar colors respond to features that frequently occur together and are therefore coupled more strongly than other nodes (thick lines).

The Challenge of Stability.

The dynamics of recurrent networks requires subtle control to assure stability and to adjust time constants to the actual processing needs. On the one hand, the network must be prevented from engaging in runaway dynamics and becoming epileptic. On the other hand, a critical level of spontaneous activity has to be maintained to accelerate the readout of stored information and the grouping of nodes. If the nodes were solely activated by sensory input, these processes would take time to self-organize or might not occur at all. Nature adjusts the dynamics of cortical networks with a number of self-regulating local mechanisms involving complex networks of inhibitory neurons (E/I balance) (134), normalization of synaptic strength (135), synaptic plasticity (136), adaptation (137), and membrane resonance (138). In addition, excitability is controlled by globally acting modulatory systems (139, 140). These control mechanisms are not only essential for stability but also allow task-dependent optimization of network dynamics for the execution of particular computations. Experimental evidence indicates that cortical networks operate in a dynamic regime close to criticality (107, 108, 141, 142). Varying the distance of the network dynamics from the critical bifurcation point toward chaotic behavior offers interesting options for computations at the edge of chaos (143). It allows control of 1) the networks “fading memory,” that is, the duration of the interval over which temporally segregated inputs can interact with one another; 2) the speed of state transitions, that is, the rate at which cooperative interactions among nodes can be established and dissolved; and 3) the distance over which nodes can interact with one another (144).

Storage and Readout of Priors

Theories of perception, formulated more than a hundred years ago (1), and a plethora of more recent experimental evidence (145, 146) indicate that perceiving is a constructivist process. An internal model of the world is used to interpret sparse and noisy input signals, to make inferences about their relevance, and to eventually identify and classify perceptual objects. Given the vast number of priors required for the interpretation of ever-changing sensory input patterns, the store harboring this model must have an immense capacity. Moreover, the store must permit ultrafast retrieval of the relevant priors to meet the constraints of processing speed (see the Introduction).

In the following, a scenario is proposed that exploits the dynamics of recurrent networks for the storage and retrieval of priors, and for the ultrafast comparison of stored knowledge with sensory evidence.

Informative Spontaneous Activity.

Spontaneous activity of the cerebral cortex is highly informative. Even in sensory areas, it reflects global states of the organism and is influenced by movements (147, 148). In addition, it exhibits a distinct correlation structure that reflects the motifs of recurrent connectivity, that is, the topology and the weight distributions of the connections among nodes (149–151). Hence, in V1, the correlation structure of spontaneous activity covertly represents priors about regularities of natural scenes (152). In addition, this correlation structure is expected to be shaped by recurrent connections from higher processing levels (153, 154) that contribute information about higher-order statistics of natural scenes (155). Hence, resting activity in V1 is supposed to be high dimensional and to occupy a vast but constrained subspace inside the universe of all theoretically possible dynamic states.

Matching Sensory Evidence with Priors.

Once sensory signals become available, they are likely to trigger the following cascade of effects: They drive, in a graded way, the subset of nodes tuned to the features contained in the actual input pattern and thereby perturb the ongoing network dynamics. If these feature constellations match some of the priors stored in the anisotropic coupling connections, then a subnetwork of nodes will immediately resonate and engage in enhanced cooperation. Consequently, the network dynamics collapses to a specific substate. This substate is bound to have a lower dimensionality than the spontaneous activity because its dynamics is constrained by the enhanced cooperation of a subset of nodes. Due to reverberation, activity will quickly spread across the recurrent network, and stimulus-specific substates will eventually involve not only the initially driven nodes but, as time elapses, also other nodes of the network. Which of these additional nodes will be engaged depends, again, on the weight distributions of the recurrent connections and on the constellation of features in the stimulus. The evolution of a particular substate is therefore determined not only by the stimulus but by the entirety of the synaptic weights of the recurrent connections, that is, the priors stored in the functional architecture. The attained substate is thus equivalent to the result of a matching operation that combines the actual sensory evidence with a vast amount of stored “knowledge.” Therefore, it contains more information than the sensory evidence alone. Because these processes occur within a very high-dimensional state space, substates induced by different input patterns are likely to be well segregated and easy to classify once the respective substates have sufficiently diverged from resting state.

In the following paragraphs, predictions are derived from this scenario and related to experimental evidence.

Predictions and Experimental Evidence

The evidence reviewed in the following section is drawn partly from the literature and partly from experiments designed specifically to test the above predictions.

Prediction i.

Knowledge about the statistical regularities of natural visual scenes should be stored in the anatomical layout and the synaptic weights of both feed-forward and recurrent connections. The regularities captured by the feed-forward architecture should be expressed explicitly by the specific RF properties of the network nodes. By contrast, the regularities stored in the weight distributions of the recurrent connections should not contribute to the classical RF of the nodes but rather mediate selective cooperative interactions. These weight distributions should, in addition, be reflected in the correlation structure of spontaneous activity.

As documented by the innate feature selectivity of nodes (14, 156), the genetically determined layout of feed-forward connections does capture certain regularities of visual scenes. As developmental studies indicate, these inborn response properties of nodes are complemented and refined by experience-dependent pruning of feed-forward connections (for a review, see refs. 157 and 158). The same holds for the recurrent connections. Their layout and synaptic weights are also specified by experience (159) and reflect statistical regularities of natural environments (86, 92–97). Most of these developmental pruning processes are self-supervised (160, 161); for a review, see refs. 157 and 158. They require activated brain states (162), require scanning of and interaction with the environment (163), and fail to occur when the available sensory evidence is in conflict with inborn response properties of the system (164, 165). However, some of these network changes can be induced even under anesthesia when stimuli are presented repetitively (162, 166). Because the recurrent connections from higher to lower cortical areas share essential features with the intraareal connections, it is likely that they are also susceptible to experience-dependent modifications, but this possibility is much less explored.

Evidence also supports the conjecture that spontaneous activity is informative and reflects the priors residing in the network architecture. The resting-state dynamics of cortical networks are high dimensional (167), exhibit spatiotemporal patterns expected to occur in recurrent networks such as traveling waves and avalanches (106–109), and operate in a dynamic range close to criticality (108, 109, 168). In agreement with the anisotropic layout of the recurrent connections, resting activity exhibits a covariance structure that reflects the differences in coupling strength among nodes (95, 150, 169). And, as expected from the experience-dependent adjustment of coupling, this correlation structure is susceptible to experience-dependent modifications during early development (170), remains malleable throughout life by learning (171, 172), and recapitulates regularities of the environment (152). Thus, prediction i is already well supported by data.

Prediction ii.

In response to structured visual stimuli, the initially unconstrained, high-dimensional internal dynamics of the recurrent network should collapse into metastable subregions of the state space. These substates should exhibit reduced dimensionality, and their dynamic signatures should be stimulus specific. Stimuli matching the priors should induce substates, whose correlation structure, consistency, and persistence should reflect the quality of the match between sensory evidence and stored priors. Substates induced by natural stimuli should carry more stimulus-specific information than substates induced by less matching stimuli. Accordingly, classifiers should perform better for substates induced by natural than unnatural stimuli.

Stimulation reduces the variance and dimensionality of the dynamics, as indicated by a decrease of the Fano factor and an increase in coherence (155, 167, 173). This is consistent with the collapse toward lower-dimensional substates. For low levels of the visual system, a particularly good match with stored priors is achieved with grating stimuli because they comply with a large number of Gestalt criteria (priors). These are continuity, colinearity, regularity of texture, symmetry and similarity in feature space (in this case in the orientation domain), and—if the grating drifts—common fate. With drifting gratings, each stimulus entering a neuron’s RF is unequivocally predicted by the context. Accordingly, these stimuli elicit particularly stable and coherent states that are characterized by sustained, highly synchronized responses that, in addition, oscillate in the gamma frequency range (see above and refs. 49 and 50). If more-complex constellations of features are presented, such as plaids or several contours drifting in different directions, the correlation structure of the substates becomes more complex and less sustained (71, 174). And, if stimuli are presented that lack any regularities, such as random dot textures, and therefore do not match any priors, the resulting population responses show only weak, if any detectable, coherence (113). Accordingly, natural stimuli induce substates whose correlation structure exhibits greater stimulus specificity than substates induced with manipulated stimuli whose statistical regularities of higher order had been removed (115, 155). Recent experiments complement these findings by showing that substates induced with natural scenes allowed for better classification of stimulus identity with linear decoders than substates induced with phase-scrambled versions of the same natural scenes. This improved classification was not due to trivial causes such as enhanced signal to noise ratios, because discharge rates and stimulus-induced reduction of response variability (Fano factor) were exactly the same for natural and scrambled images. However, natural stimuli evoked stronger gamma bursts than their scrambled versions. This suggests that good matches between evidence and priors lead to more succinct, more coherent substates than poor matches and agrees with the proposal of Vinck and Bosman (116) and the findings of Peter et al. (111) and Uran et al. (115). The likely reason is that matching stimuli activate, preferentially, subsets of nodes that are strongly coupled. As a consequence, these will engage in stronger cooperative interactions and, when driven sufficiently, in oscillatory resonance.

The prediction that priors stored in the network contribute to the stimulus specificity of substates is also supported by the finding that classifiers trained to identify stimuli often perform better on the delayed, reverberating part of the responses than on the initial transients that are dominated by direct sensory input (175). Principal component analysis (PCA) of the rate vectors showed that the segregation of the vectors induced by different stimuli can increase over a few hundred milliseconds even after stimulus offset (175). The latter finding was obtained in experiments performed under anesthesia and requires confirmation in awake preparations.

A dependence of cortical dynamics on the match between sensory evidence and stored priors has also been described for another phenomenon that is characteristic of recurrent networks: the sequential activation of nodes after perturbations. A recent study (176) showed that natural scene stimuli lead to sequential activation of neurons (nodes) in monkey V1 and V4. These sequences (songs) are stimulus specific; that is, information about the identity of the stimuli is contained not only in the rate vector but also in the rank order of the activated nodes. Hence, stimuli can be identified by decoders trained to sequence order rather than rate vectors. A dependence of rank order on the match between sensory evidence and stored priors is suggested by the finding that the temporal precision of the sequences and hence also their decodability is higher for natural stimuli than for manipulated stimuli that lack some of the higher-order regularities of natural scenes.

Together, these observations support prediction ii that stimuli matching stored priors induce a collapse of systems dynamics toward metastable, low-dimensional substates that are more informative when induced by stimuli that match stored priors.

Prediction iii.

Reverberating activity should outlast the duration of stimuli. Hence, substates evoked by successively presented stimuli should overlap. This should allow decoding of sequence order and identity of the sequentially presented stimuli at the end of the stimulation sequence.

Experiments with anesthetized cats revealed that linear decoders could determine the identity of both the first and the second stimulus as well as the sequence order of the stimuli from activity vectors persisting after the end of the second stimulus (118).

Prediction iv.

Because the synapses of recurrent connections are plastic, repeated exposure of the network to the same stimuli should engage unsupervised or self-supervised learning processes. These should favor the stabilization and segregation of stimulus-specific substates. Consequently, discriminability of substates should increase with the familiarity of stimuli. Moreover, particularly well-stabilized substates might then emerge even in the absence of stimulation and appear as stimulus-specific correlation patterns in resting state activity (replay).

After frequent exposure to the same set of stimuli, performance of classifiers trained to discriminate between stimuli improved, suggesting enhanced orthogonalization of stimulus-specific substates. Accordingly, PCA analysis of the activity vectors revealed increased segregation of representations in the high-dimensional space for stimuli that had been presented particularly frequently (175). Also in agreement with this prediction is the evidence that stimulus-specific response vectors evoked by frequently repeated stimuli are replayed in the primary visual cortex during resting state. Classifiers trained to reconstruct stimuli from spontaneously occurring response vectors achieved much better reconstructions for stimuli that had been presented frequently than for less familiar stimuli (175); see also ref. 177. These experiments were performed under anesthesia, making an involvement of supervised or reinforcement-based learning processes unlikely. This interpretation is supported by simulations of self-organizing recurrent networks. If the recurrent connections are endowed with Hebbian synapses, they reproduce this improved classification of familiar stimuli (121, 175, 178). Direct evidence for unsupervised modifications of recurrent connections by repeated stimulation is also available from the cat visual cortex of anesthetized cats (166). Repeated exposure to drifting gratings led to an expansion of those orientation domains that were responsive to the conditioning stimulus. These changes involved modifications at the level of the tangential recurrent connections, supporting further the notion that correlated activation of network nodes enhances their reciprocal coupling.

Prediction v.

Because Hebbian modifications of synaptic connections require a high amount of cooperativity (179), network states exhibiting a high degree of coherence should be particularly well suited to support this kind of nonsupervised learning.

The modifications of orientation domains induced with drifting gratings in the cat visual cortex (see preceding paragraph) occurred only when stimuli induced well-synchronized responses, in this case, based on gamma oscillations. When stimuli failed to induce synchronized activity but just induced temporally dispersed discharges of comparable frequency, the orientation domains did not change. Rather, the responses of the neurons driven by the conditioning stimulus showed a long-lasting decrease in amplitude (166) as is characteristic of habituation or long-term depression.

Prediction vi.

Top-down signals associated with attention, expectancy, and prediction should influence the network’s dynamic state and thereby facilitate convergence toward substates that serve best the respective behavioral goal. This should be reflected by changes in the dimensionality and correlation structure of network activity and the speed of convergence.

With respect to attention effects on network dynamics of V1, evidence is still sparse. There is consensus that spatial attention exerts only weak to moderate effects on rate responses and the power and frequency of synchronized oscillations in the primary visual cortex. A consistent finding is a decrease of correlations (140, 180–183). This suggests that attention shifts network dynamics toward a state that permits rapid and minimally biased transitions into stimulus-specific substates once sensory evidence becomes available. The majority of studies investigating attention effects on dynamics concentrated on changes of coherence between visual areas (184–188). These studies were inspired by the communication through coherence hypothesis (58) which postulates that synchronization between the oscillatory activity of sender and receiver facilitates communication. Attention has been shown to increase coherence of oscillatory activity between V1 and V4 whereby coherence measures were based on local field potentials (LFPs) which reflect mainly synaptic activity (see above). It has been concluded from these results that transmission of the attended signals is facilitated by increasing synchronization between V1 and V4 (184, 186). This interpretation has recently been challenged by the demonstration that increases of LFP-derived coherence measures do not necessarily imply enhanced effective communication (189). The reason is that the LFPs in the target area are, to a large extent, due to synaptic currents caused by the afferents from the sending area, and, therefore, changes of the sender´s dynamics can mimic changes in coherence measures.

By contrast, strong effects on cortical dynamics have been observed by the expectancy of having to react to a rewarding stimulus. Monkeys were shown a sequence of three temporally segregated stimuli consisting of identical drifting gratings and were trained to respond swiftly to a change in the grating´s orientation. The color of the fixation spot indicated to the monkey whether the second or the third stimulus was going to change. In this case, the responses to the cued grating were associated with a large increase (several hundredfold) in the power of the gamma oscillations and synchronized firing compared to the grating that promised no reward. Despite these drastic changes in network dynamics, the discharge frequencies of the responses to the cued and uncued stimuli were similar, confirming again that discharge rates and the correlation structure of network dynamics can change independently (110). Thus, focusing attention on a stimulus that required a swift response to obtain a reward changed the correlation structure of the dynamic state induced by this stimulus. Unlike with spatial attention, this change was particularly strong when the rewarded stimulus had to be selected from a sequence.

In conclusion, available evidence supports many of the core predictions derived from the hypothesis that, for its computations, the cerebral cortex exploits the high-dimensional state space provided by the dynamics of recurrent networks. In the sections above, only low-level functions were considered. However, the proposed computational strategy can be iterated at all levels of the processing hierarchy, because all cortical areas possess the required recurrent circuits. At early levels of processing, the weight distributions of the reciprocal connections capture only the elementary Gestalt rules of perception (190), and the resulting grouping operations can therefore support only low-level functions such as feature binding, scene segmentation, and figure–ground segregation. Grouping criteria based on higher-order statistics of natural scenes are, in all likelihood, stored in the functional architecture of downstream areas. However, these can, through feedback connections, contribute additional context for the disambiguation of grouping operations in the primary visual cortex. It is only at the highest levels of the processing hierarchy, where nodes respond to complex constellations of features due to iterative recombination of feed-forward connections, that assemblies of transiently bound nodes could represent entire perceptual objects, abstract categories, concepts, or action plans. However, apart from the evidence that recurrent connections also have a distinct topology (191) in higher cortical areas, little is known about the principles that govern the layout of these connections. In this context, it is noteworthy that comparison between artificial and natural systems has revealed that certain functional properties of nodes in higher cortical areas can only be accounted for if feed-forward processing is complemented by recurrent processing (192).

Putative Functions of Oscillatory Mechanisms

As mentioned above, stimuli that simultaneously drive a sufficient number of strongly coupled nodes induce prominent, well-synchronized oscillating responses. The reason is that interactions among nodes with a propensity to oscillate give rise to resonance and entrainment in synchronized oscillatory activity (82, 100, 193, 194). The propensity of cortical networks to oscillate is due to multiple mechanisms that operate at different scales. They range from individual neurons with pacemaker characteristics (131, 195), over microcircuits with recurrent inhibition (141, 196–206), to large-scale recurrent interactions (106). Therefore, oscillations are a hallmark of cortical activity, which raises the question of whether they are merely an unavoidable epiphenomenon of circuits serving specific functions such as gain control, normalization, noise suppression, etc., or whether they serve specific functions (refs. 77 and 78; for a review of controversial interpretations, see ref. 80).

Oscillatory mechanisms have an undisputed functional role in pattern generator networks that serve the control of movements. These networks possess neurons with pacemaker currents that generate and stabilize rhythmic activity (refs. 207–209; for a review, see ref. 61). Evidence indicates that a substantial fraction of neurons in the primary visual cortex of cats and monkeys also have properties of pacemaker cells (131, 195). These oscillate in the same frequency range as the ING and PING circuit (210), which suggests that they might serve to enhance and stabilize oscillatory resonance.

Oscillatory resonance amplifies and facilitates interactions, stabilizes dynamic states, and generates precise temporal relations among discharge patterns (205, 211). Facilitation of interactions by resonance might be required to assure communication among distant, not directly connected nodes that can only interact via polysynaptic paths. Such long-distance interactions are likely needed to accomplish context-dependent scene segmentation in large visual areas with a high magnification factor such as the primary visual cortex of cats and monkeys. This could be the reason why pacemaker cells are implemented in monkey and cat V1, which has a high magnification factor, but seem to be missing in small cortical areas such as the rodent visual cortex and higher visual areas of the monkey (131).

Another process likely to benefit from oscillatory resonance is synaptic plasticity. Its induction is facilitated by precise temporal alignment of discharges, strong cooperativity, and repetition of the change-inducing activity patterns (see above). All of these prerequisites are fulfilled if networks are seeded with oscillatory mechanisms that allow for resonance. These mechanisms stabilize dynamic states and establish precise temporal relations among discharges. The finding that experience-dependent synaptic modifications are facilitated in the visual cortex by entrainment of the network in synchronized gamma oscillations supports this interpretation (166).

Oscillatory mechanisms could also be exploited to establish precise temporal relations between responses that either lack distinct temporal structure or are of different origin. This is likely the case for all self-generated activity patterns, for example, the replay of memories, but also for responses to stationary stimuli or responses from different sensory modalities. When these activities have to be subject to binding operations for the establishment of new associations, a common temporal structure has to be imposed on neuronal discharges by internal mechanisms. This is necessary to make discharges contiguous in time and to comply with the exquisite sensitivity of synaptic plasticity mechanisms to temporal correlations. Thus, oscillatory modulation of discharges and resonance could be exploited to achieve the temporal coordination required for associative learning.

Finally, the stabilizing effect of oscillatory resonance may also serve to maintain informative states for a sufficiently long period of time to permit 1) their disambiguation by top-down signals from higher cortical areas and 2) the integration of temporally segregated stimuli for the analysis and encoding of sequences.

However, the flip side of oscillatory resonance is that it takes time to evolve and to dissolve and, if too generalized, destroys information. Thus, global entrainment of neuronal populations in synchronous oscillations as it occurs, for example, with the highly redundant grating stimuli may actually prevent transmission of signals in downstream areas. The inhibitory neurons at the input layer IV of downstream areas receive highly convergent connections from the respective upstream areas. Therefore, their tuning is broad, and they respond strongly to globally synchronized input. Thus, globally synchronized input from an upstream area likely prevents further transmission by blocking responses of the excitatory target cells in the downstream area (212). This is one of the reasons why pyramidal cells in the primary visual cortex respond poorly to diffuse flashes of light which elicit globally synchronized but spatially unstructured input from the lateral geniculate. Thus, global synchronization could act as a filter that suppresses redundant information. Whether synchronization of discharges facilitates or prevents propagation of signals may therefore depend critically on the spatial structure of the synchronized activity. Traveling waves are particularly interesting in this context, as they endow synchronized oscillatory activity with not only temporal but also spatial structure (102, 106) and have recently been shown to influence behavioral choice in primates in a signal detection task (213). In contrast to classical population codes, traveling waves are not separable as the product of a spatial function and a temporal function. This mathematical difference makes it possible to convert temporal information into a spatial code and vice versa (see ref. 102).

When discussing putative functions of oscillations, it is also important to consider that the propensity of cortical networks to engage in oscillatory activity is regulated by ascending modulatory systems (214–217) and top-down control (140, 182, 184, 203). These mechanisms allow adjustments of oscillatory resonance and thereby control the temporal and the spatial scale of interactions among distributed responses.

Conduction Delays

Coupling delays among nodes are variable because recurrent connections differ in diameter, length, and myelination, raising the question of whether variable conduction delays of axons are exploited for computation or are an unavoidable epiphenomenon of neuronal hardware that the system has to cope with.

Conduction delays, if exceeding a critical threshold, may jeopardize oscillatory resonance and synchronization, prevent effective summation of convergent inputs at conjunction-specific nodes, and pose a problem for the time-sensitive synaptic plasticity mechanisms. On the other hand, conduction delays between the nodes may have computational advantages. Recurrent systems with delay are infinite dimensional and thus offer an immensely large coding space (218). Moreover, as shown by Muresan and Savin (138), delays stabilize activity in recurrent networks and may thus be an essential ingredient of recurrent processing. Adjustment of conduction velocities can serve to render inputs from different sources contingent in time, to generate sequences, and to produce functionally relevant phase offsets. Several observations suggest that these possibilities are exploited by the brain. Through variations in axon diameter and myelination, it is assured that impulses in bifurcating callosal fibers arrive at exactly the same time in the widely distributed terminal arbors (219). Likewise, conduction times of connections in the cat visual system between area 17, the suprasylvian cortex (PLMS), and the superior colliculus are tuned in a way that assures that signals that are correlated between area 17 and PMLS arrive simultaneously in the tectum despite differing path length (220). In the auditory system, delay lines assure coincidence of binaural signals on common target cells in the mesencephalon to achieve direction tuning for auditory targets (221). These few examples indicate that neuronal connections are specific not only in the spatial but also in the temporal domain. How this specificity is achieved is little explored. One possibility is activity-dependent selection of afferents during development by the established correlation-sensitive synaptic plasticity mechanisms. As these stabilize, selectively, afferents whose activity is correlated in time, they can serve to select, among axons with different conduction velocities, those whose discharges are coincident. Another possibility is activity-dependent modification of conduction velocity. There is now ample evidence that neural activity influences myelination by stimulation of oligodendrocytes (222). Whether these modifications of conduction velocity depend on correlations between presynaptic and postsynaptic activity, which would be required to associate inputs according to the coincidence of their discharges, is unknown. The fact that learning processes are accompanied by changes in the myelination pattern of fiber tracts (223–225) suggests that myelination might indeed be controlled by associative mechanisms. Taken together, these observations suggest the possibility that neuronal networks are structured with similarly high precision in the temporal as in the spatial dimension. If this turns out to be true, conduction delays would be an important variable of computational architectures rather than an unavoidable epiphenomenon, further emphasizing the critical role of dynamics and precise control of temporal relations in information processing.

Concluding Remarks

The rapid switching or collapse of a high-dimensional resting state toward a lower-dimensional substate bears similarities to other processes that have been proposed to exploit dynamic interactions in reciprocally coupled systems for computations. In addition to the concepts derived from reservoir computing mentioned above (117–122), these comprise theories based on attractor dynamics, aliasing processes in Little−Hopfield nets and Ising models (226), and the concept of free energy reduction (227). Last but not least, there is a puzzling analogy with the processes that make quantum computing so fast and efficient. The superposition of wave functions bears similarities to the covert superposition of priors in the correlation structure of spontaneous activity, and the simultaneous and probabilistic evaluation of nested relations resembles the virtually simultaneous and holistic interaction between network nodes that represent, in a probabilistic and graded way, the presence of particular features. It would be truly fascinating if evolution had succeeded to realize, with classical mechanisms, those functions that quantum computers are particularly good at: the parallel and therefore ultrafast evaluation of the relations between a huge number of probabilistic variables.