A neural code for three-dimensional object shape in macaque inferotemporal cortex.

Previous investigations of the neural code for complex object shape have focused on two-dimensional pattern representation. This may be the primary mode for object vision given its simplicity and direct relation to the retinal image. In contrast, three-dimensional shape representation requires higher-dimensional coding derived from extensive computation. We found evidence for an explicit neural code for complex three-dimensional object shape. We used an evolutionary stimulus strategy and linear/nonlinear response models to characterize three-dimensional shape responses in macaque monkey inferotemporal cortex (IT). We found widespread tuning for three-dimensional spatial configurations of surface fragments characterized by their three-dimensional orientations and joint principal curvatures. Configural representation of three-dimensional shape could provide specific knowledge of object structure to support guidance of complex physical interactions and evaluation of object functionality and utility.

A primary goal in the study of object vision is to decipher the neural code for complex object shape. At the retinal level, object shape is represented isomorphically (i.e., replicated point-for-point) across a 2D map comprising approximately 106 pixels. This isomorphic representation is far too unwieldy and unstable (due to continual changes in object position and orientation) to be useful for object perception. The ventral pathway of visual cortex1–2 must transform the isomorphic image into a compact, stable neural code that efficiently captures the shape information needed for identification and other aspects of object vision.

Previous studies of complex shape coding have focused on 2D pattern representation. These studies have shown that neurons at intermediate (areas V2 and V4) and final (IT) stages in the monkey ventral pathway process information about 2D shape fragments. V2 and V4 neurons encode curvature, orientation, and object-relative position of 2D object boundary fragments3–7. At the population level, these signals combine to represent complete boundary shapes as spatial configurations of constituent fragments8. In posterior IT, individual neurons integrate information about multiple 2D boundary fragments, producing explicit signals for more complex shape configurations9–10. In central/anterior IT, the homologue of high-level object vision regions in human cortex11–13, neurons are selective for a variety of patterns and that selectivity is organized across the cortical surface in a columnar fashion14–18. At each stage, neurons appear to be tuned for component-level shape, although holistic shape tuning can evolve in IT through learning19. Holistic object representation may be more fully realized in medial temporal brain structures associated with long-term declarative memory20 and in prefrontal areas processing categorical object information21.

The question addressed here is whether and how complex 3D shape is encoded by IT neurons. Our specific hypothesis is that IT neurons encode 3D spatial configurations of surface fragments. Under this hypothesis, the 2D structural representations described above could be considered to occupy a subspace within the higher-dimensional 3D structure domain. (I.e., surface fragments forming the 2D self-occlusion boundary of an object would be a special case of 3D surface fragments.) This hypothesis is consistent with classic shape coding theories in which objects are represented as 3D spatial configurations of simple 3D parts22,23. The alternative hypothesis, advanced in more current theories, is that complex shape perception is based primarily on 2D image processing. According to these theories, consistent recognition of 3D objects from different vantage points is achieved by learning associations between multiple 2D views24. The multiple-views hypothesis avoids the time-consuming computational complexity of inferring 3D structure. This hypothesis is supported by psychophysical results showing that view-invariant recognition is learning-dependent25,26 and by computational studies showing that 2D image processing can support rapid, accurate object identification27,28. (However, these results and the multiple-views hypothesis itself are also compatible with 3D representation29; see Discussion.)

The classic hypothesis that complex shapes are represented as 3D spatial configurations of 3D parts has yet to be tested at the neural level. Previous studies have shown differential responses across a small number of 3D shapes30 or tuning along a single depth-related dimension31–34, but such results cannot demonstrate or explain complex 3D shape representation. (Similar responses in dorsal pathway cortex have been interpreted as signals for orientation in depth35.) Representation of 3D object shape would require neurons with much more complex, multidimensional tuning properties. That kind of tuning can only be measured with large stimulus sets in which a wide range of 3D shape elements are combined in many different ways, so that quantitative analyses can disambiguate which 3D shape factors (if any) are uniquely and consistently associated with neural responses. This has not been attempted before due to the intractable size of 3D shape space. In this virtually infinite domain, a conventional random or systematic (grid-based) stimulus approach can never produce sufficiently dense, combinatorial sampling.

We addressed this obstacle with a novel evolutionary stimulus strategy. Beginning with an initial generation of 50 random 3D shapes, stimuli evolved through multiple generations under the guidance of neural feedback. Ancestor stimuli from previous generations were probabilistically morphed, either locally or globally, to produce descendant stimuli that varied the ancestor’s shape characteristics and/or combined them with new shape features. The average neural response to each stimulus determined the probability with which it produced morphed descendants in subsequent stimulus generations. This strategy has two advantages. First, sampling becomes increasingly focused around the response range of the neuron, so that far less experimental time is spent sampling null response regions. Second, in the high-response stimulus lineages, the (initially unknown) shape characteristics encoded by the neuron are repeatedly varied and recombined with other shape features. The result is much denser, more combinatorial sampling in the most relevant region of the 3D shape domain. (In contrast, standard gradient descent search aims to identify a single, maximum response stimulus, which by itself cannot reveal what specific shape information is encoded by a neuron.) This evolutionary stimulus strategy made it possible for the first time to test the 3D configural coding hypothesis at the neural level.

RESULTS

Random 3D shape stimuli were constructed by extensively deforming a closed ellipsoidal surface ( online). These stimuli were rendered in depth by a combination of binocular disparity and shading cues. Two rhesus monkeys were trained to maintain fixation on a small spot while stimuli were flashed for 750 ms each at the center of gaze. We recorded the stimulus responses of individual neurons in central and anterior IT (between 5.8 and 21.0 mm anterior to the interaural line). Neurons were typically tested with 8–10 generations of 50 stimuli each. When recording time permitted, the entire procedure was run a second time, to verify that stimulus evolution would converge in the same direction (, left and right columns). In each run, the initial generation was completely random, and responses were generally low ().

Higher response stimuli had a greater probability of producing morphed descendants in subsequent generations, and response levels progressed across generations in a way that was sometimes gradual and sometimes punctuated (; also see Supplementary Figs. and online). In any given generation, only a few descendants produced higher responses; most descendants evoked equivalent or lower responses. These lower response samples in neighboring shape space are essential for characterizing the shoulders and boundaries of tuning functions.

For this example cell, both lineages converged toward shapes that varied on the global level but contained consistent local structure comprising sharp protrusions and indentations, oriented toward the right and positioned to the upper right of object center (). We quantified this response pattern with a multi-component model analogous to those applied recently to dorsal pathway area MT and in posterior IT cortex9–10, 36. In the MT analyses, the model components represent tuning for different movement directions in the orientation domain. Here, the model components represent tuning for different surface fragments in a 3D curvature/orientation/position domain. The two tuning models based on the two runs for this neuron () both comprised two component Gaussian functions, denoted by magenta and cyan. The magenta functions are centered on strongly positive (convex) maximum curvature and negative (concave) minimum curvature, indicating a hyperbolic or saddle-shaped surface. This surface is oriented to the right (0°) and positioned to the upper right in the x/y plane. The cyan functions are centered on convex/convex curvature, oriented downwards (270°) and positioned to the upper right.

The predicted response for a given stimulus depended on how closely any of its constituent surface fragments matched these tuning functions. The overall predicted response was a linear combination of predictions based on the two separate functions plus a nonlinear interaction term (, equations). In this case, the nonlinear interaction terms had the highest fitted weights, showing that neuron was relatively selective for the combination of both types of surface fragments. Higher response stimuli in both runs included surface fragments near these two Gaussian functions (, magenta- and cyan-tinted surface regions). In this and all subsequent stimulus plots, the tinted regions include any additional surface structure strongly correlated with the Gaussian tuning function, to better capture the entire surface configuration associated with neural responses. This is necessary because the geometry of closed, continuous surfaces imposes strong local structure correlations. For example, sharp points (tangent discontinuities) are necessarily correlated with the conjoined surfaces that define them, as in . From a mechanistic point of view, the neuron’s response could be driven by a sub-portion of that structure, but geometric constraints make that difficult or impossible to test in an experiment of this kind.

These tuning models showed strong cross-prediction of responses in the other run (r = 0.67 for run 1 model cross-prediction of run 2; r = 0.63 for run 2 model cross-prediction of run 1). The two runs provide a rigorous cross-validation test since they were generated completely independently. (Geometric similarities between runs are imposed by the neuron itself, and thus serve to confirm the generality of the tuning model.) Cross-validation analyses of all cells with double lineages (see online) established that models based on two Gaussian tuning regions had greater statistical validity than simpler models based on one Gaussian and more complex models based on three, four, or five Gaussians. Restricting models to subunits explaining at least 5% additional variance produced a corresponding predominance of 2-Gaussian models (see online). The results presented below are based on two-Gaussian models for 95 neurons that showed statistically significant cross-validation between two runs (n = 16) or 5-fold cross-validation within one run (n = 79). We corrected the 5-fold cross-validation procedure for standard error of the mean response measurements to estimate that these models accounted for 32% of the explainable response variance on average (mean r = 0.57; see ).

exemplifies the expected pattern for cells that encode 2D boundary shape. Even though the response model domain encompasses 3D surface fragments, the fitting procedure finds surfaces at the 2D occlusion boundary, with normal orientations in the image plane, essentially corresponding to 2D boundary fragments. Correspondingly, control tests showed that this cell responded nearly as strongly to the same shapes when all depth cues were removed (producing a silhouette shape with no internal detail; online; 54 spikes s-1 with stereo and shading cues, 44 spikes s-1 without). However, this result did not typify the majority of neurons in our sample.

The more common finding was tuning for surface fragments outside the image plane, as in . The highest response shapes for this neuron were characterized by a ridge facing out of the image plane (; also see online). Stimuli lacking this surface characteristic evoked little or no response (). Control tests performed on stimuli drawn from the top, median, and bottom of the response range (, top, middle, and bottom rows) confirmed that this neuron’s shape tuning was dependent on depth structure and robust to other image changes. The neuron remained responsive when either disparity or shading cues for depth were present but failed to respond when both were eliminated (; see also online). 3D shape selectivity remained consistent across lighting directions over 180° vertical and horizontal ranges (), in spite of the resulting dramatic changes in shading patterns across the object surface ( online). Selectivity was likewise consistent across changes in stereoscopic depth () and x/y position (), although not across changes in stimulus orientation (). Selectivity was consistent across stimulus size (). The response model () comprised tuning for forward-facing ridges in front (magenta) and upward-facing concavities near object center (cyan). The model for this neuron was highly nonlinear, as shown by the large AB interaction term representing combined energy in the two tuning regions. Thus, this neuron provided a relatively explicit signal for the ridge/concavity configuration. (By “explicit”, we mean having a simple, easily decoded relationship to the shape configuration in question. Linear integration of information about multiple fragments would produce more ambiguous, less explicit signals, in which the same response level could correspond to either part A or part B.) This configuration characterized the high response stimuli in the evolutionary test (; one high response stimulus is shown from both the front and above for greater visibility of the two surface components; see also online).

Tuning for 3D surface configurations (as in ) was more common in our sample than tuning for 2D boundary configurations (as in ). This was established by follow-up control tests measuring how responses depended on the presence of 3D cues (as in ). The critical comparison from these tests is between the condition in which the standard 3D cues (disparity and shading) were present and the condition in which no 3D cues were present, so that the stimulus was a plain 2D silhouette. In each condition, strength of modulation was measured by the response difference between a stimulus drawn from the top of the main test response range and a stimulus drawn from the bottom of the range, normalized by the maximum response across all conditions. Response modulation was generally strong (near 1.0) when 3D depth cues were present (, horizontal axes). When depth cues were removed (vertical axes) modulation sometimes remained strong (points near the upper right), indicating that responses were based on 2D boundary shape. More commonly, however, modulation dropped to near 0, reflecting sensitivity to 3D shape. The drop in response modulation with removal of 3D cues does not simply reflect selectivity for 2D shading patterns, since 3D shape tuning was robust to changes in shading pattern produced by different lighting directions. We quantified this by measuring the separability of tuning for shape and tuning for lighting direction (). 3D shape responses were likewise robust to changes in x/y position, stereoscopic depth and stimulus size (), analogous to previous results showing similar tolerance in 2D shape responses37.

Tuning models spanned a wide range of surface fragment configurations (). Component fragments were frequently discontiguous, consistent with a multi-fragment configural coding scheme. Even when tuning regions were extended to cover structures strongly correlated with the fitted Gaussians (as in ), only a fraction (on average 23%) of total object surface area was included ( online). Correspondingly, the global shape of high response stimuli varied at locations outside these surface regions. Thus, 3D shape representation in IT is not generally holistic. IT neurons represent spatially discrete 3D shape components, and must cooperate in a distributed coding scheme.

Tuning models showed a predictable bias toward surface positions near the front of the object, which is more visible and behaviorally relevant under normal circumstances (). Tuning was markedly biased in the curvature domain toward high values, especially on the convex end of the scale. Thus, although object surface area is dominated by flat or broad curvature (), the IT representation of 3D shape emphasizes sharper projecting points and ridges (). Tuning for non-zero curvature reflects the coding advantage of higher order derivatives38. The bias toward convexity may reflect the functional importance of protruding object parts and/or the well-established perceptual bias toward interpreting convexities as objects parts and concavities as junctions between parts39.

DISCUSSION

We tested the classic hypothesis that complex shapes are represented as 3D spatial configurations of 3D parts. This hypothesis requires that neurons encode the 3D shape, 3D orientation, and relative 3D position of object parts. Our analyses show that a substantial fraction of IT neurons do exactly that—they are simultaneously tuned for 3D shape (maximum and minimum principle surface curvatures), 3D orientation, and relative 3D position of constituent surface fragments. Moreover, they are tuned for multiple regions in this domain, i.e. they respond to shapes that include a specific configuration of particular surface features. This result supports classic theories of 3D configural shape representation22,23, and extends those theories by suggesting that neurons encode not just individual parts but configural relationships between multiple parts.

In many cases, our analyses and follow-up control tests revealed exclusive tuning for 2D boundary shape. The majority of neurons in our sample, however, were clearly tuned for 3D spatial configurations of 3D surface fragments. Our control tests eliminate explanations based on other stimulus properties besides 3D shape. For example, neural responses might have been associated with 2D boundary shape features, since these can be strongly correlated with 3D surface shape40. But our depth cue test showed that, for most cells, removing 3D information and presenting only the 2D boundary resulted in a drastic loss of tuning/responsiveness (). Alternatively, neural responses might have been associated with the 2D shading patterns we used to help convey 3D shape. But our lighting direction test showed that dramatic changes in the 2D shading pattern () had little effect on shape tuning (). Neural response differences might have reflected tuning for binocular disparity, but our depth position test showed that large changes in disparity had little effect on tuning (). These three control tests eliminate alternate explanations based on any stimulus characteristic associated with the 2D boundary, 2D shading pattern, or disparity values, and these three elements constitute all the image information present in these stimuli. Likewise, large changes in stimulus size and position typically had little effect on tuning (). No other potentially explanatory stimulus properties apart from 3D surface shape itself would have survived all these major image changes. As a further control, we performed an analysis to show that response variations could not be explained in terms of image spatial frequency content ().

Our results are largely consistent with classic theories of configural shape representation22,23. According to these theories, objects are represented as spatial configurations of canonical 3D parts. The parts are generalized cones or more complex volumetric components called “geons”. Their configuration is represented in an object-defined reference frame that translates, scales, and rotates with the object (producing invariance to viewpoint). Correspondingly, we observed explicit signals for configurations of 3D surface fragments that appear to be encoded in an object-relative 3D reference frame. By “explicit”, we mean easily decoded signals with clear, simple relationships to large-scale 3D object structure. In contrast, although the same information is necessarily present in V1 (from which IT responses ultimately derive), the V1 representation is highly implicit and difficult to decode, because it is distributed across a much larger population of neurons with complex, highly variable relationships to large-scale 3D object structure.

However, our results differed in two ways from classic theories. First, while the neural reference frame appeared to translate and scale with the object (given the consistency of responses across stimulus position and size) we did not find any evidence that it rotates with the object. Generalization across object rotations may depend on learned associations between views24,41, although there is some capacity for recognizing rotated views of novel objects42. Representation of 3D structure is not incompatible with that view29. Second, classic theories envision that individual neurons would represent single, spatially discrete parts. Instead, consistent with our previous 2D studies, we found that neurons represent configurations of multiple parts, frequently at distant, discontiguous locations on the object surface ( online). Integration across multiple boundary and surface regions to derive larger, more complex configurations may be a ubiquitous aspect of visual processing43. Complete 3D shapes might be represented in terms of a small number of component surface configurations (). A coding scheme like this still has the combinatorial productivity of parts-based representation—a finite number of signals can be combined in many different ways to represent a virtual infinity of objects. At the same time, it would constitute a step toward holistic shape coding, which has greater potential for sparseness. A sparse object representation based on just a few signals can be more efficiently stored in memory and decoded by other parts of the brain44,45. The configural coding we observe may reflect a compromise between productivity and sparseness in higher-level visual cortex.

The neural code we observed here is a 3D generalization of the 2D coding scheme we have previously described8–9. Boundary fragments in 2D generalize to surface fragments in 3D. The circular orientation domain for 2D boundary fragments in the image plane generalizes to the spherical orientation domain for 3D surface fragments. Tuning for 2D boundary fragment curvature generalizes to joint tuning for the two principal curvatures characterizing any 3D surface fragment. Tuning for relative position of boundary fragments in the 2D image plane generalizes to tuning for relative position of surface fragments in 3D space. Conceivably, the 2D shape tuning properties that we and others have studied previously are not in any way distinct, and simply occupy a subspace in the higher dimensional 3D domain. In other words, neurons tuned for 2D shape might simply be a subpopulation (within a 3D shape representation) encoding surface fragments near the occlusion boundary. The predominance of 3D shape sensitivity in our neural sample supports this suggestion. However, this bias might be due to denser sampling in the superior temporal sulcus, which has been reported to contain more 3D-sensitive neurons46 (see online).

To summarize, we used a novel evolutionary stimulus strategy to reveal for the first time an explicit structural code for 3D object shape in high level visual cortex. This code is embodied by neurons tuned for configurations of multiple object surface fragments. This result rules out the alternative that complex shape representation is primarily 2D, as in most current computational models of object vision. It supports classic theories of 3D structural representation22,23, but qualifies those theories in two ways. First, individual neurons represent not single object parts but configurations of multiple object parts, providing more explicit signals for spatial relationships between shape elements. Second, the spatial reference frame is only partially defined by the object. The reference frame is centered with respect to the object, but contrary to classic theories does not rotate with the object. Thus, neural representations are very different depending on viewpoint, and relationships between different views of the same object would have to be learned, as proposed in view-dependent theories of object representation24–26. Finally, these results apply only to the population of neurons we were able to study effectively with our stimulus strategy, and they do not rule out other object processing mechanisms in other neural populations. In particular, rapid, coarse object categorization might depend on faster, feedforward processing of 2D image information28.

Why would the brain explicitly represent complex 3D object shape, considering the computational expense of inferring 3D structure from the 2D retinal image and the higher neural tuning dimensionality required? In contrast, 2D shape representations are lower-dimensional and can be derived quickly and directly from the retinal image. Moreover, computational studies of biologically-inspired hierarchical network models show that 2D image processing alone can produce rapid, accurate object identification27,28. We speculate that representation of 3D object structure instead supports other aspects of object vision beyond identification. Direct cognitive access to 3D object structure makes even unfamiliar objects comprehensible in terms of their geometric similarity to familiar objects, inferred physical properties, potential functionality and utility, and aesthetic qualities. Knowledge of 3D structure is also necessary for accurate prediction of physical events and control over complex physical interactions with objects. These cognitive and behavioral requirements may have driven the emergence of an explicit neural code for 3D object structure in visual cortex. Similar tuning might emerge from hierarchical network models given 3D input information and more diverse task demands.

METHODS

Behavioral task

Two head-restrained rhesus monkeys (Macaca mulatta), a 9.5 kg male and a 5.3 kg female, were required to maintain fixation on a 0.1° spot within a 0.75° radius for 4 s in order to obtain a juice reward. Eye position was monitored with an infrared eye tracker (ISCAN). Separate left and right eye images were presented via mirrors. Binocular fusion was verified with a random dot stereogram search task. All animal procedures were approved by the Johns Hopkins Animal Care and Use Committee and conformed to National Institutes of Health and US Department of Agriculture guidelines.

Electrophysiological recording

The electrical activity of well-isolated single neurons was recorded with epoxy-coated tungsten electrodes (A-M Systems) and amplified and filtered in a Tucker-Davis Technologies acquisition system. We studied 250 neurons with at least 300 stimuli. Of these, 95 (59 from the male, 36 from the female) produced models that passed our statistical thresholds (Results). These neurons were sampled from the central/anterior lower bank of the superior temporal sulcus and lateral convexity of the inferior temporal gyrus (5.8–21.0 mm anterior to the interaural line; see ). IT cortex was identified based on structural MRI images.

Visual stimuli

3D shape stimuli were rendered with shading and binocular disparity cues using the NURBS facility in OpenGL (gluNurbsSurface). NURBS control point positions were varied by distorting a polar grid (see ). Stimulus depth was adjusted so that the depth of the object surface at the fovea matched fixation depth (screen distance).

Neurophysiological testing protocol

A roughly optimal stimulus color (white, red, green, blue, cyan, yellow, or magenta) was selected based on responses to stimuli under experimenter control. During all subsequent tests, following initiation of fixation, four stimuli were flashed one at a time for 750 ms each, with inter-stimulus intervals of 250 ms. Control tests of sensitivity to depth cues, x/y position, stereoscopic depth, size, orientation, and lighting direction (see ) were performed on three stimuli, drawn from the top, median, and bottom of the response range in the main experiment.

Evolutionary morphing algorithm

The first stimulus generation contained only randomly generated 3D stimuli. Stimulus responses (averaged across 5 repetitions) were ranked into 10 bins with equal numbers of stimuli. In the second generation, 10–20% of stimuli were randomly generated. The rest were morphed descendants of ancestor stimuli from the first generation, selected randomly in equal numbers from the 10 bins. Thus, a typical second generation would contain 10 stimuli generated de novo, 4 descendants of stimuli in the highest response bin, 4 descendants from the second highest bin, etc. In subsequent generations, ancestor stimuli were pooled across all preceding generations and re-binned. Descendant stimuli had an equal probability of being either locally or globally morphed (see ). The amplitude of control point changes was inversely proportional to response rate, to produce denser sampling in higher response ranges.

3D shape response models

We characterized each 3D stimulus in terms of its component surface fragments. The stimulus surface was densely sampled across the NURBS control point grid. At each point, 7 values were determined: x, y, and z positions, relative to stimulus center of mass, surface normal orientation on the x/y and y/z planes, and maximum and minimum (principal) cross-sectional curvatures, squashed to a range from −1 to 1 using a sigmoidal function. Component surface fragments were defined by mathematically fitting elliptical regions on this grid with approximately constant curvature in either the minimum or maximum curvature dimensions. Each ellipse was shifted and scaled to be as large (i.e. cover as many grid points) as possible without violating constraints on maximum deviation of curvature values. Successive ellipses were fitted to remaining regions on the object surface. Curvature maxima and minima were also used to define surface fragments. On average, a given stimulus comprised 240 component surface fragments. For each fragment, maximum and minimum curvature were averaged across the included surface points, and the position and orientation values were measured at the point of maximum or minimum curvature, yielding 7 measurements. Thus, each fragment corresponded to a point in a 7-dimensional domain, and each stimulus was represented by a constellation of such points. All the models described here were based on 7D Gaussian tuning functions (model subunits) in this domain. The predicted response component due to a given subunit was based on the Gaussian function amplitude at the stimulus point closest to the Gaussian peak: where ka, kb, Θx, Θy, rx, ry, and rz are the curvature, normal, and relative position values for a surface fragment in the stimulus, μ and σ are the fitted Gaussian peaks and the standard deviations on each of these seven dimensions, and A is the fitted Gaussian amplitude. The standard deviations in the curvature, orientation, and position dimensions were constrained to be the same (respectively) for all Gaussian functions in order to limit model complexity (i.e. only 3 standard deviation values were required for any given model, one each in the curvature, orientation and position dimensions). Including separate standard deviation parameters for each dimension slightly increased variance explained, by a mean of 3.7% for the 1-Gaussian models and 4.5% for the 2-Gaussian models (5-fold cross-validation, n = 95). Since the difference in explained variance between 1- and 2-Gaussian models increases, this would not impact our conclusion that neurons are tuned for configurations of multiple surface fragments rather than single fragments. The analyses presented here are based on the models with just 3 standard deviation parameters.

For each cell, we fitted models based on combinations of 1–5 excitatory Gaussian subunits. The predicted response to each stimulus was a weighted combination of the individual subunit responses (the linear component) and a product of subunit responses (the nonlinear interaction component). All possible combinations of subunits for nonlinear components were tested, but in the final models each subunit could participate in only one nonlinear interaction term. For the 2-Gaussian models emphasized in this report, there was only one possible interaction term. where R is the unweighted response predicted by each subunit (derived from equation 1) and R is the unweighted response predicted by each interaction term. W is the fitted weight (amplitude) for each subunit, W is the fitted weight for each interaction term, G is the overall gain, and b0 is the baseline firing rate. The total numbers of fitted parameters for the models with 1–5 subunits were 13, 21–22, 29–30, 37–39, and 45–47, respectively. The variability in parameter number for higher order models is due to variability in the number of possible interaction terms. Overfitting was controlled by cross-validation analyses described in Results and presented in detail in .

Response rates were calculated by counting the number of spikes during the 750-ms stimulus presentation period and averaged across repetitions. The Matlab function lsqnonlin was used to adjust model parameters to minimize the sum of squared differences between observed and predicted responses. The fitting procedure was repeated using 20–163 (average 93) starting points based on the constituent surface fragments for the 3 highest response stimuli. For each neuron, the best-fitting model across all starting points was used.

In some cases, these Gaussian tuning models may fail to capture the complete shape configurations associated with neural responses, instead focusing on smaller surface fragments within the complete configurations. This problem is inevitable due to the local surface structure correlations in any closed, continuous surface (Results). To better capture the complete shape configurations signaled by the neurons, we identified additional surface structure components that were highly correlated with those identified by the fitted Gaussian tuning regions. We then extended the tuning regions to include those correlated structures, up to the point at which explained variance dropped by 5%. The extended tuning regions are bound to have a non-Gaussian shape, which we approximated with a cluster of neighboring Gaussians. Predicted responses were based on average stimulus matches across the Gaussians in each cluster. These extended models were the basis for projections onto stimulus surfaces (Figs. , and ), tuning distributions (, Supplementary Figs. and ), and analysis of fractional surface coverage (). All goodness of fit results were based on the original fitted 2-Gaussian models. The computational procedure for extending tuning regions is described in Supplementary Information (see ).

Statistical analysis

Randomization analysis of cross-validation between double lineages was used to compare the statistical validity of models based on 1–5 Gaussian tuning regions (see ). A 5-fold cross-validation procedure was used to estimate explained variance without overfitting (Results).