Cortical mechanisms of visual brightness.

The primary visual cortex signals the onset of light and dark stimuli with ON and OFF cortical pathways. Here, we demonstrate that both pathways generate similar response increments to large homogeneous surfaces and their response average increases with surface brightness. We show that, in cat visual cortex, response dominance from ON or OFF pathways is bimodally distributed when stimuli are smaller than one receptive field center but unimodally distributed when they are larger. Moreover, whereas small bright stimuli drive opposite responses from ON and OFF pathways (increased versus suppressed activity), large bright surfaces drive similar response increments. We show that this size-brightness relation emerges because strong illumination increases the size of light surfaces in nature and both ON and OFF cortical neurons receive input from ON thalamic pathways. We conclude that visual scenes are perceived as brighter when the average response increments from ON and OFF cortical pathways become stronger.

INTRODUCTION

Artists and scientists have been studying for decades how size and luminance combine to influence the perception of visual brightness (Da Vinci, 1880; Gilchrist et al., 1999; Li and Gilchrist, 1999). Already in the sixteenth century, Leonardo da Vinci combined variations in size and luminance to increase visual salience in his chiaroscuro paintings (Da Vinci, 1880). Today, vision scientists continue to study the brain mechanisms of luminance perception (Chubb and Nam, 2000; Haynes et al., 2004; Nam and Chubb, 2000; Yang et al., 2022), which have important implications in visual disease and image processing analysis. Despite its clear importance, we still do not understand how the visual system drives the perception of stimulus brightness. We know that two major brain pathways process the onset of light (ON pathway) and dark stimuli (OFF pathway) in visual scenes, and that these two pathways are remarkably well preserved in the animal kingdom, from invertebrates to humans (Mazade and Alonso, 2017). However, the relative contribution of ON and OFF pathways in the perception of visual brightness remains unclear.

A common belief in visual neuroscience is that ON and OFF pathways are similarly activated with small light stimuli and, therefore, their function cannot be separately measured in the eye clinic. This belief is rooted in the assumption that light targets drive responses from both ON and OFF retinal ganglion cells (responding when the target is turned on and off, respectively), and the two responses blend in time with eye movements. Against this common belief, recent results demonstrate that cortical neurons generate different response increments when a light target is turned on (onset responses) than when it is turned off (rebound responses). In contrast with onset responses, rebound responses are weaker, slower, more variable across stimulus presentations, more dependent on stimulus duration (absent when stimulus is brief), less temporally precise, less synchronized across neurons, have poorer stimulus specificity, and are preceded by stronger response suppression (Hirsch et al., 1998; Komban et al., 2014; Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021). These pronounced differences suggest that the onset responses from ON and OFF pathways are the ones driving the perception of light and dark stimuli, whereas the rebound responses drive the perception of afterimages (Li et al., 2017; Liu et al., 2021). Consistent with this interpretation, the inactivation of the ON pathway affects the perception of light, but not dark, stimuli in both humans and macaques (Dryja et al., 2005; Schiller et al., 1986).

ON and OFF pathways segregate in visual cortex (Jin et al., 2011b, 2008; McConnell and LeVay, 1984; Norton et al., 1985; Oliveira Ferreira de Souza and Casanova, 2019; Wang et al., 2015; Zahs and Stryker, 1988) and can function relatively independently from each other (Dryja et al., 2005; Emran et al., 2007; Sarnaik et al., 2014; Schiller et al., 1986). However, their population responses combine during natural vision, and the relative dominance of each pathway changes with the stimulus properties (Dai and Wang, 2012; Jansen et al., 2019; Kremkow et al., 2014; Mazade et al., 2019; Onat et al., 2011; Taylor et al., 2018; Williams et al., 2021; Xing et al., 2014; Zweig et al., 2015). Here we demonstrate that these changes in ON-OFF response balance have important implications for the cortical processing of stimulus luminance. We demonstrate that bright stimuli smaller than a receptive field center drive opposite responses from ON and OFF cortical neurons (increased versus suppressed activity) making the ON/OFF response ratio increase with stimulus brightness. Conversely, bright large surfaces covering the receptive field center and flanks drive response increments from both ON and OFF cortical neurons, making the average ON + OFF cortical response to increase with stimulus brightness. These results reveal a neuronal mechanism for size-brightness interactions previously described by artists and scientists (Da Vinci, 1880; Gilchrist et al., 1999; Li and Gilchrist, 1999) that could help to improve current algorithms of image processing (Rahimi-Nasrabadi et al., 2021).

RESULTS

We measured the visual responses of cortical neurons to different types of stimuli. The stimuli were image patterns containing light and dark features smaller than a receptive field center (e.g., gratings, white noise) or larger homogeneous targets that could be light (239 cd/m2) or dark (0.27 cd/m2). Our results demonstrate that the relative contribution of ON and OFF cortical pathways to signaling stimulus brightness changes with stimulus size. In the following sections, we demonstrate that ON and OFF cortical pathways have different response properties and are differently modulated by a large variety of stimulus parameters, including size, duration, and luminance. We then demonstrate that ON and OFF cortical pathways respond similarly to surfaces, and the strength of the cortical response depends on surface polarity (light or dark) more than receptive field polarity (ON or OFF). At the end of the paper, we demonstrate a relation between surface brightness and size in both nature and human vision. Throughout the paper, we use the term luminance to describe the light intensity per stimulus area (a physical property of the stimulus expressed in candelas per square meter [cd/m2]) and the term brightness to describe the human perception of stimulus luminance (a subjective perception expressed in normalized units ranging from 0 for darkest to 1 for brightest).

ON-OFF cortical dominance changes with the spatial structure of the stimulus

Our measurements show that the cortical distribution of ON- and OFF-dominated receptive fields changes with the spatial structure of the stimulus. When the visual cortex was stimulated with a sequence of stationary gratings containing light and dark bars narrower than a receptive field center, the polarity of the population receptive field from a local cortical region (ON or OFF domain) was bimodally distributed (median OFF-dominance: −0.44, median ON-dominance: 0.41, Hartigan test, p < 0.001; Figure 1A). Similarly, when the stimulus was made of white noise with light and dark squares smaller than a receptive field center, the distribution was also bimodal, although the separation between ON and OFF dominance was less pronounced (median OFF-dominance: −0.53, median ON dominance: 0.33, Hartigan test, p < 0.001; Figure 1B; see also Wang et al., 2015). Conversely, when the visual cortex was stimulated with large 7° surfaces covering the entire receptive field (center and opposite-polarity flanks), the distribution of receptive field polarity became strongly unimodal (median OFF-dominance: −0.23, median ON-dominance: 0.21, Hartigan test, p = 1; Figure 1C). Moreover, if the target was reduced in size (2.8°) but was still large enough to cover the receptive field center and flanks (average receptive-field size: 2.8°, see below), the distribution remained unimodal but became biased toward OFF-dominance (median OFF-dominance: −0.29, median ON-dominance: 0.15, Hartigan test, p = 1; Figure 1D). Consistently with these findings, the dominant polarity of the cortical receptive field (ON or OFF) was similar when measured with gratings and white noise (Figures S1A–S1C) but could be very different when measured with large squares (Figures S1D–S1F). Only cortical sites strongly ON or OFF dominated at the input layers of the cortex had the same dominant polarity when measured with different stimuli (Jansen et al., 2019; Kremkow et al., 2016; Wang et al., 2015).

Figure 1.

ON-OFF receptive field dominance changes with the spatial structure of the stimulus

(A) Contrast polarity histogram of cortical responses to gratings. Contrast polarity (CP) was calculated as (ON − OFF)/(ON + OFF), where ON and OFF are maximum responses to light and dark stimuli, respectively. Positive values indicate ON dominance (red, n = 578) and negative values OFF dominance (blue, n = 741). n: number of cortical sites, p: probability that the distribution is not bimodal (Hartigan dip test).

(B–D) Same as in (A) but for white noise (B, n = 747/564 for OFF/ON), surfaces (C, n = 465/386 for OFF/ON, 7° surfaces), and sparse noise (D, n = 944/262 for OFF/ON, 2.8° targets).

(E) Single cortical sites (top) and averages (bottom) of OFF- and ON-dominated peri-stimulus time histograms (PSTHs) measured with light (red) and dark (blue) grating phases.

(F–H) Same as in (E) but for white noise (F), surfaces (G), and sparse noise (H). Notice that gratings and white noise produce bimodal distributions for contrast polarity and PSTHs with opposite responses, whereas surfaces and sparse noise do not.

See also Figure S1.

The stimulus spatial structure not only affected the dominant polarity of the receptive field (ON or OFF) but also the response polarity (increased or suppressed activity). OFF neurons increased their firing rate when their receptive field centers were optimally stimulated with the small dark features of stationary gratings or white noise (Figures 1E and 1F, left, blue lines) and suppressed their firing rate when the polarity of the features was light (Figures 1E and 1F, left, red lines). Similarly, ON neurons increased their firing rate when their receptive field centers were optimally stimulated with the small light features of stationary gratings or white noise (Figures 1E and 1F, right, red lines) and suppressed their firing rate when the polarity of the features was dark (Figures 1E and 1F, right, blue lines). Unlike gratings and white noise, large stimuli covering both receptive field center and flanks increased the firing rate of both OFF and ON cortical neurons (Figures 1G and 1H). These results indicate that the receptive field centers are not strong enough to suppress the cortical population response from the receptive field flanks, and the receptive field flanks are not strong enough to suppress the cortical population response from the receptive field centers. Therefore, the dominant receptive field polarity measured with large stimuli is a combination of receptive field centers and flanks that not always matches the polarity of the receptive field center. Note that the response suppression measured in cats under propofol + sufentanil anesthesia is comparable or slightly stronger to the response suppression measured with grating stimuli in awake macaques (Figures S1G and S1H). Therefore, our measure of response suppression is unlikely to be underestimated when compared with awake brains.

ON and OFF cortical neurons have different spatiotemporal properties

We classified cortical receptive fields into ON or OFF based on the ON-OFF bimodal distribution of receptive field polarity measured with stationary gratings (Figure 1; see STAR Methods). The average population receptive field of OFF cortical domains (obtained by spatially aligning and averaging all OFF-dominated receptive fields) had a circular center-surround structure with an OFF center and an ON surround (Figure 2A; for comparison, see Figure S2A for example cortical receptive fields included in the average and the afferent population receptive field reproduced from Jin et al., 2011b). The average receptive field of ON cortical domains (obtained by spatially aligning and averaging all ON-dominated receptive fields) also had a circular center-surround structure with an ON center and an OFF surround (Figures 2C; see also Figure S2B). OFF receptive fields were best stimulated by turning on dark bars or turning off light bars at their receptive-field centers (Figures 2A and 2B). Conversely, ON receptive fields were best stimulated by turning on light bars or turning off dark bars at their receptive-field centers (Figures 2C and 2D). Because the average receptive field of ON and OFF cortical domains had a center-surround structure similar to the receptive field from single thalamic afferents, for simplicity, we refer to population cortical receptive fields also as ON or OFF throughout the paper.

Figure 2.

ON and OFF cortical neurons have different spatiotemporal properties

(A) Average OFF-dominated receptive field (n = 741) with onset response to the dark grating phase turned on (blue) followed by rebound response to the light grating phase turned off (red). Line plots below show the normalized receptive field strength through the center of the receptive field. Scale bar: 2°

(B) Average normalized response to dark and light stimuli (left) and light-dark difference (right) in OFF cortical domains (blue).

(C and D) Same as in (A) and (B) but for average ON-dominated receptive field (n = 578). The dotted blue line is the inverted OFF response from (B) normalized to match the baseline and maximum of the ON response.

(E) Distribution of onset responses (left) and averages (right) for ON (red) and OFF (blue) cortical sites.

(F) Same as in (A) but for the rebound response (ON: rebound to dark turned off; OFF: rebound to light turned off).

(G) Same as in (A) but for the onset (O)/rebound (R) ratios.

(H–J) Same as in (A) but for receptive field onset latency (H), signal-to-noise ratio (I), and size (J). The sharp peaks in the latency histogram correspond to the temporal bins (stimulus update) used to calculate the receptive fields.

Error bars are SEM; ns: not significant. *p < 0.05, ***p < 0.001, Wilcoxon rank-sum test in all panels. See also Figure S2.

ON and OFF cortical receptive fields differed in their temporal properties when mapped with stationary gratings, as previously demonstrated with white noise (Komban et al., 2014). ON and OFF receptive fields had onset responses with similar strength (0.25 ± 0.006 versus 0.24 ± 0.005; p = 0.127, Wilcoxon test; Figure 2E), but the OFF rebounds were weaker than the ON rebounds (0.13 ± 0.003 versus 0.15 ± 0.003; p < 0.001, Wilcoxon test; Figure 2F), making the peak/rebound ratios larger in OFF than ON cortical neurons (1.96 ± 0.02 versus 1.71 ± 0.02; p < 0.001, Wilcoxon test; Figure 2G). OFF receptive fields also had shorter response latencies than ON receptive fields (39.7 ± 0.4 versus 43.9 ± 0.34 ms; p < 0.001, Wilcoxon test; Figure 2H).

The receptive fields from ON and OFF neurons also had slightly different spatial properties. When mapped with stationary gratings, ON receptive fields had slightly higher signal-to-noise ratio (5.36 ± 0.12 versus 5.07 ± 0.10; p < 0.05, Wilcoxon test; Figure 2I) and smaller size than OFF receptive fields (2.73 ± 0.03° versus 2.84 ± 0.03°; p < 0.001, Wilcoxon test; Figure 2J), just the opposite of receptive fields mapped with sparse noise (Kremkow et al., 2014; Mazade et al., 2019). These small ON-OFF differences in signal-to-noise ratio and receptive-field size across stimuli are predicted from the saturation differences between ON and OFF contrast response functions. The greater saturation of the ON contrast response function expands the size of light more than dark stimuli, making the light grating bars wider than the dark bars (Jansen et al., 2019; Kremkow et al., 2014; Mazade et al., 2019; Onat et al., 2011; Pons et al., 2017). Because the expanded light bars recruit more spatial summation than the non-expanded dark bars, gratings finer than the receptive field center drive stronger ON than OFF cortical responses. In turn, the stronger responses from the finer gratings make ON receptive fields slightly finer and stronger than OFF receptive fields, a finding that is replicated, in both thalamus and retina, when mapping receptive fields with small stimuli (Liu et al., 2021). The expansion of large light targets also drives cortical responses from a larger visual area and activates more surround suppression than the non-expanded large dark targets (Kremkow et al., 2014). Consequently, in both thalamus and cortex, ON receptive fields are larger but weaker than OFF receptive fields when mapped with large stimuli. Cortical receptive fields also had subregions ~2.5 times larger when mapped with 2.8° square targets than fine gratings (Mazade et al., 2019), as expected from the differences in stimulus size and receptive field calculation (e.g., ON or OFF sums versus ON-OFF difference). When receptive fields were calculated as stimuli sums (ON or OFF sums), the receptive field subregions were 1.4 times larger than when calculated as differences (ON-OFF), a proportion that was similar for population thalamic receptive fields (Figure S2; see also Jin et al., 2011b).

ON and OFF cortical neurons have different response time courses

Receptive fields are calculated by averaging all stimuli driving cortical responses regardless of response strength. To investigate more precisely the stimulus preferences of ON and OFF cortical domains, we measured the average response increments driven by the 10 preferred stationary gratings (Figure 3A) and the average response suppression driven by 10 non-preferred gratings with opposite polarity to preferred gratings (Figure 3D). The onset of preferred gratings drove a response increment followed by a weak response suppression (Figure 3A), whereas the onset of non-preferred gratings drove a response suppression followed by a response increment (rebound) that was ~2 and ~4 times weaker than for preferred gratings (Figure 3D).

Figure 3.

ON and OFF cortical neurons have different response time courses

(A) Representative impulse response from an example cortical site calculated as an average from the responses to the 10 preferred gratings. The five preferred gratings that generated the strongest responses are shown below marked by numbers from strongest (1) to weakest (5).

(B) Distribution of onset responses (left) and average (right) from ON (red, n = 578) and OFF (blue, n = 741) cortical sites.

(C) Same as in (B) but for suppression to preferred stimulus.

(D) Representative impulse response from an example cortical site calculated as an average from the responses to the 10 non-preferred gratings. The five non-preferred gratings that generated the strongest suppression are shown below marked by numbers from strongest (1) to weakest (5).

(E and F) Same as in (B) and (C) but for suppression (E) and rebound response to non-preferred stimulus (F, OFF: rebound response to light off, ON: rebound response to dark off).

(G) Impulse responses illustrating the measurements of response latency and duration.

(H–L) Same as in (B) but for response latency to preferred stimulus (H), suppression latency to non-preferred stimulus (I), response duration to preferred stimulus (J), suppression duration to non-preferred stimulus (K), and signal-to-noise ratio (SNR) of response to preferred stimulus (L).

Error bars are SEM; ns: not significant, *p < 0.05, ***p < 0.001, Wilcoxon rank-sum test in all panels.

The onset of preferred gratings drove similar response increments in ON and OFF neurons (42.9 ± 1.1 versus 41.9 ± 1.0 spikes [spk]/s; p = 0.30, Wilcoxon test; Figure 3B) but slightly stronger response suppression in ON neurons (8.9 ± 0.3 versus 7.7 ± 0.2 spk/s; p = 0.022, Wilcoxon test; Figure 3C). Conversely, the onset of non-preferred gratings drove similar response suppression and rebound increments in ON than OFF neurons (suppression: 16.0 ± 0.4 versus 15.5 ± 0.4 spk/s, p = 0.16; rebound increment: 11.5 ± 0.4 versus 10.8 ± 0.3 spk/s, p = 0.17, Wilcoxon tests; Figures 3E and 3F). Therefore, we conclude that preferred and non-preferred stationary gratings drive similar responses in ON and OFF cortical neurons, but the suppression to preferred gratings is slightly stronger in ON neurons.

The largest differences between ON and OFF neurons were not in response strength but in response time course (Figures 3G–3K). Both the onset response to preferred stimuli and onset suppression to non-preferred stimuli were faster (response: 48.3 ± 0.2 versus 50.8 ± 0.2 ms, p < 0.001; suppression: 46.9 ± 0.3 versus 50.9 ± 0.3 ms, p < 0.001, Wilcoxon tests; Figures 3H and 3I) and lasted longer in OFF than ON cortical neurons (preferred stimulus: 25.0 ± 0.2 versus 23.1 ± 0.3 ms, p < 0.001; non-preferred stimulus: 29.8 ± 0.4 versus 28.3 ± 0.5 ms, measured at half amplitude, p < 0.001, Wilcoxon tests; Figures 3J and 3K) even if their response signal-to-noise ratio was similar (2.1 ± 0.02 versus 2.0 ± 0.02; p = 0.61, Wilcoxon test; Figure 3L). Because the impulse responses also have shorter latency and are less biphasic in OFF than ON thalamic afferents (Jin et al., 2011a; Komban et al., 2014), we conclude that the OFF pathway drives faster and more sustained signals than the ON pathway at different levels of the thalamo-cortical pathway.

Increasing retinal illumination makes cortical responses stronger, faster, and shorter

Increasing retinal illumination made the responses from ON and OFF neurons stronger (Figures 4A, 4B, and S3). It enhanced the responses to preferred stimuli by ~8.1 spk/s per log unit of luminance (Figure 4C), the suppression to non-preferred stimuli by ~3.2 spk/s (Figure 4D), and the suppression following the onset of preferred stimuli by ~1.1 spk/s (Figure 4E). Because the response to preferred stimuli increased more than the suppression to non-preferred stimuli, the preferred/non-preferred ratio (i.e., response/suppression ratio) increased by ~0.3 per log unit luminance (Figure 4F), enhancing the cortical discrimination of preferred stimuli.

Figure 4.

Increasing retinal illumination makes cortical responses stronger, faster, and shorter

(A) Responses from an OFF-dominated cortical site to preferred (dark, blue) and non-preferred gratings (light, red) measured at multiple stimulus luminances (rows).

(B) Same as in (A), but for an ON-dominated cortical site.

(C) Average response strength to preferred gratings for OFF- (blue, n = 110) and ON-dominated (red, n = 88) cortical sites. A linear fit (not shown) can approximate the change in response strength with luminance (R2 and p values in magenta).

(D–J) Same as in (C) but for suppression to non-preferred stimulus (D), suppression to preferred stimulus (E), ratio between response to preferred stimuli (R) and suppression (S) to non-preferred stimuli (F), response latency to preferred stimulus (G), suppression latency to non-preferred stimulus (H), response duration to preferred stimulus (I), and suppression duration to non-preferred stimulus (J).

Error bars are SEM; *p < 0.05, ** p< 0.01, ***p < 0.001, Wilcoxon rank-sum test in all panels. See also Figure S3.

Increasing retinal illumination also changed the response time course of ON and OFF neurons. It decreased the average response latency to preferred stimuli and suppression latency to non-preferred stimuli by ~5 ms per log unit (Figures 4G and 4H) and also decreased the average response duration by ~8 ms per log unit (Figures 4I and 4J). The decrease in response latency was significantly more pronounced in OFF than ON neurons (Figures 4G and 4H), while the decrease in response duration was significantly more pronounced in ON than OFF neurons (Figures 4I and 4J). Therefore, increasing retinal illumination made all cortical responses stronger, faster, and more transient (Figure S3). At the same time, it made response latencies shorter in OFF than ON neurons and more transient in ON than OFF neurons, amplifying temporal differences already present in the thalamus (Jin et al., 2011a; Komban et al., 2014). The response time course also changed with stimulus size, contrast, and duration (Figure S4), replicating similar changes previously described in the retinal inputs (Levick and Zacks, 1970; Shapley and Victor, 1978). The pronounced dependency of ON-OFF temporal differences on the stimulus properties (Figures 4 and S4) together with the diversity of response time courses in the retina (Mani and Schwartz, 2017; Ravi et al., 2018) may explain why some studies found faster response latencies in OFF than ON pathways (Jin et al., 2011a; Komban et al., 2014; Li et al., 2017; Nichols et al., 2013; Norcia et al., 2020; Rekauzke et al., 2016), others found no differences (Kaplan and Benardete, 2001), and still others found faster response latencies in ON than OFF retinal neurons (Chichilnisky and Kalmar, 2002).

ON and OFF cortical domains respond similarly to surfaces

When ON and OFF cortical neurons are classified based on the strength of their responses to light and dark surfaces (Figure 1C), by definition, ON neurons respond stronger to light surfaces than OFF neurons (Figure 1G). However, this classification is misleading because surfaces cannot be used to measure the polarity of the receptive field center from most cortical neurons (Figure S1). It is also arbitrary because the distribution of polarity measured with surfaces is unimodal (Figures 1C and S1). Surprisingly, when cortical receptive fields were classified based on the bimodal response distribution to stationary gratings (Figures 1A and S1), ON and OFF cortical populations responded very similarly to large surfaces and showed remarkably similar size suppression (Figure 5). In both ON and OFF neurons, large sustained (133-ms) surfaces suppressed cortical responses three to five times more when they were dark than light (dark versus light size suppression: 0.462 ± 0.029 versus 0.157 ± 0.027 in OFF neurons and 0.394 ± 0.045 versus 0.077 ± 0.039 in ON neurons; p < 0.001, Wilcoxon tests; Figure 5A). However, if the size suppression for light and dark stimuli was combined, the average size suppression was not significantly different between ON and OFF neurons (OFF versus ON size suppression: 0.295 ± 0.021 versus 0.217 ± 0.031; p = 0.557, Wilcoxon test; Figure 5). Midgray backgrounds increased the average size suppression for light stimuli, but the size suppression from ON and OFF neurons remained similar (Figure 5B).

Figure 5.

ON and OFF cortical domains respond similarly to surfaces

(A) Left four panels: size suppression histograms from OFF (top, n = 259) and ON cortical domains (bottom, n = 224) driven by dark (blue) and light (red) surfaces presented for different durations (16–133 ms) at maximum monitor contrast (light background for dark surfaces and dark background for light surfaces). Right panel: average size suppression for dark and light surfaces presented for 133 ms in OFF and ON domains.

(B) Same as in (A) but for OFF (n = 190) and ON cortical domains (n = 131) stimulated with dark and light surfaces presented at half monitor contrast (midgray background).

(C and D) Same as in (A) and (B) but for temporal suppression from dark and light surface stimuli presented with different sizes (4°–23°).

Error bars are SEM; p values obtained with Wilcoxon rank-sum test in all panels. See also Figures S4 and S5.

Cortical responses were suppressed not only by the stimulus size but also by the stimulus duration. For each stimulus size, increasing the stimulus duration made cortical responses weaker, but the suppression was again much stronger for dark than light stimuli. The dark-light average differences increased with stimulus size (Figures 5C and 5D) and, at the largest size (23°), dark surfaces caused strong temporal suppression, whereas light surfaces caused weak temporal facilitation (dark versus light temporal suppression: 0.383 ± 0.026 versus −0.119 ± 0.023 in OFF neurons and 0.413 ± 0.024 versus −0.045 ± 0.024 in ON neurons; p < 0.001; Figure 5C). Once again, when the suppression for light and dark stimuli was combined, the temporal suppression of ON and OFF neurons was not significantly different (OFF versus ON temporal suppression: 0.130 ± 0.021 versus 0.182 ± 0.021; p = 0.075, Wilcoxon test). Midgray backgrounds also made the dark-light differences more pronounced but did not increase the ON-OFF differences (Figure 5D).

These results indicate that the strength of cortical responses to surfaces depends on stimulus polarity (light or dark) more than cortical receptive field polarity (ON or OFF). As stimulus duration, size, and luminance range (i.e., maximum minus minimum luminance) increase, cortical responses become stronger to light than dark stimuli. Conversely, as surface duration, size, and luminance range decrease, cortical responses become stronger to dark than light stimuli (see also Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021). Importantly, this shift in stimulus polarity preference is not due to a change in the ratio of ON and OFF cortical neurons responding to the stimulus but to a change in the response balance between ON and OFF responses within each cortical receptive field (see also Jansen et al., 2019). Brief small stimuli (i.e., slightly larger than one receptive field center) drive more effectively OFF receptive-field subregions in visual cortex. Conversely, long-lasting large surfaces (covering both receptive field center and flanks) drive more effectively ON receptive-field subregions (Figures 5 and S5; see also Jansen et al., 2019). If the surface properties do not change (e.g., same polarity, duration, and size), the ON-OFF response balance remains relatively constant and the population responses from ON centers + OFF flanks are roughly equal to the population responses from OFF centers + ON flanks. The OFF pathway can dominate cortical responses, as previously demonstrated in humans, non-human primates, carnivores, and rodents (Jansen et al., 2019; Jimenez et al., 2018; Jin et al., 2008; Kremkow et al., 2014; Norcia et al., 2020; Taylor et al., 2018; Wang et al., 2015; Xing et al., 2014; Yeh et al., 2009; Zemon et al., 1988; Zurawel et al., 2014)). However, this OFF cortical dominance shifts to ON dominance when the stimulus duration and size increase (Figure S5).

Cortical responses to surfaces are strongly dependent on background luminance

The strength of cortical responses to bright surfaces was modulated by background luminance. Changing the background from black to midgray increased the size suppression to light surfaces by 5 times in ON neurons and 2.7 times in OFF neurons (cf. Figures 5A and 5B), but barely affected the size suppression to dark surfaces, which only becomes weak at very low light (Mazade et al., 2019). Therefore, consistent with previous studies, midgray backgrounds made cortical responses weaker to light than dark stimuli (Kremkow et al., 2014; Pons et al., 2017; Xing et al., 2014; Yeh et al., 2009; Zurawel et al., 2014) through a reduction in luminance range (Rahimi-Nasrabadi et al., 2021). Conversely, increasing surface duration, size, and luminance range made response transients much stronger to light than dark surfaces (Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021).

Neuronal mechanisms underlying ON and OFF cortical responses to surfaces

The results above indicate that small stimuli drive opposite responses from ON and OFF cortical neurons (increment or suppression), but large surfaces drive nearly identical response increments (Figures 6A and 6B). The striking similarity in the average population response to surfaces from ON and OFF cortical neurons (Figure 6B) can only be demonstrated if their receptive fields are classified based on responses to stationary gratings (Figure 1A). If they are classified based on responses to surfaces (Figure 1C), by definition, ON neurons respond stronger to light surfaces and OFF neurons to dark surfaces. Our results demonstrate that increases in both stimulus duration and size make the average ON + OFF cortical response stronger to light than dark surfaces (Figures 6B, 6C, and S4–S6). The effect of stimulus duration can be simulated with a model that assumes a difference in temporal integration between the ON and OFF thalamic inputs diverging and converging into ON and OFF cortical neurons (Figures 6D–6F). By making the temporal integration and recovery longer in ON than OFF inputs, the model can simulate the shift from OFF to ON cortical dominance with the increase in stimulus duration (see STAR Methods for details on the simulation parameters). Moreover, by adding a gain factor for retinal illuminance, the model can also simulate the changes in cortical response strength with light level. The model can also simulate ON-OFF cortical differences in size tuning by adding a suppressive ON surround to both ON and OFF cortical receptive fields that is luminance dependent (Figure 6G). If the background luminance is zero (or much lower than the stimulus luminance), the surround suppression is weak and the cortex responds strongly to light surfaces. However, as the background luminance and retinal illumination increase, the surround suppression becomes stronger and reduces the amplitude of cortical responses (Figures 6H and 6I). This simple model accurately reproduces our previous measurements of cortical responses to stimuli with different size, duration, luminance, background luminance, and retinal illumination (Figures 6E–6I, measurements are reproduced from Mazade et al., 2019). The model is also consistent with recent measurements in awake macaques demonstrating an important role of cortical surround suppression in encoding surface luminance (Yang et al., 2022).

Figure 6.

Modeling the stimulus preferences from ON and OFF cortical neurons

(A) Average receptive fields from OFF (top, blue, n = 238) and ON (bottom, red, n = 207) cortical neurons. Scale bar: 2°.

(B) Average impulse responses from the same OFF (top) and ON neurons (bottom) measured with dark (blue) and light (red) gratings, sparse noise (2.8°/33 ms), small fast surfaces (7°/33 ms), and large slow surfaces (23°/133 ms).

(C) Scatterplot showing the light/dark response ratio from ON and OFF neurons for large slow and small fast surfaces (n = 478).

(D) Cartoon illustrating simulated temporal differences between ON and OFF pathways (ON pathways can sustain longer responses but take longer to recover).

(E) Average stimulus duration tuning in response to dark (blue, n = 239) and light (red, n = 338) surfaces (average of all surface sizes) measured on maximum contrast backgrounds under scotopic, mesopic, and photopic illumination and fit with an ON-OFF model (see STAR Methods).

(F) Same as in (E) but for stimulus duration tuning in response to dark (n = 187) and light surfaces (n = 185) measured on midgray backgrounds.

(G) Cartoon illustrating the difference between ON and OFF spatial integration for grating stimuli and the similarity in ON and OFF spatial integration for surfaces. Notice that receptive field centers measured with sparse noise are slightly larger for light than dark stimuli due to neuronal blur (Kremkow et al., 2014).

(H and I) Same as in (E) and (F) but for average stimulus size tuning (average of all surface durations) fit with an ON-OFF model measured with maximum contrast (H, dark: n = 309, light: n = 330) and midgray backgrounds (I, dark: n = 186, light: n = 185). Luminance data are adapted from Mazade et al. (2019).

See also Figure S6.

Surface brightness in nature and human vision

The response ratio between ON and OFF visual pathways provides a reliable signal of stimulus brightness when the stimulus is small (i.e., making small stimuli brighter increases the strength of ON cortical responses and the ON/OFF ratio). However, unlike small stimuli, bright surfaces drive similar response increments from ON and OFF cortical neurons through a combined stimulation of both receptive field centers and flanks (e.g., ON center from ON neurons and ON flanks from OFF neurons). Therefore, making a surface brighter does not increase the ON/OFF cortical response ratio. It increases the ON + OFF response average (Figures 7A–7C and S6), which is driven by a large number of neurons with receptive fields overlapping the large surface area. This ON + OFF response average is a reliable signal of surface brightness because it increases with the physical properties of bright surfaces in nature, such as luminance range (Rahimi-Nasrabadi et al., 2021), size, and duration (Figures 7B, 7C, and S6).

Figure 7.

Cortical processing of visual brightness

For a Figure360 author presentation of Figure 7, see https://doi.org/10.1016/j.celrep.2022.111438.

(A) Average cortical response from ON and OFF neurons to light (red, n = 304) or dark stimuli (blue, n = 325) of 2.8° and 33 ms measured under different luminance ranges (maximum minus minimum luminance in the display).

(B) Same as in (A) but for large surfaces of 23° and 133-ms duration (light: n = 306, dark: n = 218).

(C) Color plots showing the average cortical response to the onset of light (n = 304) and dark stimuli (n = 297) with different sizes (y axis) and luminance ranges (x axis) presented for 33 (left) and 133 ms (right). Blue: stronger responses to darks than lights. Red: stronger responses to lights than darks.

(D) Example natural images with high (top) and low brightness (bottom) passed through separate ON and OFF luminance/response functions (LRFs) and split into an image of darks (darkest pixels, blue) and an image of lights (brightest pixels, red).

(E) Left: scatterplot illustrating the size of the largest continuous dark and light regions of 1,314 images plotted against the mean brightness. Right: the same data binned into 10% mean luminance increments (open circles) and fit with NakaRushton functions (solid lines) (see STAR Methods).

(F) Individual luminance adjustment curves from a checkerboard brightness matching task performed by six subjects. Positive values indicate an increase in adjusted luminance to make a stimulus perceived as dim to appear as bright as the reference stimulus. Negative values indicate a reduction in adjusted luminance. Pink shading illustrates square sizes stimulating foveal receptive fields (centers + flanks). Blue shading illustrates square sizes stimulating the suppressive surrounds.

(G) Average luminance adjustment across all six subjects as a function of check size.

Error bars are SEM in all panels, ***p < 0.001, all p values calculated with Wilcoxon rank-sum test. See also Figure S7.

The combined effect of stimulus size and luminance in cortical responses is reminiscent of size-luminance interactions used in visual art by painters and photographers for many decades. Large dark surfaces are frequently used to illustrate image darkness and large light surfaces to illustrate image brightness. Therefore, we predicted that similar size-luminance interactions may already exist in nature and be used by both cortical neurons and artists to signal visual brightness. A simple analysis of visual scenes (Olmos and Kingdom, 2004) confirms this prediction. The analysis demonstrates that, as scenes become brighter, light surfaces become larger and dark surfaces smaller (Figures 7D, 7E, and S7). Such size-luminance interaction is most pronounced when images are compressed for luminance with gamma functions (a standard procedure in digital cameras and monitors) and for contrast with cortical response functions (Figure S7; see also Kremkow et al., 2014; Mazade et al., 2019; Pons et al., 2017; Rahimi-Nasrabadi et al., 2021). Consistently, without compression for luminance and contrast, natural images from outdoor scenes appear extremely dark, unnatural, and very different from the images illustrated in artistic paintings, photography, or the images that we perceive (Rahimi-Nasrabadi et al., 2021).

The image size-brightness interactions in nature should help increase the efficacy of cortical responses at signaling scene brightness. As images become brighter, the increase in luminance range should strengthen cortical responses to all stimuli (Figures 4C, 7A, and 7B). Moreover, the enlargement of light surfaces should make cortical responses stronger because bright surfaces drive stronger cortical responses than dark surfaces (Figures 7B and 7C). Also, the enlargement of light surfaces in nature should further strengthen cortical responses through an increase in stimulus duration (Figure 7C) because eye movements are more likely to keep a cortical receptive field inside a large than a small surface. Size-brightness interactions can be demonstrated in different image databases (van Hateren and van der Schaaf, 1998) (Figure S7) and, as for contrast response functions, they saturate for light more than dark stimuli (Kremkow et al., 2014; Mazade et al., 2019; Pons et al., 2017; Rahimi-Nasrabadi et al., 2021).

Another important prediction from our results is that the perception of stimulus brightness should change with stimulus size. As the stimulus size increases and the ON + OFF population response becomes stronger, the stimulus should appear brighter. Moreover, large stimuli should appear dimmer if they effectively stimulate the suppressive surround. To test these predictions, we asked human subjects to compare two checkerboard patterns of light squares with different sizes. The subjects had to adjust the luminance of light squares from a test checkerboard to match the brightness of the light squares from a reference checkerboard. The squares of the reference checkerboard were always large (2.5°), whereas the squares from the test checkerboard varied in size from 0.06° (7.8 cycles/degree) to 2.5° (0.2 cycles/degree; see STAR Methods for more details). As predicted from our results, the perceived stimulus brightness was strongly dependent on stimulus size in each individual subject (Figure 7F) and in the subject average (Figure 7G). As the squares became larger, they appeared brighter and the subjects had to decrease the luminance needed to match the large squares of the reference checkerboard. In some subjects, the brightest stimulus was the largest one (e.g., subjects 1–3), as it would be expected from weak surround suppression. In other subjects (subjects 4–6) and in the subject average, the brightest stimulus was 0.4°, which roughly matches the average population receptive field center at the human fovea when jittered by small eye movements during fixation.

DISCUSSION

We have demonstrated that ON and OFF cortical neurons generate similar response increments to large surfaces, and their average response increases with luminance range, surface brightness, size, and duration. Because surface size and duration also increase with light intensity in nature, we conclude that the average response from ON and OFF cortical neurons provides a reliable neuronal signal of scene brightness.

Cortical responses to surfaces are strong and reliable

A common belief in visual neuroscience is that cortical neurons respond poorly to surfaces, and surface perception emerges from the spread of cortical responses to the surface borders (von der Heydt et al., 2003). However, there is increasingly stronger evidence that cortical neurons respond reliably to surfaces (Dai and Wang, 2012; Haynes et al., 2004; Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021; Yang et al., 2022; Zurawel et al., 2014; Zweig et al., 2015). Moreover, while surfaces drive weaker responses than small stimuli from single neurons, the population response to surfaces can be stronger because large stimuli cover a larger number of cortical receptive fields and drive more neurons than small stimuli.

Our results demonstrate that the strength of cortical responses to surfaces increases with luminance range (Figure 7B), and the increase is more pronounced for light than dark surfaces. In cat visual cortex, light surfaces generate stronger responses than dark surfaces when the surface duration is longer than 80 ms (Figure 6B; see also Mazade et al., 2019). Similarly, in macaque visual cortex, light surfaces lasting 500 ms generate stronger response transients than dark surfaces of the same duration (Xing et al., 2014), although the sustained responses can be stronger to dark surfaces (Xing et al., 2014) because the response suppression/rebound is stronger in ON than OFF neurons (Figure 3C; see also Jin et al., 2011a; Komban et al., 2014; Liu et al., 2021; Zurawel et al., 2014).

Cortical responses to surfaces are modulated by multiple stimulus parameters

Our results demonstrate that the strength of cortical responses to surfaces depends on multiple stimulus parameters that include background luminance, luminance range, surface brightness, duration, and size. Increasing the background luminance makes cortical responses stronger to dark than light stimuli, as demonstrated in many studies using midgray backgrounds (Kremkow et al., 2014; Mazade et al., 2019; Pons et al., 2017; Rahimi-Nasrabadi et al., 2021; Xing et al., 2010; Yeh et al., 2009; Zurawel et al., 2014). Conversely, increasing stimulus luminance range, size, and duration makes cortical responses stronger to light than dark stimuli (Figure 7).

The cortical responses to light and dark surfaces can be accurately simulated with a model that includes an ON suppressive surround and ON-OFF differences in temporal integration. As the background luminance increases, the ON suppressive surround reduces the strength of simulated cortical responses to surfaces through mechanisms of light adaptation and lateral inhibition that are likely to originate at different levels of the visual pathway, from retina (Bloomfield and Volgyi, 2009; Shapley and Enroth-Cugell, 1984) to visual cortex (Yang et al., 2022). Moreover, as the surface duration increases, the longer temporal integration within ON pathways makes the simulated responses stronger to light than dark surfaces through mechanisms that could be mediated by retinal differences in metabotropic/inotropic receptor dynamics and/or synaptic pool. At the photoreceptor-bipolar synapse, OFF inotropic receptors are faster than ON metabotropic receptors, and electrotonic propagation is also faster in OFF than ON bipolar cells because OFF bipolar cells are shorter. Moreover, because the photoreceptor-bipolar synapse is closer to the synaptic ribbon in ON than OFF neurons, ON neurons may be able to use a larger synaptic pool to sustain longer responses through slow metabotropic receptors that also take longer to recover (Figure 6D). Therefore, differences already present at the photoreceptor-bipolar synapse could explain the ON-OFF temporal differences that we report and simulate.

The visual cortex increases neuronal selectivity to stimulus light-dark polarity

In his first description of ON and OFF pathways, Hartline (1938) used light stimuli to demonstrate that ON and OFF retinal ganglion cells respond to light turned on and off, respectively. Later studies using light and dark stimuli demonstrated that cortical responses are very different to stimuli turned on (onset responses) than off (rebound responses). Regardless of light-dark polarity, the responses to stimuli turned on are stronger, faster, more temporally precise, and have higher stimulus specificity than rebound responses to stimuli turned off (Hirsch et al., 1998; Komban et al., 2014; Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021). Rebound responses are also weaker in cortex than thalamus (Alonso et al., 2001; Jin et al., 2011a; Komban et al., 2014), making cortical responses more reliable than their thalamic inputs at signaling light-dark polarity (e.g., selectivity for light polarity is maximized when an ON neuron responds to a light stimulus turned on, but not to a dark stimulus turned off). Cortical responses are also more finely tuned to contrast, size, and duration in onset than rebound responses (Mazade et al., 2019; Rahimi-Nasrabadi et al., 2021).

The terminology light “increments” and “decrements” remains widely spread in the scientific literature to describe stimuli in psychophysics, physiology, and the eye clinic. However, the terms assume that turning on a light stimulus and turning off a dark stimulus drive identical cortical responses (both are light “increments”), which is not correct. Turning on a light stimulus generates a strong onset response that is perceived as the onset of a light stimulus. Conversely, turning off a dark stimulus generates a weaker rebound response that is perceived as a dark stimulus turned off. Only when a surface covers the entire visual field is a light increment always perceived as a light surface and a light decrement as a dark surface. As for stimulus orientation (Alonso et al., 2001; Chapman et al., 1991; Jin et al., 2011b; Lien and Scanziani, 2013; Reid and Alonso, 1995; Sedigh-Sarvestani et al., 2017), thalamocortical convergence may contribute to increasing the selectivity of cortical neurons to light-dark stimulus polarity by making onset responses stronger than rebound responses. Because the OFF thalamic pathway generates weaker response rebounds than the ON pathway (Figure 2; see also Jin et al., 2011a; Komban et al., 2014), the OFF-dominated thalamocortical convergence (Jin et al., 2011b, 2008) should make response rebounds weaker in cortex than thalamus, increasing even further the selectivity for light-dark polarity (Alonso et al., 2001; Jin et al., 2011a; Komban et al., 2014).

Cortical processing of visual brightness

In summary, our results demonstrate that cortical responses to light and dark surfaces are different in strength, and the light-dark difference increases with background luminance, luminance range, surface size, and duration. These results are consistent with classical studies in human perception demonstrating that image brightness is not determined by image luminance only but also by the size of light and dark image regions (Gilchrist et al., 1999; Li and Gilchrist, 1999). Whereas other cortical signals may contribute to the perception of stimulus visual brightness (Yang et al., 2022), the average ON + OFF cortical response provides a strong and reliable signal that also explains size-luminance interactions used by artists over many centuries to illustrate brightness.

Limitations of the study

Our multielectrode recordings of cortical activity were performed under propofol-sufentanil anesthesia. It is well known that some types of general anesthesia strongly affect the amplitude of neuronal responses in primary visual cortex. However, as we show in the paper, the cortical responses and response suppression measured under propofol-sufentanil are similar to those measured in awake behaving brains.

STAR★METHODS

RESOURCE AVAILABILITY

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Jose Manuel Alonso (jalonso@sunyopt.edu).

Materials availability

This study did not generate new unique reagents.

Original data have been deposited at the Open Science Framework Repository and are publicly available as of the date of publication. The DOI is listed in the key resources table.

KEY RESOURCES TABLE

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Experimental models: Organisms/strains
Adult cats, Felis catus (4–7 kg)	Liberty Research, Inc., Waverly, New York, USA	N/A
Software and algorithms
MATLAB	MathWorks	R2016–2022
Psychtoolbox-3	(Brainard, 1997)	v3.0.12
Custom Code	Open Science Framework	https://doi.org/10.17605/OSF.IO/H87DT
Other
Custom 32-channel multielectrode	NeuroNexus	N/A
OmniPlex Neural Recording Data Acquisition	Plexon	N/A

All original code has been deposited to the Open Science Framework Repository and is publicly available as of the date of publication. The DOI is listed in the key resources table.

Any additional information required to reanalyse the data reported in this paper is available from the lead contact upon request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Animal models

All experiments and procedures were performed in accordance with the guidelines of the United States Department of Agriculture (USDA) and approved by the Institutional Animal Care and Use Committee (IACUC) at the State University of New York, State College of Optometry. Adult male cats (Felis catus, 4–7 kg, n = 15) and adult male rhesus macaques (Macaca mulatta, n = 2) were housed in groups and provided enrichment. There were no assigned experimental groups. Some of the analysis and illustrations reported in this paper use data collected in previous studies. Specifically, the data used to illustrate the population receptive fields from thalamic afferents of Figure S2 was also used in Jin et al. (2011b), the analysis of ON-OFF dominance measured with white noise uses data from Wang et al. (2015), the comparison of response suppression between cat and macaques uses data from Jansen et al. (2019), the analysis of ON and OFF responses to surfaces and the fit of the computational model describing how cortical responses change with stimulus luminance, size, duration, background and retinal illumination uses data from Mazade et al. (2019).

Human subjects

We recruited six human subjects to measure the perception of visual brightness, one female and 5 males ranging in age from 27 to 57 years old. All experiments in human subjects were approved by the Institutional Review Board (IRB) at the State University of New York College of Optometry and followed the principles outlined in the Declaration of Helsinki. Informed consent was obtained from all subjects before the experiments.

METHOD DETAILS

Visual stimuli and electrophysiological recordings from anesthetized visual cortex

Adult male cats were tranquilized with an intramuscular injection of acepromazine (0.2 mg kg−1) and anesthetized with an intramuscular injection of ketamine (10 mg kg−1). One intravenous catheter was inserted into each hind limb to administer continuous infusions of propofol (5–6 mg kg−1 h−1), sufentanil (10–20 ng kg−1 h−1), vecuronium bromide (0.2 mg kg−1 h−1), and saline (1–3 mL h−1). Vital signs, including heart rate, blood pressure, electrocardiogram (EKG), temperature, expired CO2, pulse oximetry, and electroencephalogram (EEG), were carefully monitored throughout the surgery and recordings and maintained within normal physiological limits. Once the animal was under continuous anesthesia, the head was carefully secured and a small craniotomy was made over the primary visual cortex. Further details of surgical procedures have been described previously (Jin et al., 2008; Kremkow et al., 2016).

Two 32-channel linear multielectrode arrays (0.1 mm inter-electrode distance, Neuronexus) were horizontally introduced in the primary visual cortex (<5 deg angle relative to the horizontal plane and centered in layer 4) to measure cortical multiunit activity. The recordings were filtered between 250 Hz and 8 kHz, sampled at 40 kHz and collected by a computer running Omniplex (Plexon). Cortical multiunit activity was thresholded at a voltage level of approximately 60 microvolts to record from a restricted population of neurons with each electrode. The spread of this thresholded cortical multiunit is estimated to be ~50 microns (Mazade et al., 2019). Throughout the paper, we use the terms ON or OFF cortical neurons to describe small groups of neurons (multiunit) with a population receptive field that is either ON or OFF dominated. We use the terms cortical site or domain to refer to each group of cortical neurons recorded as multiunit activity with one electrode of the multielectrode probe.

Stimuli were generated with custom MATLAB code (Mathworks) with Psychtoolbox extensions and presented in a gamma corrected 24-inch LCD monitor (BenQ XL2420-B, 120 Hz, mean luminance: 120 cd/m2). In some experiments (n = 4 cats) we used a gamma corrected 22.5-inch VIEWPixx monitor (VPixx Technologies, 120 Hz, mean luminance: 50 cd/m2). Data from both monitors are shown combined in individual analyses for local feature stimuli only (e.g. gratings and white noise). However, only data obtained with the LCD monitor was used in data analysis for sparse noise and large surfaces. A surface is defined in the paper as a homogeneous target with a size larger than the average receptive field center plus flanks. Gamma correction was performed using luminance measures based on the standard V(λ) function. The monitor was 0.57 meters from the animal.

We first mapped the receptive field of each cortical site with a fast sequence of 576 static gratings (88 orientations, 41 spatial frequencies, 4 phases) randomly presented at 60 Hz in multiple repetitions (Ringach et al., 1997). The gratings had a size of 24 degrees and were presented on a midgray background of 120 cd/m2. The receptive fields were also mapped with white noise and sparse noise. The white noise was made of a sequence of 32,767 checkerboards presented at 60 Hz (16 × 16 pixels, 1.6 degrees/pixel). The sparse noise was made of large light and dark squares presented on a light, dark, or midgray background at 30 Hz (20 × 20 target positions, separation between positions: 1.4 degrees, target width: 2.8 degrees). The receptive fields mapped with sparse noise were used to find the center of mass of all receptive fields within each recording probe and center subsequent stimuli. The summed population receptive field across the 32 electrodes of a multielectrode probe had an average diameter of 10.65 degrees, which includes the average size of the population receptive field and the retinotopy changes across 3.1 mm of cortex. The average size of the population receptive field measured with sparse noise was 7.1 degrees.

We measured the size and temporal tuning of each cortical recording site with dark and light square targets (30–90 repeats) presented with different sizes (1, 4, 7, 11, 14, 17, 20 and 23 deg/side), durations (16, 33, 83 and 133 ms), and background luminances (light, dark or midgray). Each stimulus block contained a sequence of different target sizes with the same polarity, duration and background luminance. Each target was separated from the next by an interval lasting 150 ms that spanned from the end of one target to the beginning of the next. We measured cortical responses under different retinal illuminances by placing neutral density filters in front of the eye. The neutral density filters reduced the maximum luminance of the stimulus from 239 cd/m2 to 0.0239 when using the BenQ LCD monitor, and from 100 cd/m2 to 0.01 when using the VIEWPixx monitor.

Visual stimuli and electrophysiological recordings from awake visual cortex

Two macaques were implanted with a post for head fixation, a scleral eye coil for eye tracking, and a chronic array of ultrathin microelectrodes to record from single neurons in primary visual cortex (for more details, see Jansen et al., 2019; Lashgari et al., 2012). After recovering from surgery, the macaques were trained to fixate their eyes on a small 0.12° cross while presenting a fast sequence of grating stimuli centered on the neuronal receptive field. The grating sequence was presented at 80 Hz and contained 576 gratings of 19.2 degrees/side that could have 88 different orientations, 4 different phases, and 41 different spatial frequencies ranging from 0.09 to 1.07 cycles per degree.

Human psychophysics

Human subjects seated in front of a gamma corrected 22.5-inch VIEWPixx monitor (VPixx Technologies, 120 Hz, maximum luminance: 89 cd/m2, viewing distance: 1 meter) and used a chinrest to minimize the head movements. The subjects were asked to compare the brightness of two checkerboard patterns of light squares presented on the left and right side of the monitor on a black background. The two checkerboards covered the same area of visual space (7.5 × 7.5 degrees), but had different square sizes. The squares of the checkerboard used as reference were always 2.5 degrees in size and 44 cd/m2 of luminance. The squares of the checkerboard used as test varied in size from 0.06 degrees (7.8 cycles per degree) to 2.5 degrees (0.2 cycles per degree) and their initial luminance was randomly chosen from two different luminance ranges, 0–24 cd/m2 or 65–89 cd/m2 luminance (half trials for each range). The location of the test checkerboard was signaled by a green dot at the beginning of each trial and was randomized across trials (left or right of reference checkerboard). All subjects performed 40 trials per square size (40 × 5 sizes = 200 trials in total). In each trial, the subjects were free to move the eyes and inspect the two checkerboards for as long as needed while using a keypad with three buttons to adjust the luminance. One button was used to increase the luminance of the light squares, another to decrease the luminance, and another to signal that the luminance adjustment was completed and start the next trial. Each button press changed the luminance of the squares from the test checkerboard by 0.35 cd/m2.

QUANTIFICATION AND STATISTICAL ANALYSIS

Receptive field analysis

Cortical receptive fields were classified into ON and OFF based on their dominant contrast polarity. The receptive fields were calculated at different times between stimulus and response with a temporal resolution of 16.6 ms when using gratings and 33 ms when using sparse noise. The dominant contrast polarity was measured at the time interval that had the receptive field with the largest absolute value. Only receptive fields with high signal-to-noise were included in the analysis. The signal-to-noise threshold was higher for white noise (10) than gratings (2.5) and sparse noise (1.2) because the responses to white noise were weaker and strongly dominated by the frequency of the stimulus update. When stimulated with surfaces, the dominant contrast polarity was calculated directly from responses to light surfaces (ON responses) or dark surfaces (OFF responses) with 7 deg size and 33 ms duration. The dominant contrast polarity was calculated as the difference between the maximum responses to the onset of light (ON) and dark stimuli (OFF) divided by the sum. The dominant contrast polarity across all recorded neurons was also calculated for each stimulus size and duration.

The receptive field properties measured with gratings were calculated as follows. The normalized onset and rebound responses were measured as the maximum absolute value of the first phase and second phase of the receptive field across time. The onset latency was calculated as the time at the receptive-field onset with a temporal resolution equal to the stimulus temporal update (16.6 ms). The onset/rebound ratio was calculated as the ratio between the onset and rebound of the receptive field and the calculation included only cortical sites with both onset and rebound values that passed our criterion of signal-to-noise ratio (SNR, larger than 2.5). The SNR was calculated as the receptive field onset divided by the baseline. The baseline was measured as the maximum value of the receptive field at a temporal frame 16 ms before the receptive field onset. All receptive fields were baseline subtracted, normalized, and thresholded at 20% of the maximum response. Receptive field size was calculated as the receptive field diameter (sqrt (2 * pixel count/pi)) converted to degrees. In response measurements under different retinal illumination, both the SNR for selection as well as the contrast polarity of the receptive field were measured at the highest luminance (with no neutral-density filter). All receptive fields larger than 30 deg were disregarded as noise. This exclusion criterion was rarely applied but was important to remove receptive fields measured at low retinal illumination with very low spike counts and baselines close to zero. Receptive field averages displayed in figures were not thresholded since the background noise was very low due to the large number of receptive fields included in the average. All receptive fields shown in the paper were centered at the pixel that generated the maximum response (MATLAB function ‘circshift’). This centering process was done separately for receptive fields measured with different retinal illuminations.

Analysis of visual responses

Visual responses to gratings were quantified for each cortical recording site with peri-stimulus time histograms (PSTHs) calculated with 1 ms bins and smoothed with a temporal window of 20 ms (MATLAB function ‘smooth’). PSTHs for each of the 576 gratings were made around the stimulus onset with a time window of −50 to 400 ms and the 10 gratings that drove the strongest responses were selected (Williams and Shapley, 2007). The PSTHs for the 10 preferred gratings were averaged to measure the response to the preferred stimulus. The PSTHs for the 10 gratings with opposite phase to these 10 preferred gratings were also averaged to measure the response to the non-preferred stimulus. All averaged PSTHs were baseline subtracted to limit variations in multiunit spontaneous spiking across stimulation conditions, recording sites, and anesthesia levels. The response to the preferred stimulus was calculated as the maximum response between 20 and 70 ms following the stimulus onset. The response suppression to the preferred stimulus was measured as the minimum response between 70 and 120 ms following the stimulus onset. The response suppression to the non-preferred stimulus was measured as the minimum response between 20 and 70 ms following the stimulus onset. The rebound response to the non-preferred stimulus was measured as the maximum response between 70 and 120 ms following the stimulus onset. The response and suppression latencies were measured as the time of the onset response and suppression, respectively. The response and suppression duration were measured as the duration of the PSTH at 50% of the maximum response and suppression. The SNR of the PSTH was calculated as the maximum response to the preferred stimulus divided by the baseline. The baseline was defined as the average spike rate between −40 and 0 ms before stimulus onset. A cortical recording site was selected for the grating PSTH analysis if the SNR of the response was larger than 2.5. In some analyses, the SNR threshold to select significant responses was systematically varied to investigate the contribution of the strongest cortical population responses. For experiments testing the effects of mean illumination, the time window for calculating the PSTH response parameters was shifted by ~7 ms per log unit of luminance reduction to compensate for the increase in response latency at low light. Visual responses to white noise were quantified with PSTHs generated using similar procedures as for gratings. The PSTHs were calculated with 1 ms bins separately for the dark and light pixels located at the receptive field center, smoothed with a temporal window of 20 ms, and baseline subtracted. A PSTH was only included in the analysis of responses to white noise if the SNR was greater than 20 (to avoid PSTHs dominated by the frequency of the stimulus update).

Visual responses to surface and sparse noise stimuli were quantified from PSTHs with 1 ms bins, smoothed with a temporal window of 15 ms and baseline subtracted. Surface and sparse noise PSTHs required less smoothing than gratings and white noise because the responses were stronger and did not oscillate as much with the lower stimulus update (30 Hz for sparse noise and 3.5, 4.3, 5.5, and 6 Hz for surfaces). The response strength to surfaces and sparse noise was measured within a temporal window aligned with the response onset (15 ms average around the maximum value). Response latency was calculated as the time-to-peak of the PSTH with stimulus onset. The SNR was calculated as the ratio between the maximum response onset and the baseline where the baseline was defined as the average mean firing rate preceding the response onset. To minimize noisy responses at low retinal illuminations, only response amplitudes larger than 5 spk/s were included in the analysis.

To measure size tuning, a cortical recording site was selected for analysis if the response SNR to both a dark and light 7 deg target (the average receptive field size measured with sparse noise) was higher than 5. This selection process was done independently for different stimulus durations (16, 33, 83 and 133 ms) and different backgrounds (light/dark or midgray). In the analysis of temporal tuning, we selected a PSTH series (PSTHs for all stimulus durations with same stimulus size from each recording site) if the response onset to both a dark and light 16 ms target had a SNR also higher than 5. Any response with a negative value after baseline subtraction was set to zero to calculate the average size tuning and temporal tuning. Because PSTHs with zero baseline have infinite SNR, we added a small arbitrary baseline offset to all PSTHs (4 spks/s) to minimize the selection of noisy responses. To compare directly responses measured with gratings (1.25 threshold), sparse noise (1.2 threshold), and surfaces (5 threshold), cortical neurons from one recording electrode had to pass all SNR thresholds to be accepted.

Surface size suppression was measured from the size tuning as the ratio (Rp - Rt)/Rp, where Rp is the response at the peak of the tuning and Rt the response at the tail end of the tuning (Figure 5). Rp was measured as the maximum response generated by surface sizes between 4 and 7 degrees and Rt as the minimum response generated by surface sizes between 17 and 23 degrees. The temporal suppression was measured from the temporal tuning with a similar (Rp - Rt)/Rp ratio. Rp was measured as the maximum response to stimulus durations between 16 and 33 ms and Rt as the response generated by a stimulus duration of 133 ms. In both size and temporal measurements of suppression, a positive value indicates suppression and a negative value indicates enhancement. Suppression ratios were calculated independently for darks and lights and for cortical sites that were ON or OFF dominated. For luminance measurements with neutral density filters, an entire PSTH series was selected (all measurements with neutral-density filters for the same stimulus size or duration) if the PSTH measured at the highest luminance (with no neutral-density filter) passed the SNR threshold. This selection process was done independently for stimuli with different polarity (dark or light), and different backgrounds (light/dark or midgray). To compare between responses to dark and light surfaces of different sizes and stimulus durations across max stimulus luminance, PSTH responses were binned into 5 log-unit luminance ranges (0.007, 0.07, 0.7, 7 and 70 cd/m2). To adjust the parameters of the computational model, the spatiotemporal tuning responses were binned into 3 log-unit luminance ranges corresponding to scotopic, mesopic and photopic conditions, as in previous work (Mazade et al., 2019).

ONOFF model of cortical responses to light and dark stimuli

We simulated the cortical responses to light and dark stimuli with a model that has four different terms: spatial summation, temporal summation, contrast summation and retinal illumination. The model takes as input stimuli with different light-dark polarity, size, duration, contrast, background luminance and retinal illumination. Then, it calculates separately the response to light (RL) and dark stimuli (RD) with several Naka-Rushton functions that simulate spatial summation, temporal summation and contrast summation. As shown in Equation 1, the parameters of the model are the stimulus size (S), stimulus temporal duration (T), size of receptive field center (RFC = 2), size of suppressive surround (SS = 10), receptive-field peak temporal summation (RFT = 20 ms), the exponents for spatial and temporal summation (a = 3, b = 2 for size tuning and 0.3–3 for light-dark temporal tuning), the background for light (BL) and dark stimuli (BD), the retinal illumination (I), and the contrast for light (CL) and dark (CD) stimuli. The background for light (BL) and dark stimuli (BD) changes with background luminance (BGLum) and maximum scene luminance (MaxLum), as illustrated in Equation 2. The retinal illumination (I) changes with the mean scene luminance (MeanLum), as illustrated in Equation 3. The contrasts for light (CL) and dark (CD) stimuli are calculated as Naka-Rushton functions with the parameters adjusted for retinal illumination (Rahimi-Nasrabadi et al., 2021), as illustrated in Equation 4.

Analysis of light and dark region size and luminance in visual scenes

To investigate the relation between luminance and size in the visual scene, we analyzed 1,314 calibrated images obtained from the McGill Calibrated Colour Image Database (Olmos and Kingdom, 2004) and 4,163 images from the van Hateren database (van Hateren and van der Schaaf, 1998). The images were processed by converting them to grayscale and passing them through different ON and OFF luminance/response functions modeled as Naka-Rushton functions (Kremkow et al., 2014; Pons et al., 2017). The output for each pixel, R(x,y), was determined by the pixel luminance of the original gray image, L(x,y), the exponent of the Naka-Rushton function (n), the luminance that generated 50% of the maximum response, L50, and the maximum response, Rmax, as shown in Equation 5. The L50 and n values were lower for the ON than the OFF pathways to simulate the larger ON than OFF luminance/response saturation (L50: 0.1, n: 1.6, Rmax: 1 for the ON pathway; L50: 0.5, n: 2.5, Rmax: 1 for the OFF pathway).

The pixel outputs, RON (x,y) and ROFF (x, y) were normalized by their maximum luminance value (maximum pixel value within each image). Then, the pixels of RON with the 10% largest values were used to make an image of lights and the pixels of ROFF with the 10% lowest values were used to make the image of darks. The 10% brightest pixels from RON represent the highest ON temporal contrast driving the strongest ON visual responses during eye movements. The 10% darkest pixels from ROFF represent the highest OFF temporal contrast driving the strongest OFF visual responses. Any negative value in ROFF (x,y) was set to 0, which is the maximum darkest contrast possible. The images of lights and darks were then converted into binary images (MATLAB function ‘im2bw’) and continuous/connected regions of light or dark pixels were identified (MATLAB function ‘bwlabel’) and quantified in number and size.

Statistical analysis

All data presented show the mean across multiple stimulus repetitions and cortical recording sites. All error bars are ± SEM (standard error of the mean) across multiple recording sites. The sample sizes shown in the figure legends indicate the number of cortical recording sites included in the average. All comparisons between darks, lights, target sizes, target durations, and luminance conditions were done using the Wilcoxon RankSum test unless specified otherwise. For testing significance of contrast polarity distributions, the Hartigan test for bimodality was used. Significance is marked in the figures as follows: * p<0.05, ** p<0.01 and *** p<0.001.