Abstract representations of small sets in newborns.

From the very first days of life, newborns are not tied to represent narrow, modality- and object-specific aspects of their environment. Rather, they sometimes react to abstract properties shared by stimuli of very different nature, such as approximate numerosity or magnitude. As of now, however, there is no evidence that newborns possess abstract representations that apply to small sets: in particular, while newborns can match large approximate numerosities across senses, this ability does not extend to small numerosities. In two experiments, we presented newborn infants (N = 64, age 17 to 98 h) with patterned sets AB or ABB simultaneously in the auditory and visual modalities. Auditory patterns were presented as periodic sequences of sounds (AB: triangle-drum-triangle-drum-triangle-drum …; ABB: triangle-drum-drum-triangle-drum-drum-triangle-drum-drum …), and visual patterns as arrays of 2 or 3 shapes (AB: circle-diamond; ABB: circle-diamond-diamond). In both experiments, we found that participants reacted and looked longer when the patterns matched across the auditory and visual modalities - provided that the first stimulus they received was congruent. These findings uncover the existence of yet another type of abstract representations at birth, applying to small sets. As such, they bolster the hypothesis that newborns are endowed with the capacity to represent their environment in broad strokes, in terms of its most abstract properties. This capacity for abstraction could later serve as a scaffold for infants to learn about the particular entities surrounding them.

Introduction

From the very first days after birth, newborn infants are not tied to respond to low-level, modality-specific aspects of their environment: they can represent abstract properties, and access these representations from several modalities (for reviews see Bahrick, Lickliter, & Flom, 2004; Guellaï et al., 2019; Lewkowicz & Ghazanfar, 2009; Streri, de Hevia, Izard, & Coubart, 2013). Hence, since the pioneer studies of the 1980s, we know that newborns react specifically when the sounds they hear and the images they see are temporally synchronous or emanate from the same place (Butterworth, 1983; Filippetti, Johnson, Lloyd-Fox, Dragovic, & Farroni, 2013; Lewkowicz, Leo, & Simion, 2010; Morrongiello, Fenwick, & Chance, 1998; Slater, Brown, & Badenoch, 1997; Slater, Quinn, Brown, & Hayes, 1999), an indication that they perhaps project visual and auditory information onto an amodal sense of time and space. While this evidence is only suggestive (and may simply reflect the existence of special detectors for temporal and spatial synchrony), several studies have now unambiguously demonstrated that newborns can recognize some amodal properties across senses: for example, direction of movement (Orioli, Bremner, & Farroni, 2018), shape (Sann & Streri, 2007; Streri & Gentaz, 2004), texture (Kaye & Bower, 1994; Sann & Streri, 2007), or approximate numerosity (Coubart, Izard, Spelke, Marie, & Streri, 2014; Izard, Sann, Spelke, & Streri, 2009; McCrink, Veggiotti, & de Hevia, 2020). In other experiments, newborns were even found to react to correspondences between dimensions of different nature, thus accessing even more abstract invariants. For example, newborns can map numerical quantity or temporal duration with spatial extent (de Hevia, Izard, Coubart, Spelke, & Streri, 2014), elevation in visual space with auditory pitch (Walker et al., 2018), and left-right positions in visual space with auditory (de Hevia, Veggiotti, Streri, & Bonn, 2017) or visual (Di Giorgio et al., 2019) numerical quantities.

As part of a larger project looking for amodal representations in newborns (Bonn, Netskou, Streri, & de Hevia, 2019; Coubart et al., 2014; de Hevia et al., 2014; de Hevia et al., 2017; Izard et al., 2009; Streri et al., 2013), here we tested whether they possess abstract representations that apply to small sets. Indeed, in all the studies that demonstrated abstraction of magnitudes in newborns (Coubart et al., 2014; de Hevia et al., 2014; de Hevia et al., 2017; Di Giorgio et al., 2019; Izard et al., 2009; McCrink et al., 2020), participants were exclusively presented with large sets. Older infants however use two different systems to process the numerosity of small and large sets, at least from the age of 5 months. Hence, infants show different behavioral signatures in the small and large numerosity ranges (ratio-based discrimination and little access to individual item features for large numerosities vs. set size capacity limit and sensitivity to individual items for small numerosities; for reviews see Carey, 2009; Feigenson, Dehaene, & Spelke, 2004; Hyde, 2011), and moreover, they often fail to compare a small vs. a large numerosity (Cordes & Brannon, 2009; Coubart, Streri, de Hevia, & Izard, 2015; Feigenson & Carey, 2005; Xu, 2003). This last phenomenon is already observable at birth: in the same experimental paradigm, newborns failed to match numerosities 2 vs. 6 across audition and vision (a contrast between a small vs. a large numerosity), while they successfully detected cross-modal matches for pairs of larger numerosities differing in a 1:3 ratio (e.g. 3 vs. 9, 4 vs. 12, Coubart et al., 2014). Just like older infants, newborns thus process small and large sets differently, with a switching point between set sizes 2 and 3. Still, while these findings indicate that newborns' responses to large numerosities do not extent to small sets, to this date evidence that newborn infants are able to process small sets in an abstract way is lacking.

At first view, it may seem possible to address this question by adapting the task developed by Izard et al. (2009) to test newborns' abstract processing of large numerosities. Given the findings of Coubart et al. (2014), to remain within the range of small numerosities newborns should be tested with numerosities 1 and 2, that is, they should be presented with sequences of 1 or 2 sounds in the auditory modality conjointly with visual arrays of 1 or 2 items, while their looking times to congruent vs. incongruent trials are being recorded. However, we intuited that this strategy may fail to yield interpretable results: infants may not process isolated items as sets of numerosity 1, but rather focus on analyzing the features of these items. To circumvent this issue, we decided to present our participants not with simple arrays of 1 or 2 items, but with patterned sets AB vs. ABB. We reasoned that the presence of a pattern may encourage participants to process relations between items, and thus to encode set properties rather than item properties. Importantly, patterns AB vs. ABB cannot be discriminated on the basis of newborns' known representations of large approximate numerosities: while these patterns differ both in terms of their maximum by-item numerosity (1 vs. 2) as well as in global numerosity (2 vs. 3), these two numerosity contrasts are beyond newborns' known ability for numerical abstraction. Demonstrating crossmodal matching of AB vs. ABB would thus reveal the existence of abstract representations that differ from the representations evidenced in previous studies of newborns' numerical abilities.

Patterns AB vs. ABB also differ along another amodal dimension: in the presence vs. absence of a repetition. Importantly, just like in the case of small numerosities, as of now there is no evidence that newborn infants can represent repetitions in a way that is abstract, shared between modalities. Two different lines of research have provided evidence that infants are sensitive to repetition patterns, e.g. ABC vs. ABB (e.g. Bulf, Brenna, Valenza, Johnson, & Turati, 2015; Bulf, de Hevia, Gariboldi, & Macchi Cassia, 2017; Dawson & Gerken, 2009; Ferguson, Franconeri, & Waxman, 2018; Gerken, Dawson, Chatila, & Tenenbaum, 2015; Johnson et al., 2009; Marcus, Fernandes, & Johnson, 2007; Marcus, Vijayan, Rao, & Vishton, 1999; Saffran, Pollak, Seibel, & Shkolnik, 2007; for reviews see de la Cruz-Pavía & Gervain, 2021; Rabagliati, Ferguson, & Lew-Williams, 2019), or more generally, that they can contrast arrays made of same vs. different items (e.g. Ferry, Hespos, & Gentner, 2015; Hochmann, Mody, & Carey, 2016; for reviews, see Hespos, Gentner, Anderson, & Shivaram, 2021; Hochmann, 2021). In particular, the newborn brain is sensitive to the presence of adjacent repeated elements in the auditory modality (Gervain, Berent, & Werker, 2012; Gervain, Macagno, Cogoi, Peña, & Mehler, 2008). From the age of 3 months, infants not only detect repetitions, but also generalize a familiar pattern to test stimuli made of novel items, both within the auditory (4mo: Dawson & Gerken, 2009; 7mo: Marcus et al., 2007; Marcus et al., 1999; 9mo: Gerken et al., 2015) and visual (3–4mo: Anderson, Chang, Hespos, & Gentner, 2018; Ferguson et al., 2018; 7mo: Bulf et al., 2015; Bulf et al., 2017; Ferry et al., 2015; Saffran et al., 2007; 8- and 11mo: Johnson et al., 2009) modalities. At the age of 7 months, infants can even generalize a familiar pattern to new stimuli of different nature: from spoken syllables to tone pitches, animal sounds or sound timber (Marcus et al., 2007), from sequences of visual numerosities to sequences of visual shapes (Bulf, Capparini, Nava, de Hevia, & Macchi Cassia, 2022).

However, to this date, evidence that infants can generalize repetition patterns across modalities is scarce. A few studies reported that infants (age 5 months and older) benefit from the redundant presentation of a pattern across modalities (Frank, Slemmer, Marcus, & Johnson, 2009; Thiessen, 2012; Tsui et al., 2016; see also Lew-Williams, Ferguson, Abu-Zhaya, & Seidl, 2019). These findings are however compatible with several interpretations, and do not necessarily indicate that infants represent patterns in a way that is abstract, shared between modalities. For example, it is possible that participants were more likely to access pattern information in the redundant conditions simply because several modalities carried information to detect the changes at test, as opposed to just one. As a stronger test of amodal pattern representations in infants, Bulf et al. (2021) studied whether 7-month-olds could transfer a rule across the visual and auditory modalities. The study yielded some evidence of successful transfer; however this evidence was fragile, as it was only found in one direction (from vision to audition) and only in a subset of the participants (infants habituating faster, in the sense of less accumulated looking time; median split).

Interestingly, the same gap is found in the animal literature: while evidence that non-human animals represent repetition pattern within the auditory or visual modality is plentiful (for review see Milne, Wilson, & Christiansen, 2018), only one study (Ravignani & Sonnweber, 2017) reported a cross-modal pattern congruency effect in an animal species1. In this study, two chimpanzees were found to respond faster to symmetrical visual patterns (i.e., ABA amongst AAB or ABB distractors) when the visual stimuli were preceded by a congruent ABA auditory patterns, compared to incongruent ABB or AAB patterns (Ravignani & Sonnweber, 2017). Although suggestive, again the evidence reported is not very strong, as acknowledged by the authors themselves: the reverse response rules could not be tested, and the tendency to select congruent patterns across audition and vision was not significant. These gaps in the developmental and comparative literature, together with results showing that human adults can easily track two different grammars in parallel in two modalities (Conway & Christiansen, 2006), led some authors to propose that algebraic patterns are usually not represented in an abstract format, but rather are induced as domain-specific rules (Frost, Armstrong, Siegelman, & Christiansen, 2015; Milne et al., 2018).

In order to assess whether newborns possess abstract representations of small sets, here we tested whether they are able to match patterns AB vs. ABB across modalities. Infants aged 0 to 4 days were presented with audio-visual stimuli displaying either the same pattern in the visual and auditory modalities (congruent AB or ABB), or different patterns across modalities (AB and ABB paired together; incongruent). As newborns generally display a preference for stimuli that match across the auditory and visual modalities (e.g. Addabbo, Colombo, Picciolini, Tagliabue, & Turati, 2021; Aldridge, Braga, Walton, & Bower, 1999; Coubart et al., 2014; de Hevia et al., 2014; Filippetti et al., 2013; Guellaï, Streri, Chopin, Rider, & Kitamura, 2016; Izard et al., 2009; Lewkowicz et al., 2010; Orioli et al., 2018; Walker et al., 2018), we predicted that infants should look longer at congruent than incongruent stimuli if they detected the matching patterns.

Experiment 1

In Experiment 1, newborns were presented with a succession of congruent and incongruent trials displaying AB/ABB patterns simultaneously in the visual and auditory modalities. Care was taken to use A and B items that were discriminable by newborn infants. The visual arrays were composed of items differing in shape (circle vs. diamond; see Sann and Streri, 2007, Sann and Streri, 2008 for a demonstration that newborns discriminate angled vs. smooth shapes), color (white vs. red; see Adams, Maurer, & Davis, 1986 for a demonstration that newborns discriminate red vs. grey), and size (in a 1:2.9 ratio; see de Hevia et al., 2014 for a demonstration that newborns discriminate between rectangles in a 1:3 area ratio). Visual items were presented simultaneously to facilitate the perception of a pattern (see Ferguson et al., 2018 for a demonstration that simultaneous presentation facilitates pattern perception in infants; and Conway & Christiansen, 2009 for a similar result in adults). The auditory stimuli were constructed with two different sounds differing markedly in timber and pitch (bass drum vs. triangle; for evidence that newborns and fetuses are sensitive to pitch see Lecanuet, Graniere-Deferre, Jacquet, & DeCasper, 2000; Walker et al., 2018), and cycled to form a rhythmic pattern (for evidence that newborns are sensitive to periodicity in musical rhythm, see Stefanics et al., 2007; Winkler, Haden, Ladinig, Sziller, & Honing, 2009).

Methods

Participants

Families were recruited directly in the maternity ward, before they were discharged from the hospital. Informed written consent was obtained from each family, and a caregiver was systematically present during the test. The project was approved by the Comité de Protection des Personnes Ile de France II (IRB 00001072, project 2010–6).

We initially included 16 healthy newborn infants (age 15–83 h; 8 female). All infants were born after 37 weeks of gestation (range 37:1–41:2 weeks:days), weighted at least 2500 g at birth (range 3028-4130 g), and had an Apgar score of at least 8 after 10 min (all scores were 10). Forty other infants were excluded because they were not in a quiet, alert state (crying: 12, sleeping: 7; hunger: 7; hiccup and defecating: 1), because they failed to look at the stimuli on at least one trial (1), because of a technical problem with the stimuli presentation or with the video recording (6), or because an experimenter intervened while the infant was looking at the screen (4). Lastly, in two cases, the infant was excluded because coders did not agree on looking times (see data recording and analysis section).

As the results revealed an interaction between the variables of congruency and trial order instead of the expected main effect of congruency (see Supporting Online Material for the results of the first sample of 16 infants), we decided to double the sample size with another 16 infants (age 17–98 h; 13 female; born at 37:5–41:5 week:days of gestation; weight at birth 2900-3760 g; Apgar score 9–10 after 10 min). For this second sample, we reduced the number of trials from 6 to 4 in a hope to reduce attrition. Still, while testing this second sample we had to exclude 25 infants for crying (6), falling asleep (7), regurgitating (1), hiccup (1), failing to look at the stimuli on at least one trial (7), or because of technical problems with the video camera (3).

Stimuli

Infants were presented with videos showing a horizontal array of one red circle and one or two white diamonds (an AB or ABB array), while a soundtrack repeatedly played a triangle-drum (AB) or a triangle-drum-drum (ABB) sequence (Fig. 1). Two types of videos were created: congruent videos where the pattern matched across the auditory and visual modalities (an AB or ABB pattern in both modalities), and incongruent videos (AB in one modality and ABB in the other). The soundtrack played continuously, each cell (AB or ABB) lasting 2 s, such that infants presented with auditory AB or ABB sequences received equal exposure. As a result, the rate of presentation of individual sounds was faster in ABB than AB sequences. In contrast, in the visual modality the spacing between the shapes was maintained constant across stimuli, and therefore ABB arrays were more extended in space than AB arrays. To facilitate the extraction of the repeated auditory sequence, the visual arrays were animated with a stroboscopic movement and moved each time the A sound (triangle) played. Importantly though, the visual array always moved together as a whole, such that temporal synchrony did not provide any cue to differentiate congruent and incongruent videos.

Fig. 1

Auditory and visual components of the stimuli.

Procedure

For each infant, the auditory pattern remained the same throughout the whole experiment (either AB or ABB). Infants were first familiarized with the auditory stimulus for 20s, while the computer screen remained black. They then received congruent and incongruent trials in alternation. Infants tested in the original sample received a total of 6 trials, while the number of trials was reduced to 4 in the second sample. Order of presentation of the congruent and incongruent stimuli (congruent first or incongruent first) and auditory condition (AB or ABB) were fully crossed and counterbalanced across participants.

During the experimental session, an experimenter recorded whether the infant looked at the stimuli in real time. The experimental condition (auditory AB or ABB; trial order with congruent first or incongruent first) was chosen by the program, such that the coder did not know when the infant was looking at a congruent or an incongruent stimulus. Each trial ended when the infant looked away for 2 s or when he or she had accumulated 60s of looking, whichever came first. A second experimenter stood behind the infant to monitor any sign of discomfort.

Data recording and analysis

Data from the first 4 trials of all infants were recoded offline by a second observer with substantive experience coding newborns' looks. Trials where the looking time recorded online and offline differed by more than 5 s (20/128 trials, 15.6%) were recoded again by another trained observer (usually the experimenter who live-coded during the experiment). For two participants, this third measurement differed by more than 5 s from the two first measurements in at least one trial: these two infants were excluded from the sample and replaced.

The two closest measures of looking times were averaged (by virtue of our recoding and exclusion policy these two measures differed by less than 5 s) and entered in an ANOVA with the two within-subject variables of trial pair (first or second) and congruency (Congruent or Incongruent), and the two between-subject variables of auditory condition (AB or ABB) and trial order (Congruent first or Incongruent first). This first analysis yielded residuals that departed significantly from normality (Shapiro test, W = 0.948, p < .0001), thus the final analysis was performed on log-transformed looking times (residuals of the ANOVA of log-transformed looking times did not depart from normality: Shapiro W = 0.986, p = .19). Significant interactions were explored post-hoc by computing simple contrasts on the Congruency variable, Holm corrected for multiple comparison (using afex and emmeans packages in R; Lenth, 2022; Singmann, Bolker, Westfall, Aust, & Ben-Shachar, 2021). To compensate for the fact that the decision of doubling the sample size was taken after we had looked at the results of the first sample, all p-values reported are adjusted for 2 comparisons using Bonferroni's correction. A sensitivity analysis (conducted in G*Power, Faul, Erdfelder, Buchner, & Lang, 2009) indicates that with a total sample size of 32 participants, and with the Bonferroni correction applied, Experiment 1 can detect effects of size η2p = 0.17 with a power of 80%. This experiment is thus powered to detect large effects only.

Lastly, we also ran an exploratory analysis with a third between-subject variable for sample (first or second), but there was no significant effect or interaction associated with this variable (main effect and interactions, ps > 0.11, η2ps < 0.10). The data and the scripts for the analyses can be found at https://osf.io/zdvnh/

Results

While the main effect of congruency was not significant (looking times of M = 27.8 s vs. 22.9 s for congruent vs. incongruent trials; F(1,28) = 4.5, p = 0.088, η2p = 0.14), the analysis yielded a significant interaction between congruency and trial order (F(1,28) = 14.0, p = 0.002, η2p = 0.33), as well as a significant interaction between congruency and trial pair (F(1,28) = 6.4, p = 0.035, η2p = 0.19). These findings are illustrated on Fig. 2, which displays responses to congruent and incongruent trials by pair and trial order. To explore the two significant interactions, we computed contrasts comparing looking times to congruent vs. incongruent videos, first in the two subgroups receiving either a congruent or an incongruent stimulus first (interaction between congruency and trial order), and then, for all infants, on the first and second pairs of trials (interaction between congruency and test pair). Infants who received a congruent trial first showed markedly longer looking times at congruent audio-visual stimuli than at incongruent stimuli (M = 32.6 s vs. 18.9 s; Post-hoc contrast p = 0.001; 14/16 infants looked longer at the congruent trial), while in contrast, the infants who started the experiment with an incongruent trial did not develop any preference (M = 22.9 s vs. 26.9 s for congruent vs. incongruent trials; Post-hoc contrast p = 0.52; 7/16 infants looked longer at the congruent trial). Moreover, analyses by trial pair showed that across groups, newborns displayed longer looking times at congruent videos in the second pair of trials (M = 29.5 s vs. 20.4 s; Post-hoc contrast p = 0.002; 22/32 infants looked longer at the congruent trial), while looking times at congruent vs. incongruent videos did not differ in the first pair of trials (M = 26.0 s vs. 25.4 s; Post-hoc contrast p = 1.0; 16/32 infants looked longer at the congruent trial). As illustrated on Fig. 2, during the first pair of trials all infants tended to display a transient response to the very first stimulus encountered, irrespective of the nature of this stimulus; while in the second pair of trials this preference was maintained only in the subgroups of infants who had first received a congruent trial, yielding a main effect of test congruency.

Fig. 2

Looking times in Experiment 1.

Note. Looking times are presented separately for each order (whether infants received a congruent or an incongruent stimulus first), trial pair, and congruency level. The order of the bars shows the successive trials presented to the infants, starting either from a congruent or from an incongruent trial. Error bars are 95% CI.

In addition to these effects, the ANOVA identified a significant four-way interaction between auditory pattern, order, congruency and pair (F(1,28) = 6.1, p = 0.040, η2p = 0.18). Exploring this interaction revealed that amongst infants presented with a congruent video first, preference for the congruent stimulus was significant in the first pair of trials for the group hearing an AB pattern (M = 39.0 vs. 13.2 s; Post-hoc contrast p = 0.036; 7/8 infants looked longer at the congruent trial), and in the second pair of trials for the group hearing an ABB pattern (M = 30.1 vs. 12.3 s; Post-hoc contrast p = 0.011; 6/8 infants looked longer at the congruent trial). In all other cases (including in the groups presented with an incongruent video first), no significant preference emerged (Post-hoc contrasts ps > 0.20; 2–7/8 infants looked longer at the congruent trial).

No other effects or interactions were significant in the ANOVA (ps > 0.30, η2ps < 0.09).

Discussion

The findings of Experiment 1 provide evidence that newborns can match patterned sets AB vs. ABB across audition and vision as, in general, they looked longer at congruent stimuli in the second pair of trials. Importantly, our stimuli were conceived to exclude several deflationary interpretations and ensure that the visual and auditory displays could only be matched based on set-level properties. First, while newborns tend to rely on temporal synchrony to form audio-visual associations (Lewkowicz et al., 2010; Morrongiello et al., 1998; Slater et al., 1997), in our videos the ‘B' sounds of the AB and ABB auditory patterns were not associated with any visual event, excluding temporal synchrony as a cue. Second, the task could also not be solved on the basis of associations between visual and auditory features. Suppose for example that newborns are prone to spontaneously associate drum sounds (the ‘B' items of our auditory patterns) with white diamonds (our visual ‘B's)2. Detecting such correspondences between auditory and visual features would not be sufficient to solve our task, because all our stimuli contained the same features (both A and B items were presented in both modalities), and thus supported the same associations. To distinguish between congruent and incongruent stimuli, infants needed to differentiate between patterns containing one vs. two ‘B' items, and they further needed to associate the auditory and visual patterns that contained the same number of ‘B's. Consequently, representing isolated items was not sufficient; as such, our findings demonstrate that newborns are able to represent properties of small sets in an abstract way.

In detail, infants showed different responses to congruent vs. incongruent audio-visual patterns only when the very first stimulus they received was congruent: infants presented first with a congruent trial looked longer at congruent than incongruent audio-visual patterns throughout the two pairs of trials presented, while in contrast infants presented first with an incongruent trial did not develop any stable preference. The very first trial presented thus had an influence on participants' analysis of the subsequent stimuli. We suspect that this first trial oriented participants to attend to different aspects of stimuli: while a first congruent stimulus led newborns to focus on set properties and respond to audio-visual congruency, infants presented first with an incongruent stimulus may have developed different approaches. For example, as the visual arrays moved systematically when the first sound of the auditory pattern was played in all our videos (‘A' sound in AB or ABB), some infants may have been content to discover this relation of temporal synchrony and subsequently narrowed their analysis of all stimuli to this aspect only, thus looking equally at all trials. Others may have formed alternative interpretations of the mapping between sounds and shapes, for example associating one drum sound with two white diamonds, or the reverse3.

Experiment 2 builds on the findings of Experiment 1 to further explore the nature of the relation extracted by infants when they were presented with a first congruent stimulus. More specifically, we asked whether the relation extracted by infants is specific to one particular pattern, or whether infants who learn a relation of correspondence for one pattern subsequently can generalize this relation to a new pattern.

Experiment 2

Experiment 2 used the same audio-visual stimuli as Experiment 1. Two groups of newborns were first familiarized with either a congruent or an incongruent video, and then tested with new stimuli in two trials (one congruent, one incongruent). Crucially, and in contrast to Experiment 1, the auditory pattern changed systematically between the familiarization and the test trials. Given the results of Experiment 1, and in line with previous findings obtained in this paradigm (de Hevia et al., 2014; de Hevia et al., 2017), we expected that infants familiarized with an incongruent stimulus would not display any preference between the two test trials. Infants familiarized with a congruent stimulus may look longer at the congruent test, if they are able to generalize the perceived cross-modal correspondence to a new pattern.

A sample of 32 newborn infants (age 21–98 h, average 54.8 h; 13 female; born at 37:5–42:0 week:days of gestation; birth weight 2650-4370 g; Apgar score of 10 after 10 min) were included in Experiment 2. Another 42 infants were excluded, because they were not in a quiet, alert state (hungry: 3; crying: 11; falling asleep: 7; hiccup: 2), because they failed to look at the screen on at least one trial (5), because the phone of a parent rang during the experiment (1), or because of technical problems with the presentation of the stimuli (1). Five infants were excluded because they did not look sufficiently at the familiarization stimulus (see below), and 6 infants were excluded because they reached the maximum looking time (60s) on both test trials. Lastly, one infant was excluded because coders disagreed on their estimation of looking times on one trial by more than 5 s.

Experiment 2 used the same video stimuli as Experiment 1. Infants were first familiarized with a congruent or an incongruent video, played for a fixed duration of 60s. In contrast to Experiment 1, Experiment 2 did not include a phase where auditory stimuli were presented in isolation. In order to ensure that infants encoded the familiarization video, they were included in the final data set only if they had looked during at least 50% of the familiarization trial (30s). In accordance with the results of Experiment 1 (no difference in looking between congruent and incongruent videos in the first trial), looking times at the familiarization video were similar in all subgroups (average looking times ranging from M = 46.7 s to 50.9 s across groups; no effect of auditory pattern, visual pattern, and no interaction, all ps > 0.26).

After the familiarization, infants were presented with two test trials: one congruent trial, and one incongruent trial (Fig. 3). The auditory pattern changed systematically between the familiarization and the test trials, from AB to ABB, or from ABB to AB. In the two test trials, this new auditory pattern was paired successively with an AB and an ABB visual pattern (in counterbalanced order).

Fig. 3

Procedure of Experiment 2.

Note. Three variables were fully crossed between participants: the familiarization trial was either congruent or incongruent, the auditory familiarization played either pattern AB or ABB, and the first test trial was either congruent or incongruent. For the sake of simplicity, here we did not illustrate this last manipulation (only conditions where the first test trial is congruent are shown).

Analyses

Looking times were recoded following the same procedure as in Experiment 1. For the infants included, 12 out of 96 trials (12.5%) were recoded twice as the first offline coding deviated from the online looking time measure by more than 5.0 s. Test trial looking times were analyzed in an ANOVA with three between-subject variables for Familiarization congruency (Congruent, Incongruent), Auditory familiarization pattern (AB or ABB) and Test trials order (Congruent First, Incongruent First), and one within-subject variable for Test congruency (Congruent, Incongruent). Residuals did not deviate from normality (Shapiro test W = 0.968, p = .10), hence no transformation was applied to looking times. A sensitivity analysis indicates that with a total sample size of 32 participants, Experiment 2 can detect effects of size η2p = 0.15 with a power of 80%; thus, again this experiment is powered to detect large effects only. The data and the scripts for the analyses can be found at https://osf.io/zdvnh/.

Results are displayed on Fig. 4. As predicted, infants from the two groups showed different responses to the congruent and incongruent trials (interaction between Familiarization condition and Test congruency, F(1,24) = 8.6, p = .007, η2 = 0.27). Infants familiarized with a congruent stimulus looked longer at the congruent test trial, even though the pattern presented at test was different from familiarization (M = 45.5 s vs.24.3 s; Post-hoc contrast p = .007, 14/16 infants looked longer at the congruent trial). In contrast, the looking times of the infants familiarized with an incongruent stimulus did not differ between the two test trials (M = 34.5 s vs. 40.6 s; Post-hoc contrast p = .36, 7/16 infants looked longer at the congruent trial). No other effect or interaction were significant (all ps > 0.12, η2s < 0.10).

Fig. 4

Looking Times at Test in Experiment 2.

Note. Looking times are presented separately for each familiarization condition (Congruent or Incongruent) and Test condition (Congruent or Incongruent). Error bars are 95% CI.

Experiment 2 yielded results consistent with our predictions, and with Experiment 1. Again, infants who first experienced an incongruent stimulus in familiarization subsequently looked equally at congruent and incongruent test trials, while infants familiarized with a congruent stimulus looked longer at the congruent than at the incongruent test. While in Experiment 1 the congruent trials presented the same matching pattern throughout the whole experiment (either AB or ABB), here the familiarization and test trials presented different congruent patterns (AB in familiarization and ABB at test, or vice-versa). These results suggest that newborns can learn to recognize a matching pattern across audition and vision, and then generalize this audio-visual correspondence to a new pattern.

However, the ANOVA results are compatible with a second interpretation: perhaps, newborns simply looked longer when the visual pattern presented was novel with respect to the familiarization trial (looking times to visually novel vs. familiar stimuli: M = 43.1 s vs. 29.4 s). Indeed, if we use a variable encoding the two test trials as visually familiar vs. novel instead of congruent vs. incongruent, the observed interaction between familiarization condition and test congruency requalifies as a main significant effect of visual novelty, while the non-significant main effect of test congruency translates into a non-significant interaction between visual novelty and familiarization group. The ANOVA results thus leave us with two possible interpretations: either that newborns familiarized with a congruent pattern later discriminated congruent vs. incongruent patterns at test; or that newborns in general looked longer at test trials presenting visually novel patterns with respect to familiarization, whether the familiarization was congruent or not4.

What conclusions can be drawn from Experiment 2, if we accept that these two interpretations of the results are possible? Interestingly, the behavior of the group familiarized with congruent stimuli contrasts with the findings of Experiment 1 where, rather than seeking novelty, newborns presented first with a congruent stimulus later displayed longer looking to (congruent) stimuli that were visually familiar. The designs of our two experiments differ in that, in Experiment 2, the auditory stimulus changed between the first (familiarization) and the second (test) trials, while the auditory pattern remained constant throughout the whole experiment in Experiment 1. When considered in the light of Experiment 1, the results of Experiment 2 thus demonstrate a minima that newborns can discriminate our AB vs. ABB stimuli within the auditory and the visual modalities, and that they expect a change of auditory input to be accompanied by a change in visual input – at least when they detect a match in pattern across audition and vision in the first stimulus.

General discussion

In two experiments, we presented newborns with stimuli displaying AB and ABB patterns simultaneously in the auditory and visual modalities. Auditory patterns were played as a cycling sequence of sounds (AB: triangle-drum-triangle-drum-triangle-drum …; ABB: triangle-drum-drum-triangle-drum-drum-triangle-drum-drum …), while an array of two or three visual shapes was displayed on a screen (AB: red circle + white diamond; ABB: red circle + white diamond + white diamond). Across trials, newborns were thus presented with the same auditory and visual items, albeit in different quantities. To solve our task, infants needed to represent properties of groups of items within each modality (numerosity and/or repetition), and detect whether these group properties matched across audition and vision.

In Experiment 1, newborns were presented with congruent and incongruent trials in alternation: the same auditory pattern was played throughout the whole experiment, while visual patterns alternated between AB and ABB. After a transient response to the first trial, infants looked longer at congruent audio-visual stimuli in the second pair of trials. In detail however, this preference was only present when the very first stimulus encountered was congruent; infants presented first with an incongruent stimulus did not develop any preference. This pattern of interaction was found again in Experiment 2, which used the same video displays but introduced a change in auditory pattern between a first familiarization trial and two ensuing test trials.

A minima, these results support four conclusions. First, in both experiments one subgroup of infants (infants presented first with a congruent stimulus) showed different looking times to displays that differed in their visual component only. This response shows that newborns can discriminate between our AB vs. ABB visual displays, corroborating previous findings about newborns' perception of numerosity (Antell & Keating, 1983; Turati, Gava, Valenza, & Ghirardi, 2013) and spatial extent (Turati et al., 2013) in small visual arrays. Second, infants presented with a first congruent stimulus showed different reactions to subsequent visual patterns in our two experiments: they looked longer at the familiar visual pattern in Experiment 1 (where auditory pattern had remained constant), and looked longer at the novel visual pattern in Experiment 2 (where auditory pattern had changed). These different responses indicate that newborns detected the change in auditory stimulation, and therefore that they discriminated our AB and ABB auditory stimuli. This again corroborates previous findings on newborns' perception of numerosities (Bijeljac-Babic, Bertoncini, & Mehler, 1993), temporal rate (Gardner, Lewkowicz, Rose, & Karmel, 1986) and item repetitions (Gervain et al., 2008; Gervain et al., 2012) in auditory sequences. Third, the different patterns of preference displayed by newborns in our two experiments provide a first indication that they did not process the auditory and visual components of our stimuli independently from each other. Rather, they reacted to associations between their auditory and visual input.

Fourth, and most importantly, the results of Experiment 1 show that infants can recognize matching AB or ABB patterns across audition and vision. Newborns thus possess representations that differentiate these patterns, and these representations are abstract enough to apply to stimuli of very different nature: periodic sequences of sounds on one hand, and arrays of visual shapes on the other hand. Importantly, these findings provide the first evidence that newborns possess abstract representations to match AB vs. ABB patterns across senses. As detailed in the introduction, studies testing newborns' numerical abstraction abilities found success only with large sets, and only when the set numerosities differed by a 1:3 ratio (Coubart et al., 2014; de Hevia et al., 2014; Izard et al., 2009; McCrink et al., 2020). Other studies found that newborns are sensitive to repetitions (Gervain et al., 2008) or that they can discriminate between sets of numerosity 2 vs. 3 (Antell & Keating, 1983; Bijeljac-Babic et al., 1993; Turati et al., 2013), but these abilities were only tested within the auditory or visual modalities. Our findings thus provide the first demonstration that newborns can represent a property of small sets in a way that is abstract, shared between modalities.

What kind of abstract representations did newborns deploy to solve our task? First, it should be noted that the stimuli presented in our study had no particular ecological relevance. It thus appears unlikely that the representations implicated in our task were selected to apply precisely to the kind of sound or shapes we presented. Just like they can process quantity for various kinds of stimuli (Bonn et al., 2019; de Hevia et al., 2014; de Hevia et al., 2017; Di Giorgio et al., 2019; Izard et al., 2009), newborns probably can deploy their small sets representations in response to a broad range of sensory inputs.

Second, newborns could have used several cues, either in isolation or in combination, to discriminate between our AB and ABB patterns: the numerosity of the set (2 vs. 3, abstracting away the difference between A and B items), the numerosity of the B items (1 vs. 2), and/or the mere presence of repeated items. As a fourth possibility, it is also possible that newborns quantified our stimuli not in terms of numerosity but in terms of the total amount of “stuff” presented in the auditory and visual modalities – given that we did not vary the size of visual shapes or the duration of sounds across AB and ABB patterns (relatedly, for evidence that older infants often respond to total continuous extent when presented with small homogeneous sets, see Feigenson, Carey, & Spelke, 2002).

The literature does not provide any strong argument to accept or reject the interpretations in terms of numerosity or repetition. On one hand, Coubart et al. (2014) found that newborns failed to match sets of 2 vs. 3 items across senses, which may be taken to suggest that newborns did not analyze our patterns in terms of numerosity. However, it remains possible that newborns quantified the ‘A' and ‘B' items separately in our stimuli, and processed the difference between AB and ABB as a contrast between numerosities 1 and 2 – a presumably easier numerical contrast than 2 vs. 3, especially if numerosity 3 exceeds the limit of newborns' small set system. Moreover, it is also possible that our stimuli provided a better context than Coubart et al. (2014)’s stimuli for newborns to respond to small numerosities: in line with the “Intermodal Redundancy Hypothesis” (Bahrick et al., 2004), the temporal synchrony between the ‘A' sounds and the movements of the visual arrays may have oriented participants to focus on amodal properties in our displays, leading them to attend to numerosity.

Similarly, one could argue that our participants did not respond to repetitions, given the absence of evidence for modality-abstract representations of repetitions in older infants or non-human animals. However, compared to more experienced infants and animals, young infants can sometimes be surprisingly responsive to crossmodal correspondences. Various reasons have been offered to explain these abilities (for a review, see Bremner, Lewkowicz, & Spence, 2012): for example newborns may suffer less interference from modality-specific properties due to the incomplete development of their senses (Turkewitz, 1994), they may be specifically tuned to detect crossmodal redundancies in their sensory input (Bahrick et al., 2004; Gibson, 1969; Lewkowicz & Ghazanfar, 2009; Streri et al., 2013), or representations of amodal, abstract, properties may develop faster because they apply to a broader range of experiences (a phenomenon described in the Bayesian modeling literature as the “blessing of abstraction”, Goodman, Ullman, & Tenenbaum, 2011).

Lastly, while numerosity and repetition both could be plausible drivers of our participants' responses, we doubt that newborns matched our auditory and visual stimuli based on their total continuous extent (total amount of “stuff”). First, while older infants often encode small homogeneous sets in terms of continuous extent rather than numerosity (Feigenson et al., 2002), they preferentially respond to numerosity when sets are made of different items (Feigenson, 2005). As our patterns mixed repetitions and item variations, it is not clear whether they would typically elicit a response to continuous extent in infants. Second, and most importantly, in Experiment 1 each participant received only one type of auditory pattern (either AB or ABB), and thus could not match the auditory and visual stimuli on the basis of relative quantities, i.e. mapping the smaller visual area with the smaller amount of sound duration, and the larger visual area with the larger amount of sound duration. Claiming that newborns solved Experiment 1 on the basis of continuous extent would imply the existence of absolute correspondences between auditory duration and visual extent at birth – and we would furthermore need to assume that, out of pure luck, our stimuli were rightly calibrated on these correspondences. We regard this possibility as highly unlikely.

As newborns may have matched our stimuli on the basis of numerosity, repetitions, or both, further research will be needed to determine which abstract property(ies) of small sets newborns can encode. For example, to determine whether newborns can encode abstract numerosities, they could be tested either with fully homogeneous or fully heterogeneous sets (e.g. A vs AA vs. AAA, AB vs. ABC). In reverse, to determine whether newborns encode repetitions irrespective of numerosities, they could be tested with homogeneous vs. heterogeneous sets (e.g. AA vs. AB, AAA vs. ABC). Note however that designing these experiments would raise some challenges, especially if, as the findings of Coubart et al. (2014) suggest, the small sets system is limited to numerosities 1 or 2 at birth5.

As a third line of discussion, the representations supporting newborn's matching of auditory and visual stimuli in our task may have been of two kinds. On one hand, perhaps newborns represented the amodal properties of the auditory and visual sets in an abstract format, detached from the items forming the sets (just like the symbols “1”, “2”, “repetition” convey information about sets without referring to set items). If so, newborns would simply need to compare these representations across their visual and auditory input, to find whether they matched or not. Alternatively, it is possible that newborns did not represent abstract properties such as numerosity or repetitions per se, but instead encoded the stimuli as arrays of multimodal objects – the kind of representations deployed by older infants when they process small sets (Carey, 2009; Feigenson et al., 2004)6. Interestingly, while the idea that infants represent small sets as arrays of individual objects was first introduced in the field of numerical cognition (in order to explain why infants fall prey to set size effects), it is possible that infants also encode repetitions within this format, i.e. in a representation of the form of {X X}, where the two X's are constrained to be identical objects (Hochmann et al., 2016). Under that view, when encoding small sets as arrays of objects, infants would be forming an integrated representation combining information about both repetitions and numerosity – and these two aspects may not be separable. For example, to compare auditory and visual sets infants may attempt to establish a one-one correspondence between objects perceived in the auditory and visual modality, and this process could be sensitive to relations between items (identical visual items should be put in correspondence with identical auditory items), in addition to numerosity (every visual item should have an auditory counterpart, and vice versa). In that case, infants would not be able to detect matches in numerosity when patterns of repetitions mismatch, and they also would fail to recognize patterns of repetitions in sets that vary in numerosities.

Lastly, at a more general level, our findings also illustrate how context can shape newborns' behavioral responses in quite sophisticated ways. Historically, demonstrations of habituation and familiarization effects already established that previous experiences can modulate newborns' subsequent responses, when infants are presented again with the same stimulus (e.g. Slater, Mattock, & Brown, 1990; Slater, Mattock, Brown, & Bremner, 1991). More recently, several studies found that a first stimulus can exert a durable influence on newborns' responses based on a more complex relation with subsequent test displays (Bonn et al., 2019; de Hevia et al., 2014; de Hevia et al., 2017). In these studies, newborns reacted specifically when the test and familiarization stimuli could be related in a consistent way across several dimensions: for example, newborns looked longer when both numerosity and spatial extent increased between familiarization and test, or when they both decreased. Furthermore, in one specific case (when numerosity was paired with luminance), newborns' analysis of relations between two dimensions appeared quite flexible, as they responded to mappings associating greater numerosities either with more luminous or with darker displays, depending on the experimental condition (Bonn et al., 2019). Far from an automatic response, in this particular case newborns' interpretation of the stimuli thus appeared to have been formed as the experiment unfolded, such that the succession of displays constrained the direction of the mapping they constructed.

In the present experiments, again the first stimulus influenced newborns' subsequent responses; however this time, it is the nature of this very first stimulus (congruent or incongruent) that determined subsequent responses. We suspect that infants engaged in a process searching for the best interpretation of this first stimulus. When the stimulus was congruent, this process converged on a representation of set properties – perhaps because representing both the visual and auditory inputs as reflecting the same set provides a parsimonious, yet powerful account of the perceptual input. In contrast, when infants were confronted with a first incongruent stimuli, this process did not seem to converge on the same interpretation across individuals, yielding chance responses at the group level. Some infants may have settled on aspects of the stimuli that are constant across congruent and incongruent trials (e.g. temporal synchrony between sounds and visual movements), thus looking equally in all trials. Others may have formed alternative interpretations of the relations between sounds and images (e.g. one drum sound associated with two diamonds, or the reverse), preferring incongruent stimuli. Yet others may have extended their search for a best interpretation and converged on set properties in the following trials.

Experiment 2 explored this process further and tested whether infants' preference would transfer to a new congruent pattern. Newborns responses were compatible with several interpretations. It is possible that the auditory change led infants to quit analyzing set properties as a relevant dimension, and to seek visual stimuli that simply differed from the stimuli associated with the familiarization auditory sequence in any respect. It is also possible that infants persisted in analyzing set properties across senses despite the change in auditory stimulation, and successfully detected congruence in test stimuli despite the change in pattern. Note that responding to a new congruent pattern supposes a mind with a sophisticated level of organization and abstraction, whereby representations of different sets, despite having different contents, are nonetheless treated as representations of similar kinds of contents. Quite suggestively, several studies have found that the infant brain is organized in functional areas processing different categories of stimuli; if this organization is already present at birth, it could support generalizations by stimulus categories (Blasi et al., 2011; Deen et al., 2017; Dehaene-Lambertz et al., 2006; Dehaene-Lambertz, Hertz-Pannier, & Dubois, 2006).

In summary, we provided evidence for a new type of abstract representations in newborns, applying to small sets. Together with a recent line of studies (de Hevia et al., 2014; de Hevia et al., 2017; Izard et al., 2009; Sann & Streri, 2007; Streri & Gentaz, 2004), these findings contribute to draw a picture where newborns are representing their environment in broad strokes, in terms of its most abstract and general properties. This early capacity for abstraction may later serve as a scaffold for infants to learn about the particularities of the various entities that surround them.

CRediT authorship contribution statement

Lucie Martin: Conceptualization, Methodology, Investigation, Data curation, Writing – review & editing. Julien Marie: Conceptualization, Methodology, Resources, Investigation, Data curation, Writing – review & editing. Mélanie Brun: Investigation, Data curation, Writing – review & editing. Maria Dolores de Hevia: Conceptualization, Writing – review & editing. Arlette Streri: Conceptualization, Writing – review & editing, Project administration. Véronique Izard: Conceptualization, Methodology, Software, Investigation, Formal analysis, Visualization, Writing – original draft, Writing – review & editing, Supervision, Funding acquisition, Project administration.

Declaration of Competing Interest

We have no known conflict of interest to disclose.