The Processing of Biologically Plausible and Implausible forms in American Sign Language: Evidence for Perceptual Tuning.

The human auditory system distinguishes speech-like information from general auditory signals in a remarkably fast and efficient way. Combining psychophysics and neurophysiology (MEG), we demonstrate a similar result for the processing of visual information used for language communication in users of sign languages. We demonstrate that the earliest visual cortical responses in deaf signers viewing American Sign Language (ASL) signs show specific modulations to violations of anatomic constraints that would make the sign either possible or impossible to articulate. These neural data are accompanied with a significantly increased perceptual sensitivity to the anatomical incongruity. The differential effects in the early visual evoked potentials arguably reflect an expectation-driven assessment of somatic representational integrity, suggesting that language experience and/or auditory deprivation may shape the neuronal mechanisms underlying the analysis of complex human form. The data demonstrate that the perceptual tuning that underlies the discrimination of language and non-language information is not limited to spoken languages but extends to languages expressed in the visual modality.

Introduction

Despite the wide variation of sound forms used in the world's languages, there are clear instances where highly salient acoustic signals are not incorporated in the repertoire of language sounds (e.g. a cough, a laugh, “the raspberry”, etc.), even though these are also generated by vocal action. Speakers/listeners are exquisitely sensitive to differences between speech and non-speech sounds (Benson et al., 2001; Mattingly, Liberman, Syrdal, & Halwes, 1971; Remez, Rubin, Berns, Pardo, & Lang, 1994; Vouloumanos, Kiehl, Werker, & Liddle, 2001) and preferences are established early in life. Studies of pre-linguistic infants show an initial bias towards language-like stimuli with subsequent exposure to one's native language shaping the processing system in a language-specific manner (Kuhl, 2000; Werker & Tees, 1992). Neuroimaging studies of adults have reported auditory regions in the bilateral superior temporal lobe that differentiate between speech and non-speech sounds, even in cases in which artificial non-speech stimuli preserve important spectral-temporal cues found in natural languages (Binder et al., 2000; Liebenthal, Binder, Spitzer, Possing, & Medler, 2005). Electrophysiological studies have observed rapid differentiation of speech and non-speech forms occurring as early as 100–150 ms (Aulanko, Hari, Lounasmaa, Näätänen, & Sams, 1993; Parviainen, Helenius, & Salmelin, 2005; Phillips et al., 2000; Shtyrov, Kujala, Palva, Ilmoniemi, & Näätänen, 2000; Vihla, Olli, & Salmelin, 2000) and report sensitivity between native and non-native phonemic contrasts (Näätänen et al., 1997). Collectively, these studies suggest that ontogenetically there is a high degree of sensitivity to speech and that consistent language exposure ultimately leads to perceptual tuning of early auditory systems used in language processing, with concomitant propensity to process non-linguistic signals differently.

Whether the perceptual sensitivity to language and non-language forms is solely a property of speech/auditory-based languages or whether it also extends to signed languages that are expressed in the visual modality is not known. Here we explore whether users of signed languages show evidence for perceptual tuning in the visual modality that leads to the ability to differentiate biologically possible and impossible signs. Signed languages used in deaf communities are instances of naturally occurring human languages that are expressed by changes in articulatory postures of hands and arms and entail movements of the body and face in the expression of linguistic form. The present study was designed to investigate the sensitivity of deaf signers and hearing non-signers in the discrimination of possible and impossible signs. The inclusion of non-signers in our paradigm allows us to determine general perceptual capabilities that are brought to bear during complex sign discrimination. The inclusion of expert signers allows us to ascertain whether lexical knowledge of American Sign Language (ASL) modulates the discrimination of possible and impossible signs. The use of stimuli that depict possible and impossible sign articulations provides a means to link studies of sign language processing to the growing literature on human form processing.

Sign language recognition

Psycholinguistic studies of sign language recognition have reported a range of processing effects that are commonly observed in studies of spoken language processing. These include effects of lexicality, neighbourhood density, word frequency, semantic priming and form-based interference effects (see Corina & Knapp, 2006 for a review). These studies suggest that, broadly speaking, the architecture of lexical recognition is likely similar for spoken and sign languages. Recent electrophysiological studies have explored the differential processing of signs, pseudo-signs and non-linguistic gestures presented in a sentence context. Grosvald, Gutierrez, Hafer, and Corina (2012) examined EEG data collected for target words that were semantically appropriate (baseline), semantically inappropriate, or a pseudo-sign that was formationally possible (i.e. a permissible articulation) but a non-existing ASL sign. Similar to what has been reported for spoken languages (Bentin, 1987; Bentin, McCarthy, & Wood, 1985; Hagoort & Kutas, 1995; Holcomb & Neville, 1990; Kutas & Hillyard, 1980), relative to the appropriate ending conditions, the two anomalous sign conditions elicited a greater negativity at approximately 400 ms after stimulus onset. Moreover, the formationally possible non-signs (i.e. pseudo-signs) produced a greater negativity than the semantically inappropriate sign. Importantly, the response to the pseudo-sign condition was clearly distinct from a fourth condition in which the critical ending sign was replaced with a non-linguistic “self-grooming” gesture (e.g. scratching one's face, rubbing one's eye, etc.), which bore no systematic relationship to the phonotactic properties of ASL. The appearance of a non-linguistic gesture produced a broad positivity beginning at 400 ms that was clearly different from the modulations of the N400 components evoked by different degrees of linguistic well-formedness. Thus, deaf subjects showed a clear sensitivity to linguistic properties of the sign stimuli and rejected non-sign gestures that were nevertheless executed with movements of the arms and hands. This study demonstrates electrophysiological evidence for stages of language processing that involve sensitivity to formational properties of ASL signs. These effects may be observed quite early in language processing (see also Gutierrez, Williams, Grosvald, & Corina, 2012) and are clearly distinct from non-linguistic gesture processing. However, this study provides an index of lexical integration of ASL signs within a sentential context and thus is likely to include cognitive processes beyond those used solely for sign recognition.

A small number of studies have directly compared the processing of sign language and other gestural actions in paradigms that permit the testing of both sign experts and sign-naive participants. Corina, Grosvald, and Lachaud (2011) reported data from a repetition priming paradigm for signs and non-linguistic self-grooming gestures that varied in viewpoint. Data from deaf signers and sign-naive hearing subjects showed equivalent patterns of priming for each class of stimuli across the varying viewpoints; however, deaf signers were uniformly faster and more accurate than hearing non-signers in categorising both sign and non-linguistic gestures. Evidence of subtle differences between sign experts and sign-naive subjects were seen in a more demanding task of recognition and lexical decision under conditions of prime and target inversion (Corina & Grosvald, 2012). Deaf subjects were adept at utilising both inverted signs and self-grooming gesture primes for subsequent target decisions, facilitating response times. In contrast, hearing subjects showed priming effects only for the self-grooming gestures. In addition, data from the categorisation of inverted targets indicated that deaf subjects used a configural processing strategy in the recognition of signs compared to hearing non-signers. These data suggest that sign expertise may lead to perceptual modifications of a general-purpose human action recognition system that for deaf signers allows for the rapid recognition of human actions, including linguistic actions (i.e. signs) and non-linguistic self-grooming human actions.

The present study adds to this growing literature by examining the processing of possible and impossible sign forms using magnetoencephalography (MEG), an electrophysiological technique with high temporal resolution that can provide information about the temporal dynamics that give rise to sign language processing. Our study draws upon data from human form processing to test a range of evoked response fields with known cortical generators that may be associated with the modification of perceptual systems for sign language. More generally, this study contributes to our understanding of how human languages may impact perceptual sensory systems.

Possible and impossible sign recognition experiment

In order to identify the neural responses and regions associated with recognition of possible and impossible signs we used MEG. Our study capitalises on the fact that the cortical dynamics associated with visual human form processing are reflected in a series of evoked responses – the M100, M130, and M190 – each of which can be inferred to underlie either basic category-unselective visual processes (e.g. M100, M130) or category-specific visual computations (e.g. M190) based on their cortical origins: The M100 response is usually localised to V1 or its vicinity (Ishizu, Amemiya, Yumoto, & Kojima, 2010; Taylor, Bayless, Mills, & Pang, 2011), while the origin of the M130 lies near the occipitotemporal cortex (Tarkiainen, Cornelissen, & Salmelin, 2002), and the M190 is generally localised to the extrastriate body area (EBA) (Ishizu et al., 2010; Peelen & Downing, 2007; Pourtois, Peelen, Spinelli, Seeck, & Vuilleumier, 2007; Taylor, Roberts, Downing, & Thierry, 2010). To assess whether our plausibility manipulation differentially affected lexical access, we examine the M400-evoked response. The M400 response reflects facilitated access of stored information (Lau, Almeida, Hines, & Poeppel, 2009).

We tested deaf signers and normal-hearing non-signing participants during a newly developed categorisation task of biologically possible and impossible sign language forms. The stimuli were pictures of ASL signs half of which had been edited such that the position of either the arms or the hands was inverted (left became right and right became left), resulting in an impossible human body configuration (Figure 1(a) and 1(b)). These stimuli were generated in order to preserve the gross characteristic of articulatory postures used in ASL while rendering some forms decidedly impossible. We chose signs that were easily identified in the absence of movement. Our choice to use static emblems of sign form rather than moving forms was strategic in that we wished to use stimuli that would maximally elicit a time-locked evoked response, rather than a response whose timing may be less determinant as the temporal course of sign articulation unfolds in time. Prior psycholinguistic studies have used static depictions of sign forms to ascertain hemispheric and lexical processing in deaf signers (see, e.g. Poizner & Lane, 1979). fMRI data from the last author's laboratory indicate that static representations of ASL signs produce activation in responses canonical language regions, including activation in motion sensitive area MT+ (Spotswood & Corina, 2009) similar to that observed in the perception of static representations of dynamic events (Kourtzi & Kanwisher, 2000). Note further that the use of representative static depictions of dynamic behaviours is nearly ubiquitous in the study of facial expression (Ekman, 1993).

Figure 1.

(a) Example of possible gesture. (b) Example of impossible gesture. (c) Reaction time results. Error bars represent standard errors of each condition. (d) Distribution of d' scores for the discrimination between possible versus impossible gestures.

If our hypothesis is on the right track that sign language expertise modifies general-purpose recognition systems used in the service of human form recognition, both behavioural and neurophysiological indices of this effect should be observed. The predicted behavioural effects are that deaf signers should be more accurate in their discrimination abilities and overall faster in distinguishing possible from impossible body configurations than non-signers. The neurophysiological predictions are more complex, due to the uncertainty surrounding which brain mechanisms could reflect the predicted behavioural modification. Since the visual processing of human forms is supported by both domain-general and domain-specific visual cortical regions, there are at least three alternatives regarding the brain mechanisms that could be modified due to sign language experience: The changes could involve only domain-specific visual regions, only domain-general visual regions, or both. If the faster and better detection of biologically possible and impossible signs predicted to be found in the deaf-signer group is subserved by changes in domain-specific cortical regions (e.g. EBA), one would expect to see effects reflected in the M190 response in the deaf-signer group. Alternatively, if only domain-general visual cortical regions are implicated, one expects to see the effects of the impossible sign manipulation in the M100, M130, or both, and exclusively in the deaf group. If the performance enhancement predicted for the deaf-signer group is subserved by changes in both domain-general and domain-specific visual cortical regions, the effects of the sign configuration manipulation are expected in all three evoked responses in deaf-signer group. Finally, we examined whether M400 responses differentiated deaf signers from sign-naive hearing subjects. Such effect may be expected based upon prior ERP studies examining lexical processing in ASL (Gutierrez et al., 2012)

We recorded the magnetoencephalogram from our participants while they were performing the discrimination task between possible and impossible signs. Event-related fields (ERFs) were computed from the onset of stimulus presentation. A series of topographic permutation tests was performed on the ERF data. Topographic tests provide a summary of the strength of a particular experimental manipulation over the whole sensor space, which, unlike individual sensors, can be meaningfully compared between groups of participants whose head position is not standardised in MEG recordings. Three types of topographic analyses were performed: simple topographic tests (Karniski, Blair, & Snider, 1994) and cluster tests (Maris & Oostenveld, 2007), sensitive to changes in both the amplitude and the pattern of activity of the underlying brain generators, and topographic shape tests (Desmedt & Chalklin, 1989; Greenblatt & Pflieger, 2004; Tian & Huber, 2008), sensitive only to the latter.

Methods

Participants

Thirteen native English speakers (five women) from the University of Maryland, College Park community participated in the hearing group (mean age: 22, age range: 18–31). Three additional subjects were excluded from this group due to failure to comply with the task (n = 1), and excessive blinking and motion artefacts (n = 2). Participants in the hearing group reported no history of hearing problems. Thirteen deaf native ASL signers (eight women) from the Gallaudet University community participated in the signer group (mean age: 25, range: 20–34). We considered ASL signers native if they had been exposed to ASL from birth by at least one parent who was a fluent ASL signer. All participants were right-handed, had normal or corrected–to–normal vision, and no reported history of language disorders or mental illness. Participants gave their written informed consent and were paid for their participation.

Materials

The stimuli were 150 greyscale still frames extracted from videos of ASL signs recorded by two (one male and one female) native ASL users (75 signs each), corresponding to the first frame where lexical recognition was unambiguous (Figure 1(a)). Our individual sign emblems were produced in citation form with minimal facial expression. Our behavioural responses showed no indication that one signer's images were recognised differently from another, and therefore all responses were collapsed across signers. During stimulus development, two native signers were asked to select individual frames from videos that would faithfully represent a known ASL sign, thus we selected still frames whose signs could be recognised in the absence of movements. Perusal of text-based sign language dictionaries will reveal that this is a common manner for displaying and conveying otherwise moving stimuli. Half of the images were edited (Adobe Photoshop 9.0) such that the position of either the arms or the hands was inverted (left became right and right became left), creating an impossible human body configuration. Great care was taken to ensure that the edited pictures did not introduce extraneous artefacts. The use of greyscale images aided in the construction of the novel stimuli (Figure 1(b)).

Procedure

The experiment was conducted in one of the blocks of a two-block experiment, where the other block was a separate face perception study reported in the supplementary material for this article. The order of blocks was randomised across participants. In the gesture categorisation block, participants were asked to indicate, by button-press, whether a static sign picture was an anatomically possible or impossible gesture, a task that does not require linguistic knowledge. Each gesture was presented once. The order of trials was randomised for each participant, and each trial started with the presentation of a fixation point (“+”) for 500 ms followed by the presentation of a gesture, which remained until the participant responded or the time out limit of 3500 ms was reached. Participants performed the task while comfortably laying supine in a bed in the magnetically shielded room. Stimuli were presented on a rear-projection screen inside the room.

MEG recordings

MEG data were recorded using a 160-channel axial gradiometer system (KIT, Kanazawa, Japan). Data were acquired continuously, with bandwidth between DC-200 Hz and sampling rate of 1 kHz. Prior to offline averaging, noise reduction was achieved using a multi-time-shift PCA filter (de Cheveigné & Simon, 2007). Individual 800 ms epochs (starting 100 ms prior to stimulus onset) were visually inspected. The epochs were removed from the analysis if they contained blinks, saccades, and other recording artefacts. In addition to the exclusion of artefact-contaminated trials, incorrect responses and trials where subjects failed to respond were also excluded from both behavioural and MEG data analysis, resulting in the exclusion of 20% of epochs from the signer group and 29% of the epochs from the hearing group. Following averaging, the data were baseline corrected (using 100 ms pre-stimulus interval) and band-pass filtered (.03–40 Hz).

Behavioural data analysis

Discrimination accuracy in the behavioural task was assessed using signal detection methods (d' scores). To correct for Hit Rates of 1, these were substituted by (1–.5)/N, where N is the number of observations. Conversely, False Alarm Rates of 0 were substituted by .5/N (Kadlec, 1999)

MEG data analysis

For the MEG analyses, the identification of the early evoked responses of interest (M100, M130, and M190) was based on windows around the peaks of the grand-average root mean square (RMS) waveforms of each condition within each group. This strategy was chosen over the more traditional method of selecting a fixed temporal window applicable to all conditions because there were large differences in the timing of the RMS peaks across groups (Table 1 and Figure 3). Due to their short time courses, a window of 30 ms was used for the M100 and M130 responses, while a longer window of 100 ms was chosen for the more sustained M190. The M400 response was quantified using a canonical 200 ms window between 300 and 500 ms (e.g. Almeida & Poeppel, 2013; Lau et al., 2009). The cosine similarity between the topography of the peak of each early evoked response and the other time points within their respective temporal windows was larger than .95, indicating that the windows comprised stable topographic patterns.

Table 1.

Condition-specific temporal windows (in ms) based on the peak of the grand-average RMS waveform for the first three evoked responses.

	M100 (30 ms)		M130 (30 ms)		M190 (100 ms)		M400 (200 ms)
	Possible	Impossible	Possible	Impossible	Possible	Impossible	Possible	Impossible
deaf	69–99	74–104	125–155	116–146	202–302	183–283	300–500	300–500
hearing	91–121	86–116	140–170	146–176	213–313	203–303	300–500	300–500

Note: The sensor space topographies at the time points within each window were all highly similar (>.95 cosine similarity) to the topography corresponding to the peak RMS that defined each window.

Figure 3.

(Top) Grand-averaged neuromagnetic fields (dark lines) for each condition, plotted with their RMS (red lines). Coloured bands indicate the temporal windows used in the analyses (green for M100, orange for M130, blue for M190 and pink for M400). (Bottom) Thin-plate spline interpolation of the four average event-related fields (ERFs: M100, M130, M190, and M400) for each condition in each group.

Permutation tests of planned comparisons

We tested the planned within-group comparisons for gesture type (possible and impossible) in all of the relevant evoked responses (M100, M130, M190, and M400) using three types of topographic analyses: simple topographic tests (Karniski et al., 1994), cluster tests (Maris & Oostenveld, 2007), and topographic shape tests (Desmedt & Chalklin, 1989; Greenblatt & Pflieger, 2004; Tian & Huber, 2008). Both the simple topographic test and the cluster test are sensitive to changes in the amplitude and/or the pattern of activity of the underlying brain generators, while the topographic shape test is sensitive only to the latter, and can thus help narrow down the interpretation of the former two tests.

Simple topographic tests (Karniski et al., 1994) are computed by performing t-tests over the whole sensor space followed by the summing of their square values in order to obtain a summary statistic (Tsum2) that reflects the strength of the experimental treatment effect over the sensor space. A permutation test using Tsum2 can then be performed across conditions to derive its empirical distribution under the null hypothesis of no treatment effect, called the reference distribution. A p-value can then be computed by calculating the proportion of results in the reference distribution that is equal to or more extreme than the Tsum2 observed in the data; if lower than .05, the result can be considered to be statistically significant.

The cluster test is a modification of the simple topographic test that is potentially more powerful (Maris & Oostenveld, 2007): First, a t-test is performed for each MEG sensor. Second, a cluster-forming t-statistic threshold is set, and clusters of at least three neighbouring channels with supra-threshold t-statistics are summarised by their size – the number of channels within the cluster – and their mass – the sum of the absolute t-values within the cluster. The choice of the t-statistic threshold is often arbitrary, and in our case it was set for all time windows to |t(12)| = 1.58 (equivalent to p = .07) for purely practical reasons: that was the lowest threshold with which clusters of the minimum required size (n ≥ 3) were found in every time window for the hearing group; a lower threshold would therefore precluded us from performing the cluster test, as no clusters would have been observed in the data from the hearing group. Third, in our particular implementation of the cluster test, the two cluster-level summary statistics (size and mass) are combined into a single summary statistic using the non-parametric combination methodology (Hayasaka & Nichols, 2004; Pesarin, 2001). The results reported here were based on the Fisher combining function, but the use of the Tippett function led to the same conclusions regarding statistical significance. The logic of the permutation test from this point onwards is identical to the simple topographical test but relies on the combined cluster statistic instead of Tsum2. Furthermore, since the early visual responses are known to arise from bilateral dipolar sources (Liu, Harris, & Kanwisher, 2002) that are easily separable in the MEG sensor space, only the two largest clusters were considered for each time window, if more than two clusters were found, as these would be the most likely to reflect the relevant bilateral ERF activity (Figures 3 and 4). In the specific case of the M100, only one cluster was found for the hearing group, and therefore the cluster test included only the largest cluster found for the deaf group.

Figure 4.

Results of cluster tests for the M100 and M130. Head plots represent the distribution of p-values from t-tests performed in individual channels for the gesture-type comparison (possible and impossible). A cluster-forming threshold of |t(12)| = 1.58 (equivalent to p = .07) was used across all conditions, and the channels comprising the observed clusters are marked in red. As can be seen, the clusters in the deaf group are much larger both in size and in mass. Waveforms represent the RMS of activity in all channels present in the clusters. The larger amplitudes of the waveforms of hearing group are due to the much smaller number of channels being averaged. Grey bars represent the temporal windows used in the analysis. Statistically significant cluster comparisons (combining cluster size and mass) are marked with an asterisk in the cluster waveforms.

The topographic shape test (Desmedt & Chalklin, 1989; Greenblatt & Pflieger, 2004; Tian & Huber, 2008) is selectively sensitive to changes in the number, location, or orientation of the underlying cortical sources. In other words, if two experimental conditions elicit solely a change in the strength of activation of an otherwise identical source configuration, the topographic shape test will not detect it. Therefore, the topographic shape test can aid in refining the interpretation of the results of the other two topographical tests, which cannot distinguish a simple change in the strength of activation of a common source configuration from a change in the actual configuration of these sources. The results reported here for the topographic shape test were obtained adopting the cosine similarity metric to compare the shapes of the topographies across gesture type (possible and impossible), but tests using the spatial correlation coefficient yielded identical results.

Permutation tests of interaction

Because there are debates about the correct test for the interaction in the permutation test framework (Edgington & Onghena, 2007), we decided to base our tests for the interaction between gesture type (possible and impossible) and group (deaf and hearing) on the difference of Tsum2 and the cluster statistics (size and mass) between the groups (with the two difference cluster statistics being combined into a single summary statistic using the non-parametric combination methodology; cf. Hayasaka & Nichols, 2004; Pesarin, 2001; see Bemis & Pykkänen, 2011 for a similar formulation of the test of interaction). We chose to perform two different tests because there are no direct comparative studies establishing the relative power of tests of interaction based on Tsum2 and those based on cluster statistics. Crucially, the permutation test for the interaction required the reshuffling of subjects across groups, followed by conditions within subjects. For each complete data reshuffling, a simple topographical test and a cluster test were performed for each group, and the difference between the tests of each group was calculated. A reference distribution for the two difference statistics (ΔTsum2 and Δcombined-cluster-statistic) was created with 10,000 data permutations, and the p-values were calculated by computing the proportion of data permutations that yielded interaction statistics equal to or more extreme than the ones observed in the data.

Results

Behavioural results

As predicted, deaf signers were more accurate (mean d'deaf = 3.9) than the hearing participants (mean d'hearing = 2.2) at discriminating possible signs from impossible signs (t(15.89) = 4.484, p < .001, with Welch's correction for heterogeneous variance, see Figure 1(d)). A 2 (gesture type: possible, impossible) × 2 (group: deaf, hearing) repeated measures ANOVA of the reaction times also confirmed the prediction that the deaf group would be overall faster than the hearing group in the discrimination task: deaf participants were in general 281 ms faster than the participants in the hearing group (F(1,24) = 16.923, p < .001; Figure 1(c)). A significant main effect of gesture type was also observed, with responses to possible signs gestures being in average 220 ms faster than responses to impossible signs (F(1,24) = 38.9761, p < .001). The interaction between gesture type and group was not significant, and planned comparisons within each group revealed faster responses to possible signs compared to impossible signs by both deaf signers (t(12) = 4.798, p < .001) and hearing participants (t(12) = 4.002, p = .002; Figure 2).

Figure 2.

(a) Average reaction time (RT) of the five participants with worst d′ scores (all hearing) (left side), the four hearing participants with highest d′ scores (middle), and four deaf participants with matched d′ scores to the former (right). (b) Linear trend lines of RT based on discrimination ability (d′ scores), with 95% confidence bands for each Group (deaf, hearing) and Gesture Type (possible, impossible) combination. The subjects marked in red are the four hearing participants with highest d′ scores and their matched d′ scores deaf counterparts.

Contrary to the deaf group, who was very homogeneous in their discrimination performance, the hearing group exhibited significant variation (Figure 1(d)). This raises the possibility that the observed difference between groups may not be due to language experience per se, but rather to a significantly worse performance by a subset of the hearing participants. To assess this, we matched the only four hearing participants who had d' scores in the range of the deaf group (i.e. d' >= 3) with the deaf participants who had the closest d' scores to each one of them. Despite being matched by their discrimination abilities (mean d'deaf = 3.51 and mean d'hearing = 3.59), the deaf participants were still substantially faster than their hearing counterparts (184 ms for possible gestures and 143 ms for impossible gestures; see Figure 2), suggesting that deaf signer's language experience, and not just discrimination performance, is the basis for these results.

MEG results

The Tsum2-based test revealed a significant interaction between group (deaf and hearing) and sign type (possible and impossible) for the M100 (p = .001), and a marginal interaction for the M130 (p = .1), but no interaction for the M190 (p = .48) nor M400 (p = .21; see Table 2 for full results). The cluster-based test seemed to be less powerful and revealed only a marginal interaction between group (deaf and hearing) and sign type (possible and impossible) for the M100 (p = .06), and no interaction for the M130 (p = .34), M190 (p = .67), and M400 (p = .56).

Table 2.

Results from the topographic tests.

Group (deaf vs. hearing) × gesture type (possible vs. impossible)
Interaction statistic	M100		M130		M190		M400
ΔTsum2	195	(.001)*	68	(.1)•	3	(.48)	35	(.21)
ΔCluster 1 (size, mass)	14, 35	(.06)•	3, 6.9	(.34)	-3, -7.1	(.67)	-1, -0.3	(.56)
ΔCluster 2 (size, mass)	11, 28.8		5, 13.2		0, 0		1, 2.7

Notes: p-Values are in parentheses. Significant p-values are marked by an asterisk (*), and marginal p-values by a black dot (•). Tsum2 is the sum of squared t-tests for every sensor. Size indicates the number of channels within each cluster. Mass indicates the sum of the absolute t-values within each cluster. The topographic shape dissimilarity metric ranges from 0 (equal shape) to 2 (inverted shape). ΔTsum2, ΔCluster 1 and ΔCluster 2 are the differences between the respective gesture-type tests of each group (deaf and hearing).

Planned comparisons for all evoked responses using the simple topographic test within each group established that only the deaf participants showed effects of gesture type, which was marginal in the M100 (p = .07) and marginal in the M130 (p = .1; Figure 4). In contrast with the pattern found in the interaction test, the cluster test was more powerful in the planned comparisons, establishing significant effects of gesture type for the deaf group in the M100 (p = .03) and a marginal effect in the M130 (p = .1). Finally, the topographic shape test within each group established that only the deaf participants showed effects of gesture type, which was significant only in the M130 (p = .044; Figure 3).

Discussion

Our results demonstrate a remarkable enhancement in the visual discrimination abilities of deaf signers compared to normal-hearing monolingual English-speaking participants: Deaf participants are both more accurate and faster than their normal-hearing counterparts at discriminating possible signs compared to biologically impossible sign configurations (Figure 1). This perceptual enhancement co-occurred with group-specific differences in the earliest cortical ERF visual responses (M100 and M130, see Figures 3 and 4), with no difference found in latter M190 and M400 responses. Because these differences were found only in the deaf-signer group, these results suggest that (1) the M100 and M130 effects found in the deaf group are not solely driven by low-level physical stimulus properties and (2) extensive sign-language exposure can lead to the tuning of very early, putatively domain-general visual areas for complex human form processing. The M100 amplitude modulation in the deaf group did not co-occur with a topographic shape difference (Figures 3 and 4), which probably reflects a change in the strength of the activity in a common generator pattern, plausibly in the primary visual cortex (Ishizu et al., 2010; Taylor et al., 2010). Consistent with this observation, recent voxel-based morphometry studies have reported a greater volume of cortex surrounding the calcarine sulcus in congenitally deaf signers relative to hearing non-signers (Allen, Emmorey, Bruss, & Damasio, 2013; Lepore et al., 2010).

In contrast, the marginally significant M130 amplitude modulation did co-occur with a significant topographic shape difference (Figures 3 and 4), suggesting a qualitative change in the generator patterns between the two conditions by 130 ms in the visual processing stream, plausibly in the occipitotemporal cortex (Tarkiainen et al., 2002). Studies of body posture violations in normally hearing subjects have reported increased activity in extrastriate visual regions in response to distorted and impossible human postures relative to natural postures (Costantini et al., 2005. Cross, Mackie, Wolford, & de C. Hamilton, 2010). Costantini et al. (2005) report a strongly right-hemisphere asymmetrical response to impossible finger movements in the temporo-occipital region located near the EBA and suggest this greater activity may reflect the neural matching of visual and somatic information concerning observed and inner simulated actions. Avikainen, Liuhanen, Schürmann, and Hari (2003) used MEG to explore neural responses to normal and distorted figure postures. This study reported no significant differences in the right and left occipital regions of interest in a 100–200 ms time window; however, between 260 and 700 ms, extrastriate occipital areas of both hemispheres were activated more strongly by distorted than natural finger postures. These authors attributed this latter activation to early top-down effects of emotional valence on the processing of unusual hand shapes in the extrastriate visual cortex. In contrast to the current study, in which we asked for speeded plausibility judgements, the subjects in the Avikainen et al. (2003) study performed either an n-back task or were asked to intermittently reproduce possible hand postures. This difference in task design makes direct comparison of these paradigms difficult.

One question that arises is whether these effects are specific to sign language processing or rather reflects some more general perceptual differences in deaf signers. In an effort to ascertain the specificity of the sign language effects, we investigated deaf and hearing MEG response to upright and inverted faces and examined M100 and M170 responses (see supplementary materials online). Both groups exhibited similar patterns in the evoked responses in the face-viewing task: no effect on the M100 followed by M170 latency shifts for the face-inversion manipulation (Itier, Herdman, George, Cheyne, & Taylor, 2006). The data suggest that the current results are not simply general enhanced perceptual effects from any natural human forms but are likely specific to body poses.

We consider two interpretations of our results. The first involves differential processing related to the lexical status of the ASL signs. The static presentation of such forms may have allowed for ultra-rapid lexical recognition of the signs by the deaf group, in turn leading to faster categorisation of signs as biologically possible or impossible. Although little is known about the speed of lexical access from static presentations of ASL signs, extensive reading experiments with hearing subjects place lexical access only after the first 200 ms, considerably later than the time windows of the M100 and M130 modulations observed in the deaf group. However, this view cannot be ruled out and, if correct, would suggest that ASL lexical recognition can occur on the basis of a single static image in less than 150 ms. Indeed, signed language may be unique in their ability for a single image to engender the lexical recognition of a word/sign form. A recent ERP study provides evidence for very early effects of sign language processing. Gutierrez et al. (2012) recorded ERPs to critical signs that were consistent with the sentence context (baseline) (e.g. Last-night I had a drink of WINE, it was delicious!) or were semantically (s) and phonologically (p) consistent with the expected baseline target (e.g. BEER, +s, +p). Additionally, items could share the expected articulatory location of the baseline sign but are semantically unrelated (e.g. CANDY, -s, +p), or were semantically related to the baseline form but had a phonological form that differed from the expected sign target (e.g. MILK, +s, -p). Finally, a fifth condition tested words that were completely unrelated to the sentence context (e.g. JUDGE, -s, -p). Most pertinent to the present study was the finding that ERP responses showed evidence for an early effect of semantic pre-activation of plausible candidates (i.e. (BEER, +s, +p), and (MILK, +s –p) at 150–250 ms. Similar semantic pre-activation has been observed in some visual word recognition studies (Dambacher, Rolfs, Göllner, Kliegl, & Jacobs, 2009; Kim & Lai, 2012). As proposed by Kim and Lai (2012), the context may generate an early “blurry” pre-activation of the upcoming visual lexeme that is sensitive to small deviations from the prediction. The static images of signs used in the present experiment may share properties with orthographic forms, for example the simultaneous availability of lexical information. However, in contrast to the above-mentioned studies, in the present experiment there is no sentential contextual information. Thus, if some type of lexical semantic pre-activation is operative, it must be based on some highly general expectation of sign well-formedness that is established in the context of single signs.

While it is possible that early lexical effects may underlie the processing for possible and impossible forms, the lack of robust differences in later M190 and, especially, the M400 response suggests that the lexical effects may be quite limited. This may be related to the fact that in the present experiment lexical effects are implicit. Recall the response required of subjects was to determine whether the signs were possible or impossible, and not an explicit judgment of lexical status. Previous work using implicit studies of lexical processing in ASL have reported effects weaker than those that might be expected from explicit judgments (Corina & Grosvald, 2012).

The reaction time data for making the plausibility were on the average of 746 ms in the deaf participants. This would suggest that there may be additional cognitive processes that are not being fully resolved by the ERF data. Nevertheless, the fact that native signers were 281 ms faster than the hearing subjects indicates that these early effects are reflected in the processing speed. One possibility is that these post-sensory but pre-behavioural processes are poorly time-locked to stimulus presentation, and thus cannot be properly resolved in ERF analyses. Future exploratory work on the time–frequency representation of the epochs may help uncover differential induced responses.

A second, more provocative interpretation is that signers, due to extensive experience in analysing gestures for linguistic communication, have developed a different, visually driven internal forward model for human gestures used in the service of linguistic communication, recruiting primary visual areas. In the context of reading, highly predicted linguistic information has been shown to modulate early, putatively general visual areas (Dikker, Rabagliati, Farmer, & Pylkkänen, 2010), and sensitivity of V1 cells due to perceptual learning has been observed even in single cell recordings (Gilbert, Sigman, & Crist, 2001; Hua et al., 2010; Li, Piëch, & Gilbert, 2004), lending plausibility to this hypothesis.

Recent studies further support our findings. Deaf signers show enhanced recognition of dynamically presented signs and, compared to normal-hearing listeners, are approximately 100 ms faster in the recognition of non-language gestures (Corina & Grosvald, 2012). Positron Emission Tomography (PET) imaging studies (Corina et al., 2007; Emmorey, Xu, Gannon, Goldin-Meadow, & Braun, 2010) have reported that deaf signers show enhanced activation in the inferior temporal visual area when processing non-linguistic actions, which differs significantly from the frontal–parietal activation observed in hearing non-signers. In light of the groups differences observed for the M130 effects in the current study, it is noteworthy that in these PET studies congenitally deaf signers showed greater bilateral activation in occipital–temporal regions to human actions relative to ASL signs, while hearing non-signers showed more restricted right-hemisphere activation for this same contrast (Corina et al., 2007).

An important question that we cannot address in the current study is whether these effects are due to sign language experience or deafness per se. Congenital deafness has been shown to modify allocation of peripheral attention (Bavelier & Neville, 2002; Dye, Hauser, & Bavelier, 2009; Neville & Lawson, 1987). It is possible that the enhanced discrimination reported here reflects visual attention differences between deaf and hearing subjects. Future studies that include hearing individuals whose native language is American Sign Language may help further determine the locus of the present effects.

Conclusion

We report compelling behavioural evidence for enhanced sensitivity to American Sign Language signs in deaf signers that co-occurs with specific changes in the two earliest evoked responses elicited by the visual presentation of human bodies (M100 and M130). This neurophysiological effect was found in the deaf group alone. We suggest that these results are consistent with the hypothesis that deaf signers make use of general-purpose visual recognition routines in the interpretation of sign language, but that early visual processing may be modified by linguistic experience resulting in a more efficient coding of complex articulatory body postures. These results suggest that linguistic experience and/or early profound auditory deprivation shapes early neuronal mechanisms that underlie the analysis of visual communication, likely on the basis of highly articulated, predictive internal models of gesture and language processing. These data indicate that the perceptual tuning that underlies the discrimination of language and non-language forms is not limited to spoken languages but also extends to languages expressed in the visual modality.

Click here for additional data file.