Dynamics of Speech Perception in the Auditory-Visual Mode: An Empirical Evidence for the Management of Auditory Neuropathy Spectrum Disorders.

BACKGROUND AND OBJECTIVES: The present study probed into the relative and combined contribution of auditory and visual modalities in the speech perception of individuals with auditory neuropathy spectrum disorders (ANSD). Specifically, the identification scores of consonantvowel (CV) syllables, visual enhancement (VE), and auditory enhancement in different signal to noise ratios (SNRs) were compared with that of the control group. Subjects and.
METHODS: The study used a repeated measure standard group comparison research design. Two groups of individuals in the age range of 16 to 35 years participated in the study. The clinical group included 35 participants diagnosed as ANSD, while the control group had 35 age and gender matched individuals with typical auditory abilities. The participants were assessed for CV syllable identification in auditory only (A), visual only (V), and auditory-visual (AV) modalities. The syllables were presented in quiet and at 0 dB SNR.
RESULTS: The speech identification score was maximum in AV condition followed by A-condition and least in V condition. This was true in both the groups. The individuals with ANSD were able to make better use of visual cues than the control group, as evident in the VE score.
CONCLUSIONS: The dynamics of speech perception in the AV mode is different between ANSD and control. There is definite benefit of auditory as well as visual cues to individuals with ANSD, suggesting the need to facilitate both the modalities as part of the audiological rehabilitation. Future studies can focus on independently facilitating the two modalities and testing the benefits in the AV mode of speech perception in individuals with ANSD.

Introduction

Poor speech perception is one of the cardinal features of typical individuals with auditory neuropathy spectrum disorders (ANSD) [1,2]. Perception of speech in these individuals is compromised more in adverse listening conditions, particularly in the presence of background noise [1,3,4]. Earlier studies in the literature have attributed the auditory perceptual deficits found in individuals with ANSD to the disrupted temporal processing [4-8]. It was found that the poor speech identification of these individuals was related to their temporal processing deficits assessed in terms of just noticeable difference and temporal modulation transfer function [6].

In instances where speech perception through auditory modality is compromised, some of the missing information is compensated by visual cues [9,10]. Earlier studies in individuals with normal hearing have shown that auditory-visual (AV) mode facilitates better speech perception compared to auditory mode in the degraded listening conditions [11-13]. The benefit drawn from visual modality for speech perception is also known to be dependent on the speech to noise ratio in the auditory mode [14,15]. Understanding the dynamics of AV integration in persons with hearing impairment is of significance, as their auditory input is compromised which in turn hinders optimum speech perception. Grant, et al. [16] assessed integration abilities in adult hearing-impaired listeners using a variety of AV integration measures and reported a significant AV benefit even with extremely reduced auditory signal. The improvement in AV mode was seen not only in degraded acoustic environment but also in clear speech. Similar results have been reported in children with hearing impairment [17].

Individuals with ANSD are reported to compensate for their speech perception difficulty by focusing primarily on visual cues. Ramirez and Mann [18] found no significant difference in scores between visual alone and AV condition in identifying the place of articulation indicating that individuals with ANSD rely solely on visual cues in identifying place of articulation, except at high levels of noise. At high levels of noise, individuals with ANSD extracted more information about place of articulation in AV mode compared to visual mode.

Even though the perceptual deficits in individuals with ANSD and its underlying mechanisms are well understood, management of their speech perception difficulty remains a challenge for audiologists. Individuals with ANSD get limited benefit with hearing aids as these devices do not compensate for their temporal processing deficits. Although frequency modulated (FM) devices are reported to be more beneficial than conventional hearing aids [19], utility of FM devices is limited to only a few listening conditions. Similarly, despite reports of improved speech perception in ANSD through acoustic enhancements [8], improvement is not appreciable in those with poor speech identification scores. Therefore, in instances of limited benefit even with enhanced auditory modality, cues from visual modality serve to supplement and thereby enhance speech perception. The present study aimed to investigate relative benefits of auditory and visual cues in individuals with ANSD for the perception of speech.

Furthermore, considering that the auditory input is distorted in ANSD, there could be mismatch in the input from the auditory and visual modalities during the process of AV integration in these individuals. This could possibly lead to inherent incongruence between the two modalities which in turn could negatively influence speech perception. Therefore, dynamics of AV speech perception in individuals with ANSD needs to be evidenced before advising it as a rehabilitative option. Hence the study also aimed to document the dynamics of AV speech perception in individuals with ANSD.

Subjects and Methods

The study incorporated a standard group repeated measure design to test the null hypothesis that there is no significant difference in the AV integration during speech perception in individuals with ANSD compared to participants with normal hearing.

Participants

Two groups of participants, namely experimental group and control group, were included in the study. The experimental group had 35 individuals with ANSD (17 male and 18 female) in the age range of 16 to 35 years. Their hearing sensitivity ranged from normal hearing to sensorineural hearing loss of moderate degree [with pure tone average (PTA) of 500, 1,000, 2,000, and 4,000 Hz of up to 55 dB HL]. Majority of the individuals with ANSD had predominant low frequency hearing loss with normal or near normal hearing thresholds at higher frequencies. Word identification scores ranged from 0% to 100% across the participants. The minimum duration of hearing loss was one year and the onset of ANSD was postlingual (around 13 years) in all of them. They had normal middle ear functioning as on immittance evaluation. All of them had transient otoacoustic emissions of amplitude more than 6 dB signal to noise ratio (SNR) indicating normal outer hair cell function. Auditory bainstem responses (ABRs) were absent in all participants indicating retrocochlear dysfunction. Further, neurological examination in these individuals revealed diagnosis of primary auditory neuropathy and ruled out the presence of space occupying lesions. The control group had age and gender matched participants with normal hearing. They also had normal speech-language development and abilities. Their hearing thresholds were within 15 dB HL at octave frequencies between 250 Hz and 8 kHz [20].

Participants in both groups had normal visual acuity (6/6) in a Snellen chart. They were native speakers of Kannada (language predominantly used in the state of Karnataka, India). It was ensured that all participants were literate and were able to read non-meaningful consonant-vowel (CV) syllables written in Kannada. All participants signed an informed consent form before conducting the experiment. The procedures were approved by the ethical committee of All India Institute of Speech and Hearing [21].

Test stimuli

Six CV syllables were used as target test stimuli. Vowel /a/ was used in all the syllables whereas the consonants included were velar /k/, retroflex /ṭ/, and bilabial /p/, and their voiced counterparts /g/, /d/, and /b/. These monosyllables were non-meaningful in Kannada language.

Audio-video recording and editing of test stimuli

The test stimuli were both audio and video recorded in an audiometric room. The audio recording was carried out using a unidirectional microphone connected to the computer with adobe audition software (version 3, Adobe Systems Incorporated, San Jose, CA, USA). Each of the six syllables was spoken by five adult males who were native speakers of Kannada and had clinically normal speech. The syllables were digitized at a sampling frequency of 44,100 Hz and 16 bit digitization. The recorded syllables were normalized in order to minimize differences in the stimulus energy. They were then ensured for their clarity through quality judgement by 10 native speakers. The best of the five samples were chosen for the study.

The video recording of the test stimuli was done by a professional videographer using a high definition camera (Recording frame rate: AVCHD FX (24 Mbps) 1,920×1,080/50i/16:9) in an audiometric room with appropriate lighting. A white screen was used as the background. Five adult native Kannada speaking males with clinically normal speech were chosen for generating the video stimuli. Prior to the video recording, the speakers were made to listen to the audio sample several times. They were instructed to produce the CVs in the same rate as in the audio samples and also to minimize eye blinks and head movements while recording. Out of the five sets of samples, the sample best in terms of articulation and video clarity was selected for the study. The AV samples were prepared by dubbing the recorded audio stimuli with the corresponding visual stimuli using video pad editor software (version 4.2.2, NCH Software, Canberra, Australia). The procedure followed is similar to that in the earlier studies of AV speech perception [12,18].

The recorded AV stimuli were presented to five experienced audiologists to rate the quality of video and also the synchrony of the audio video sample on a five point rating scale. The audio video sample which was rated as ‘good’ by all the five audiologists was considered for the study. The representative static images derived from the video recording of production of /pa/ are shown in Fig. 1.

Fig. 1.

Representative picture sequence showing the production of /pa/.

Procedure for the assessment of speech perception

Speech perception of nonsense CV syllables was assessed through closed set listening in three modalities; auditory only (A), visual only (V), and AV modes. In the A and AV modalities, perception was tested in quiet, and also in the presence of noise at 0 dB SNR. On the other hand, perception was tested only in quiet condition in the V mode. The procedure of testing speech perception in the V mode only in quiet whereas different SNRs were used to test the auditory and AV speech perception is in line with the earlier studies [12,13,18,22].

Visual stimuli were presented through Paradigm software (version 2.5.0.68, Perception Research System Inc., Lawrence, KS, USA) using Samsung 21 inch LCD screen, kept at 0° azimuth of the participant. The LCD screen was connected to a Sony Vaio Laptop computer via a VGA cable.

The audio stimuli were presented at the most comfortable level for each participant through a loud speaker kept at 45° azimuth. The participant was seated one meter away from the LCD screen (placed in front of the participant at eye level) and also from the speaker. The laptop was connected to the audiometer to route the audio signal to the sound field speaker and also to the LCD screen to direct the visual stimuli.

Each syllable in each SNR and in each modality was presented 10 times. Considering that there were 6 syllables, 3 modalities and 2 SNRs (except for V in quiet only), the total number of presentation was 300 for each participant. The participants were instructed about the stimulus type and the response task. Kannada script of the six CV syllables (/pa/, /ṭa/, /ka/, /ba/, /da/, and /ga/) was displayed on the screen. They were asked to click the corresponding CV, using the left click button of the mouse. Participants were given a two-minute break after each stimulus condition (60 presentations) and the entire testing was completed in a single session. The order of presentation was randomized across participants.

Scoring

Each correct response was provided a score of one, and an incorrect response was given a score of zero. The total number of correct responses of each participant in each condition was noted as the raw score. Separate total scores were obtained for each stimulus condition. The difference between the scores in AV mode and A mode was taken as the visual enhancement (VE) score [VE=(AV-A)]. Similarly, the difference between the scores in AV mode and V mode was considered as the auditory enhancement (AE) score [AE=(AV-V)]. VE and AE scores were calculated separately for each participant in each SNR.

Results

Effect of modality on speech identification scores in ANSD and control groups

The total identification scores of each participant in ANSD and control group in quiet and 0 dB SNR conditions are shown in Fig. 2 and 3, respectively. The mean and standard deviation (SD) of syllable identification scores of control and experimental groups in all conditions are given in Table 1. Mean identification scores in the control group was higher than ANSD group in all the three modalities.

Fig. 2.

Individual identification scores of participants in ANSD and control groups in the A, AV, and V modalities in quiet condition. Maximum score is 60. ANSD: auditory neuropathy spectrum disorders, A: auditory only, AV: auditory-visual, V: visual only.

Fig. 3.

Individual identification scores of participants in ANSD and control groups in the A and AV modalities in 0 dB SNR condition. Maximum score is 60. ANSD: auditory neuropathy spectrum disorders, A: auditory only, AV: auditory-visual, SNR: signal to noise ratio.

Table 1.

Mean and SD of syllable identification scores (raw scores) in the two groups in the A, AV, and V modalities in quiet and 0 dB SNR conditions

SNR	Group	Modality
		A		AV		V
		Mean	SD	Mean	SD	Mean	SD
Quiet	Control	59.45	1.01	59.48	0.81	28.06	4.96
	ANSD	33.20	11.89	41.29	10.19	23.46	5.58
0 dB SNR	Control	52.05	5.91	55.71	3.21	NT
	ANSD	17.74	7.24	28.34	5.98	NT

NT-Not tested as presentation of stimulus in this condition was not part of the experimental design. Maximum score was 60 (6 CV×10 times). SD: standard deviation, ANSD: auditory neuropathy spectrum disorders, A: auditory only, AV: auditory-visual, V: visual only, SNR: signal to noise ratio, CV: consonant-vowel

The effect of stimulus modality on speech identification was tested using repeated measures analysis of variance (ANOVA) with group as between-subject factor and modality was within subject factor. This was done separately for quiet and 0 dB SNR conditions. Results of quiet condition showed a significant main effects of both modality [F(2, 136)=366.11, p<0.001] and group [F(1, 68)=165.71, p<0.01] on the identification scores. The interaction between modality and group was also found to be significant [F(2, 136)=62.89, p<0.001]. Subsequent Bonferroni adjusted multiple comparisons revealed significant differences in syllable identification scores across all three modalities (A, AV, and V).

Similarly, results of analysis in 0 dB SNR condition showed a significant main effects of modality [F(2, 136)=212.56, p<0.001] and group [F(1, 68)=503.93, p<0.01] on speech identification scores. There was also a significant interaction between the modality and group [F(2, 136)=192.99, p<0.001].

Owing to the significant interaction effect of group and modality, the effect of modality was further assessed separately in the two groups using repeated measures ANOVA. This was done separately in quiet and 0 dB SNR conditions. There was a significant main effect of modality in quiet [F(2, 68)= 1367.17, p<0.01] and 0 dB SNR [F(2, 68)=448.19, p<0.01] conditions in control as well as ANSD [F(2, 68)=47.87, p<0.01 in quiet & F(2, 68)=37.61, p<0.01 in noise] groups. On Bonferroni adjusted multiple comparisons, it was found that in quiet, A and AV modality yielded comparable scores in the control group (p>0.05). The mean scores in the V modality was significantly lower than both A and AV modalities (p<0.01). On the other hand, there were significant differences across all three modalities (p<0.01) in the group of participants with normal hearing when tested at 0 dB SNR, and in the ANSD group when tested in the quiet and 0 dB SNR conditions.

Comparison of AE and VE scores between ANSD and control groups

Mean and SD of AE and VE scores in different stimulus conditions for control group and experimental group are given in the Table 2. One-way repeated measures ANOVA was carried out to compare between mean AE and VE scores with group as between-subject factor. This was done separately for quiet and 0 dB SNR conditions. Results of analysis in quiet condition showed a significant main effect of type of enhancement [F(1, 68)=312.15, p<0.01] and significant interaction between type of enhancement and group [F(1, 68)=86.49, p<0.01]. Similarly, at 0 dB SNR, there was a significant main effect of type of enhancement [F(1, 68)=101.94, p<0.01] and significant interaction between type of enhancement and group [F(1, 68)=269.19, p<0.00]. There was a significant main effect of group in quiet [F(1, 68)=4.75, p<0.05] as well as 0 dB SNR [F(1, 68)=48.34, p<0.01].

Table 2.

Mean and SD (in parenthesis) of AE and VE in the two groups of participants, in quiet and 0 dB SNR

Stimulus conditions	Control group	ANSD group
AE	31.46 (4.88)	17.83 (11.74)
AE (0 dB SNR)	27.66 (5.71)	4.89 (7.37)
VE	0.06 (1.30)	8.09 (6.89)
VE (0 dB SNR)	3.66 (4.57)	10.60 (6.34)

Max score is 60. SD: standard deviation, AE: auditory enhancement, VE: visual enhancement, SNR: signal to noise ratio,ANSD: auditory neuropathy spectrum disorders

Owing to significant interaction of group, AE and VE were compared using paired t test, separately for control and ANSD groups. There was a significant difference between mean AE and VE scores in control (t=37.01, p<0.01 in quiet, t=19.65, p<0.01 in noisy condition) as well as ANSD groups (t=4.49, p<0.01 in quiet, t=-4.20, p<0.01 in noisy condition).

Correlation of AE and VE scores with the duration of hearing loss, pure tone average and speech identification scores in ANSD

The secondary purpose of the study was to determine the relationship of AE and VE scores in the two stimulus conditions (quiet and 0 dB SNR) with the degree of hearing loss (based on PTA), reported duration of ANSD and speech identification scores (SIS). This was assessed using Spearman rank-order correlation and the results (Table 3) showed that there is no significant correlation of AE and VE scores with any of the three parameters tested (PTA, duration of hearing loss, and SIS).

Table 3.

Results of Spearman’s rank-order correlations in the ANSD (n=35)

Variable	Correlation	AE score		VE score
		Quiet	0 dB SNR	Quiet	0 dB SNR
PTA	r	0.067	-0.129	-0.024	0.181
	p	0.703	0.461	0.889	0.299
SIS	r	0.075	0.279	-0.062	0.033
	p	0.667	0.104	0.725	0.850
Duration of HL	r	0.081	0.017	-0.290	0.058
	p	0.643	0.924	0.091	0.741

PTA and SIS mentioned is the ear having better PTA and SIS and the test was done in the sound field. ANSD: auditory neuropathy spectrum disorders, AE: auditory enhancement, VE: visual enhancement, SNR: signal to noise ratio, PTA: pure tone average, SIS: speech identification scores, HL: hearing loss

Discussion

The study aimed to understand the dynamics of AV speech perception in individuals with ANSD. The results showed a definite difference in the way visual cues contributed to speech perception in the AV mode between control and ANSD individuals. The dynamics of AV speech perception was also influenced by the presence of noise.

Speech perception in the control group was observed to be similar in both A and AV modalities in the quiet condition. This could be attributed to the ceiling effect in the A mode in this condition. Nonetheless visual cues did improve speech perception in the presence of noise. These results are in agreement with the earlier studies [9-13]. In the presence of noise, weak consonantal cues are masked which in turn poses a challenge in the identification of consonants. The addition of visual cues primarily supplement place of articulation information and is therefore expected to help in auditory closure, facilitating speech perception. However, dependency of the control group was primarily on the auditory modality as evidenced by marginal VE score in this group even in the presence of noise (0 dB SNR). This is further supported by significantly higher AE scores in control group compared to ANSD group, in both quiet and 0 dB SNR conditions.

Speech perception in individuals with ANSD was poorer compared to control group, both in quiet as well as 0 dB SNR condition. Poor speech perception is the cardinal feature of ANSD attributable to impaired temporal processing [4-6]. The perception further deteriorated in the presence of noise both in A and AV modalities. The effects of noise appeared to be greater in individuals with ANSD compared to control group which is in agreement with the earlier studies [1,3,4].

The dynamics of AV speech perception was found to be different in ANSD compared to individuals with normal hearing. The addition of visual cues significantly enhanced speech perception abilities of individuals with ANSD both in quiet as well as 0 dB SNR condition. Hearing sensitivity in participants with ANSD ranged from normal hearing to moderate degree of sensorineural hearing loss. Three of them had normal hearing and six of them had minimal hearing loss. Inspection of individual data showed that even these individuals did benefit from the AV mode compared to auditory mode. Although it cannot be concluded from our finding that benefit derived has relation to temporal processing deficit in ANSD, the presence of benefit in normal hearing and minimal hearing loss suggests that ANSD individuals without significant hearing loss but with significant speech perception deficit do benefit with AV mode. This is an empirical evidence to support recommendation of AV mode of speech perception as a management strategy in ANSD.

The performance of individuals with ANSD was poorer compared to individuals with normal hearing in V mode alone. This is in agreement with earlier study and hints at deficit visual processing in individuals with ANSD [23]. In spite of lower scores in the V mode, ANSD group showed higher VE scores than the control group. This means that the addition of visual cues benefitted individuals with ANSD more than individuals with normal hearing. This finding can be taken as further support to the use of AV mode in individuals with ANSD. The VE was higher both in quiet and 0 dB SNR in ANSD compared to the control group. Within the two listening conditions, VE was higher in the presence of noise. These findings suggest that individuals with ANSD are able to make better use of visual cues compared to individuals with normal hearing, particularly in the presence of noise. Although the precise reason for this is not known, one can speculate that lower scores in the auditory mode could be one of the important influencing factors. Studies in the literature have shown that the importance of visual cues increase as the listening environment becomes more challenging [9,10,24]. Considering that perception through auditory modality is compromised in ANSD, the role of visual cues seems to be crucial. Further, the reported duration of onset of ANSD in the present study was up to 20 years. In view of the compromised auditory input, it is plausible that these individuals would have become adept in better utilizing the available visual cues as a compensatory mechanism. However, such an inference is not without a caveat. It should also be noted that ANSD group had poorer mean identification scores in the visual alone mode compared to individuals with normal hearing.

The present findings suggest that individuals with ANSD are able to utilize auditory as well as visual cues and are not dependent on only one of these modalities. The finding is in contradiction to an earlier study reporting no difference in the performance of individuals with ANSD in AV mode compared to V mode [18]. The finding of the present study is derived from data of 35 individuals with ANSD, while Ramirez and Mann [18] had reported their findings from 4 individuals with ANSD. The differences in the range of SIS across subjects and the difference in the scoring pattern would have contributed to the varied findings in the two studies. However the exact reason for this difference is to be ascertained empirically. Nonetheless given that both auditory and visual cues play a role in the speech perception abilities of ANSD, it is necessary to facilitate both the modalities to the best possible extent in these individuals. The auditory modality can be enhanced through signal enhancement strategies such as FM devices [19], companding [25] and envelope enhancement [8]. Similarly, visual modality can be boosted either through training in speech reading using standardized methods or through anticipatory compensatory strategies [26,27].

Individuals with ANSD are known to have erroneous auditory perception. In our study it was hypothesized that the erroneous auditory perception when combined with visual cues may result in McGurk like effect in their perception. The support for this notion can be drawn from the consonant confusion matrix in described in Kumar [28]. During the closed set identification of consonants, there were consonant substitutions and the substituted consonants differed from the targets in terms of place of articulation and voicing. Therefore, if an individual with ANSD individual focuses on auditory as well as visual cues during the AV mode of presentation, information from the two modalities may be perceived as incongruent, resulting in distorted perception. Enhanced perception in the AV mode observed in the current study suggests that the proposed McGurk like effect is less likely and even if present, does not influence speech perception to a large extent.

Apart from the above findings, the results also suggested that audiological characteristics such as degree of hearing loss, unaided speech identification and the duration of the condition do not relate to AE and VE. The findings were similar in quiet and 0 dB SNR. This may be due to the heterogeneity in the individual audiological profile which is typical of ANSD. These findings suggest that an individual with ANSD will benefit from the AV mode without significant influence of their degree of hearing loss, SIS, and duration of the condition.

In conclusion, the findings in the study suggest that cues from both auditory and visual modalities are important for speech perception in individuals with ANSD. The advantage of visual modality is more in the presence of noise. Therefore, attempts should be made to enhance auditory as well as visual input during the audiological rehabilitation in individuals with ANSD. The benefit derived from the AV mode is not related to the degree of hearing loss, SIS, and duration of the condition. Further, the study emphasizes the need to implement auditory signal enhancement and visual cues both independently and in unison. It also reflects on the utility of training methods such as speech reading to enhance visual cues in addition to training with acoustically enhanced signal to improve speech perception in ANSD, especially in adverse listening conditions.