Real-time feedback control of voice in cochlear implant recipients.

OBJECTIVES: To evaluate feedback-dependent vocal control in cochlear implant patients using pitch-shifted auditory feedback.
METHODS: Twenty-three CI recipients with at least 6 months of implant experience were enrolled. Vocal recordings were performed while subjects repeated the vowel /e/ and vocal signals were altered in real-time using a digital effects processor to introduce a pitch-shift, presented back to subjects using headphones. Recordings were analyzed to determine pitch changes following the pitch-shifted feedback, and results compared to the magnitude of the shift as well as patient demographics.
RESULTS: Consistent with previous results, CI patients' voices had higher pitches with their implant turned off, a change explainable by increases in vocal loudness without the CI. CI patients rapidly compensated for pitch-shifted feedback by changing their vocal pitch, but only for larger shifts. Considerable inter-subject variability was present, and weakly correlated with the duration of implant experience and implant sound thresholds.
CONCLUSIONS: CI patients, like normal hearing individuals, are capable of real-time feedback-dependent control of their vocal pitch. However, CI patients are less sensitive to small feedback changes, possibly a result of courser CI frequency precision, and may explain poorer than normal vocal control in these patients. LEVEL OF EVIDENCE: Level 3b.

INTRODUCTION

Cochlear implants have been an important advance in hearing rehabilitation, and have been extensively studied to assess their benefit in auditory perception. However, global communication improvements of cochlear implant (CI) recipients also require accurate speech production in addition to perception. Understanding of the speech and voice productive abilities of CI recipients has not received the same attention as measurements of perceptual performance. Better assessment of these abilities may allow further improvements in communication for patients with hearing loss.

The importance of hearing in vocal production, including the control of both speech and voice, is well accepted. Patients with congenital deafness have difficulty acquiring and maintaining normal speech, , and patients with hearing loss acquired later in life also exhibit more subtle degradations. , Cochlear implantation partially restores these changes, with improvements in the control of pitch, loudness, vowel formants, and many other parameters of speech. However, despite the improvements seen after CI, the vocal communication abilities of recipients still often fail to match those of normal hearing individuals.

There has been recent increasing interest into the role of hearing in the control of speech and voice. Normal hearing individuals exhibit robust control of vocal production and, when faced with errors or altered auditory feedback of their voice, rapidly adjust their production to compensate. , This control is evidence that speakers use hearing on a moment‐to‐moment basis to control their speech and voice, though the underlying mechanisms are uncertain. One robust behavior observed in normal hearing individuals is a pitch‐control reflex, wherein subjects rapidly adjust in the pitch of their voice in the opposite direction of artificially perturbed feedback. , , , , , In contrast, observations of short‐term pitch control in CI patients have been limited to turning the implant on and off. , , , ,

In this pilot study, we investigated the control of vocal pitch in a cohort of CI recipients. We performed vocal recordings with their CI turned on and off, and during a pitch‐shift perturbation task in which vocalizing subjects heard their voice shifted up or down in pitch. Results were analyzed to determine pitch changes under varying conditions to demonstrate the presence of real‐time vocal control in these CI patients.

MATERIALS AND METHODS

Patients

A total of 23 patients were enrolled in this study and recruited from the CI program at our institution. All subjects had post‐lingually acquired hearing loss and had undergone a CI placement with at least 6 months of use prior to vocal testing. To reduce variations due to implant design and programming strategies, all subjects had implants form a single manufacturer. Following the completion of testing, demographic and audiologic information was extracted from the medical record. Patient demographics and audiology data (pre‐ and post‐implant) are listed in Table 1. For patients with multiple post‐implant performance measurements, the most recent assessment was used. Etiology for hearing loss and duration of pre‐implant hearing loss was not available for all subjects. Operative records were available for 22 of the 23 subjects, all indicated full electrode insertions with all 22 electrodes intra‐cochlear, 21 of which via a round window or extended round‐window approach. Patients were tested using their primary implant program, all used an ACE speech coding strategy, and all with monopolar stimulation. All experiments were conducted under approval by the institutional review board and all subjects gave written informed consent.

TABLE 1

Patient demographics and implant performance

Gender	13 Male (56.5%)	10 Female (43.5%)
Implant side	9 Right (39%) 6 Left (26%)	8 Bilateral (35%)
Contra‐lateral hearing aid	11 Yes (48%) 5 No (21%)
Electrode type	8 Precurved (35%)	15 Straight (65%)

Vocal recordings

Vocal recordings took place within a quiet room in the audiology suite of our outpatient clinic. Subjects were instructed to repeat the vowel /e/ for several seconds at a time and to maintain an even tone and loudness to their voice. A microphone (AKG C1000S) was placed ~1 ft from the subject and used to record vocal sounds onto a PC for later analysis. Experiments began with recordings under normal conditions (CI on), with 20 vowel repetitions. Subjects were then instructed to remove their implant (CI off) and the process repeated, after which the CI was replaced and recorded again. Subjects using a contralateral hearing aid removed the aid for the duration of the testing.

Feedback perturbation

Real‐time vocal control was measured using a pitch perturbation task, a commonly used method to assess feedback‐dependent vocal control. , , , , , Subjects were instructed to hold a custom modified headphone (Sennheiser HD280PRO) over their implant speech processor, or both processors in bilateral implant recipients. The headphone was carefully positioned to completely cover the microphones of the speech processor, and subjects queried that they could not hear their own voice with the headphones in place, but not connected. The experiment consisted of 100 to 120 vowel repetitions, as above. Vocal production was captured by microphone and passed through a commercial effects processor (Eventide Eclipse v4). An attached PC detected the onset of phonation, and triggered the processor to change the pitch of the acoustic signal. These perturbations lasted 200 ms and were randomly timed to begin between 300 ms and 800 ms after voice onset, to reduce predictability. Similar to past studies, pitch shifted signals were amplified to a level + 10 dB relative to the level at a subjects' lips, presented back to the subject through the headphones. The use of headphones, rather than a direct line input to the speech processor, introduces the possibility of a subject hearing their unaltered voice through the air, and is the reason for the +10 dB amplification of feedback, a potential shortcoming shared with previous studies in normal hearing individuals. On a random subset (20%‐30%) of trials, no pitch perturbation was performed to serve as a control. Recordings were performed in blocks lasting 60 to 90 seconds at a time, allowing the subject to rest in‐between.

To allow sufficient samples, each subject was tested with only 2 pitch change magnitudes. We initially tested subjects with feedback pitch shifts of +200 and −200 cents (2/12 of an octave). After interim review of the data failed to show a consist pitch shift reflex, we tested an additional cohort of subjects at +600 (1/2 octave) and +1200 cents (1 octave), and later a small number with +400 cents.

Data analysis

We extracted individual vowel phrases from the raw audio recordings, and then calculated the time‐course of pitch changes using an autocorrelation‐based method. Mean pitch and sound pressure level (SPL) were calculated for averaging across the total duration of each phrase for use in CI On/Off comparisons. A small number of trials were excluded due to pitch calculation errors or extreme pitch instability (<10%). Trial to trial variability was determined as the SD across multiple trials (phrases). To determine pitch compensation during the perturbation task, an analysis window was extracted to include 200 ms before and 700 ms after shift onset. Changes in vocal pitch relative to pre‐shift baseline were calculated as Cents = 1200 × log2(pitch/mean baseline). Reponses to upward pitch‐shifts were flipped such that any compensatory response opposite the shift was positive (and any imitation downward). Because subjects normal vocal pitch contours were not flat, we performed a normalization procedure to removed expected pitch changes from the perturbation responses. For each perturbation, the time of perturbation onset was used to select a matching analysis window from the non‐shifted control trials. These control pitch contours were averaged and then subtracted from the perturbation response. The resulting response would therefore show changes during/after the pitch shift beyond those expected from normal variation in vocal pitch over time.

We quantitatively measured the magnitude of the pitch shift compensation for each subject and shift magnitude. We averaged the pitch change contours across multiple trials for a subject, and examined the time window from 50 to 600 ms after shift onset to determine the peak pitch change. Statistical analysis was performed for individual subjects and shifts by comparing the pitch change magnitude of individual trials at this peak time point, relative to control trials at the same time point, using a two‐sided t‐test, and False Discovery Rate (FDR) corrected for multiple comparisons. Peak pitch changes were compared between different pitch shifts using an ANOVA with post‐hoc Bonferroni corrections. Comparisons between subjects' pitch changes and demographic variables were performed using Pearson's correlation coefficients, for continuous variables, and ANOVAs for categorical variables. Comparisons were performed first for all subjects, and then only for +600 and +1200 feedback conditions, to eliminate a feedback confound. P‐values ≤.05 were considered to be statistically significant.

RESULTS

Short‐term auditory deprivation

We first performed vocal recordings to determine the effects of short‐term auditory deprivation and to compare results with previous studies. We first recorded subjects' voices during repetition of the vowel /e/ while the subjects wore their CI. Recording was repeated after the subject removed their CI processor, and again after replacing it. With the CI off, subjects exhibited an increase in average vocal pitch by 11.1 Hz (116.2 cents) compared to baseline (Figure 1A). Following implant replacement, the pitch returned back towards baseline (Figure 1A), consistent with previous results. , , Statistical testing showed these pitch changes to be significant (ANOVA, df = 2, F = 5.16, P = .008). We also noted a non‐significant (P > .05) increase in phrase‐to‐phrase pitch variability during the short‐term auditory deprivation (Figure 1B).

FIGURE 1

Changes in vocal acoustics during short‐term auditory deprivation. Mean vocal pitch increased when the CI was turned Off, A, and returned back towards normal after the CI was turned back On (P < .001). Absolute pitch is shown left, pitch change in cents on the right. There was a nonsignificant increase in vocal pitch variability (trial‐to‐trial SD) with the CI off, B. During the CI Off condition, there was also an increase in mean vocal SPL, C. Vocal SPL and pitch correlate across multiple utterances during both CI On and Off conditions (D). Correlation coefficients indicated on the plot. Error bars are SE. (**P < .001)

To determine whether or not these vocal pitch changes might be attributable to increased vocal effort of a subject unable to hear themselves, we measured vocal loudness (SPL) during the CI on/off conditions (Figure 1C). We found a significant increase in mean vocal SPL with the CI turned off (+3.6 dB, F = 14.2, P < .001). As there is a well‐described correlation between vocal effort, loudness, and pitch, we further compared acoustic parameters for each individual phrase (Figure 1D). This analysis demonstrated a significant correlation between SPL and pitch during baseline CI On conditions (slope 17.1 cents/dB, 95% CI [13.1 21.1]; r = .38, P < .001). This correlation was even stronger with the CI turned Off (28.5 [23.7 33.4]; r = .49, P < .001).

Pitch‐shifted feedback

Because vocal pitch variation resulting from changes in hearing status might be attributed to vocal effort, rather than more precise vocal self‐monitoring, we performed an experiment to record vocal pitch during a pitch‐shifted feedback task. Figure 2 shows the average vocal pitch response from a single subject following a brief (200 ms) +1200‐cent pitch shift. As a result of the altered feedback, the subject compensated by changing their voice in the opposite direction with a peak magnitude of 53 cents (P < .01, t‐test with FDR correction). This vocal compensation is rapid, with a latency of 180 ms and compensation peak occurring at 590 ms, consistent with previous results in normal hearing subjects. , ,

FIGURE 2

Sample compensatory changes in vocal pitch during a +1200 shifted feedback task for a single subject. Mean and SE range is shown for pitch changes over time (red), relative to that start of the shift onset (vertical line). Variability in control trials is shown (grey). Bottom bar marks the duration of significant differences between compensation and control (P < .01), beginning 180 ms after shift onset

Individual subjects were tested with two different pitch shift magnitudes or directions (Figure 3). Initial subjects were tested with +200 and −200 cents, however we found no systematic vocal compensation across the tested subjects (P > .05, t‐test). A second cohort of subjects was tested with larger shifts (+400, +600, +1200 cents). We found that these subjects exhibited larger pitch changes of +32, +38, and + 39 cents. This compensation was significant for larger pitch shifts (P < .001), but not for +400 (P > .05), though only three subjects were tested in this intermediate condition. There was considerable inter‐subject variability in these responses, with response standard deviations of 31, 20, 35, 19, and 29 cents (for −200, +200, +400, +600, +1200 cents). Overall, the size of the pitch shift had a significant effect on the degree of vocal pitch compensation (df = 4, F = 4.09, P = .007, ANOVA). We did not, however, find any relationship between compensation timing and the degree of shift (P = .74). These results demonstrate that CI patients, like normal hearing individuals, are capable of real‐time control of vocal pitch when tested with changes in vocal feedback.

FIGURE 3

Box and whisker plot showing vocal pitch compensation for different feedback pitch shifts. Circles indicate individual peak compensations, filled circles are individuals with statistically significant compensation (P < .01). Significant average compensation across subjects for each pitch shift is indicated below (**P < .001). Comparisons between different shifts is shown above (*P < .05)

Pitch control, patient demographics, and CI performance

Although these results demonstrate vocal compensation for large feedback shifts on average, close examination of Figure 3 reveals considerable inter‐subject variability for both large and small feedback pitch shifts. To understand the origins of this variability between subjects, we compared vocal pitch compensations to patient demographic factors and CI performance (Table 2). We found no significant correlations between vocal compensation and demographic factors, including age, gender, side of implant (left, right, bilateral), contra‐lateral hearing aid use, or CI electrode type. There was a weak correlation between vocal compensation and duration of implant use (r = .36, P = .014). There was similarly no correlation between pre‐ and post‐CI audiometry, with the exception of a weak correlation with post‐implant low frequency pure tone thresholds (P = .011).

TABLE 2

Comparison of vocal compensation, demographics, and implant performance

		P‐value	P‐value (+600/1200 only)
Gender		.78	.87
Implant side (R, L, or Bilat)		.67	.67
Contra‐lateral hearing aid		.95	.70
Electrode type		.20	.30
	Correlation Coef	P‐value	P‐value (+600/1200 only)
Age	−0.15	.32	.39
Duration of implant use (mo)	0.36	.014	.06
PTA (pre)	0.10	.50	.24
PTA (post)	−0.10	.51	.011
250 Hz threshold (pre)	0.12	.43	.16
250 Hz threshold (post)	−0.18	.23	.12
Azbio (%, pre)	−0.02	.88	.53
Azbio (%, post)	−0.07	.68	.33
CNC Phonemes (%, pre)	−0.14	.38	.13
CNC Phonemes (%, post)	0.24	.12	.95

Significant p‐values shown in bold.

DISCUSSION

We investigated the role of auditory feedback in the control of vocal pitch in cochlear implant recipients. Results from this pilot study suggest that CI patients are capable of feedback‐dependent vocal control, but require large perturbations to evoke a compensatory behavioral response. There was considerable inter‐subject variability, with only weak correlations to the duration of implant experience and CI pure‐tone thresholds.

There has been relatively little prior investigations of speech and voice control in cochlear implants. These previous studies have suggested that over the long‐term, CI patients do better than prior to their implant, but often fail to match the voice control seem in normal hearing individuals. The mechanisms and acoustic features by which CI patients use the hearing afforded by the implant in their vocal control are unknown. Previous attempts to examine the effects of short‐term perturbations have largely been limited to brief hearing deprivation, turning the implant on and off. , , , , These studies found that turning the implant off generally resulted in increased pitch and vocal loudness. Here we confirmed these previous findings with a similar comparison. However, we also demonstrated a strong correlation between vocal pitch and loudness. These results suggest that it may be subjects speaking more loudly that resulted in the increase in vocal pitch, rather than fine feedback control of the voice. Increased vocal loudness in hearing loss and conditions of degraded feedback, as in background or masking noise, have been well described. , , Interestingly, we also noted that the strength of the correlation and slope between vocal pitch and loudness increased with patients' CIs turned off, suggesting that counter‐acting vocal control mechanisms to maintain pitch may be active when using the implant, and absent without the implant.

To better evaluate the use of CI auditory feedback in vocal control, we performed a pitch‐shift task. Similar feedback manipulations have been extensively used in the study of vocal control in normal hearing healthy subjects , , and select normal‐hearing patient populations. , , In the presence of pitch‐shifted feedback, these subjects compensate for the shift by changing the pitch of their voice in the opposite direction. This behavior is thought to be reflexive and mediated by the central auditory system. Our CI patients also exhibited a similar compensation to pitch‐shifted feedback. However, unlike normal hearing subjects which will compensate for shifts as small as 25 to 50 cents, our CI subjects did not exhibit a significant response until shifts reached 600 cents (1/2 octave) or more. This higher threshold is, perhaps, not surprising given the poorer frequency resolution afforded by a CI compared to a normal cochlea. Studies of vocal pitch perception have suggested that CI patients are often unable to detect changes less than 900 cents, similar to the range of our findings, in contrast to greater perceptual sensitivities in normal hearing listeners.

Another interesting observation was that, although CI patients required a larger feedback shift to evoke a response, the magnitude of their compensation was larger than typically seen in normal individuals. Here we observed average compensations between 50 and 60 cents, in contrast to more typical response of 10 to 20 in previous studies of normal subjects. , It is possible that this reflects methodologic differences, as we did not have a control subject arm in this pilot study. However, this larger effect in CI is potentially important, as even normal hearing subjects undercompensate the feedback pitch‐shift, and their compensation does not increase with the magnitude of the shift. The origin of this under‐performance has been the subject of some debate, and might be theorized to reflect somatosensory feedback or patients hearing both shifted feedback by headphone and un‐shifted feedback by bone‐conduction hearing. As CI patients would not generally perceive bone‐conducted sound, this is not an issue in the current study, and the larger compensation may therefore reflect our ability to better control the auditory feedback. Both groups, however, may potentially get unaltered feedback by air conduction, reducing compensation, something that might be investigated in the future using a direct line input to the CI speech processor. Another possible explanation for the increased effect in CIs is the contributions of central auditory plasticity due to long‐term deafness in our subjects. Duration of hearing loss was not well characterized in our cohort. How such auditory changes might affect interactions with the vocal motor system are unknown, but might be investigated in the future using implant patients with more recent onset or sudden hearing loss.

We also observed considerable variability in the vocal compensation in our subjects, irrespective of size of the feedback pitch‐shift. Indeed, past studies have suggested inter‐subject standard deviations of 13 to 15 cents, while our CI subjects exhibited 19 to 31 cents, with many subjects exhibiting following rather than opposing (compensating) responses, particularly for small feedback shifts. While the smaller, sub‐threshold, pitch perturbations like two semitones may have simply resulted in random voice fluctuations, the origin of this variability at higher perturbations is unclear. There were weak correlations with the duration of implant use and the pure‐tone thresholds perceived with the CI. It is likely that there are more significant, un‐measured factors in this performance. For example, we do not know the frequency locations of the electrode contacts or depth of insertion, which may differentially affect pitch coding and speech perception. The latter is of particular interest as it may affect low‐frequency cochlear coverage and therefore pitch coding. We also do not know the pitch perceptual abilities of our subjects, and it is possible that this would better predict their performance on the vocal pitch task. Previous studies of pitch perceptual thresholds of cochlear implant subjects have shown considerable inter‐subject variability, , any may account for the observed variability in pitch control during vocal production. , This represents the largest shortcoming of the current investigation. It is also important to note that, in this pilot study, we probed a variety of pitch shifts, and the larger shift magnitudes were only tested for one direction (positive) of shift, rather than bidirectional shifts. Such upward shifts place feedback acoustics into higher frequencies more likely to be covered by the cochlear implant. Future work will need to more directly measure pitch perception in these patients, and correlate just‐noticeable pitch differences with the threshold for vocal feedback compensation as well as pitch shifts in both directions.

CONCLUSIONS

Hearing plays an important role in the control of speech and voice in normal hearing individuals, but this role is less well understood in patients with hearing loss and rehabilitation. Previous studies of CI patients have demonstrated improvements in vocal control, but fall short of the precise control seen in normal individuals. In this pilot study, we demonstrate that CI patients are capable of rapid, real‐time control of their vocal pitch, but require large changes in feedback to evoke a compensatory vocal response. These findings are evidence that vocal control abilities are present in CI patients, but better understanding of the underlying mechanisms and limits are needed. Such work may allow development of new programming strategies or therapies to improve vocal communication in patients with hearing loss.

CONFLICT OF INTEREST

The authors have no other funding, financial relationships, or conflicts of interest.