An overview of the major phenomena of the localization of sound sources by normal-hearing, hearing-impaired, and aided listeners.

Localizing a sound source requires the auditory system to determine its direction and its distance. In general, hearing-impaired listeners do less well in experiments measuring localization performance than normal-hearing listeners, and hearing aids often exacerbate matters. This article summarizes the major experimental effects in direction (and its underlying cues of interaural time differences and interaural level differences) and distance for normal-hearing, hearing-impaired, and aided listeners. Front/back errors and the importance of self-motion are noted. The influence of vision on the localization of real-world sounds is emphasized, such as through the ventriloquist effect or the intriguing link between spatial hearing and visual attention.

Introduction: The Auditory Cues to Direction

In many simple listening circumstances, a normal-hearing person’s percept of the position of a sound source is reasonably veridical: the perceived direction corresponds closely to the actual direction, and the perceived distance is at least passably accurate. For hearing-impaired listeners, however, their performance in experimental spatial-hearing tasks is usually worse, and often hearing aids exacerbate the decrements in performance. There are situations in which even normal-hearing percepts become noticeably inaccurate (e.g., in large amounts of reverberation or at adverse signal-to-noise ratios [SNRs]), entirely wrong (e.g., perceiving a source to be behind when it was really in front), or overridden by vision (e.g., ventriloquism), and in many of these domains there is experimental evidence that hearing-impaired listeners perform even less well. This article outlines some of the major experimental results in these areas, concentrating on the fundamental physics and psychophysics of localization, the effects of hearing impairment and hearing aids, and the importance of vision in the real world. The detailed theoretical reasons for the effects will be saved for a future work.[1]

The fundamental auditory cues to direction arise because the ears are on either side of the head (e.g., Blauert, 1997; Moore, 2012; Warren, 1999). The sound from a source on the right side of the head will arrive at the left eardrum after it arrives at the right eardrum, because the left ear is further away, and it will also be lower in level at the left than at the right because the head is a solid object and so casts an acoustic shadow. These differences are termed, respectively, interaural time differences (ITDs) and interaural level differences (ILDs). The auditory system likely uses some form of mapping to decode ITDs (though quite how is unresolved: e.g., Carr & Macleod, 2010; Harper & McAlpine, 2004), whereas the analysis of ILDs may be as a simple as a comparison of the sound levels at the two ears (Hartmann & Constan, 2002). But it is important to remember that though ITDs and ILDs are usually described separately, and often manipulated separately in experiments, the sound from any real source must have both an ITD and an ILD, even if either one (or both) is zero. The auditory system combines the two cues into a percept of a direction.

To a first approximation, the relationship between direction and ITD can be found from a simple geometrical calculation of the additional distance to the far ear divided by the speed of sound (Blauert 1997; Moore, 2012; Woodworth, 1938). If the head is assumed to be spherical in shape, and the source of sound is sufficiently far away for the wavefronts to be planes, then the additional distance is given by rθ + rsinθ, where r is the radius of the head (about 9 cm) and θ is the azimuth of the sound source, in radians. For instance, a source located slightly to the side, say 10° to the right, gives an additional distance of 3 cm and an ITD of 9 μs, but a source located as far to the left as possible (−90°) gives an ITD of −670 μs. Woodworth’s formula rθ + rsinθ accounts very well for measurements of the ITDs of clicks from loudspeakers (Feddersen, Sandel, Teas, & Jeffress, 1957) and of high-frequency pure tones, but it fails for low-frequency pure tones (Kuhn, 1977), in that the measured ITDs are larger than the formula predicts by a few hundred microseconds (Aaronson & Hartmann, 2014). Empirical observations also show that the presence of the torso and any clothing on the subject can affect the values of ITD (Kuhn, 1977; Treeby, Pan, & Paurobally, 2007). These effects are relatively small though, generally no more than a hundred microseconds, and so, given its simplicity, Woodworth’s formula is commonly used to convert azimuths to ITDs and vice versa in experimental work.

The magnitudes of ILDs also vary with direction and frequency. They are generally larger at higher frequencies and are mostly larger at larger azimuths. However, unlike ITDs, there are sharp dips in ILD at some frequencies but strong peaks at others, and an ILD found for one direction may bear little resemblance to the ILD found for a neighboring direction (e.g., Shaw & Vaillancourt, 1985). The dips and peaks occur because of diffraction and reflection of the incoming sound with the torso, head, and pinnae and can be of the order of 20 dB. These are crucial for differentiating up versus down and front versus back (e.g., Zhang & Hartmann, 2010). There is no simple formulae that describe how they vary with frequency and direction, though they can be computed from boundary-element models (e.g., Kreuzer, Majdak, & Chen, 2009) or calculated analytically if simplifying assumptions are made, such as the head being an exact sphere (e.g., Macaulay, Hartmann, and Rakerd, 2010; Rayleigh, 1894/1945).

Many experiments have measured the magnitudes of the just-noticeable differences (JNDs) that listeners can detect in ITD, ILD, or actual direction, while varying the frequency, duration, overall intensity, waveform, onset, offset, masker type, signal-to-masker ratio, and so forth of the stimuli (e.g., Durlach & Colburn, 1978). A few important results are considered here. For normal-hearing listeners, the JND for ILD is of the order of 1 dB (e.g., Grantham, 1984; Hartmann & Constan, 2002; Yost & Dye, 1988). There is little frequency dependence to the JND, except a slight worsening around 1 kHz by no more than a quarter of a decibel. In contrast, the data for the JND for ITD show an extreme frequency dependence. For pure-tone stimuli, the JND is about 60 μs for a frequency of 250 Hz, 10 μs for 1000 Hz, 20 μs for 1250 Hz, but then essentially becomes unmeasurably large for frequencies above about 1500 Hz (e.g., Brughera, Dunai, & Hartmann, 2013; Klump & Eady, 1957). The rate of change of ITD JND with frequency around 1500 Hz is perhaps one of the steepest functions in all of auditory psychophysics. Nevertheless, it is not the case that ITDs are impossible to detect at high frequencies: if complex stimuli are used in which the ITDs are carried by modulations in the temporal envelope, then JNDs can still be measured (e.g., Bernstein & Trahiotis, 2010; Henning, 1980).

The JND for actual direction (known as the minimum audible angle, MAA) is, at best, about 1°. This is found for pure-tone stimuli at around 750 Hz, using sound sources directly ahead and for changes in direction limited to the horizontal plane (e.g., Mills, 1958). It reaches a maximum (about 3°) at frequencies around 2000 Hz before reducing again for frequencies up to about 8 kHz. The MAAs are much higher for changes in azimuth for sound sources located to the side (Mills, 1958) or for MAAs for changes in elevation for any direction (e.g., Grantham, Hornsby, & Erpenbeck, 2003).

The standard explanation of the midfrequency maximum is that people use ITDs to locate sound sources at lower frequencies but not at higher frequencies and ILDs at higher frequencies but not at lower. The theory is known as the Duplex Theory, which was developed around a century ago by JW Strutt, Lord Rayleigh (1894/1945, 1907) and is described in many standard textbooks (e.g., Moore, 2012; Plack, 2005; Yost, 2007). Its explanation at low frequencies is that the magnitudes of the ITDs can be relatively large, whereas the magnitudes of the ILDs are always relatively small as the wavelengths are too long for diffraction to be substantial. At high frequencies, its explanation is that the magnitudes of the ILDs can be much larger as the wavelengths are far shorter, meaning that the head shadow becomes a more significant factor and there is more scope for constructive or destructive interference, whereas though there are still substantial ITDs, the neural mechanisms for decoding them fail to work (at least with high-frequency pure tones). Over a century after its formulation, the low frequency = ITD versus high frequency = ILD dichotomization of the Duplex Theory remains valid—at least for pure-tone stimuli (e.g., Macpherson & Middlebrooks, 2002). But the experiments cited above on the JNDs for ITD and ILD demonstrate that the auditory system can discriminate ITDs and ILDs at all frequencies, as there is no frequency at which either JND is impossible to measure. Presumably, the system can therefore use the information at any frequency to help determine the direction of a sound source: for instance, envelope ITDs at high frequencies may be useful for locating sources in reverberation (Ruggles, Bharadwaj, & Shinn-Cunningham, 2012). This would make sense as a strategy, as in most natural listening circumstances the target sound may not simply be a low-frequency or high-frequency pure tone presented in quiet. It is more likely to be mostly broadband, partially masked by various backgrounds, and continually changing in instantaneous level and spectrum, with ITD cues and ILD cues at all frequencies. The cues will be changing moment-to-moment, frequency-to-frequency. Even after a century of research, however, there remains much that is unknown about quite how they are combined into a single percept of a direction.

The Effects of Hearing Impairment and Hearing Aids

In general, hearing-impaired listeners perform worse in spatial-hearing experiments than those with normal hearing. For example, Hausler, Colburn, and Marr (1983) measured minimal audible angles for white-noise stimuli as part of a comprehensive set of experimental tests on spatial hearing. For presentation from the side, they found that the smallest MAA for a group (n = 14, each tested twice) of bilaterally sensorineural-loss listeners was 7°, and about half of the listeners gave MAAs of 30° or more. In contrast, all the normal controls (n = 26) gave MAAs of 12° or less. A second example comes from Neher, Laugesen, Jensen, and Kragelund (2011), who measured the highest frequency at which listeners could discriminate an interaural phase difference (IPD) of 0° from 180° for a pure-tone stimulus. They found a mean highest frequency of just 850 Hz in a group of 23 older hearing-impaired listeners (mean age 67 years), whereas they found a mean highest frequency of 1230 Hz for a control group of eight younger normal-hearing listeners (mean age 35 years). Moreover, the across-listener range of highest frequencies was wider in the hearing-impaired listeners: around 300 to 1250 Hz, whereas it was 900 to 1500 Hz in the control group. Background sounds generally exacerbate the decrements in performance by hearing-impaired listeners, even over what would be expected simply from reduced audibility. One example is from Lorenzi, Gatehouse, and Lever (1999), who measured the accuracy of reporting the spatial direction of a 300-ms, broadband click train when partially masked by a spatially diffuse, white noise. At a SNR of −6 dB, the mean error in reported direction from their four hearing-impaired listeners was about 50°, but it was only about 25° for a control group of four normal-hearing listeners. A second example is from Best, Carlile, Kopco, & van Schaik (2011). They measured the accuracy of reporting the spatial direction of a target—the single word “two”—presented over loudspeakers, either in quiet or when partially masked by four other words. A group of hearing-impaired listeners (n = 7) were worse at this task than a control group of normal hearers (n = 7) by about 1° in quiet and by about 7° in noise. However, though most experiments on spatial hearing by hearing-impaired listeners have reported some decrement in performance, it is not true that hearing impairment leads to substantial spatial impairment in all tasks. Another one of Hausler et al.’s (1983) results demonstrates this. For measurements of the MAAs of sounds presented from the front, they recorded values of 6° or less for a group of bilaterally sensorineural-loss listeners and 4° or less for the normal-hearing listeners. The mean difference is negligible: It is about the visual width of the thumb when held at arm’s length (O’Shea, 1991). Also, given the wide individual variability in hearing-impaired results (e.g., see Hausler and Neher’s results referred to above), one needs to be careful of the ecological fallacy, that is of assuming that the mean results of a group apply to every individual member of that group.

Hearing aids do not improve the localization of sound sources: indeed, in many cases, they interfere. A few examples follow; all found larger errors in horizontal localization for aided than unaided listening. First, Drennan, Gatehouse, Howell, van Tasell, and Lund (2005) compared the accuracy in localization for aided versus unaided listening, using single words in a speech-shaped noise at a SNR of 0 dB. Despite 10 to 15 weeks of acclimatization to the hearing aids, the localization errors when the listeners (n = 7) were tested aided were generally equal or larger (about 30°–35°) than when tested unaided (about 30°–35°). Before acclimatization, aided accuracy was poorer still, reaching as much as 45°. Second, Van den Bogaert, Klasen, Moonen, van Deun, and Wouters (2006) measured localization accuracy for older listeners (n = 10, aged 44–79) with their hearing aids set for adaptive-directional mode, omnidirectional mode, or not used. They used various stimuli, including telephone rings—notable for being a stimulus that has a clear ecological validity for localization, for when one hears a telephone ring, one often wants to know where it is. Aided performance was, on the whole, worse than unaided: the mean errors were 18° (adaptive-directional), 16° (omnidirectional), and 13° (unaided) but was just 4° for a control group of younger normal-hearing listeners (aged 20–25). Third, Keidser, O’Brien, Hain, McLelland, and Yeend (2009) compared the accuracy for aided versus unaided versus normal listening, using a variety of different kinds of stimuli. The mean errors in localization were about 20°, 20°, and 10°, respectively. A follow-up experiment with different kinds of hearing-aid microphone modes showed that performance interacted between stimulus and directionality: With a pink noise, a fully directional microphone gave remarkably high localization errors at around 35°, even after 3 weeks of acclimatization, whereas omnidirectional microphones (or two types of partially directional microphones) gave errors around 20° to 25°. Fourth, Best et al. (2010) compared the accuracy for localizing single-word stimuli by aided listeners with two types of hearing aid, unaided but impaired listeners, and normal-hearing controls. They found mean errors in accuracy of about 14°, 14°, and 8° respectively. The directional microphones often found on hearing aids can also interfere with the perception of direction. A recent example was reported by Brimijoin, Whitmer, McShefferty, and Akeroyd (2014), who measured how people orientated to new targets in multitalker, multiangle babble. Two groups of aided listeners participated, one group whose own hearing aids were fairly directional (n = 8) and one whose aids were not directional (n = 7). Brimijoin et al. found that the directional group generally took longer to orientate to the target (for some target angles, by as much as half a second) and made more initial misorientations. The misorientations were especially dramatic: when the target sound was offset by 120° or more from the direction the listener was pointing before the target started, those listeners with directional hearing aids initially moved their head in the wrong direction on at least one third of the trials.

A domain in which hearing aids can cause particular problems is in distinguishing sound sources in front from those behind. A confusion between whether a source is ahead or behind occasionally happens to normal-hearing listeners—for example, people often comment how hard it is to locate the emergency siren of an ambulance or fire engine. The reason is because the head and ears are fairly front/back symmetric. Indeed, Woodworth’s rθ + rsinθ formula is perfectly front/back symmetric, as it assumes that the head is a perfect sphere, the ears are diametrically opposite, and the pinnae are not present. Whatever value of ITD the formula gives for a sound from a source at a certain left or right angle in front, it will give exactly the same for a sound from a source at the same left or right angle from behind (or indeed from above or below too). Fortunately, our heads and ears are not as symmetric as Woodworth (1983) assumed, and the pinnae effect sounds from the front differently to sounds from the back, resulting in changes to ILDs, especially at high frequencies (e.g., Shaw & Vaillancourt, 1985).

Laboratory experiments on aided listeners indicate high numbers of front/back errors. Best et al. (2010), in the same experiment described earlier, found that in normal-hearing control listeners, a front-to-back (or a back-to-front) error was made in about 5% of the trials, whereas it was about 12% in unaided, impaired listeners and between 25% and 45% in aided, impaired listeners. A second example is from Vaillancourt, Laroche, Giguere, Beaulieu, and Legault (2011). They found the proportion of front/back errors (for a short broadband noise stimulus) for a large group (n = 57) of hearing-impaired police officers wearing their own aids were 22% aided but 15% unaided. Also, they observed a substantial dependence on hearing loss: the correlation of unaided front/back error rate with unaided hearing level (across 3–6 kHz) was about 0.5 but the correlation of aided front/back error rate with aided hearing level was only about 0.2—the reason being that very many listeners were much worse in the aided conditions.

Outside the laboratory, however, it is uncertain whether front/back errors are as prevalent as those experiments would seem to indicate. The reason is because people are continually moving their heads, even if just fidgeting, and head movements can resolve front/back errors (Brimijoin & Akeroyd, 2012; Wallach, 1940). Whenever someone rotates their head by any amount (e.g., 10° to the left), then from their perspective, the auditory world rotates by an equal and opposite amount (i.e., 10° to the right). To illustrate, imagine someone perceives a given source of sound to be directly ahead. Could that person have made a front/back error—was it really, physically ahead (correct) or was it physically behind but was perceived ahead (error)? The issue can be resolved by a head rotation: if the apparent direction of the source moves by the same amount and in the same direction, then the source was actually behind, but if instead the apparent direction moved by the same amount in the opposite direction, then it really was in front (if the apparent direction moves by any other amount, provided the source is not perfectly directly above or below, then something has gone quite wrong in its localization).

The importance of motion for resolving front from back can be demonstrated by using motion to invoke front/back errors (Brimijoin & Akeroyd, 2012). This can be done by playing a sound from a loudspeaker that is physically behind someone, who then moves their head, say to the left by x°, and at that exact same moment the direction of the sound is moved by 2x° in the same direction as the head movement. This procedure thus sets up in opposition two cues to direction: the change in geometry only makes perceptual sense if the sound source was in front whereas the acoustic cues themselves (especially the ILDs for any high-frequency components in the signals) only make perceptual sense if the sound source was behind. Brimijoin and Akeroyd (2012) found that for low-pass filtered (500 Hz) signals, their listeners (normal hearing) primarily reported the source as in front, so indicating that their perception was mainly based on motion cues.

Distance, Vision, and the Real World

The phenomena described earlier have been concerned either with the perception of direction or with the underpinning ITD and ILD cues. But in the real world, there is the third dimension of distance, which is crucial to giving the auditory perception of a scene a naturalness. Distance is inherently linked to level: the closer a source is, then the more intense it is (e.g., Blauert, 1997; Zahorik, Brungart, & Bronkhorst, 2005). Normal-hearing listeners do not perceive distance linearly, in that the psychophysical function relating physical distance to perceived distance is compressive: distant sources are reported as closer than they physically are, but close sources are reported as slightly further away than they physically are (Zahorik, 2002). But hearing-impaired listeners may perceive a substantially contracted world, with external sources perceived closer than in reality (Boyd, Whitmer, Soraghan, & Akeroyd, 2012). The JND for changes in distance is distance dependent and in some conditions is affected by hearing loss (Akeroyd, Gatehouse, & Blaschke, 2007). Given that distances are cued by levels, and hearing-aid level compressors change levels, one would expect hearing-aid compressors to affect distance perception, but in a specific test of distance JNDs in aided listening, we did not find any effect of compression ratio (Akeroyd, 2010).

There is another aspect of real-world listening that is generally excluded from many experiments, namely vision: one can often see any sources of sound. There are various reasons for believing that spatial hearing is inherently linked to vision. One is based on the smallness of the MAA, which for pure-tone stimuli in quiet presented from in front can be as little as 1°. This corresponds to about the width of the index fingernail held at arm’s length (O’Shea, 1991). Though it is always possible that the evolutionary pressure for such good directional hearing may be a need to accurately locate sounds in quiet, it is perhaps more likely that it is a corollary (or a spandrel as argued by Gould & Lewontin, 1979) of other reasons, such as giving some capacity for localization of sounds in noise, or binaural signal detection through spatial release from masking, or tracking of moving sources, or compensation for the effects of head motion. None of these suggestions can directly account for why performance should be best for sound sources in front, but the idea that spatial hearing is linked to vision can. The argument is that accurate sound localization is there to set the direction of visual attention (e.g., Hafter & de Maio, 1975; Heffner and Heffner, 1992; Perrott, Saberi, Brown, & Strybel, 1990; Pumphrey, 1950). It can be illustrated by imagining an animal in a forest. The visual scene is cluttered: there are trees, branches, and leaves everywhere. Everything is at different distances. Locating another animal (be it food, a predator, or a mate) by vision alone could be difficult, especially if it is camouflaged or hiding, and would be harder still if it was twilight or raining. But the task made considerably easier if the listening animal can locate the target animal by sound and so focus its attention. That is, the more accurate auditory localization, the finer visual attention can be. There are data to support this: Heffner (2004) demonstrated that the accuracy of sound localization versus the width of field of best vision, across 29 species of mammals, correlated at 0.9: those animals with high accuracy also had the narrowest best fields. Humans are at the extremely good end of both dimensions (localization thresholds of about 1° and best visual fields of about 0.5°, whereas pocket gophers are at about 180° for both). Though correlation is not causation—nor can one be sure of the direction of causation anyway, as perhaps the distance and direction of a potential source are solved by vision before any auditory processing—there is presumably some reason why evolution has given humans both extremely good localization and a narrow field of best vision.

A second demonstration of the importance of vision for spatial hearing comes from the ventriloquist effect. The purpose of the act of a good ventriloquist is to make the audience think that the sound comes from the mouth of the ventriloquist’s dummy, not from its actual source, the mouth of the ventriloquist. There are many laboratory experiments quantifying its strength in terms of localization (e.g., Alais & Burr, 2004; Bertelson, 1999; Bertelson, Vroomen, de Gelder, & Driver, 2000; Jackson, 1953), though it would be of particular interest to know how much it influenced perception of sound sources in everyday life (it would also be of interest to know its strength in hearing impairment or aided listening, as there are few, if any, ventriloquist experiments which have used such listeners). To take just one example, imagine the situation of someone sat at the back of a large hall, listening to a lecturer in front but with a ceiling-mounted public-address system providing the sound heard, so the physical direction of the sound is overhead. It is arguable that the listener might not notice that the acoustic direction is quite different to the visual direction, and even if it is noticed, it may be tolerated then quickly ignored. Does this reflect some form of ventriloquist processing?

Taken together, these effects indicate that auditory perception of spatial position is not solely based on the sounds arriving at the ears, instead vision plays at least some role. Thus, to understand how hearing-impaired listeners perceive the location of sound sources in their everyday, real-life listening, we need know more about the localization of sounds when, like in real life, they can see a potential source. We also need experiments to study how tolerable are mislocations, and, vice versa, when mislocations are annoying. If the importance of localization lies in locating (in both direction and distance) sound sources in noise, dealing with motion, both of sources and of one’s self, and in guiding vision, then these are the domains to ask scientific questions in, to run experiments in, and in which to judge the success of hearing aids. The challenge is then to ask new science questions about how people deal with localization—and mismatched localization—in real, visual environments.