A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution.

The phenomenology of the blind has provided an age-old, unparalleled means of exploring the enigmatic link between the brain and mind. This paper delves into the unique phenomenological experience of a man who became blind in adulthood. He subsequently underwent both an Argus II retinal prosthesis implant and training, and extensive training on the EyeMusic visual to auditory sensory substitution device (SSD), thereby becoming the first reported case to date of dual proficiency with both devices. He offers a firsthand account into what he considers the great potential of combining sensory substitution devices with visual prostheses as part of a complete visual restoration protocol. While the Argus II retinal prosthesis alone provided him with immediate visual percepts by way of electrically stimulated phosphenes elicited by the device, the EyeMusic SSD requires extensive training from the onset. Yet following the extensive training program with the EyeMusic sensory substitution device, our subject reports that the sensory substitution device allowed him to experience a richer, more complex perceptual experience, that felt more "second nature" to him, while the Argus II prosthesis (which also requires training) did not allow him to achieve the same levels of automaticity and transparency. Following long-term use of the EyeMusic SSD, our subject reported that visual percepts representing mainly, but not limited to, colors portrayed by the EyeMusic SSD are elicited in association with auditory stimuli, indicating the acquisition of a high level of automaticity. Finally, the case study indicates an additive benefit to the combination of both devices on the user's subjective phenomenological visual experience.

Credit author statement

Amber Maimon: Writing - Original Draft, Writing - Review & Editing, Conceptualization, Visualization, Project Administration; Or Yizhar: Writing - Review & Editing, Conceptualization, Formal Analysis, Visualization; Galit Buchs: Conceptualization, Investigation; Benedetta Heimler: Conceptualization, Investigation; Amir Amedi: Writing - Original Draft, Writing - Review & Editing, Project Administration, Supervision, Resources, Conceptualization, Investigation, Methodology, Funding acquisition.

Introduction

“You don’t see with the eyes. You see with the brain.” (Bach-y-Rita, 1972)

The phenomenology of the blind has fascinated scientists and philosophers alike for centuries, serving as a paradigm for exploring the link between the brain/physical body and the conscious creations of the mind. As far back as the 17th century, rationalist Descartes enlisted the visual-to-tactile substitution of blind walking stick users in arguing for the dominance of reason over sensory experience (Descartes, 2001), while the classic empiricist philosophers historically argued that the blind cannot fully grasp visual sensory phenomena (Locke, 1847), and that they are missing an impression of true visual sensation (Hume, 1882). Even in modernity, it has been argued that the blind will necessarily be missing an essential aspect of visual phenomenology (Nagel, 1974). Furthermore, it has been claimed that the subjective phenomenal experience of the blind is inaccessible to others (Nagel, 1974) making the phenomenology of the blind notoriously difficult to explore in objective ways by modern scientific means. It is thought that the seemingly impenetrable experiential questions, can truly be explored only by rigorous inquiry into the first-person subjective experience of the blind.

Vision is our dominant sensory modality (Hutmacher, 2019). It is the primary sense we depend upon when interacting with the world around us (Pike et al., 2005) and is considered the most essential and complex of the senses (Pike et al., 2005; Gerrig and Zimbardo, 2010). As such, vision loss can lead to a profound deterioration in quality of life (WHO -Blindness and vision impairment, 2021; Bourne et al., 2021). In particular, late-onset blindness can have a detrimental effect on one's independence and frequency of activity (Good et al., 2008).

With cases of blindness onset in childhood (before the age of 15) representing only 5% of the cases of blindness worldwide (Steinkuller et al., 1999), the majority of the blind are late blind, the implications of which are profound. The experience of the late blind is significantly distinct to that of the early or congenitally blind, both quantitatively and qualitatively, due to the familiarity of the late blind with visual sensory experience prior to vision loss. Late blind are less capable of adapting to their circumstances than early blind, as they are less likely to develop compensatory strategies, both psychologically and physiologically (Wan et al., 2010; Dormal et al., 2016; Scheller et al., 2021). The affliction of late onset blindness is so harrowing that the late blind are willing to hypothetically trade off remaining years of their lives (Brown et al., 2001; Wagle, 2015), and many are willing to expose themselves to considerable risk of death in exchange for treatment that would restore their vision even partially (Wagle, 2015).

One common cause of late-onset blindness is retinitis pigmentosa (RP), an inherited disorder characterized by progressive retinal dystrophy due to gradual degeneration of the photoreceptors (Hamel, 2006). It is commonly thought that vision loss for individuals with RP is ultimately inevitable (Hamel, 2006) though some clinical trials involving gene therapy provide promise of visual recovery (Sahel et al., 2021; Cehajic-Kapetanovic et al., 2020). The circumstances of those with RP are uniquely grim in that they experience the entire degradation of their vision as a process that begins with night blindness, continues with a decline in vision often over decades, and ultimately concludes with legal blindness in later life (Hartong et al., 2006). As the blind, the late blind in particular, find themselves in a dire situation, cutting-edge technology is being developed with the aim of restoring their vision, yet what is the nature of the subjective experience, the qualia of the person using these technologies? Is there a difference in the qualia associated with different approaches to sensory restoration? This is an intriguing question both scientifically, philosophically and practically (so that patient will know what to expect qualia wise from the procedure they choose). So far, the matter remains to be thoroughly explored.

As RP is caused by damage to the retina, without cortical lesions, a possible solution for visual restoration is the implantation of an epiretinal visual prosthesis that replaces the degenerated photoreceptors. The first such device to get an FDA approval in the USA is the Argus II retinal prosthesis that consists of implanted electrodes and an external processing unit worn by the user. Visual information is obtained and processed externally and sent wirelessly to the implanted electrodes that stimulate the retina. The electrically elicited phosphene patterns (Ahuja et al., 2011) are consistent with the pattern of brightness found in the visual image (Ho et al., 2015).

GB, the subject of the present case study, became blind due to RP. His case is novel as he was the first person residing in Israel to undergo a visual prosthesis implant procedure, one of only around 350 users of the Argus II retinal prosthesis worldwide (Second Sight Medical) and then to also participate in extensive training with an SSD device, thus comparing both devices. GB describes his motivation for going through with the implant procedure despite the uncertainty involved in the procedure as follows: “I believed that I could get something, a pinch of vision, through the implant.” Yet, the visual function allotted by the Argus II system is relatively restricted. Other retinal prostheses are also in development and hold great promise, some taking full advantage of recent technological advancements, for example the incorporation of artificial intelligence algorithms into their devices, such as the PRIMA system (Hornig et al., 2017). A key advance offered by the PRIMA system is the small size of the electrodes used, that provides a high-resolution interface with the retina (Palanker et al., 2020, 2022). A recent study by Palanker et al. (2020) reported that a patient was able to recognize letters and that several patients could discriminate bar orientation, representing a significant advancement in the abilities allotted by newer retinal prostheses.

Retinal prostheses have the immense potential and promise of stimulating immediate visual percepts in the form of phosphenes, yet it has been suggested that by electrical stimulation alone, like that provided by retinal prostheses (particularly early models such as the Argus II) it may be a challenge to create and maintain meaningful visual percepts. This is due to two main mechanisms. One is possible interference by way of neuroplastic changes that take place in the brain as a result of sensory loss. As visual areas are rearranged, they are no longer correctly interpreting the information from the electrical stimulation of the prosthesis. But there is another challenge, which is that stimulation itself creates electrical signals that are very different from the usual code used by the retina (Merabet et al., 2005). Retinal prostheses stimulate the retina directly, activating many neurons in synchrony in an unregulated way. This is not the way the brain is used to receiving information, but rather is an unnatural way of activation. We don't currently have the tools, nor do we know the neural codes, to be able to electrically stimulate the retina specifically in a manner that the brain is used to receiving visual information and would know how to interpret (Merabet et al., 2005). This is an inherent problem with retinal prostheses irrespective of the number of electrodes that will continue to be problematic for future prostheses as well, as long as we don't know the neural “code” that the brain is used to receiving (Merabet et al., 2005). This is represented even in the phenomenological experience of the subject described in this paper. Yet the relevant technology is constantly improving, alongside the advancement of neuroscientific research and understanding, such that one day this challenge may be overcome.

A completely different approach is using sensory substitution devices (SSDs) that may serve to reignite the visual cortex for processing vision, thus priming it for interpreting information from invasive technologies such as retinal prostheses (Merabet et al., 2005), in addition to putatively providing a standalone benefit to the blind (Maidenbaum and Amedi, 2012). An expansive body of research shows that as the result of visual deprivation and blindness in particular, neuroplastic changes take place in the visual cortex, including the recruitment of the visual cortex for processing information from other senses (in the sighted when blindfolded, the early blind, and the late blind), which in turn may fundamentally change the visual cortex's ability to respond to the direct retinal stimulation. Neuroplasticity begins to take place almost immediately upon vision deprivation (even in sighted individuals when blindfolded) (Pascual-Leone et al., 2005). As such, it has been suggested that restoration of retinal activation alone would not result in true visual perception (Merabet et al., 2005). This has been shown in the deaf with relation to cochlear implant integration. Research indicates that the onset of massive cross modal neuroplastic processes (such as those which underly auditory cortex activation in correlation with sign language) adversely effect the ability to successfully install and use a cochlear implant (Lee et al., 2001). Similarly, neuroplasticity may prove maladaptive for the introduction of retinal prosthesis.

Yet, neuroplasticity is not necessarily maladaptive. As we and others have shown, the visual cortex can be recruited for processing information originating from other sensory modalities, and it has been suggested that the processing in the visual cortex does not depend on the sensory input, rather on the computational task. As long as proper training is provided, the computation can bring about the task, For example, in the case of object recognition. It has been shown that the lateral-occipital tactile-visual area (LOtv), an area activated by visual and tactile exploration of objects, can be activated in the blind for processing object shapes after training with a visual to auditory SSD, indicating that this area is involved in the task of processing the geometry and shape of objects, irrespective of the sensory modality through which the information was conveyed (Amedi et al., 2007). This task specific sensory independent form of cross modal neuroplasticity has also been shown with relation to number form extraction (Abboud et al., 2015), and letter form extraction (Reich et al., 2011; Striem-Amit et al., 2012a, 2012b, 2012c) among others. See also reviews (Maidenbaum et al., 2014), (Ricciardi et al., 2014) and (Kupers and Ptito, 2014). Recent research shows that, given appropriate training, the visual cortex of even sighted individuals responds to task-related information from other sensory modalities (Striem-Amit et al., 2012a, Striem-Amit et al., 2012b, Striem-Amit et al., 2012c). Moreover, in the case of the sighted, this task specific recruitment was not associated with a discernible drop in the visual perceptual abilities. As such, this type of cross modal plasticity would not prove maladaptive and would even be adaptive in the case of consistent and complementary inputs from both the visual and auditory pathways, taking full advantage of the computation that the area specializes in, for example in the case of the direct retinal stimulation from the prosthesis, coupled with the auditory input from the SSD. (Merabet et al., 2005).

Visual-to-auditory SSDs, such as the vOIce or the EyeMusic, which the subject of this case study was trained with, transport visual information through the auditory system by converting the images into “soundscapes.” (Meijer, 1992; Abboud et al., 2014; Macpherson, 2018). The soundscape is constructed by “sweeping” the image by column from left to right, with a novel sound frequency combination representing the image in the soundscape using a conversion algorithm specific to the SSD (Abboud et al., 2014).

GB, the subject of the current case study, has both an Argus II visual prosthesis (which he sometimes colloquially refers to as a “bionic eye”) and extensive training using the EyeMusic SSD. Moreover, GB became completely blind in adulthood and is therefore in the remarkable position of being able to judge his phenomenal experience with both the Argus II visual prosthesis and the EyeMusic SSD as compared to his prior visual qualia. GB presents an unprecedented opportunity, as the first and only such case at the present time to our knowledge, to explore whether the phenomenal experience provided by one or the other devices is genuinely visual and what are the similarities and differences in the qualia and use of the devices (at least in this unique case study).

Bach-y-Rita would have answered the phenomenal question concerning the qualia of SSD use being genuinely visual as opposed to a merely cognitive understanding of the input in the affirmative, arguing that: “If a subject without functioning eyes can perceive detailed information in space, correctly localize it subjectively, and respond to it in a manner comparable to the response of a normally sighted person, I feel justified in applying the term ‘vision’” (Bach-y-Rita, 1972, p. ix). But the matter cannot be settled and cast aside so quickly, for perhaps the phenomenal experience is not visual at all? The very slow adaptation to SSD devices despite the higher resolution, lower price and non-invasiveness of the procedure over the last several decades suggests that the answer to this question is much more complex. In addition, maybe the task of identifying SSDs with either vision or audition alone should be abandoned altogether in favor of a multisensory and multimodal interpretation (Deroy and Auvray, 2014; Lerousseau et al., 2021).

There are widely opposing views on the matter, and it is currently at the center of a heated debate. Supporters of what is known as the “deference” view argue that SSDs do indeed provide their users with genuine visual qualia (Hurley and Noë, 2003; Ptito et al., 2008; O'Regan, 2011; Pence, 2020) (albeit the quality thereof is determined by the technological capabilities and limitations of the device). On the other hand, what is known as the “dominance” view also garners vehement support (Keeley, 2002; Block, 2003; Prinz, 2006) as it represents the classic interpretation of our sensory systems in the brain. According to the dominance view, the phenomenology experienced by SSD users is based on the substituting domain. Supporters of the dominance view claim that people who use SSDs to “see with their ears” are experiencing qualia belonging to the auditory domain, not the visual (for more information on the different views see Deroy and Auvray, 2014; Macpherson, 2018; Pence, 2020).

If one is truly interested in the subjective aspect of the phenomenal experience, one must go straight to the source and inquire into the “what is it likeness” (Nagel, 1974) of the sensory substitution experience. Is the subjective experience of the SSD user perception or mental imagery? A previous case study on two long-term blind users of the vOICe visual-to-auditory SSD (Ward and Meijer, 2010) concluded that expert users have true visual phenomenology and that they experience visual qualia. Moreover, that the users maintain the capability of “seeing sound” even when not using the device due to their acquiring a form of synesthesia with prolonged use of the SSD (Ward and Meijer, 2010; Ward and Wright, 2014).

This in itself raises a number of questions regarding SSD use, does the SSD use become transparent? Transparency is the attribution of a distal sensory stimulus or device into one's sensory experience in a “second nature” manner (Auvray and Myin, 2009; Kirsch et al., 2020). Moreover, while synesthetes often exhibit idiosyncrasy regarding their perceptual experience (a somewhat distinctive or different experience for each individual user), the possibility of idiosyncrasy in the SSD user would intuitively seem to be strictly limited by the constraints of the SSD algorithm (Kirsch et al., 2020). Can there be idiosyncrasy in the experience of an SSD user despite these constraints?

The current paper aims to contribute to the as-yet limited body of research into the qualia, the phenomenology, experienced by visual prosthesis and SSD users, in this case specifically the Argus II visual prosthesis and the EyeMusic SSD. In addition, we explore for the first time in the literature to our knowledge, whether the whole is greater than the sum of its parts with regard to vision restoration in the blind. We investigate whether there is an added benefit to retinal prosthesis and SSD use in conjunction, in this case specifically the Argus II retinal prosthesis, and the EyeMusic SSD, above and beyond what is provided by one or the other in isolation.

Methods

Case description

GB was born in 1959 with low vision and became totally blind - both legally and clinically blind, with no light perception (NLP) at the age of 44 after the deterioration of his vision due to retinitis pigmentosa (RP). GB was otherwise healthy. Prior to his training with the EyeMusic SSD, GB had an Argus retinal prosthesis implanted in his left eye. With the implant, he can differentiate between light and dark, detect the location of light, and partially recognize shapes and motion (Second Sight Medical). GB has no detection of color using the prosthesis. He is also assisted by a guide dog. This study received full ethics approval from the Hebrew University of Jerusalem Institutional Review Boards (IRB). The participant reported no other neurological disorders and signed a consent form in the company of an impartial witness.

Training with the EyeMusic SSD

GB met with an EyeMusic trainer twice a week at his home, for around nine months. Each training session lasted one to 3 h. One weekly session was devoted to training with the EyeMusic SSD alone, and the other session included both the EyeMusic and the Argus II prosthesis. GB's training consisted of object identification and counting tasks, accompanied with spatial localization tasks. The translation of the visual image to auditory soundscapes was carried out on a dedicated EyeMusic website or mobile app and played back through a computer (laptop or desktop) or a mobile device (for a full description of EyeMusic training programs, see Buchs et al., 2021). The resolution of the image as depicted by the EyeMusic SSD was 1500 pixels (50px wide, 30px high). In the object identification tasks, we trained GB on the following item categories: clothing, body parts, facial expressions, kitchenware, fruits and vegetables, animals, and familiar people (such as the trainers themselves). Typically, a trainer would place an item in front of a seated GB and he would be asked “what is the item?“. After providing a response, GP would receive feedback from the trainer. In the tasks involving identification of people, one of two trainers would stand in front of GB in one of four positions (see Fig. 1). GB had to identify and locate the trainer, after which he would receive feedback. During the counting tasks, GB would sit at a table with the trainer beside him. The trainer then placed between two to four real-life objects on the table and GB would be asked “how many items are on the table?” (Fig. 1). Under the same procedure, we asked GB to spatially locate the position of a target object on the living room table. Before each trial, the trainer asked GB to report the location of a target (e.g., “where is the banana?“). Similar to the counting task, the number of objects ranged from two to four.

Fig. 1

GB training on a counting task (how many items?) (left), GB training on a localization task (where is the person) followed by an identification task (who is the person) (right).

Training with the argus II prosthesis

GB met with an Argus II trainer once a week for two and a half hours, for several months. With the trainer, he was trained with a kit consisting of black geometric shapes presented on a white background (see Fig. 2). GB would be asked to locate the shape/s, after which he was asked about the shape's position, for example, he would be asked: “what orientation is the shape in?” In addition, he would walk outside with the trainer who would train him to walk along a path while avoiding obstacles.

Fig. 2

GB's training kit for the Argus II retinal prosthesis consists of dark geometric shapes presented on a light background.

Task procedure

There were two types of localization tasks. Each of the two tasks was performed in three different scenarios: (a) we instructed GB to use only the EyeMusic, (b) GB made use of the retinal prosthesis without the EyeMusic, and (c) GB used both the retinal prosthesis and the EyeMusic. In the first task, GB spatially located the position of the trainer in the room. Before each trial, the trainer stood in one of four possible locations (near/far, right/left) inside the living room while GB sat on the couch. During the trial, GB reported the location of the trainer's spatial position. In the second task, GB spatially located the position of a target object on the living room table. Before each trial, the trainer placed two or three real-life objects (e.g., a banana) on the table. The trainer then asked GB to report the location of a target (e.g., “where is the banana?“). Overall, GB participated in 20 trials in the EyeMusic scenario (12 from the first task), 28 trials in the prosthesis scenario (18 in the first task), and 27 in the combination scenario (10 in the first task). It warrants mention that the aim of the quantitative data is limited and meant to provide some credence to the subject's claims and to corroborate his actual abilities. GB was not interested in taking part in a controlled study, and as such, the main focus of the meetings was casual training sessions, in which he provided a phenomenological glimpse into his unique experience through these informal sessions and conversations in his home. The quantitative data cannot be generalized and represents only this unique individual case.

Statistical analysis

We first analyzed GB's correct responses on all three scenarios separately. First, we calculated the appropriate chance level for each scenario based on the mix of trials from the two tasks. The chance level was 31.65% for the EyeMusic scenario, 27.68% for the prosthesis scenario, and 34.89% for the combination scenario. Next, we tested and found that the number of overall trials per scenario would be sufficient for a normal approximation to the binomial (, which allows for a standardized z-test. Per scenario, we performed a one-way z-test on the number of correct responses . We conducted all the above-mentioned analyses using the MATLAB software (MathWorks), statistical tests were corrected for multiple comparisons using False Discovery Rate (⟨ = 0.05).

Results

GB Can Localize Objects and Persons Using the EyeMusic, the Prosthesis, or a Combination of Both. We analyzed GB's success in localization tasks using three different setups (Fig. 2). Correct responses were significantly above chance level for the EyeMusic (), the prosthesis (), and the combination of both the prosthesis and the EyeMusic ().

Phenomenological reports

During a public lecture that he gave, GB described his phenomenology using the Argus II retinal prosthesis as follows:

“I usually don’t say that I see [with the prosthesis]. I can distinguish. Because I once saw, and I have an idea about what is sight, I can’t say that with the Argus, I see.”

On the other hand, when GB speaks of his experience with the EyeMusic SSD, he often refers to the experience as sight, using terminology associated with the phenomenal experience of vision, employing words such as “see” and “look” as follows (Fig. 3):

Fig. 3

Percent of correct responses for the three scenarios. GB's responses in the spatial localization task were significantly above chance when using the EyeMusic (n = 20), the Argus prosthesis (n = 28), and the combination of both (n = 27). Dashed lines indicate the chance level in each scenario, statistical tests were corrected for multiple comparison using False Discovery Rate (⟨ = 0.05).

GB: I’m sure that there is a big difference between one year ago and today. A big difference. It’s not a nuance; I feel it, really. Also, when I recollect looking at the banana [with the EyeMusic] when I recreated it, it was clear to me that what I saw was a banana (see Video 1).

Supplementary video related to this article can be found at https://doi.org/10.1016/j.neuropsychologia.2022.108305

The following is/are the supplementary data related to this article:

Video 1

2

Video 1 (Video 1): GB training with the EyeMusic SSD and then speaking about his phenomenology concerning this training. This video is edited to represent aspects of the training corresponding to the phenomenological description. In the training sessions presented, GB was told that there was an object on the counter, but identification of the object was left to him to determine. The process of identifying -in this case a banana - took GB 43 s.

Moreover, he states that:

GB: I wouldn’t be able to tell you that it’s a green shirt without seeing a green shirt, it can’t happen. It’s not that … I guess that this thing existed in me before, but now after the training, that’s mainly what was strengthened, that when I think about the shape, color, etc. the image comes up in my mind really very fast. There’s no such thing that I would recognize something without seeing it, at the very same second.

In addition to coding for shape and dimension, the EyeMusic also codes for color information in the visual scene. This is important for many aspects of visual experience including figure ground segregation and parsing of visual objects (Ostrovsky et al., 2009). This can be compared to retinal prostheses which to the best of our knowledge currently cannot decode color, though may do so in the future. As such, color experience was one of the most meaningful aspects of GB's use of the device. He describes the experience of integrating prior color knowledge (from the sight he had in his early life) with the EyeMusic. GB further claims that his ability to ‘see sounds’ in color sometimes persists even when not using the device.

GB: I suddenly discovered that when you talk about yellow, I right away see yellow. It’s not obvious. I also see lines and triangles and so on, and red and green.

INTERVIEWER: Does this happen to you only in the colors of the EyeMusic?

GB: It happens more in the colors of the EyeMusic, but if I have to imagine purple, then it is a bit slower, but I can imagine purple [the color yellow is represented by the EyeMusic SSD, while the color purple is not]. I sometimes have to concentrate and see, say, turquoise it will take me a long time but say yellow and any square shape, diagonal line, it is very clear to me that this is its [the EyeMusic’s] job.

INTERVIEWER: Is it nice?

GB: Yes, more than nice it is something very deep.

When asked about color and the Argus II retinal prosthesis (which does not allow for perception of color) GB describes his experience as a form of imagining, as follows:

INTERVIEWER: What do you mean about color and the prosthesis? Because it doesn’t have color.

GB: It’s only imagined. When I know the door is brown, say I asked my daughter what the color of the door is, now if you knock on the door, I will immediately see the door in brown kind of, because I know the door is brown.

GB occasionally describes his perceptual experience with the prosthesis as “flashes”, or a sense of something shimmering, this is an interesting representation of the phosphene patterns electrically elicited by the device.

Standing in front of the door to his apartment:

GB: I can identify the peephole of the door. It’s not that I see a peephole, I get a kind of flash.

INTERVIEWER: What do you mean?

GB: I see like a flash there, I don’t see a round hole … I see these lines, the borders of the door, the borders of the wall, I can stand here and scan and get an idea of the [light] switches and reach out to them.

Standing in front of a decorative tapestry rug on his wall:

GB: I can recognize the borders of the wall, then I can go into the rug, I can’t see its content, I get a lot of indications but not visual, like flashes according to the changes in color, but I can’t see the shapes or something.

GB has a strong desire to regain his independence, to be able to venture beyond the confines of his home independently, unassisted. Though he is usually assisted by a guide dog, he describes a time when he went to the grocery store on his own, aided only by the retinal prosthesis.

GB: I went to the grocery store and I even had to cross a street, and the bionic eye allowed me to see the line, but not always.

INTERVIEWER: Which line?

GB: The line of the hedge, or the line of the curb, it gave me the chance to notice a big garbage container. I went slowly.

INTERVIEWER: How was the experience?

GB: Nice.

INTERVIEWER: How was it at the grocery store?

GB: I asked for milk, and they gave me milk and then I left.

INTERVIEWER: Could you see the shade of the people?

GB: The people? Partly. I can also see part of the shadow of you if I use it. I have to scan. To move the camera.

INTERVIEW: Does it help? To know that there is a person so you don’t bump into them?

GB: Yes, sometimes, but it depends on the color of the background.

INTERVIEWER: And when you compare this to going to the grocery store without the bionic eye, did you walk faster or slower with the bionic eye?

GB: I likely walked faster, and felt safer.

INTERVIEWER: But can you perceive motion? Can you see if something is static or moving?

GB: With the bionic eye? Yes, a little.

A central feature of GB's experience is his ability to compare and contrast his phenomenology with the EyeMusic to that of the Argus II retinal prosthesis. The Argus II allows GB to discern rough shapes and forms, specifically borders and edges, and to orient himself in his surroundings as long as he is “sweeping” the scene by moving his head (not his eyes). He describes this as a cumbersome process. Using the Argus II retinal prosthesis, he cannot perceive the precise, finer details of the visual scene that surrounds him. He claims that the EyeMusic, on the other hand, allows him to recognize subtle details of a visual scene or image on which he focuses his attention.

GB: The [experience with the] bionic eye is different [than the EyeMusic]. In bionic eyes, what develops in my experience is related to a sense of the situation … How my body stands relative to the sidewalk and relative to what surrounds me. My orientation, left, right, west, east. When I walk on the sidewalk, and there’s a hedge on the left, I can recognize that it’s a hedge because it’s darker, pretty quickly the hedge becomes green, and it’s also because of this [pointing to the EyeMusic].

INTERVIEWER: Ok, and do you see the effects of the bionic eye even during times when you’re not using it?

GB: No, no, the bionic eye, maybe it happens, but I don’t feel it. I feel the change in the recall of visual memories due to the EyeMusic. It’s completely clear to me.

GB also describes the possible integration of the Argus II retinal prosthesis and the EyeMusic in a complementary manner. The prosthesis allows GB to differentiate borders and shade differences (lighter/darker) when actively scanning the scene (which involves moving his head consistently from side to side), but he cannot access precise visual details.

Standing in front of a painting:

GB: When I stand in front of the wall [with the retinal prosthesis] when I move from the wall to the painting, I recognize the borders of the painting and then I go with the painting, and I discern these borders, this border between the painting and the wall. Basically, when I run my scanner I can stop on the painting and see its borders, but I can’t see its content. I see it’s borders. I can scan the borders and tell you the top border is here [pointing to the top of the painting].

Standing in front of the door:

GB: I can identify the handle of the door, this line, and all the time I have to scan, yes? If I stop for two seconds, the picture disappears; it just stops working. So I need to scan.

INTERVIEWER: Wow, it’s tiring.

GB: It’s very tiring, that’s why I said that the combination of getting you in my frame [of the bionic eye] and then I can stand, give up on the bionic eye, and hear the details with the musical eye [the EyeMusic].

INTERVIEWER: So, if you don’t move your head, the image still stays?

GB: No, no, I can look at you with the bionic eye, and the image disappears, but I can hear it with the EyeMusic. Using only with the musical eye I would have trouble getting you in the frame. With this [the bionic eye], I can put you in the frame.

Looking out the window GB continues:

GB: I can see the window frame.

INTERVIEWER: Can you know if it’s open or closed?

GB: No but I can distinguish the vertical line. What would the EyeMusic do with a vertical gray line or dark gray?

INTERVIEWER: It depends on the lighting, I don’t know to tell you which color it would be exactly, it depends also on the angle.

GB: If we go out on the balcony … By the way at night the system works better [the retinal prosthesis].

INTERVIEWER: But you turned on the light in your house, so how?

GB: When I turned on the light it gave me better contrasts between black and white, it works on contrasts between black and white. That’s why at night in the dark outside the contrast is bigger, because there is light.

INTERVIEWER: So outside it would work better at night.

GB: Yes, now I can distinguish even buildings that are very far, say this building that I’m pointing to, it’s something I’ve been told like 500–600 m, I can distinguish the buildings. I don’t see anything of them; I can only discern them. I get a wide vertical silhouette, you can understand that, it’s working very weak today, If I go for this building … and I look at it and recognize it with the bionic eye, and after I recognize it, I stay on it, I assume that the musical eye would give me much more detail about windows, about colors, and so on.

Finally, GB likens his experience with the EyeMusic to a new kind of experience altogether, something that he wasn't familiar with in the past, the activation of a dormant ability:

GB: You need to develop a new organ in there. You know what I mean? Like I didn’t have, I’ve been dealing with music for years and literature. This experience [with the EyeMusic] is a whole new space, like you need to activate a different part of the brain that has slept for years.

He also speaks of a positive affective experience with the EyeMusic in particular, stating:

GB: I still don’t understand it, but it [the EyeMusic] generates a very good experience for me, that I don’t completely know how to put into words … I can explain part of the advantage, but it’s more deep. I can’t define it precisely in words, but it’s a powerful experience.

Discussion

When speaking of phenomenology, particularly the phenomenology of a single individual, one must tread carefully. On the one hand, one must be wary of hasty generalizations. Yet on the other, single case studies have provided major steppingstones in the history of psychology and neuroscience, paving the way for significant advances in research. We believe that this case study can provide the foundations for future research, psychological and neurological, philosophical, and not less importantly, in the rehabilitative sciences.

One of the most intriguing features of retinal implants is the fact that they provide immediate visual qualia, while SSDs require many hours of training to elicit visual qualia. This presents one of the main motivations for adopting an expensive and invasive approach such as a retinal prosthesis. However, this case study indicates that the intuitive advantage provided by the retinal prosthesis (specifically the Argus II) is not something that can be taken for granted on its own. People are not always aware that retinal implant patients also need extensive training to perceive meaningful visual percepts, and after passing a certain threshold of training (which remains to be empirically determined), in the case of GB, the pendulum seems to have swung back in favor of the SSD in terms of the qualia he experienced. One advantage of SSDs is the constancy of the perception, which does not fade. Whereas Argus II vision is not sustained, but rather fades in and out. The constancy in perception allowed by the SSD also permits the user to learn more effectively. Another advantage of SSDs is that they make use of a different modality (other than vision) to transfer the information, it may therefore be easier for the user because there is no competing visual input. This is unlike the case of retinal prostheses, where the process of eliciting artificial vision itself may interfere with forming a visual image. GB's experience indicates that the EyeMusic SSD became more transparent to him than the Argus II retinal prosthesis, contributing to the overall ease of use and positive affective experience attributed to the device. GB was both highly experienced and motivated regarding training on both devices, and yet for him, transparency could not be achieved with the Argus II, as he was required to constantly scan the visual scene, moving his head from side to side in an unnatural manner which he refers to as “tiring”.

GB feels that there is an additive advantage to using both devices, and it seems that GB developed unique and beneficial strategies for combining both the SSD and the Argus II retinal prosthesis. It is important to note, that repeating this study with the combination of an SSD and a newer generation or future retinal prosthesis device with higher resolution (due to a higher number of electrodes) is warranted, though this is beyond the scope of the current paper.

This paper aimed to provide a glimpse into the unique phenomenal experience of a blind sensory substitution and retinal prosthesis user. The retinal prosthesis stimulates the retina directly, eliciting grayscale visual perception in the form of phosphene patterns, while the EyeMusic converts the visual scene into soundscapes that also include color information. This case study is a novel and preliminary investigation into the use of both a prosthesis and an SSD by the same individual with strong emphasis on the phenomenology and practicality of each of the devices (alone and in combination). GB was born with vision, providing him with memories of actual visual phenomenal experiences to which he can compare the experience of both the prosthesis and the SSD. He had been deemed totally blind, so he was both legally and clinically blind, with no light perception (NLP) a decade before his experience with the retinal implant and the EyeMusic SSD, thereby ruling out the possibility that visual qualia experienced by him is based on residual vision or is a consequence of the ongoing deterioration of vision. In fact, bare to no light perception is one of the minimal requirements in the screening process for the Argus II retinal prosthesis which was implanted in GB's eye (Finn et al., 2018).

One insight that can be gained with regard to GB's experience relates to the possibility of transparency, the automaticity of what he perceives when using the SSD (Deroy and Auvray, 2014; Kirsch et al., 2020). A transparent SSD experience is one that arises passively without being preceded by a conscious decision process (Kirsch et al., 2020). GB claims to have spontaneous perception of visual, specifically color experience, in association with auditory stimuli (human speech, for example). These perceptions are more pronounced in relation to colors represented by the EyeMusic SSD, thereby seemingly exhibiting transparency. This supports a previous phenomenal report by Ward and Meijer (2010) in which the users reported “seeing sounds” that are elicited automatically in response to certain noises in their environment, specifically in frequencies represented by the vOICe algorithm, even when not using the device (Video 2). Here it seems that the transparency via SSD is even higher than that achieved with over a year use of the bionic eye.

Supplementary video related to this article can be found at https://doi.org/10.1016/j.neuropsychologia.2022.108305

The following is/are the supplementary data related to this article:

Video 2

3

This kind of automaticity of the perception leads to the question of possible idiosyncrasy in the nature of the stimulus experienced by the user. The SSD soundscape and algorithm, as they are predefined, limit the possibility of idiosyncrasy (Kirsch et al., 2020) (for example, in the case of the EyeMusic SSD, each color is mapped to a specific sound timbre). Despite this, GB claims to perceive automatically elicited colors not depicted in the EyeMusic. Interestingly, one of Ward and Meijer's (2010) vOICe SSD users also claimed to develop color qualia after extensive use of the SSD, despite the vOICe's lacking a color dimension in its algorithm. Taken together, these reports would seem to indicate relative idiosyncrasy of the experience associated with long-term SSD use, though clearly more cases must be explored in this regard.

Following from the findings presented in their report, Ward and Meijer suggested that continued use of sensory substitution devices can bring about a form of acquired synesthesia (Ward and Meijer, 2010), defined as the perception of one sense, brought about by and coinciding with the physical stimulation of another sense (Proulx and Stoerig, 2006). Acquired synesthesia can be attributed to the internalization of a system of rules for linking between audition and vision resulting from long-term neuroplastic changes in the brain (Ward and Meijer, 2010; Ward and Wright, 2014). According to the classic definition of synesthesia, the inducing stimulus, in this case of auditory origin, triggers a concurrent visual stimulus. This indeed seems to match GB's descriptions, as he claims that the content of a person's speech, such as verbally mentioning or naming a specific color (GB describes seeing yellow when he hears talk of the color yellow) can elicit a visual experience of color, meaning he has concurrent access to the substituting sense and the substituted.

Yet it has been argued that the acquired synesthesia interpretation is countered precisely by the automaticity of the SSD use, as automaticity represents a blurring or breakdown of the borders between the senses, rather than the simultaneous perception of them both. If the experience is transparent, it is not synesthetic, and vice versa (Kirsch et al., 2020). Another criterion of synesthesia not fulfilled by GB's experience is that of consistency. GB describes his perceptual experience as developing and enriching over time rather than remaining consistent, as is the experience in synesthetes (Video 2).

GB's phenomenal experience also shines the spotlight on a greater quandary, the question of whether his experience is visual perception or visual mental imagery. When using the retinal prosthesis, GB attributes any color experience to his imagination. But interestingly, with regard to the EyeMusic GB wonders about the distinction himself, as he references imagining with relation to the color phenomenology he experiences when not using the SSD. The question of color perception and blindness is a time old one. It was widely held that color qualia exemplifies the explanatory gap between the objective and the subjective (Levine, 1983; Jackson, 1986; Chalmers, 2003), as the blind cannot conceptualize color (Nagel, 1974) and people who have not been exposed to color are missing a fundamental aspect of the color experience (Jackson, 1986). Yet recent research begs to differ, specifically regarding the paradigm of sensory experience discussed by the empiricists, color. Research indicates that although there are differences in the color knowledge of the blind and the sighted (Connolly et al., 2007; Kim et al., 2021), there is a deeper understanding of color that they share (even in cases of congenital blindness) (Kim et al., 2021). Such research is paving the way for SSDs that incorporate dimensions of color such as the EyeMusic, and the SoundSight, another up and coming SSD (Hamilton-Fletcher et al., 2021). Moreover, research on color involvement in synesthesia indicates that there is a minimal experiential color “circuit” underlying color perception, involving awareness, affect, and memory, which is further enhanced and expanded in cases of cross modal color processing such as that experienced by people with synesthesia (Eagleman and Goodale, 2009). GB is adamant about perceiving true color qualia due to his SSD use (Video 3), thus strengthening the idea that color information can be transmitted through alternate modalities. But is this visual perception or mental imagery?

Supplementary video related to this article can be found at https://doi.org/10.1016/j.neuropsychologia.2022.108305

The following is/are the supplementary data related to this article:

Video 3

4

The classic definition of mental imagery is of quasi-perception rather than perception per se (Thomas and Zalta, 2021), as it is associated with perceptual information in the absence of relevant external stimuli (visual input) (Kaski, 2002; Thomas and Zalta, 2021). This debate rages on concerning synesthesia, which has, on the one hand, been described as multimodal mental imagery, and on the other hand as perception (Ward, 2013; Nanay, 2020). Yet the strict distinction between perception and mental imagery as a whole is now being challenged, with current research indicating that they cannot be disentangled as easily as once was thought (Kosslyn et al., 1993; Dijkstra et al., 2019; Kirsch et al., 2020). It was once suggested that the activation of visual areas in the brain correlated with the soundscapes produced by SSDs was in fact the result of mental imagery. Yet for this to be so, the activation would need to be caused by different underlying mechanisms (Amedi et al., 2007). In contrast, there is now an extensive body of research indicating that perception and imagery are based on similar neural mechanisms (Kosslyn et al., 1993; Dijkstra et al., 2019), and it is suggested that both externally stimulated and internally stimulated forms of visual experience (the former commonly known as perception and the latter mental imagery) rely on similar processes (Kosslyn et al., 1993; Mitchison, 1996; Dijkstra et al., 2019). For example, the mental imagery underlying the artistic abilities of a blind painter has been correlated with the activation of visual areas in his brain, overlapping the same areas that the sighted use for processing what they visually perceive (Amedi et al., 2011). Perhaps, in accordance with Occam's razor “plurality should not be posited without necessity” with regard to mental imagery and visual perception, yet this is clearly a matter that warrants further inquiry.

It must be noted that there is a commonplace association between imagery and voluntary action (Thomas and Zalta, 2021) that is not upheld in the case of GB, as he asserts the involuntary elicitation of visual color experience, and its automaticity. Moreover, GB interprets his phenomenal experience using the SSD as visual perception and relates to it as such. For example, he states straightforwardly that since using the SSD “I have more visual experience than one year ago.” An additional characteristic of true perceptual experience is the perceiver's attributing emotional value to the experience (Auvray and Myin, 2009). The perception GB acquires using the SSD elicits clear emotional responses from him; he speaks with enthusiasm about the “deep” aspects of the felt experience and the pleasure associated with visualizing the finer details of an image or scene. This positive affective response can possibly be attributed to the activation of systems associated with memory, as GB claims that his ability for “retrieving visual memories” is improved after training with the SSD, and that “it's clear that the information is extracted from the memory and flows through the brain's wires more quickly.” Or possibly it can be attributed to activation of the mirror system, since part of GB's training with the SSD involved facial expression recognition and mimicking of expressions and gestures (Video 3). Clearly these hypotheses remain to be tested both behaviorally and by employing neuroimaging methods.

In light of the aforementioned, GB's reports simultaneously strengthen deference views and weaken dominance views regarding the experience's being correlated with vision. This builds on previous phenomenal reports in support of the visual qualia interpretation of SSD use, for example, a congenitally blind EyeMusic SSD user was quoted as saying that “I have the feeling that I'm actually seeing this thing, it's a weird feeling, it's very new” (Amedi, (unpublished), p.3; Pence, 2020). When speaking of the subjective aspect of experience - the visual qualia - it has been suggested that one should accept that the SSD user is the “epistemic authority”, particularly when the individual can rely on her or his own past visual experience in comparison (Pence, 2020). GB can relate to and identify true visual experience from the period prior to going blind, and as such can be trusted in his introspective assessment (Pence, 2020).

Moreover, GB's phenomenology provides support for stances that suggest abandoning our classical definitions of sensory modalities altogether and adopting a new perspective on specialization in the brain (Reich et al., 2012; Deroy and Auvray, 2014; Heimler et al., 2015). GB, like the congenitally blind EyeMusic user quoted previously, indicates that he experiences something new going on, “a whole new space,” which he ties to the activation of a part of the brain he feels lay dormant. This backs up the idea that the canonical sensory modality interpretation of the brain warrants revision. As neuroscientist David Eagleman states “You can actually plug in completely new kinds of data streams. The brain will say, ‘Oh, oh, I get it! It's correlated with reward or with this’ and it figures out how to use it” (Mason, 2020, 00:17).

A viewpoint that seemingly accounts for this type of phenomenology and has been tied to sensory substitution is the task specific sensory independent interpretation, alongside similar interpretations such as the metamodal theory of brain organization (Pascual-Leone and Hamilton, 2001) and the sensory independent supramodal interpretation (Cecchetti et al., 2016; Kupers and Ptito, 2011). The task specific sensory independent interpretation posits that while brain regions show a preference for information related to a particular sense, the higher-order areas are still capable of performing their designated task if relevant information is transmitted to them through an alternative sense (Reich et al., 2012; Heimler et al., 2015). It follows from this view that the brain is not structured in a sensory modality-dependent manner to begin with (Ptito et al., 2008; Reich et al., 2012; Heimler et al., 2015). In line with the task specific sensory independent interpretation among others, sensory substitution utilizes and possibly even enhances neuroplastic mechanisms which involve the unmasking of pre-existing connections, present before SSD use, between what were commonly considered visual and non-visual areas (Amedi et al., 2005; Ptito et al., 2008). The aforementioned is not restricted to the visual-to-auditory domains. There is a growing body of research on visual-to-tactile (Ptito et al., 2005) and auditory-to-tactile (Eagleman, 2020; Perrotta et al., 2021) SSDs which points to similar conclusions.

Finally, GB's experience provides a unique opportunity to consider the potential of combining visual prostheses with SSDs. On the one hand, retinal prostheses alone provide immediate visual percepts by way of electrically stimulated phosphenes elicited by the device, which can allow for an immediate psychological gain. SSDs on the other hand, require tens of hours of training, spread out over weeks or even months, before visual percepts are elicited. But following an extensive training program with the EyeMusic SSD, GB expressed that the SSD alone allowed him for a richer, more complex perceptual experience. According to GB's subjective interpretation, a synthesis of both systems could provide a notable benefit to the blind (Video 2), beyond the gain provided by either of the devices alone. He feels that the Argus II prosthesis allows him to perceive the overall ‘gist’ of the visual surroundings and get a feeling for his orientation within the visual scene, while the EyeMusic SSD allows him to perceive the rich, more intricate details of the visual experience and possibly provides him an ongoing developmental gain via neuroplasticity. The benefit afforded by the combination of these devices may be even more pronounced if SSDs will be combined with future more advanced, newer generation, retinal prosthesis devices. This combination would potentially achieve an important milestone and cross a highly coveted threshold, that of genuinely improving the quality of life of the blind.