Multisensory stimulation decreases phantom limb distortions and is optimally integrated.

The multisensory integration of signals from different senses is crucial to develop an unambiguous percept of the environment and our body. Losing a limb causes drastic changes in the body, sometimes causing pain and distorted phantom limb perception. Despite the debate over why these phenomena arise, some researchers suggested that they might be linked to an impairment of multisensory signals inflow and integration. Therefore, reestablishing optimally integrated sensory feedback could be crucial. The related benefits on sensory performance and body self-representation are still to be demonstrated, particularly in lower-limb amputees. We present a multisensory framework combining Virtual reality and electro-cutaneous stimulation that allows the optimal integration of visuo-tactile stimuli in lower-limb amputees even if nonspatially matching. We also showed that this multisensory stimulation allowed faster sensory processing, higher embodiment, and reductions in phantom limb distortions. Our findings support the development of multisensory rehabilitation approaches, restoring a correct body representation.

Introduction

Humans perceive the environment through multiple senses, which often provide them with redundant information. The brain uses this redundancy as an advantage and merges the information arriving from different sensory modalities into a robust unambiguous perception (Ernst and Banks, 2002) through a mechanism of multisensory integration. The optimal integration theory provides a theoretical framework to describe this phenomenon of integration. It states that the best way to interpret a stimulus is by performing a weighted estimation between the sensory cues, where the weights should be proportional to the reliability of the cues (Ernst and Banks, 2002). This mechanism of integration is defined as optimal because it produces not only lower but the lowest-variance estimate that can possibly be achieved by the Nervous System.

Noteworthily, the process of integration is not merely related to the perception of external stimuli, but it is also considered as the basis for important aspects of bodily self-consciousness such as body ownership (Blanke et al., 2015). Indeed, when facing an amputation, the body encounters dramatic changes (Makin et al., 2010). The loss of somatosensory (i.e., tactile and proprioceptive) sensations implies the loss of one of the most immediate and influential sources of multisensory information, thus compromising the individual’s ability to recognize, grasp, and manipulate objects (Jacobsen et al., 1986). Amputation may further lead to the emergence of debilitating symptoms, such as phantom limb sensations and pain (Ehde et al., 2000; Kikkert et al., 2018).

Therefore, the restoration of artificial sensory feedback, which allows a multisensory integration with the residual sensory channels, is crucial for amputees. Recent studies with upper-limb amputees yielded encouraging results demonstrating the optimal integration of artificial sensory feedback with visual information (Dadarlat et al., 2015; Marasco et al., 2018; Risso et al., 2019). It is important to note that the artificial somatosensory signal differs in important ways from the natural signal occurring when touching an object. For example, an important feature of natural perception is that a stimulus delivered to the body is perceived simultaneously (temporally matched) and in the same localization (spatially matched) in all the different sensory modalities. Previous research has shown that redundant information from different sensory modalities should be spatially correlated for integration to occur (Dadarlat et al., 2015; Ernst and Banks, 2002; Welch and Warren, 1980). In this study, the multisensory information was perceived simultaneously. However, the electro-cutaneous signal was perceived on the residual part of the stump closer to the missing limb (i.e., the thigh), whereas the visual information was presented virtually at the foot. Thus, the multisensory information provided to the participants was not spatially matched. Furthermore, the functional and cognitive advantages that can be derived from the use of a system allowing for the optimal integration of sensory information are still to be explored. In particular, such mechanisms have not been reported in the lower limbs, which have different tactile receptor densities and placements, together with different cortical representation compared to the upper limbs.

At lower levels of cognition, the expected benefit of optimal integration is an improvement of sensory information processing, leading to higher perceptual performances (e.g., lower discrimination thresholds, faster reaction times) (Alais and Burr, 2004).

At a higher level of cognition, thanks to the interplay between multisensory integration and self-body experience, such a beneficial stimulation has been proposed to reduce altered body representation in amputees (Boesch et al., 2016; Martini, 2016; Preatoni et al., 2021; Rognini et al., 2018).

Here, we developed a multimodal therapeutic framework to investigate the optimal multisensory integration in lower-limb amputees, and in parallel we investigate the potential benefits of such stimulation on information processing and on distorted phantom limb sensations (PLS).

We combined two sensory channels: vision through Virtual Reality (VR) and touch through Electro-Cutaneous Stimulation (ECS) (Video S2). These channels were used to perform a two-alternative forced-choice task either in unimodal (visual or tactile) or bimodal (visuo-tactile) conditions. Their integration was investigated by implementing a rigorous psychophysics model assessing their optimal (or Maximum Likelihood; MLE) estimate (Ernst and Banks, 2002). Furthermore, we tested the hypotheses that artificial multisensory stimulation decreases the participants’ reaction times in a discrimination task and allows a reduction of phantom limb distortions.

Results and discussion

Optimal multisensory integration in lower-limb amputees

To assess the optimality of visuo-tactile integration, we adapted the procedure used in Ernst and Banks’ 2002 study (Ernst and Banks, 2002), which proposed a method to describe the mechanism of optimal integration using a statistical approach. The authors demonstrated that humans integrate multisensory information in a statistically optimal fashion by performing a weighted estimation between the available sensory cues. Accordingly, the sensory reliabilities of the unisensory cues can be assessed, manipulated, and then be used to compute a mathematical prediction of optimal behavior. The predicted bimodal values correspond to the lowest-variance estimate possible, given the reliability of the unimodal sensory signals. Thus, i) the predicted bimodal value can be compared with the observed bimodal value to check whether the prediction of optimality is verified; ii) the reliability of the available sensory modalities can be manipulated to check that the bimodal thresholds change accordingly (Ernst and Banks, 2002; Risso et al., 2020). Subjects in this study performed a two-alternative forced-choice task (2-AFC), where they judged which of the two virtual bars presented consecutively (i.e., one standard and one comparison stimulus), was perceived as vibrating faster (Figure 1A; Video S1). The participants can perceive the stimuli through one sensory modality (visual or tactile) or multisensory (visually and tactually simultaneously; Figure 1B). In natural conditions, the visual cue often dominates over the tactile one, and when one modality is much more reliable than the other, the optimal integration principles prescribe almost the exclusive reliance on the most reliable modality. Accordingly, the predicted behavior of integration is compatible with both the optimal integration hypothesis and the winner-take-all hypothesis, in which one modality dominates over the other (see section Experimental Procedure for more details). Accordingly, following other studies (Ernst and Banks, 2002; Risso et al., 2020), we included a condition in which the vision was blurred in order to be reliable as much as the tactile cue. The responses of the participant in each unisensory and multisensory condition were fitted with a cumulative normal probability distribution using maximum likelihood estimation to obtain a psychometric function representing the probability of judging the comparison stimulus as larger than the standard stimulus. The Likelihood Ratio statistics were significant for all psychometric functions (p < 0.001), indicating a good fit with the data (Figure 1C). For each psychometric curve, we computed the point of subjective equality (PSE), i.e., the period of vibration of the bar that was perceived as faster than the standard in 50% of the trials. We also computed the discrimination threshold — or Just Noticeable Difference (JND) (i.e., the smallest difference between two stimuli that can be reliably discriminated (Ernst and Banks, 2002)) — as the difference between the PSE and the period of vibration that is perceived to be faster than the standard in 84% of the trials (see Supplementary Information for more details on PSEs and JNDs, see Table S3). The visual JND in the no blur situation (subj 1 JNDV = 0.11; subj 2 JNDV = 0.04) was significantly smaller (subj 1 p < 0.001; subj 2 p = 0.02) than that in the tactile condition (subj 1 JNDT = 0.28; subj 2 JNDT = 0.10), confirming that in this natural condition, the vision gave a more reliable estimate of the bars’ vibration than the somatosensory channel. As expected, the bimodal JND (subj 1 JNDVT = 0.11; subj 2 JNDVT = 0.05) was not significantly different from the most reliable visual cue and not significantly different from the predicted cue (subj 1p > 0.99 for both JNDs, subj 1 p = 0.77 for both JNDs). If one cue is more reliable than the other, optimal integration principles give a very large weight to this cue (Ernst and Banks, 2002). To check whether the margin of improvement predicted by the MLE followed the weights of the unisensory cues, we manipulated the reliability of the cues and added blurring to the visual condition. The improvement predicted by the MLE was maximized because the blurring meant there was not a dominant modality and the cues had almost the same weights, i.e., the visual JND (subj 1 JNDVB = 0.51; subj 2 JNDVB = 0.06) was not significantly different from the tactile one (subj 1 p = 0.07; subj 2 p = 0.13; see Figure S2). In line with the theory, the bimodal JND (subj 1 JNDBM = 0.16, subj 2 JNDBM = 0.03) was significantly smaller than the unisensory JND (subj 1 and subj 2 p < 0.01) and not significantly different from the prediction (subj 1 JNDMLE = 0.25; p = 0.07; subj 2 JNDMLE = 0.05; p = 0.21) (Figure 1D).

Figure 1

Optimal multisensory integration in lower-limb amputees

(A) Experimental set-up. The subject is sitting on a chair with his prosthesis leaning straight on a stool while performing the Two-Alternative Forced Choice (2-AFC) task. He is equipped with a head-mounted display that immerses him in a virtual scenario. The electrical stimulator delivers the tactile stimuli on his stump.

(B) Experimental protocol, consisting of five conditions: T = tactile; V = visual; VB = visual-blurred; VT = visuo-tactile; VBT = visuo-blurred-tactile.

(C) Psychometric curves: the x axis represents the bar’s vibrating period (in s). The y axis corresponds to the proportion of responses in which the comparison stimulus was larger than the standard. The Likelihood Ratio statistics were significant for all psychometric functions (p < 0.001), indicating a good fit with the data The vertical dashed line that corresponds to a proportion of 50% of ‘faster’ responses indicates the Point of Subjective Equality (PSE). The distance between the two vertical dashed lines corresponds to the Just Noticeable Difference (JND), which is the difference between the PSE and the bar’s vibrating period that is perceived to be faster than the standard in 84% of the trials JND and PSE are reported on the bottom right of the plot. Upper row: subject 1; Lower row: subject 2.

(D) The vertical bars represents: Single subject unimodal (UM; visual blurred and tactile mean performance) and bimodal (BM; visual-blurred-tactile performance) JNDs and the predicted (MLE = maximum likelihood estimation) JND for the bimodal condition (mean ± MAD). Horizontal bars denote statistically significant differences (False Discovery Rate adjusted bootstrap p value ∗p < 0.05 and ∗∗∗p < 0.001). Upper row: subject 1; lower row: subject 2. Left column: no blur condition. Right column: blur condition.

Multisensory integration benefits on sensory performance

After assessing the optimal integration of the two sensory channels, we explored the benefits of the same visuo-tactile stimulation in a sensory discrimination task (Figure 2A). Subjects had to discriminate which of two stimuli presented simultaneously in two different stump locations was perceived as vibrating faster (See Supplementary Information and Experimental Procedure Section). In the unimodal condition (UM), the individuals had to use vision, whereas the visual cues were coupled with the ECS stimulation in the bimodal condition (BM). The reaction times (RT) were significantly lower in the BM condition compared to the UM condition, both for subj. 1 (RTum = 3.4835 ± 1.56, RTbm = 2.2693 ± 1.39, p < 0.001 Wilcoxon signed-rank test) and subj. 2 (RTum = 1.67 ± 0.79, RTbm = 1.21 ± 0.47, p < 0.001 Wilcoxon signed-rank test). A mixed-model ANOVA between stimuli and condition showed a main effect of the condition in both subjects (F[1,174] = 38.41, p < 0.001 for subj.1; F[1,180] = 14.86, p = 0.003 for subj.2), whereas the interaction was significant for subject 2 (F[9,180] = 3.89, p < 0.001) but not for subject 1(F[9,174] = 1.32, p = 0.22). The difference in RT for each stimulus between the two conditions (Figure 2C), suggests a functional benefit of the BM condition that is spread across the whole range of vibration frequencies (i.e., not concentrated in a particular group of stimuli). Thus, even when the task was easier (i.e., comparison between the slowest or fastest stimuli with the reference one), the BM condition still provided perceptual benefits compared to the UM condition. In addition, the accuracy of the subjects in the UM and BM conditions (Figure 2D) was not statistically different (Wilcoxon signed-rank test, p = 0.75 for subj. 1 and p = 0.25 for subj. 2), ruling out the possibility that the outcome of the RT measurement was affected by a speed-accuracy trade-off (i.e., reacting quicker at the price of a higher number of errors).

Figure 2

Multisensory integration benefits on sensory performance (A–C) and multisensory stimulation benefits investigation on embodiment and phantom limb representations (E–I)

(A) Protocol. UM, unimodal; BM, bimodal.

(B–D) Results for the discrimination task. Upper row: subject 1. Lower row: subject 2. (B) Reaction times. Boxplots of reaction time (s) for UM (blue) and BM (purple) (mean ± std) (p value ∗∗∗p < 0.001, Wilcoxon signed-rank test). (C) Difference (s) of reaction times between UM and BM. (D) Accuracy (%) of UM (blue) and BM (purple) (mean ± CI) (Wilcoxon signed-rank test).

(E) VR Environment. The subject sees his legs (both intact) and one foot is intermittently touched by an incoming wave.

(F) Experimental conditions. Synchronous (blue, left) and Asynchronous (red, right).

(G) Embodiment questionnaire results. Blue bars show the synchronous condition and red bars show the asynchronous condition (mean ± std) (p value ∗∗p < 0.01, Wilcoxon signed-rank test).

(H) Results for the telescoping measurement. Plots show the mean and standard deviation. Blue bars show the synchronous condition and red bars show the asynchronous condition. The dashed line indicates the total length of the leg – stump (i.e., length of the phantom without telescoping). (mean ± std) (p value ∗∗p < 0.01, Wilcoxon signed-rank test).

(I) Results for the proprioceptive displacement measurement. Blue bars show the synchronous condition and red bars show the asynchronous condition. VR= virtual reality; Sync= synchronous; Async= asynchronous; Emb= embodiment; UM= unimodal; BM= bimodal.

Multisensory stimulation benefits on embodiment and phantom limb representations

We then investigated whether this multisensory artificial stimulation could positively influence the virtual leg (Figure 2E) embodiment and PLS in three transfemoral amputees (an additional subj. 3 was recruited). The temporal congruency of multisensory signals is a fundamental feature for perceptual integration to happen (Dadarlat et al., 2015; Ernst and Banks, 2002; Welch and Warren, 1980). Accordingly, we manipulated the temporal synchronization between tactile and visual stimuli to provide two different experimental conditions: a synchronous stimulation condition, which was coherent with the setup of Experiment 1, and a control asynchronous stimulation condition, in which the tactile and visual stimuli were not simultaneous. The asynchronous condition consisted of a phase shift (5 s) between the beginning, peak, and end of the tactile and visual stimulus (Botvinick and Cohen, 1998; Maselli and Slater, 2013) (Figure 2F). Thus, we tested whether the synchronous stimulation provided through our visuo-tactile stimulation platform would lead to a higher embodiment of the virtual leg and could reduce the distorted body representations in amputees. The results show that in the synchronous condition the subjects had a significantly higher embodiment (Embsync = 1.0833 ± 0.66, p = 0.0078 Wilcoxon signed-rank test) compared to the asynchronous condition (Embasync = −1 ± 0.522) (Figure 2G). In addition, the control questions were not significantly different, allowing us to rule out the suggestibility of the subjects to the paradigm (Figure 2G). These results are comparable to those of healthy control subjects, who also showed a significantly higher (p < 0.001) embodiment in the synchronous condition compared to the asynchronous one (Figure S3). The measurements of distorted phantom limb perception were performed in Subject 2 and Subject 3, who reported perceiving an altered phantom limb feeling. In particular, after the synchronous multisensory stimulation, Subject 2 reported his shortened phantom limb as significantly longer (more similar to the length of the healthy limb) (Lengthpost = 28.9 ± 2.9 cm, p = 0.0057 Wilcoxon signed-rank test; statistical power = 0.98) compared to the pre-intervention assessment (Lengthpre = 24.7 ± 1.8 cm) (Figure 2H). This phenomenon was absent in the asynchronous condition (Lengthpre = 23.9 ± 3.9 cm; Lengthpost = 23.5 ± 2.2 cm). On the other hand, Subject 3 had a significant decrease (p = 0.0317 Wilcoxon signed-rank test; statistical power = 0.84) of the proprioceptive displacement (the spatial difference between the position of the phantom hallux after the multisensory paradigm and the real position of the prosthesis) only after the synchronous condition (Displpre = 4.02 ± 2.2 cm; Displpost = 0.92 ± 0.9 cm) (Figure 2I). This phenomenon was not observed after the asynchronous condition (Displpre = 3.68 ± 1.7 cm; Displpost = 4.9 ± 1.04 cm).

Discussion

We combined immersive digital technology and electro-cutaneous sensory feedback and demonstrated with a psychophysical method that two lower-limb amputees optimally integrate visuo-tactile artificial feedback. We have shown with a carefully designed technological set-up, allowing the maximum control over the manipulated variables (time and location of the stimulation), that it is possible to achieve the optimal integration processing of visuo-tactile information similar to how it occurs naturally in the intact nervous systems (Ernst and Banks, 2002).

Then, we showed that this stimulation led to benefits not only in the lower cognitive process, allowing the subjects to shorten the sensory processing time in a discrimination task, but also in higher ones, increasing the embodiment of the virtual leg in all the three participants and when reported, reducing their phantom limb distortions. These results add knowledge to previous studies investigating multisensory integration mechanisms (Christie et al., 2019; Dadarlat et al., 2015; Ernst and Banks, 2002; Marasco et al., 2018; Risso et al., 2019). We demonstrated the possibility of achieving an optimal integration in lower-limb amputees with a noninvasive and remapped sensory feedback approach, in contrast to invasive approaches (Dadarlat et al., 2015; Petrini et al., 2019a; Risso et al., 2019). In addition, we showed that this stimulation leads to measurable benefits in cognitive processing. Our findings confirm previous evidence from upper-limb amputees that encourage the adoption of multisensory stimulation approaches for the treatment of phantom limb distortions (Rognini et al., 2018).

Related to the occurrence of optimal integration in presence of remapped (i.e., somatosensory information provided on the thigh instead of the foot), therefore spatially mismatched signals, it is important to note that it is not a foregone result. In cognitive neurosciences, long-standing interest has been given to the roles of various bottom-up and top-down factors which influence multisensory perception in humans (Stein and Meredith, 1993; Ernst and Bülthoff, 2004; Welch and Warren, 1980). Seminal studies on the intersensory discrepancy stated that spatial congruence between the multisensory cues is a fundamental feature for perceptual integration to happen (Welch and Warren, 1980). In this study, the sensations elicited on the thigh by the ECS stimulation were not spatially congruent with the artificial image of the limb being stimulated on the foot. Nevertheless, the multisensory information was optimally integrated by the participants. In this preliminary study, we showed that optimal integration in amputees can also happen in peculiar conditions where the spatial localization of the sensory information does not overlap, such as those provided by our immersive virtual reality experience. To interpret the result, we refer to the growing literature that has been investigating top-down factors modulating the merging of multisensory cues over the last decade (Chen and Spence, 2017; Feldman, 2013; Senkowski et al., 2008; Welch and Warren, 1980). One of such high-level factors is the Unity Assumption (Chen and Spence, 2017), which has been proposed as a crucial mechanism used by the human brain to solve the problem of binding single modality inputs (Feldman, 2013; Senkowski et al., 2008). The assumption states that the processing of multisensory cues is modulated by the observers’ belief that the unisensory cues belong to the same object or event (Chen and Spence, 2017). Given this, the optimal integration observed in the current study may indicate that participants were likely to believe that both the visual and tactile cues belong to a unique object, possibly their own leg. This process is fundamental for amputees to perceive a prosthesis as part of their bodies (Raspopovic, 2020). Indeed, researchers have shown that stronger feelings of embodiment (Makin et al., 2017) arise not through motor or somatosensory processing alone but through multisensory integration of visual, tactile, and proprioceptive information (Blanke et al., 2015). Accordingly, in this study, we found an increased embodiment of the virtual leg when the multisensory cues were provided synchronously compared to a conflicting condition. In addition, we found that the multisensory stimulation allowed the amputees to reduce their phantom limb distortions, suggesting the feasibility of a “cognitive multimodal neuroprosthesis,” which could solve such conditions through therapy sessions (Raspopovic et al., 2021). Notably, previous studies have found that subjects with a disturbed body integrity react differently to these types of multisensory stimulation (Ehrsson et al., 2008; Lenggenhager et al., 2015). Future studies should also test healthy subjects in conditions of not spatially matched stimulation to better understand whether this integration is achieved because the amputees are “more plastic” in their body representation or because of the immersive experience provided by this multimodal platform.

Taken together, our results shed light on the mechanisms underlying optimal integration and on the possibilities of noninvasive approaches for multisensory stimulation, opening up relevant opportunities toward the perceptual rehabilitation (Raspopovic, 2021) of neurologically disabled individuals.

Limitations of the study

Even though the results seen in this study provide initial interesting results, there are several important limitations. Surely, the sample size is one of these. Given that the evidence found comes from only a few amputees, it can not be extended to the general population. Future studies should examine these aspects with a larger sample size to explore its repeatability.

In addition, the results seen in the two experiments can not be directly linked together. In experiment 1, the participants perceived bar’s vibration-related features, whereas in experiment 2 they were in a different VR scenario (i.e., different visual information), and the vibration information was modulated in intensity. Therefore, we cannot state with certainty that the visuo-tactile stimulation in experiment 2 was optimally integrated as in experiment 1. Nevertheless, both experiments combined in the same participants, the same sensory channels in the same locations using the same platform, surely relating the results to a multisensory integration.

Moreover, it would be interesting to investigate in patients affected by chronic pain conditions the extent to which an optimally integrated multisensory stimulation can provide therapeutic benefits. Finally, the effect of training in the likelihood of optimal integration of artificial sensory signals would be an interesting topic for future research. Indeed, the participants reported in this study have already exploited the remapped artificial stimulation in combination with a prosthetic device (Basla et al., 2021). This previous experience might influence the sensory integration process. Indeed, when referring to the MLE model, the integration of sensory information is mostly described as a bottom-up process. Prior knowledge is important to interpret ambiguous sensory information as considered by the Bayesian inference framework (Ernst and Bülthoff, 2004). Accordingly, future studies should investigate the effect of sensory training on remapped feedback not only in amputees but also in healthy subjects.

STAR★Methods

Key resources table

Resource availability

Lead contact

Further information and requests for resources should be directed to the lead contact, Stanisa Raspopovic (stanisa.raspopovic@hest.ethz.ch).

Materials availability

This study did not generate new unique reagents.

Experimental model and subject details

Subjects

Three subjects with unilateral lower-limb amputation at transfemoral level (Table S1) participated in this study. All amputees were male and were respectively 31, 54, and 35 years old. The subjects had undergone a traumatic amputation because of accidents in 2018, 2013, and 2007. All the three subjects already exploited a remapped sensory feedback system on their prostheses in some motor tasks (Basla et al., 2021). Because the amount of time necessary for the experiments did not comply with his job, Subject 3 preferred not to participate in experiment 1. As a control condition for Experiment 2, 12 healthy subjects were recruited. All subjects read and signed the informed consent. This study was approved by the ETH Ethical Commission (EK-2019-N-97).

Method details

Optimal multisensory integration assessment

The model

In 2002, Ernst & Banks proposed a model that describes in a statistical sense a mechanism of optimal integration (Ernst and Banks, 2002). This optimal integration model states that the benefit of multisensory integration derives from a reduction in the variance of the final perceptual estimate. Indeed, according to the Maximum Likelihood Estimation (MLE), the final integrated estimate SVT derived by visual (SV) and tactile (ST) cues corresponds to a weighted average between the individual sensory signals:where the weights WV and WH sum up to unity (WV + WH = 1; [2]) and should be proportional to the reliability of the stimulus:where the reliability (R) is simply the variance, that is the inverse of the noise of the corresponding cue:

Assuming that the unimodal cues are independent and normally distributed the variance of the optimal MLE estimate integrating the cues SVH corresponds to the lower than that of the single cue:

It is easy to show that the improvement brought by the integration depends on the distribution of the weights among the available sensory modalities. Indeed, when one modality is much more reliable than the other, the optimal integration principles prescribe almost the exclusive reliance on the most reliable modality. In natural conditions, the visual cue often dominates over the tactile one. Accordingly, the margin of improvement predicted by the MLE model is reduced, and the predicted behavior of integration is compatible with both the optimal integration hypothesis and the winner-take-all hypothesis, in which one modality dominates over the other. Conversely, when the reliabilities of the two sensory cues are equal, the improvement brought by the integration is maximal and the difference between the integrated estimate and the single modalities should be more clearly distinguishable. Accordingly, following other studies [1,7,25], we included a condition in which the vision was blurred to avoid the exclusive reliance on the visual modality and to observe the benefit of optimal integration in the condition where such a benefit is maximized.

Optimal multisensory integration calibration procedure

Two couples of superficial electrodes of 1 cm2 were placed on the stump in the most distal way allowed by the prosthesis socket (i.e., without being compressed by it). One pair was attached in the anterior part, and the other in the posterior part of the stump. The couples were approximately 1 cm distant from each other. Right before the experiment, the subjects underwent a calibration procedure to find the proper intensities for the electrical stimulation. The subjects were instructed to aim for a sensation very precise in space (i.e., not spread), with a vibration-like perception (Figure S1). They were also instructed to avoid in any way pain and muscle contractions. The same procedure was done for healthy subjects (except for the stimulation site which was on the foot, i.e. somatotopically matched).

The stimulation was increased in terms of amplitude while keeping pulse-width fixed at 40μs and the frequency 4.1 Hz. The trains of biphasic-balanced stimuli lasted 2 s and were paused for 1 s. Every iteration, the amplitude increased of 1 mA, starting from 2 mA. As the perception is subjective and depends on the stimulation intensities, the resulting parameters were: for the frontal channel 13 mA (Subject 1), 15 mA (Subject 2), and 30 mA (Subject 3); for the back channel 22 mA (Subject 1), 17 mA (Subject 2), and 22 mA (Subject 3) (Table S2).

Once the subject reported the sensation for which he was previously instructed to aim, the pair of electrodes would be changed to the other channel, and the paradigm repeated for the remaining channel.

The stimulation was performed using a RehaMove3 device (Hasomed GmbH, Germany) connected to a PC with an ad-hoc developed C++ software.

Subsequently, to avoid a mismatch between the two channels (i.e., one predominating the other), they were activated simultaneously. The subject was asked to report whether he felt an unbalanced perception, and in that case, the amplitudes were adjusted to reduce this mismatch. All participants underwent a brief learning session (<30 min) to help map the stimulation location with respect to the vibrating bars. They quickly expressed confidence in understanding the feedback. Neither of our participants reported changes in sensation intensity for the duration of our experiment (trials were lasting on the order of minutes). This indicated that there was no adaptation to the stimulation.

For Experiment 2, five repetitions of monotonically increasing stimulation ramps were performed for every channel: the subjects received stimuli of the duration of 1 s, spaced out by 1 s of pause. The pulse-width of the stimulation was increased by 20 μs in every step. The stimulation was paused when subjects reported the perceptual threshold. The mean of the stimulation values for this condition was calculated and used for the minimum stimulation intensity. Then, the stimulation intensity was increased until the subjects reported the maximum level of the sensation (below the pain threshold). Also this procedure was repeated 5 times, and the mean was used as the maximum stimulation level (corresponding to the maximum overlap between the wave and the foot).

Virtual reality system and calibration

The virtual reality was implemented with the HTC VIVE headset, while the 3D world and event scripting was programmed using Unity3D. The connection between the stimulating system and VR was achieved via Pipeline Server allowing a precise temporal relation between visual and electro-tactile events. In detail, Unity scripts write to the pipe server via a dynamic-link library (DLL) that is imported into the C script. This library allows the C script in Unity to write to a “pipe” created by the TENS control program.

Before each experiment, the subject wore the headset and was sitting on a chair with his prosthesis elevated on a stool (Figure 1A). The subject familiarized themself with a demo 3D world to make sure they would not suffer from cybersickness. Furthermore, using the proper screw on the headset, the lenses were corrected for the pupil distance to avoid that the environment was seen out of focus.

Subsequently, the subject was immersed in the actual VR scenario. The environment the subject was immersed in was composed of a chair on which a virtual avatar would sit on. The point of view of the subject was always posed from the head of the avatar. The leg of the avatar matching the side of the amputation was representing an intact straight limb and placed over a plank in front of him (see Figure 1A). The position allowed the subject to perfectly see his leg and foot and the surrounding ambient was made by rocks and trees. Behind the virtual foot, either one or two bars were placed (depending on the experiment), one behind the heel (lower bar) and one behind the phalange. The bar colors allowed a good contrast to the floor.

Stimuli selection

The headset internal screen had a refresh rate of 90 Hz, which meant for the Nyquist rule that it can display signals that are lower or equal to 45 Hz. This was the highest frequency of possible stimuli (i.e. 0.0222 s of period), and indeed was the lowest limit of used periods. All the other possible stimuli needed to be multiples of this period. Due to the inverse relationship between frequency and period, it resulted that low periods were more different between them than higher periods. To overcome this limitation, the number of different stimuli levels with respect to the reference was chosen asymmetrically. Hence, the number of comparisons between the reference and lower periods was set at 4, while the comparisons between the reference and higher periods were set at 6.

Experimental procedure

The first task explored multisensory integration using visual and electro-cutaneous (ECS) feedback. We adapted the size discrimination task used by Ernst & Banks in their seminal study (Ernst and Banks, 2002). The task was a two-alternative forced-choice task (2-AFC), and the subjects had to judge which between two consecutively presented bars vibrated faster. Information about the bar vibration was provided visually or by stimulating the stump through ECS stimulation. The ECS feedback was provided using only the frontal couple of electrodes (i.e. one channel), while visual information corresponded to an intermittent flickering of the bar, and was provided through VR and the subjects saw one bar behind their virtual foot. Each trial consisted of the presentation of two stimuli: the first stimulus lasted for 2 s and after 0.5 s the second stimulus was presented again for 2 s. The period of vibration of the target stimuli varied randomly from trial to trial (from 0.0222 to 0.711 s), while the period of vibration of the reference stimulus did not vary across trials (0.2442 s; see Supplemenal information for more details on the stimuli selection). These vibration periods correspond to the lowest frequency of 1.4 Hz (corresponding to 0.711 s) and the highest of 45 Hz (corresponding to 0.0222 s). During each trial, the intra-order of couples (i.e., the order between the comparison and the standard) was randomized and each comparison stimulus was tested 10 times, resulting in a total of 110 trials for each condition.

At the end of each trial an audio alarm played, and the subject had to indicate which bar vibrated faster. The answer was provided by pressing up or down arrows on a keyboard. The up arrow indicated that the first vibration was faster, and the down arrow indicated the opposite.

The experiment included unimodal conditions in which participants performed the task using only visual (V or VB) or tactile (T) information, and bimodal conditions in which both information were presented simultaneously (VT or VBT) (Figure 1B). The tactile information was conveyed through an electro-cutaneous stimulation that elicited a vibration sensation, which was homologous to the visual cues (Figure S1). The parameters of the stimulation were defined thanks to a thorough characterization procedure (See Supplementary information and Table S2). To verify whether the integration of the stimuli followed a weighted estimation between the sensory cues, we considered two situations in which the weight given to each sensory cue was varied by adding blur to the visual condition (VB). This was done in line with the optimal integration model (Ernst and Banks, 2002) to avoid a situation where the subjects would rely only on the strongest stream of input. In the no blur situation, we added no noise to the visual cue (V) and to the somatosensory one (T), such that the visual cue was weighted more than the somatosensory one. In the blur condition, we added noise to the dominant channel (vision), by blurring the information (VB), such that the tactile and visual cues had the same weights (Figure 1D). The blurring was implemented as two half-transparent white layers placed between the foot and the virtual bar. In the case of multisensory conditions (i.e., VT, VTB) the visual clue of vibration was linked with a synchronous electro-cutaneous stimulation over the frontal part of the stump and the participants did not report any delay between the somatosensory and visual feedback. The intensity of the stimulation was previously obtained by calibration, while the frequency-matched the visual information.

Point of subjective equalities and just noticeable difference

For each sensory modality condition and each blurring situation, we fitted a cumulative normal probability distribution using maximum likelihood estimation, i.e. with a Generalized Linear Model (GLM) with the comparison stimulus as the predictor, and a probit link function to obtain separate psychometric functions representing the probability of judging the comparison stimulus as having a higher vibrating period than the standard stimulus. For each psychometric curve, we computed the point of subjective equality (PSE), and the Just Noticeable Difference (JND)

We computed the 95% confidence interval for the PSE and JND from 5000 parametric bootstrap samples with x fixed (Davison and Hinkley, 1997). In Table S3, we report the JND and PSE for each single and merged (when not statistically different one from the other) unisensory and multisensory JND and the bootstrap 2.5% and 97.5% of confidence intervals.

Sensory performance assessment

Subjects had to discriminate which of two stimuli presented simultaneously in two different stump locations was perceived as vibrating faster (See Supplementary Information). In the unimodal condition (UM), the individuals had to use vision, which was the sensory channel identified as predominant during the optimal integration assessment (Figure 1D). In the bimodal condition (BM), the visual cues were coupled with the ECS stimulation. The visual stimuli consisted of two vibrating bars under the virtual foot (under the heel and toes). The accuracy and reaction times of the subjects were measured. The couples of stimuli tested were the same as in Experiment 1, except for the couple of two identical stimuli (comparison stimuli) since it would not give any useful information in terms of accuracy (no correct answer). The 10 couples were repeated 10 times each, yielding a total of 100 trials per subject/condition. However, the analyses did not include the few trials where the subjects answered in more than 8 s. This resulted in a total of 196 trials for Subject 1 and 200 trials for Subject 2. The reaction times and accuracy between the UM and BM conditions were analyzed and are displayed in Figures 2C and 2D. Additionally, we subtracted the RTs of the BM condition from the UM condition to see if the possible benefits of the bimodal condition were focused on a particular group of stimuli.

Measures of embodiment and phantom limb distortions

In a second VR scenario, the subjects observed both their legs as intact limbs with incoming water waves touching the virtual foot of the subjects (corresponding to the prosthetic foot in the real world) (Figure 2E, Video S1), from a first-person perspective. Healthy subjects participated in this experiment and followed the same procedure as amputees (except for the stimulation delivered on the foot). A Transcutaneous Electrical Nerve Stimulator (TENS) generated the tactile feedback in the same location tested during the first experiment (optimal integration assessment) (Figure 1B). The stimulation was modulated in intensity to be either synchronous or asynchronous (Botvinick and Cohen, 1998) (Figure 2F) with the visual event of an incoming wave touching their leg. The VR and the TENS devices were the same ones used in the first phase of the study. The RehaMove3 device (Hasomed GmbH, Germany) was connected to the computer with a USB cable. A previously custom-made software, running at the same time as the virtual scenario, allowed for real-time control of the device and synchronization with the events in the virtual world. The virtual environment consisted of a beach scenario, while the TENS parameters were chosen after a calibration procedure as done in Experiment 1 (see Supplementary Information).

The subjects underwent each condition (synchronous and asynchronous) once in a randomized order. Trials lasted 5 min, which was expected to induce the illusion (Botvinick and Cohen, 1998; Slater, 2009), divided by 15 min wash-out time for the questionnaire, measurements, and rest. After the conditions, electrodes were removed, and the area was checked for any adverse skin reaction. In none of the subjects (both amputees and healthy individuals) this was the case.

To assess the strength of the illusion, questionnaires for the embodiment of the virtual leg were collected in amputees and healthy subjects (Botvinick and Cohen, 1998; Crea et al., 2015). Control questions to assess subjects’ suggestibility were included (Botvinick and Cohen, 1998). For measuring the phantom limb representations in amputees, telescoping (Rognini et al., 2018) and proprioceptive displacement (Petrini et al., 2019b) assessments were performed. For the former, the subject indicated first where he felt the end of the stump, and afterwards where he felt the end of the phantom foot, as in (Rognini et al., 2018). The perceived length of the phantom was then estimated as the difference between the perceived position of the stump and the phantom foot. For the latter, the position of the hallux of the prosthesis was noted. The subject had to instruct the experimenter to move a ruler either to the left or to the right until he thought the ruler was in the exact position where he felt the hallux of the phantom foot (Petrini et al., 2019b). This measurement was repeated ten times both before and after every experiment condition tested. The perceived length of the phantom was then estimated as the difference between the perceived position of the end of the stump and the perceived position of the end of the phantom foot (Rognini et al., 2018). For the proprioceptive displacement, the subject was sitting with the prosthesis placed on a plank. The position of the hallux of the prosthesis was noted. The subject’s view of the stump and the prosthesis was hidden. Starting from a random position, the experimenter moved a second ruler which was placed vertically and perpendicularly to the plank. The subject had then to instruct the experimenter to move the ruler either to the left or to the right until he thought the ruler was in the exact position where he felt the hallux of the phantom foot (Petrini et al., 2019b).

Quantification and statistical analysis

Statistical analysis: optimal integration

For each sensory modality condition and each blurring situation, we fitted a cumulative normal probability distribution using maximum likelihood estimation, i.e. with a Generalized Linear Model (GLM) with the comparison stimulus as the predictor, and a probit link function to obtain separate psychometric functions representing the probability of judging the comparison stimulus as having a higher vibrating period than the standard stimulus. The Goodness Of Fit was assessed with the Likelihood Ratio test. For each psychometric curve, we computed the point of subjective equality (PSE), i.e. the period of vibration of the bar that was perceived as faster than the standard in 50% of the trials. We also computed the discrimination threshold - or Just Noticeable Difference (JND) (i.e. the smallest difference between two stimuli that can be reliably discriminated (Ernst and Banks, 2002))– as the difference between the PSE and the period of vibration that is perceived to be faster than the standard in 84% of the trials. 84% JND corresponds to the standard deviation of the normal distribution underlying the psychometric function and is an estimate of the noise associated with the unimodal or bimodal cues. From the results in the unimodal conditions, we obtained a prediction of optimal integration behavior (JNDMLE) that we compared with the participants’ actual bimodal performance.

We computed the 95% confidence interval for the PSE and JND from 5000 parametric bootstrap samples with x fixed (Wichmann and Hill, 2001). In the Results, we report the JND of the bootstrap distribution. To compare the difference in JNDs we performed pairwise comparisons by computing the resampling distribution of the difference and the p value that corresponded to the largest equi-tailed confidence interval that excluded zero (i.e. results occurring under the null hypothesis of no difference between groups). To account for the multiple comparisons, we then adjusted this p value for False Discovery Rate (Benjamini and Hochberg, 1995).

All data from the first experiment were analyzed using R software version 3.5.1.

Statistical analysis: discrimination task

After checking for normality (Kolmogorov Smirnov), the reaction times (RT) were compared between the unimodal and bimodal conditions with the Wilcoxon sign rank test. Additionally, a two-way ANOVA was performed with factors Stimuli and Condition. For the accuracy results, after checking for the normality of the distributions (Kolmogorov Smirnov), the correct/incorrect percentage distributions were statistically compared with a Wilcoxon sign rank test and are reported in Figure 2D (mean ± CI). All data from the second experiment were analyzed using the built-in function in Matlab R2017b.

Statistical analysis: phantom limb distortions

The data were analyzed and plotted using Matlab R2019b software. Post-hoc G-power was calculated using GPower 3.1 software. The bar plots for the embodiment questionnaire show the mean and standard error of the mean. The bar plots for the telescoping and proprioceptive displacement show the mean and standard deviation. Non-parametric Wilcoxon tests were used.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
MATLAB R2016b	MathWorks	https://www.mathworks.com
R 3.5.1	R foundation	https://www.r-project.org
Unity	Unity Technologies	https://unity.com
Other
Rehamove3	Hasomed GmbH	https://hasomed.de
Circle Electrodes Pads (25 mm)	Tenscare	https://tenscare.co.uk
HTC VIVE	VIVE	https://vive.com