Haptic contents of a movie dynamically engage the spectator's sensorimotor cortex.

Observation of another person's actions and feelings activates brain areas that support similar functions in the observer, thereby facilitating inferences about the other's mental and bodily states. In real life, events eliciting this kind of vicarious brain activations are intermingled with other complex, ever-changing stimuli in the environment. One practical approach to study the neural underpinnings of real-life vicarious perception is to image brain activity during movie viewing. Here the goal was to find out how observed haptic events in a silent movie would affect the spectator's sensorimotor cortex. The functional state of the sensorimotor cortex was monitored by analyzing, in 16 healthy subjects, magnetoencephalographic (MEG) responses to tactile finger stimuli that were presented once per second throughout the session. Using canonical correlation analysis and spatial filtering, consistent single-trial responses across subjects were uncovered, and their waveform changes throughout the movie were quantified. The long-latency (85-175 ms) parts of the responses were modulated in concordance with the participants' average moment-by-moment ratings of own engagement in the haptic content of the movie (correlation r = 0.49; ratings collected after the MEG session). The results, obtained by using novel signal-analysis approaches, demonstrate that the functional state of the human sensorimotor cortex fluctuates in a fine-grained manner even during passive observation of temporally varying haptic events. Hum Brain Mapp 37:4061-4068, 2016.

INTRODUCTION

Previous brain imaging studies have shown that observing another person's actions and feelings activates brain areas that support similar functions in the observer. Such automated, vicarious activations contribute to our understanding of the mental and bodily states of other people and facilitate social interaction [for reviews, see, e.g., Frith and Frith, 2007; Hari and Kujala, 2009]. Although the most prominent vicarious activations have been demonstrated in pre‐motor cortical areas [for a review, see, e.g., Rizzolatti and Craighero, 2004], also somatosensory cortices are engaged during observation of actions [e.g., Avikainen et al., 2002; Möttönen et al., 2005; Oouchida et al., 2004; Rossi et al., 2002] or while seeing other people being touched [e.g., Blakemore et al., 2005; Ebisch et al., 2008; Keysers et al., 2004; Martínez‐Jauand et al., 2012; Meyer et al., 2011; Pihko et al., 2010; Schaefer et al., 2009].

In every‐day life, events that elicit vicarious activation are inseparably intermingled with a complex and rapidly changing social environment, and such experimental settings are thus difficult to bring into a brain‐imaging laboratory. Our solution for studying the neural underpinnings of life‐like social perception is to image brain function of subjects who are viewing movies that are rich in their composition, but can—in contrast to real‐life events—be repeated in identical form to each individual viewer.

Vicarious activation of the sensorimotor cortex during movie viewing has been demonstrated previously by functional magnetic resonance imaging (fMRI). For example, hand movements in the movie activated the postcentral sulcus in the vicinity of cytoarchitectonic area 5 [Hasson et al., 2004], and the primary and secondary somatosensory cortices were activated when the movie displayed a boxing match [Nummenmaa et al., 2014].

Compared with the sluggishness of the hemodynamic changes that are measured with fMRI, magnetoencephalography (MEG) measures directly neuronal activity that can be tracked with a millisecond‐range temporal accuracy. In the current study, we used MEG to explore how haptic events in a movie affect the spectator's sensorimotor cortex. We probed the functional state of the sensorimotor cortex by delivering tactile stimuli to subjects’ fingers once per second throughout the movie. We then compared the moment‐by‐moment modulations of the resulting 880 single evoked responses per subject with the level of the haptic contents present in the movie.

We demonstrate time‐varying and fine‐grained vicarious modulation of the sensorimotor cortex that is concordant with the viewers’ moment‐by‐moment reports on their own haptic engagement in the movie.

MATERIALS AND METHODS

Subjects

Sixteen healthy adults (8 females, 8 males; 2 left‐handed; mean age 28 years, range 20–60 years) participated in the study. All subjects had normal or corrected‐to‐normal vision. The study had a prior approval by the ethics committee of the Hospital District of Helsinki and Uusimaa. All participants gave written informed consent prior to the study.

Stimuli

During the MEG recordings, the subjects watched a 15‐min silent black‐and‐white film (“At Land” by Maya Deren, 1944). The movie depicts a female character engaged in a wide range of bodily activities in her environment, such as crawling, climbing, or touching things. The film was presented at 24 frames/s using Presentation software (Neurobehavioral systems; http://www.neurobs.com/) and projected to a back‐projection screen located 1.25 m in front of the subject (viewing angle 13° horizontal, 10° vertical; screen size 28 cm × 22.5 cm).

Throughout the movie, the viewers received pneumatic tactile stimuli on the volar distal phalanges of their five left‐hand fingers, one finger at a time. These probe stimuli (total duration 178 ms) were produced by expanding a thin plastic membrane with an air puff [Mertens and Lütkenhöner, 2000]. The fingers were stimulated in random order once every 1,005 ms. Each subject received 880 stimuli over the 15‐min movie. Figure 1 illustrates the measurement setup.

Figure 1

Pneumatic tactile stimuli were delivered to the five left hand fingers in random order every 1,005 ms while the subject was watching the movie on the screen. Simultaneously, whole‐scalp MEG was measured. Both the movie and tactile stimuli timings were aligned with the MEG acquisition system by trigger signals, and were thus identical across all the subjects.

The timings of both the video and the tactile stimuli were aligned with MEG acquisition system and were identical for all subjects.

MEG and MRI Recordings

MEG signals were recorded in a magnetically shielded room (MEG Core, Aalto NeuroImaging, Aalto University) with a 306‐channel whole‐scalp neuromagnetometer comprising 204 planar gradiometers and 102 magnetometers at 102 sensor units (Elekta NeuromagTM, Elekta Oy, Helsinki, Finland). MEG was bandpass‐filtered from 0.03 to 330 Hz and sampled at 1,000 Hz. The head position with respect to the MEG sensors was continuously followed by using head‐position‐indicator (HPI) coils. Vertical and horizontal electro‐oculograms (EOGs) were recorded at the same time.

For source‐level analysis, T1‐weighted anatomical MRIs were acquired at the Advanced Magnetic Imaging Centre of Aalto Neuroimaging, Aalto University, or previously acquired MRIs were used. In total, MR images were available for 10 out of 16 subjects.

Preprocessing of MEG Data

MEG data of each subject were preprocessed to compensate for head movements inside the sensor helmet, and to suppress magnetic interference by temporal signal‐space separation (tSSS) method [Taulu and Simola, 2006] implemented in Maxfilter software version 2.2 (Elekta Oy, Helsinki, Finland). Default parameter settings of the software were used.

Figure 2 summarizes all pre‐processing steps and the extraction of consistent single‐trial waveforms across subjects over the entire movie. Data from 204 planar gradiometers were analyzed. The raw data were bandpass‐filtered from 1 to 40 Hz and, on the basis of stimulus triggers, split into 400‐ms epochs containing the evoked response. The mean value was removed from each epoch for the subsequent multi‐set canonical correlation analysis (MCCA). As the further analysis was insensitive to occasional artifacts in individual single‐trials, and because signal decomposition based on MCCA is inherently resilient to artifacts that are not exactly time‐locked across subjects, no further artifact rejection was needed.

Figure 2

Preprocessing and multiset canonical correlation analysis (MCCA). (A) Continuous data from 204 MEG channels for each subject were used in the analysis. (B) Preprocessing steps included artifact reductions, filtering to 1–40 Hz band, splitting data to 400 ms epochs (time‐locked to onset of tactile stimuli), and reducing the data dimension from 204 to 67 by spatial PCA (resulting data matrix denoted by ). (C) MCCA was used to find optimal spatial filters for each subject. In MCCA, the projections are formed so that the overall correlation between all and ( is maximized for each component . (D) As a result, the most consistent components across the subjects were extracted. The figure illustrates two first MCCA components, with data from all subjects superimposed.

In the last step of preprocessing (step B in Fig. 2), spatial principal component analysis (PCA) was applied to reduce data dimensionality from the original 204 down to , corresponding to the degrees of freedom of the data matrix remaining after the tSSS artifact reduction.

MCCA‐Based Spatial Filtering

We aimed to follow waveform changes of the single‐trial tactile responses throughout the movie. As the subjects viewed the movie only once, it was not possible to improve signal‐to‐noise ratio by signal averaging. Instead, we applied multi‐set canonical correlation analysis [MCCA: Kettenring, 1971; Li et al., 2009] in the spatial filtering scheme at sensor‐level to uncover consistent single‐trial responses across subjects at 1‐s intervals over the entire movie [Lankinen et al., 2014].

MCCA is a data‐driven analysis method that finds maximally correlated signal components across data sets, or here, across MEG recordings of individual subjects (Fig. 2). In our setting, spatial filtering refers to projection where is the weighted sum of the preprocessed MEG data , superscript m indicating the subject (m = 1 … M, number of subjects). and are D × T matrices, corresponding to the MEG signal dimension (D = 67) and to the number of concatenated samples over time (T = 352,000, consisting of 880 epochs, each with 400 samples). is a D × D weighting matrix estimated by MCCA so that the projections that is the canonical variates, are maximally correlated across subjects but mutually uncorrelated within the dataset m. In other words, MCCA optimizes an objective function, here MAXVAR [Kettenring, 1971], so that the canonical variates achieve the maximum overall correlation. The performance of MCCA improves as the number of datasets increase [Li et al., 2009].

The key advantages of the MCCA approach are that (i) the method reduced noise in individual evoked responses, thereby enabling the inspection of characteristics of single‐trial signals throughout the entire movie without averaging; (ii) it revealed a set of signal components that highlighted complementary temporal and spatial characteristics of the responses; and (iii) the method could be applied easily at sensor level, so that the resulting signal components of different subjects covary in common signal subspace.

The training set of the MCCA model consisted of the first 440 trials out of 880 from all the subjects. For validation, the MCCA coefficients were applied to a test set, which consisted of the last 440 trials. To evaluate the statistical significance of the resulting components for the test data, we calculated pair‐wise correlations of eight randomly selected independent pairs (sampled without replacement from the pool of 16 subjects) for each component. We applied the t‐test to find out whether the mean of these correlations deviated significantly from zero. The significance level after Bonferroni‐correction was P < 0.05/D.

Principal Component Analysis for Single‐Trials

After finding the most consistent time‐courses across subjects by MCCA‐based spatial filtering, we quantified the changes in each single‐trial response throughout the movie by applying (temporal) principal component analysis (PCA) (Fig. 3). First, separately for each MCCA component, we formed a matrix , the columns representing 400 time samples per trial, and the rows corresponding to the number of single‐trial responses pooled over subjects (880 trials/subject over 16 subjects) (matrix transposed in Fig. 3A). Next we applied PCA to the matrix and selected the first PCA eigenvector e (size 400 × 1; Fig. 3B) and the corresponding first PCA projection (PCA score; Fig. 3C) explaining the largest variance in the signals across trials. Practically, the first eigenvector revealed parts of the single‐responses that explain the largest variance in the data across the movie, and PCA projection to this eigenvector quantifies the amount of change in the responses at each trial.

Figure 3

Temporal principal component analysis (PCA) was applied separately to the MCCA1 and MCCA2 components projections. (A) Data matrix contained single‐trial MCCA projections of all subjects. PCA uncovers the direction of maximum variance in the data. (B) Eigenvectors e reveal the parts of the response accounting the largest variability in . (C) The corresponding projections (PCA scores) quantify the changes in single‐trials during the movie viewing.

In the subsequent analysis, separately for the first three MCCA components, the first PCA score of each subject were separated from the vector and averaged across the subjects. The resulting mean time series, denoted by (dimension 1 × 880), was also detrended to compensate very slow linear drifts.

Ratings of Tactile Engagement with the Movie

After the MEG recordings, the subjects watched the movie once again on the computer screen and rated their level of engagement with the haptic contents of the movie by shifting a cursor up and down on a scale presented on the screen. The scale was continuous from 0 to 1 and the ratings were sampled at 5 Hz. In each subject, the ratings were linearly transformed to range from 0 to 1. Supporting Information Figure S1 represents the average ratings across all 16 subjects together with 25th and 75th percentiles. For further analysis, the ratings were averaged across the subjects, and downsampled to correspond to the number of responses (=880).

Comparison of MEG Responses and Ratings

Next, we computed correlation between mean PCA score and averaged behavioral ratings. For this analysis, both and ratings were low‐pass filtered at 0.1 Hz, as the changes in manual ratings were relatively slow. Correlation between and the ratings was computed with time lags from −20 to 20 s separately for the first two MCCA components.

We used non‐parametric stationary block bootstrapping to determine the confidence intervals for the correlation values, thus retaining temporal dependences in time‐series as well as stationarity in the data [Politis and Romano, 1994]. The average block lengths (between 38 and 46 samples) were estimated by the automatic optimization method presented by Politis and White [2004] and Patton et al. [2009]. We determined 95% confidence intervals for the correlation coefficient between and the average of the behavioral ratings by repeating bootstrapping 10,000 times (Supporting Information Fig. S2).

Source Localization

To verify that the MCCA components reflect activity in feasible brain areas, we inspected the spatial‐filter weights from the MCCA‐model in source space. The weights (D × D) were transformed back to the 204 dimensions, , corresponding to the original number of sensors. The spatial‐filter weights were converted to activation patterns (forward models) , where and are covariance matrices of the data and projections [Haufe et al., 2014].

The resulting sensor‐level activation patterns were further transformed to the anatomical source space by employing minimum‐norm estimates [Hämäläinen and Ilmoniemi, 1994] with MNE Suite software package (http://www.martinos.org/mne/). Thus, for one MCCA component, the input for MNE was a 1 by 204 vector. MNE was calculated at discrete locations separated by 7 mm on the cortical surface, with “loose factor” 0.4 to favor the dipole component normal to the surface, and with “depth weighting” to reduce the bias toward superficial currents. For illustration, individual maps were morphed by linear mapping to a common template (“fsaverage” in FreeSurfer package) and averaged across subjects.

RESULTS

Intersubject Correlation of MEG Time‐Courses

Applying MCCA to the MEG data uncovered the consistent signal components across the subjects during the movie. Components MCCA1 and MCCA2 exceeded the significance level with corresponding mean intersubject correlations (ISC) of 0.17 and 0.10 for the test data. For comparison, the correlation coefficients were with the training data 0.30 and 0.21, respectively, which shows that the spatial filters from the training data can be generalized to the test data. Figure 4 illustrates the MCCA time‐series. Notably, different components revealed complementary response characteristics and showed more consistent waveforms across subjects than the original raw signals.

Figure 4

Raw MEG gradiometer signals and components MCCA1 and MCCA2 for all 16 subjects superimposed (10 first trials of test data). Corresponding trial averages across the whole movie are represented on the right panel, along with the time points of the main peaks. It is noteworthy that the sign of both MCCA components is arbitrary.

Correlation Between MEG Responses and Ratings

Figure 5A illustrates the mean behavioral ratings (red) of the manually rated haptic engagement with the movie content together with the mean PCA scores (black) (both averaged across all 16 subjects). The maximum correlations between the ratings and MCCA1–MCCA2 were 0.49 and 0.38, respectively (see Supporting Information Fig. S2 for statistical testing). The corresponding time‐lags were 7 and 5, the ratings lagging behind the PCA scores.

Figure 5

Modulation of evoked response characteristics along with subjective ratings of the engagement to the haptic movie contents. (A) PCA scores for the components MCCA1 and MCCA 2 (black) together with the subjective ratings (red). The ratings are shifted in time to correspond to the lag with maximum correlation. The amplitudes are normalized to zero mean and standard deviation of 1. All the signals represent the average across all 16 subjects. (B) The eigenvectors of the MCCA components accounting for the highest variance in the responses.

By visual inspection, the highest peaks both in the ratings and in the PCA scores matched with the scenes where the main character was, for example, crawling, climbing, pulling herself up, picking up stones, or feeling a rough rock surface against her cheek, or where her body was gently moved by waves rolling on the shore.

Figure 5B represents the first PCA eigenvectors e for components MCCA1 and MCCA2. Notably, these eigenvectors account for the variance in the single‐trial responses most clearly at latencies 85–125 ms, and 125–175 ms for MCCA1 and MCCA2, respectively.

Source Estimates

To validate the feasibility of the obtained spatial filter weights, we inspected their distribution both in sensor and source space. The MEG channels mostly contributing to MCCA components were found over the right hemisphere, contralateral to the stimulated hand, with slightly different distributions for the different MCCA components (see Supporting Information Fig. S3 for sensor level maps). Figure 6 shows the averaged source level maps for 10 subjects whose MR images were available. Corresponding individual source estimates are shown in Supporting Information Figure S4. Expectedly, these maps verified the right sensorimotor cortex as the source of MCCA components.

Figure 6

Average source level activation patterns over 10 subjects (arbitrary units; activation pattern strength) showing areas that most contribute to MCCA components. The dark gray color represents sulci and the light gray gyri.

DISCUSSION

We demonstrated that the haptic content of a movie modulates dynamically the state of the viewer's sensorimotor cortex as established by fine‐grained changes in single‐trial responses to simultaneously presented tactile probe stimuli. This modulation covaried with the viewer's moment‐by‐moment ratings of haptic engagement with the movie.

Vicarious activation of the sensorimotor cortex has been demonstrated earlier (see references in the “Introduction” section and Keysers et al., 2010 for a review) but never in such naturalistic experimental settings as used in the present study. Ours is also the first study to demonstrate that the vicarious brain activity changes dynamically in concordance with the subjects’ perceived and rapidly varying haptic engagement. Using novel analysis of the time‐sensitive MEG measures we were thus able to establish that the abundance and richness of observed haptic interaction between other people and their environment vicariously modulate the state of the viewer's sensorimotor cortex.

Vicarious somatosensory activation has been previously linked mainly to area 2 of the primary somatosensory cortex, whereas area 3b in the posterior wall of the central sulcus (which is the main generation site of responses to tactile stimuli applied to the own body) has been regarded as “private” and thus insensitive to observed touch [for a review, see, e.g., Keysers et al., 2010]. However, when the observed stimuli are haptic (involving action in addition to touch), also area 3b can be vicariously affected, as has been shown previously in association with observed manipulatory finger movements [Avikainen et al., 2002], and, in the current study, with the haptic contents of a movie.

Movies mimic real life, where the stimuli of interest (here, haptic events) are intermingled with other sensory features. Accordingly, the present movie “At Land” by Maya Deren (1944) contains multiple overlapping sensory events that can be perceived simultaneously (see [Kauttonen et al., 2015] for detailed annotation). Consider, for instance, the main character picking up stones while walking, or climbing up barefoot an upright‐standing driftwood. These actions and haptic events engage somatosensory, proprioceptive, and motor functions, which is concordant with our finding that MEG signals were modulated in large regions of the sensorimotor cortex.

The single‐trial evoked responses characterized in the present study are noisy, and we thus used a recently described [Lankinen et al., 2014] MCCA‐based procedure to find common response waveforms across subjects following each single stimulus. This analysis was applied to sensor‐level signals in a data‐driven way, and it was informative about vicarious perception in a naturalistic experimental setting. As MCCA finds similar temporal patterns in the recordings over subjects, we acknowledge the possibility that if the responses would highly differ in some subsets of subjects, their responses might split into different MCCA components. However, in further examination (not shown) we found that such splitting did not occur in the present study; instead, the individual MCCA1 and MCCA2 waveforms were highly concordant with the group averages of the corresponding components. Thus, the MCCA‐based approach revealed the most consistent components across subjects and enabled moment‐to‐moment comparison of these single trials to the behavioral ratings during the movie viewing.

CONCLUSION

In our everyday world, we are surrounded by continuously changing multisensory stimuli. The current study demonstrates that observed haptic events modulate dynamically the functional state of the viewer's sensorimotor cortex. Our results thus expand the current understanding of the nature of fine‐grained and time‐varying vicarious brain activations in naturalistic settings.

Supporting Information

Click here for additional data file.

Supporting Information

Click here for additional data file.

Supporting Information

Click here for additional data file.

Supporting Information

Click here for additional data file.