The heaviness of invisible objects: Predictive weight judgments from observed real and pantomimed grasps.

Observation of others' actions has been proposed to provide a shared experience of the properties of objects acted upon. We report results that suggest a similar form of shared experience may be gleaned from the observation of pantomimed grasps, i.e., grasps aimed at pretended objects. In a weight judgment task, participants were asked to observe a hand reaching towards and grasping either a real or imagined glass, and to predictively judge its weight. Results indicate that participants were able to discriminate whether the to-be-grasped glass was empty, and thus light, or full, and thus heavy. Worthy of further investigation, this finding suggests that by observing others' movements we can make predictions, and form expectations about the characteristics of objects that exist only in others' minds.

Introduction

The behavior of others supplies a rich source of information about the world around us. The ability to process this information is key for learning about the properties of objects acted upon, as well as to read others’ intentions and expectations (Cavallo, Koul, Ansuini, Capozzi, & Becchio, 2016; for review, see Ansuini, Cavallo, Bertone, & Becchio, 2015). By observing another person grasping and lifting a cup, for example, we can immediately deduce whether the cup is full or empty, even when we cannot see inside the cup (Bingham, 1987, Hamilton et al., 2004, Maguinness et al., 2013). Through this, we may also perceive whether the other person had a correct or false expectation about the weight of the cup (Finisguerra et al., in press, Runeson and Frykholm, 1983), and use this information to reduce ‘surprise effects’ in our own interactions with the environment (Meulenbroek, Bosga, Hulstijn, & Miedl, 2007).

As such, observing other people acting upon objects involves a form of experience sharing (Brown and Brüne, 2012, Limanowski and Blankenburg, 2013): we can learn about the properties of a given object through others’ action, without needing to have first hand experience. In this way, supposedly hidden, internal properties of objects, such as weight, become available for perception (Runeson, 1985).

The question addressed in the present study is whether a similar form of shared experience may be gleaned from the observation of pantomimed actions, i.e., actions aimed at imagined, rather than real objects. Put simply: can we share through others’ action the characteristics of an object that is not there?

The hypothesis that pantomimed actions contribute to shared experience of imagined objects is motivated, in part, by studies investigating the kinematics of pantomimed grasping actions. When pretending to pick up imagined objects, we move and shape our hands quite differently from when we grasp real objects (Cavina-Pratesi et al., 2011, Goodale et al., 1994). Still, pantomimed actions demonstrate at least some perceptual features of the pretended object. For example, during pantomimed grasping, grip width depicts the width of the imaginary object (Goldenberg, Hartmann, & Schlott, 2003). Moreover, there is evidence that, early on in the movement, the kinematics of both real and pantomimed movements is scaled to the weight of the object to be grasped (e.g., Ansuini, Cavallo, Campus, et al., 2016, Eastough and Edwards, 2007). This raises the possibility that, even before contact, observers can take advantage of kinematic information in order to form a shared representation of the object acted upon – be it real or imagined. The present study aimed to test this hypothesis by asking participants to make predictive weight judgments from the observation of real and pantomimed reach-to-grasp movements.

Material and methods

Participants

Twenty-four participants took part in the experiment (12 females; M age = 24; age range = 19–30 years old). The sample size was determined in advance by power analysis using effect sizes observed in a pilot study. A sample size of 24 was calculated to detect a Cohen’s d of 0.70 with alpha set at 0.05 (one-sided), and power set at 0.90. All participants were right handed, with normal or corrected-to-normal vision, and with no history of either psychiatric or neurological disorders. The research was approved by the local ethical committee (ASL 3 Genovese), and was carried out in accordance with the principles of the revised Helsinki Declaration (World Medical Association General Assembly, 2008). Written informed consent was obtained from each participant.

Experimental stimuli: video capturing, selection and editing procedure

To create the stimuli to be used in the main experiment, we filmed 15 agents (10 females; M age = 28.8; age range = 24–32 years old) performing real and pantomimed reach-to-grasp movements. For real reach-to-grasp movements, participants were requested to reach towards, and grasp, either an empty glass (139 g) or a glass filled with iron screws (838 g), placed at a distance of 48 cm from the participant’s body midline. For pantomimed reach-to-grasp movements, the glass, either empty or filled, was positioned at a displaced location (12 cm away from the target position). Participants were instructed to imagine that an identical glass was positioned at the target position, and were asked to pretend to perform the very same action sequence towards the imagined glass.

Reach-to-grasp movements were filmed from a lateral viewpoint using a digital video camera (Sony Handycam 3D, 25 frames/s; Sony Corporation, Tokyo, Japan). Simultaneously, hand movement kinematics were recorded using a near-infrared camera motion capture system (frame rate: 100 Hz; Vicon Motion Systems Ltd, Oxford, UK). To assess the availability of weight information over time, a set of kinematic variables was calculated using a custom Matlab (MathWorks, Natick, MA, USA) script (see Table S1 for a detailed description of the kinematic variables). All variables were computed only considering the reach-to-grasp phase of the movement, i.e., from ‘reach onset’ (i.e., the first time point at which the wrist velocity crossed a 20 mm/s threshold and remained above it for longer than 100 ms) to ‘reach offset’ (i.e., the time at which the wrist velocity dropped below a 20 mm/s threshold) at an interval of 10% of the normalized movement time (see Ansuini, Cavallo, Campus, et al., 2016, Ansuini, Cavallo, Koul, et al., 2016 for further details).

With respect to the stimulus selection, we proceeded as follows: first, we submitted the computed kinematic variables of real and pantomimed reach-to-grasp movements to separate linear discriminant analyses (LDAs) to find the linear combinations of features that, for each type of movement, separated between heavy and light objects. Kinematic data from one participant were discarded due to technical problems with video recording. Discriminant function analyses using a leave-one-out cross validation method (Efron, 1982) revealed that classification of object weight was significantly above chance level (i.e., 50%) for both real and pantomimed reach-to-grasp movements (see Table 1 for details). This conclusion was supported by the results of permutation tests (1000 simulations for each LDA model) (all p values < 0.001).

Table 1

Confusion matrix from LDAs for real and pantomimed reach-to-grasp movements directed at light and heavy objects. Bold values indicate cross-validated grouped cases that were correctly classified. Actual number of observations is shown in parentheses.

	Real reach-to-grasp movements			Pantomimed reach-to-grasp movements
	Light object	Heavy object	Total	Light object	Heavy object	Total
Light object	91.4% (288)	8.6% (27)	100% (315)	67% (221)	33% (109)	100% (330)
Heavy object	10.8% (35)	89.2% (288)	100% (323)	33.4% (108)	66.6% (215)	100% (323)

The kinematic variables that contributed the most to weight classification were grip aperture, wrist velocity and thumb/index finger vertical displacement for real reach-to-grasp movements, wrist velocity and thumb vertical displacement for pantomimed reach-to-grasp movements. Fig. S1 provides a visual summary of how each kinematic variable contributed to the classification of object weight over time for real and pantomimed movements.

With the new space defined via the LDA, we next selected, for each type of reach-to-grasp movements (real, pantomime) and for each weight (light, heavy), the 50 movements that minimized the within-weight distance, i.e., the distance from the mean variate score of heavy versus light objects. This procedure allowed us to identify a final set of 200 representative movements (50 real reach-to-grasp/light; 50 real reach-to-grasp/heavy; 50 pantomimed reach-to-grasp/light; 50 pantomimed reach-to-grasp/heavy).

The 200 unique video clips corresponding to the selected movements were edited using Adobe Premiere Pro CS6 (.avi format, disabled audio, 25 frames/s; Adobe Systems Software Ltd, Dublin, Ireland). To produce spatial occlusion of the to-be-grasped object, a grey rectangular mask (height = 51.5 mm; length = 31.1 mm) was superimposed onto the target object location. The size and the position of this mask were kept constant across participants. Each video was edited so as to begin at reach onset and to end at reach offset (see Video S1). Movement durations (from reach onset to reach offset) did not differ significantly between light and heavy objects, both for real (Light object: M = 869.20 ms, 1SE = 25.76; Heavy object: M = 927.40 ms, 1SE = 27.02) (t(98) = −1.56, p = 0.122, d = 0.31, 95% CI [−132.28, 15.88]) and for pantomimed reach-to-grasp movements (Light object: M = 933.80 ms, 1SE = 29.36, Heavy object: M = 923.20 ms, 1SE = 30.96) (t(98) = 0.25, p > 0.250, d = 0.05, 95% CI [−74.07, 95.27]).

Procedure and measures

The experiment was carried out in a dimly lit room. Participants sat in front of a 17-in. computer screen (resolution: 1280 × 800; frame rate: 75 Hz) at a viewing distance of 50 cm. They were presented with video clips of the reach-to-grasp phase of the selected movements (see ‘Experimental stimuli: video capturing, selection and editing procedure’ section). A one-interval discrimination design was employed (see Fig. 1).

Fig. 1

Illustration of an experimental trial. Each trial started with word cues for the two weights (light versus heavy), followed by the video clip of a reach-to-grasp movement. Participants were free to respond at any time after video stimulus onset during video presentation, and the subsequent 3000-ms response interval. After their response, participants were then asked to rate how confident they felt in their decision on a 4-point scale (from 1  =  least confident, to 4  =  most confident).

After each video, participants were asked to judge as accurately and as quickly as possible the weight of the object towards which the movement was directed (i.e., light versus heavy object). Responses were given by pressing one of two keys on a keyboard. For half of the participants, the Italian word ‘leggero’ (light) on the left prompted a button press with the index finger on the left button of a wireless keyboard touchpad, while the word ‘pesante’ (heavy) on the right prompted a button press with the middle finger on the touchpad right button. The position of the two words was counterbalanced within and across participants. Participants were instructed to respond either during the video, or within a maximum of 3000 ms after the video ended. To ensure that movement sequences could be temporally attended, that is, to provide participants enough time to focus on movement start and prevent anticipation, +13 up to +28 static frames in step of +1 were added at the beginning of all video clips. To equate stimulus duration within each type of reach-to-grasp movement (i.e., real and pantomimed), static frames were also added at the end of the videos in a compensatory manner (+14 up to +29 in step of +1). In this way, each real movement clip lasted exactly 2520 ms and each pantomimed movement clip lasted exactly 2600 ms. After indicating a response, participants were requested to rate the confidence of their decision on a 4-point scale by pressing a key (from 1 = least confident, to 4 = most confident; see Fig. 1). Participants were encouraged to use the entire confidence scale.

To avoid a ‘dual-task’ situation (Pashler, 1994), whereby the implicit categorization of reach-to-grasp movements as real versus pantomimed may have interfered with the explicit categorization of weight information, stimuli displaying real and pantomimed movements were administered to participants in separate sessions on two consecutive days. In each session, participants completed two blocks of 100 trials. The videos were pseudo-randomized over the two blocks so that each block included one repetition of each movement. At the beginning of each experimental session, participants were presented with two movement samples (i.e., one for each object weight), so that they could see the phase during which the agent grasped (or pretended to) the glass, and lift it. Further, at the beginning of each experimental session, participants performed a small practice (4 trials each, for the two weights). Each experimental session lasted about 50 min. Stimuli presentation, timing, and randomization procedures were controlled using E-prime version 2.0.10.242 (Psychology Software Tools, Inc, Sharpsburg, PA, USA). The order of experimental sessions was counterbalanced across participants.

Signal Detection Theory (SDT) was used to analyze weight judgments parameters. Reach-to-grasp movements aimed at a light object were designated as ‘signal’ and reach-to-grasp movements aimed at a heavy target were designated as ‘noise’. The proportion of hits and false alarms was calculated for each participant, and combined with confidence ratings to determine points on an empirical receiver operating characteristic (ROC) curve. The ROC curve plots the hit rate as a function of the false alarm rate at different degrees of confidence. Because each response (light, heavy) had four ratings associated with it, there were eight possible responses for each trial (graded from the most confident first interval response to the most confident second interval response), resulting in seven points on the ROC curve. The area under the curve (AUC) equals the proportion of times participants would correctly identify the target, if the target and non-target were presented simultaneously. The AUC can be any value between 0 and 1. A diagonal curve, which coincides with an AUC of 0.50, corresponds to a situation where the number of hits and false alarms are equal, showing a chance level classification score. On the contrary, an AUC of 1.00, which corresponds to a ROC curve on the left upper bound of the diagonal, indicates a perfect positive prediction with no false positives. Importantly, unlike average accuracy, AUC is a measure of sensitivity unaffected by response bias, robust to imbalanced problems and independent of the statistical distribution of the classes (for a similar approach, see Azzopardi and Cowey, 1997, Charles et al., 2014, Tamietto et al., 2015, Van den Stock et al., 2014).

The AUCs were estimated for each participant and above-chance significance across participants was computed, separately for real and pantomimed reach-to-grasp movements, by means of one-sample t-tests. To verify whether the ability to infer object weight differed depending on the type of reach-to-grasp movement being observed, AUC values of real and pantomimed movements were then compared by means of a paired-sample t-test. To aid comparison with previous works, we also calculated the sensitivity (d′), and criterion (c).

In order to control for different movement durations, reaction times were normalized by dividing the actual reaction time by the duration of each specific movement video clip. Participants’ correct responses whose normalized RTs deviated by more than 2.5 SD were treated as outliers, and removed from further analyses. Outliers and no-response trials accounted for less than 4% in real and pantomimed reach-to-grasp movements. Normalized RTs of real and pantomimed reach-to-grasp movements were then submitted to a paired-sample t-test. For all statistical tests the alpha level of significance was set to 0.05.

Results

Full results are reported in Table 2. AUC values were significantly above the chance level of 0.50 for both real (M = 0.55, 1SE = 0.01) (t(23) = 4.50, p < 0.001, d = 0.92, 95% CI [0.02, 0.07]) and pantomimed reach-to-grasp movements (M = 0.53, 1SE = 0.01) (t(23) = 3.29, p = 0.003, d = 0.67, 95% CI [0.01, 0.05]); Fig. 2). This indicates that participants were able to identify the weight of the to-be-grasped object from both occluded real and pantomimed movements, solely using available kinematic information. For both real and pantomimed reach-to-grasp movements, the fitted ROC curves did not differ significantly from the empirical curves (χ2 (5) = 5.37, p > 0.250 and χ2 (5) = 2.11, p > 0.250, respectively). Comparison of AUC values between real and pantomimed reach-to-grasp movements revealed no significant difference (t(23) = 1.01, p > 0.250, d = 0.21, 95% CI [−0.01, 0.04]), suggesting that the ability to predictively judge object weight did not differ depending on the type of movement being observed.

Table 2

Results from one-sample t-tests on AUC, d' and c values. M = Mean; SE = Standard Error; t = t-test; d = Cohen’s d; 95% CI = 95% Confidence Interval of the difference from the test value (i.e., 0.50 for AUC and 0 for d’ and c).

Real reach-to-grasp movements						Pantomimed reach-to-grasp movements
	M ± 1SE	t	p value	d	95% CI	M ± 1SE	t	p value	d	95% CI
AUC	0.55 ± 0.01	4.50	<0.001	0.92	0.02, 0.07	0.53 ± 0.01	3.29	0.003	0.67	0.01, 0.05
d'	0.16 ± 0.05	3.31	0.003	0.67	0.06, 0.25	0.13 ± 0.05	2.72	0.012	0.56	0.03, 0.22
c	0.00 ± 0.04	0.05	>0.250	0.01	−0.08, 0.09	0.01 ± 0.03	0.40	>0.250	0.08	−0.06, 0.08

Fig. 2

Weight discrimination from observed real and pantomimed reach-to-grasp movements. Results for AUC for real (a) and pantomimed (b) reach-to-grasp movements at individual (grey lines) and group (black line) level. Participants were able to correctly discriminate the weight of the to-be-grasped object from the observation of both real and pantomimed movements (p < 0.001 and p = 0.003, respectively). The dashed line indicates a random guess performance (AUC = 0.50).

Analysis of RTs also revealed no difference between discrimination of weight from real (M = 2.05, 1SE = 0.13) and pantomimed reach-to-grasp movements (M = 1.99, 1SE = 0.12) (t(23) = 1.05, p > 0.250, d = 0.21, 95% CI [−0.06, 0.17]).

Discussion

In the present research, we investigated whether observers can take advantage of information gleaned from the observation of real and pantomimed reach-to-grasp movements to form a shared representation of the properties of the object to-be-grasped – be it real or imaginary.

Perceptual weight-judgment tasks require deriving a hidden state (i.e., the weight of the object) from the kinematics of an observed action. Previous studies indicate that participants are able to do so from observing a person lifting an actual object (e.g., Bingham, 1987, Hamilton et al., 2007, Runeson and Frykholm, 1981). Our results extend these findings in two ways.

First, we demonstrate that observers are able to predictively extrapolate the weight of the to-be-grasped object from advanced kinematic sources, i.e., pre-contact kinematics. Early in the movement, hand kinematics conform to the properties of the object to be grasped (Ansuini, Cavallo, Campus, et al., 2016, Ansuini, Cavallo, Koul, et al., 2016). Our findings imply that observers are able to detect differences between movements tailored to different anticipated weights, and predictively link these differences to different weight representations to anticipate whether an occluded object is light or heavy. Put simply: we can share through others’ actions the characteristics of an object yet to be grasped.

Second, we show that predictive weight judgments are also possible from pantomimed actions aimed at imagined objects. Observers are able to judge whether an imagined object is light or heavy solely by using available kinematic information. This finding has implication for our understanding of how object knowledge is converted into pantomimes in that it suggests that, similarly to real grasps, pantomimed grasps are planned by specifying the features of the intended, albeit imaginary, goal-object. What is more, it implies that, well before grasp, the distinctive features of the imagined object become available for perception. In other words, pantomimed reach-to-grasp movements do not only demonstrate the interaction of the hand with a pretended light or heavy object, but also create the expectation of an interaction with an object of varying weight. It is tempting to speculate that the power of empty-handed gestures to convey aspects of imaginary objects might be related to this anticipatory component.

These results extend the limits of experience sharing, suggesting that observers are able to use early kinematic information to make predictions, and form expectations about the characteristics of not only real objects, but also objects that exist only in others’ minds. Apparently, the weight of an imagined object was no more difficult to anticipate than the weight of a real occluded object. It will be important for future work to quantify the resolution of movement observation, i.e., how accurately weight can be estimated from observation of real and pantomimed grasps, and to determine the exact kinematic features used to extract object-related information in the two domains. Finally, it will be important to consider these results from the perspective of the underlying computational and neural mechanism. Perceptual weight judgments have been proposed to involve the mapping of a perceptual representation of the observed action onto a representation of the appropriate motor pattern for the same action in the observer (e.g., Alaerts et al., 2010, Alaerts et al., 2010, Bosbach et al., 2005, Hamilton et al., 2004). More specifically, judging the weight of an object acted upon would require a detailed motoric simulation of the changes in kinematic parameters over the course of the observed action (Pobric & Hamilton, 2006; but see Hamilton et al., 2007). It will be important for future studies to determine which specific features of the observed actions are simulated and the extent to which simulation of pantomimed actions requires experience of physical performing the actions. This could be tested in professional magicians regularly using pantomimed actions. Magicians have been shown to be better than controls at calibrating pantomimed grasps (Cavina-Pratesi et al., 2011). If motor experience influences motor simulation, then we would expect them to be also more accurate in predictively estimating the weight of a pretended object.

Declaration of conflicting interests

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.