When Does One Decide How Heavy an Object Feels While Picking It Up?

When lifting an object, it takes time to decide how heavy it is. How does this weight judgment develop? To answer this question, we examined when visual size information has to be present to induce a size-weight illusion. We found that a short glimpse (200 ms) of size information is sufficient to induce a size-weight illusion. The illusion occurred not only when the glimpse was before the onset of lifting but also when the object's weight could already be felt. Only glimpses more than 300 ms after the onset of lifting did not influence the judged weight. This suggests that it takes about 300 ms to reach a perceptual decision about the weight.

In order to make perceptual decisions about properties in our environment, we combine sensory information with expectations based on prior experience (Kersten, Mamassian, & Yuille, 2004; Summerfield & de Lange, 2014). For instance, prior experience with one of an object’s properties, such as its material or size, influences how heavy the object feels (Buckingham, 2014; Buckingham, Cant, & Goodale, 2009; Buckingham & Goodale, 2010a; de Brouwer, Smeets, & Plaisier, 2016; Ellis & Lederman, 1998, 1999; Ross, 1969). The best-known example of this is the size-weight illusion: a large object is perceived to be lighter than a smaller object of the same weight (for a recent review, see Saccone & Chouinard, 2018). The size-weight illusion is a robust effect that occurs even if the perceiver knows that both objects have the same mass (Flournoy, 1894). It also occurs when heaviness is judged by pushing an object instead of lifting it (Plaisier & Smeets, 2012; Platkiewicz & Hayward, 2014), and it occurs when size is felt instead of seen (Ellis & Lederman, 1993; Plaisier & Smeets, 2015). As is the case with influences of other priors, it is possible to alter the size-weight illusion by training (Flanagan, Bittner, & Johansson, 2008). Size can affect perceived weight even if the object is shown only prior to lifting (Buckingham & Goodale, 2010b), suggesting that weight expectations prior to lifting might play a role (but see Masin & Crestoni, 1988, for counterevidence). Direct somatosensory information about an object’s weight becomes available as soon as the object loses contact with its supporting surface. After more time passes, we reach a decision as to how heavy the object feels. Our question is what is the time course of this perceptual decision-making process?

Prior to “liftoff,” the only information that one has about an object’s mass are expectations based on what it looks like and a statistical relation between its appearance and weight. After liftoff, the gravitational and inertial forces provide unambiguous sensory information about the object’s mass. If seeing the size of the object influences the judged weight because it provides an expectation of the force required to achieve liftoff, it should become much less effective as soon as expectations become irrelevant, that is, after the liftoff has already occurred. If so, information about size presented after liftoff should not influence the perceived weight. Alternatively, if size information is considered throughout the judgment, there is no reason to expect the moment of liftoff to have a special relevance, so presenting size information will remain effective until the decision has been made.

Perceptual decision making is usually studied in situations in which a choice needs to be made between two alternatives (Shadlen & Kiani, 2013): a one-bit decision. Other judgments involve more alternatives, for instance, three for judging the color of a traffic light or four for judging the suit of a playing card. One can interpret the number of bits of information as the number of binary decisions underlying the judgment (e.g., two bits for judging the suit of a playing card). Object properties such as size or weight can vary on a continuous scale, so a judgment of such properties could involve an infinite number of alternatives. However, given the finite precision of such a judgment, one can regard them as the outcome of a set of binary decisions, with the number of decisions corresponding to the relative precision expressed as bits of information (Fitts, 1954; Summerfield & de Lange, 2014).

The time needed for decisions that are more complex than a binary decision is known to scale with the number of bits of information. For instance, choice reaction times increase linearly with the number of bits of information processed (Hick, 1952; Hyman, 1953). Therefore, we can expect the time needed to reach a perceptual decision on a continuous scale to increase with the relative precision of the percept (expressed in bits). Here, we monitored the process of judging an object’s weight by varying the time at which visual information about its size was made available during a lifting action. Using three experiments that differed in when participants were allowed to view the object they were lifting and what happened after they lifted the object, we determined the time course of when visual size information can influence weight judgments.

Method

Participants

Ten participants (2 male; all right handed; age: M = 22 years, SD = 3) were recruited for Experiment 1. A second group of 10 participants (3 male; all right handed; age: M = 28 years, SD = 3) was recruited for Experiment 2. A third group of 12 participants (6 male; 2 left handed; age: M = 25 years, SD = 4) was recruited for Experiment 3. Each participant only completed one of the experiments. None of the participants was aware of any relevant sensory or motor deficits. All participants were naive as to the purpose of the experiments. They were treated in accordance with the local ethical guidelines and gave informed consent prior to participating. We used 10 participants on the basis of earlier experience that this sample size allowed for an easy detection of the illusion using the present stimuli (Plaisier & Smeets, 2012). We included 2 more participants in Experiment 3 after observing the results of Experiment 2. The study was part of a program that was approved by the Scientific and Ethical Review Board of the Faculty of Behavioural and Movement Sciences of Vrije Universiteit Amsterdam.

Stimuli and setup

We used objects of two sizes: small (6 × 6 × 6 cm) and large (6 × 6 × 9 cm; Fig. 1a). A plastic handle was attached to the top of each object. We let participants lift the objects by a handle so that they could not deduce the size from the grip aperture when holding the object. We made sure that wielding the object could not provide information about its size (Amazeen & Turvey, 1996; Kingma, van de Langenberg, & Beek, 2004) by connecting the handle to the object by a rotatable joint. In Experiment 1, we used two pairs of objects (one pair of 260 g and one pair of 210 g, including the handle); in Experiments 2 and 3, only the pair of objects weighing 260 g was used. An infrared marker was attached to the surface of each object at the center of one side. Its position was tracked using an Optotrak 3020 system (Northern Digital, Waterloo, Ontario, Canada). The objects were placed on a force sensor so we could measure the lifting force (ATI Industrial Automation, Apex, NC; Nano17 F/T Sensor). The position and force-sensor signals were sampled synchronously at 500 Hz. Participants wore computer-controlled PLATO visual-occlusion goggles (Translucent Technology, Toronto, Ontario, Canada).

Fig. 1.

Stimuli and procedure. Participants were asked to lift small and large objects (a) using handles connected to each object by a hinge. Lifting objects in this way removed all haptic size cues. An infrared LED was attached to each object to track its position. In Experiment 1, there were three conditions (b), which differed in the timing of when the occlusion goggles worn by participants opened (the gray shading in the figure indicates when they were closed). In the two conditions in which the occlusion goggles opened, the object had to be placed on the square that corresponded to its size. Participants lifted the object off a force sensor, allowing us to precisely determine the time of liftoff. In Experiment 2 (c), the procedure was largely the same as in Experiment 1, except that the goggles opened for 200 ms at varying moments with respect to liftoff. The procedure for Experiment 3 (not illustrated) was the same as for Experiment 2, except that participants did not place the object back on the table.

Procedure

Participants were seated at a table with the occlusion goggles closed. The experimenter placed an object in front of the participant and indicated that he or she could grasp the handle with the dominant hand. The experimenter manually guided the participant’s hand to the handle. Participants were instructed to wait while holding the handle until an auditory go cue sounded. At that moment, they were to lift the object straight up without shaking or rotating it. In Experiments 1 and 2, they subsequently placed it back on the table at a specific position. In Experiment 3, the experimenter removed the object from the participant’s hand, so participants never moved the object down after lifting it. If the object was to be placed on the table, participants had to complete the whole movement within 3 s. Otherwise they had to reach maximum height within 2 s. After completing each trial, participants were asked to indicate the object’s weight using a method of free magnitude estimation (Zwislocki & Goodman, 1980). Participants performed 10 practice lifts to become acquainted with the task prior to starting the main experiment. Practice was performed with an object that was not part of the stimulus set.

In Experiment 1, there were three conditions: no vision, late vision, and continuous vision (Fig. 1b). In the no-vision condition, the goggles remained closed throughout the trial. In the late-vision condition, the goggles opened as soon as the object was raised 5 mm above the table surface. In the continuous-vision condition, the goggles opened roughly 0.5 s prior to the go cue. This experiment consisted of three blocks of trials. The first and third block each consisted of 20 no-vision trials (5 per object). In these blocks, participants placed the object on the table in front of them. In the second block, participants performed a total of 80 late-vision and continuous-vision trials (10 per object in each condition), which were randomly interleaved. During this block of trials, drawings of a large and small square on the table indicated on which side (left or right) to place each object. Halfway through the block, these locations were reversed. Participants placed the object at the correct side on all trials so we could be sure that they had taken note of the size of the object.

In Experiment 2 (Fig. 1c), the goggles opened for 200 ms during every trial. The moment at which the goggles opened was varied with respect to the auditory go cue. The goggles could open 200 ms prior to the go cue or 100 ms, 400 ms, 700 ms, or 1,000 ms after the go cue. Given the variability in response times, these opening times resulted in a more or less uniform distribution of times of visual information relative to lift onset throughout all phases of lifting. Each of the five opening times was presented 10 times for both objects, resulting in 100 trials per participant. Trials were performed in blocks, with one trial of each opening time for each object randomly interleaved within each block to ensure an even distribution of all opening times throughout the experiment. Participants placed the small object on the left and the large object on the right. We did not switch left and right placement halfway through, as in Experiment 1, because in this case the goggles were always closed during this part of the trial. Thus, participants could not see the drawings of the small and large rectangles on the table and had to remember where to place which object size. On average, participants did this correctly in 98.4% of the trials (minimum individual trials correct was 92%).

Experiment 3 was identical to Experiment 2 except that after lifting, participants did not place the object back on the table but held it in the air until a second auditory cue (2 s after the go cue) indicated that the experimenter was going to remove it from their hand. To ensure that participants noticed the size of the object, we asked them to report whether it was a large or a small object after giving their heaviness rating. On average, participants judged the size correctly in 98.5% of the trials (minimum was 93%).

Analysis

We first converted heaviness ratings into z scores for each participant individually to be able to compare the heaviness ratings across participants. To this end, we took the heaviness ratings for all trials of an individual participant and calculated the mean and standard deviation across all trials. To arrive at the z scores, we subtracted the mean from each heaviness rating and divided the result by the standard deviation.

In Experiments 2 and 3, we determined the moment of liftoff from the force-sensor signal with a method that we adapted from the recommendation of Oostwoud Wijdenes, Brenner, and Smeets (2014). We fitted a line through the signal between 50% and 80% of the maximum force. We used this period because it was the smoothest part of the force profile. We excluded a trial if the R2 value of the fit was below .6 (this happened in 1.8% of the trials in Experiment 2 and 3.1% of the trials in Experiment 3). We took the intersection between the fit line and a line at the level of no force (the average of the last 100 samples during which there was no object on the force sensor) as the moment of liftoff. In the late-vision condition of Experiment 1, the opening of the goggles happened 120 ms (between-participants SD = 30) after the moment of liftoff that was determined in this way.

In Experiment 1, we calculated a single illusion magnitude for each participant, object mass, and condition by subtracting the z scores for the large object from those for the small object of the same mass. We subsequently performed a repeated measures analysis of variance (ANOVA) on the illusion magnitude with object mass and condition as factors. We followed this up with post hoc paired-samples t tests to determine whether the illusion magnitude differed between the conditions. In all statistical tests, we considered p < .05 to be significant.

In Experiments 2 and 3, we determined the time at which the visual size information was provided (“time of visual information”) for each trial as the difference between the time of liftoff and the center of the 200-ms time window during which the goggles were open. We subsequently transformed the heaviness ratings (expressed as z scores) to smooth functions of the time of visual information for each participant by calculating a Gaussian weighted average for each instant and object. The Gaussian function had a standard deviation of 50 ms and was shifted in steps of 1 ms. Within the range that we show in the figures, there was at least 1 data point within every 100-ms interval for each participant and object size. We calculated illusion magnitude as a function of time of visual size information for each participant by subtracting the heaviness rating function for the large object from that for the small object. In order to relate the heaviness ratings to the lifting movement, we determined three parameters in addition to the moment of liftoff: loading-phase onset, time of half height, and time of maximum height. We used the moment at which the loading force first exceeded 0.2 newton as the loading-phase onset. Time of half height and time of maximum height were determined in a straightforward manner from the Optotrak position signal. These two experiments were exploratory: No hypotheses are tested; 95% confidence intervals around the mean are provided as an indication of precision.

Results

In Experiment 1, we tested whether the size-weight illusion occurs if size information is provided only immediately after liftoff, when the decision process has just started. As was to be expected, we did not find an illusion in the no-vision condition, and we found a clear illusion with continuous vision. Limiting vision to the period after liftoff reduced the illusion to less than half of its magnitude with continuous vision (late-vision condition). A repeated measures ANOVA on the illusion magnitude showed a significant effect of condition, F(2, 18) = 12.18, p < .001, η2 = .575, no effect of object mass, and no interaction effects. Post hoc paired-samples t tests with Bonferroni correction showed a significant difference between the no-vision and continuous-vision conditions, t(9) = 4.12, p = .008, and between the late-vision and continuous-vision conditions, t(9) = 3.94, p = .010, but not between the late-vision and no-vision conditions, t(9) = 1.55, p = .47. Thus, the size-weight illusion decreased considerably when visual information about the object’s size was available only after the decision process had started, so much so that performance was statistically indistinguishable from having no visual size information.

Although the illusion effects in the late-vision and the no-vision conditions were indistinguishable, we cannot conclude that visual size information was ignored from the moment of liftoff, when the decision-making process presumably started. The size of the illusion effect and its associated 95% confidence interval in Figure 2a leave the possibility open that the illusion did not disappear completely in the late-vision condition (despite the magnitude not being significantly different from that in the no-vision condition). It is possible that visual information influenced perceived weight even after liftoff up to a certain moment during the decision-making process. To test this hypothesis, we conducted a more detailed investigation of how visually presenting size information at different times during the decision process influences the judged weight.

Fig. 2.

Results (averaged across participants). For Experiment 1 (a), the illusion magnitude is shown for each of the three conditions. Data bars show means, circles indicate values for individual participants, and error bars show 95% confidence intervals. For Experiment 2 (b) and Experiment 3 (c), the illusion magnitude is shown as a function of the time when visual information was provided with respect to liftoff. The full illusion magnitude as found in the continuous-vision condition of Experiment 1 is indicated by the red dashed line. The gray horizontal bar indicates the loading phase, and the black dots indicate the moment at which the half height and maximum height were reached. Shaded bands indicate 95% confidence intervals. Note that in Experiment 3, the full illusion effect did not occur for very late presentations because participants did not place the object back on the table.

In Experiment 2, the goggles opened very briefly (200 ms) once every trial. Despite this very short presentation of visual size information, the illusion was strong. If visual size information was provided before liftoff, the participants in Experiment 2 were influenced by the short window of visual information to a similar extent as the participants in Experiment 1 were influenced by continuous vision of the object (Fig. 2b; curve slightly above the red dashed line). The illusion magnitude remained approximately the same when vision was provided up to 300 ms after liftoff. Visual size information thus influenced the perceived weight until well after the start of the decision-making process.

The size-weight illusion was reliably lower than for the full illusion in Experiment 1 only when the visual size information was provided between 330 ms and 500 ms after liftoff (when the object had already reached more than half of its maximal height). Surprisingly, the illusion returned to its full magnitude when vision was provided around 600 ms after liftoff, at about the moment at which the maximum height was reached. At that moment, the object was being held more or less stationary in these trials, because participants were waiting for the visual information to appear in order to decide on which square they should place the object. Possibly, the start of the downward movement induced a reevaluation of the perceptual decision, which might have been responsible for the illusion also occurring in this situation.

In Experiment 3, we tested the robustness of our results and investigated the occurrence of the illusion when size information is provided very late without a new movement possibly tempting one to reevaluate the decision. To do so, we repeated Experiment 2 but without letting participants place the objects back on the table. They were instructed to lift the object and hold it in the air until the experimenter removed the object from their hand. In Experiment 3, we replicated the results of Experiment 2: The illusion decreased only when vision was provided well after liftoff (Fig. 2c). The illusion persisted for even slightly later moments of providing visual size information than in Experiment 2 (up to 400 ms after liftoff). In line with our explanation for the reoccurrence of the illusion in Experiment 2, the illusion did not return to its full magnitude when vision was provided later after liftoff.

Discussion

The size-weight illusion was markedly reduced when visual size information became available only after liftoff in Experiment 1 (Fig. 2a), suggesting that the use of prior information stopped when sensory input about weight became available. By providing only a short glimpse of visual information, we could determine the timing at which this reduction occurred more precisely in Experiments 2 and 3 (Figs. 2b and 2c). We found that the illusion did continue to occur for visual information that was provided briefly up to 400 ms after liftoff (Figs. 2b and 2c). We can thus conclude that information related to prior experience affected the decisions well after sensory input about weight became available and thus after the decision-making process had started. We can also conclude that the decision process took at least 330 ms and 400 ms in Experiments 2 and 3, respectively.

At first glance, this interpretation of Experiments 2 and 3 might seem inconsistent with the results of Experiment 1. In the late-vision condition of Experiment 1, the illusion was considerably reduced when visual information was continuously available after the object had moved 5 mm upward, about 120 ms after liftoff. In Experiments 2 and 3, we found a full-strength illusion when visual information was provided briefly at that time. This difference is probably due to the fact that we did not control when participants determined the size of the objects in Experiment 1 as precisely as we did in Experiments 2 and 3. In the latter experiments, participants had to look at the objects during the brief exposure in order to know the size, while in Experiment 1, they could have looked at the object at any time after the goggles opened and knew that they could do so. This could be why the average illusion effect in the late-vision condition of Experiment 1 was in between no effect and a full-strength illusion.

In Experiment 2, the decision about weight appears to have been reached 70 ms earlier than in Experiment 3. We argued in the introduction that the time needed for a perceptual decision on a continuous scale depends on the precision of the percept. If the perceptual decision was indeed made more quickly in Experiment 2 than in Experiment 3, one would expect that the participants in Experiment 2 would have been less precise than those in Experiment 3. We therefore determined the precision for each participant on the basis of the variation of the responses for all trials for a single object in which the visual information was provided before liftoff. We indeed found that this coefficient was higher (less precise) in Experiment 2 (coefficient of variation = 0.15) than in Experiment 3 (coefficient of variation = 0.12).

Our data show that it takes at least 330 ms to reach a decision on how heavy an object feels. We cannot exclude the possibility that the decision-making process was still in progress after 330 ms. On the other hand, one third of a second has been claimed to be the typical duration of embodied decisions (Ballard, Hayhoe, Pook, & Rao, 1997). Is 330 ms indeed a reasonable time for a perceptual decision of this precision? The observed values for the coefficient of variation in the perceptual judgments correspond to about three bits of information (Welford, 1960). If the decision-making process would indeed have finished at the moment visual information about size ceased to have an effect, the information-processing capacity would have been about 10 bits per second, which seems a reasonable value for human sensorimotor processing (Fitts, 1954; Welford, 1960). So it is likely that the time it took to reach a decision indeed coincided with the time that visual information had an effect after liftoff.

We interpreted the fact that visual information affected weight perception for more than 300 ms after the haptic information became available as indicating that the indirect size information was combined with haptic information to judge heaviness even when it was presented considerably after direct weight information became available. One could argue that this is not necessarily the case: If tactile information were processed more than 300 ms slower than visual size information, the visual size information might have been available to the relevant parts of the brain before the haptic weight information. We consider this to be unlikely because tactile information is known to be processed within 100 ms to stabilize the grasp (Johansson & Flanagan, 2009). It is known that the judged timing of signals can shift to some extent with repeated exposure when judging simultaneity (Sugita & Suzuki, 2003), but it is also known that we do not correct for processing-time differences when using signals to control goal-directed movements (van Mierlo, Louw, Smeets, & Brenner, 2009), so we may also not adjust the timing for making judgments on the basis of lifting movements. Even if the timing of signals would be shifted, it is very unlikely that such a shift would influence our conclusions substantially, as reported shifts were less than 100 ms. Note that in the above-mentioned cue-combination studies, the temporal-integration window was also clearly less than 100 ms, so a sluggish temporal integration also cannot explain our finding that visual information presented 300 ms after liftoff affected heaviness ratings.

There are two approaches to explain the size-weight illusion: a top-down and a bottom-up approach. The top-down approach involves expectations (Buckingham, 2014; Ross, 1969), quantified as anti-Bayesian (Brayanov & Smith, 2010) or Bayesian priors (Peters, Ma, & Shams, 2016). Our results are clearly in conflict with such explanations because the visual information that is supposed to set the prior was just as effective when it was presented after the haptic information. The results are in line with an explanation in terms of a bottom-up combination of a direct and an indirect cue (Anderson, 1970; Masin & Crestoni, 1988). For this approach, one needs to identify the indirect cue. One suggestion is that object density is this indirect cue (Wolf, Bergman Tiest, & Drewing, 2018). However, the size-weight illusion is equally strong for objects that differ in size but clearly not in amount of material and thus not in density (Plaisier & Smeets, 2015). So this explanation of the size-weight illusion is still lacking a convincing candidate for the indirect cue.

In summary, our results show that perceptual decisions can be affected by prior knowledge that is invoked at a moment at which direct sensory information is already available. However, once a perceptual decision has been reached, prior knowledge does not lead to a reevaluation of the decision. Changes in direct sensory information, for instance due to a new motor action, could lead to reevaluation of the decision, in which recently invoked prior knowledge is also considered. Overall, this study provides a first account of the time course of the use of prior knowledge in making perceptual decisions on a continuous scale.