Using a robotic teleoperation system for haptic exploration.

INTRODUCTION: When children with physical impairments cannot perform hand movements for haptic exploration, they miss opportunities to learn about object properties. Robotics systems with haptic feedback may better enable object exploration.
METHODS: Twenty-four adults and ten children without physical impairments, and one adult with physical impairments, explored tools to mix substances or transport different sized objects. All participants completed the tasks with both a robotic system and manual exploration. Exploratory procedures used to determine object properties were also observed.
RESULTS: Adults and children accurately identified appropriate tools for each task using manual exploration, but they were less accurate using the robotic system. The adult with physical impairment identified appropriate tools for transport in both conditions, however had difficulty identifying tools used for mixing substances. A new exploratory procedure was observed, Tapping, when using the robotic system.
CONCLUSIONS: Adults and children could make judgements on tool utility for tasks using both manual exploration and the robotic system, however they experienced limitations in the robotics system that require more study. The adult with disabilities required less assistance to explore tools when using the robotic system. The robotic system may be a feasible way for individuals with physical disabilities to perform haptic exploration.

Introduction

Many children with physical disabilities have a limited ability to manipulate objects due to limited gross and fine motor movements which may cause them to miss out on opportunities to play and learn about the properties of objects.[1,2] If children are unable to ascertain information about object properties such as rigidity, texture, or weight of objects through play, it may limit their ability to make judgments about objects that can be used as tools.[3-5]

Haptic exploration occurs when individuals manipulate objects and use their sense of touch to determine the physical characteristics of the objects.[6] Exploratory procedures (EPs) are the defined movement patterns of the hands that individuals use to extract information about specific object properties, for example; lateral motion is for determining texture, pressure for hardness, enclosure for shape and volume, static contact for temperature, and contour following for shape.[7,8] When individuals use EPs, they perceive object properties which in turn provide clues as to how to use objects as tools.[5,8,9]

Lederman and Klatzky examined EPs in a series of studies. In the first, adults were blindfolded and asked to match objects to a sample on a particular dimension (e.g., shape or texture), and their hand movements were observed.[7] Participants performed EPs corresponding to the object knowledge that was required for the match. Klatzky, Lederman, & Manikinen[8] later confirmed the role of haptic exploration in a Function Judgement Task. They asked 4-year-old children and adults to make decisions about the appropriateness of a tool to perform a functional task, i.e., sticks of varying rigidity to mix either sugar or gravel (Mixing subtask) or spoons of varying sizes to carry a small and large piece of candy (Transport subtask). Results showed that participants were able to perform perceptual analysis to judge if a tool was appropriate for the task through visual or haptic exploration, without needing to carry out the actual task. A later study replicated the Klatzky et al.[8] study with children 3 to 5 years old and adults,[9] however the participants were constrained to only use haptic exploration to judge the tool’s utility. The 3-year-olds explored objects less and were less accurate in their responses than the older children and adults. The results of the 4-year-olds were the same as the 4-year-olds in the Klatzky, Lederman, & Manikinen[8] study. Five-year-olds demonstrated adult-like EPs and were very accurate in their responses.

Individuals with disabilities could engage in haptic exploration with assistive robots. Switch controlled mobile robots have been shown to provide children with a means to manipulate objects and toys.[10] Likewise, robotic arms, often mounted on wheelchairs, allow individuals with disabilities to manipulate objects (i.e., pick-up a TV remote control) to perform daily activities.[11] However, so far the robotic interfaces used in such studies have not given the user the haptic sensory feedback about the object manipulated. Haptics-enabled robots controlled through teleoperation[12] could allow an individual to perform EPs and acquire sensory information about an object’s properties in order to determine how to use objects as tools.

This study explored if a haptics-enabled robotic teleoperation system could enable participants to perform haptic exploration and how exploration resembled or differed from haptic exploration using the hands. The robot system consisted of a stylus probe to keep the system simple (to resemble haptic exploration using one finger) and cautious gain parameters to ensure the system was safe and stable. These features make a system simpler and safer to implement, but it remains to be determined if they are sufficient to determine the physical characteristics of objects. The purpose of the study was to examine if haptic exploration using a robotic teleoperation system is a viable alternative to haptic exploration using the hands, for when haptic exploration using the typical EPs is not possible for people with physical disabilities. Study 1 was performed with adults and children without disabilities to examine robot capabilities and if children would perform EPs and haptic exploration with the system similarly to adults as in Lederman, Klatzky & Mankinen.[8] Study 2 was a case study with an adult with cerebral palsy to examine robot use by someone with a physical impairment. Study 2 was performed to explore how the system could be used by a participant with physical disabilities that affected their ability to freely manipulate objects and perform haptic exploration with hands. Cerebral palsy is a common disorder affecting approximately 1 in 1000 children, with a significant proportion of children having limitations in upper limb function.[13] The long term goal of this robotic development program is to assist children with physical impairments with environmental exploration. The Function Judgment Task used in the studies described above was replicated.[8,9] The following research questions were addressed in both Study 1 and 2:

How do participants respond on the Mixing and Transport subtasks when they use a robotic teleoperation system for exploration, and how does that compare to when they use their hands?

How do participants vary their EPs as a function of task (Mixing or Transport) when they use a robotic teleoperation system, and how does that compare to when they use their hands?

Study 1 – Adults and children without disabilities

Method

Study design

A crossover study design was used where participants performed the task in two conditions, No Tech (exploration with the hands) and Tech (exploration with the robotic system). The order in which the conditions were presented to participants was counterbalanced, so that half of the participants started with the No Tech condition, and the other half started with the robotic system condition. Ethics approval was sought from and granted by the University of Alberta Health Research Ethics Board, University of Alberta (Approval number Pro00049751).

Participants

A convenience sample of 24 adults (ages 19-52 years) and 10 five-year-old children participated in the study. The number of participants in the adult group was chosen based on the study performed by Kalagher.[9] In Kalagher 2015, 25 adults did the same Function Judgment Task and the study found significant differences in the participants’ responses. Twenty-four participants were recruited in the current study order to counterbalance participants' first condition. The number of participants in the child group was chosen based on Klatzky, Lederman, & Manikinen (2005) who performed a study using similar methods and tasks without a robot and attained significant differences with 10 children (aged 3 years, 11 months to 4 years, 11 months). Five-year-old children were included in the current study because previous research indicated that they would be able to perform appropriate EPs and tool selections based on manual haptic exploration,[9] thus giving us an opportunity to examine if the robot posed additional challenges. Participants had no sensory, physical or cognitive disabilities, and no experience using robotic teleoperation systems. Children were able to understand instructions and provide a “yes” or “no” response (assessment based on Kalagher[9])

Materials

The Function Judgement Task consisted of two subtasks, Mixing and Transport. Two target objects and five tools were needed for each sub task. For the Mixing subtask the target objects were a container filled with sugar and a container filled with gravel. The tools were five sticks constructed of plastics with varying degrees of rigidity and balsa wood for the most rigid stick (see Figure 1 upper left). The rigidity of the sticks was determined by how much they deflected when they were secured with 7 cm overhanging off a table, and with a weight of 120 grams hanging from them. The deflection of the sticks was 5.0 cm, 3.5 cm, 1.25 cm, 0.5 cm and 0 cm (called R1, R2, R3, R4, and R5, respectively, from least to most rigid). The task was to examine the stick and say if they thought it could stir the sugar or gravel. For the Transport subtask, the target objects were a round candy of approximately 4 cm diameter and a round candy of approximately 8 cm diameter (see Figure 1, upper right). The tools consisted of 5 spoons with circular bowls with different diameters, 2 cm, 3 cm, 4 cm, 6 cm and 8 cm (called S1, S2, S3, S4, and S5, respectively, from smallest to largest size). The task was to examine the spoon and say if they thought it could transport the small or big candy.

For the No Tech condition, a box with openings on opposite sides was used (see Figure 1, bottom left). One opening was covered with a curtain so that participants could place their hands inside to explore a tool without being able to see it. The other opening allowed the researcher to put the different tools inside. A video camera was placed facing the inside of the box to capture the participant’s EPs.

The teleoperation system in the Tech condition consisted of a user-side robot that controlled the movement of an environment-side robot, where the tools were placed (see Figure 1, bottom right). The robots were two 6-DOF Phantom Premium 1.5 A (3-DOF rotational and 3-DOF translational) haptic robots (Geomagic, Cary, NC). The movement of both haptic robots was constrained to a horizontal plane, to help the participants more easily interact with the objects. A panel was used to block the participant’s view of the environment during the tasks. A video camera was placed on the environment side of the teleoperation system facing the end effector of the robot to capture the EPs.

Figure 1.

Materials Function Judgment Task: Sticks (upper left) and spoons (upper right); box for No Tech condition (lower left); teleoperated robot for Tech condition (lower right). The participant is moving the end effector of the user-side robot on the right side of the barrier which moves the end effector of the environment-side robot on the left side of the barrier. The researcher on the left is stabilizing a stick in place while the user explores it. A bowl of sugar is visible to the user on the right side of the panel.

Procedures

A practice phase was given before the Tech condition to allow participants to freely explore the tools from the Mixing and Transport subtasks using the haptic robotic system. This was so participants could experience how to perform manipulation with the system and how much force to apply. We expected that the haptic experience would let them encode the object characteristics that would then support recognition of how the objects feel when using the robot.[14,15]

Participants performed the Mixing and Transport subtasks in both conditions (No Tech or Tech) in the order they were randomly assigned. The Mixing subtask always preceded the Transport subtask, as in Kalagher.[9] The order in which the two target objects and the five tools were presented within each subtask was randomized. The participants were given breaks as requested.

The general procedure was the same for each subtask. The target object was placed in the participant's view. While pointing at the target object, the researcher set the context: 1) her friend wanted to make a cake (for the sugar) or a mud pie (for the gravel), and she needed help finding a stick to mix the sugar or gravel; or 2) his friend wanted to fill a bowl with candy and needed a spoon to carry the candy in. The tools were presented to the participants one at a time, and participants were allowed to interact with the tool for a maximum of 10 seconds. In the No Tech condition, the tools were placed inside the box, and participants put their hands into the box to feel the tool. In the Tech condition, the tools were secured to the environment-side at the same marked point each time to have consistency between tools and sessions. The researcher placed the environment-side robot’s end effector against the tool and informed the participant that the end-effector was touching the tool. Then the participant moved the user-side robot, which moved the environment-side robot, to examine the tool. After each tool the participants were asked, “Do you think your friend can use this?” and participants provided a “Yes” or “No” response.

Data collection

There were two dependent variables, response (“yes” or “no”) and types of EPs. The responses about each tool and target object were recorded on a scoring sheet during the experiment. For the Mixing subtask, “yes” responses were expected to be more frequent with the sugar target object than the gravel target object, and to increase as the sticks became more rigid. For the Transport subtask, “yes” responses were expected to be more frequent for the small candy target object than the big candy target object, and to increase as the spoons became larger.[8,9]

The EP coding was done from video recordings, based on the methods used by Kalagher.[9] Whenever a participant produced any of the EPs defined by Lederman & Klatzky[7] it was recorded on a score sheet. An EP was only counted once as long as the participant continued performing it without stopping or switching to a different EP. For example, if the participant ran the robot effector along the stick several times without stopping, it was counted as one Lateral Motion. However, if the participant performed Lateral Motion, then switched to Pressure and returned to Lateral Motion, it was counted as two Lateral Motions and one Pressure. An EP occurred that was not noted in the Klatzky, Lederman & Mankinen[8] or Kalagher[9] studies, but was previously described by Lederman & Klatzky[7] in a matching task. The EP was called Function Test and when participants performed it, they executed movements related to the object and the task goal. The movements of interest in the current study were pretending to use the stick to stir something in the “air” or pretending to carry an imagined object with the spoon.

An EP that was not mentioned in the Klatzky and Lederman study[15] was observed in the Tech condition, which we called “Tapping”. It consisted of participants gently tapping a point of the spoon tool with the robot’s end-effector and then moving in a straight line until they tapped the opposing point of the tool, often repeating the movement multiple times; it seemed that participants were doing this to determine the distance between the two points. This EP was determined to be separate from “Pressure”, which is defined by Klatzky and Lederman as “applying torque or normal forces to one part of the object, while another part of the object is stabilized or an opposing force is applied. This can be seen by obvious movement, as in poking, or by signs of force evident in the fingers and hand.”[15] If participants did not show signs of force and were actively moving back and forth between two points the movement was determined to be “Tapping” and not a variant of the “Pressure” EP. Likewise, “Tapping” was determined to be separate from Contour following, which is defined as ”a dynamic EP in which the hand maintains contact with a contour of the object. Typically, the movement is smooth and nonrepetitive within a segment of object contour, stopping or shifting direction when a contour segment ends”.[15] If participants did not maintain contact with a contour of the object throughout the movement it was determined to be “Tapping”.

The first author coded all the videos, and the second author coded 30% of the videos. Comparing the EP coding point by point, inter-rater reliability on the type and frequency of the EPs was 90%. Based on Klatzky, Lederman, & Manikinen[8] and Kalagher[9] rigidity was the relevant perceptual dimension for the Mixing subtask, and therefore it was expected that participants would execute more of the Pressure EP. In the Transport subtask, size was the relevant perceptual dimension, and therefore, it was expected that participants would execute the Contour Following EP.

Data analysis

GraphPad Prism (GraphPad, San Diego, USA) software was used to complete all statistical analyses. Statistical comparisons of the frequency of “yes” responses and EPs were performed between target objects and conditions. When data were found to be normally distributed, a paired t-test was conducted for comparisons between two groups and a repeated measures one-way ANOVA was conducted for comparison between three or more groups. When data were not found to be normally distributed nonparametric tests were run: the Wilcoxon matched-pairs test was conducted for comparisons between two groups, and Friedman’s Test was conducted for comparisons between three or more groups. Significance was set at p < 0.05. Preliminary analyses revealed that there was no significant effect for the order in which participants performed the Tech and No Tech conditions (p > 0.13), therefore the data were combined for each condition.

Results

Figure 2 shows the average frequency of “yes” responses for each tool in the Tech and No Tech conditions. Figure 3 shows the mean frequency and standard deviation of adults’ and childrens’ “yes” responses in each condition for each subtask.

Figure 2.

Frequency of “yes” responses for each tool in the Mixing Subtask for (a) adults with No Tech, (b) children with No Tech, (c) adults with Tech, and (d) children with Tech. Frequency of “yes” responses in the Transport Subtask for (e) adults with No Tech, (f) children with No Tech, (g) adults with Tech, and (h) children with Tech. R1, R2, R3, R4, and R5, are the sticks from least to most rigid and S1, S2, S3, S4, and S5, are the spoons, from smallest to largest size.

Figure 3.

Mean frequency and standard deviation of adults’ and childrens’ “yes” responses in No Tech and Tech conditions for the Mixing (gravel and sugar) and Transport (big and small candy) subtasks.

As expected, we can see in Figure 3 that when adults and children explored with their hands (No Tech condition), they responded “yes” significantly more frequently for sticks to stir the sugar than the gravel, and for spoons to carry the small candy than the big candy. When using the robot (Tech condition), adults answered “yes” significantly more frequently for spoons for the small candy than the big candy, but there was no significant difference in responses for sticks to stir the sugar and gravel. There were no significant differences in either subtask for children in the Tech condition. Comparing responses in the Tech versus No Tech condition, the only significant difference was that both adults and children answered “yes” significantly more frequently for tools for the big candy in the Tech condition than in the No Tech condition.

Table 1 shows the EPs that were performed by the participants for each subtask during the Tech and No Tech conditions. In the Mixing subtask, adults and children performed Pressure, the expected EP for that subtask, more often than the other EPs in both the Tech and No Tech conditions. Both adults and children performed it significantly more frequently in the Tech condition than the No Tech condition. Both adults and children also performed Lateral Motion significantly more frequently in the Tech condition than the No Tech condition. Function Test was performed by adults significantly more frequently in the No Tech condition than the Tech condition. In the Transport subtask, adults and children performed Contour Following, the expected EP for that subtask, more often than the other EPs in both the Tech and No Tech conditions. Children performed Contour Following significantly more frequently in the Tech condition than the No Tech condition. Adults and children performed Enclosure in the No Tech condition, which was not possible in the Tech condition, making the difference significant. Tapping was used significantly more frequently in the Tech condition than the No Tech condition by both adults and children.

Table 1.

Mean count and standard deviation (mean (standard deviation)) of EPs performed by the participants for each subtask during the Tech and No Tech conditions in the Mixing and Transport subtasks.

	Mixing				Transport
	Adults		Children		Adults		Children
	No Tech	Tech	No Tech	Tech	No Tech	Tech	No Tech	Tech
Lateral motion	0.33 (0.82)a	2.71 (4.32)a	1.20 (2.82)a	6.50 (2.99)a	0.04 (0.20)	0.17 (0.48)	0.50 (0.85)	0.90 (1.66)
Pressure	10.29 (1.23)a	11.79 (2.47)a	9.80 (1.03)a	15.1 (3.54)a	0 (0)	0 (0)	0 (0)	0.60 (1.26)
Static contact	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0.10 (0.32)	0 (0)
Unsupported holding	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Enclosure	0 (0)	0 (0)	0 (0)	0 (0)	4.33 (3.05)a	0 (0)a	5.60 (3.78)a	0 (0)a
Contour following	0.13 (0.61)	0 (0)	0 (0)	0 (0)	7.50 (2.38)	8.54 (2.83)	7.20 (2.94)a	10.10 (1.10)a
Function test	1.04 (1.81)a	0.04 (0.20)a	0 (0)	0 (0)	0.04 (0.20)	0 (0)	0 (0)	0 (0)
Tapping	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)a	3.96 (4.69)a	0 (0)a	2.30 (3.02)a

aStatistical difference between No Tech and Tech (p < 0.05).

Discussion

This study set out to determine if participants responded the same and used similar EPs when they used a robotic teleoperation system compared to when they used their hands to explore tools for functional subtasks. When adults and children used their hands to do the task (No Tech condition) the “yes” responses for sticks were significantly more frequent for the sugar than the gravel target object, and increased as the tools became more rigid. Likewise, “yes” responses were significantly more frequent for spoons with the small candy than the big candy, and increased as the spoons became bigger. Participants understood that the gravel target object required more rigid tools than the sugar target object, and that the big candy target object required larger spoons than the small candy target object, therefore a wider variety of tools would function to meet task demands with the sugar and small candy target objects. These results are the same as those from the previous studies that the current study was based on.[8,9]

However, in the Tech condition, the only significant difference in “yes” responses was for adults on the Transport subtask, where they had significantly more frequent “yes” responses for the small candy than the big one. Visual analysis of the frequency of “yes” responses per tool in Figure 2 shows a general trend that as sticks became more rigid, or spoons became larger, the “yes” responses increased for both adults and children. Thus, though not as accurate as in the No Tech condition, their responses were still somewhat sensitive to the tool’s characteristics.

The lack of a significant difference between “yes” responses between target objects for adults and children on the Mixing subtask and for children on the Transport subtask could be due to a robot limitation. Due to the cautious gain value, the user-side robot and the environment-side robot did not follow each other exactly, and when the environment side-robot end-effector was pushing against something hard, the participant would sometimes apply more force and move the user-side robot further, thus creating a rigidity distortion (i.e., at the user-side the tool could seem spongy, rather than rigid). This could explain why both adults and children had difficulty differentiating the sticks that were close in rigidity, resulting in less difference between responses for tools for sugar or gravel. Children were observed to use more force than the adults, so as they pushed further at the user-side robot the spoons may have seemed bigger than they really were, making them think a big candy could be transported in them.

It follows that the rigidity distortion could also be why there were significantly more “yes” responses to spoons for the big candy in the Tech condition than the No Tech condition for both adults and children (Figure 3). The spoons could have seemed bigger than they really were because of the flexibility of the teleoperation system. Adults still sensed differences in the spoon size, as seen by the steady upward trend of “yes” responses for increasing size of spoons for adults (Figure 2), and a significant difference in responses for tools for the small candy than the big candy (Figure 3), so the distortion did not affect them as much as children. Children's data did not exhibit a steady trend, and there was no significant difference in responses between tools for the small and big candy. Unfortunately, it is not trivial to improve this factor in the robotic system. A higher gain in the control system would improve distortions, but then there is the possibility of instability in the system. In this system, robot control parameters were chosen to err on the side of safety.

Adults and children performed the expected EPs more often than the other EPs for both subtasks in both conditions i.e., Pressure for determining rigidity in the Mixing subtask, and Contour Following for determining size in the Transport subtask, as determined by Kalagher[9] and Klatzky, Lederman, & Mankinen.[8] Except for Enclosure and Function Test, which were not possible with the robot, the number of EPs performed in the Tech condition were greater than in the No Tech condition for both adults and children, some significantly higher. This could be due to participants needing to perform the EPs multiple times in order to extract the required information. By performing more EPs, individuals are able to obtain better information about object properties[9]; therefore, by repeating the EPs multiple times, participants may have been able to compensate for the limitations in attaining information that the robotic teleoperation system imposed.

Gibson[16] states that an object’s affordances determines how a person explores the object. It is possible that the affordances of the robot may have led to a difference in usage of the two additional EPs that were not described in the previous Klatzky, Lederman, & Manikinen[8] or Kalagher.[9] Function Test, which was mostly performed by adults during the Mixing subtask, was not actually possible with the robot system. Though participants could not “hold” the stick and “stir”, one adult performed a circular motion of the user-side effector, possibly visualizing how the presented tool could perform the mixing subtask. Function Test was never observed in the absence of Pressure, indicating that although it may be used to extract information about an objects’ function, it was not a useful EP to determine a tools’ rigidity. The Tapping EP was observed only in the Tech condition during the Transport subtask. In the Tech condition, participants were not able to execute the Enclosure EP, but Tapping seemed to allow participants to acquire the relevant information about the size of the circular spoons instead. Although Tapping replaced Enclosure, it was still not used as frequently as Contour Following, the expected EP.

As in Kalagher,[9] the 5-year-old children demonstrated adult-like EPs in manual exploration in the two properties that were tested (rigidity and size). In this study, the EPs they performed in the Tech condition were consistent with the adult's EPs.

Study 2 – Adult with cerebral palsy

Study 1 demonstrated that a haptic robotic system could be used to perform haptic exploration, however the results with adults and children without disabilities cannot necessarily generalize to persons with physical impairment. Study 2 was performed to explore how the system could be used by a participant with physical disabilities.

Methods

An exploratory case study was conducted to examine telerobotic haptic exploration by an individual with physical disabilities. The case was a 40-year-old woman with cerebral palsy categorized as spastic quadriplegia. She is right handed with limited range of motion in her upper limbs, and she was classified as MACS IV, meaning that she can manipulate a limited selection of objects but requires continuous assistance and adapted equipment.[17] She has no sensory or cognitive impairments, and no experience using teleoperated robotic systems, as self-reported. She uses a speech-generating communication device, but was able to give verbal “yes” or “no” responses.

Materials and set up were the same as in Study 1, with some modifications to accommodate the participant's abilities. Because she was unable to reach out and grasp the tools in the box, it was not used in the No Tech condition. Instead the participant was first shown the target objects (i.e., sugar or gravel, or small or big candy) and then was blindfolded while she explored the tools. She was able to hold on to the tools once they were placed in her hand. For the Tech condition, the user-side robot was placed as close to her right side as possible and she used a lateral grasp of the robot end effector between her ring and middle finger. A rubber band was placed on the distal part of her fingers in order for her to more easily hold on to the robot’s end effector. She had sufficient range of motion to move the robot in the required workspace (approximately 12 cm x 12 cm).

The procedure was the same as in Study 1, starting with practice using the teleoperated robot system before performing the tasks. She was also given the opportunity to stir the gravel and the sugar using her finger in order to acquire information about the demands of those target objects in the Mixing subtask. This was done because her mother reported that the participant had probably never felt them before. The participant then performed the Mixing and Transport subtasks first in the Tech condition and then in the No Tech condition with breaks as requested. Data was collected as in Study 1, but no statistics were calculated.

Figure 4 presents the participant’s responses for each tool when she performed the task with each target object in the Mixing and Transport Subtasks.

Figure 4.

Frequency of “yes” responses in the mixing and transport subtasks for the adult with disabilities. R1, R2, R3, R4, and R5, are the sticks from least to most rigid and S1, S2, S3, S4, and S5, are the spoons, from smallest to largest size.

Table 2 shows the participant's EPs in the No Tech and Tech Conditions for each subtask. For the Mixing subtask, she performed the Pressure EP most often in both conditions, but needed physical assistance in the No Tech Condition (i.e., the researcher holding one end of the stick). In the Transport subtask, the participant performed mostly the Static Contact EP in the No Tech condition. She did this by placing her closed fist over the spoon's cup or holding on to the side of the spoon’s cup after it was placed in her hand. She also performed Static Contact when she placed one finger in the spoon. Two times she rubbed her finger along a limited distance on the side of the spoon, but it was coded as Lateral Motion, since the distance was not sufficient to be coded as Contour Following. Enclosure was performed when the participant was able to hold the entire cup-end of the spoon in her hand. In the Tech conditions she mostly performed Tapping, and some Static Contact.

Table 2.

The EPs performed by the participant with disabilities in the No Tech and Tech Conditions for each subtask.

	Mixing		Transport
	No Tech	Tech	No Tech	Tech
Lateral motion	0	3	2	1
Pressure	10	11	0	0
Static contact	0	0	9	5
Enclosure	0	0	1	0
Contour following	0	0	0	1
Function test	0	0	0	0
Tapping	0	0	0	9

Study 2 examined if a participant with disabilities would respond the same and use similar EPs when she used a robotic teleoperation system compared to when she used her hands to explore tools in the subtasks. To the best of our knowledge, there are no previous studies exploring the experience of people with disabilities in these function judgement tasks when they used a haptic robotic system, therefore results were compared to those of adults without disabilities in Study 1. The participant's responses were closer to those we would expect when she used the teleoperation system compared to when she used her hands for exploration in these tasks. Since she performed the tasks first in the Tech condition, there was no learning effect. The total number of “yes” responses in Figure 4 for the No Tech condition seemed to be as expected, with more “yes” responses for sticks that could stir the sugar than the gravel (i.e., 3 > 2) and more spoons that could transport the small candy than the big one (4 > 2). However, when looking at her responses for each tool, there was no consistent pattern of “yes” responses increasing as tools became more rigid or larger. Therefore, it was not possible to confirm that the participant was able to acquire knowledge about the perceptual properties of the tools when she performed haptic exploration in the No Tech condition. In the Tech condition, there was a consistent trend of “yes” responses increasing as tools became more rigid or larger, and the total “yes” responses was greater for the small candy than the big one (i.e., 4 > 2), however, the “yes” responses were equal for gravel and sugar (1 = 1). When performing pressure in both the No Tech and the Tech conditions, it appears it was difficult to tell the difference in rigidity in the sticks for any but the hardest one.

The difficulty distinguishing the difference between tools for the gravel and sugar in the Mixing subtask could be attributed to the participant having limited experience with haptic exploration and the objects. The additional step to explore the sugar and gravel with her hands before making judgments about the tools was needed because the participant indicated that she had never felt them before. When children have physical disabilities their play oftentimes is reduced and therefore, they miss out on opportunities for exploration.[1] The participant may have had limited haptic exploration in her childhood, so she may not have been able to understand the requirements of the sugar and gravel to know how rigid sticks needed to be to stir them. In contrast, the information about the size of the candies could be determined through visual exploration.[8] Although possibly useful, it was not imperative for the participant with disabilities to explore the target candies beforehand using her hands. In fact, it was not possible for her to perform haptic exploration on the large candy target object because it required a large spherical grip that was difficult for her to do. She surprisingly answered “no” for the biggest spoon being able to transport the big candy, but this was likely due to difficulty performing EPs to attain the required size information.

Because affordances of objects influence the way they are explored by the hands[16] it is possible that the affordances of the robot may have also led to the EPs used by the participant to determine tool properties. In the Mixing subtask, she performed only Pressure in the No Tech condition, but both Pressure and Lateral Motion in the Tech condition. In the Transport subtask, she mostly performed Static Contact with her hand, not the expected optimum EP of Contour Following. By contacting one side of the spoon, it may have been possible for her to acquire information about size by feeling the spoon’s curvature on that specific spot. In the Tech condition, she did do Contour Following, and also Tapping. It is possible that the movement required for Tapping was easier for her to perform than that for Contour Following. Tapping only required her to move back and forth between two points on the spoon, unlike Contour Following, which required more complex movements of the shoulder-hand system. The participant could perform gross motor movements in her shoulder and elbow, but fine motor movements with her fingers were difficult. Thus, she could move the robot end effector back and forth and side to side. When new actions become available, it is possible to learn about object’s properties through haptic exploration.[16] The robotic teleoperation system allowed new actions to become available to the participant (i.e., Tapping EP to determine size), which could explain why she appropriately answered “yes” to only the larger spoons for the big candy.

In general, the participant required more assistance when she performed the task in the No Tech condition than in the Tech condition. When she used her hands for exploration, the participant required physical assistance from the researcher in order to grasp the tool and explore it. With the robot system, after the end effector was placed between her fingers, the participant was able to explore more independently by initiating and ending exploration of the tools. The participant's unique grasp and small range of motion could have influenced the types of EPs that she used. Other individuals with different abilities may employ different EPs. Physical disabilities, including cerebral palsy, encompass a wide array of abilities and impairments that would require this technology to be adapted for various ways to position the effector, grasp it and scale the range of motion of the environment-side robot up or down, depending on requirements. Likewise, tactile impairments would need to be considered to determine the viability of using a haptic robotic teleoperation system. In this study the participant had no known sensory impairment, but additional modifications in the system would be needed to amplify or reduce sensations to address the needs of individuals with impairments in sensation. The design of haptic robots to provide compensation to accomplish functional tasks is not as frequently studied as haptic robots for assistance or resistance for exercise therapy, but one study where an individual with cerebral palsy performed an object sorting task determined that the system needed to consider the individual's preferred trajectory pattern,[18] which may interfere with EP patterns.

Overall discussion and conclusions

In Study 1 typically developing children and adults without physical disabilities used a haptic robotic system to perform haptic exploration in a Function Judgement Task, and their responses were compared to when they did the tasks with manual exploration. The results were as expected for manual exploration: the knowledge acquired through manual haptic exploration about the perceptual properties of the tools for Mixing and Transport influenced their judgment and their responses were sensitive to the constraints on each tool’s function. However, the robotic system posed some challenges. Adults were able to compensate for the robot system's rigidity distortions, and detect constraints in tools' function, and obtain the expected results for the Transport subtask, but children had trouble with both subtasks. More work is needed to ensure the positions and forces of the teleoperated robot better represent haptic experience, yet are still safe.

Study 2 was an exploratory study where an adult with physical disabilities used a haptic robotic system and her hands to perform the Function Judgement Task. The improved responses when using the teleoperation system compared to when she used her hands could be due to being able to better perform EPs to extract the information she needed to judge the tools based on their rigidity and size. The Tapping EP that all participants performed when exploring with the robot was effective in determining size. Finally, the adult with disabilities required less assistance to be able to explore the tools, so the use of the robotic system may be a feasible way for individuals with physical disabilities to perform haptic exploration in play and functional tasks, but this requires further investigation.

While participating in the study it was revealed that the adult with disabilities missed out on childhood opportunities to perform haptic exploration, such as manipulation of gravel or sugar. In Study 1 there was no significant effect for the order in which participants without disabilities performed the Tech and No Tech conditions. It is possible that the practice phase, where they explored the tools using the haptic robotic system, was not needed. We expected that the experience would support recognition of how objects feel when using the robot, but they had the ability and previous experience of haptic exploration of various objects through their development and could perhaps better correlate what they felt through the system to previous haptic experiences. The participant with cerebral palsy in Study 2 had no previous experience of haptic exploration of the presented objects, either with her hands or through the system, and so had no previous experiences to draw upon in relation to object properties or correlating the sensations felt in both conditions. It is possible that if children with physical disabilities are given opportunities to experience haptic feedback through a robotic teleoperation system during play activities, it could provide a means, in addition to visual and manual exploration, to perform EPs and practice perceiving object properties that are required to make judgements about tools and possibly contribute to their independence when participating in play.

The studies had some limitations yet to be mentioned in addition to the aforementioned rigidity distortion in the teleoperated robot system. Only one adult with disabilities was recruited and no children with disabilities, the eventual target population. With this sample, it is not possible to generalize about how children with disabilities will perform with a haptic robotic system. No assessment was performed with the adult with disabilities regarding her sensory abilities; therefore, it is not possible to know if the participant’s performance on the task was influenced by sensory impairment, or strictly by her motor limitations. In general, all participants needed to do more EPs with the robot before giving their response, which could be due to needing more exploration compared to when using the hands, but since this was the first time they used a haptic robotic system, it is possible that their performance in the tasks would have been different with more practice.

These findings are a first step towards the development of robotic teleoperation systems for haptic exploration for individuals with physical disabilities. Future studies reducing the rigidity distortion, as well as testing alternate end effectors so individuals can more easily “grasp” and move them could further guide the development of assistive robots. In addition, studies recruiting children with physical disabilities to explore robotic use for haptic exploration are imperative to understand how limited mobility while haptic exploration is still developing will influence performance using a haptic robotic system. Also further research is necessary with a larger sample size and a wider range of participants (i.e. different types of motor impairments and levels of functioning affecting their ability to manipulate objects) to understand how the haptic robotic teleoperation systems can influence haptic exploration in this population.