Data-driven body-machine interface for the accurate control of drones.

The accurate teleoperation of robotic devices requires simple, yet intuitive and reliable control interfaces. However, current human-machine interfaces (HMIs) often fail to fulfill these characteristics, leading to systems requiring an intensive practice to reach a sufficient operation expertise. Here, we present a systematic methodology to identify the spontaneous gesture-based interaction strategies of naive individuals with a distant device, and to exploit this information to develop a data-driven body-machine interface (BoMI) to efficiently control this device. We applied this approach to the specific case of drone steering and derived a simple control method relying on upper-body motion. The identified BoMI allowed participants with no prior experience to rapidly master the control of both simulated and real drones, outperforming joystick users, and comparing with the control ability reached by participants using the bird-like flight simulator Birdly.

Teleoperation, a subfield of human–machine interaction (HMI), describes the control at a distance of an actuated device (1). Typical applications include deployments in environments where it is not desirable or possible to send a human operator, such as nuclear plants (2, 3), scenes of natural hazards, or more generally in search and rescue missions (4–6). The use of teleoperated systems can augment human dexterity and precision, which are fundamental abilities in those and other fields of application, such as minimally invasive surgery (7) or microfabrication (8). Patients suffering from neurological disorders may benefit as well from teleoperated systems to substitute for lost body functions by controlling wheelchairs (9, 10), telepresence systems (11, 12), or robotic manipulators (13).

Successful teleoperation requires robust and reliable control interfaces. A well-defined interaction should be transparent (14, 15), rely on intuitive command inputs to ensure rapid proficiency and minimize the task-associated workload (16), and provide appropriate feedback (visual, auditory, haptic) to strengthen the awareness of the operator (17). A number of existing interfaces already allow interactions with robotic devices. However, simple third-party devices such as a joystick show limited performance even with systems with few degrees of freedom (DOFs). The development of intuitive commands becomes yet more challenging in “nonhomologous” interactions, that is, when the operators’ command behaviors significantly differ from the machine’s realizable behavior, or when their physical abilities are restricted.

A possible approach to address this issue comes from brain–computer interfaces (BCIs), which bypass behavioral output by directly retrieving the desired information from the cerebral activity patterns, often relying on mental imagery. Successful examples include the control of humanoids (18), unmanned aerial vehicles (UAVs) (19–21), wheelchairs, and telepresence systems for motion-impaired individuals (9–12). BCIs do nonetheless come with certain limitations, which may prevent their widespread utilization. Firstly, the noninvasive signal acquisition is associated with a low signal-to-noise ratio and thus a high sensitivity to perturbations. The use of these systems is therefore limited to relatively controlled environments and may not be suited to everyday activities. Another limitation of this approach comes from the execution of motor imagery tasks, which strongly constrains the user’s focus on the completion of the control task. The system is therefore prone to errors in case of unpredicted and undesired stimuli and a long-term operation is likely to be cognitively demanding.

Recent and promising developments suggest that body–machine interfaces (BoMIs) are a valuable alternative to BCIs for able-bodied or partially impaired persons. Instead of neural activity patterns, these systems retrieve information from body motion or from the underlying muscular activities (22). The broad spectrum of applications ranges from the control of assistive devices by neurological patients (23–25) to the control of UAVs (26–29). BoMIs present one unambiguous advantage over BCIs: they exploit the fine control the operators can have over their body, while operating a BCI requires to actively modulate the activity of designed cerebral areas, a task for which humans have no preexisting ability (24). BoMIs have therefore the potential to be more robust to external sensory perturbations and more straightforward to master.

Among the various types of mobile robots, drones have the potential to extend human perception in unprecedented ways for several civilian applications and disaster mitigation scenarios (30), but proficient teleoperation of drones with a standard joystick may require extensive training sessions. Therefore, the identification of more intuitive BoMIs between humans and drones could reduce training time, improve flight efficiency, and allow the operators to shift their focus from the control task to the evaluation of the information provided by the drone.

Over the last decade, gesture-based interfaces for the control of flying robots have gained increasing interest, including approaches using commands from the head (31, 32), the hand (26, 33, 34), the upper body (5, 27, 35–39), or even the entire body (40). Systems using head gestures involve a first-person view (FPV) visual feedback presented to the user through a head-mounted display (HMD) and are thus highly intuitive, yet this approach prevents the user from visually exploring the environment without affecting the flight trajectory. Relying on other body parts for the control decouples the steering from the visual exploration. However, aside from one exception (36) these implementations only offer a visual feedback in third-person view with the drone flying in the user’s field of view, thus restricting the level of immersion offered to the operator. Another important limitation comes from the implementation of discrete commands instead of a continuous mapping of the user’s movements to the drone’s actions. This characteristic also reduces the immersion experienced by the user, as modulations of the control input have no effect on the trajectory of the drone (41). Only one approach interfaced Microsoft’s Kinect with a quadcopter, with a direct translation of the user’s gestures into velocity commands for the drone (42). Lastly, the majority of the HMIs for the control of drones make use of preselected movements, that the participants have to reproduce. The definition of the control strategy may thus be biased by the designers’ preferences and fail to encounter patterns that are intuitive to potential operators. Recent works addressed this issue through Wizard-of-Oz sessions (35, 37, 39) or interviews (43) to develop more intuitive interfaces, yet still with a focus on discrete commands.

In this work, we present a structured methodology to identify intuitive communication strategies for nonhomologous continuous HMIs, and we detail this approach in the case of immersive gesture-based drone control. Briefly, we first recorded the upper-body kinematics and muscle activities during the generation of movements that would imitate the behavior of a flying drone. After identifying two main interaction strategies used by the participants, we assessed the capacity of potential users to actively steer the path of the virtual drone employing these two strategies. Eventually, we evaluated the transferability of the skills acquired during simulation training to the control of a real drone. provides an overview of the study.

Results

Experiment 1: Identification of Preferred Human Flight Strategies.

The participants wore an HMD through which they were shown simulated flight sequences in FPV and were asked to follow the actions of the aircraft executing five distinct behaviors (constant forward motion, right-banked turn, left-banked turn, upward pitch, and downward pitch; ) with self-selected, flight-like movements. Upper-body kinematics and electromyographic (EMG) activities of the subjects were recorded during the execution of the task (Fig. 1 and ) as these are often used as sources of information for BoMIs.

Fig. 1.

Experimental setups. (A) Identification of the preferred movement strategies. (B) Gesture-based control of a virtual drone using the Torso and Torso and Arms strategies. (C) Gesture-based control of a real drone using the Torso strategy.

Simulated human flight gives rise to two major distinct movement strategies.

Out of the 17 participants who completed the experiment, 15 selected similar strategies which resembled gliding flight, and symmetrically involved both limbs (Fig. 2). The two outliers, which were excluded from subsequent analyses, performed very distinctive movements: one participant employed solely the arms, pointing toward the direction of change, and the second one impersonated a form of active, powered flight (Fig. 2 and Movie S3).

Fig. 2.

Representative traces of the muscle activities (Top: rhomboid major, Middle: deltoid anterior, Bottom: triceps brachii lateral head) and shoulder abduction angles during the execution of the open-loop task. (A) Participant using only the torso. (B) Participant actively using both the torso and the upper limbs. (C) Participant categorized as outlier. (D) Participant clustering according to the selected movement strategies.

We categorized the participants according to their upper-limb usage, with a k-means classification on the norm of the root-mean-square angular velocities of the different joints and found that 10 subjects used predominantly their torso but did not actively involve their arms (i.e., their arms were either rested on their thighs or extended sideward like wings; see Fig. 2 and Movie S1), while five subjects combined torso and arm movements (Fig. 2 and Movie S2). To account for these different patterns, we further analyzed both groups separately (hereafter “Torso” and “Torso and Arms” groups).

Muscle activity patterns confirm the presence of two movement strategies.

We assessed the activity of major upper-body muscles groups during the imitation task and compared their contribution to the overall data variability across subgroups using nonnegative sparse principal component analysis () (44, 45).

The Torso group displayed a sparse pattern, with six pairs of muscles located on the superior back (sternocleidomastoid, trapezius superior, latissimus dorsi) and on the upper arm (deltoid anterior, deltoid medialis, triceps brachii; Fig. 3 and ) found to significantly contribute to the overall variability. The Torso and Arms group in turn showed a more uniform pattern, with 10 pairs of muscles out of 16 considered as carrying the relevant variability. All of the contributors for the Torso group were also selected here. The additional muscles were located on the forearm (extensor digitorum communis) and in the upper trunk (infraspinatus, rhomboid major, pectoralis major; Fig. 3 and ). These results confirm the existence of two distinct motion strategies, one involving only the torso, the other one including both torso and arm movements.

Fig. 3.

Cumulated factor loadings showing the contribution of each muscle to the overall variance in the EMG dataset. (A) Participants using only their torso. (B) Participants using their torso and their arms. The retained muscles (indicated by the black stars) are STRCM, TRAP, LAT, DANT, DMED, and TRI for the Torso group; STRCM, TRAP, INFRA, LAT, RHO, PEC, DANT, DMED, TRI, and EXDC for the Torso and Arms group. The bar graphs represent the means + SEM over four (Torso group), respectively, five subjects (Torso and Arms group).

Kinematic variables show uniform levels of discriminant information.

We evaluated and compared the amount of discriminant information provided by all considered upper-body segments, that is the torso, both upper arms, and both forearms, as defined by the 3D position of their center of mass (COM) (46), as well as the absolute orientation angles for the torso and the shoulder and elbow angles (47, 48). We used the Reliable Independent Component Analysis (RELICA; ) (49, 50) to parse the multivariate dataset into independent components, and to identify the variables carrying the relevant information.

The averaged segment scores for the Torso group show that the information is uniformly distributed across all variables. In particular, we found no significant difference between the amount of information held by the torso COM and torso angles (ITorsoCOM = 9.25 ± 1.00, ITorsoAngles = 10.16 ± 0.65, P = 0.098), which indicates that the positional and angular variables are of equal interest for decoding the movements of this strategy (Fig. 4). Similarly, the level of information was nearly uniformly distributed across the individual segments for the Torso and Arms group. We also assessed the difference between the cumulated informativeness carried by all COMs and all joint angles. Once more, we found that the two subsets of variables held equivalent levels of information (IallCOMs = 51.93 ± 1.33, IallAngles = 48.07 ± 1.98, P = 0.062, Fig. 4).

Fig. 4.

Information levels held by the upper-body segments in terms of three-dimensional position or joint angles. (A and B) Participants using only the torso. The three-dimensional displacement and the angles of the torso appear to hold equivalent levels of information. (C and D) Participants using the torso and the arms. The informativeness is equally shared between the positional and angular data of all segments and joints. The bar graphs represent the means + SD over 10 (Torso group) and 5 (Torso and Arms group) subjects.

The discriminant information thus appeared to be equally distributed across all kinematic variables. In view of future applications, we decided to restrict the subsequent steps of our analysis to the angular data, which can be more robustly extracted with wearable sensors (51).

Selected kinematic variables lead to higher decoding performances than selected muscles.

Next, we assessed the decoding power held by the full sets of kinematic variables and the muscular activities, and by the reduced (selected) sets of both types of signals as identified in the previous section. For each subject, we implemented linear discriminant analysis (LDA) classifiers employing one set of variables with leave-one-out cross-validation. Eventually, we performed a generalized classification, using the data of all-but-one subjects to build the classifier, which was then tested on the data of the remaining subjects (Fig. 5 and ).

Fig. 5.

Classification accuracy as percentage of correctly classified samples. (A–D) Subject by subject and (E) generalized classification for the Torso group. (F–I) Subject by subject and (J) generalized classification for the Torso and Arms group. The selected EMG were STRCM, TRAP, LAT, DANT, DMED, and TRI for the Torso group and STRCM, TRAP, INFRA, LAT, RHO, PEC, DANT, DMED, TRI, and EXDC for the Torso and Arms group. The selected kinematic variables consisted of the (absolute) torso rotation angles for the Torso group and the torso, and bilateral shoulder and elbow angles for the Torso and Arms group. The bar graphs represent the means + SD over four (Torso group) and five (Torso and Arms group) subjects.

The EMG-based classification for the Torso group yielded low accuracies, with similar results for the entire dataset (ATorso_allEMG = 37.65 ± 26.42%), and the selected variables (ATorso_selectedEMG = 34.87 ± 27.29%, P = 0.6). In particular, the “Forward” and “Up” commands were poorly recognized, with scores in the range of chance level. In contrast, the data of the Torso and Arms group led to satisfying performances, again with similar results for the entire dataset (ATorsoArms_allEMG = 76.11 ± 6.23%), and the selected variables (ATorsoArms_selectedEMG = 72.15 ± 10.70%, P = 0.77); with all movements equivalently well decoded.

The kinematics-based classification yielded outcomes comparable with the EMG-based decoders. The accuracy obtained for the Torso group was similar when using the full dataset (ATorso_allKin = 55.90 ± 8.51%) and the selected variables (ATorso_selectedKin = 60.13 ± 17.16%, P = 0.93). Likewise, the decoding power for the Torso and Arms group was in the same range for the full dataset (ATorsoArms_allKin = 76.13 ± 14.43%) and for the selected variables (ATorsoArms_selectedKin = 75.08 ± 16.11%, P = 0.12). The generalized classification led to lower, yet not significantly different accuracies, for the Torso group (ATorso_gen = 53.02 ± 14.88%, P = 0.175) and Torso and Arms group (ATorsoArms_gen = 40.69 ± 8.78%, P = 0.021, not significant at the corrected Bonferroni level).

These results confirm the good decoding power of the selected datasets as we observed only minor changes in the decoding performance when the full sets of variables were reduced to the selected subsets, indicating that the retained factors carried the discriminant information. In general, we obtained higher decoding accuracies for the Torso and Arms group than for the Torso group. This reflects the higher intermovement variability displayed by the Torso and Arms group, due to the higher number of degrees of freedom.

Overall, the selected kinematic variables, i.e., the joint angles, yielded the best decoding ability. Therefore, we used the joint angles (torso angles for the Torso group; torso, shoulder, and elbow angles for the Torso and Arms group) as inputs for a closed-loop implementation.

Experiment 2: BoMI Control of a Simulated Drone.

The subjects were shown a virtual environment displaying a series of waypoints to follow (Fig. 1) through a HMD. The first 9 min of virtual flight were considered as a training period. The final performance was evaluated at the end of the training, on an additional series of waypoints (Movie S4; see also ). We compared the outcomes of the gesture-based control to a previous study in which the participants used a standard joystick or Birdly, an immersive platform simulating a bird’s flight (52). Additionally, we compared the evolution of the steering performance between the gesture-based controls and the joystick over three training sessions on consecutive days on a subset of those subjects.

Participants steering the aircraft using only their torso outperformed those using their torso and arms.

Over a single practice session, all participants displayed a continuous performance improvement. The final performance, evaluated at the end of the session, was significantly higher for the group using only the torso (PerfTorso = 84.58 ± 17.79%) than for the group using the Torso and Arms strategy (PerfTorsoArms = 62.59 ± 25.88%, P = 0.004) or the joystick (PerfJoystick = 59.42 ± 31.35%, P = 0.029). The performance of the Torso group was however comparable to the results obtained by the participants using Birdly (PerfBirdly = 93.01 ± 5.87, P = 0.43; Fig. 6).

Fig. 6.

Control of the simulated drone. (A) Performance evolution during the training phase and final evaluation after one session. The dotted lines represent the performance averaged across participants, the full lines the modeled learning curves, and the diamonds indicate the mean performance (*P < 0.05, **P < 0.01). (B) Final evaluation on three training sessions on consecutive days. The diamonds indicate the mean performance.

Three-day training leads to a homogeneous performance for the torso strategy.

After three training sessions, the participants using only their torso showed lower variability in their final performances than those using the Torso and Arms strategy (F5,5 = 10.96, P = 0.02) and the joystick (F6,5 = 19.88, P = 0.005). This was observed even if, with this reduced number of participants, there were no differences in terms of performances across the three strategies (P = 0.25).

Experiment 3: BoMI Control of a Real Drone.

We evaluated the transferability of the skills acquired during the virtual reality (VR) training to the control of an actual drone using the Torso strategy, as this approach proved to be superior for the control of a virtual aircraft (). The participants began with a 9-min VR training as described previously. Afterward, they were given the control of a real quadcopter with FPV video feedback, which they could freely fly for 2 min to get used to its dynamics. Eventually, they were asked to steer the drone through six gates arranged along an eight-shaped trajectory (Fig. 1; see also ).

After completing the VR training and the free flight, the participants were able to steer the quadcopter along the defined path with an average percentage of validly crossed gates (PVC, 19) of 87.67 ± 9.88% (Movie S5). This result suggests a transfer of the control skills acquired in simulation and confirms the usability of the Torso strategy for the steering of a real drone.

Discussion

We proposed a systematic selection process to identify effective body movement patterns in nonhomologous HMIs and to reduce the sensor coverage necessary for the acquisition of the discriminant information. We applied the described method to the specific case of flight and derived a simple BoMI interface for drone control. We found that, despite the noninnate nature of flying, two common motives emerged during the spontaneous selection of congruent movements during a virtual imitation task. These two major patterns proved to be valid command inputs for the control of a virtual drone, with the simpler strategy involving only movements of the torso leading to higher performances than the strategy employing both the torso and the arms. Eventually, we demonstrated that a real quadcopter could be controlled with the first, simpler strategy.

When using only their torso to steer the trajectory of the simulated drone in the virtual environment, inexperienced participants needed less than 7 min of practice to reach a performance of 84.58% (Fig. 6). By comparison, users performing the same task with a joystick typically used for piloting drones only reached an average score of 59.42% (Fig. 6). Furthermore, the performance level obtained using the identified BoMI is comparable to the performance of subjects using the bird-flight simulator Birdly to steer the virtual drone (52) (Fig. 5). However, the participants using this platform displayed higher initial performance and a steeper improvement. But the Birdly platform provides haptic and vestibular feedbacks in addition to the visual information used in this study, factors known to improve the execution of teleoperated tasks (51). The lying position imposed by Birdly may also have affected the rapid proficiency, since this platform allows the entire body to move as a whole, and this posture may be more closely associated with the idea of flying. Nonetheless, the comparable final steering performance suggests that the identification of intuitive BoMIs can compensate to a certain extent the absence of additional sources of feedback, while requiring only minimal recording apparatus. Moreover, the Torso control method led to 87.7% of gates crossed without collisions during the steering of a real drone along a complex trajectory following a 9-min training in simulation.

In a single session, the two implemented gestural strategies led to significantly different performance levels, with the participants using the Torso strategy outperforming those using the Torso and Arms approach (Fig. 6). This difference was expected, since the Torso and Arms strategy was derived from the movement patterns displayed by 5 out of 15 participants of Experiment 1, being therefore less representative of the population. Additionally, while the Torso strategy mapped three body DOFs (torso rotations) to two drone DOFs, the participants using their torso and arms had to correctly coordinate 13 DOFs to control the two rotations of the aircraft. Such an approach may however be of interest in the perspective of an extension of this work including additional commands or behaviors.

All of the subjects who trained for three consecutive days improved their performance, confirming the importance of practice. However, the intragroup variability significantly differed across the control methods after the third training session, as the steering ability displayed by some participants using either the joystick or the Torso and Arms strategy remained low (Fig. 6). Instead, all of the subjects using the Torso strategy displayed a final performance above 77% and the overall performance variability significantly decreased over time. Therefore, the Torso strategy was the only approach which all participants managed to master following the 3-d training, suggesting that this method may be suited to a broader range of users.

Surveying the spontaneous interaction strategies selected by nontrained users is a concept that has already been applied for the development of intuitive controllers for UAVs, either by means of interviews (43) or through Wizard-of-Oz experimentations (37, 39). However, these systems focus on the identification of discrete commands and have the user interact with the drone from an external perspective. Conversely, our work presents a case of a data-driven, gesture-based interface for the continuous and immersive control of drones using an immersive visual feedback. Our present approach could easily be translated into a wearable implementation using an inertial measurement unit to acquire the three-dimensional torso angles. This would provide a substantial benefit over HMIs using video-based motion tracking, which imposes constraints on lighting conditions in the operating environment and on the users’ freedom of displacement, and thus limit the applicability of such a controller in natural environments.

A possible limitation of this study could be found in the mapping (scaling and offset constants) used to translate upper-body movements into commands of the simulated and real drones. The chosen mapping has shown to be sufficiently sensitive to steer the drone along the relatively smooth waypoint paths used in the experiments described here. However, we cannot exclude that sinuous trajectories involving sharp changes of directions may require different or even adaptive mapping values. Indeed, it is known that humans make directional errors when relying only on proprioception to estimate the spatial location of their limbs, and that these errors are proportional to the distance to the body centerline (53, 54). Building on this knowledge, previous studies showed that nonlinear transformations of the users’ arm movements led to faster and more precise control of a robotic arm than a simple scaling (14, 55). Further studies will be needed to understand the role of more complex mappings to extend the results of this work.

Another limitation comes from the small diversity of our study population, which consisted mainly of young, male university students. It is unknown to which extent experience and observation shape the human representation of noninnate behaviors such as flight. We can therefore not exclude that factors such as age, gender, physical condition, or familiarity with technology could lead to the identification of different body motion patterns. However, such discrepancies may highlight interesting causes in motor learning and representation rather than invalidating the proposed identification method.

Conclusion

The results of this study have a significant importance for the field of teleoperation and more generally HMIs. Often, control strategies are predefined and selected to comply with existing interfaces rather than derived from spontaneous representations of the interaction. The implementation of a methodology to identify body–machine patterns for specific applications could lead to the development of more intuitive and effective interfaces, which could in turn reduce the training time required to reach proficiency, limit the workload associated with the operation of the system, and eventually improve the reliability of teleoperated missions. Moreover, the method described in this article could be extended to different populations, machines, and operations, including individuals with limited or impaired body functions.

Methods

Seventeen subjects participated in Experiment 1, in which they were asked to produce self-selected upper-body movements corresponding to predefined drone commands. For Experiment 2, 44 new participants were asked to steer a virtual drone using the Torso or the Torso and Arms strategy during a single session. Sixteen randomly selected participants repeated this task on two additional, consecutive days. Ten new participants were recruited for Experiment 3. They were first asked to control a virtual drone using the Torso strategy and afterward to steer a real drone through circular gates (see for details). The experiments were approved by the École Polytechnique Fédérale de Lausanne Brain Mind Institute Ethics Committee for Human Behavioral Research and the Ethics Committee Geneva.