The Optimal Adaptive-Based Neurofuzzy Control of the 3-DOF Musculoskeletal System of Human Arm in a 2D Plane.

Each individual performs different daily activities such as reaching and lifting with his hand that shows the important role of robots designed to estimate the position of the objects or the muscle forces. Understanding the body's musculoskeletal system's learning control mechanism can lead us to develop a robust control technique that can be applied to rehabilitation robotics. The musculoskeletal model of the human arm used in this study is a 3-link robot coupled with 6 muscles which a neurofuzzy controller of TSK type along multicritic agents is used for training and learning fuzzy rules. The adaptive critic agents based on reinforcement learning oversees the controller's parameters and avoids overtraining. The simulation results show that in both states of with/without optimization, the controller can well track the desired trajectory smoothly and with acceptable accuracy. The magnitude of forces in the optimized model is significantly lower, implying the controller's correct operation. Also, links take the same trajectory with a lower overall displacement than that of the nonoptimized mode, which is consistent with the hand's natural motion, seeking the most optimum trajectory.

1. Introduction

In many countries, population aging leads to a decrease in productivity of useful work, and this will cause serious problems. Many robots are designed and employed for self-rehabilitation of elderly, disabled, damaged people in daily activities [1-11]. The hand is one part of the body that is frequently involved and employed in most individuals' daily activities. Each individual performs different daily activities such as reaching and lifting with his hand that shows the important role of robots designed in this field to estimate the position of the forces exerted by the hand. There is a growing trend worldwide for the application of handling machines, inspired by human arms, in all industrial sectors, to carry materials from one destination to the other under limited operating conditions. Advances in manipulators are manifested both in their high technical level and growing economy and safety [12]. In the robotic human arm, two links are usually used as the arm and forearm segments with two-degree-of-freedom (DOF), and at least four muscle elements are used for moving it in the 2D space. The inverse dynamic model is applied to generate joint torques in this robot [13]. The motion or force predetermined and designed by powerful controllers is used in rehabilitation applications. The training is an important factor for controlling the arm to achieve a static goal, and the body's musculoskeletal system gradually gains this capability through interaction with the surrounding environment. For example, a soccer player performs a series of random activities to deliver the ball to the gate, but the more professional he becomes in this field, the faster and more efficient he hits the ball [14, 15]. This is achieved by the gradual training of the muscles' kinematic, and the related information can be saved and used in the future [16]. Therefore, understanding the training mechanism of the musculoskeletal system of the body can lead us to employ a powerful controller for body rehabilitation robotics. Many researchers used the training controls, which will gradually train the arm controller [17-21]. Golkhou et al. [22] employed an improved Actor-Critic algorithm for the controller of a single-link musculoskeletal arm with two extensor and flexor muscles during vibrational motion. A CMAC controller was applied to the Critic section to estimate the optimal activities and update the Actor section's coefficients. Zacharie et al. [23] applied an advanced logic-based neural network to a robotic hand. The logical function was determined based on the endpoint of the arm's arbitrary trajectory in space to compute the possible conditions of the neuron's activity to respond to the desired field. Bouganis and Shanahan [24] presented a neural network that could automatically learn to control a robotic hand with 5 degrees of freedom and the motor's initial time conditions. Kambara et al. [25] proposed a control model for motion training based on the inverse static model, direct dynamic model, and feedback control combined with Actor-Critic. Their model supported the trajectory prediction of a 2-DOF arm with six artificial muscles. Thomas et al. [26] applied an improved learning controller based on a proportional derivative control technique (PD) to control a robotic hand with four muscles for conducting the Reaching activity. Dong et al. [27, 28] implemented an adaptive sliding mode control strategy on a 2-DOF robotic hand with biarticular muscles so that the dynamic parameters were updated, which caused the input disturbances and stimulations of the system to be considered. Zadravec et al. [29] implemented an optimal controller, whose cost function was to minimize the joint torques, on a 2-DOF robotic hand. In this study, the authors could predict optimum trajectories along with the functional constraints of the muscles.

This model requires accurate dynamic parameters; however, accurately determine these parameters for different people is impractical. According to the literature review above, adaptability and optimality are the basic characteristics of the human brain, and the lack of a powerful controller that can implement the control strategy of the brain to some extent is very noticeable. In the present study, first, the equations governing the 3-link human arm's motion and the related dynamic equations are expressed in Section 2. An adaptive neurofuzzy controller is presented in the next section. The results obtained from the simulation of controllers with/without optimization are presented in Section 4. Finally, the concluded remarks of this study are described in Section 5.

2. The 3-DOF Human Arm Musculoskeletal Model

The multibody planar model of the human arm with 3-DOF is presented in Figure 1, in which the upper arm, forearms, and hand are considered three rigid links. This model considers the planar motion around three revolute joints at the shoulder, elbow, and wrist and neglects the gravitational effects. As shown in Figure 1, this model consists of six muscles that can only apply tensile forces so that each joint rotates by some of these related muscles. Muscles are assumed to be without weight and designed based on the Hill model, which are directly connected to links as [30]: f denotes the output force of ith muscle,  f0 is the maximum contractile muscle force, α expresses the activation level of controlled muscle, and is the contractile muscle velocity. b, c is also the muscle damping coefficients and stiffness, respectively. Considering the number of six muscles, the matrix form of Eq. (1) is

Figure 1

Schematic view of the 3-DOF musculoskeletal model.

The following equation expresses the relation between the position vector of the end effector of the arm and joint angles: where L1, L2, and L3 represent the first, second, and third links, respectively. θ1, θ2, and θ3 are also the relevant link's angle to the x-axis, the second and third link. The velocity at the end effector of the arm, which is dependent on angular velocities, are expressed as follows:

J ∈ R2×3 is the Jacobian matrix that shows the relation between the arm's end effector's linear velocities and angular velocities. The length vectors of the muscles are defined as and where r1−6 and s1−6 represent the torque surfaces, as shown in Figure 1. The following equation is obtained by taking the time derivative of the above equation to time:

W ∈ R6×3 is the Jacobian matrix, which relates the muscles' contractile rate to the joints' angular velocity, and represents the stretch rate of muscles. By applying the principle of virtual work, the work done by muscle torque is defined as follows:

where is the vector represents the tensile forces of muscles and is the joint torque vector. As depicted in Table 1, by putting muscle parameters in Eq. (6), W is defined as follows:

Table 1

Numerical values related to the muscle's geometrical shape.

Muscle	Value (m)
l 1	r 1 = 0.055	s 1 = 0.080
l 2	r 2 = 0.055	s 2 = 0.080
l 3	r 3 = 0.030	s 3 = 0.120
l 4	r 4 = 0.030	s 4 = 0.120
l 5	r 5 = 0.035	s 5 = 0.220
l 6	r 6 = 0.040	s 6 = 0.250

Using Lagrange's equations [29]

H is a symmetric matrix representing the mass momentum, and C is a skew-symmetric matrix of Coriolis, centrifugal, and friction torques. By substituting Eq. (8) into the above equation, the dynamic equations of the musculoskeletal system are obtained as

3. Controller Design

The controller design's main purpose is to use appropriate motion commands for each muscle in the process of interacting with the environment and learning the kinematics of the arm in the movement toward a fixed target. Neurofuzzy systems are a combination of neural networks with fuzzy logic systems and utilized to simplify problems and apply the subjective, complex rules and concepts. To mimic the human brain's function in these systems, which consists of a set of artificial neurons, an artificial neural network is used with fuzzy logic rules. Ghanooni et al. [31] found that the adaptive multicritic neurofuzzy control framework can help identify the unknown systems and suggested that the computational load required for this controller's parameters compatibility is lower than the conventional neurofuzzy controllers, and this is one of the advantages of this controller in real-time applications. They also claimed that their controller would benefit from the reinforcement learning compared to supervisory learning in the online evaluation of the output, which led to the capability of controlling any uncertainty in the system.

A new structure of adaptive neurofuzzy control framework composed of several inputs and outputs based on reinforcement learning was investigated by Balaghi et al. [32]. Their study aimed to control the motion trajectory by optimizing a 2-DOF model of the human arm's contractile muscle forces. The “critic estimates the system's achievement,” and the “actor” updates the controller parameters by generating the associated signal. They argued that the difficulty of determining the precise arm's biological specification values such as mass and inertia made them use this controller because it is independent of the model parameters. Moreover, this controller's generated inputs are optimum, which is significant in the musculoskeletal system due to the biological limitations of human muscle limitations. This controller is implemented for the existing 3-DOF model in this study because of the advantages mentioned above. The model's endpoint has to be directed on the arbitrary trajectory for all initial values in the X and Y direction by multiple muscle contractile forces. Hence, a multiple-input and multiple-output system (MIMO) consisting of muscle inputs and endpoint outputs should be considered.

3.1. Neurofuzzy Network

Fuzzy systems consist a fuzzification unit, a defuzzification unit, a fuzzy rule base, and an inference engine. The fuzzy system can be regarded as performing a real and nonlinear mapping from an input vector x ∈ R to an output vector y = f(x) ∈ R, where m and n are the dimensions of the input and output vectors, respectively. The bitwise interfaces of the real and fuzzy worlds are fuzzifier and defuzzifier, respectively. The earlier addresses real inputs to the associated fuzzy sets, and the latter serves to address the fuzzy sets of output variables to the associated real outputs in the reverse direction.

Two types of fuzzy systems, called Takagi-Sugeno-Kang (TSK) and systems with fuzzifiers and defuzzifiers (Mamdani), are more common in the literature, and the TSK type is used in this study for adaptive neurofuzzy control framework. The multi-input single-output (MISO) neurofuzzy system—including N rules—is defined as follows:

Rule: if (u1isA) and if (u2isA) and … and if (uisA)

then ify = G(u1, u2, ⋯, u)

where i is the rule number, u are the inputs with m number, A indicates the fuzzy set for inputs, and G which is the linear relation of inputs evaluated as a crisp function as

Consequently, the TSK neurofuzzy output can be expressed as

In Eq. (12), M is the number of rules, and μ is the membership function for the ith rule.

The inputs of the adaptive critic-based neurofuzzy controller applied to the endpoint of the human arm model in this study are e, , e, and as where (x, y) and (x, y)  are the desired and real output of the system in the 2-D workspace, respectively. and are also the desired and real velocity of the arm's endpoint in the task space. As a consequence, the vector form of the TSK neurofuzzy output for M rules is calculated as

The fuzzy system in an adaptive neural network is a standard TSK system, which leads to the formation of a four-layered network. In the first layer, all inputs are directed into the [-1, 1] scope of the membership function. Based on Figure 2, three membership functions were determined for each input and labeled using N, Z, and P, representing the negative, zero, and positive expression, respectively. Also, the fuzzification and defuzzification process is performing in the second and fourth layers, respectively. The third layer performs decision-making with Max-Product law. Therefore, there are 81 rules for each controller of the TSK system.

Figure 2

The input membership function of the TSK fuzzy system.

3.2. Adaptive Critic

The critic agent is the main part of any learning system. Each critic agent examines a system's state by evaluating its output and generates a critic signal called r. The signal r is a real number in the range of [-1, 1] and is implemented by the learning process to train and adjust the TSK fuzzy system's parameters to minimize the signal to reach zero value indicates that the system does not require more training. In multicritic systems, the evaluation of a system's performance is carried out by each agent separately. Accordingly, all critic signals should become zero, which indicates the critic is satisfied by the system's performance. Here, two cost functions are studied to gratified the critics by minimizing as [33]

where e and are position and velocity tracing error of the arm's endpoint, k and k are the critics' weight, which indicates the component preferences in the cost function, h1 and h2 scale variables bring the items in [-1, 1], and  α as mentioned before is the activation level of controlled muscle. In Eq. (16), the second term of the right-hand side is the TSK system's optimization, which minimizes the muscles' tensile forces. The reform of the above equation for the number of m muscles and s system's output is represented as where f is the contractile force of jth muscle, f is the max amount of f, k and k are the critic weights, and d is an arbitrary positive number. As stated, the aim of controlling the musculoskeletal system is that the arm's endpoint reaches the desired position simultaneously with minimizing the contractile muscle forces; thus, in Eq. (17), s = 2  and m = 6. The block diagram and critic rules of the controller are shown in Figure 3.

Figure 3

Controller block diagram and system critic rules.

3.3. Learning System

As previously described, the primary purpose of the learning mechanism is to minimize the error function's critic effects and satisfy all critic's criteria. In a learning system, updating neurofuzzy control parameters by critical signals is called emotional training. Therefore, emotional training aims to minimize the cost function E in Eq. (17). By using the Newton gradient descent method, the variation in critic weights should conform to the following rule: where η is the learning rate of the corresponding neurofuzzy controller and ω is the adjustable parameter of the controller. Substituting Eq. (17) and Eq. (18) in the above equation and using chain rule yields in

m is the number of inputs to the model and the term, and ∂τ/∂f is the Jacobian matrix in Eq. (7). According to the method in Ref. [33], which proposes a matrix by implementing a neural network, the Jacobian term (∂θ/∂τ)  is obtained as where H(θ) is the mass momentum in the dynamic equation of the system. Also, by taking Eq. (1), into account, the term ∂f/∂ω is calculated as

Eq. (19) updates the coefficients of the TSK controller as the critic rule.

4. Results

In this section, the 3-DOF model, along with the neurofuzzy critic-based controller allocated individually to each of the muscles, is simulated numerically. First, the limits of membership functions e and are determined for the TSK system. Simulation of the model by arbitrary shows that the values of e = e = 0.2, as well as , can be acceptable. In the next step, the initial values selected randomly in the range of [−100,100] are assigned to matrix ω in Eq. (15), for six muscles. These coefficients are updated by Eq. (19) in each step to minimize the cost function value. The minimization is originally conducted by minimizing the error values of e and that finally resulted in the appropriate system's performance. The parameters related to the controller are selected as

To indicate the controller's performance without considering the effect of muscle optimization, the process is performed also with k = zeros1.6 × 10−5 (for j = 6). The model parameters and the values associated with the joint types of muscles are listed in Table 2 and Table 1.

Table 2

Numerical values of the model.

	Length (m)	Mass (kg)	Inertial moment (kg·m2)	CoM position (m)
1st link	0.31	1.93	0.0141	0.165
2nd link	0.27	1.32	0.0120	0.135
3nd link	0.15	0.35	0.0010	0.075

For evaluating the controller's performance, a semicircular trajectory is applied by the following equation in the workspace:

The total simulation time is assumed to be T = π (s), and during the aforementioned period, the model is expected to fully go through the trajectory. To show the controller's robustness against the system uncertainties, a 10% diversion is considered for the values of the mass and inertia in the model. Figure 4 displays the arm model's motion trajectory in both cases with and without considering muscles' optimization. As it is depicted, both models can follow the desired trajectory with acceptable accuracy. It should be noted that the following error is a little less for nonoptimized mode. This is because the focus, in this case, is only on reducing the trajectory error, and the model is not seeking to optimize the muscle forces.

Figure 4

The moving trajectory of the model with/without optimizing the muscular forces.

Figure 5 shows the magnitude of forces applied to each muscle during the motion. The muscle forces' values are significantly lower in the optimized mode, showing the controller's correct performance. The maximum values of the forces are also in the intended range and controlled properly. These limited values in muscle forces are one of the main features that resulted from applying optimal control to the model.

Figure 5

The input forces of each muscle during the motion.

Finally, Figure 6 illustrates how each joint displaces during motion. The proposed values imply that the two cases have select completely different configurations to go through the trajectory. In the case of optimized muscles, the displacement of muscles is lower, i.e., links got through the trajectory with a lower overall displacement than the optimized muscles. The obtained results are in good agreement with the hand's natural motion, which is always sought the optimal trajectory of motion. This figure shows the advantage of the muscle optimization method.

Figure 6

The displacement of each joint during motion.

5. Conclusion

The given trajectory was followed properly by controllers with/without muscle optimization. However, the tracking error was slightly lower in the absence of optimization, caused by the controller's focus to track the desired trajectory without minimizing the muscle forces. In conjunction with the optimal controller, the muscle forces were much lower than those of the nonoptimal controller, suggesting a significant role of muscle optimization in improving the controller's performance. The maximum values of muscle forces were also in the desired range and well-controlled. This limited force is one of the main features of the optimal control strategy applied to the model. In the case of optimized muscles, the joints displacement was lower, i.e., links go through the trajectory with a lower overall displacement compared to nonoptimized muscles case, and this shows the good agreement of results with the natural motion of the hand, which is always sought the optimal trajectory of motion. We intend to enable the movement of the arm exactly along complex trajectories as well as the compensation of dominant external disturbances [34, 35]. Moreover, future research will mainly aim to experimentally analyze the results obtained. The feasibility of the proposed neurofuzzy control system is proposed for future researches. The proposed neurofuzzy controller should contain essential features such as adaptivity and muscle force optimization. Moreover, other methods such as machine learning and artificial intelligence can be applied to reach optimum results.