Self-learning variable structure control for a class of sensor-actuator systems.

Variable structure strategy is widely used for the control of sensor-actuator systems modeled by Euler-Lagrange equations. However, accurate knowledge on the model structure and model parameters are often required for the control design. In this paper, we consider model-free variable structure control of a class of sensor-actuator systems, where only the online input and output of the system are available while the mathematic model of the system is unknown. The problem is formulated from an optimal control perspective and the implicit form of the control law are analytically obtained by using the principle of optimality. The control law and the optimal cost function are explicitly solved iteratively. Simulations demonstrate the effectiveness and the efficiency of the proposed method.

Introduction

With the development of mechatronics, automatic systems consisting of sensors for perception and actuators for action are more and more widely used in applications [1-4]. Besides the proper choices of sensors and actuators and an elaborate fabrication of mechanical structures, the control law design also plays a crucial role in the implementation of automatic systems especially for those with complicated dynamics. For most mechanical sensor-actuator systems, it is possible to model them in Euler-lagrange equations [4,5]. In this paper, we are concerned with the sensor-actuator systems modeled by Euler-lagrange equations.

Due to the importance of Euler-lagrange equations in modeling many real sensor-actuator systems, much attention has been paid to the control of such kind systems. According to the type of constraints, the Euler-lagrange system can be categorized into Euler-lagrange system without nonholonomic constraints (e.g., fully-actuated manipulator [6,7], omni-directional mobile robot [8]), and the system subject to nonholonomic constraint [9] (e.g., the cart-pole system [10], the under-actuated multiple body system [11]). For Euler-lagrange system without nonholonomic constraints, the dimension of inputs are often equal to the dimension of output and the system are often able to be transformed into a double integrator system by employing feedback linearization [12]. Other methods, such as control Lyapunov function method [13], passivity based method [14], optimal control method [15], etc., are also successfully applied to the control of Euler-lagrange system without nonholonomic constraints. In contrast, as the dimension of inputs is lower than that of outputs, it is often impossible to directly transform the Euler-lagrange system subject to nonholonomic constraints to a linear system and thus feedback linearization fails to stabilize the system. To tackle the difficulty, variable structure control based method [16], backstepping based control [17], optimal control based method [18], discontinuous control method [19], etc., are widely investigated and some useful design procedures are proposed. However, due to the inherent nonlinearity and nonholonomic constraints, most existing methods [16-19] are strongly model dependent and the performance are very sensitive to model errors. Inspired by the success of human operators for the control of Euler-lagrange systems, various intelligent control strategies, such as fuzzy logic [20], neural networks [21], evolutionary algorithms [22], to name a few of them, are proposed to solve the control problem of of Euler-lagrange systems subject to nonholonomic constraints. As demonstrated by extensive simulations, these type of strategies are indeed effective to the control of Euler-lagrange systems subject to nonholonomic constraints. However, rigorous proof on the stability are difficult for this type of methods and there may exist some initializations of the state, from which the system cannot be stabilized.

In this paper, we propose a self-learning control method applicable to Euler-lagrange systems. In contrast to existing work on intelligent control of Euler-lagrange systems, the stability of the close loop system with the proposed method is proven in theory. On the other hand, different from model based design strategies, such as backstepping based design [17], variable structure based design [16], etc., the proposed method does not require information of the model parameters and therefore is a model independent method. We formulate the problem from an optimal control perspective. In this framework, the goal is to find the input sequence to minimize the cost function defined on infinite horizon under the constraint of the system dynamics. The solution can be found by solving a Bellman equation according to the principle of optimality [23]. Then an adaptive dynamic programming strategy [24-26] is utilized to numerically solve the input sequence in real time.

The remainder of this paper is organized as follows: in Section 2, preliminaries on Euler-lagrange systems and variable structure control are given briefly. In Section 3, the problem is formulated as a constrained optimization problem and the critic model and the action model are employed to approximate the optimal mappings. The control law is then derived in Section 4. In Section 5, simulations are given to show the effectiveness of the proposed method. The paper is concluded in Section 6.

Preliminaries on Variable Structure Control of the Sensor-Actuator System

In this paper, we are concerned with the following sensor-actuator system in the Euler-Lagrange form, where q ∈ ℝ(q) ∈ ℝn× is the inertial matrix, C(q,q˙) ∈ ℝ×(q) ∈ ℝ and u ∈ ℝ. Note that the inertial matrix D(q) is symmetric and positive definite. There are three terms on the left side of the above equation. The first term involve the inertial force in the generalized coordinates, the second one models the Coriolis force and friction, the values of which depend on q̇ and the third one is the conservative force, which is in correspondence to the potential energy. The control force u applied on the system drives the variation of the coordinate q. It is also noteworthy that we assume the dimension of u is equal to that of q here. This definition also admits the case for u with lower dimension than that of q by imposing constraints to u, e.g., the constraint u = [u1,u2, …,u] with u1 = 0 restricts u in a n – 1 dimensional space. Defining state variables x1 = q and x2 = q, the Euler-Lagrange Equation (1) can be put into the following state-space form: Note that the matrix D(x1) is invertible as it is positive definite. The control objective is to asymptotically stabilize the Euler-Lagrange system (2), i.e., design a mapping (x1,x2) → u such that x1 → 0 and x2 → 0 when time elapses.

As an effective design strategy, variable structure control finds applications in many different type of control systems including the Euler-Lagrange system. The method stabilizes the dynamics of a nonlinear system by steering the state to a elaborately designed sliding surface, on which the state inherently evolves towards the zero state. Particularly for the system (2), we define s = s(x1,x2) as follows: where c0 > 0 is a constant. Note that s = c0x1 + x2 = 0 together with the dynamics of x1 in Equation (2) gives the dynamics of x1 as ẋ) = –c0x1 for c0 > 0. Clearly, x1 asymptotically converges to zero. Also we know x2 = 0 when x1 = 0 according to s = c0x1 + x2 = 0. Therefore, we conclude the states x1, x2 on the sliding surface s = 0 for s defined in Equation (3) converge to zero with time. With this property of the sliding surface, a control law driving the states to s = 0 definitely grantees the ultimate convergence to the zero states. Accordingly, the stabilization of the system can be realized by controlling s to zero. To reach this goal, a positive definite control Lyapunov function V(s), e.g., V(s) = s2, is often used to design the control law. For stability consideration, the time derivative of V(s) is required to be negative definite. In order to guarantee the negative definiteness of the time derivative of V(s), exact information about the system dynamics (2) is often necessary, which results in the model based design strategies.

About the Euler-Lagrange Equation (1) for modeling sensor-actuator systems, we have the following remark:

Remark 1 In this paper, we are concerned with the class of sensor-actuator systems modeled by the Euler-Lagrange

Problem Formulation

Without losing generality, we stabilize the system (1) by steering it to the sliding surface s = 0 with s defined in Equation (3). Different from existing model based design procedures, we design a self-learning controller, which does not require accurate knowledge about D(q), C(q,q̇) and ϕ(q) in Equation (1). In this section, we formulate such a control problem from the optimal control perspective.

In this paper, we set the origin as the desired operating point, i.e., we consider the problem of controlling the state of the system (1) to the origin. For the case with other desired operating points, the problem can be equivalently transformed to the one with the origin as the operating point by shifting the coordinates. At each sampling period, the norm of s = c0x1 + x2, which measures the distance from the desired sliding surface s = 0, can be used to evaluate the one step performance. Therefore, we define the following utility function associated with the one-step cost at the ith sampling period, with where s is defined in Equation (3) and s = [s1,s2, …, s], |s| denotes the absolute value of the ith component of the vector s, the parameter δ 0 for i = 1, 2,…, n. At each step, there is a value U and the total cost starting from the kth step along the infinite time horizon can be expressed as follows, where x(k) is the state vector of system (1) sampled at the kth step with γ is the discount factor with 0 < γ < 1, ū(k) = (u1,…,u∞) is the control sequence starting from the kth step. Note that for the deterministic system (1), the preceding states after the kth step are determined by x(k) and the control sequence ū. Accordingly, J is a function of x(k) and ū(k) with J = J(x(k), ū(k)). Also note that both the cost function J and the utility function U are defined based on the discrete samplings of the continuous system (1). Now, we can define the problem of controlling the sensor-actuator system (1) in this framework as follows, subject to: where U is defined by Equations (4) and (5), τ > 0 is the sampling period, the set Ω defines the feasible control actions, J0 is the cost function for k = 0 in Equation (6). It is worth noting that J0 is a function of ū(0) = (u0, u1,…, u∞) and x(0) according to Equation (6). The optimization in Equation (7) is relative to ū(0) with a given initial state x(0). Also note that in the optimization problem in Equation (7), the decision variable u(0),u(1), …,u(∞) are defined in every sampling period. The control action keeps the value in the duration of two consecutive sampling steps. This formulation is consistent with the real implementations of digital controllers.

Remark 2 There are infinitely many decision variables, which are u(0), u(1), …, u(∞), in the optimization problem in

Model-Free Control of the Euler-Lagrange System

In this section, we present the strategy to solve the constrained optimization problem efficiently without knowing the model information of the chaotic system. We first investigate the optimality condition of Equation (7) and present an iterative procedure to approach the analytical solution. Then, we analyze the convergence of the iterative procedure and the stability with the derived control strategy.

Optimality Condition

Denoting J* the optimal value to the optimization problem in Equation (7), i.e.,

subject to: (7b); (7c)

According to the principle of optimality [23], the solution of Equation (7) satisfy the following Bellman equation: where z is the solution of Equation (7b) at t = k + 1 with x(k) = y and the control action u(t) = u for kτ ≤ t < (k + 1)τ. Without introducing confusion, we simply write Equation (9) as follows

Define the Bellman operator ℬ relative to function h(z) as follows

Then, the optimality condition in Equation (10) can be simplified into the following with the Bellman operator,

Note that the function U is implicitly included in the Bellman operator. The Equation (12) constitutes the optimality condition for problem in Equation (7). It is difficult to solve the explicit form of J* analytically from Equation (9). However, it is possible to get the solution by iterations. We use the following iterations to solve J*, subject to: (7b); (7c)

The control action keeps constant in the duration between the kth and the k + 1th step, i.e., u*(t) = for kτ ≤ t < (k + 1)τ. can be obtained from Equation (9) based on Equation (13),

Approximating the Action Mapping and the Critic Mapping

In the previous sections, the iteration (13) is derived to calculate J* and the optimization (14) is obtained to calculate the control law. The iteration to approach J* and the optimization to derive u* have to be run in every time step in order to obtain the most up-to-date values. Inspired by the learning strategies widely studied in artificial intelligence [26,28], a learning based strategy is used in this section to facilitate the processing. After a enough long time, the system is able to memorize the mapping of J* and the mapping of u*. After this learning period, there will be no need to repeat any iterations or optimal searching, which will make the strategy more practical.

Note that the optimal cost J* is a function of the initial state. Counting the cost from the current time step, J* can also be regarded as a function of both the current state and the optimal action at current time step according to Equation (10). Therefore, ĵ(n), the approximation of J*, can also be regarded as a function relative to the current state and the current optimal input. As to the optimal control action u*, it is a function of the current state. Our goal in this section is to obtain the mapping from the current state and the current input to ĵ (n) and the mapping from the current state to the optimal control action u* using parameterized models, denoted as the critic model and the action model, respectively. Therefore, we can write the critic model and the action model as J( x) and (x) respectively, where W is the parameters of the critic model and W is the parameters of the action model.

In order to train the critic model with the desired input-output correspondence, we define the following error at time step n + 1 to evaluate the learning performance,

Note that Bĵ(n) is the desired value of ĵ(n + 1) according to Equation (13). Using the back-propagation rule, we get the following rule for updating the weight W of the critic model, where l(n) is the step size for the critic model at the time step n.

As to the action model, the optimal control u* in Equation (14) is the one that minimizes the cost function. Note that the possible minimum cost is zero, which corresponds to the scenario with the state staying inside the desired bounded area. In this regard, we define the action error as follows,

Then, similar to the update rule of W for the critic model, we get the following update rule of W for the action model, where l(n) is the step size for the action model at the time step n.

Equations (16) and (18) update the critic model and the action model progressively. After W and W have learnt the model information by learning for a long enough time, their values can be fixed at the one obtained at the final step and no further learning is required any longer, which is in contrast to Equation (14) requiring to solve an optimization problem even after a long enough time.

Simulation Experiment

In this section, we consider the simulation implementation of the proposed control strategy. The dynamics given in Equation (1) model a wide class of sensor-actuator systems. Particularly, to demonstrate the effectiveness of the proposed self-learning variable structure method, we apply it to the stabilizations of a typical benchmark system: the cart-pole system.

The cart-pole system, as sketched in Figure 1, is a widely used testbed for the effectiveness of control strategies. The system is composed of a pendulum and a cart. The pendulum has its mass above its pivot point, which is mounted on a cart moving horizontally. In this part, we apply the proposed control method to the cart-pole system to test the effectiveness of our method.

Figure 1.

The cart-pole system.

The Model

The cart-pole model used in this work is the same as that in [29], which can be described as follows. where with the following values of the parameters:

g: 9.8 m/s2, acceleration due to gravity;

m: 1.0 kg, mass of cart;

m: 0.1 kg, mass of pole;

l: 0.5 meter, half-pole length;

μ: 0.0005, coefficient of friction of cart on track;

μ: 0.000002, coefficient of friction of pole on cart;

F: ±10 Newtons, force applied to cart center of mass.

This system has four state variables: y is the position of the cart on track, θ is the angle of the pole with respect to the vertical position, and ẏ and θ̇ are the cart velocity and angular velocity, respectively.

Define , , , , A5 = m + m, A6(θ,θ̇) = mlθ̇ sinθ, A7(θ) = –ml cosθ. With these notations, Equation (19) can be re-written as:

By choosing the system of Equation (19) coincides with the model of Equation (1). Note that the input u in this situation is constrained in the set Ω = {u = [u1, u2]1 = u2 ∈ ℝ}.

Experiment Setup and Results

In the simulation experiment, we set the discount factor γ = 0.95, the sliding surface parameter k = 10, δ1 = 2, δ2 = 24. The feasible control action set Ω in Equation (7) is defined as Ω = {u = [u1,u2]1 ∈ ℝ,u2 ∈ ℝ,u1 = u2 = ±10 Newtons}. This definition corresponds to the widely used bang-bang control in industry. To make the output of the action model within the feasible set, the output of the action network is clamped to 10 if it is greater than or equal to zero and clamped to – 10 if less than zero. The sampling period τ is set to 0.02 seconds. Both the critic model and the action model are linearly parameterized. The step size of the critic model, which is l(n) and that of the action model, which is l(n) are both set to 0.03. Both the update of the critic model weight W in Equation (16) and the update of the action model weight W in Equation (18) last for 30 seconds. For the uncontrolled cart-pole system with F = 0 in Equation (19), the pendulum will fall down. The control objective is to stabilize the pendulum to the inverted direction (θ = 0). Time history of the state variables are plotted in Figure 2 for the system with the proposed self-learning variable structure control strategy. From this figure, it can be observed that θ is stabilized in a small vicinity around zero (with a small error of ±0.1 rads), which corresponds to the inverted direction.

Figure 2.

State profiles of the cart-pole system with the proposed control strategy.

Conclusions and Future Work

In this paper, the self-learning variable structure control is considered to solve a class of sensor-actuator systems. The control problem is formulated from the optimal control perspective and solved via iterative methods. In contrast to existing models, this method does not need pre-knowledge on the accurate mathematic model. The critic model and the the action model are introduced to make the method more practical. Simulations show that the control law obtained by the proposed method indeed achieves the control objective. Future work on this topic includes the theoretical proof of the convergence and exploration on the performance limit of the proposed strategy. Also, the control of other mechanical systems modeled by Euler-Lagrange system, such as manipulators etc., will be explored in our future work.