Optimization of fractional-order chaotic cellular neural networks by metaheuristics.

Artificial neural networks have demonstrated to be very useful in solving problems in artificial intelligence. However, in most cases, ANNs are considered integer-order models, limiting the possible applications in recent engineering problems. In addition, when dealing with fractional-order neural networks, almost any work shows cases when varying the fractional order. In this manner, we introduce the optimization of a fractional-order neural network by applying metaheuristics, namely: differential evolution (DE) and accelerated particle swarm optimization (APSO) algorithms. The case study is a chaotic cellular neural network (CNN), for which the main goal is generating fractional orders of the neurons whose Kaplan-Yorke dimension is being maximized. We propose a method based on Fourier transform to evaluate if the generated time series is chaotic or not. The solutions that do not have chaotic behavior are not passed to the time series analysis (TISEAN) software, thus saving execution time. We show the best solutions provided by DE and APSO of the attractors of the fractional-order chaotic CNNs.

Introduction

Chaos is a discipline that has shown novel applications in different engineering areas [1]. Nowadays, fractional-order chaotic systems are a hot topic for research, but yet the solution of the dynamical system is a challenge, because there is not an analytical method to solve them. Almost all dynamical systems can be converted from its integer-order models to fractional-order ones. In both cases, one can evaluate their performances as dynamical characteristics that are associated to Lyapunov exponents, entropy, and Kaplan–Yorke dimension (). These characteristics can be optimized by metaheuristics [2-4]. For instance, among the available optimization algorithms, differential evolution (DE) and particle swarm optimization (PSO) have shown advantages in maximizing fractional-order systems [5]. In this paper, we apply DE and accelerated PSO (APSO) to optimize fractional-order neural networks that can have commensurate (if the derivatives have the same fractional-order) or incommensurate (if the derivatives have different fractional-order) orders.

Artificial neural networks (ANNs) are associated to model the behavior of a biological neural network, as in the human brain [6]. They are modeled by mathematical equations, some of them involving continuous-time and other discrete time variables. Chaotic neural networks have shown advantages in generating random binary sequences to encrypt color images, as shown in [7]. Other applications can be found in [8], where the authors show recent developments on fuzzy logic, neural networks and optimization algorithms, as well as their hybrid combinations, and their application in areas such as intelligent control and robotics, pattern recognition, medical diagnosis, time series prediction, and optimization of complex problems. More recently, it has been proven that neural networks are a good option for the analysis of the coronavirus pandemic problem [9]. One can list a huge number of applications of neural networks, but in this paper, the efforts are focused on problems involving fractional-order issues [10]. For instance, the following fractional-order neural networks can be suitable for optimization: the fractional-order residual convolutional neural network introduced in [11]; the distributed-order neural networks introduced in [12]; the fractional-order neural network with time-varying delays given in [13]; the fractional-order quaternion-valued neural network given in [14]; the fractional-order complex-valued neural network with impulsive effects given in [15]; the fractional-order memristive neural synaptic weighting given in [16]; the fractional-order complex-valued neural network given in [17], and so on.

As it is well known, the challenge on applying metaheuristics is the reduction of the execution time in the optimization loop. In this manner, as the evaluation of the dynamical characteristics of a fractional-order chaotic system consumes large time, we introduce a procedure to eliminate the evaluation of of a fractional-order CNN when the behavior is not chaotic. Henceforth, the proposed method consists on applying Fourier transform to the chaotic time series, to evaluate if the CNN generates chaotic behavior or not. If the CNN is chaotic, its time series are introduced to the time series analysis (TISEAN) software to evaluate . If the CNN is not chaotic, the individual in DE or particle in APSO is eliminated and this saves execution time in the optimization loop.

The organization is done as follows: Sect. 2 describes the mathematical model of the cellular neural network (CNN) in its fractional-order version. Section 3 describes the metaheuristics, namely: DE and APSO algorithms that are used to optimize the CNN. The formulation of the optimization problem is shown in Sect. 4, where the goal is maximizing , and we show the application of Fourier transform to reduce execution time of DE and APSO. Section 5 summarizes the optimization results and lists the best individuals provided by DE and solutions by APSO along their corresponding chaotic attractors. Finally, the conclusions are given in Sect. 6.

Fractional-order chaotic cellular neural network

Some recent fractional-order chaotic systems have been introduced in [18, 19], and the commensurate and incommensurate characteristics have been described in [20, 21]. This paper focuses on the Cellular Neural Network (CNN) that was proposed in [22]. Basically, it consists of a series of modeled cells by an analog nonlinear circuit. In [23], the authors analyzed the chaotic behavior of the autonomous CNN composed by two or three cells. For instance, the CNN modeled by three cells is given in (1). In this mathematical model: x, y, and z are neural states; f(x) , f(y) , and f(z) are neural activation functions shown in (2); and , , , s, and r are the weights or coupling forces of the network, which values generating chaotic behavior are set to , , , , and As one sees, (1) is a dynamical system of integer order, and it can be transformed to its fractional-order version, as shown in (3). It can be observed that the differential equations changed to have fractional orders denoted by . One can have two cases for the optimization process of the fractional orders: if the CNN has commensurate orders, otherwise, the CNN has incommensurate orders, i.e., In general, the optimization of a continuous-time system can have infinite possibilities for the design variables, in this case the coefficients and fractional orders of the derivatives. However, not all the possibilities will lead to generate chaotic behavior, so that one can estimate reduced search spaces for , , , s, and r, and for the fractional orders, one can also establish fractional ranges for , , and , as listed in Table 1, which have high possibilities of generating chaotic behavior than other random values.

Table 1

Proposed search space ranges of the design variables for the optimization process

Design Variable	Range
p_1	[1.00000000, 1.50000000]
p_2	[0.50000000, 1.50000000]
p_3	[0.50000000, 1.50000000]
s	[2.50000000, 4.00000000]
r	[4.00000000, 5.00000000]
q_i	[0.60000000, 0.99999999]

The authors in [24] demonstrated that for a chaotic oscillator consisting of three fractional-order derivatives, and considering different types of model nonlinearities, and using the proper control parameters, chaotic attractors are obtained with system orders as low as 2.1. Consequently, those authors introduced a conjecture that third-order systems can still produce chaotic behavior with a total system order higher than 2. In this manner, the fractional-order chaotic CNN modeled by (3) has three differential equations, and its minimum fractional order to generate chaotic behavior must be higher than two. This can be accomplished if all the commensurate fractional orders of the derivatives have values equal or higher than 0.7. However, for incommensurate fractional orders, one can have combinations of values, so that the sum of the three fractional orders be higher than two. To know the dimension of the problem: By setting the search spaces from 0.60000000 to 0.99999999, it leads us to have possibilities of having . This number increases exponentially when varying , , , s, and r. For this reason, metaheuristics are a good option to optimize a fractional-order chaotic CNN.

Metaheuristics

The pseudocodes of DE and PSO algorithms have already given in [5] for the optimization of fractional-order chaotic oscillators. In this paper, DE and APSO are applied to optimize a fractional-order chaotic CNN. The pseudocodes are summarized in the following subsections.

Differential evolution (DE)

DE algorithm is a metaheuristic that uses real numbers in its representation and then can be applied to optimize continuous-time problems. DE requires as input, the size of the population, number of generations, the value for the recombination constant (CR), the value for the differential constant (F), and the threshold value s. The pseudocode for the optimization of the chaotic CNN is given in Algorithm 1. The evaluation of is performed, as shown in Sect. 4.2, including the verification of chaotic behavior of the times series using the spectrum.

The generations are updated by creating new vectors by performing mutation (4) and crossover (5) operations. Where, a, b, and c are different numbers randomly chosen; g denotes the current generation; is the differential constant; ; is the jth evaluation of a generated random number; is a crossover coefficient chosen by the user; and is an index randomly generated [5]

Accelerated particle swarm optimization (APSO)

Similar to DE, APSO algorithm is a metaheuristic that performs a direct random search in an intelligent sense. Generally, APSO requires the following parameters as input: size of the population, number of generations, , and . The pseudocode of APSO for the optimization of the chaotic CNN is given in Algorithm 2. The evaluation of is performed, as shown in Sect. 4.2, including the verification of chaotic behavior of the times series using the spectrum.

The particle behavior is defined by two equations: velocity (6) and position (7). Where, i is the index of the particle; j its dimension; the best position found in i; the best position found during the optimization; is the inertial weight; the acceleration constant; and a random number generator with uniform distribution [5]

Optimizing of a chaotic CNN by DE and APSO

This section shows the fitness function of the optimization process, and the handling of constraints highlighting the use of Fourier transform to verify if the time series is chaotic or not and, therefore, save execution time when evaluating by TISEAN, who is time-consuming.

Fitness function

In chaotic systems, is evaluated considering the number of ordinary differential equations. In this case, the chaotic CNN has three equations and its dimension can be evaluated by applying (8). This dimension is related to the Lyapunov exponents , which are equal to the number of state variables. In this case, the chaotic CNN has three state variables x, y, z, so that it has three Lyapunov exponents, and to show a chaotic behavior, one must be positive, one zero, and another negative, and therefore, [25]. Maximizing leads us to deal with the fitness function given in (9)Among the available methods to evaluate Lyapunov spectrum, one can use the online available software called TISEAN, who requires the chaotic time series of the associated state variable x, y, z, as input. This software is used herein to evaluate , but is time-consuming, so that we propose to verify if the chaotic time series is chaotic or not by computing the Fourier transform, and it is taken as a constraint.

Verification of chaotic behavior using the spectrum

The optimization algorithms based on populations as DE and APSO are time-consuming, so that to reduce the execution time, we propose the application of Fourier transform to select those individuals in DE or particles in APSO that can be passed to TISEAN to compute . Besides, the first constraint is imposed by (10) for both commensurate or incommensurate fractional ordersA second constraint consists on verifying the eigenvalues that must be complex to generate oscillating behavior. If the mathematical model does not have equilibrium points, as the case of the fractional-order chaotic CNN given in (3), then one can apply a numerical method as Newton–Raphson [26], which is given in (11). This method is very fast, so that the execution time of the metaheuristic is not increased as for the one taken by TISEANGuaranteeing the constraint in (10) and that the eigenvalues are complex conjugated may not lead to chaotic behavior. In fact, in many cases, the time series may have periodic behavior or they can be damped. In this manner, as DE and APSO work with populations, if the individuals are verified to have chaotic behavior, the execution time can be reduced, because is evaluated by TISEAN and it is the bottleneck in the optimization loop. See for example the spectrum of a chaotic time series shown in Fig. 1, and the spectrum of a non-chaotic time series shown in Fig. 2. It is very notable that the spectrum of a chaotic time series has higher amplitude than a non-chaotic one. For this reason, we propose to evaluate the Fourier transform of each time series before the individual or particle pass to TISEAN and this is decided by setting a threshold for the spectrums. In this work, the threshold was set to 0.2.

Fig. 1

Fourier transform of a chaotic time series

Fig. 2

Fourier transform of a non-chaotic time series

The pseudocode including the Fourier transform as constraint is given in the procedure named Evaluate(), shown below. This procedure is executed in Algorithm 1 at step 3, and in Algorithm 2 at step 4 to evaluate . As one can see, the first step is verifying the fractional order to accomplish (10). If this is accomplished, the eigenvalues are computed applying Newton–Raphson. If the eigenvalues are complex conjugated, the chaotic CNN is simulated by applying a numerical method, as the Grüwald–Letnikov [1]. The generated time series is evaluated with the Fourier transform to double check that is chaotic behavior. Finally, the chaotic time series is processed by TISEAN computing .

Results

Both DE and APSO were programmed in python language. The simulation of the fractional-order CNN modeled in (3) is done applying Grünwald–Letnikov with a step-size and a short memory length of of the total length, e.g., for 20, 000 samples. The initial conditions for all the cases were set to , , and .

Best feasible solutions provided by DE

The input parameters that were used for DE algorithm are listed in Table 2. It can be noted that it includes the parameters given in Subsect. 3.1

Table 2

DE input parameters

Parameter	Value
Population size	40
Number of generations	20
Number of variables	8
Difference constant	0.4
Recombination constant	0.1
min_p_1[1], max_p_1[1]	1.00000000, 1.50000000
min_p_2[2], max_p_2[1]	0.50000000, 1.50000000
min_p_3[3], max_p_3[1]	0.50000000, 1.50000000
min_s[4], max_s[1]	2.50000000, 4.00000000
min_r[5], max_r[1]	4.00000000, 5.00000000
min_q_i[6-8], max_q_i[6-8]	0.60000000, 0.99999999

The four best feasible solutions provided by DE are given in Table 3. It can be noted that solution 2 has incommensurate, while solutions 1, 3, and 4 have commensurate fractional order, but the other parameters are totally different, and all these best solutions have good values for the fitness function, i.e., .

Table 3

Four best solutions provided by DE

Variable	Solution 1	Solution 2	Solution 3	Solution 4
p_1	1.2249	1.22556604	1.19199827	1.23340808
p_2	1.08	1.13855862	1.1	1.17102638
p_3	1.0	0.96259797	1.0	1.0
s	3.2	3.19792497	3.02918670	2.95860449
r	4.4	4.55205362	4.4	4.4
q_1	0.99	0.98373450	0.99	0.99
q_2	0.99	0.99857154	0.99	0.99
q_3	0.99	0.98559591	0.99	0.99
D_KY	2.41112811	2.52850903	2.59403606	2.73015194

Figure 3 shows the chaotic attractors of the solutions provided by DE for the fractional-order CNN. It can be appreciated that they have very similar ranges in the state variables x and y.

Fig. 3

Chaotic attractors for the solutions given in Table 3: a Solution 1, b Solution 2, c Solution 3, and d Solution 4

Best feasible solutions provided by APSO

APSO algorithm was executed by setting the input parameters, as shown in Table 4. It includes the values of the search spaces ranges of the design variables given in Sect. 3.2.

Table 4

APSO input parameters

Parameter	Value
Population size	40
Number of generations	20
Number of variables	8
β	0.3
α	0.7
min_p_1[1], max_p_1[1]	1.00000000, 1.50000000
min_p_2[2], max_p_2[1]	0.50000000, 1.50000000
min_p_3[3], max_p_3[1]	0.50000000, 1.50000000
min_s[4], max_s[1]	2.50000000, 4.00000000
min_r[5], max_r[1]	4.00000000, 5.00000000
min_q_i[6-8], max_q_i[6-8]	0.60000000, 0.99999999

Table 5

Four best solutions provided by APSO

Variable	Solution 1	Solution 2	Solution 3	Solution 4
p_1	1.2249	1.21524764	1.28832790	1.21529417
p_2	1.08	1.02487968	1.14787860	1.02466857
p_3	1.0	1.16548903	1.05392120	1.16572212
s	3.2	3.20087397	3.10834142	3.20088438
r	4.4	4.60504481	4.44069088	4.46546546
q_1	0.99	0.99057132	0.94994949	0.99040112
q_2	0.99	0.98569995	0.97372856	0.98570651
q_3	0.99	0.97725932	0.94780070	0.97719746
D_KY	2.41112811	2.63592020	2.64600627	2.83383367

The four best solutions provided by APSO are given in Table 5. In this case, just one commensurate case associated with solution 1 is listed. the other three solutions 2, 3, and 4 have incommensurate fractional order, but in all cases, again the fitness function is high. In fact, APSO provided higher than DE.

Figure 4 shows the chaotic attractors associated to the four best solutions provided by APSO.

Fig. 4

Chaotic attractors for the solutions given in Table 5. a Solution 1, b Solution 2, c Solution 3, and d Solution 4

Conclusions

This paper showed the optimization of a fractional-order cellular neural network by applying metaheuristics, such as DE and APSO. The fitness function was associated to maximize the Kaplan–Yorke dimension, which is evaluated by TISEAN program. To reduce execution time of the optimization loops in both metaheuristics, we proposed the evaluation of the Fourier transform to the time series of the CNN and apply a threshold to the spectrum to verify chaotic behavior. However, to verify that the individual or particle in the population will generate chaotic behavior, some constraints were evaluated, namely: verification that the fractional order is higher than two, verification that the eigenvalues are complex, and verification that the Fourier transform has high amplitude values in the spectrum, e.g., 0.2, in this work. These constraints driven DE and APSO algorithms to maximize , providing feasible solutions whose attractors have similar ranges in their phase portraits. The best results provided by the metaheuristics were associated to incommensurate fractional-order CNNs. They can be implemented on embedded systems for the development of real applications as encryption systems under a protocol for IoT.