Generalized stability conditions for coupled neural networks with delay feedbacks.

We present a generalization of the existing models of delayed neural networks (DNNs) with positive delay feedback. A generalized criterion for stability of the system of delay differential equations (DDEs), which governs the dynamics of DNNs, around the trivial local equilibrium is also provided.

Introduction

An Artificial Neural Network (ANN) is a paradigm of the information processing systems, which are designed to operate in accordance with the complex biological learning systems such as human brains. The architecture of a neural network consists of non-linear information processing elements (neurons) and the interconnections between them (synapses). The neurons are generally arranged in layers and this arrangement is referred to as the topology of a neural network. According to this topology, a neural network can be classified into feed-forward network, in which the signals are propagated in only one direction, and feedback network, wherein the signals are allowed to propagate in both directions by introducing loops. The feedback network is comparatively closer to the resemblance of a biological neural network due to the presence of feedback, which is an intrinsic component of every physiological system.

In a biological neural network, there are time delays involved in the propagation of signals along the axons and dendrites. These delays are incorporated in ANNs and such networks are known as delayed neural networks (DNNs). They immensely affect the dynamics of the network and the effects are quite complicated. While it is shown in [1], [6], [7], [8], [9] that a time delay leads to distortion of stability and convergence properties of a neural network thereby creating oscillations and chaos, an increased stability of several dynamical systems due to a time delay is also presented in [11], [12], [13], [14], [19], [34]. Furthermore, Marcus and Westervelt [2] has found out that a single time delay or identical multiple time delays can destabilize the network as a whole and create oscillatory behavior when the connection matrix is symmetric whereas Campbell [3] has observed Hopf bifurcation of codimension two and Hopf-Hopf bifurcations in neural networks with distinct multiple delays.

The dynamics of natural systems—physical, chemical and biological—involve time delays in the evolution of a stable state and, therefore, can be modeled by delay differential equations (DDEs), which are also referred to as the retarded functional differential equations. These equations allow us to understand the critical role of a time delay in the evolution of a state of the system. The most fundamental functional differential equation is the first order linear DDE given in [5] as: where and are real-valued functions dependent on time t, x is an element of a vector space of dimension n (say) over (the set of complex numbers), and τ is the time delay involved in the time evolution of x. The mathematical models governed by DDEs are very useful in describing the dynamics of neuron interactions with time delays which are generally reflected in experimental data such as EEG, EMG, fMRI, DTI, etc. For coupled systems of DNNs, the dynamics include the excitatory and inhibitory influences and the activation level of one neuron is affected by the local feedback from the other neurons [6]. To gain a better insight of the characteristics of such dynamical systems under perturbations, one needs to focus on the equilibria and their stability.

The stability analysis allows us to understand the near-equilibrium properties of the flows of the systems under perturbations. An equilibrium is locally stable if a dynamical system near the equilibrium approaches towards it and exhibits its critical behaviors near that equilibrium point. The behaviors may be different for different local equilibrium points. The local stability analysis, in general, refers to the linear approximation of a non-linear model in the neighbourhood of a local equilibrium point and analyzing its critical behaviors near the point. The local stability of a system can be characterized by calculating the eigenvalues from the Jacobian of the corresponding linear system. There are several applications [4] of this analysis in various dynamical systems such as exponential and logistic models of population growth, models of natural selection, delayed models of neural networks, etc.

Neural networks have a remarkable ability to learn from the input patterns. The learning is conceptually similar to the way the biological neural networks are trained to retain memories in the human CNS. They are trained with learning algorithms such as Backpropagation (supervised), K-means clustering (unsupervised), etc. However, these networks pose several challenges while using them for data mining. Neural networks can take a long time to train due to large computing resources and a great deal of data for learning. They normally require specifications of the architecture in advance. Moreover, the information stored in neural networks cannot be easily translated into comprehensible knowledge. In spite of these challenges, the study of neural networks is pivotal for data mining as these networks are excellent in solving a variety of problems in pattern recognition, prediction, associative memory, optimization, etc.

The modeling of DNNs has been a cutting-edge research in the area of neuroscience. Both single variable as well as multivariate discrete and continuous time models for DNNs have been developed and investigated in different dimensions. For instance, Liao et al. [7] has studied a single delayed neuron model, while elaborate discussions of two-neuron models have been presented in several works [8], [9], [10], [11], [12]. Moreover, tri-neurons networks have also been studied with different feedback models [5], [6], [13], [14], [15], [16], [17] and four-neuron BAM models [18], [19], [20] have been discussed up to some extent. However, there has not been a generalized model of neural network from which a simpler model can be deduced, and the characteristic equation of the Jacobian associated with the governing system of DDEs is yet to be determined.

We present an N-dimensional DNN as a generalization of the existing models, where N is an arbitrary positive integer, and delineate its canonical properties. It is noteworthy that the perspective of generalization draws references from some of the existing models. Our work focuses on the local stability analysis of the systems of DDEs, which govern our DNN models, around the trivial local equilibrium. We also determine the conditions for the subsistence of multiplicity 2 of the zero root. A system whose associated characteristic equation has a zero root of multiplicity 2 leads to Bogdanov-Takens bifurcation [6], [21]. One can use the centre manifold reduction [22], [23], [24], [25], [26], [27] and the normal form method [28], [29], [30], [31] to compute the simpler normal form of the governing DDEs and analyze the dynamic behaviors of DNNs [6]. Finally, we observe a similarity in the Jacobians and the corresponding characteristic equations of the systems for all of our DNN models. We then generalize, from this standpoint, the form of the Jacobian and the corresponding characteristic equation and provide a generalized stability criterion by imposing certain conditions.

Methodology

The local stability analysis of a system of DDEs such as (1) is performed by linearizing the non-linear functions, which appears in the DDEs of DNN models, through the Taylor series approximation around a local equilibrium of the system. We assume that the non-trivial solutions of the linearized system exist as , where λ is the eigenvalue of the linearized system and C is a constant in . We then substitute a non-trivial solution and compute the Jacobian, where 's and 's are the coordinates of x and C respectively. Here, n is also the number of neurons involved in the model. The characteristic equation of the linearized system is obtained from the above Jacobian and its zero root can be determined. This procedure is illustrated in the following example:

Let us consider the system of constant-coefficient DDEs of the following form: The corresponding characteristic equation is obtained in view of the above linearization: We then check for the existence of the zero root of (3) and the necessary conditions for the subsistence of multiplicity of the roots. This model is studied in [21]. The explicit stability conditions are obtained with the help of the following elementary lemma.

If is a root of of multiplicity r, then is a root of having a multiplicity r is a single root of and not a root of , where denotes the derivative of F w.r.t. x of order .

Model-1: 1D delayed neural network model

We consider a one-dimensional neural network which is slightly different from that investigated by Liao et al. in [7]. We take into account a time delay of the network in the model equation, which is given below: where is the feedback strength having a delay . Let us assume that the system (4) possesses an equilibrium at the origin, and the solution exists in the linear form as x = . The model equation is reduced to the linear form: using Taylor series approximation on the sigmoid function ‘tanh’ around the trivial equilibrium. The characteristic equation of the linear form is then obtained using the local stability analysis as:

λ = 0 is a single root of (6) if and only if α = 1.

If λ = 0 is a single root of (6), then which gives . Conversely, one has and , which completes the proof by using Lemma 1. □

Model-2: 2D delayed neural network model

There have been extensive studies [8], [9], [10], [11], [12] for numerous models of two-neuron networks. We consider the DNN model studied by Fan et al. [21]. The network is modeled, along with the introduction of delayed self-feedback and a delayed connection from the other neuron, by a coupled system of DDEs as follows: where and are the connection strengths with and as the respective connection delays and α > 0 is the feedback strength having a delay . The system (7) linearizes to the following: The characteristic equation in λ for the system (8) is then obtained as: where .

The characteristic equation (9) has a zero root (λ = 0) of multiplicity 2 if and only if

The maximum multiplicity of the zero root is 2.

Let us assume that λ = 0 is a root of (9) having multiplicity 2. Then, one has: which leads to: Solving these equations yields the conditions (10). For the other direction, it suffices to show, using Lemma 1, that = 0 for and ≠ 0 when and satisfy the conditions (10). Evidently, , and .

We prove this by contradiction. Suppose is a multiple root of (9) having multiplicity 3. This necessarily yields: = 0 for . As a result, and satisfy the conditions (10) and . However, upon substituting the values of and in , we obtain: which contradicts the results of (10). □

Model-3: 3D delayed neural network model

In the three-neuron DNN model, each neuron has the ability to activate itself and each new activation is dependent on the history of its previous activation [6]. The axonal and dendritic propagation time, also called the synaptic delay, is considered to be associated with the local positive feedback, which is biologically termed as ‘reverberation’ [34]. The model is described by the following system of equations: where for all . Here, represents the activation level of the ith neuron with the activity coefficient , β denotes the inhibitory influence measure of the past history, and is the reverberation for . The linearization of equation (11) allows us to obtain the following system of equations: and the corresponding characteristic equation at the trivial local equilibrium point is obtained from the Jacobian, as the following functional form: If β = 1 - and 1 < α < 4, where α = , then has a zero root of multiplicity two and no purely imaginary roots. The conditions of single root and double root are briefly discussed in [6].

Model-4: 4D delayed neural network model

The delayed ‘bidirectional associative memory’ (BAM) neural networks of four neurons with time delays have been investigated extensively in [18], [19], [20]. We now present an innovative approach to the modeling of DNNs containing four neurons by extending the 3D DNN model. The features and the functional forms are equivalent, but cross-linkages among the neurons are introduced in the 4D model. Due to these new connections, there arise different measures of the inhibitory influence of the past history as shown in Fig. 1. The different measures signify that the inhibitory influences vary with different time delays. Moreover, the local positive feedback is also increased. The model equation is given by: where for and any integer k, (mod 4), and are the measures of inhibitory influences during time delays and respectively, and are the reverberations, and and have their usual meanings. Let us set = = τ. The system (12) is then reduced to the following linearized form: The corresponding Jacobian matrix of the model is given by where = , for . Evidently, we arrive at the following characteristic equation: where . We now provide a few stability conditions of the 4-D neuron model around the trivial equilibrium.

Figure 1

Architecture of the four-neuron model.

Suppose = 1 and = 1 - . Then,

λ = 0 is a zero root of multiplicity 1 if and only if and

λ = 0 is a zero root of multiplicity 2 if and only if where = - , = and = .

We use Lemma 1 for the proof. We note that Substituting the supposed values of and ultimately yields . And, Again, upon substitution of the values of and , we obtain: If , it eventually leads to: , ≠ 0. This completes the proof. □

Model-5: 5D delayed neural network model

The 5D DNN model is an extension of our 4D DNN model. The increase of one neuron adds up a set of cyclic pathways of neurons to the network. The new pathways result in an increase of influences and feedbacks from other neurons and their past history. The model is governed by the following system: where for all (the set of integers), , and ,

(mod 5),

is the measure of the inhibitory influence during time delay ,

are the reverberations for , and

the other notations have their usual meanings.

The linear form of (14) is correspondingly obtained as follows: Assuming that the delays are identical and the non-trivial solutions exist in the linear form as in the previous models, the corresponding Jacobian is given by: The characteristic equation of the system (15) is then obtained as follows: We set: As in Proposition 2, one can similarly obtain a stability condition here also by setting: However, this approach would be increasingly cumbersome as the dimension of the model increases. In light of this, we introduce another approach by imposing strict conditions on 's and their coefficients. Under these conditions, the new approach would serve as a technique for determining stability conditions of a generalized DNN.

In this new approach, we consider all the measures of inhibitory influences of the past history to be identical and place a constraint on , for all , as follows: This reduces the characteristic equation of (15) to:

If is a root of unity and , the reduced characteristic equation (16) has a zero root of multiplicity:

1 if and only if and

2 if and only if .

Using equation (16), we have: Since the sum of n roots of unity is zero for all , This reduces F to: Thus, 0 is a root of (16) when . To find the multiplicity of the zero root, we note that: which gives: Thus, when . □

Generalization of the DNN models

We finally introduce a generalized model of DNN by increasing the number of neurons to an arbitrarily large positive integer N. The model comprises of a network of N neurons with cross-linkages among them. For all , the local feedback responses are cyclic networks consisting of neurons. Every pathway within a cyclic network of neurons is assumed to have a different measure of the inhibitory influences of the past history which is denoted by , where , except for the largest cyclic network of N neurons wherein is the measure for all pathways. Although the time delays are set to be identical ( = τ), the synaptic delay of the neurons differs from each other. Thus, the N-neuron model is described by the following system: where (mod N) and the term symbols have their usual meanings. Similarly, the system (17) linearizes to the following: Let us assume as before that the non-trivial solutions exist in the form: , where λ is the eigenvalue of the linearized system and C is a scalar in . We note here that the corresponding Jacobian is not straightforward. The following lemma explains the features of the Jacobian. We now place constraints on 's and as follows: We note that for all , there exists at least one such that . This reduces the generalized characteristic equation to: The generalized stability condition is then obtained as follows:

The Jacobian of (18) has the following characteristics:

For lower triangular region of the matrix, each entry of the diagonal having () elements have the following form all the way alongside the main diagonal:

For upper triangular region of the matrix, diagonal having () elements have the entries alongside the main diagonal in the form of and

For i.e. diagonal elements, each entry is of the form () where λ is the eigenvalue.

The characteristic equation for the linear form of the system of a generalized DNN is of the form: where, for all and ,

, where and 's are permutations of ,

's are symmetric functions of 's of degree k, and

, where 's are integers between 0 and N such that for each , , and 's are permutations of .

Assume is the Nth root of unity for and . Then, the reduced generalized characteristic equation has a zero root of multiplicity:

1 if and only if and

2 if and only if .

Using the reduced generalized characteristic equation, one has: Thus, 0 is a root when . Upon differentiating the reduced generalized characteristic equation, one obtains: which gives: Hence, if which, upon substituting , reduces to □

Discussions and conclusions

A generalized DNN model is constructed through an extension of the 3D model. Noting that 1D is the simplification of the 2D model, one could extend the 2D model to obtain another generalized DNN model. The functional form of characteristic equation for the corresponding linearized system is expected to be similar to that in Theorem 2. The investigation of the subtle differences in the dynamics of DNN due to the difference in the model equations is a good topic for follow-ups.

The characteristic equation in Theorem 2 does not have a general formula for its solution when due to an important result in Galois theory that a polynomial equation of degree greater than 4 cannot have a general formula for its solutions. One has to use numerical methods such as “Newton-Raphson” to determine the solutions of the polynomial formula in Theorem 2.

We have also developed an approach to obtain a generalized stability criterion for the system of DDEs governing a DNN. The set of conditions used in determining the stability criterion is not exhaustive. An another way would be changing the constraints on 's and 's, where and . However, this approach becomes tremendously cumbersome to analytically determine the stability criterion when is not the Nth root of unity, for all .

The local stability analysis has led us to an observation that for all of our DNN models, the characteristic equation of the governing system of DDEs can have a zero root, whose multiplicity is dependent on the conditions imposed. If we wisely impose the conditions in such a way that the characteristic equation, except in case of 1D DNN, has a zero root of multiplicity 2 and no other purely imaginary roots, then the associated system exhibits Bogdanov-Takens bifurcation [6].

It is important to note that our work does not include the stability analysis around the non-trivial equilibria of the system of governing DDEs. It is far from trivial to determine the stability of such system around a non-trivial local equilibrium. In fact, the dynamics and stability of coupled systems are characterized by transcendental eigenvalue problems, with transcendental characteristic equations. These transcendental problems could, however, be transformed into algebraic problems by the use of finite element or finite difference methods as shown in [33]. The use of these methods to investigate the stability of a system of DDEs around a non-trivial local equilibrium could potentially align along a direction of further work.

Declarations

Author contribution statement

Sanjeet Maisnam, R.K. Brojen Singh: Conceived and designed the analysis; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.