(Multiscale) Cross-Entropy Methods: A Review.

Cross-entropy was introduced in 1996 to quantify the degree of asynchronism between two time series. In 2009, a multiscale cross-entropy measure was proposed to analyze the dynamical characteristics of the coupling behavior between two sequences on multiple scales. Since their introductions, many improvements and other methods have been developed. In this review we offer a state-of-the-art on cross-entropy measures and their multiscale approaches.

1. Introduction

To quantify the asynchronism between two time series, Pincus and Singer have adapted the approximate entropy algorithm to a cross-approximate entropy (cross-ApEn) method [1]. Then, other cross-entropy methods—that improve the cross-ApEn—have been developed [2,3,4,5,6,7]. Furthermore, additional cross-entropy methods have been introduced to quantify the degree of coupling between two signals, or the complexity between two cross-sequences [8,9,10]. Cross-entropy methods have recently been used in different research fields, including medicine [5,11,12], mechanics [13], and finance [7,10].

The multiscale approach of entropy measures was proposed by Costa et al. in 2002 to analyze the complexity of a time series [14]. In 2009, Yan et al. proposed a multiscale approach for cross-entropy methods to quantify the dynamical characteristics of coupling behavior between two sequences on multiple scale factors [15]. Then, other multiscale procedures have been published with different cross-entropy methods [16,17]. Multiscale cross-entropy methods have recently been used in different research fields, including medicine [18,19,20,21], finance [6,9], civil engineering [22], and the environment [23].

Cross-entropy methods and their multiscale approaches are used to obtain information on the possible relationship between two time series. For example, Wei et al. applied percussion entropy to the amplitude of digital volume pulse signals and changes in R-R intervals of successive cardiac cycles for assessing baroreflex sensitivity [18]. Results showed that the method is able to identify the markers of diabetes by the nonlinear coupling behavior of the two cardiovascular time series. Moreover, Zhu and Song computed cross-fuzzy entropy on a vibration time series to assess the bearing performance degradation process of motor [13]. Results showed that the method detects trend for bearing degradation process over the whole lifetime. In addition, Wang et al. applied multiscale cross-trend sample entropy to analyze the asynchrony between air quality impact factors (fine particulate matters, nitrogen dioxide, …), and air quality index (AQI) in different regions of China [23]. Results showed that the degree of synchrony between fine particulate matter and AQI is higher than the other air quality impact factor which reveals that fine particulate matter has become the main source of air pollution in China.

Our paper presents a state-of-the-art in three sections: First, the cross-entropy methods are introduced. We detail, in the second section, different multiscale procedures. A multiscale cross-entropy generalization is presented and other specific multiscale cross-entropy algorithms are proposed in the third section.

2. Cross-Entropy Methods

In this section, we classify cross-entropy methods according to their entropy measures: Cross-approximate entropy, cross-sample entropy, and cross-distribution entropy. Other methods that use different cross-entropy-based measures are also detailed. Table 1 shows the twelve measures that are detailed in this section.

Table 1

Cross-entropy measures, in chronological order, that are presented in this review. Authors, year, reference, and section location are indicated for each item.

Method	Authors	Year	Ref.	Section
Cross-approximate entropy	Pincus and Singer	1996	[1]	Section 2.1.1
Cross-conditional entropy	Porta et al.	1999	[8]	Section 2.4.1
Cross-sample entropy	Richman and Moorman	2000	[2]	Section 2.2.1
Cross-fuzzy entropy	Xie et al.	2010	[3]	Section 2.4.2
Modified cross-sample entropy	Yin and Shang	2015	[4]	Section 2.2.2
Binarized cross-approximate entropy	Škorić et al.	2017	[5]	Section 2.1.2
Modified cross-sample entropy
based on symbolic	Wu et al.	2018	[6]	Section 2.2.3
representation and similarity
Kronecker-delta based cross-sample entropy	He et al.	2018	[7]	Section 2.2.4
Permutation based cross-sample entropy	He et al.	2018	[7]	Section 2.2.5
Cross-distribution entropy	Wang and Shang	2018	[9]	Section 2.3.1
Permutation cross-distribution entropy	He et al.	2019	[10]	Section 2.3.2
Cross-trend sample entropy	Wang et al.	2019	[23]	Section 2.2.6
Joint permutation entropy	Yin et al.	2019	[24]	Section 2.4.3

2.1. Cross-Approximate Entropy-Based Measures

2.1.1. Cross-Approximate Entropy

Cross-approximate entropy (cross-ApEn), introduced by Pincus and Singer [1], allows to quantify asynchrony between two time series. For two vectors and of length N, cross-ApEn is computed as: where and is the number of sequences, of m consecutive points, of that are approximately (within a resolution r) the same as sequences, of the same length, of . One major dawback of this approach is that should not be equal to zero. This is why cross-ApEn is not really adapted for a short time series. Furthermore, it is direction-dependent because often is generally not equal to its direction conjugate [2]. The value of cross-ApEn computed from two signals can be interpreted as a degree of synchrony or mutual relationship.

2.1.2. Binarized Cross-Approximate Entropy

Binarized cross-approximate entropy (XBinEn), introduced by Škorić et al. [5] in 2017, is an evolution of cross-ApEn to quantify the similarity between two time series. It has the advantage of being faster than cross-ApEn. XBinEn encodes a time series divided into vectors of length m. For two vectors and of length N, the XBinEn algorithm follows these six steps:

Binary encoding series are obtained as: where , , and .

The time lag t allows a vector decorrelation to be performed;

Vector histograms and are computed as: where , and is a function that is equal to 1 if the indicated condition is fulfilled;

The probability mass functions are obtained as: where ;

A distance measure is applied: where ;

The probability that a vector is within the distance r from a particular vector is estimated:

XBinEn is finally obtained as: where .

This method gives almost the same results as cross-ApEn for a non-short time series. However, it is computationally more efficient than cross-ApEn. Its main disadvantage is that it cannot identify small signal changes. XBinEn is adapted to environments where processor resources and energy are limited but it is not a substitute to cross-ApEn [5]. It is proposed when the cross-ApEn procedure cannot be applied. The value of XBinEn computed from two signals can be interpreted as a degree of relationship between a related pair of time series.

2.2. Cross-Sample Entropy-Based Measures

2.2.1. Cross-Sample Entropy

Cross-sample entropy (cross-SampEn) quantifies the degree of asynchronism of two time series. This method was introduced by Richman and Moorman in 2000 to improve the cross-ApEn limitations (see Section 2.1.1) [2]. Cross-SampEn is a conditional probability measure that quantifies the probability that a sequence of m consecutive points (called sample) of a time series —that matches another sequence of the same length of another time series —will still match the other sequence when their length is increased by one sample (). For two vectors and , cross-SampEn is computed as: where m is the sample length, N is the vectors ( and ) length, and are, respectively, the probability that a sequence of and a sequence of will match for and m points (within a tolerance r).

For two time series and of length N, cross-SampEn can also be described as: where represents the total number of sequences of m consecutive points of that match with other sequences of m consecutive points of .

The main difference between cross-ApEn and cross-SampEn is that cross-SampEn shows relative consistency whereas cross-ApEn does not. Unlike cross-ApEn, cross-SampEn is not direction-dependent. However, cross-SampEn generates, sometimes, undefined values for short time series. The value of cross-SampEn computed from two time series can be interpreted as a measure of similarity of the two time series.

2.2.2. Modified Cross-Sample Entropy

Modified cross-sample entropy (mCSE), introduced by Yin and Shang in 2015, has been developed to detect the asynchrony of a financial time series [4]. Inspired by the generalized sample entropy, proposed by Silva and Murta, Jr. [25], the authors proposed to adapt this method to cross-SampEn. The method combines cross-SampEn and nonadditive statistics. For two vectors and of length N, mCSE is computed as: where m is the sample length, q is the entropic index, and is the number of times that the distance between vectors and is less than or equal to the tolerance r. The distance is calculated with .

The value of mCSE computed from two time series can be interpreted as a degree of synchrony between the two time series and it can illustrate some intrinsic relations between the two time series.

2.2.3. Modified Cross-Sample Entropy Based on Symbolic Representation and Similarity

Modified cross-sample entropy based on symbolic representation and similarity (MCSEBSS), introduced by Wu et al. in 2018, has been developed to quantify the degree of asynchrony of two financial time series with various trends (stock markets from different areas [6]). In comparison with cross-SampEn, this method reduces the probability of including undefined entropies and it is more robust to noise. For two vectors and of length N, MCSEBSS is computed as: where m is the sample length and is the number of template matches by comparing and r. For and ( and ), the similarity function is calculated as: where is obtained by the function f defined as:

The parameter r must be fixed between and , where n is the maximum number of zeros obtained with to consider and similar.

The value of MCSEBSS computed from two time series can be interpreted as a degree of asynchrony of the two time series. A low cross-entropy value indicates a strong synchrony between two signals.

2.2.4. Kronecker-Delta-Based Cross-Sample Entropy

The Kronecker-delta-based cross-sample entropy (KCSE), introduced by He et al. in 2018, has been developed to define the dissimilarity between two time series [7]. KCSE is based on the Kronecker-delta function that returns 1 if two variables are equal and 0 otherwise. For two vectors and of length N, KCSE is calculated as: where and . The dissimilarity, between and , is calculated as:

Authors show that KCSE is better to classify financial data than multidimensional scaling based on the Chebyshev distance method [7]. The value of KSCE computed from two time series can be interpreted as a degree of irregularity between the two time series.

2.2.5. Permutation-Based Cross-Sample Entropy

The permutation-based cross-sample entropy (PCSE), introduced by He et al. in 2018, is quite similar to KCSE (see Section 2.2.4) [7]. A permutation step has only been added. For two vectors and of length N, PCSE is calculated as: where and . The KrD function is defined in Section 2.2.4. The two vectors and are obtained by a permutation algorithm defined with the permutation entropy [26]. The Video S1 shows an example of a permutation algorithm.

PCSE shows better results than KCSE for synthetic data (ARFIMA model) [7]. However, the two approaches give the same results for financial data [7]. Authors show that KCSE is better to classify financial data than multidimensional scaling based on the Chebyshev distance method [7]. The value of PCSE computed from two time series can be interpreted as degree of irregularity between the two time series.

2.2.6. Cross-Trend Sample Entropy

Inspired by MCSEBSS (see Section 2.2.3), Wng et al. developed the cross-trend sample entropy (CTSE) to quantify the synchronism between two time series with strong trends [23]. For two time series and of length N, CTSE is calculated with the following four steps algorithm:

The two time series are symbolized as: where and are, respectively, the trend of and obtained by polynomial fitting (linear, quadratic or higher order).

The template vectors and are constructed as: where and .

The similarity between and is calculated as: where the i-th symbol vector is determined with f, a symbolic function between two template vectors and , as:

CTSE is finally computed as: where is obtained by comparing within a tolerance r for .

CTSE has two advantages over MCSEBSS: It is more sensitive to the difference of dynamical characteristic between two signals, and it works well with signals with trends (linear, quadratic, cubic, and sinusoidal) [23]. The value of CTSE computed from two time series can be interpreted as an indicator of dynamical structure regarding the two time series with potential trends.

2.3. Cross-Distribution Entropy-Based Measures

2.3.1. Cross-Distribution Entropy

In 2018, Wang and Shang introduced the cross-distribution entropy (cross-DistEn) to quantify the complexity between two cross-sequences [9]. To generalize the standard statistical mechanics, the authors replaced the standard distribution entropy (DistEn) based on Shannon entropy by DistEn based on Tsallis entropy [9]. The authors showed that cross-DistEn better illustrates the relationships between two vectors than cross-SampEn does [9]. For two times series and of length N cross-DistEn follow these four steps:

The state-space is reconstructed by building vectors and with …, , and …, . m is the intended size of the vectors and ;

The distance matrix is built by defining the distance matrix with being the Chebyshev distance between the vectors and defined as:

The probability density is estimated by computing the empirical probability density function of the matrix by applying the histogram approach. If the histogram has M bins, the probability of each bin will be with ;

The cross-distribution entropy based on the Tsallis entropy is computed as: where q is the order of the Tsallis entropy and a the logarithm base of the entropy computation.

The main advantage of cross-DistEn is that it is adapted for short time series. With financial data, cross-DistEn illustrates better the relationship between signals than cross-SampEn [9]. The value of cross-DistEn computed from two time series can be interpreted as a degree of linkage of the two time series.

2.3.2. Permutation Cross-Distribution Entropy

The permutation cross-distribution entropy (PCDE), introduced by He et al. in 2019, is a variant of cross-DistEn (see Section 2.3.1) [10]. The permutation allows to characterize fluctuations and prevents the impact of spatial distances on results. The PCDE algorithm is the same as the one of cross-DistEn, detailed in Section 2.3.1. However, an additional step is added before step 2 to permute and with the permutation algorithm mentioned in Section 2.2.5. The distance matrix is therefore constructed with the permuted vectors. The value of PCDE computed from two time series can be interpreted as a degree of dissimilarity between the two time series.

2.4. Other Cross-Entropy-Based Measures

2.4.1. Cross-Conditional Entropy

Cross-conditional entropy (CCE), introduced by Porta et al. in 1919, quantifies the degree of coupling between two signals [8]. A corrected conditional entropy has been introduced to improve the approximate entropy that suffers from limitations when a finite number of sample is considered [27]. CCE is an adaptation of the corrected conditional entropy. For two signals and , CCE is computed as: where L is the length of the pattern extracted to be compared, is the joint probability of the pattern , and is the probability of the sample given the pattern . If a mixed pattern, composed by samples, of and : , is defined and with the Shannon entropy , CCE can also be described as:

For a limited amount of samples, the approximation of CCE always decreases to zero while increasing L. To solve this problem, a modification has been introduced as: where is the ratio of mixed patterns found only once over the total number of mixed patterns, and are, respectively, the estimates of the and based on the considered limited dataset.

CCE can be defined as a measure of unpredictability of one signal when the second is observed because it quantifies the amount of information carried by one signal which cannot be derived from the other. It is not fully a measure of synchronization. The main disadvantage of CCE is that it is not totally adapted for short time series.

2.4.2. Cross-Fuzzy Entropy

Cross-fuzzy entropy (C-FuzzyEn), introduced by by Xie et al. in 2010 [3], is an adaptation of fuzzy entropy, introduced by Chen et al. [28], that quantifies the synchrony or similarity of patterns between two signals [3]. C-FuzzyEn is an improvement of cross-SampEn that is more adapted for short time series and more robust to noise. For two times series and of length N, C-FuzzyEn is obtained with the following three steps algorithm:

The distance between and is computed as: where m is the number of consecutive data to compare, , and . and are calculated as: , and ;

The synchrony or similarity degree is computed as: , where is the fuzzy function obtained as: where r and n determine the width and the gradient of the boundary of the exponential function, respectively;

Finally, C-FuzzyEn is computed as: where , and .

The value of C-FuzzyEn computed from two time series can be interpreted as the synchronicity of patterns.

2.4.3. Joint-Permutation Entropy

Joint permutation entropy (JPE), introduced by Yin et al. in 2019, quantifies the synchronism between two time series. It is based on permutation entropy that consists of comparing neighboring values of each point and mapping them to ordinal patterns to quantify the complexity of a signal [26]. For two signals and , JPE is computed as the Shannon entropy of the distinct motif combinations : where d is the embedded dimension and is the joint probability of appearing in the and and it is defined as: where , corresponds to the map from pattern space to symbol space, and corresponds to the cardinality of a set.

The main advantages of JPE are the simplicity, the robustness, and the low computational cost. The value of JPE computed from two time series can be interpreted as a degree of correlation between the two time series [29].

3. Multiscale Procedures

To study entropy or cross-entropy measures of time series across scales, a multiscale procedure can be used. In this part we detail, in chronological order, three multiscale methods: The coarse-grained, the time-shift, and the composite coarse-grained approaches.

3.1. Coarse-Graining Procedure

In 2002 Costa et al. introduced the coarse-graining procedure to analyze the complexity, defined by the analysis of the irregularity through scale factors [14]. This method is an improvement, more adapted for a biological time series, of the coarse-graining procedure introduced by Zhang [30]. This procedure has been used in multiscale entropy and cross-entropy methods [6,9,15,20,31,32,33]. For each scale factor, this procedure derives a set of vectors illustrating the system dynamics. For a monovariate discrete signal of length N, the coarse-grained time series is calculated as: where is the scale factor and . The length of the coarse-grained vector is . An example of coarse-graining procedure is presented in Figure 1A.

Figure 1

Examples of multiscale procedures for the ten first points of a time series . (A) represents the coarse-graining procedure (modified from [34]), (B) shows the time-shift procedure, and (C) illustrates the composite coarse-graining procedure (modified from [35]).

3.2. Time-Shift Procedure

As for the coarse-grained procedure, the time-shift procedure is used to decompose a signal through different scale factors and to perform a multiscale analysis. While coarse-graining procedure uses the averaging of time series on several interval scales, the time-shift procedure applies time shifting in time series. The main disadvantage of a coarse-graining procedure is the loss of pattern information hidden in the time series. To overcome this limitation, Pham used the Higuchi’s fractal dimension (HFD) [36] and proposed a new multiscale analysis [37]. The time-shift procedure illustrates the fractal dimension of a signal. This method has been recently used with entropy and cross-entropy measures [17,37,38,39]. HFD shows stable numerical results for stationary, non-stationary, deterministic, and stochastic time series [40]. For a monovariate discrete signal of length N, the time-shift signal is calculated as:

For each time scale , time-shift time series are computed (). An illustration of the time-shift procedure is presented in Figure 1B.

3.3. Composite Coarse-Graining Procedure

The coarse-graining procedure, introduced by Costa et al. [14], increases the variance of estimated entropy values at large scale. To overcome this limitation, by Wu et al. introduced in 2013 a composite coarse-graining procedure [35]. This method has been used with entropy and cross-entropy measures [16,32]. For a monovariate discrete signal of length N, the k-th composite coarse-grained time series is computed as: where . For each time scale , k composite coarse-grained time series are computed (). An illustration of the composite coarse-graining procedure is presented in Figure 1C.

4. Multiscale Cross-Entropy Methods

4.1. Generalization

Multiscale cross-entropy (MCE) methods consist of applying a cross-entropy measure for each scale factor obtained by a specific procedure. For each scale factor , MCE is computed as: where and are computed with a multiscale procedure (see Section 3), k is the number of time series that are generated by the multiscale procedure ( for the coarse-graining procedure and for the time-shift and the composite coarse-graining procedures), and crossEn is the cross-entropy method used (see Section 2). Table 2 shows the multiscale cross-entropy methods that can be generalized with Equation (35). Before the computation of MCE, a pre-treatment can be performed. For example, the asymetric multiscale cross-SampEn (AMCSE) method [33] decomposes each signal into two, one for the positive trends and the other for the negative trends, before applying a coarse-graining procedure and cross-SampEn.

Table 2

Multiscale cross-entropy methods, in chronological order, that can be generalized with Equation (35). For each method, the multiscale procedure and the cross-entropy measure used and the reference are mentioned.

Method	Multiscale Procedure	Cross-entropy Measure	Reference
Multiscale cross-SampEn	Coarse-grained	cross-SampEn	Yan et al., 2009 [15]
Multiscale cross-ApEn	Coarse-grained	cross-ApEn	Wu et al., 2013 [31]
Asymetric multiscale cross-SampEn	Coarse-grained	cross-SampEn	Yin and Shang, 2015 [33]
Composite multiscale cross-SampEn	Composite coarse-grained	cross-SampEn	Yin et al., 2016 [16]
Multiscale cross-DistEn	Coarse-grained	cross-DistEn	Wang and Shang, 2018 [9]
Modified multiscale cross-SampEn
based on symbolic	Coarse-grained	MCSEBSS	Wu et al., 2018 [6]
representation and similarity
Modified multiscale cross-SampEn	Coarse-grained	mCSE	Castiglioni et al., 2019 [20]
Time-shift multiscale cross-SampEn	Time-shift	cross-SampEn	Jamin et al., 2019 [17]
Time-shift multiscale cross-DistEn	Time-shift	cross-DistEn	Jamin et al., 2019 [17]
Multiscale cross-trend SampEn	Coarse-grained	CTSE	Wang et al., 2019 [23]
Multiscale joint permutation entropy	Coarse-grained	JPE	Yin et al., 2019 [24]

4.2. Particular Cases

Some multiscale cross-entropy methods cannot follow the generalization previously introduced. In this part we detail three particular methods, in chronological order: The adaptive multiscale cross-SampEn, the refined composite multiscale cross-SampEn, and the percussion entropy.

4.2.1. Adaptive Multiscale Cross-Sample Entropy

The adaptive multiscale cross-sample entropy (AMCSE), introduced by Hu and Liang in 2011, assesses the nonlinear interdependency between different visual cortical areas [41]. The method uses the multivariate empirical mode decomposition (MEMD), introduced by Rehman and Mandic [42], to decompose two time series into intrinsic mode functions (IMFs) that represent the oscillation mode embedded in the data. For two time series and , AMCSE is calculated with the following three steps algorithm:

The MEMD on and is performed to obtain N IMFs;

The scales of data are computed in two directions, fine-to-coarse and coarse-to-fine , with the following two equations:

The two directions can be used separately or used in tandem to reveal the underlying dynamics of complex time series;

For each scale factor , the cross-SampEn (see Section 2.2.1) is applied between the two scales of data ( and ) extracted from vectors and .

4.2.2. Refined Composite Multiscale Cross-Sample Entropy

Yin et al. introduced in 2016 the composite multiscale cross-sample entropy (CMCSE) that follows the generalization (see Section 4.1), where the composite coarse-graining procedure and cross-SampEn are used [16]. The main disadvantage of this method is that cross-SampEn generates some undefined values when the number of matched sample is zero. To overcome this limitation, Yin et al. introduced the refined CMCSE (RCMCSE). This method leads to better results with short time series. For two times series and of length N, RCMSE is computed with the following three steps algorithm:

Coarse-grained time series are obtained with the composite coarse-graining procedure detailed in Section 3.3;

For a scale factor , the number of matched vector pairs, and , are calculated for all coarse-grained vectorsl

For each scale factor , RCMCSE is computed as: where m is the dimension and of the matched vector pairs and r is the distance tolerance for the matched vector pairs.

4.2.3. Percussion Entropy

Wu et al. introduced, in 2013, the multiscale small-scale entropy index () that is obtained by summing the values of entropy for the first five scale factors [43]. Percussion entropy, introduced by Wei et al. in 2019, allows one to quantify a percussion entropy index (PEI) [18]. The method has been introduced to assess baroreflex sensitivity. PEI compares the similarity in tendency of change between two time-series. This index has been compared to . For two time series and of length N, PEI is computed with the following three steps algorithm:

A binary transformation of and is used to obtain and :

The percussion rate for each scale factor is computed as: where m is the embedded dimension vectors and represents the match number between and ;

PEI is calculated as: where and is the number of scales to consider. Wei et al. [18] have chosen in accordance with .

This algorithm is a generalization of the method developed by Wei et al. [18] for a specific time series, amplitudes of successive digital volume pulse signals and changes in R-R intervals of successive cardiac cycles. At the moment, it has not been used to process other kinds of signals.

5. Conclusions

In this review we proposed a state-of-the-art of cross-entropy measures, multiscale procedures, and multiscale cross-entropy methods. Multiscale cross-entropy methods offer other interesting perspectives for time series analysis. Furthermore, all the cross-entropy methods, detailed in this review, can be translated into multiscale cross-entropy methods with the multiscale procedures presented in this review.