Capacity of very noisy communication channels based on Fisher information.

We generalize the asymptotic capacity expression for very noisy communication channels to now include coloured noise. For the practical scenario of a non-optimal receiver, we consider the common case of a correlation receiver. Due to the central limit theorem and the cumulative characteristic of a correlation receiver, we model this channel noise as additive Gaussian noise. Then, the channel capacity proves to be directly related to the Fisher information of the noise distribution and the weak signal energy. The conditions for occurrence of a noise-enhanced capacity effect are discussed, and the capacity difference between this noisy communication channel and other nonlinear channels is clarified.

It is well known that, for an additive Gaussian noise channel and an energy constrained input signal, the channel capacity can be explicitly calculated1234. In practical applications, however, communication systems frequently encounter non-Gaussian noise environments, for instance, underwater acoustic noise and low-frequency atmospheric noise567. Of all channels with power-constrained noise, the capacity of a Gaussian channel is the smallest1234. Thus, the capacities of non-Gaussian channels are of great interest56789101112131415. Moreover, from theoretical and practical viewpoints, a very interesting topic is the investigation of the channel capacity with very weak input signals, e.g. deep space communication channels29 and qubit depolarizing channels15. A very noisy channel was introduced by Reiffen8, and extended by Gallager2 and Majani9 to model many physical communication channels operating at very low signal-to-noise ratio (SNR). “Very noisy” channels with very low capacity are of significant interest to communications, since Shannon’s theorem guarantees reliable communication as long as the capacity is nonzero1234916. Following the approaches developed in289 and using a power series of characteristic functions, Nirenberg5 derived a simple formula of the capacity for the coherent threshold channel with an optimum receiver. For memoryless channels with very weak inputs, Kullback10, Verdú11 and Prelov12 explicitly expressed the asymptotic expressions of the channel capacity closely related to the Fisher information matrix. Recently, Kostal and Lansky14 presented an approximate expression for the information capacity in a broad class of discrete-time channels under the constraint of vanishing input amplitude or power, which allows us to analyse the capacity of channels with memory in a convenient way1314.

In this paper, under the assumption of low SNR, we will further derive the capacity of a very noisy communication channel, wherein the optimum receiver may be unavailable and noise is not restricted to be white. Based on the central limit theorem, we argue that, for sufficiently large observation times and with the constraint of weak signal energy, the receiver output tends to be Gaussian distributed, and the channel capacity is then computed by a simple formula being directly related to the Fisher information of the noise distribution. We demonstrate the enhancement of capacity via stochastic resonance will not occur in very noisy communication channel with an optimum receiver, but it can occur with generalized correlation receivers suited for practical implementation. Finally, we compare the asymptotic capacity expressions of this noisy communication channel with other capacity formulas in refs 10, 11, 12, 13, 14.

Results

Channel capacity for coloured noise

For the M-ary communication channel shown in Fig. 1, the observation data vector X contains the additive noise vector Z and the signal vector S, . With the assumptions of white noise and very low SNR, Nirenberg5 derived the capacity for the coherent threshold channel with an optimum receiver. We briefly present the conclusions of ref. 5 for reference (see Methods). However, the idealized assumption of white noise is unpractical, and the coloured noise has practical significance234. We here further derive a general asymptotic expression of the channel capacity for coloured noise, which applies to not only the optimum receiver but also an arbitrary correlation receiver.

Figure 1

Mutual information I(Φ, Ω) of the communication channel and I(Φ, Ψ) of the nonlinear channel.

In the case of coloured noise and for very low SNR, the conditional probability function can be expanded to the first order

where the operator and the statistic . Here, from an information theory point of view by Reiffen8 and Gallager2, the module indicates the channel is very noisy in the sense that the channel output is almost independent of the input. For M equiprobable signals S, the receiver takes the maximum likelihood rule

to optimally choose mth signal517. Substituting equation (1) into equation (2), the optimum receiver

enables us to decide if the mth signal was transmitted. For clarity, we state that the statistic Γ(s, x) and the maximum likelihood decoding rule of equation (2) compose an optimum correlation receiver. The channel output is the decoding signal ω of the receiver, as shown in Fig. 1.

Then, supposing the zero-mean E(s) = 0 and extending the very noisy vector channel 2589, the mutual information between the input signal space Φ and the channel output space Ω is given by

where the Fisher information matrix of the noise distribution is defined as37

It is noted that J(f) is also called the Fisher information of a location parameter or the shift-invariant Fisher information36718, which can be viewed as a special case of the Fisher information measuring the statistical information contained in data about an unknown parameter. Therefore, with the energy constraint of E(ss) ≤ ε and for the standardized vector , the channel capacity can be expressed as

where Λ is the largest eigenvalue of the matrix J(f) and u takes the corresponding eigenvector.

For positive definite matrixes A, and an arbitrary column vector , the inequality X(A − B)X ≥ 0 is abbreviated as . Then, for the positive semidefinite matrix

and the noise covariance matrix ∑ = E(zz), we have

where the equality occurs for N-dimensional Gaussian distribution with its Fisher information matrix . Thus, equation (7) indicates the maximum eigenvalue of is less than that of Fisher information matrix of non-Gaussian noise. This result extends the conclusion of equation (36) by Nirenberg5, and also confirms that, in terms of the channel capacity, zero-mean Gaussian noise is the worst case given that the noise vector has a fixed covariance matrix34.

However, we note the channel capacity of equation (6) is achieved by the optimum receiver of equation (3). In many practical cases, the optimum receiver may be not implementable for the unknown noise distribution or the non-closed form of distributions (e.g. α-stable noise19). Thus, we further consider the generalized correlation receiver

where the coefficient vector and the function g(x) is not restricted to be memoryless. For the zero-mean vector of E[g(z)] = 0 (for a shift in mean)6 under f and for very low SNR, g(x) can be expanded to the first-order

Then, for a large observation size N, the statistic T has the mean and the variance . Using the Cholesky decomposition of the symmetrical matrix V = E[g(z)g(z)] = LL, the output SNR of the receiver can be calculated as

by optimally choosing . Then, we argue that, for sufficiently large observation times and with the constraint of weak signal energy, the receiver output tends to be Gaussian distributed, and the capacity can be approximately calculated as

where Λ is the largest eigenvalue of the matrix . Observing

and for the positive semidefinite matrix

we have

with and the equality occurring for . This inequality (13) indicates that the eigenvalue Λ of J(f) is not less than the eigenvalue Λ of the matrix . Therefore, based on equations (6), (11) and (13), we find

which extends the conclusion of ref. 5 to the case of coloured noise. In addition, the equality in equation (13) also demonstrates the receiver of equation (8) is optimal when , i.e. the optimum receiver of equation (3).

We argue that the asymptotic capacity expression of equation (11) has a broader applicability for an arbitrary correlation receiver operated in coloured or white noise environments. As a simple check for the consistency of the results from equation (11) to equation (14), we consider the case of white noise. Immediately, due to the statistical independence of g(z), the expectation matrices and V = E[g2(z)]I. Here, the derivative g′(z) = dg(z)/dz and I is the unit matrix. Therefore, the matrix in equation (11) has N identical eigenvalues , and the channel capacity becomes

where the eigenvalue Λ = J(f) corresponds to the Fisher information matrix J(f) in equation (6). Using the Cauchy-Schwarz inequality and integration by parts , and the equality in equation (15) occurs when that specifies the optimum receiver in the presence of white noise5.

Conditions for noise-enhanced capacity

Since the emergence of the concept of stochastic resonance20, the employment of noise in enhancing the performance of nonlinear systems has become an interesting option131421222324252627282930313233343536. Initially, the mechanism of stochastic resonance manifests itself as a time-scale matching condition for the noise-induced characteristic time of systems and the signal period2027. Later, the notion of stochastic resonance has been widened to a number of different mechanisms, e.g. aperiodic stochastic resonance22 and suprathreshold stochastic resonance31. For such stochastic resonance effects2231, there is no matching time-scale that corresponds to the input aperiodic or information-carrying random signal, but the system performance still reaches a maximum at an optimal non-zero noise level. Therefore, the noise-enhanced effect, instead of stochastic resonance, becomes a more appropriate term for describing the enhancement effect of system responses via the addition of noise. Here, if the channel capacity reaches a maximum at an optimal non-zero noise level, then the noise-enhanced capacity effect occurs. Otherwise, upon increasing the noise level, the channel capacity monotonically decreases, this is to say, the noise-enhanced capacity effect does not exist.

There are two approaches for varying the noise in stochastic resonance. One is tuning the noise level but not changing the noise type, and the other is adding extra noise to a given noisy signal, while the extra noise type may be different form the original one. Next, we will demonstrate the occurrence or nonoccurrence condition of the noise-enhanced capacity effect by the above mentioned methods.

First, we will prove that no noise-enhanced capacity effect exists for tuning the scaled noise level in an optimum receiver. For the scaled noise vector Z = DZ, the covariance matrix ∑ can be factored as ∑ = DD and the standardized noise vector Z has a covariance matrix being the unit matrix 7. A well-known scaling property of the Fisher information matrix is71837383940

which implies the largest eigenvalue Λ of J(f) is a monotonically decreasing function of Λ/det(∑) for the determinants det2(D) = det2(D) = det(∑). Here, the largest eigenvalue of is Λ that is a fixed quantity for Z. For such a channel with its optimum receiver, equation (11) indicates the channel capacity monotonically decreases as the noise intensity increases. Thus, no noise-enhanced capacity phenomenon will occur by tuning the noise level.

For instance, we consider a threshold receiver based on the function g(x) = sign(x) and the Laplacian white noise with its distribution . We note that the threshold receiver is optimum for the Laplacian noise, and satisfies the equality condition in equation (15). In this case, the channel capacity in equation (15) can be calculated as , which monotonically decreases as the noise level σ increases. Thus, there is no noise-enhanced capacity effect.

Secondly, we usually have a given signal corrupted by noise, and the initial noise level is unadjustable. We will prove that the addition of extra noise cannot further improve the channel capacity achieved by the optimum receiver. Under this circumstance, we add an extra noise vector W, independent of Z and S, to the observation X, and the updated data vector is

where the composite noise vector U = Z + W with its distribution f. In this case, we should employ the statistic to specify the optimum receiver, and the corresponding capacity is then given by

with the largest eigenvalue of the Fisher information matrix J(f). For any nonsingular matrix , the Fisher information matrix inequality337383940 holds for

we then find the largest eigenvalue Λ of J(f) is not less than the largest eigenvalue of J(f) and

This result of equation (20) clearly shows that stochastic resonance cannot further improve the channel capacity achieved by the optimum receiver, regardless of adding white or coloured noise vector W.

Thirdly, we note that the above two negative conditions of the noise-enhanced capacity effect arise with the optimum receiver matched to the distribution of the background noise. By Contrast, if the generalized correlation receivers of equation (8) are not optimal for the background noise, stochastic resonance may play an important role in the enhancement of capacity. For example, we consider non-scaled Gaussian mixture noise vector W with its distribution6212833

where the variance and parameters μ, ζ ≥ 0. A useful coloured noise model of the first-order moving-average41 as

where the correlation coefficients are ρ1,2 and is an independent identically distributed (i.i.d.) random vector. For small values of ρ1,2 , the dependence among noise samples Z will be weak41. The signum function g(x) = sign(x) is adopted to construct the generalized correlation receiver of equation (8), which is not optimal for the coloured noise Z. The optimum receiver indicated in equation (3) for the coloured noise Z is rather complicated, since the distribution f does not have a tractable analytic expression41. Using the approach developed in ref. 41, we have the expectation matrix

with the unit matrix I and , and the matrix V becomes tridiagonal with elements

for , and other elements are higher-order infinitesimal of ρ1 + ρ2 . Then, we calculate the largest eigenvalue of the matrix as

where the error function . In Fig. 2, we show the capacity per signal energy C/ε = Λ/2 in equation (11) versus the noise parameters μ and ζ in equation (21). Here, the correlation coefficient ρ1 = 0.2 and ρ2 = 0 in the coloured noise model of equation (22). We regard the parameters ±μ as the peak locations of the Gaussian mixture distribution in equation (21), while the parameter ζ as the noise level. It is then clearly shown in that Fig. 2, upon increasing ζ for a fixed value of μ (the noise variance also increases), the noise-enhanced capacity effects exist. The corresponding maxima of C/ε versus optimal values of ζ are also marked by squares in Fig. 2.

Figure 2

Stochastic resonance effect of the capacity per signal energy C/ε = Λ/2 in equation (11) versus the noise parameters μ and ζ in equation (21).

Here, the correlation coefficient ρ1 = 0.2 and ρ2 = 0 in the coloured noise model of equation (22). The corresponding maxima of C/ε versus optimal values of ζ are also marked by squares.

We emphasize that the above noise-enhanced capacity effect is an illustrative case of stochastic resonance that exists for a suboptimal receiver not matching the background noise. However, this mismatch condition is not the decision criteria for the occurrence of the noise-enhanced effect, since the example illustration is under the assumptions of a small signal and a correlation receiver with a large observation size. Beyond these restrictive assumptions, the noise-enhanced effect has been frequently observed21242528293031. For instance, the noise-enhanced effect has been demonstrated for non-weak signals in threshold neurons252931, where an optimal matching condition is inapplicable to the neuronal model immersed in complex noisy environments. It is sufficiently recognized that a well-established criterion for the noise-enhanced effect is to observe an optimal noise level at which the system response can be optimized.

Discussion

In this paper, we analyse the capacity of a very noisy communication channel with correlation receivers. With the weak signal energy constraint and for very low SNR, we generalize an asymptotic expression of capacity achieved by the optimum receivers in a coloured noisy environment. Moreover, for the case when the optimum receiver is unavailable in practice, a capacity formula is presented for the communication channel with a generalized correlation receiver. We further discuss the occurrence condition of the noise-enhanced capacity effect in the considered communication channel.

A similar asymptotic expression of capacity is also obtained in memoryless1011 or memory additive-noise channels121314. We emphasize the asymptotic capacity expressions of equations (6) and (11) are different from that in previous literature1011121314. In Fig. 1, for the channel output Y = g(X), these studies assume the conditional probability density as f(y|s). Then, the Fisher information matrix is defined as1011121314

with the operator . Then, for the zero-mean signal vector E(s) = 0 and the weak signal energy ε, the mutual information between the input space Φ and the output space Ψ is approximated as1011121314

which is different from the mutual information I(Φ, Ω) of equation (4) based on the Fisher information matrix J(f) of the noise distribution f. It is shown in Fig. 1 that the receiver multiplies nonlinear transformation g(x) with optimized coefficients, and obtains a cumulative statistic T that decides whether the mth signal S is sent or not. Then, the considered communication channel chooses an optimal signal S from the signal space to maximize the average mutual information. Since the receiver collecting the weighted nonlinear outputs as the statistic , and for any nonlinear function g, the distribution of T tends to be Gaussian. This leads to the asymptotic expressions of capacity of equations (6) and (11). We recognize the asymptotic capacity expressions in equations (6) and (11) have application in the context of a very noisy communication channel with a correlation receiver. As a new analytical result of the channel capacity, it has theoretical significance and deserves some exposition.

We also note that, for the linear transfer function of Y = Z + S, the conditional probability density f(y|s) = f(y − s), the Fisher information matrix of equation (27) becomes

where the differentiation operator with respect to S is equivalent to differentiation with respect to Z3. Therefore, for the linear additive-noise channel, the considered communication channel has the same capacity as that denoted in refs 10, 11, 12, 13, 14.

Besides a linear channel capacity defined and calculated by Shannon1, only a few analytical results exist for a variety of different nonlinear channel models. We argue that our asymptotic capacity expression for a nonlinear channel may be valuable for practical channels and coding techniques developed for communication applications in order to approach the established linear Shannon limit, and deserves further extensive study. We here only consider a single correlation receiver for detecting the weak signal, however recent studies in general provide evidence that, besides an optimal noise intensity, an optimal network configuration exists, at which the best system response can be obtained22314243444546. Thus, an interesting extension for future work is to investigate the capacity of a very noisy communication channel with receivers connected in various network configurations.

Methods

Very noisy communication channel model

Consider a coherent M-ary communication channel transmitting M possible signals S for , as shown in Fig. 1. In an interval, the observation vector

where contains the noise vector and the signal vector . Then, a receiver multiplies the transformation g(X) with optimized coefficients, resulting in a cumulative statistic T(X) for deciding whether the mth signal S is sent or not. The capacity C of a communication channel is given by the maximum of the mutual information I(Φ, Ω) between the input signal space Φ and the channel output space Ω

where the maximization is with respect to the input distribution f over the signal space Φ12345.

Nirenberg’s approach for white noise

The white noise Z has the multivariate distribution with zero-mean and variance . Let the statistically independent signal components be constrained to satisfy , and the total signal energy has a constraint . Then, for very low SNR of , the conditional probability density can be approximated as

with the first two terms of Taylor series. Here, and the statistic 5. Using the maximum likelihood rule17, the conditional probability density on the knowledge that the mth signal satisfies

which leads to the optimum receiver

to decide if the mth signal was sent.

To simplify the mathematical manipulations, Nirenberg5 assumes the even noise distribution function f(z) = f(−z) and a very noisy channel 289 yielding the mutual information between the output space Ω and the input signal space Φ as

where the Fisher information of the noise density f and the expectation . Since the same bias does not affect the decision of inequality of equation (34), it may be conveniently assumed to be zero5. Then, over the class of signal distributions f, the channel capacity is computed as5

which is applicable to various white noise types.

Furthermore, for a fixed noise variance and an arbitrary noise density function f, 17, where the equality occurs for Gaussian distribution with its Fisher information 71837. Accordingly, the additive Gaussian noise channel is the worst one, and has the minimum capacity, as indicated in equation (36). It is well known that, for very low SNR of , the capacity of Gaussian vector channel is approximately calculated as234

which accords well with equation (36)5.

Additional Information

How to cite this article: Duan, F. et al. Capacity of very noisy communication channels based on Fisher information. Sci. Rep. 6, 27946; doi: 10.1038/srep27946 (2016).