Medical Image Fusion Based on Sparse Representation and PCNN in NSCT Domain.

The clinical assistant diagnosis has a high requirement for the visual effect of medical images. However, the low frequency subband coefficients obtained by the NSCT decomposition are not sparse, which is not conducive to maintaining the details of the source image. To solve these problems, a medical image fusion algorithm combined with sparse representation and pulse coupling neural network is proposed. First, the source image is decomposed into low and high frequency subband coefficients by NSCT transform. Secondly, the K singular value decomposition (K-SVD) method is used to train the low frequency subband coefficients to get the overcomplete dictionary D, and the orthogonal matching pursuit (OMP) algorithm is used to sparse the low frequency subband coefficients to complete the fusion of the low frequency subband sparse coefficients. Then, the pulse coupling neural network (PCNN) is excited by the spatial frequency of the high frequency subband coefficients, and the fusion coefficients of the high frequency subband coefficients are selected according to the number of ignition times. Finally, the fusion medical image is reconstructed by NSCT inverter. The experimental results and analysis show that the algorithm of gray and color image fusion is about 34% and 10% higher than the contrast algorithm in the edge information transfer factor QAB/F index, and the performance of the fusion result is better than the existing algorithm.

1. Introduction

Medical imaging attracts more and more attention due to the increasing requirements of clinic investigation and disease diagnosis [1]. Imaging of different modalities can reflect different information about the lesion. For example, CT images have a clear bone image, but soft tissue imaging is blurred; MRI images can obtain multiangle and multiplane detail information of soft tissue, but the skeleton imaging is blurred; PET images can present metabolic activity of human cells, but the anatomical structure is not clear. Therefore, the medical images of different modalities are fused to improve the accuracy and recognition of lesion location, which provides a more effective imaging reference for clinical diagnosis of modern medicine [2].

Compared with wavelet transform and Contourlet transform, NSCT transform has the advantages of multiscale and multidirection analysis, and anisotropy and translation invariance [3]. However, the low frequency subband coefficients obtained by NSCT decomposition are not sparse, and the fusion of them directly is not conducive to maintain the characteristics of the source image. Sparse Representation (SR) can extract deeper structural features between low frequency subband coefficients and express or approximate it in a linear combination of a few atoms [4]. Compared with other artificial neural networks, PCNN has an incomparable advantage over other traditional artificial neural networks [5]. The PCNN model has global coupling and pulse synchronization, which can combine the input high frequency subband coefficients with human visual characteristics to obtain richer detail information [6]. Therefore, NSCT transform, sparse representation, and PCNN model image fusion method are gaining more and more attention. Chun-hui and Yun-ting [7] propose a fast image fusion algorithm based on sparse representation and nonsubsampled Contourlet transform. This algorithm greatly improves the efficiency of image fusion, but it gives only four-directional sparse representation in the low frequency subband, which cannot fully represent the characteristics and details of the source image. Shabanzade and Ghassemian [8] propose a multimodal image fusion algorithm based on NSCT and sparse representation. This algorithm uses sparse representation to perfectly approximate the low frequency subband coefficients. However, the rule based on the larger local energy or variance is used for high frequency subband coefficients, which cannot effectively solve the problem of image detail smoothing caused by sparse representation. Gong et al. [9] propose an image fusion method based on the improved NSCT transform and PCNN model, and this method can preserve the image structure better so that the fusion image is more in line with the human visual nervous system. However, the mutual information of the fusion image is relatively less. Mohammed et al. [10] propose a medical image fusion algorithm based on sparse representation and dual input PCNN model. This algorithm has a high fusion performance and adapts to the human visual nerve system. However, it needs to train medical image database to get an overcomplete dictionary; in addition, PCNN model applies a dual input, which presents the high complexity and low integration efficiency of the algorithm.

In order to obtain the medical fusion image with high fusion performance and high fusion efficiency, and to help it adapt to human visual nervous system, this paper, by aiming at the above research situation and existing problems and combining the sparse representation with PCNN simplified model, proposes the medical image fusion algorithm based on NSCT and SR-PCNN, hereinafter referred to as NSCT-SR-PCNN fusion algorithm.

2. Nonsubsampled Contourlet Transform

The NSCT transform consists of two steps: the nondownsampling pyramid (NSP) decomposition and the nondownsampling direction filter bank (NSDFB). NSP decomposition is the process of decomposing the source image into low and high frequency subbands through the nonsubsampling tower filter bank to ensure the characteristic of NSCT multiscale transformation. The NSPFB filter is decomposed as shown in Figure 1.

Figure 1

NSPFB filter decomposition.

NSDFB is a two-channel nondownsampling filter bank, and it decomposes the high frequency subband image decomposed by NSP in level-one NSDFB direction, so it can produce 2l different directional subband image. NSDFB filter is decomposed as shown in Figure 2.

Figure 2

NSDFB filter decomposition.

3. Sparse Representation

Sparse representation means that the natural signal can be represented or approximated by a linear combination of a small number of atoms in the overcomplete dictionary D ∈ R; then the sparse coefficient of the signal x can be obtained bywhere D is a prespecified dictionary; A is a sparse coefficient vector; ‖A‖0 stands for the count of nonzero entries in A; ε is the bounded representation error. NSCT-SR-PCNN algorithm uses K-SVD method to train the dictionary and uses the orthogonal matching tracking optimization (OMP) algorithm to estimate the sparse coefficient A [11].

4. Pulse Coupled Neural Network

PCNN simplified model is a feedback neural network model proposed by simulating the signal processing mechanism of cat visual cortex [12]. In the simplified model, the partial simplification of the parameters makes the generality of the model well guaranteed. However, there is a great difference in the response of the visual system to the different feature regions in the image. In the PCNN model, this difference is mainly reflected in the setting of the parameters, and the flexible changes in the parameters still affect the final fusion results. Therefore, this paper uses the most commonly used discrete mathematical iterative model. The simplified model is shown in Figure 3.

Figure 3

PCNN simplified model.

The mathematical expression of the PCNN simplified model can be expressed bywhere n is the number of iterations; I is the external input; Y is the output of neuron; U is the internal behavior of neurons; F is feedback input excitation; L is the input of neuron's link; W is the weight coefficient of the connection between neurons; β is the link strength coefficient; θ is the output of variable threshold function; V and α are, respectively, the signal amplification factor and attenuation time constant of neuron's link; V and α are, respectively, the signal amplification factor and decay time constant of variable threshold function.

5. Medical Image Fusion Algorithm Based on NSCT-SR-PCNN

NSCT-SR-PCNN algorithm firstly uses NSCT transform to decompose the source image after registration to obtain the low frequency and high frequency subband of the source image; secondly the fusion method based on sparse representation is used to fuse the low frequency subband, and the fusion method based on PCNN simplified model is used to fuse the high frequency subband; finally NSCT inverse transform is used to reconstruct the fused subband coefficients to obtain the medical image of fusion. The specific implementation process of NSCT-SR-PCNN medical image fusion algorithm is shown in Figure 4.

Figure 4

NSCT-SR-PCNN medical image fusion algorithm flow.

5.1. The Rules of Low Frequency Subband Coefficient Fusion

Low frequency subband coefficient fusion is achieved by using sparse representation fusion. First of all, blocks taken from the image to be fused form a training sample set, and secondly K-SVD algorithm is used to train a complete dictionary, and then the Batch-OMP [13] optimization algorithm is used to estimate the sparse coefficient; finally, the sparse coefficients are adaptively fused according to image features. The specific steps are as follows.

Step 1 .

Use the NSCT transforms to decompose, respectively, the source images A and B with M × N size after registration to obtain the low frequency and high frequency subband coefficients.

Step 2 .

Segment the low frequency subband coefficients L and L by using the sliding window with the steps of S pixels and the size n × n, and obtain the (N + n − 1)×(M + n − 1) subblocks; transform the image subblocks into column vectors to form the sample training matrix V and V.

Step 3 .

Average the sample training matrices V and V to obtain the mean matrices and ; average of the sample training matrices V and V is removed to obtain the sparse representation of the sample matrices V′ and V′.

Step 4 .

Use the K-SVD algorithm to iterate the sample matrix to obtain the overcomplete dictionary matrix D of the low frequency subband coefficient.

Step 5 .

Use the Batch-OMP optimization algorithm to estimate the sparse coefficients of V′ and V′ and obtain the sparse coefficient matrices α and α. According to the value of L1 norm, the sparse coefficient matrix of column i is fused by applying the rules of

Among them, α can be seen from

Step 6 .

The choice of the fusion mean matrix is given by

Step 7 .

Multiply the overcomplete dictionary matrix D with the fusion sparse coefficient matrix α and then add fusion mean matrix . The fusion sample training matrix V is given by

Step 8 .

Convert the columns of the fusion sample training matrix V into data subblocks, and reconstruct the data subblocks to obtain the fusion coefficients of the low frequency subbands.

The implementation of low frequency subband coefficients fusion based on sparse representation is shown in Figure 5.

Figure 5

Low frequency subband coefficient SR fusion process.

5.2. The Rules of High Frequency Subband Coefficient Fusion

According to the characteristics of human visual system, the spatial frequency (SF) reflects the local area characteristics and details of the image. The high frequency subband coefficient fusion selects SF as the neuron feedback input to stimulate the PCNN simplified model. The neuron feedback input is expressed by

Among them are the window size 3 × 3, RF, and CF; from formula (8) we can see:

In the PCNN model, the value of β determines the strength of the coupling relationship of the neurons, and the high frequency subband coefficient fusion selects the Laplacian energy (EOL), the visibility (VI), and the standard deviation (SD) that can measure the neighborhood characteristic information, respectively, as the linking strength values of PCNN corresponding neurons, and EOL, VI, and SD are expressed bywhere (u, v) is a pixel point of the image; f(u, v) is the pixel value; w is the window of size l × l; m is the pixel gray level average; N is the number of pixels in the window; α is a constant.

For fusion based on PCNN simplified model, SF is used as the neuron feedback input to excite each neuron, and EOL, VI, and SD are selected as the linking strength values of the corresponding neurons; then the corresponding ignition map is obtained by the PCNN ignition, and the new ignition map of the source image is constructed by the weighting function. Finally, the fusion coefficient is selected according to the number of ignition frequencies. Specific implementation steps are as follows.

Step 9 .

According to formula (7), calculate the neighborhood spatial frequency SF and SF of the high frequency subband coefficients HA and HB and then normalize SF and SF, and mark them as SF′ and SF′, respectively; SF′ and SF′ are used as neuron feedback input to motivate the PCNN simplified model.

Step 10 .

According to formula (9), calculate EOL, VI, and SD of high frequency subband coefficients HA and HB (which is recorded as β, β, β, β, β, and β) and take them, respectively, as the linking strength value of corresponding neurons.

Step 11 (initialization setting).

L (0) = U(0) = 0, θ(0) = 1; at this time the neuron is in the flameout state; that is, Y(0) = 0, so the number of pulses generated is O(0) = 0.

Step 12 .

According to formula (2), calculate L[n], U[n], θ[n], and Y[n].

Step 13 .

The output of the PCNN simplified model iteration run is as follows: O, O, O, O, O, and O; use the weighting function to obtain the new ignition map O and O which corresponds to high frequency subband coefficients HA and HB: O = w1O + w2O + w3O, O = w4O + w5O + w6O; w  (i = 1,2, 3,4, 5,6) is given by

Step 14 .

Compare the ignition time output threshold values (ignition frequencies) at the new ignition map pixel; the high frequency subband fusion coefficient H(i, j) is given by

The adaptive fusion implementation process based on PCNN simplified model is shown in Figure 6.

Figure 6

High frequency subband coefficient PCNN fusion process.

6. The Results and Analysis of Experiments

In order to verify the effectiveness of the proposed algorithm, five kinds of contrast algorithms are selected to conduct gray and color medical image fusion experiments. The medical images of each group are obtained from http://www.med.harvard.edu/AANLIB/home.html page. Objective evaluation of quality is made in terms of 7 indexes, such as the information entropy (IE), spatial frequency (SF), mean gradient (AG) [14], clarity (MC), mutual information (MI), standard deviation (SD) [15], and edge information delivery factor (QAB/F high weight evaluation index) [16-19]. The visual information fidelity (VIFF) and structural similarity model (SSIM) were used to evaluate the visual effect of human eyes. The contrast algorithm 1 is a medical image fusion study based on NSCT transform (referred to as NSCT fusion algorithm) proposed in the paper [20]. The contrast algorithm 2 is the multifocus image fusion (referred to as the SR fusion algorithm) based on the fragmented complete sparse representation proposed in the paper [21]. The contrast algorithm 3 is the image fusion by using pulse coupled neural network (referred to as PCNN fusion algorithm), proposed by the paper [22]. The contrast algorithm 4 is a multifocus image fusion based on NSCT and sparse representation (referred to as NSCT-SR fusion algorithm) proposed in the paper [11]. The contrast algorithm 5 is an improved algorithm based on NSCT and adaptive PCNN medical image fusion (referred to as NSCT-PCNN fusion algorithm) proposed in the paper [23]. NSCT transformation parameters setting: the class of decomposition is 4, the scale decomposition filter selects “pyrexc” filter, and the direction filter selects “vk” filter [24]; SR parameters setting: sliding window size is 8 × 8, step size is 1, the number of dictionary training iteration is 30 times, and sparse error is 0.01; PCNN parameters setting: V = 1, V = 20, α = 1, α = 0.2, Nmax = 100, and(see [25]).

6.1. Gray Image Fusion Experiment

Gray image fusion experiment is conducted by selecting the images of four groups of brain under different states as the images to be fused. The results of the fusion of the various algorithms are shown in Figures 7–10, and the objective evaluation indexes for quality of the various algorithms are shown in Tables 1–4.

Figure 7

CT/MRI medical image fusion results.

Figure 8

MR-PD/MR-T1 medical image fusion results.

Figure 9

MR-PD/MR-T2 medical image fusion results.

Figure 10

MR-T1/MR-T2 medical image fusion results.

Table 1

Quality evaluation of CT/MRI medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.1312	0.1498	0.1000	0.3965	0.0620	0.1634
MC	2.6621	2.3780	2.9962	2.3718	2.4343	2.4396
MI	3.4885	2.8146	1.8082	2.1005	1.7618	2.2426
SD	56.6060	37.9078	59.5349	44.0044	46.2033	55.4061
Q AB/F	0.6572	0.5441	0.4020	0.6794	0.3502	0.6899
VIFF	0.6598	0.5688	0.3922	0.7074	0.4201	0.6677
SSIM	0.9385	0.8775	0.6674	0.8520	0.7896	0.9567

Table 2

Quality evaluation of MR-PD/MR-T1 medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9999	0.9904	0.9579	0.9922	0.9999	0.9963
MC	2.2233	2.4200	4.3005	2.4234	2.7476	2.6054
MI	3.0047	2.4785	2.1328	2.3858	2.2826	2.5474
SD	54.6893	45.1990	56.0878	45.4664	46.5353	51.4444
Q AB/F	0.5648	0.5669	0.4345	0.5399	0.4205	0.5887
VIFF	0.5722	0.5675	0.3408	0.4964	0.4228	0.5991
SSIM	1.4165	1.5332	0.9745	1.4869	1.3444	1.4279

Table 3

Quality evaluation of MR-PD/MR-T2 medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9730	0.9768	0.9928	0.9777	0.9859	0.9768
MC	2.5614	2.5205	3.0647	2.4164	2.4802	2.5033
MI	2.3808	2.7229	2.3961	2.5019	2.4154	2.5378
SD	56.4522	46.3235	59.0546	47.5818	52.1596	56.6857
Q AB/F	0.6120	0.6491	0.4912	0.6385	0.4612	0.6530
VIFF	0.7091	0.7528	0.5534	0.6966	0.6267	0.8150
SSIM	1.6297	1.6524	1.5168	1.6552	1.6306	1.7148

Table 4

Quality evaluation of MR-T1/MR-T2 medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9936	0.9907	0.9979	0.9920	0.9974	0.9956
MC	2.7955	2.6224	3.2850	2.6689	2.8966	2.8338
MI	1.8604	2.4779	2.1623	2.2494	2.0621	2.2047
SD	53.9478	47.4264	72.9638	49.5285	54.9129	58.7637
Q AB/F	0.4321	0.5076	0.4357	0.5754	0.4007	0.5732
VIFF	0.3003	0.4529	0.3316	0.3400	0.3400	0.4769
SSIM	1.2341	1.3590	1.2475	1.2379	1.2379	1.2812

The NSCT-SR-PCNN algorithm has better fusion and better fusion performance than the five contrast algorithms, from the human visual effects in Figures 7–10 or the evaluation index from Tables 1–4. The reason is that, for NSCT algorithm, NSCT decomposition of the low frequency subband coefficient is not sparse, and direct low frequency coefficient fusion is not conducive to the retention of the source image features; for SR algorithm and the PCNN algorithm, the image fusion is based on the spatial domain implementation, and the spatial domain fusion method fails to express details, so the fusion image has low contrast, fuzzy details, and block artifact, and other problems. For NSCT-SR algorithm, it solves the problem that low frequency subband coefficient is not sparse, but the fusion of high frequency subband is only conducted based on the direction featured principle, which cannot completely present the details of the image information. For NSCT-PCNN algorithm, it can adapt to the human visual system, but the low frequency subband coefficients have no sparseness. For NSCT-SR-PCNN algorithm, it not only solves the problem of the detail loss of the wavelet transform and the sparseness loss of low frequency subband coefficient of the NSCT, but also improves the comprehensive performance of the fusion results by using the spatial frequency of the high frequency subband coefficients to impel input and by using EOL, VI, and SD to strengthen their links with the corresponding neurons. NSCT-SR-PCNN algorithm can obtain better performance for CT/MRI, MR-PD/MR-T1 and MR-T1/MR-T2 medical image fusion in the comprehensive analysis of evaluation indexes. The NSCT-SR-PCNN algorithm can achieve better performance for CT/MRI, MR-PD/MR-T1 and MR-PD/MR-T2 medical image fusion according to edge information delivery factor of QAB/F index.

6.2. Color Image Fusion Experiment

Color image fusion experiment selects three groups of brain under different images as the images to be fused. The fusion results of various algorithms are shown in Figures 11–13, and the objective evaluation index for quality of various algorithms is shown in Tables 5–7.

Figure 11

MR-PD/PET medical image fusion results.

Figure 12

MR-T1/PET medical image fusion results.

Figure 13

MR-T2/PET medical image fusion results.

Table 5

Quality evaluation of MR-PD/PET medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9977	0.9761	0.9677	0.9791	0.9954	0.9801
SF	6.0266	5.7013	6.9747	5.8537	5.7295	5.9952
AG	4.8934	4.9109	6.9312	4.8133	4.2310	4.9744
MC	1.9402	1.7960	2.5601	1.8627	1.8753	1.9409
MI	2.7632	2.7705	2.6597	2.6677	2.6979	2.7879
SD	70.2719	50.0841	70.9987	50.8343	63.5030	67.2682
Q AB/F	0.5414	0.5411	0.4311	0.5295	0.3908	0.5718

Table 6

Quality evaluation of MR-T1/PET medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9992	0.9879	0.9755	0.9897	0.9975	0.9923
SF	6.6819	6.9951	7.9528	6.9423	6.7408	6.9475
AG	6.5924	8.0276	9.9589	7.5923	6.2566	7.5961
MC	2.2740	2.3869	3.0047	2.3844	2.4070	2.4067
MI	2.5493	2.7738	2.5631	2.7455	2.5485	2.7559
SD	70.8814	60.3674	77.6841	62.4699	63.5939	67.0245
Q AB/F	0.4333	0.5505	0.4521	0.5611	0.3908	0.5681

Table 7

Quality evaluation of MR-T2/PET medical image fusion.

Evaluation index	NSCT	SR	PCNN	NSCT-SR	NSCT-PCNN	NSCT-SR-PCNN
IE	0.9953	0.9697	0.9622	0.9734	0.9935	0.9769
SF	6.7434	6.9677	8.0595	6.6804	6.7986	6.9365
AG	6.7453	8.0305	10.3970	7.1794	6.0200	7.4678
MC	2.2195	2.2513	3.0121	2.1432	2.2898	2.3136
MI	2.5588	2.7566	2.5835	2.6558	2.6199	2.6076
SD	70.6653	54.3247	82.1519	56.6147	63.2384	65.1174
Q AB/F	0.3985	0.5125	0.3832	0.5449	0.3166	0.5451

Compared with the five contrast algorithms in terms of the human visual effect of Figures 11–13 or the comprehensive analysis of the evaluation index and the QAB/F index of the edge information delivery factor, the NSCT-SR-PCNN algorithm can provide better performance for MR-PD/PET, MR-T1/PET, and MR-T2/PET medical image fusion. Based on the experimental data, the NSCT-SR-PCNN algorithm proposed in this paper can make the fusion image obtain high fusion performance in the aspect of texture clarity, gray scale variation, and contrast ratio and can realize no color loss or distortion in transmission.

7. Conclusion

The NSCT-SR-PCNN algorithm effectively combines NSCT transform, sparse representation, and pulse coupled neural network to overcome the shortcomings of wavelet transform, which cannot reflect the holistic characteristics, and solves the problem that the low frequency subband coefficient is not sparse. In addition, this algorithm collects the image texture, the degree of change in edge and details, and other information, which improves the comprehensive performance of the fusion results. The experimental data show that although not all the evaluation indexes of NSCT-SF-PCNN algorithm rank the first, the evaluation indexes of the NSCT-SF-PCNN algorithm are all in the top three and the top four; besides this, the comprehensive indexes are the number one, and the edge information delivery factor QAB/F of the high weight evaluation index is higher than that of the five contrast algorithms, and the edge and detail information of the source image is better preserved, and the human visual effect is better. Certainly, the NSCT-SR-PCNN algorithm also needs to be improved. For example, through the online dictionary learning method, to obtain a complete dictionary D still requires further study.