Improving Empirical Mode Decomposition Using Support Vector Machines for Multifocus Image Fusion.

Empirical mode decomposition (EMD) is good at analyzing nonstationary and nonlinear signals while support vector machines (SVMs) are widely used for classification. In this paper, a combination of EMD and SVM is proposed as an improved method for fusing multifocus images. Experimental results show that the proposed method is superior to the fusion methods based on à-trous wavelet transform (AWT) and EMD in terms of quantitative analyses by Root Mean Squared Error (RMSE) and Mutual Information (MI).

Introduction

Due to the limited depth-of-focus of optical lenses, cameras cannot be focused simultaneously on all objects at different distances from them to gain a clear image [1]. On the other hand, many pattern-related processing tasks, such as machine vision and target tracking, are better implemented using focused images rather than defocused ones [2]. Therefore, it is often advantageous to construct an image with every object in focus using image fusion methods by fusing the multifocus images taken from the same view point under different focal settings [3].

Up to now, various methods at pixel, feature or decision levels have been presented for image fusion [3-5]. Arithmetic algorithms at the pixel level often cause undesirable side effects such as reduced contrast [6]. Another alternative approach using image block and spatial frequency suffers from a tradeoff between block size and the quality of the fused image. That means using a large image block will lead to a less clear image while using a small image block may lead to saw-tooth effect [7].

Another family of methods has been explored based on undecimated ′à-trous′ wavelet transform (AWT) [8], [9]. The basic idea is to implement an AWT on each multifocus image, and then fuse all wavelet coefficients by their magnitudes to produce one composite wavelet representation, from which the focused image can be recovered by performing the inverse AWT (IAWT) [2].

Empirical mode decomposition (EMD) is a more recent signal processing method for analyzing nonlinear and nonstationary data, which was developed by Huang et al. [10, 11]. The final representation of signal is an energy-frequency distribution that gives not only sharp identifications of salient information but also other “smoother ” part of the signal. The EMD is a highly efficient and adaptive method and offers higher frequency resolution and more accurate timing of nonlinear and nonstationary signal events than traditional integral transform techniques [12-16]. In this paper, a combination of EMD and support vector machines (SVMs) is proposed to produce a better EMD representation of the fused image from fusing multifocus images.

The SVM is a supervised classification method that outperforms many conventional approaches in many applications [2]. The improvement of the EMD based multifocus image fusion using the SVM is presented in Section 2. An experiment in Section 3 illustrates that the presented method produces the focused images better than the traditional fusion methods based on EMD and AWT quantitatively.

Fusion Principle

Here, the processing of two images A and B is considered, though the algorithm can be extended to handle more than two. Each multifocus image is firstly decomposed by EMD into one residue and a series of intrinsic mode functions (IMFs). Then a SVM is trained to determine which IMF plane is clearer at each location at each level. In the end, the focused image is recovered by carrying out the inverse EMD (IEMD).

EMD-based multifocus image fusion using the SVM

The EMD can represent the details and smooth part of an image and this framework is well suitable to fuse images by managing different IMFs [12-17]. For a two-dimensional image, the EMD process that generates the IMFs is summarized as follows [12]:

Treating the original image I as the initial residue I0.

Connecting all the local maxima and minima along rows using constructed smooth cubic splines to get upper envelope uer and lower envelope ler. Similarly, upper envelope uec and lower envelope lec along columns are also obtained. The mean plane ul is defined:

Then, the difference between I0 and ul is

This is one iteration of the sifting process. Because the value of ul decreases rapidly for the first several iterations and then decreases slowly, this suggests that the appropriate number of iterations can be used as the stopping criterion. Hence, the appropriate number of iterations to build IMFs is used in this paper. This sifting process is ended until ω1 becomes an IMF. The residue is obtained by:

Treating the residue as the new input dataset. A series of {ω}1≤≤ is obtained by repeating 2) untilI is a monotonic component (J denotes the decomposition levels). I can be recovered by IEMD:

Figure 1 shows one example of the EMD. The original image is downloaded from http://www.inria-lpes.fr/is2/people/pgoncalv(accessed in April 2007). The original image contains three kinds of patterns. The two modes and the residue show that patterns vary in size from the smallest to the largest.

Figure 1.

(a) the original image; (b) IMF1; (c) IMF2; (d) the residue.

Multifocus image fusion method based on the EMD is to fuse the residues and the IMFs by the activity levels to produce a composite decomposition of the fused image. However, this simple fusion rule sometimes may not produce optimal EMD representation of the fused image when adjacent EMD coefficients are jointly considered to take fusion judgment where a decision fusion rule is needed. With the SVM, one expects much room for improvement over the activity level based fusion schemes.

The SVMs are a set of related supervised learning methods used for classification and regression. Interested readers may consult [18] for details. Given a group of labeled patterns {(x, y)}. x and y are the pattern and the corresponding class label, respectively. Training a SVM is equivalent to tackling a quadratic programming problem (QPP) in a number of variables equal to the number of patterns. The solution to the QPP has the following form:

K(x, x) is the kernel function used to calculate the inner production of x and x which means respectively the support vector and validated input vector. L is the number of support vectors. ॅ is the coefficient corresponding to x. C is a user-defined regularization parameter. y is different from y.

Based on the outputs of the SVM corresponding to the inputs, the activity level based fusion rule can be upgraded to the decision fusion rule in such a way that the trained SVM can be used to pick out the focused EMD coefficients for preserving the salient information at each pixel location at each level.

The procedure of the proposed method

The proposed method (Figure 2) takes the following steps:

Figure 2.

Schematic flowchart of the proposed algorithm.

Extract generalized spatial frequency (S) of each pixel of A and B using a small window (W) centered at the current pixel position according to formula (6). In this paper, the W of 3×3 is used. Let I and I(m, n) denote A or B and its gray value at (m, n), respectively. Then S(m, n) is given by:

S is used to measure the overall activity level of a pixel value because it is a manner that gray value switches to its neighbors.

Collect training patterns as follows: where M and N are the dimensions of A and B.

Train a SVM using the training patterns obtained 2). The kernel function used has the following form: where xi and xj denote the training patterns given by equations (7) and (8).

Decompose A and B with EMD along rows and columns to J levels, resulting in a residue and a total of J IMF planes, respectively.

Derive the S value of the EMD coefficients of A and B at each position at each level according to formula (6), denoted by and

Perform the fusion based on the outputs of the SVM. If the SVM output is positive, coefficients for the corresponding position of the fused image will come from A, and vice versa. In other words, the fused coefficient at level j is given by: where and are the outputs of the SVM obtained in 3) by inputting the S value obtained in 5).

Finally, the fused image is recovered by implementing IEMD according to formula (4). In Figure 2, the position (m, n) has been omitted in order to be concise.

Experiments

In this section, multifocus image fusion based on the AWT, the EMD, and the proposed method is tested on two sets of images: green pepper (512×512) and leopard (480×360). Each reference image [Figures 3(c) or 3(i)] contains one or more objects that are all in good focus. Two pairs of out-of-focus images [Figures 3(a) and 3(b); Figures 3(g) and 3(h)] are produced from the reference images by first blurring one object to obtain an image, and then blurring another object to produce second image.

Figure 3.

Reference images and source images of green pepper and leopard. (a) Focus on the front green pepper; (b) focus on the behind green pepper; (c) reference green pepper image; (d) fused image using AWT; (e) fused image using EMD; (f) fused image using EVM (C=5500); (g) focus on the right top part; (h) focus on the left bottom part; (i) reference leopard image; (j) fused image using AWT; (k) fused image using EMD; (l) fused image using EVM (C=6500).

When performing the AWT based fusion algorithm, because multiresolution analysis based on à trous filter can preserve translation invariance, short decomposition/reconstruction filters are needed to avoid ringing artifacts [19]. Max scheme choosing is used to select the significant coefficient, à trous filter 2-1/2 (1/16, 1/4, 3/8, 1/4, 1/16), together with a decomposition level of three, coefficient based activity. For the EMD, cubic spline function, along with two levels of decomposition and coefficient based max scheme is used. For performing the proposed method termed EVM (Empirical support Vector Machine), the SVM20 with the radial basis function is used, and this software was downloaded from http://liama.ia.ac.cn/PersonalPage/lbchen/svm20.zip (accessed in 2004). Based on formulae (6), (7), and (8), the training patterns are abstracted from the input images. In this experiment, each pixel in the multifocus images generates one training pattern. The fused images produced by the three methods are shown in Figures 3(d)-(f) and Figures 3(j)-(l).

Two evaluation criteria are used. They are the Root Mean Squared Error (RMSE) between the fused image F and the reference image I, and the Mutual Information (MI)

Here, h is the normalized joint gray level histogram of images F andI, h and h are the normalized histograms of F and I, and L is the number of gray levels. RMSE is used to measure the difference between F and I. MI is used to measure the reduction in uncertainty about I compared with F, so a less RMSE and a larger MI are preferred. Quantitative comparison of their performance is shown in Tables 1 and 2.

Table 1.

Performance of the three fusion methods on processing Figure 3(a) and (b)

	AWT	EMD	EVM
RMSE	5.2075	3.0118	2.6166
MI	2.5338	3.8520	3.9093

Table 2.

Performance of the three fusion methods on processing Figure 3(d) and (e)

	AWT	EMD	EVM
RMSE	3.8077	3.2249	2.7220
MI	1.7062	3.2331	3.4211

As can be found from Tables 1 and 2, the EVM exhibits significant improvements over the AWT and EMD. The fused images produced by the EVM are nearly a combination of the good-focus parts of the input images. In comparison, the fused images produced by the AWT and EMD are inferior.

The key reason for the superiority of the EVM over the AWT and EMD is the usage of generalized spatial frequency in representing image clarity, which produces good input features for the SVM in deciding which input image has the better focus at a specific pixel position.

The SVM requires the presetting of a regularization parameter [C in formula (5)] that trades off the margin with training errors. In general, using the C value too large or too small is undesirable, and this is corroborated by Figures 4(a) and (b), which show the effects of the C on the RMSE and MI by the EVM with radial basis function (Kernel) and linear basis function (Linear) on processing Figures 3(a) and (b), respectively. Initially, the parameter C is set to 5000. Then the C value is added and subtracted by 2000 each time. Figures 4(a) and (b) are plot using the RMSE and MI values of the fused images corresponding to the values of the C. In general, using a C value too large or too small is undesirable because the performance is relatively stable over a large range of C.

Figure 4.

(a) The effect of the C on the RMSE; (b) the effect of the C on the MI.

Conclusions

In this paper, we study the wedding of EMD and SVM for fusing images with different focuses of the same scene in order to get an image with every object in focus. The EMD is used for the multiresolution decomposition, while the SVM is employed to find the multifocus image with the better focus at a given pixel position. Based on the outputs of the SVM, the fusion scheme based on the activity level of the EMD coefficients can be improved to the decision fusion rule. This fusion scheme is used to select the source multifocus image that has the best focus at each pixel location. Experiments corroborate that the proposed method does better than the traditional AWT and EMD based fusion schemes in fusing multifocus images in terms of the evaluation based on RMSE and MI. By working on the EVM fused image rather than on the original defocused image, vision-related processing tasks can be expected to yield more accurate results. Compared with the separate AWT and EMD based methods, the EVM based method is more computational intensive when implemented to perform real-time image fusion. However, overall evaluation shows that it is a promising method.

In remote sensing community, one of the most challenging tasks is fusion of images with different imaging geometry and spatial resolution, for example, synthetic aperture radar images and Landsat Thematic Mapping images. In the future, we intend to extend the proposed fuser to merge multisensor images. Another is the fusion of images with obviously different pixel sizes and spectral properties, such as Moderate Resolution Imaging Spectroradiometer (MODIS) images and TM images [20]. Here, the fusion problem for the SVM then becomes how to choose the input image with the best spectral and spatial response at each location.