The Tradeoff Analysis for Remote Sensing Image Fusion Using Expanded Spectral Angle Mapper.

Image fusion is a useful tool in integrating a high-resolution panchromaticimage (HRPI) with a low-resolution multispectral image (LRMI) to produce a highresolutionmultispectral image (HRMI). To date, many image fusion techniques have beendeveloped to try to improve the spatial resolution of the LRMI to that of the HRPI with itsspectral property reliably preserved. However, many studies have indicated that thereexists a trade- off between the spatial resolution improvement and the spectral propertypreservation of the LRMI, and it is difficult for the existing methods to do the best in bothaspects. Based on one minimization problem, this paper mathematically analyzes thetradeoff in fusing remote sensing images. In experiment, four fusion methods are evaluatedthrough expanded spectral angle mapper (ESAM). Results clearly prove that all the testedmethods have this property.

Introduction

In optical remote sensing, with a combination of a set of observational constraints imposed by the acquisition system, detector specifications and satellite motion, among others, some satellite sensors supply low-resolution multispectral images (LRMIs) needed to identify features spectrally but not spatially, while other satellite sensors supply high-resolution panchromatic images (HRPIs) characterizing features spatially but not spectrally [1,2]. In addition, an increasing number of applications, such as urban mapping, feature detection and change identification, often demand the high resolution multispectral images (HRMIs). In general, there exist two types of approaches for acquiring such images. One is to increase the sensitivity of the photo detector of the satellite sensors, but the photons are expensive to collect, making long exposure multispectral observations unusual. Another comes from the field of image fusion [3].

To date, various remote sensing image fusion methods have been proposed in the literature [1,4,5]. These methods inject high frequency features from a HRPI into every LRMI trying to improve the spatial resolution of the LRMI to that of the HRPI with its spectral property reliably preserved. The objective is to obtain the HRMI that would be observed by a sensor with the same spectral response (i.e., spectral sensitivity and photon efficiency) as the multispectral sensors and the same spatial response (i.e., spatial detail and texture structure) as the panchromatic sensor [2]. However, it is difficult for the existing methods to do the best in both aspects. Some methods, such as intensity-hue-saturation (IHS) [6], provide superior HRMIs visually but inferior spectrally, while such HRMIs can not satisfy the requirement of most remote sensing applications based on spectral signatures, such as vegetation analysis [1,7]. On the contrary, some methods, such as orthogonal wavelet transform (OWT) [8], and à trous algorithm based wavelet transform (AWT) [9], produce better HRMIs spectrally but worse spatially. It was found that remote sensing image fusion is a tradeoff between the spectral information of the LRMI sensor and the spatial information of the HRPI sensor [1,2,5,8].

However, no explicit theoretical analysis of the tradeoff property has been presented in the existing literature. In Section 2, we present a dissertation which does demonstrate the tradeoff in fusing remote sensing images. Furthermore, the analysis framework makes it convenient to compare and evaluate the current image fusion methods and it is also open to include new fusion methods in the future. In this Section, spectral angle mapper (SAM) is improved to expanded spectral angle mapper (ESAM) for evaluating the performance of fusion methods.

To validate the theoretical analysis, the tradeoff property is tested by using ESAM to evaluate the spectral and spatial quality of the HRMIs produced from fusing QuickBird images using four fusion methods in Section 3. Finally, conclusions are given in Section 4.

The Tradeoff Analysis Based on RMSE

Because the analytical model of image fusion does not exist, the difference between two images is employed to assess the tradeoff property. First, we define the root mean squared error (RMSE) between images U and V as:

where {u1, u2,⋯, un} and {v1, v2,⋯, vn} denote the spectral vectors of U and V, respectively. The greater the RMSE, the higher the difference between U and V will be. Before the analysis, the LRMI is first resampled to the same spatial resolution of the HRPI and HRMI. Let T, P, and F denote the LRMI, HRPI, and HRMI. Then based on RMSE, let us consider the following minimization problem:

The terms RMSE and RMSE are used to estimate the spectral and spatial quality of the HRMI, respectively. The subsequent result is:

where t represent the pixel values of the LRMI, the HRPI and the HRMI. The solution for (2) is that the HRMI is equal to the (LRMI+HRPI)/2.

From formula (3), we can see that for the HRMI obtained from any image fusion method, the value of is equal to or greater than . It is impossible for fusion methods to sharp the spatial resolution of the LRMI to that of the HRPI with its spectral property completely preserved. Whenever the difference between the HRMI and HRPI is lower, the amount of the detail information of the HRMI is larger while its color distortion will be larger accordingly. On the contrary, whenever the difference between the HRMI and LRMI is lower, the color distortion of the HRMI is less whereas the amount of the detail information of the HRMI is less, either. As a result, a tradeoff occurs between the detail and color information of the HRMI. Therefore, we can draw a conclusion that any given fusion method cannot produce such image with the same amount of the detail information as the HRPI and the same color information as the LRMI.

In order to quantitatively assess the performance of image fusion methods, many objective image quality indexes have been proposed in the literatures [5] for objective evaluation of the HRMIs. Among these quality indexes, correlation coefficient (CC) and SAM [9] are widely used for measuring spectral similarity between the HRMI and LRMI. where μ are the mean values of U and V.

However, CC is insensitive to a constant gain and bias between the two images and does not allow a subtle discrimination of possible fusion artifacts [10]. Similar limitations happen to SAM. ESAM is introduced to measure the similarity between two images: Compared with the general used CC and SAM, ESAM is more sensitive to the difference between two images because the value of the ESAM is equal to 0 only for two identical images whereas the values of the SAM and CC can be equal to 0 for two similar images. As to the ESAM and SAM, for example, U={u, u,⋯, u}, V={v, v,⋯, v}, and u ≠ v, the value of the SAM is 1, but the value of the ESAM is less than 1; As to the CC and ESAM, for example, U={u+l, u−1,⋯, u+ (−1)n}, V={v+1, v−1, ⋯, v+(−l)n}, and u≠v, the value of the CC is 1, but the value of the ESAM is less than 1. Therefore, the ESAM is more informative than the SAM and CC in terms of measuring how close the pixel values of the two images are.

The ESAM measures only information from isolated pixels, and does not consider the neighboring pixel relationships, which are more important for structures and textures. Therefore, it would be convenient to take into account the local pixel relationship during comparing two images. As image quality is often space dependent [11], the ESAM is computed using a sliding window approach. In this work, sliding windows with a size of 16 x 16, 32 x 32, 64 x 64, and 128 x 128 pixels are used. Considering that the ESAM can only be applied to monochromatic images, the average ESAM (AE) is used as an overall quality index for measuring the difference between the HRMI and the LRMI, HRPI. The higher the AE, the less the similarity of two images will be.

Experiments

The raw images are downloaded from http://studio.gge.unb.ca/UNB/images. These remote sensing images are taken by QuickBird satellite sensor which collects one panchromatic band (450-900 nm) of the 0.7 m resolution and blue (450-520 nm), green (520-600 nm), red (630-690 nm), near infrared (760-900 nm) bands of the 2.8 m resolution. The coverage of the images was over the Pyramid area of Egypt in 2002. Before the image fusion, the raw LRMIs were resampled to the same spatial resolution of the HRPI in order to perform image registration. The test images of size 1024 by 1024 at the resolution of 0.7 m are cut from the raw images. In order to save space, only the natural color LRMI (red-green-blue composite) is shown in Figure 1(a). The near infrared band and the corresponding HRMIs obtained using different methods are not shown but are processed and numerically evaluated. The HRPI is shown in Figure 1(b). Obviously, more details can be found from the HRPI than the LRMIs because of its finer spatial resolution.

Figure 1.

Fusion results of different methods. (a) the LRMIs as a colour composite; (b) The HRPI; (c) The HRMIs produced by the IHS method; (d) The HRMIs produced by the OWD method; (e) The HRMIs produced by the AW method; (f) The HRMIs produced by the MAIM method.

The tradeoff property is tested on these images obtained applying the following standard fusion methods:

Intensity Hue Saturation transform (IHS) [1].

OWD [8,12].

Additive Wavelet-based method (AW) [13,14].

Multiresolution Analysis-Based Intensity Modulation (MAIM) [1].

The spatial resolution ratio between the QuickBird HRPI and the LRMIs is 1: 4. Therefore, in the OWT method, Daubechies ‘db4’ filter, a decomposition level of three, coefficient coefficiency based activity, and choose max scheme are used. In the AW and MAIM methods, a trous filter 2-1/2 (1/16, 1/4, 3/8, 1/4, 1/16), together with a decomposition level of two, is employed to abstract the high frequency information of the HRPI. Fused HRMIs using different algorithms are shown in Figures l(c)-(f).

In this experiment, the quality of the fused HRMIs is estimated both qualitatively and quantitatively. Visual inspection is used for qualitative estimation. Both ESAM and RMSE are used for quantitative evaluation.

Visual inspection

Visual inspection provides the analyst an overall impression of the HRMI with respect to the LRMI. Visually, some small spatial structure details, such as edges, lines, which are not discernible in the LR- MI (Figure 1(a)), can be identified individually in each of the HRMIs (Figures 1(c)-(f)). This means the spatial quality of the LRMI has been improved via the fusion process.

It can be seen from Figures 1(c)-(f) that the HRMI produced by the IHS method shows the most obvious color distortion. The HRMI of the OWD method shows the second obvious color distortion. The MAIM and AW methods have slight color distortion. Conversely, from Figure 1(c) to Figure 1(f), the spatial resolution decreases. For the IHS, this is due to the fact the HRPI is directly substituted to the low resolution intensity component. The critically-sampled multiresolution analysis in the OWT method affects the spectral preservation and the spatial resolution improvement of the LRMI. The AW and MAIM methods produce slight color distortion because the additive methods have not considered the differences in high-frequency information between the HRPI and the LRMI.

Therefore, we can draw a conclusion that the closer the spatial resolutions of the HRMI and HRPI, the larger the spectral distortion of the HRMI. When the higher the similarity of the HRMI and LRMI, the less the spectral distortion of the HRMI but the lower its spatial resolution. This result means that the tradeoff property exists in fact in terms of visual inspection. It is impossible for a fusion method to improve the spatial resolution of the LRMI to that of the HRPI without altering its spectral property.

Quantitative analysis

In addition to the visual inspection, the tradeoff property has been analyzed quantitatively using ESAM and RMSE. Tables 1 and 2 show the results.

Table 1.

ESAM values between the LRMIs, HRPI and HRMIs

	IHS	OWD	AW	MAIM
AETF16×16	22.71°	13.34°	9.59°	13.28°
AETF32×32	21.74°	13.28°	8.79°	12.33°
AETF64×64	20.83°	13.29°	8.19°	11.57°
AETF128×128	20.09°	13.41°	7.85°	10.97°
AEFP 16×16	4.99°	20.14°	17.84°	16.23°
AEFP32×32	4.57°	19.00°	16.87°	15.63°
AEFP64×64	4.21°	17.93°	15.92°	15.00°
AEFP 128×128	3.87°	16.95°	15.01°	14.29°

Table 2.

RMSE values among the LRMIs, HRPI and HRMIs

		RMSETF	RMSEFP	RMSETFP	RMSE_TP/2
IHS	B1	49.82	13.52	63.34	34.77
	B2	49.44	12.24	61.69	30.47
	B3	48.56	17.38	65.94	28.28
	B4	34.31	33.83	68.13	28.61
OWD	B1	32.27	47.09	79.36	34.77
	B2	31.65	41.32	72.97	30.47
	B3	31.74	38.33	70.07	28.28
	B4	21.62	36.78	58.40	28.61
AW	B1	17.13	44.94	62.08	34.77
	B2	17.43	37.98	55.40	30.47
	B3	17.17	36.69	53.86	28.28
	B4	14.86	31.24	46.10	28.61
MAIM	B1	19.93	45.87	65.79	34.77
	B2	19.25	39.96	59.21	30.47
	B3	18.51	37.52	56.03	28.28
	B4	22.99	35.69	58.69	28.61

From Table 1, it can be found that the IHS method has the highest AE, followed by the OWD and MAIM methods, the AW method has the lowest AE. On the other hand, the IHS method has the lowest AE, followed by the MAIM and AW methods, the OWD method has the highest AE. The higher the AE, the lower the similarity of two images is. Therefore, in terms of transferring details, the performances of the IHS, MAIM, AW, and OWD methods decrease; in terms of preserving spectral property, the order is the AW, MAIM, OWD, and IHS methods.

Similar results can be found from Table 2. In Table 2, B1, B2, B3, and B4 denote the red, green, blue, and near infrared bands, respectively; the RMSE is the sum of the RMSE and the RMSE; the is the minimum RMSE value for any fused HRMI. From the fifth and sixth columns, it can be found that all the RMSE values of the HRMIs exceed the corresponding values. Based on the RMSE, the grade of the extent that the HRMI is close to the LRMI is the AW, MAIM, OWD, and IHS methods. Based on the RMSE, the order of the extent that the HRMI is close to the HRPI is the IHS, AW, MAIM, and OWD methods. This indicates that the tradeoff property exists in terms of the RMSE index.

Based on Tables 1 and 2, the comprehensive analyses of the four fusion methods are shown in Table 3. From Table 3, we can conclude that the AW method is the best among the four when considering both spatial enhancement and spectral preservation. Conversely, the OWD is the method to avoid because both the spatial and spectral qualities of the HRMIs obtained are worse. It will be very interesting to know this kind of information for the image fusion community or uses.

Table 3.

The results obtained based on Tables 1 and 2

	IHS	OWD	AW	MAIM
Spatial Improvement	best	worst	better	worse
Spectral Preservation	worst	worse	best	better

Based on Tables 1, 2 and 3, in addition to the conclusion that the tradeoff property exists during fusing remote sensing images, another is that a given fusion method can do better than another one in both aspects, but it could not do the best in both. For example, the AW method does better than the OWD method spectrally and spatially, however, it does worse than the IHS method spatially. Therefore, it could not obtain a HRMI that the multispectral sensor would capture at the spatial resolution of the panchromatic one through fusion methods.

Conclusions

Based on the minimization problem (formula (2)), this paper mathematically demonstrates the tradeoff property during fusing the remote sensing images. The analysis shows that the Euclidean distance sum between any fused HRMI and the LRMI, HRPI will exceed the value given by formula (3). Using the ESAM and RMSE indices, four existing image fusion methods, IHS, OWD, AW, and MAIM, are experimented on QuickBird images to test this conclusion.

The experimental results show that some fusion methods can produce better the HRMI spectrally and spatially than other fusion methods. However, fusion methods often attend to one thing and lose another in terms of compromising the spatial and spectral properties of the HRMI. Therefore, they can be strong spectrally or spatially, but it is impossible for them to produce one HRMI with simultaneously high spatial and spectral resolution through fusion procedure.