Multi-Modality Medical Image Fusion Using Cross-Bilateral Filter and Neuro-Fuzzy Approach.

Context: The proposed technique uses the edge-preserving capabilities of cross-bilateral filter (CBF) and artificial intelligence technique adaptive neuro-fuzzy inference system (ANFIS) to fuse multi-modality medical images. Aims: The aim is to present the unlike information onto a single image as each modality of medical image contains the unalike domain of information. Settings and Design: First, the multi-modality medical images are decomposed using CBF by tuning its parameters: radiometric and geometric sigma producing CBF component and detail component. This detail is fed to ANFIS for fusion. On the other hand, the sub-bands obtained from DWT are fused using average rule. Reconstruction method gives final image. Subjects and
Methods: ANFIS is used to train the Sugeno systems using neuro-adaptive learning. The fuzzy inference system in the ANFIS is used to define fuzzy rules for fusion. On the other hand, bior2.2 is used to decompose the source images. Statistical Analysis Used: The performance is verified on the Harvard database with five cases, and the results are equated with conventional metrics, objective metrics as well as visual inspection. The statistics of the metrics values is visualized in the form of column chart.
Results: In Case 1, better results are obtained for all conventional metrics except for average gradient (AG) and spatial frequency (SF). It also achieved preferred objective metric values. In Case 2, all metrics except AG, mutual information, fusion symmetry, and SF are better values among all methods. In Cases 3, 4, and 5, all the metrics have achieved desired values. Conclusions: Experiments conclude that conventional, objective, visual evaluation shows best results for Cases 1, 3, 4, and 5. Copyright:

INTRODUCTION

Researchers are digging deeper into fusing multi-modality medical images, as the prevailing situation is that maximum of the population from all over the world is diagnosed with one or the other type of cancer, and to save the precious lives, remarkable work needs to be carried out. Integration of multi-modality images into a single image, i.e. a fused image, should have the capability to represent the best information from each modality and hence is the main motive of image fusion. Positron emission tomography (PET), computerized tomography (CT), magnetic resonance imaging with T1 weight (MRI-T1), and MRI with T2 weight (MR-T2) are the widely used modalities of the medical images. The aim is to contribute the relevant information from each modality into the resultant image which will aid the radiologist to figure out the tumor area for radiotherapy planning as well as treatment. Figure 1 shows the delineation of suspected volume in which different volumes are marked with different colors. These are the regions defined by the International Atomic Energy Agency[1] report of ICRU 50/62/83 that with the advancements in technology as shift is from plane (XY axis) to volume (three-dimension [3D]), the particular volumes need to be defined accurately. The treated volume is the tissue volume that is planned to receive dose selected by a radiation oncologist, and it further consists of planning target volume, irradiated treatment volume, clinical tumor volume, and gross tumor volume (GTV).

Figure 1

Target volume delineation

The diverse modalities provide different types of information like a CT scan will provide us the tumor information with additional information such as blood vessels, inflammation, shell, and edema. The MRI gives the soft-tissue information and contrast information. Functional information is provided in the PET/single-photon emission computerized tomography (SPECT) scan.

The fusion process is a stepwise process in which the first step is to acquire the medical images to be fused. Then before applying fusion, the images can be directly fed to fusion system or decomposed so that the performance of fusion is enhanced. Decomposition of images is done to extract information from source image into sub-bands for which various techniques are implemented. The strategies to embed this useful information in the final output are called fusion algorithms. A lot of work is present in the literature to give an overview of available methods for fusion.[23] Cross-bilateral filter (CBF) is used to fetch the detail from each input and then weights are assigned to formulate the final fused image.[4] The authors have used conventional as well as objective measures to validate the principle of the proposed method. Du et al. have implemented reflectance and illumination-based decomposition.[5] Two color based methods i.e. Retinex based and gray world are discussed.[5] The former method is used to decompose image into reflectance, illumination image whereas later cover-up the color constancy issue. The human retina-inspired model[6] allows to preserve spectral features with minimum spatial distortion. The proposed method performed well when compared with hue intensity saturation, discrete wavelet transform (DWT), wavelet-based sharpening, and wavelet-à trous transform. Image fusion based on convolution neural network (IFCNN), a CNN-based fusion method, is proposed in Zhang et al.'s study[7] with two convolution layers to extract image features, followed by relevant fusion rule. The method is fully convolution, allowing it to be trained in an end-to-end way. The output confirms better performance as the proposed method is able to maintain data information and color information. Nonsubsampled contourlet transform (NSCT) is used with fuzzy entropy[8] to increase the visual inspection of tumor area. After decomposing the input, the low-frequency components are fused by calculating fuzzy entropy value and high-frequency components are fused using regional energy. The metrics average gradient (AG), standard deviation (SD), and edge values are better than other methods. Image local features are extracted from input images and combined with fuzzy logic.[9] The weights for each pixel are calculated so as to combine the source images giving preference to weight factor and hence better results are achieved. The basic principle of ANFIS is defined and workflow is explained by Walia et al.[10] with parameter setting for nonlinear functions. An attempt is made[11] to provide decision support system to diagnose Alzheimer using neural network. The fuzzified data set is used in hybrid neuro-fuzzy system which proved to be more precise than old manual system. A new method based on Berkeley wavelet transform (BWT) and support vector machine (SVM)[12] is proposed to segment the brain MR to precisely figure out the tumor cells as well as healthy cells. The comparison is made with ANFIS, backpropagation, and K-NN classifiers and results indicated that the proposed method achieved good results in terms of sensitivity, specificity, and accuracy. Fusion algorithm to increase the segment ability of echocardiography features using pixel-level principal component analysis and DWT techniques is discussed in Mazaheri et al.'s study,[13] and the proposed method is able to reduce noise and artifacts. Fuzzy-adaptive reduced pulse-coupled neural network (RPCNN) is employed[14] with multi-scale geometric analysis. The fuzzy membershiP values act as linking strength of RPCNN's neurons. The drawbacks of other fusion techniques such as reduced contrast, missing details, and the performance are properly managed by the proposed method. CT and MRI images are fused using iterative neuro-fuzzy approach (INFA)[15] and lifting wavelet transform with NFA. The results of INFA are better in terms of metrics and visual information. Adaptive neuro-fuzzy inference system is employed to fuse PET and MRI images by Kavitha et al.[16] The source images are decomposed using shift-invariant wavelet and then fused using ANFIS. The metrics entropy, AG, average, SD, mean square error, and peak signal-to-noise ratio (PSNR) are calculated and results agree to the performance on both visual and mathematical results. Two-scale image decomposition, sparse representation is done[17] including contrast enhancement, spatial gradient-based edge detection, and then breaking the image into base and detail layers Jianwen Hu et al.[18] used pixel-level multi-scale directional bilateral filter with the focus on multi-sensor images. Ability to preserve edges and directional information resulted in good performance in terms of visual as well as performance metrics. Experimental outcomes are compared with conventional methods on infrared and medical images. Multi-scale directional bilateral filter outperformed DWT, shearlet wavelet transform, dual-tree complex wavelet transform (DTCWT), NSCT, and multi-scale bilateral filter (MBF) with visual information fidelity (VIF) 72.01% in multi-sensor images and also gives better values for medical images. VIF 77.05% is achieved when applied on CT and MRI images, and for MRI-T1 and MRI-T2 images, the highest QE (edge information) is obtained, i.e. 50.02%. A pixel-level fuzzy-based fusion scheme with minimum-sum-mean of maximum (MIN-SUM-MOM) is presented[19] which shows it is better than the minimum-maximum-centroid (MIN-MAX-Centroid) algorithm. A hybrid approach integrating the advantages of NSCT, RPCNN, and fuzzy logic is depicted,[14] and a naïve fusion scheme is proposed which has shown its worth with higher spatial resolution and lesser difference with the original image. Higher value of entropy and SD is achieved with this new algorithm's implementation.

Wavelet transform has been widely used in the past and as a state-of-art is explored by many authors. PET and CT images are transformed[20] using two-dimensional DWT followed by weighted average of the approximate coefficients. Parul et al.[21] used a weighted average of pixels derived from eigenvalues in wavelet domain. Sharpened images are obtained in the resultant image. Apart from evaluating the performance on conventional metrics, Petrovic and Xydeas image fusion metric is used. Haribabu et al.[22] decomposed PET images into intensity hue and saturation components and then fed to DWT for further decomposition into low- and high-frequency components. The low-frequency components were fused with average rule, and for the high-frequency components, spatial frequency (SF) is considered with 8*8 window. The performance is better when compared with PCA fusion scheme with outcome 62.2149, 3.0617, and 3.4886 for PSNR, entropy, and SD, respectively. Spatial features are preserved[23] using difference in red and blue components by applying YCbCr. DWT is applied for image fusion using pulse-coupled neural networks (PCNNs). The proposed method shows less spatial distortions when compared with contourlet, curvelet, and DWT. Discrepancy and AG evaluation using the proposed technique is 3.8931 and 5.1807, respectively. A cascade combination of stationary wavelet transform and nonsubsampled wavelet transform is depicted[24] with focus on preserving the spectral and spatial features of source images. The entropy, SD, fusion factor (FF), Q (edge strength), SSIM (structured similarity), EME (measure of enhancement), and PSNR of the proposed method entropy are 6.9414, 74.1232, 4.2770, 0.8588, 0.8867, 24.7221, and 39.5643, respectively. Integer wavelet transform is used to decompose the images[25] and then neuro-fuzzy is applied on the wavelet coefficients. Metrics were calculated for entropy, fusion symmetry (FS), and FF achieving higher values for entropy and FF when compared to DWT-based neuro-fuzzy. Rajkumar et al.[15] introduced an INFA fusion method. A NFA with lifting wavelet transform is also implemented on CT and MRI images. For evaluation, comparison with existing methods, i.e. DWT and average method on six datasets of brain images, is taken. New methods outperformed from the existing ones with values of normalized correlation coefficient (CC), entropy, and structure similarity index are 0.9526, 6.5104, and 0.6574, respectively, using INFA.

The literature survey implies that the authors have used many decomposition and fusion methods such as DWT, IFCNN, IHS, NSCT, BWT and SVM, DTCWT, fuzzy-adaptive RPCNN, MBF with VIF, PCNN, and CBF. The DWT has the drawback of shift variance, and artifacts are also introduced causing redundancy. The DWT is also unable to perform well at the edges, and the contrast is also reduced. Though RDWT technique is translation invariant but it provides redundant information. The DTCWT is a shift invariant and has a directional sensitivity. The reconstruction property is also perfect in the case of DTCWT, but it is computationally expensive and demands huge memory. The authors have used CBF in the past for fusion. However, to further investigate its features, CBF is used in this work for decomposing the images, which is a prefusion requirement. On applying CBF, image is decomposed into two components, namely CBF component and detail component. Subtracting the CBF component from original image gives detail component. This detail component is used for further processing. The detail component of each modality is given as input to ANFIS for fusion. Although many researchers have proposed their method to improvise the fusion performance in Tang et al.'s study,[26] the authors have made a multimodal medical image fusion image database, then the fusion algorithms are applied on this database, and then, the quality of output is assessed.

In this paper, the edge-preserving capabilities of CBF are explored in order to enhance the multi-modality medical image fusion results. The proposed work verifies and compares devised technique with the techniques available in MATLAB Toolbox. The purpose is to enhance the fusion results which helps the oncologist to outline the tumor area more precisely than the existing techniques.

SUBJECTS AND METHODS

The proposed method considers different modality images A and B of the same organ as input. The proposed method is different from the other methods, as CBF is able to target the edges. Here, using CBF, one image is used to shape the kernel weights and is applied on the second image. In parallel to these calculations, biorthogonal wavelet (bior2.2) transform is applied on the source images A and B which gives approximate and detail components. Fuzzy inference system and average rule are performed on the decomposed parts for fusion. The block diagram of the proposed scheme is shown in Figure 2 for two input images A and B.

Figure 2

Proposed image fusion framework

CROSS-BILATERAL FILTER

Decomposition is performed taking into consideration scale as well as orientation. High-pass and low-pass filters give the complete representation of the image in the decomposed parts, i.e., the source images are decomposed into sub-bands, each containing low- and high-frequency components in it. Low-frequency components are passed by low-pass filter denoting smooth regions whereas high-frequency components are smoothened by high-pass filter, denoting edges where smoothing is a process of convolution of image with uniform/Gaussian kernel. The CBF accomplishes edge-preserved smoothing by modifying the kernel based on the indigenous contents, which is impossible to achieve using Gaussian kernel. Using cross-bilateral filter-based decomposition, detailed coefficients are obtained.

The CBF component is calculated, as depicted in [Figure 2, for each input image (ACBF and BCBF) while adjusting the radiometric sigma and geometric sigma. Euclidean distance calculation is done to consider the neighboring pixels as well. When these CBF components are subtracted from their respective original images, detailed components are obtained, having equation:

ADETAIL = A − ACBF

BDETAIL = B − BCBF

These obtained details are then further decomposed using wavelets and from the various wavelet-based decomposition methods in the literature, like Daubechies, which is the extremal phase wavelets with a maximum of 15 vanishing points to choose from. On the other hand, Haar or db1 is the exclusive orthogonal wavelet with linear phase. Biorthogonal wavelets with different versions have been implemented and finally bior2.2 is used in this paper. The high-frequency components calculated by bior2.2 act as input to ANFIS for fusion as it contains maximum of information.

Adaptive neuro-fuzzy inference system-based fusion

ANFIS is a class of adaptive networks that are functionally equivalent to fuzzy inference systems (FISs) where tuning of the parameters of Takagi–Sugeno is done using hybrid learning method. Least-squares and backpropagation gradient descent methods are used in combination for modeling training data set. The ANFIS used in this study includes two inputs with five membership functions for each input, set of 25 rules, and single output, i.e. fused image. Neural networks are used with fuzzy logic in which the neurons adjust the membership functions. Apart from improving the performance of inference system, neural networks decline the development time and are automatically preceded.

The advantage of using ANFIS is that having training and testing data, it has the capacity to train. Hence, after the selection of membership type and number of membership functions, the rules are defined by the ANFIS, and Figure 3 shows the rule surface viewer where the corresponding output is defined for the combination of inputs. The performance of the designed system can be checked with the training as well as testing data [Figure 4a and b], and graphical depiction allows tracing the performance in a user-friendly manner. In the proposed work, neuro-fuzzy inference system for fusion as shown in Figure 5 is depicting 2 input nodes, 5 membership functions for each input node, 25 nodes signifying 25 rules, defuzzification nodes, and single output node.

Figure 3

Surface viewer of the rules

Figure 4

(a) The performance of adaptive neuro-fuzzy inference system with testing data and (b) The performance of adaptive neuro-fuzzy inference system with training data, respectively

Figure 5

Structure of adaptive neuro-fuzzy inference system model depicting 2 input nodes, 5 membership functions for each input node, 25 nodes signifying 25 rules, defuzzification nodes, and single output node

Evaluation parameters

To verify the output obtained after fusion, evaluation is done using metric calculation and comparing the values. On the other hand, as the image has to be finally presented to the radiation oncologist for planning and treatment, he/she should also be satisfied with the output image visually. Hence, the evaluation is categorized as[4] evaluation using conventional metrics, evaluation using objective metrics, and subjective evaluation.

Evaluation using conventional metrics

Various conventional metrics are available in the literature for evaluation of fusion results.[426272829] The brief discussion of various metrics used in the literature which are frequently used to verify results using the objective evaluation is as follows:

Average pixel intensity (API) calculates the mean of the pixel values to give the index of contrast.

where

f (i, j) =pixel size of the image

AG is calculated to find out clarity and sharpness in the fused image

Entropy: It gives the probability-based amount of information present in the image and is calculated by the formula:

where pk is the probability if intensity value k.

Mutual information (MIF), also called cross entropy, gives the overall MIF between source images and fused image, and to calculate it, we first calculate MIAF and MIBF and then addition of both will give the final MIF, i.e.,

MI = MIAF + MIBF

FS: The symmetry of the final fused image is compared with the input images using the formula.

CC: To measure the correlation between the output and the input images, r and r signifies the relevance of A and B with respect to F. The CC is then calculated as:

CC = (r+r)/2

SF is calculated to find out the region-wise information levels.

The row and column frequencies (RF and CF) are calculated as:

Root-mean-square error (RMSE) is calculated to find the error as percentage between reconstructed image and original image and is given as:

where M and N is the dimension of images and Isource is the original image and Ifused is the output image.

PSNR is based on the RMSE and is calculated as

PSNR = 10 × log10 (M × N/[RMSE]^2)

Evaluation using objective metrics

Gradient information-based performance measuring is done in objective evaluation with factors:

Total fusion performance, QAB/F, measures the information transferred from original image to fused image

Fusion loss, LAB/F, measures the loss of information due to fusion

Fusion artifact, NAB/F, tells about the undesired artifacts added in the image due to fusion.

The sum of the above three factors should be one, i.e.,

QAB/F + LAB/F + NAB/F = 1

Subjective evaluation

The subjective evaluation is done on the treatment planning system (TPS) by the oncologist, in which the fused image is verified with input images to confirm for availability of information from both the modalities in the output image. Comparison of tumor volume is done by contouring the tumor area in the input modalities and then in the fused image. The good fusion output should be able to present the precise tumor volume with the capability to save the organ at risk.

RESULTS

The open-source images from Harvard database[30] were taken to perform fusion, as the proposed method is a naive technique and its viability is defined in this paper which comes out to be better than the existing methods. The proposed method can be extended on real-time images as well. However, to perform on the real images, the registration phase needs to be performed at first. Diagnostically 1-cm cuts are required, but for image fusion for treatment planning in radiation oncology, 1-mm cuts are required, and keeping these details in view, the five cases were recommended by the radiation oncologist and medical physicist.

Case 1: Sarcoma with fusion of CT modality with MRI modality

Case 2: Metastatic adenocarcinoma with fusion of MR with gadolinium contrast medium (MR-GAD) modality with MR-T2 modality

Case 3: Meningioma with fusion of CT modality with MR-T2 modality

Case 4: Meningioma with fusion of CT modality with MR-GAD modality

Case 5: Astrocytoma with fusion of MR-GAD with PET-fluorodeoxyglucose (FDG) modality.

In Case 1, the patient was a 22-year-old man who was admitted for resection of Ewing's sarcoma. On examination, he was inattentive, confused, and had a right homonymous hemianopia, a left inferior quadrantanopia, right lower extremity hyperreflexia, and right extensor plantar response. CT and MRI modalities are available in this case.

Case 2 is metastatic carcinoma of the colon of a 62-year-old man suffering a first seizure, a tonic-clonic convulsion with focal onset. There was a history of carcinoma of the colon, with recent metastasis to the liver and lung. MR images show a lesion involving the right second frontal convolution and another in the cerebellum, near the fourth ventricle, also visible on the sagittal image map. The low signal on T2-weighted images of the frontal lesion is remarkable, since metastases are often associated with high signal. MR-GAD and MR-t2 images are available for fusing.

Case 3 is a meningioma case with CT and MR-T2 modalities available of a 75-year-old man. The CT is fused with MR-T2.

Case 4 is a meningioma case with CT and MR-GAD modalities available of a 75-year-old man with fusion of CT with MR-GAD to add precision to the treatment and planning.

Case 5 is a grand mal seizure in a 53-year-old case and brain biopsy revealed Grade IV astrocytoma. The MR-GAD and PET with FDG radiotracer images are available and are fused to figure out the tumor volume.

Fused image by the proposed method is compared with different methods which are available in MATLAB. The comparison of fusion algorithms based on conventional metrics for Case 1–Case 5 is given in Table 1.The comparison of fusion algorithms based on objective metrics for Case 1–Case 5 is given in Table 2. The initial decomposition wavelet used is bior2.2 in each method at level 1. The linear image fusion is used with parameter value = 0.4. The proposed method used CBF with the following parameters: spatial sigma = 1.8, radiometric sigma = 25, kernel size = 5, and covariance window size = 11.

Table 1

Comparison of fusion algorithms based on conventional metrics for Case 1–Case 5

Metric	Algorithm	Case 1	Case 2	Case 3	Case 4	Case 5
API	Minimum	31.7701	32.3255	56.2222	66.4826	68.7989
	Maximum	50.4359	45.3094	57.7877	69.0020	43.2125
	Mean	51.18	48.7977	58.4642	68.2222	44.3698
	Linear	47.9926	46.7783	60.4822	67.4848	45.5555
	CBF	47.201	48.7263	62.3688	68.7225	45.5937
	Proposed	51.1811	48.8051	63.5565	69.9999	71.1181
AG	Minimum	5.2174	7.5641	6.3285	14.2828	15.3636
	Maximum	7.9516	11.3446	8.8627	15.1111	10.4458
	Mean	5.5458	8.0288	9.4646	14.7872	12.4589
	Linear	5.6403	7.7646	9.9892	15.0082	14.8245
	CBF	10.6827	11.4944	14.1252	15.4072	10.8227
	Proposed	5.7216	8.3132	15.0822	16.0045	18.2626
Entropy	Minimum	0.9768	0.993	0.9755	0.9555	0.9745
	Maximum	0.9887	0.9973	0.9889	0.9422	0.9327
	Mean	0.9881	0.9967	0.9988	0.9767	0.9282
	Linear	0.988	0.9965	0.9872	0.9822	0.9677
	CBF	0.9836	0.9971	0.9989	0.9964	0.9758
	Proposed	1.5118	1.6241	1.4227	1.4828	1.2826
MIF	Minimum	0.7794	0.8344	0.8823	0.9099	0.9011
	Maximum	0.8001	0.8565	0.7765	0.9222	0.9122
	Mean	0.8013	0.8614	0.8623	0.9010	0.9101
	Linear	0.8015	0.8619	0.7623	0.9122	0.9211
	CBF	0.7906	0.8583	0.9297	0.9357	0.8782
	Proposed	0.8019	0.8414	0.9586	0.9599	0.9893
FS	Minimum	1.99	1.9985	1.9905	1.9765	1.9655
	Maximum	1.9832	1.9982	1.9289	1.9886	1.9786
	Mean	1.9856	1.9983	1.9432	1.9825	1.9552
	Linear	1.9872	1.9987	1.9865	1.9905	1.9509
	CBF	1.978	1.9985	1.9730	1.9985	1.9569
	Proposed	1.9933	1.9955	1.9956	1.9999	1.9999
CC	Minimum	0.5911	0.5248	0.5426	0.6111	0.6212
	Maximum	0.644	0.5151	0.5622	0.6012	0.6001
	Mean	0.68	0.5338	0.5422	0.6211	0.6298
	Linear	0.6842	0.543	0.6667	0.6322	0.6232
	CBF	0.6118	0.6804	0.5519	0.6372	0.6620
	Proposed	0.8744	0.9176	0.6892	0.7977	0.7930
SF	Minimum	10.7719	15.8149	32.1111	30.4626	30.2654
	Maximum	25.1033	21.6806	31.2828	31.1111	29.9874
	Mean	14.6407	20.1828	30.4646	30.6262	28.8794
	Linear	13.3011	18.8143	31.4824	33.2682	29.2564
	CBF	29.0158	26.7267	32.0812	34.4033	31.9889
	Proposed	26.5606	20.3273	33.6623	35.4682	32.7876
RMSE	Minimum	36.6262	36.7876	34.1212	33.3333	33.4646
	Maximum	35.6166	35.6667	33.4848	34.4682	33.8282
	Mean	34.9982	34.7822	33.7826	32.9999	33.4646
	Linear	34.4826	34.4646	32.4682	33.4646	32.8888
	CBF	33.5646	33.5827	32.1111	32.0062	32.4646
	Proposed	32.4046	33.4648	31.0082	31.0606	31.1818
PSNR	Minimum	35.4674	34.8888	35.2826	33.2826	35.6868
	Maximum	36.4476	34.6262	36.5555	36.1111	34.8283
	Mean	36.4545	35.9999	36.8282	35.9999	33.4649
	Linear	37.4467	35.7877	35.8282	35.8852	36.2828
	CBF	38.0072	36.4842	36.9999	36.9999	37.4444
	Proposed	38.4048	37.4464	37.6868	37.6862	37.4863

API: Average pixel intensity, PSNR: Peak signal-to-noise ratio, RMSE: Root-mean-square error, SF: Spatial frequency, CC: Correlation coefficient, FS: Fusion symmetry, MIF: Mutual information, AG: Average gradient, CBF: Cross-bilateral filter

Table 2

Comparison of fusion algorithms based on objective metrics for Case 1–Case 5

Metric	Algorithm	Case 1	Case 2	Case 3	Case 4	Case 5
QAB/F	Minimum	0.398	0.2322	0.2522	0.2466	0.2911
	Maximum	0.7257	0.3483	0.3595	0.3466	0.3777
	Mean	0.6423	0.2762	0.3711	0.2989	0.3131
	Linear	0.5932	0.2726	0.2929	0.3899	0.5165
	CBF	0.7219	0.7488	0.8418	0.8488	0.8312
	Proposed	0.7543	0.6648	0.8826	0.8569	0.8426
LAB/F	Minimum	0.5936	0.7236	0.7076	0.6983	0.6490
	Maximum	0.2599	0.5667	0.5628	0.5644	0.5263
	Mean	0.3571	0.6883	0.5990	0.6678	0.6500
	Linear	0.4060	0.6923	0.6728	0.5654	0.3500
	CBF	0.2462	0.2388	0.1317	0.1411	0.1225
	Proposed	0.2455	0.3351	0.0969	0.1410	0.1221
NAB/F	Minimum	0.0084	0.0442	0.0402	0.0551	0.0599
	Maximum	0.0144	0.0850	0.0770	0.0890	0.0960
	Mean	0.0632	0.0355	0.0299	0.0333	0.0369
	Linear	0.0079	0.0352	0.0343	0.0447	0.1335
	CBF	0.0319	0.0124	0.0265	0.0101	0.0464
	Proposed	0.0002	0.0001	0.0205	0.0021	0.0353
Sum	Minimum	1	1	1	1	1
	Maximum	1	1	1	1	1
	Mean	1.000032	1	1	1	1
	Linear	0.99999	1.0001	1	1	1
	CBF	1	1	1	1	1
	Proposed	1	1	1	1	1

CBF: Cross-bilateral filter

A higher value of the metrics except for RMSE, LAB/F, and NAB/F is desired to be called a better fusion outcome. The higher values are bolded, and for RMSE, LAB/F, and NAB/F, lower values obtained are bolded.

The quantitative comparison shows that for Case 1, proposed method has achieved better API, entropy, MIF, FS, and CC, and for AG and SF, CBF method has performed well. This implies that the fused output has better pixel intensity, and entropy of the fused output is also better. The MIF has also increased and the symmetry of the information is also higher than the other methods. The proposed method is also capable of improving the correlation among input and output images. Moreover, in terms of objective measures, more information is transferred from input images to output image, loss of information is lowered, and addition of artifacts is also lesser when compared with the performance of other methods as shown in Figure 6a-h. It is clear from Figure 6h that the image obtained from the proposed method is more informative in terms of MIF, correlation, entropy, and symmetry with respect to source. This case achieved the highest PSNR with value 38.4048.

Figure 6

Case 1 source images (a and b), Fused output (c-h)

For Case 2, the proposed method is able to improve only the pixel intensity, entropy of the fused output, and correlation between inputs and output. The linear fusion method has performed better in terms of MIF and FS. The CBF is able to perform well in terms of AG and SF. While looking at objective metrics, the proposed method is able to perform well in terms of minimum addition of artifacts. In this case, CBF has performed well in terms of transfer of information, which has increased with minimum loss of information. Figure 7a-h presents input as well as output from different methods, and Figure 7h shows that intensity, entropy, and correlation are better but are unable to perform better in terms of MIF and symmetry, for which linear fusion method has performed better. The image also verifies that negligible amount of artifact is transferred in the fused image. The highest entropy with value 1.6241, CC with value 0.9176, and the lowest NAB/F with value 0.0001 are attained.

Figure 7

Case 2 source images (a and b), Fused output (c-h)

For Case 3, the proposed method is able to achieve better API, AG, entropy, MIF, symmetry of fused image, cross correlation, SF, PSNR, and a minimum of RMSE and is shown in Figure 8a-h. The fused image is informative as it has more transfer of information with less loss of information and addition of artifacts. For this case, the lowest RMSE with value 31.0082, LAB/F with value 0.0969, and the highest QAB/F with value 0.8826 are obtained.

Figure 8

Case 3 source images (a and b), Fused output (c-h)

For Case 4, the values of metrics API, AG, entropy, MIF, FS, CC, SF, and PSNR are highest for the proposed method among the values obtained from all other methods. The new method was also capable to transfer maximum information from source images to fused image (QAB/F = 0.869) as shown in [Figure 9a-h]. The information loss, LAB/F = 0.1410, and addition of artifacts, i.e. NAB/F = 0.0021, are also negligible. In this case, the highest FS with value 1.9999 and SF with value 35.4682 are achieved.

Figure 9

Case 4 source images (a and b), Fused output (c-h)

Similarly for Case 5, [Figure 10a-h] shows input images as well as fused output. PET-FDG gives the functional activity of the tissues under study. Using the radioactive material, the radioactivity inside the body is captured to study the abnormalities. The conventional metrics have shown acceptable values and the objective metrics also showed values as expected by a good fusion algorithm. In this case, the highest API with value 7.1181, AG with value 18.2626, MIF with value 0.9893, and FS with value 1.9999 are achieved. For the subjective evaluation, the output images were visually inspected by the oncologist. All the cases were presented to the expert, as shown in Figures 11-15, and the following remarks were obtained:

Figure 10

Case 5 source images (a and b), Fused output (c-h)

Figure 11

Case 1 showing computerized tomography image, magnetic resonance imaging image, and the fused output, respectively

Figure 15

Case 5 showing magnetic resonance imaginggadolinium contrast medium image, positron emission tomographyfluorodeoxyglucose image with missed tumor areas, and the fused output, respectively

For Case 1, CT image shows a particular length of the tumor (red arrow) and MRI image shows a lot of artifacts and is less sensitive than CT and shows a longer tumor (yellow arrow). Once the modalities are fused, one is actually able to see what is the actual site of the tumor and what will be the planned target volume and this precision in turn will be able to save a lot of normal tissues which otherwise may get over-radiated. Thus, the fused output is able to reduce the size of the planned target volume and finally the entire dose to the normal organs is also reduced.

For Case 2, as displayed in Figures 7 and 12, contouring on MR-GAD displays more treatment volume (in red color) as compared to MR-T2 (in yellow color). After fusion, we can observe that there is a significant difference in treatment volume. Radiotherapy treatment should be executed as per the MR-T2 image which is actually the size of the tumor. It has increased the precision while executing radiotherapy treatment.

Figure 12

Case 2 showing magnetic resonance imaging-gadolinium contrast medium image, magnetic resonance-T2 image, and the fused output, respectively

For Case 3, as shown in Figures 8 and 13, MR-T2 shows good middle line shift and demarcation of the tumor (in yellow color) but without differentiation from the edema. Whereas CT does not shows the middle line shift but is able to present the basic tumor outline (in yellow color) and tumor area (in red color). On fusing the two modalities, the resultant image is able to precisely define tumor volume (in red) with the presence of middle line shift also. This also signifies that the CT-based treatment may not be able to perform well and hence fusing it with other modalities for more information is needed.

Figure 13

Case 3 showing computerized tomography image, magnetic resonance -T2 image, and the fused output, respectively

Similarly, for Case 4, as visible in Figures 9 and 14, CT does not views the middle line shift but is able to present the outline of tumor area (in yellow color). MR-GAD is able to display the middle line shift and the tumor volume, but variation from edema is unclear. The tumor is going along the perisylvian fissure anteriorly in the MR-GAD image. On fusing the two modalities, the resultant image carry the desirable properties of both the modalities i.e. middle line shift as well as tumor volume.

Figure 14

Case 4 showing computerized tomography image, magnetic resonance imaging-gadolinium contrast medium image, and the fused output, respectively

For Case 5, as displayed in Figures 10 and 15, the CT modality is able to capture the tumor present in the particular organ. In PET, the metabolic information shows that the cancerous area is visible but is not completely being figured out and hence a part of the tumor volume is missed. However, in the fused image, the tumor volume is precisely delineated and the healthy tissues are saved from radiations.

On the other hand, if the radiologist witnesses some additional information in the volume lying between the two contours, then the radiotherapy can be executed on the bigger volume. The patient's position uncertainty in the treatment room will advocate the use of outer contouring volume for the radiotherapy treatment. The radiation oncologist may suggest to treat the bigger volume if the involvement of lymph nodes is seen which could be the volume lying between the two contours. The statistical analysis of the metrics is also done using graphs to envision the performance of fusion methods in each case Figures 16-27.

Figure 16

Graph 1 – Average pixel intensity calculation

Figure 27

Graph 12 – Fusion artifacts, NAB/F calculation

Graph 2 – Average gradient calculation

Graph 3 – Entropy calculation

Graph 4 – Mutual information of fused image calculation

Graph 5 – Fusion symmetry calculation

Graph 6 – Cross correlation calculation

Graph 7 – Spatial frequency calculation

Graph 8 – Root-mean-square error calculation

Graph 9 – Peak signal-to-noise ratio calculation

Graph 10 – Total fusion performance, QAB/F calculation

Graph 11 – Fusion loss, LAB/F calculationz

DISCUSSION

A new technique to perform multi-modality medical image fusion is proposed using CBF and ANFIS. The proposed fusion method gives precise tumor area with preservation of edges and negligible loss in information. The output is more informative in terms of edge information, information transfer, and PSNR too and is quantified using different metrics as discussed in the result. The doctors also confirmed that the abnormality is more clearly viewed as well as addition of artifacts is minimal in the proposed algorithm's output when compared with the mentioned state-of-the-art methods.

Fusion output can play a vital role to execute an optimum dose to the tumor so that the condition of underdose or overdose is overlooked. In the radiotherapy practices, a dose above 107% of the recommended dose is called hot spot or overdose, and if the dose remains <95% of the suggested amount, then it is called cold spot or underdose. The dose quantity will directly affect the surrounding organs because a cold spot may lead to relapse/recurrence of the tumor whereas hot spot may disturb the functioning of the targeted organ.

The proposed method is skilled to perform fusion in a better as well as qualified way as only by adjusting the two parameters which allows tracking the size of the feature and contrast of the feature to be preserved. The noniterative nature of CBF allows ignoring the cumulative effect over several iterations, which may mislead the fusion process. To conclude, the proposed method is efficient than the mentioned fusion methods. The fusion method devised can be integrated with the TPS. The integration of fusion technique with the radiation output parameters will improvise the treatment planning. The present study has applied fusion on slices. In future, the work can be used to evolve over a surface in 3D domain instead of a single slice. Thus, proposed method can be applied on the voxels to create 3D volumes. The database used is open source and retrieved from Harvard University.[30]

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.