Image quality improvement in cone-beam CT using the super-resolution technique.

This study was conducted to improve cone-beam computed tomography (CBCT) image quality using the super-resolution technique, a method of inferring a high-resolution image from a low-resolution image. This technique is used with two matrices, so-called dictionaries, constructed respectively from high-resolution and low-resolution image bases. For this study, a CBCT image, as a low-resolution image, is represented as a linear combination of atoms, the image bases in the low-resolution dictionary. The corresponding super-resolution image was inferred by multiplying the coefficients and the high-resolution dictionary atoms extracted from planning CT images. To evaluate the proposed method, we computed the root mean square error (RMSE) and structural similarity (SSIM). The resulting RMSE and SSIM between the super-resolution images and the planning CT images were, respectively, as much as 0.81 and 1.29 times better than those obtained without using the super-resolution technique. We used super-resolution technique to improve the CBCT image quality.

INTRODUCTION

Cone-beam computed tomography (CBCT), which can reveal much information related to internal organs and tissues, has been used increasingly in medical applications in recent years. For example, CBCT has been used in angiography CT, CT fluoroscopy, and 4D CT [1-4]. In particular, CBCT is extremely useful for radiation treatment. The application of CBCT to radiation treatment has been developed for the use of image-guided radiotherapy (IGRT), enabling high-accuracy radiotherapy [5, 6].

Recently, many approaches for the implementation of adaptive radiotherapy (ART), which allows real-time treatment adaptation based on the most recent patient anatomy, have been investigated for use in providing higher accuracy treatment. Although CBCT images provide the most up-to-date information for a human body, they are adversely affected by noise and artifacts, which degrade the image quality. The image quality is often so low that CBCT images cannot be used directly to compute the dose distribution for treatment planning.

A conventional method using low-quality CBCT images for ART transforms planning CT images to CBCT images using non-rigid functions. The method then uses these images for re-planning [7-9]. Nevertheless, it is difficult to ensure high positional accuracy using this method with soft tissues or flatus with particularly large positional displacement between two images.

Another method for improving CBCT images suppresses scatter or cupping artifacts. The dose distribution is computed directly from those improved images [10-12]. However, those methods are applicable only to images before reconstruction. Furthermore, a specific filter must be used to improve image quality.

As described herein, we propose an image-improving method using a super-resolution technique for upsampling a low-resolution image (or noisy image) by combining it with a corresponding high-resolution image (or clean image) [13]. Two benefits are gained when applying this method to CBCT images. (i) No misregistration occurs between the CBCT image and the reconstructed image, which is useful for radiotherapy that requires high irradiation accuracy. (ii) This method can be adapted to the reconstructed image. Therefore, no special equipment or reconstruction technique is necessary in order to improve image quality.

An earlier study by Karimi et al. [14] examined reduction of the radiation dose by application of a super-resolution technique to CT images. Their method uses a few projections as low-resolution images and numerous projections as high-resolution images. Images reconstructed from a few projections were clarified using this technique. Most studies using super-resolution techniques have adopted this approach for image improvement for the same modality.

There are two fundamental differences between the study by Karimi et al. and our study. The specifications of the CBCT used for our study differ from the specifications of the CT used for the other study. The CBCT image quality is much lower than the CT image quality because of the effects of the scattered X-rays, which are greater for larger irradiated areas.

Another important difference is the purpose of using super-resolution. The method reported by Karimi et al. uses a few projections as low-resolution images and numerous projections as high-resolution images. Actually, the purpose of their study was to produce high-dose CT images from low-dose images by application of the super-resolution technique. By contrast, the purpose of the present study was to estimate high-quality CBCT images from other devices, and to use planning-CT images as high-resolution images.

We propose a method of improving CBCT image quality by application of the super-resolution technique using images taken from two modalities: planning CT and CBCT.

MATERIALS AND METHODS

Overview of the conventional super-resolution method

We start with a description of an overview of the conventional super-resolution method, which usually requires numerous high-resolution images. As a first step, the super-resolution method generates a matrix consisting of high-resolution image bases: ‘atoms’. This matrix is called a high-resolution dictionary (DH). The K-SVD [15] algorithm has been used to generate the dictionary. Then, the super-resolution method generates a low-resolution dictionary (DL) from undersampling or by adding blur to high-resolution atoms in DH. The DL represents the low-resolution image atoms. A low-resolution image is represented as a linear combination of low-resolution atoms. Finally, the super-resolution image is inferred by replacing low-resolution atoms with high-resolution ones. Consequently, a high-resolution image is generated from a low-resolution one.

Overview of the proposed method

A schematic representation of our approach is presented in Fig. 1. As a first step, high-resolution images and low-resolution images must be acquired to construct dictionaries. For this study, the planning CT and CBCT images were regarded, respectively, as high-resolution and low-resolution images. As a second step, the dictionaries are created from those images. Then an input CBCT image is represented as a linear combination of the atoms in the low-resolution dictionary. Finally, the super-resolution image is inferred by replacing the atoms in the low-resolution dictionary with the atoms in the high-resolution dictionary. Details of this method are described in the following sections.

Fig. 1.

Schematic representation of our proposed algorithm.

Database preparation

For this study, a set of full-pelvis-region images (planning CT and CBCT) of 30 patients with prostatic tumor was acquired using planning CT (Aquilion LB; Toshiba Medical Systems Corp.) and CBCT (On-Board Imager; Varian Medical Systems Inc.) after anonymization. The CBCT images were acquired 8.0 days (median; range, 7–37) after the planning CT image acquisition. They were resized down to 140 × 270 × 50 voxels to reduce the calculation time. Table 1 presents the imaging parameters.

Table 1.

Imaging parameters

	Planning CT	CBCT
Slice thickness [mm]	2.0	2.5
Pixel width [mm] range	0.877–1.074	0.651–1.172
X-ray tube voltage [kVp]	120	125
X-ray exposure [mAs] median (range)	250 (223–600)	682 (679–724)

We regarded CBCT images as low-resolution images and regarded planning CT images as high-resolution images. To generate a super-resolution database, planning CT images were registered to corresponding CBCT images using B-spline deformable registration. The resulting images are portrayed in Fig. 2. Then we extracted patches of size from planning CT and CBCT images, where denotes the number of elements in each patch. All patches were extracted with overlaps to increase the number of patches. Each element in the patch was represented as a vector ( and ) and was arranged in a column of a matrix. Thus, we constructed the matrices and , where t denotes the total number of patches within an image. We assumed that the k-th patches in these matrices (i.e., and ) are extracted from the same position of those images. By adapting this operation to all images, we generated a high-resolution database () and a low-resolution one (), where s denotes the total number of images, and where and denote the matrix of the k-th image (). The high-resolution database structure is presented in Fig. 3.

Fig. 2.

Preparation images for database creation: CBCT image as a low-resolution image (a), planning CT image as a high-resolution image (b), alpha blending image of CBCT image (red) and planning CT image (green) before registration (c), and alpha blending image after registration (d).

Fig. 3.

Structure of high-resolution database : denotes the vector of the extracted patch from the planning CT image (); denotes a matrix having in each column (); and is the augmented matrix ().

Construction of the low-resolution dictionary

This section presents a method for generating a dictionary from a prepared database . We assumed that input signal , which is extracted from an input image (low-resolution image), can be represented sparsely by over the dictionary , as where . Here, denotes the number of nonzero elements in the vector, called -norm. In addition, is a sparse coefficient vector representing the input signal’s coordinates with fewer than non-zeros.

For this study, we substituted a low-resolution database for and substituted the matrix of combination coefficients for in the equation. A conceptual scheme of this equation is depicted in Fig. 4. The dictionary is learned under the constraint that the combination coefficient is sparse. This problem is presented by the following equation: Therein, denotes the i-th column of with only l or fewer non-zero coefficients. When the sparsity constraint is the -norm as in Eq. (2), this optimization problem is classified as NP-hard [16]. Therefore, we relaxed Eq. (2) under the -norm constraint to solve the problem as shown below. Here, denotes the number of atoms in , represents the k-th column of , and is a regularization parameter.

Fig. 4.

Conceptual scheme showing relations between matrices: is a low-resolution database; is a low-resolution dictionary; and is a combination coefficient.

Then, we adopted functions of Mini-batch Dictionary Learning in scikit-learn (a machine learning library in Python) [17] to optimize Eq. (3) and to generate the low-resolution dictionary . Here, regularization parameter is a fixed numerical value of 1.0. Inferred in Eq. (3) is no longer used in the following sections because it is a by-product in the construction of a low-resolution dictionary.

Construction of the high-resolution dictionary

We solved the following equation and obtained coefficient matrix of sparse representation corresponding to database . In this dictionary learning procedure, the orthogonal matching pursuit (OMP) [16] was used for sparse coding as

In that equation, denotes the i-th column of . This equation presented above is intended to solve the objective function in Eq. (3) as an -norm minimization problem for the coefficient matrix. We can control sparsity (number of non-zero elements) by parameter l.

Finally, we constructed the high-resolution dictionary [13]. We assumed that the high-resolution database is reconstructed using a linear combination of atoms in . Therefore, we sought dictionary , which satisfies the following relation:

The solution of the problem in Eq. (5) is given as the following formula [13]:

Here, represents a Moore–Penrose inverse matrix of . We generated from Eq. (6). The resulting image of dictionary learning is presented in Fig. 5.

Fig. 5.

Estimated high-resolution dictionary (left) and low-resolution dictionary (right). We present two sets of 200 chosen atoms from those dictionaries that have 400 atoms. Each atom is 20 × 20 pixels. Atoms of the same locations in all dictionaries mutually correspond.

Improved image estimation

Image reconstruction to improve the image quality of CBCT was done using the following steps.

Extract patches from an input image in the manner described in ‘Preparation of the database.’ Each patch is arranged in a column (). Here, denotes the number of elements in a patch. denotes the number of patches. Then, we obtained the set of patches .

Apply the OMP algorithm on and find a sparse representation matrix , in Fig. 6 and Fig. 7.

Fig. 6.

Conceptual scheme showing relations among matrices: is an input image; is a low-resolution dictionary; and is a combination coefficient.

Fig. 7.

Visualization of the combination coefficients obtained by orthogonal matching pursuit by atoms = 400 and l = 20. Mapping of the combination coefficients multiplied by the corresponding atoms in (left) is shown (right): mapped zero coefficients are white; non-zero coefficients are gray.

Multiply the matrix of combination coefficients by high-resolution dictionary , and infer a high-resolution set of patches using .

Reconstruct the super-resolution image reconstructed from an input image by replacing the patches in with the patches in and by averaging the overlapped regions.

Quantitative evaluation

For the quantitative evaluation of registration accuracy, we computed normalized mutual information (NMI) and the Dice similarity coefficient (DSC) between the CBCT image and planning CT after registration. For similarity evaluation between images obtained from different modalities, NMI [18] is widely used. From two body contours ( and ), which were extracted by threshold processing, DSC (i.e. ) was computed. The larger NMI and DSC signify that the two images are more similar. Furthermore, for quantitative evaluation of the output images, we computed the root mean square error (RMSE) and the structural similarity (SSIM) [19] between the images improved by super-resolution and the planning CT images.

Root mean square error

The RMSE is the standard deviation of the residuals between two images. Here, we designate a planning CT image as and for a super-resolution image as Here, n denotes the number of pixels. Also, denote the i-th pixels of those images. In general, a smaller RMSE reflects that fewer differences exist between two images.

Structural similarity

The SSIM is one method of evaluating the similarity between two images. Here, suffixes x and y, respectively, denote the super-resolution image and the planning CT image. The average of all pixel values is represented by , the standard deviation of the pixel values by , the covariance of two images by , and fixed numbers by and . In this study, a higher SSIM indicates higher similarity between a super-resolution image and a planning CT image.

Statistical analysis

For this study, Student’s paired t test was used to assess differences between means in pixel values before and after reconstruction. Furthermore, Welch’s t test was used to assess differences between mean in image Improvement and Non-improvement groups. All analyses were conducted using R ver. 4.3.2. Results for which P < 0.05 were inferred as significant.

RESULTS

Registration accuracy

The registration accuracy was computed between the CBCT image and planning CT after registration using NMI and DSC. The mean of NMIs was 0.231 (range 0.209–0.262). That of DSCs was 0.674 (range 0.546–0.794). Table 2 presents Welch’s t test results for those evaluation indexes.

Table 2.

Assessment of differences between NMI and DSC means in images Improvement and Non-improvement groups using Welch’s t test

	n	NMI			DSC
		Mean (min–max)	P value	95% CI	Mean (min–max)	P value	95% CI
I	27	0.232 (0.21–0.26)	0.10	(−0.04, 0.03)	0.675 (0.55–0.79)	0.79	(−0.12, 0.15)
U	3	0.220 (0.22–0.23)			0.664 (0.59–0.70)

CI = confidence interval, I = Improvement group, U = Non-improvement group.

Optimization of parameters in dictionary learning

We demonstrated the application of our algorithm to whole-pelvis CT images of 30 patients. To obtain better results, we optimized the parameters in dictionary learning. One parameter is a patch size for extracting patches from images. During the optimization procedure, we attempted to set the patch sizes in the ranges of 5 × 5 to 50 × 50 pixels while maintaining all other parameters as fixed (Table 3). We evaluated the imaging quality using RMSE and SSIM. Figure 8 shows the improvement of image quality by comparison of those values. The optimum patch size is approximately 15 × 15 to 20 × 20 in Fig. 8. Refined images with a 20 × 20 patch size are presented in Fig. 9. These figures exhibit remarkable improvement in image quality in terms of suppression of cupping artifacts and enhancement of the organ boundaries.

Table 3.

Fixed parameters of dictionary learning

Parameters	Values
Number of slices s	50
Number of atoms m	400
Regularization parameter λ	1.0
Iteration	5000

Fig. 8.

Quantitative evaluation comparison for 30 patients. Patch sizes range from 5 × 5 to 50 × 50. Shown are the root mean square error (RMSE) (a) and the structural similarity (SSIM) (b).

Fig. 9.

Example images of application to pelvis CT images (a). Images of prostate level (left column). Images of bladder level (right column). CBCT images before processing (top row). Processed super-resolution images (middle row). Planning CT images with the same slice as the CBCT image (bottom row). The worst result of quantitative evaluation (b).

Quantitative evaluation result

Results of the quantitative evaluation of the proposed method showed that the values of RMSE between the original CBCT images and the planning CT images at bladder and prostate levels were 72.51–186.55 (minimum–maximum value) for all patients. The values of SSIM were 0.31–0.59. The values of RMSE between the reconstructed images and the planning CT images were 63.79–158.72. Additionally, the values of SSIM were 0.37–0.64. From the quantitative evaluation results, 56 regions out of 60 (two regions per patient) show image quality improvement. Of 30 patients, 27 showed improved image quality in both regions. Those results demonstrate that the values of RMSE and the SSIM were 0.81 and 1.29 times at the maximum (0.90 and 1.13 times at the mean) better than those original CBCT images, respectively. Furthermore, assessment of differences between means before and after reconstruction using Student’s paired t test and 95% confidence intervals (CIs) revealed that the quality of each reconstructed image was improved significantly (Table 4). Figure 10 and Fig. 11, respectively, present quantitative comparisons of pelvis CT images.

Table 4.

Assessment of differences between means in images before and after reconstruction using Student’s paired t test (bladder, n = 30; prostate, n = 30)

		n	RMSE			SSIM
			Mean (min–max)	P value	95% CI	Mean (min–max)	P value	95% CI
B	BR	30	95.4 (72.5–186.5)	0.15	(−2.89, 17.98)	0.46 (0.31–0.57)	<0.01	(−0.08, 0.02)
	AR	30	87.9 (63.8–158.7)			0.51 (0.37–0.59)
P	BR	30	100.8 (75.3–137.5)	<0.01	(4.38, 18.46)	0.46 (0.35–0.59)	<0.001	(−0.10, −0.04)
	AR	30	89.3 (69.8–121.4)			0.53 (0.43–0.64)

CI = confidence interval, B = bladder, P = prostate, BR = before registration images, AR = after registration images.

Fig. 10.

Quantitative comparison of qualities of pelvis CT images with scatter plots of root mean square error (RMSE) (left) and structural similarity (SSIM) (right) for 30 patients. Image quality was evaluated at the prostate and bladder. Markers show that RMSE and SSIM were improved by application of the proposed method to images. CBCT = cone beam CT image, SR = super-resolution image.

Fig. 11.

Root mean square error (RMSE) (left) and structural similarity (SSIM) (right) for 30 patients: pelvis CT images at different slice levels.

Moreover, assessment of differences between two groups (with improvement and without improvement) using Welch’s t test and 95% CIs revealed that the evaluation indexes of original CBCT images in the Non-improvement group showed better values than those of the Improvement group (Table 5).

Table 5.

Assessment of differences between means in images of the Improvement and Non-improvement groups using Welch’s t test

		n	RMSE			SSIM
			Mean (min–max)	P value	95% CI	Mean (min–max)	P value	95% CI
B	I	28	96.2 (72.5–186.5)	0.32	(−31.24, 54.44)	0.46 (0.31–0.57)	0.53	(−0.17, 0.13)
	U	2	84.6 (77.2–92.0)			0.47 (0.45–0.49)
P	I	28	102.3 (79.3–137.5)	<0.05	(7.95, 39.25)	0.46 (0.35–0.59)	<0.05	(−0.08, −0.03)
	U	2	78.7 (75.3–82.1)			0.51 (0.51–0.52)

CI = confidence interval, B = bladder, P = prostate, I = Improvement group, U = Non-improvement group.

DISCUSSION

Artifacts affected CBCT images, thereby reducing their image quality considerably. This paper presents a method for suppression of these artifacts using a super-resolution technique, thereby improving the CBCT image quality. The results of RMSE and SSIM in the super-resolution images were, respectively, as much as 0.81 times and 1.29 times better than those obtained from the planning CT images without the super-resolution technique. In addition, our method achieved high positional accuracy by including the latest internal body information at the time of CBCT imaging.

The results of the visual and quantitative evaluation (RMSE and SSIM) confirmed the image quality improvement. However, in some cases, the image quality was not improved greatly by quantitative evaluation. A possible reason is that those cases have low registration accuracy in the dictionary learning section. From Welch’s t test results, we were unable to find a significant difference between NMI/DSC in the Improvement and Non-improvement groups (Table 2). For three patients in the Non-improvement group, however, NMI were 0.229, 0.216 and 0.215, and DSC were 0.704, 0.695 and 0.592. The NMIs in the Non-improvement group were lower than the mean NMI (0.231). Moreover, the DSC of one case was lower than the mean of DSC (0.674). Therefore, we infer that the registration error influenced those results.

Another possible reason is that the original CBCT images before application of the super-resolution method closely resembled the planning CT images, meaning that the original CBCT images were not degraded to any great degree in comparison with the planning CT images. Table 5 shows that the Non-improvement group has smaller RMSEs and higher SSIM values than those of the Improvement group. Therefore, the original CBCT images in the Non-improvement group were similar to the planning CT images. Such image quality was not improved with respect to the quantitative evaluation. Nonetheless, as presented in Fig. 9b, the results showed suppression of the cupping artifacts and enhancement of the organ boundaries.

The proposed method based on the super-resolution technique achieved higher positional accuracy than a conventional method using a non-rigid function [7-9]. As shown in Fig. 12, it is noteworthy that our method was able to include the latest internal body information at the time of CBCT imaging, e.g. indicating position changes due to flatus or organ movement of the patient. Therefore, this method can play an important role in radiation therapy that requires highly accurate irradiation.

Fig. 12.

Checkerboard images: upper left, a checkerboard of the CBCT image and planning CT image after registration; lower left, a checkerboard of the CBCT image and output image; right, an enlarged view of the part flatus in the left images.

Although we demonstrated the capabilities of our method using whole-pelvis CT images in this study, our method is not restricted to pelvic images. The quality of CBCT images taken from areas of the body other than the pelvis is also expected to be improved using dictionaries constructed from their CBCT images and planning CT images.

Several candidates for improving image quality exist. One approach is to increase the number of images for dictionary composition. For this study, we used a set of patient images per dictionary. The set included 50 images, providing a set of 1 287 750 patch-pairs. In general, the image quality becomes higher when using more data for dictionary learning [13]. If more images are used for learning, then it will be possible to construct a dictionary that can provide higher accuracy. In addition, the reconstructed image quality will be improved by increasing the number of atoms in the dictionary, which would also require increased computation time for the reconstruction [14].

Another approach is to acquire better registration accuracy. The conventional super-resolution method does not rely on the assumption that misregistration exists between image pairs used for dictionary learning. Therefore, misregistration might engender erroneous correspondence of atoms and might influence the reconstructed image quality. For this study, planning CT images were registered to corresponding CBCT images using B-spline deformable registration, but a limit was imposed to ensure high positional accuracy in soft tissues or flatus, as shown in Fig. 2. Even though registration was not perfect, our method showed a marked improvement in CBCT image quality, as shown in Fig. 9. Better registration accuracy will engender better performance of the algorithm.

Our study revealed that the super-resolution technique improves CBCT images from the corresponding planning CT images. This study specifically examined a CBCT image quality improvement technique. The proposed method is applicable not only to parts of the body other than the pelvis, but also to images obtained using other modalities.