A novel adaptive cubic quasi-Newton optimizer for deep learning based medical image analysis tasks, validated on detection of COVID-19 and segmentation for COVID-19 lung infection, liver tumor, and optic disc/cup.

BACKGROUND: Most of existing deep learning research in medical image analysis is focused on networks with stronger performance. These networks have achieved success, while their architectures are complex and even contain massive parameters ranging from thousands to millions in numbers. The nature of high dimension and nonconvex makes it easy to train a suboptimal model through the popular stochastic first-order optimizers, which only use gradient information.
PURPOSE: Our purpose is to design an adaptive cubic quasi-Newton optimizer, which could help to escape from suboptimal solution and improve the performance of deep neural networks on four medical image analysis tasks including: detection of COVID-19, COVID-19 lung infection segmentation, liver tumor segmentation, optic disc/cup segmentation.
METHODS: In this work, we introduce a novel adaptive cubic quasi-Newton optimizer with high-order moment (termed ACQN-H) for medical image analysis. The optimizer dynamically captures the curvature of the loss function by diagonally approximated Hessian and the norm of difference between previous two estimates, which helps to escape from saddle points more efficiently. In addition, to reduce the variance introduced by the stochastic nature of the problem, ACQN-H hires high-order moment through exponential moving average on iteratively calculated approximated Hessian matrix. Extensive experiments are performed to access the performance of ACQN-H. These include detection of COVID-19 using COVID-Net on dataset COVID-chestxray, which contains 16 565 training samples and 1841 test samples; COVID-19 lung infection segmentation using Inf-Net on COVID-CT, which contains 45, 5, and 5 computer tomography (CT) images for training, validation, and testing, respectively; liver tumor segmentation using ResUNet on LiTS2017, which consists of 50 622 abdominal scan images for training and 26 608 images for testing; optic disc/cup segmentation using MRNet on RIGA, which has 655 color fundus images for training and 95 for testing. The results are compared with commonly used stochastic first-order optimizers such as Adam, SGD, and AdaBound, and recently proposed stochastic quasi-Newton optimizer Apollo. In task detection of COVID-19, we use classification accuracy as the evaluation metric. For the other three medical image segmentation tasks, seven commonly used evaluation metrics are utilized, that is, Dice, structure measure, enhanced-alignment measure (EM), mean absolute error (MAE), intersection over union (IoU), true positive rate (TPR), and true negative rate.
RESULTS: Experiments on four tasks show that ACQN-H achieves improvements over other stochastic optimizers: (1) comparing with AdaBound, ACQN-H achieves 0.49%, 0.11%, and 0.70% higher accuracy on the COVID-chestxray dataset using network COVID-Net with VGG16, ResNet50 and DenseNet121 as backbones, respectively; (2) ACQN-H has the best scores in terms of evaluation metrics Dice, TPR, EM, and MAE on COVID-CT dataset using network Inf-Net. Particularly, ACQN-H achieves 1.0% better Dice as compared to Apollo; (3) ACQN-H achieves the best results on LiTS2017 dataset using network ResUNet, and outperforms Adam in terms of Dice by 2.3%; (4) ACQN-H improves the performance of network MRNet on RIGA dataset, and achieves 0.5% and 1.0% better scores on cup segmentation for Dice and IoU, respectively, compared with SGD. We also present fivefold validation results of four tasks. It can be found that the results on detection of COVID-19, liver tumor segmentation and optic disc/cup segmentation can achieve high performance with low variance. For COVID-19 lung infection segmentation, the variance on test set is much larger than on validation set, which may due to small size of dataset.
CONCLUSIONS: The proposed optimizer ACQN-H has been validated on four medical image analysis tasks including: detection of COVID-19 using COVID-Net on COVID-chestxray, COVID-19 lung infection segmentation using Inf-Net on COVID-CT, liver tumor segmentation using ResUNet on LiTS2017, optic disc/cup segmentation using MRNet on RIGA. Experiments show that ACQN-H can achieve some performance improvement. Moreover, the work is expected to boost the performance of existing deep learning networks in medical image analysis.

INTRODUCTION

Deep learning for medical image analysis attracts great attention and achieves great success. It can take care of the simple repetitive and time‐consuming process, help in diagnosing disease, evaluate prognosis, and plan operation. Among numerous medical image analysis tasks, classification and segmentation are the most attractive. Great efforts have been made on designing high‐performance deep neural networks (DNNs), which can even exceed human recognition ability. One of leading DNNs for medical image segmentation is UNet. It adopts an encoder–decoder structure, which enables recovering full spatial resolution. Many variants of UNet , , , , have been proposed for medical image analysis. For instance, H‐DenseUNet makes full use of adjacent computer tomography (CT) volume for model training and achieves improved performance on liver segmentation task. ResUNet introduces a semantic segmentation model with context multiimages input and utilizes a new loss function that combines Dice loss with cross‐entropy loss, which brings faster convergence speed. MRNet explores the utilization of rich annotation information from multiple experts and incorporates the multirater (dis‐)agreement cues which help to generate better prediction.

Recently, COVID‐19 ravages the world. The medical system suddenly suffers great pressure from exponentially increasing number of infections. This inspires the research on deep learning based COVID‐19 automated diagnosis. COVID‐Net is the first open‐source convolutional neural network designed for COVID‐19 detection and achieves good precision. COVID‐Net and its many variants , , augment the traditional healthcare strategy for tackling COVID‐19, while they can hardly be applied to segment infected regions from CT slices faces. This may partly be due to high variation in infection characteristics, low‐intensity contrast between infections and normal tissues, and lack of labeled data. To solve these problems, Inf‐Net is proposed to automatically identify infected regions from chest CT slices. It utilizes implicit reverse attention and explicit edge attention to improve the identification of infected regions.

Most of DNNs mentioned above are always complex with massive parameters ranging from thousands to millions in numbers. The high dimension and nonconvex nature of DNNs make them hard to optimize. , The most popular optimizers are the first‐order ones that are based on first‐order Taylor expansion of loss function. For many applications, SGD, Adam, and AdamW are the default optimizers because of their simplicity and efficiency. Recently, due to its good performance, Adam also engenders an ever‐growing list of modifications, such as AdaBound, Radam, Adabelief, and Adax. However, they are easy to be trapped in suboptimal solutions in algorithmic iterations, which largely due to only utilizing gradient information. Instead, stochastic second‐order optimizers can capture and exploit curvature properties of the loss landscape by incorporating both gradient and Hessian information, leading to better performance. AdaHessian, as an adaptive Hessian‐based optimizer, estimates the Hessian matrix diagonally, Hutchinson's method. The method incorporates spatial averaging for Hessian diagonal which helps in denoising local Hessian information and enables AdaHessian to achieve better generalization. Instead of directly obtaining Hessian information, an alternative is a class of stochastic quasi‐Newton optimizers, , , which approximate the curvature of objective function only using gradient information. Generally, second‐order optimizers require much more computational resource on both time and memory to calculate the Hessian matrix, while quasi‐Newton optimizers are more applicable as they are able to balance performance and efficiency. As an example, for experiments on ImageNet using ResNext, the time cost of second‐order optimizer AdaHessian can reach up to 11.78 and 9.58 times larger than those of first‐order optimizer SGD and recent proposed stochastic quasi‐Newton optimizer Apollo, respectively. Meanwhile, the memory cost of AdaHessian can reach up to 2.51 and 2.39 times larger than those of SGD and Apollo, respectively.

In this work, we propose a novel adaptive cubic quasi‐Newton optimizer with high‐order moment (termed ACQN‐H) for medical image analysis. Different from existing stochastic quasi‐Newton optimizers which usually approximate Hessian only using the curvature of the loss function, the proposed optimizer incorporates both the curvature of the loss function and the norm of difference between previous two estimates. Besides, ACQN‐H hires high‐order moment through exponential moving average on iteratively calculated Hessian approximations to reduce the variance introduced by the stochastic nature of the problem. The performance of ACQN‐H has been validated in four tasks including detection of COVID‐19, COVID‐19 lung infection segmentation, liver tumor segmentation, and optic disc/cup segmentation. Moreover, the work is also expected to boost the performance of other existing deep learning networks in medical image analysis.

We use italics letters such as ε and β to denote scalars, bold lowercase letters x and y to denote vectors, and bold uppercase letters H and D to denote matrices.

METHODOLOGY

In this section, we first provide the formulation of the cubic quasi‐Newton method in Subsection 2.1. Then, we describe the updated process for approximated Hessian matrix in Subsection 2.2. Finally, the form of high‐order moment applied for ACQN‐H is discussed in Subsection 2.3.

Formulation of cubic quasi‐Newton method

Generally, the updated rule of the Newton method can be written as where is the parameter vector updated at kth iteration, and are the gradient vector and Hessian matrix, respectively. Acquiring the exact Hessian needs high computation cost. Instead, the quasi‐Newton method approximates the second derivative of loss function as a series sum of first‐order gradient information from prior iterations, and this is much more efficient. The curvature of the loss function can be acquired through a second‐order Tayler expansion where is the loss function and is the weights of DNNs. The weight update process is shown as where represents the set of values where the object function attains the minimum, is an approximation to the Hessian matrix at . To further enhance the global convergence, the cubic regularization is introduced and the optimal weights can be acquired through finding the minimizer of cubically regularized second‐order Taylor expansion is a sufficient large hyperparameter. By first‐order optimality conditions, we set the derivative of the objective to zero, which immediately yields which is a nonlinear system and can be approximated by a linear one as follows: yielding a novel update: where is the identity matrix. Comparing with (3), (7) additionally makes use of the norm of difference between previous two estimates, leading to better performance. Further, to guarantee the positive‐definiteness, the Newton update in (7) is combined with rectifying operation and becomes where where θ is a positive parameter, operation enables to prevent the step size from becoming arbitrary large since there exists zero value in .

Updating

For simplification, the matrix in (8) is approximated by a diagonal matrix. Thus, acquiring at every iteration becomes computationally feasible since is also a diagonal matrix. Here, can be updated according to the quasi‐Cauchy equation : where and . The solution to the above problem with the Frobenius matrix based on the variational technique in Zhu et al. is given by where is the diagonal matrix with diagonal elements from vector .

High‐order moment

To reduce the variance and further improve the performance, we adapt the moments for both gradient and diagonally approximated Hessian. The first moment is defined as where β1 is the first moment hyperparameter. The high‐order moment is shown as follows: where is the hyperparameter and h represents the order of the second moment. The second moment utilizes historical second‐order derivatives to smooth the noisy curvature information. Generally, in many image classification and segmentation tasks, Adam, AdamW, and AdaHessian would set , . We use the same setting to enable a fair comparison.

To summarize, the complete algorithm of adaptive cubic quasi‐Newton optimizer with high‐order moment (ACQN‐H) is given in Algorithm 1. In which, at most first‐order gradients are required, and , , and are all diagonal. Therefore, ACQN‐H iteratively updates with linear complexity for both time and memory.

Performance evaluation

To access the performance of ACQN‐H, the optimizer is extensively tested on a wide range of learning tasks: detection of COVID‐19, COVID‐19 lung infection segmentation, liver tumor segmentation, and optic disc/cup segmentation. The results in each task are compared with stochastic first‐order optimizers like Adam, SGD, AdaBound, and stochastic second‐order optimizer Apollo. Among them, Adam and SGD are the most common and default optimizers for these tasks, AdaBound is a recently proposed first‐order optimizer that works well. For each task, fivefold cross‐validation results are reported. The tested learning tasks are briefly explained below:

Detection of COVID‐19: We experiment on the COVID‐chestxray dataset using COVID‐Net. The training set consists of 7966 normal chest X‐rays and 8599 X‐rays of negative samples, while the test set contains 885 normal X‐rays and 956 negative X‐rays. Besides, to broadly test its performance, we set VGG16, ResNet50, and DenseNet121 as backbones instead of only one default ResNet50 backbone. Besides, the total training epoch is 300. Although there are image samples from the same patient, we only want to test the performance of the optimizer under the same conditions as those of COVID‐Net.

COVID‐19 lung infection segmentation: We report the performance of ACQN‐H on the COVID‐CT dataset using a supervised version of the Inf‐Net model. COVID‐CT is a COVID‐19 CT segmentation dataset with 100 labeled CT slices, which consists of 45 CT images for training, 5 CT images for validation, and the remaining 50 images for testing. Additionally, the total training epoch is 100.

Liver tumor segmentation: We use ResUNet on the LiTS2017 dataset, which also served as a segmentation challenge during MICCAI 2017. The training set of LiTS2017 contains 50 622 abdominal scan images of 130 CT scans from 91 patients while the test set contains 26 608 images of 70 CT scans from 40 patients. The total training epoch is 300.

Optic disc/cup segmentation: We report experiments using the MRNet model on the RIGA dataset, which contains in total of 750 color fundus images. Followed with the experiment setting in MRNet, 655 samples are selected as the training set and 95 samples consist of the test set. Moreover, the total training epoch is 60.

Experiment environment. The deep neural network framework we experiment on is Pytorch1.7.1 with python3.6 and is GPU‐accelerated. The hardware is a single RTX 3090Ti with I9‐10920X CPU, while the RAM is 32GB.

Experiment setup. We perform a careful hyperparameter tuning in experiments as follows:

ACQN‐H: We set , , , , . As if a diagonal element of the approximated Hessian matrix is less than 1, the corresponding element in becomes 1. Thus, the update of this element can work as that of SGD and prevent the step to be arbitrarily large. Besides, we do not tune ρ and θ on different problems, which may help to reach a better result. The learning rate η is set to 0.1. In addition, for each task, we search the best order h from 2.0 to 10.0.

SGD: The momentum is set to 0.9, while the learning rate is searched among , where and .

Adam, AdaBound, and Apollo: The learning rate is searched as SGD, and other parameters are set as their own default values in the literature.

Evaluation metrics. In the medical image classification task detection of COVID‐19, the commonly used classification accuracy is utilized as the evaluation metric. For the other three medical image segmentation tasks, we integrate the default evaluation metrics into the following seven evaluation metrics, that is, Dice, structure measure (SM), enhanced‐alignment measure (EM), mean absolute error (MAE), intersection over union (IoU), true positive rate (TPR), and true negative rate (TNR). Among these metrics, Dice, SM, EM, MAE, and IoU can measure the similarity between the result and ground truth. TPR means the correct rate of correctly segmented pixels of a target region, and TNR represents the correct proportion of background pixels that are segmented correctly. Assuming and represent the normal region and the ground truth (GT), respectively, is the predicted normal region, and means the predicted segmentation region, means the number of matrix elements. The seven evaluation metrics can be formulated as follows:

Dice:

Structure measure: where , is the object‐aware similarity and represents the region‐aware similarity.

Enhanced‐alignment measure: where w and h are the width and height of the input CT image, respectively, and represents the enhanced alignment matrix.

Mean absolute error:

Intersection over union:

True positive rate:

True negative rate:

RESULTS

Detection of COVID‐19

To assess the generalization performance of ACQN‐H on medical image classification, we also use COVID‐Net with VGG16, ResNet50, and DenseNet121 as backbones on the COVID‐chestxray dataset, and results are shown in Table 1, ACQN‐H outperforms other optimizers on classification accuracy in all experiments and achieves 0.38%, 0.16%, and 0.16% higher accuracy than Apollo with the backbones VGG16, ResNet50, and DenseNet121, respectively. Test accuracy curves are reported in Figure 1. As can be seen, the test accuracy of ACQN‐H is better than that of other optimizers. Moreover, fivefold cross‐validation results using ACQN‐H are reported in Table 2. It can be found that the average Dice values among different folds vary slightly.

TABLE 1

Test accuracy of COVID‐Net with COVID‐chestxray

Backbone (%)	Adam	SGD	AdaBound	Apollo	ACQN‐H
VGG16	95.44	95.44	96.25	96.36	96.74
ResNet50	94.41	94.19	95.49	95.44	95.60
DenseNet121	95.17	95.27	95.82	96.36	96.52

FIGURE 1

Test accuracy curves of COVID‐Net on COVID‐chestxray using VGG16, ResNet50, and DenseNet121 as backbone

TABLE 2

Quantitative results of fivefold cross‐validation using COVID‐Net with COVID‐chestxray

	Validation set			Test Set
Backbone(%)	VGG16	ResNet50	DenseNet121	VGG16	ResNet50	DenseNet121
Fold‐0	99.01	98.95	99.21	97.21	95.95	97.61
Fold‐1	98.55	98.15	98.65	96.05	95.15	95.97
Fold‐2	97.62	98.02	98.82	95.92	95.32	96.32
Fold‐3	97.86	97.36	97.93	97.86	95.36	97.13
Fold‐4	98.66	97.72	98.54	96.66	96.22	95.57
Avg	98.34	98.04	98.63	96.74	95.60	96.52

COVID‐19 lung infection segmentation

In the COVID‐19 lung infection segmentation task, we experiment on the Inf‐Net model with the COVID‐CT dataset. The performance of ACQN‐H is evaluated through six widely adopted metrics, that is, Dice, TPR, TNR, SM, EM, and MAE. For metric MAE, lower is better and for other metrics, higher is better.

Table 3 presents quantitative results of COVID‐19 lung infection segmentation. It shows that ACQN‐H has the best scores in terms of Dice, TPR, EM, and MAE. Particularly, ACQN‐H achieves 1.0% better Dice as compared to Apollo. Figure 2 also gives some visual comparison examples of COVID‐19 lung infection segmentation. Comparing with other optimizers, ACQN‐H yields infection segmentation results with more accurate boundaries. Besides, Table 4 shows quantitative fivefold validation results of COVID‐19 lung infection segmentation in terms of Dice. As can be seen, the variance on the test set is much larger than on the validation set. Moreover, the performance on the test set drops about 9% on average. The potential reason could be the small size of the dataset.

TABLE 3

Assessment of Inf‐Net with COVID‐CT. (For metric MAE, lower is better, for other metrics, higher is better)

(%)	Adam	SGD	AdaBound	Apollo	ACQN‐H
Dice	68.7	57.4	64.8	69.0	70.0
TPR	68.1	79.2	65.9	68.2	68.4
TNR	94.9	85.7	94.3	95.6	95.4
SM	76.5	63.8	75.0	76.9	76.0
EM	84.9	72.4	83.8	87.3	88.7
MAE	7.7	14.9	8.9	7.7	7.4

FIGURE 2

Segmentation results on COVID‐CT with Inf‐Net

TABLE 4

Quantitative results of fivefold cross‐validation using COVID‐CT with Inf‐Net

%	Validation set	Test set
Fold‐0	78.50 ± 0.10	68.51 ± 0.57
Fold‐1	77.90 ± 0.14	70.87 ± 0.47
Fold‐2	79.71 ± 0.10	65.45 ± 0.90
Fold‐3	80.17 ± 0.10	73.31 ± 0.65
Fold‐4	81.77 ± 0.07	71.86 ± 0.62
Avg.	79.61 ± 0.11	70.02 ± 0.59

Liver tumor segmentation

We experiment with the ResUNet model on the LiTS2017 dataset and evaluate the results with assessments Dice, IoU, TPR, and TNR. For all these evaluation metrics, higher is better.

Quantitative results of liver tumor segmentation are shown in Table 5. As can be seen, ACQN‐H achieves the best results and outperforms Apollo in terms of Dice by 2.3%. Figure 3 gives some visual comparison examples of liver tumor segmentation. We can see that the segmentation results using ACQN‐H are more similar to the GTs than other optimizers. Besides, quantitative fivefold validation results of liver tumor segmentation in terms of Dice are shown in Table 6. As can be seen, the variance among results in each fold is low. Moreover, the relative performance on the test set drops only about 2.14%.

TABLE 5

Assessment of ResUNet on LiTS2017. (For Dice, higher is better, while for other evaluation metrics, lower is better)

(%)	Adam	SGD	AdaBound	Apollo	ACQN‐H
Dice	91.22	90.87	90.62	91.20	93.46
TPR	97.53	98.48	98.40	98.77	98.77
TNR	86.90	84.77	84.80	85.37	89.26
IoU	84.43	83.58	83.20	84.14	88.03

FIGURE 3

Segmentation results on LiTS2017 with ResUNet. The red pixels denote the liver region.

TABLE 6

Quantitative results of fivefold Cross‐validation using ResUNet with LiTS2017

%	Validation set	Test set
Fold‐0	95.03 ± 0.09	94.04 ± 0.04
Fold‐1	95.12 ± 0.09	93.13 ± 0.04
Fold‐2	96.68 ± 0.09	94.01 ± 0.04
Fold‐3	95.45 ± 0.09	93.11 ± 0.04
Fold‐4	95.72 ± 0.09	93.01 ± 0.04
Avg	95.60 ± 0.09	93.46 ± 0.04

Optic disc/cup segmentation

In the optic disc/cup segmentation task, we experiment on the RIGA dataset with the MRNet model, and the performance is evaluated through IoU and Dice. We present quantitative experiments in Table 7. Obviously, ACQN‐H has advantage on cup segmentation and achieves 0.6% and 1.3% better scores for Dice and IoU, respectively, when comparing with Apollo. ACQN‐H also achieves comparable results on disc segmentation. Figure 4 presents some visualized segmentation results. From the first row, all optimizers achieve excellent performance on optic disc segmentation, and their results are similar to the GT. Meanwhile, the second row shows that the optic cup segmentation result using ACQN‐H achieves superior performance.

TABLE 7

Assessment of MRNet with RIGA. (Higher is better)

(%)	Adam	SGD	AdaBound	Apollo	ACQN‐H
Disc Dice	94.9	97.7	97.5	97.6	97.6
Disc IoU	90.5	95.5	95.1	95.3	95.4
Cup Dice	83.3	83.3	81.9	83.2	83.8
Cup IoU	73.2	73.2	71.5	72.9	74.2

FIGURE 4

A sample of segmentation result on RIGA with MRNet. (a) represents the original image (above) and ground truth (below). The segmentation boundaries of GT (green) and the predicted optic disc (red) for different optimizers are shown in the first row of (b)–(f), while the results of cup segmentation are shown in the second row.

We also present quantitative fivefold validation results of optic disc/cup segmentation in Table 8. It can be found that optic disc segmentation can achieve high performance with low variance. However, the performance of optic cup segmentation is not as good as optic disc segmentation, and the average Dice drops about 14% on the test set with obvious larger variance. This indicates that optic cup segmentation remains a challenging problem.

TABLE 8

Quantitative results of fivefold Cross‐validation using MRNet with RIGA

	Optic disc		Optic cup
Subtask (%)	Validation set	Test set	Validation set	Test Set
Fold‐0	99.01 ± 0.02	97.81 ± 0.02	88.36 ± 0.14	84.24 ± 0.68
Fold‐1	98.70 ± 0.03	97.15 ± 0.02	88.18 ± 0.14	82.31 ± 0.71
Fold‐2	98.57 ± 0.01	96.47 ± 0.02	86.97 ± 0.10	83.55 ± 0.62
Fold‐3	99.25 ± 0.01	98.32 ± 0.02	86.13 ± 0.14	83.97 ± 0.53
Fold‐4	99.07 ± 0.01	98.25 ± 0.01	87.61 ± 0.17	84.93 ± 0.57
Avg	98.92 ± 0.02	97.60 ± 0.02	87.45 ± 0.15	83.81 ± 0.64

DISCUSSION

In this paper, we present a novel ACQN‐H for medical image analysis. Our method can capture the curvature of the loss function by diagonally approximated Hessian and the norm of difference between previous two estimates. Additionally, ACQN‐H hires high‐order moment through exponential moving average on iteratively calculated Hessian approximations. The method can help to escape from saddle points more efficiently and train a DNN with better performance.

ACQN‐H is evaluated through a wide range of medical image analysis tasks using state‐of‐the‐art models. These include COVID‐chestxray for the detection of COVID‐19 using COVID‐Net, COVID‐CT for COVID‐19 lung infection segmentation using Inf‐Net, LiTS2017 for liver tumor segmentation using ResUNet and RIGA for optic disc/cup segmentation using MRNet. For the detection of COVID‐19, quantitative results are reported in Table 1. As a medical image classification task, ACQN‐H consistently achieves the best accuracy with different backbones. Test accuracy curves are reported in Figure 1, from which we can see that ACQN‐H has a better convergence than Adam, SGD, and AdaBound. Quantitative results of COVID‐19 lung infection segmentation are shown in Table 3, from which ACQN‐H achieves the best score in terms of dice, EM, and MAE, only slightly inferior to Apollo in SM and TNR. This implies the segmentation results from ACQN‐H are more similar to GTs when comparing with Adam, SGD, and AdaBound. In terms of TPR and TNR, the resulting image of ACQN‐H has the least proportion of missegmented pixels. In addition, Figure 2 also gives some visual comparison examples. The result of ACQN‐H segment the regions infected with COVID‐19 more accurately. Results of liver tumor segmentation are shown in Table 5 and Figure 3. Obviously, ACQN‐H outperforms other optimizers in terms of quantization and segmentation. As quantitative results of optic disc/cup segmentation shown in Table 7, ACQN‐H achieves the best dice and IoU on optic cup segmentation, which implies that ACQN‐H can better segment optic cups as GT. For optic disc segmentation, ACQN‐H is only slightly lower than Adam which is better than other optimizers. Figure 4 gives some visualized segmentation results. ACQN‐H has a visually better optic cup segmentation result, while its disc segmentation result is comparable.

We find that the segmentation results do not match the GTs well on COVID‐19 lung infection segmentation and optic cup segmentation. This has resulted from two main reasons. At first, the datasets COVID‐CT and RIGA have limited labeled cases. Second, the test samples are complex from a visual point. For example, the samples in the COVID‐19 lung infection segmentation task include many small infections with irregular margins. For the optic cup segmentation task, the cases fail most often due to a weak boundary between the optic disc and optic cup. Thus, these two tasks are much more challenging, while ACQN‐H still achieves the best results in visual.

Figure 5 shows the violin plots of COVID‐19 lung infection, liver tumor, and optic disc/cup segmentation results using different optimizers in terms of Dice. It gives the summary statistics and the entire distribution of the quantitative results. As can be seen, ACQN‐H achieves the best lower quartile, median, and upper quartile in all tasks, which indicates that most cases segmented using ACQN‐H get higher Dice.

FIGURE 5

The violin plots present the dice of different optimizers for COVID‐19 lung infection segmentation, liver tumor segmentation, and optic disc/cup segmentation.

CONCLUSION

We have proposed ACQN‐H, a novel and efficient adaptive cubic quasi‐Newton optimizer with a high‐order moment for medical image analysis, and its superiority is demonstrated on four types of datasets. ACQN‐H only requires at most first‐order gradients and updates with linear complexity for both time and memory, thus it is quite suitable for large‐scale deep learning based medical image analysis and is expected to boost the performance of existing DNNs for medical image analysis.

CONFLICT OF INTEREST

The authors have no conflict to disclose.

Require: n_g        //Mini‐batch size

Require: η        //Stepsize

Require: β_1,β_2∈[0,1)        //Exponential decay rates for the moment estimates

Require: ε,ρ,θ,h        //Positive parameters

Require: x 0, g 0, B 0, m 0, V 0        //Initialize variables

Require: k←0        //Initialize timestep

1: while x k not converged do

2:  k←k+1

3:  g_k←∇f_i(x;ξ_i)        //Stochastic gradient at timestep k

4:  s_k←x_k−x_k−1

5:  y_k←g_k−g_k−1

6:  B_k←B_k−1+skTy_k−skTB_k−1s_k||s_k||44+εDiag(sk2)        //Update diagonal Hessian

7:  D_k←max(abs(B_k+ρ2||x_k−x_k−1||·I),θ·I)        // Rectify for positive‐definiteness

8:  m_k←(1−β1k−1)β_1m_k−1+(1−β_1)g_k1−β1k        //Update first moment

9:  V_k←(1−β2k−1)β_2V_k−1+(1−β_2)Dkh1−β2k            //Update high‐order second moment

10:  x_k+1←x_k−ηV_k^−1hm_k        //Update parameters

11: end while

12: return x_k+1