Automatic Detection and Tracking of Marker Seeds Implanted in Prostate Cancer Patients using a Deep Learning Algorithm.

PURPOSE: Fiducial marker seeds are often used as a surrogate to identify and track the positioning of prostate volume in the treatment of prostate cancer. Tracking the movement of prostate seeds aids in minimizing the prescription dose spillage outside the target volume to reduce normal tissue complications. In this study, You Only Look Once (YOLO) v2™ (MathWorks™) convolutional neural network was employed to train ground truth datasets and develop a program in MATLAB that can visualize and detect the seeds on projection images obtained from kilovoltage (kV) X-ray volume imaging (XVI) panel (Elekta™).
METHODS: As a proof of concept, a wax phantom containing three gold marker seeds was imaged, and kV XVI seed images were labeled and used as ground truth to train the model. The projection images were corrected for any panel shift using flex map data. Upon successful testing, labeled marker seeds and projection images of three patients were used to train a model to detect fiducial marker seeds. A software program was developed to display the projection images in real-time and predict the seeds using YOLO v2 and determine the centers of the marker seeds on each image.
RESULTS: The fiducial marker seeds were successfully detected in 98% of images from all gantry angles; the variation in the position of the seed center was within ± 1 mm. The percentage difference between the ground truth and the detected seeds was within 3%.
CONCLUSION: Our study shows that deep learning can be used to detect fiducial marker seeds in kV images in real time. This is an ongoing study, and work is underway to extend it to other sites for tracking moving structures with minimal effort. Copyright:

INTRODUCTION

External beam radiotherapy has become a standard treatment modality for the management of cancer patients; more than 60% of patients receive radiotherapy either alone or in combination with other treatment modalities such as chemotherapy, immunotherapy, and surgery. In most centers, external beam radiotherapy delivered under image guidance has become the standard procedure for managing most radiotherapy patients. Image guidance aids in reducing patient set-up errors and monitors any changes in patient position during treatment. Cone-beam computed tomography (CBCT) is commonly used in radiotherapy for setting up the patient to the treatment isocenter and provides guidance for defining site-specific margins for generating planning target volume from either gross tumor volume (GTV) or clinical target volume (CTV). Stereotactic ablative radiotherapy (SABR) is a type of treatment technique where high doses of radiation are administered to small tumor volumes with high precision in a small number of fractions, typically between 1 and 5. It can be used for a wide variety of malignancies such as lung, liver, prostate, spine, kidney, pancreas, and bone. The delivery of high doses per treatment fraction demands tight margins to minimize the probability of normal tissue complications for surrounding organs at risk. Tight margins around the GTV or CTV are achievable if tumors are static during treatment delivery. For tumor sites such as the prostate, liver, kidney, and pancreas which are affected by gas, respiration, or bowel movements, it is more challenging as they are highly prone to movement during radiotherapy. Prostate cancer is the second most common cancer in Australia, with 19,500 cases diagnosed in 2019.[1] The goal of treatment is to deliver a therapeutic dose to the tumor to effectively reduce its size while sparing the neighboring organs and tissues, typically the rectum, bladder, and seminal vesicles.[2] One of the main issues with this objective is that due to being made up of soft tissue, the prostate can be difficult to visualize on scans or images. Another problem is that the male prostate is the size of a walnut and continues to grow gradually to the size of a lemon by the age of 60. Bladder and rectal filling can very easily displace the position of the prostate during treatment, causing the dose to be administered to the normal tissues instead of the tumor volume.[3] This can have severe implications on the patient's health, causing unnecessary side effects and potentially compromising the effectiveness of the treatment. For this reason, a lot of research is being undertaken to develop tools and techniques that can help radiation oncologists visualize and predict the position of the prostate to provide effective treatment during radiotherapy.

Some of these real-time imaging techniques are ultrasound imaging,[45] electronic portal imaging device (EPID) (megavoltage [MV]) based tracking,[67] kilovoltage (kV)/MV image-guided tracking,[8] the Calypso electromagnetic system,[9] SeedTracker,[1011] kV Intrafraction Monitoring (KIM),[1213] and magnetic resonance imaging (MRI)/linac integration system.[1415]

SABR has a much higher dose per fraction than conventional techniques and requires superior accuracy and intrafraction motion tracking. In 2004, Keall et al.[6] developed an EPID based tracking system for tumor targeting. In 2007, Kupelian et al.[9] presented the first report on the clinical use of the Calypso system in radiotherapy of prostate cancer patients. The Calypso system is based on electromagnetic detection of transponders implanted in the prostate. Another approach was demonstrated by Cho et al. in 2009[8] in which fiducial markers were tracked using kV and MV imaging and a dynamic Multileaf Collimator (MLC) beam. In 2010, Schlosser et al.[4] designed and built a robotic ultrasound imaging system that can be used for real-time soft-tissue imaging.

Radio-opaque fiducial markers are often used to find the position of the prostate prior and during treatment fractions using CBCT, giving high contrast for image guidance.[151617] Marker seeds enable tracking of tumor volumes on a linac with an onboard imaging system as (1) soft-tissue contrast is relatively poor and (2) only a two-dimensional image is available for real-time tracking using the onboard imager. The use of fiducial markers has become a standard practice in most centers to monitor intrafraction motion. The fiducial markers are implanted through an invasive procedure placed inside the prostate using transrectal ultrasound while the patient is sedated.[18] Once all three marker seeds are placed inside the prostate, a CT scan is done after 1 week to check their positions relative to prostate anatomy. Software such as SeedTracker and KIM are currently being evaluated to monitor the motion of prostate fiducial markers using the onboard kV imaging system.[10111213] The KIM system developed by Keall et al.[19] uses three-dimensional (3D) probability maps to find the position of the prostate volume, while the SeedTracker uses a marker enhancement filter (MEF) developed by Peshko et al.[20] to outline and detect the seed during treatment. Gehrke et al. developed a model to automatically track seed implants in prostate and lung cancer patients.[16] Monitoring the tumors allow real-time adjustment of treatment to ensure that the maximum dose is delivered to the targeted tumor while minimizing the dose to the surrounding healthy tissues. Typically, this is implemented by halting treatment delivery if seeds move beyond specified tolerances and then correcting the patient set-up for the remainder of the delivery. This translates into reducing short- and long-term side effects, improving cancer outcomes. The use of mathematical methods is mostly based on image contrast, and it should be noted that marker seeds' position tends to vary as the gantry rotates around the patient and so does the contrast due to wide-ranging radiological thicknesses. Mathematical methods do fail when seeds overlap and situations where it gets obscured at certain gantry angles. The above studies use simple mathematical functions to calculate the seed positions but struggle to identify the markers at lateral gantry angles due to the patient's lateral separation and superimposition of the right and left femur. Moreover, the above seed detection software is yet to be released for routine clinical use. Artificial intelligence (AI) is gaining significance in all fields and replacing traditional methods. Deep learning, a subset of AI, is a potential tool that could be employed to tackle most of the issues associated with applying mathematical models for real-world problems.

Deep learning

Deep learning is a fast-growing machine learning field with multiple uses in fields such as image recognition, image generation, self-driving vehicles, and medical sciences. As computer hardware and processing speed increase, more and more data are available to train and verify deep learning networks. Deep learning architecture consists of multiple interconnected layers of weighted elements analogous to neurons. Convolutional neural networks are often used for image processing, as the use of small convolutional filter sizes allows efficient encoding of spatial information in the data processed by the network.

In a convolutional neural network, the input layer contains all the input data, such as pixels in an image, and passes them on to the next layer. Subsequent layers convolve different filters to the input data to find specific features such as edges or lines. Each layer works on different features and has connections to nodes in subsequent layers. The final output layer gives a probability score on what the image contains, based on the ground truth. The network is trained by adjusting the weights and biases throughout the network and then comparing the output with the ground truth.

You Only Look Once v2 Architecture

You only look once (YOLO) is a real-time object detection system introduced and created by Redmon et al. in 2016.[21] In the YOLO model, a single convolution network predicts multiple bounding boxes. It also predicts the class probabilities for those boxes. The YOLO v2 architecture consists of 24 convolution layers. Batch normalization is applied to all convolution layers in the network to improve model convergence. Anchor boxes are predicted by fully connected layers on top of convolution layers.[2122] Anchor boxes refer to a set of predefined bounding box shapes selected to match ground truth bounding box sizes.

In previous detection systems such as region-based convolution neural networks, a search algorithm is used to generate around 2000 regions in an image and then classify them.[23] Significant postprocessing was required to ensure that no region in the image has duplicate detections. This was a very time-consuming process and did not work well on real-time data. YOLO, on the other hand, looked at any image once and predicted where the objects are along with their probability scores. This meant that this new system could be used in real-time scenarios where quick detection is needed.

In YOLO, an image is split in a grid of S × S pixels, and B bounding boxes are taken within each grid of S × S. For each box, a class probability C is given as an output, and the boxes having probabilities higher than a threshold value are chosen.

For example, a bounding box will have these 5 predictions (x, y, w, h, confidence). The (x, y) values are the top-left coordinates of the bounding box, w is the width and h is the height of the box. Confidence is scored of intersection over union (IoU) within the predicted and ground truth bounding boxes if the object is present. If the object is not present, the confidence is scored as zero. The confidence score tells us the likelihood of the box containing an object and accuracy of the bounding box.[22]

It is given by Confidence = P(object)* IoUtruthpred where P (object)= 1 if the object exists in the box, P (object)= 0 or otherwise.

The following equation gives us the class-specific confidence scores for each box:

Pr (Classi Object) * Pr (Object) * Pr (Object) * IOUtruthpred=

Pr (Classi) * IOUtruthpred (1)

Imager flex map

The force of gravity acting on the imaging system mounted on a linear accelerator changes with respect to gantry angle, causing the imaging system to flex. To quantify this flex with the gantry rotation, a ball-bearing phantom is placed at the isocenter, and it is imaged through deliverable gantry angles.[2425] Deviations in the ball-bearing position in the images are used to generate a flex map [Figure 1] used as a correction factor to find the true position of the seed centers.

Figure 1

Flex (gantry sag) in the kilovoltage panel from gantry angles − 180° to + 180°

Deep learning models can detect implanted marker seeds in locations where the conventional algorithms struggle to identify the fiducials. This is different from the relatively straightforward task of reconstructing seed positions from a CBCT taken before patient treatment for image guidance. In this study, we have developed an object detector using the YOLO v2 deep learning algorithm and applied it to CBCT images to detect fiducial marker seeds. Our object detector is applied to the images obtained from the XVI imager to draw a bounding box around each seed. The center of the seeds is calculated based on the pixel position of the center of the bounding box, and imager flex correction is applied to find the deviation between the seeds as the gantry angle changes during treatment. The software program used for this project was written in MATLAB[2126] using the YOLO v2 deep learning toolbox.

METHODS

The images used for this research were intrafraction images from a single fraction acquired from 3 patients during prostate treatments monitored using the SeedTracker system. In addition, a set of data was acquired using a wax phantom with implanted seeds. Software design and implementation were performed using MATLAB R2019b software (MathWorks, Natick MA, USA).[26] The CBCT images were acquired from the kV X-ray volume imaging (XVI) system of the Elekta linear accelerator. These images were acquired in the proprietary .his format was then converted into png images so they could be easily read by the software. The images were named using gantry angle and sequence information in the filename from the frames.xml file generated by XVI during image acquisition. For example, GA_1.5_313.png tells us that this image was taken at a gantry angle of 1.5° and the sequence according to the frames.xml file was 313.

These. png images were then cropped using the imcrop function to focus on the seeds and reduce the image size from 512 × 512 pixels to 201 × 201 pixels. They were then preprocessed using the imadjust function to increase the contrast in the image. The repmat function was used to convert the gray scale images into RGB images with identical red, green, and blue channels. RGB images were necessary for use with the pretrained model used for this study. The resultant images were 201 × 201 × 3 pixels.

The cropped images were divided into three sets, 60% of the images were used for training the model, 20% were used for model validation, and 20% were used for testing. For simplicity sake, three separate folders, TrainingImages, ValidationImages, and TestImages, were created in the MATLAB workspace. The first step in training the model was to label the seeds, and this was done using the Image Labeler app in MATLAB. A total of 1500 images were manually labeled by drawing bounding boxes around each of the 3 marker seeds in every image. The average size of a bounding box was 10 × 10 pixels as the seeds were very small in size.

A pretrained network (ResNet-50) was used as the base network for feature extraction. ResNet-50 has a network structure composed of a series of 13 skip connected building blocks, each consisting of three repeated Convolution, Batch Normalization, and Rectified Linear Unit (ReLU) layers. The feature extraction layer, representing the input for the YOLO v2 network, was chosen to be “activation_40_relu,” representing the ReLU layer beyond the 13th building block. The YOLO v2 network consisted of a set of two repeated Convolution, Batch Normalization, and Rectified Linear Unit (ReLU) layers.

The YOLO v2 layer function was used to create the network, and the parameters passed into this function were the image size, number of classes, anchor box size, base network name, and the feature layer. To detect seeds, the number of classes was restricted to 1 (representing the seed) in our case. The size of the anchor boxes was selected based on the size of the seeds; the initial anchor box size was set to (10 × 10). The computer vision toolbox was used to help estimate the anchor box size.

The detector was trained using the trainYOLOv2ObjectDetector function. The “Adam” optimizer was used as a solver, the initial learning rate was set to 1e-4, and 50 epochs were chosen for the training. The total training time taken for 1500 images was 38 h on a single central processing unit. At the end of the training, the detector was saved to the workspace. This could then be saved as a detector and used for validation and testing of the test image datasets.

The software program was designed in the MATLAB App Designer module to access the images from the original seed location, process them, and display with the centers of the seeds marked. The center of each seed was assumed to be the center of the seed's bounding box. Two plots below the image show the seed centers in the superior-inferior plane and the anterior-posterior/lateral plane using the coordinates obtained from the image on detection. Flex map data as acquired from the frames.xml file were used to adjust all values.

To determine the agreement or shift between the planned and treatment fiducial markers, the coordinates of the contoured fiducial markers were obtained directly from the treatment planning system (TPS) and projected onto the XVI images using the following method.

We transformed the coordinate system by a rotation matrix shown below:

where θ is the gantry angle, x, y, and z and are the local coordinates of the seeds, and x',y',z' are the transformed coordinates of the seeds with the reference frame rotated by angle θ.

The transformed coordinates of the seeds were then scaled for projection on the XVI imaging panel, and a flex map correction was applied.

Source-image distance (SID) is the distance from the source to the imaging panel. Source-to-object distance (SOD) is the distance from the source to the isocenter. x˝ and y˝ are the coordinates after the scaling.

RESULTS

The training was done using 1500 images, including both patient and phantom images, with a single label applied for all seeds. Some of the datasets were imaged with clockwise (CW) gantry rotation, and some were imaged with counter CW gantry rotation. This was considered when the flex map corrections were applied. Detection was accurate, with 98% of the seeds detected using the object detector.

The images in Figures 2 and 3 show an example of the results obtained for the datasets using our software. Figure 2a and b depict the original and detected seeds for the gantry angle at 0.3°, Figure 2c and d depict the seeds at gantry angle 45.3°, Figure 2e and f show the seeds at gantry angle 90.2°, and 2 g and h show the seeds at gantry angle 134.9°. Similarly, the images in Figure 3a-h show the seeds at various gantry angles.

Figure 2

Original and detected images from gantry angles 0.3 degrees (a & b), 45.3 degrees (c & d), 90.2 degrees (e & f) and 134.9 degrees (g & h)

Figure 3

Original and detected images from gantry angles 180.2 degrees (a & b), 225 degrees (c & d), 270.1 degrees (e & f) and 315.5 degrees (g & h)

Flex map adjustment result

Figure 1 shows the flex map plot for the Elekta Axesse linear accelerator used to correct the position of the seed centers.

To find the value of the seed center, the following equation has been used to correct for the flex in our calculation.

Where x and y are the centers of the seeds in the x and y direction.

x and y are the pixel coordinates of the seeds in the x and y direction.

x and y are the interpolated flex coordinates of the seeds in the x and y direction.

P is the mm value of each pixel.

M is the magnification factor given by

SID is the distance from the source to the imaging panel.

SOD is the distance from the source to the Isocenter.

The value of P was found to be 0.518 mm from the XVI panel. The source to imaging panel distance was 1536 mm, and the source to isocenter distance was 1000 mm. In this case, the flex map values were found to be within the ± 3 mm range, as shown in the plot in Figure 1.

Difference between the ground truth and detected values

Various gantry angles were chosen to determine the difference between the ground truth and the detected seed coordinates. Figures 4, 5 and 6 show the variation in the seed coordinates in the Y (superior/inferior [SI]) and X/Z (AP/Lateral) directions for random gantry angles. All three plots show that the percentage difference is within a range of 3%. Tables 1 and 2 display the mean and standard deviation error between ground truth and detected values in the Y and X/Z directions for three patients.

Figure 4

Plot for the percentage variation in the ground truth and detected seed 1 coordinates for various gantry angles

Figure 5

Plot for the percentage variation in the ground truth and detected seed 2 coordinates for various gantry angles

Figure 6

Plot for the percentage variation in the ground truth and detected seed 3 coordinates for various gantry angles

Table 1

Mean and standard deviation between ground truth and detected values (superior/inferior direction) for various gantry angles

	Seed 1 (mm)		Seed 2 (mm)		Seed 3 (mm)
	Ground truth	Detected	Ground truth	Detected	Ground truth	Detected
Patient 1	42.2±1.4	42.3±1.1	50.9±0.7	50.5±0.5	59.9±1.1	59.6±0.7
Patient 2	41.2±0.6	40.9±0.6	49.8±0.8	49.5±0.7	58.9±0.8	58.6±0.5
Patient 3	32.9±0.6	32.7±0.4	42.4±0.7	42.1±0.4	48.1±0.6	47.9±0.4

Table 2

Mean and standard deviation between ground truth and detected values (anterior-posterior lateral direction) for various gantry angles

	Seed 1 (mm)		Seed 2 (mm)		Seed 3 (mm)
	Ground truth	Detected	Ground truth	Detected	Ground truth	Detected
Patient 1	53.9±10.9	53.6±10.6	43.1±4.9	42.9±4.8	46.5±7.7	46.1±7.6
Patient 2	43.5±12.9	43.2±13.1	51.8±4.9	51.5±4.9	46.7±7.8	46.6±7.5
Patient 3	37.6±7.5	37.7±7.6	24.8±4.6	24.6±4.4	33.1±2.4	32.9±2.4

Software program results

Figure 7 shows the image obtained upon running the software to go through each cropped image, detecting the seeds using the YOLO v2 detector, and plotting the seed center at corresponding gantry angles. The plot at the top shows the movement of the seed centers in the SI direction, and the one at the bottom shows the movement of the seed centers in the AP/lateral (AP/L) direction. It was observed that the movement in the SI and AP/L directions was within 3 mm.

Figure 7

The software showing the position of the center of the seeds along with the gantry angle

As shown in Figure 8, the software reads each image from the source folder and runs the detector on the image in real time to display the seed centers and the plots in the x and y directions.

Figure 8

The software showing the position of each of the seeds along with the gantry angle

As each image is depicted, the corresponding gantry angle and kV source angle are also visible with the three seed positions and the center of the three seeds.

DISCUSSION

Conventional mathematical algorithms are used to track the fiducial marker seeds except for the situation where the imager passes through the hips, leading to insufficient contrast between the seeds and bone in most cases. In this project, we have used the YOLO V2 deep learning algorithm to detect the position of the fiducial marker seeds and corrected for the flex map to determine the true position of the detected seeds. The MATLAB app developed in our study was able to successfully display the images, seed positions, and calculated center positions.

For comparison, SeedTracker software uses the MEF method proposed by Peshko et al.[20] to perform the auto segmentation of the marker seeds. They used an intensity thresholding method to convert the MEF image into a binary image by setting the seed regions to “1” and other regions to “0.” In KIM, the segmentation was performed with a robust marker detection algorithm where the markers were identified in a 3D marker constellation method proposed by Fledelius et al.[27] Gehrke et al.[16] used Otsu's method to perform contrast thresholding, and then, the images were enhanced using the Prewitt and Roberts edge detection methods to define each seed location.

As observed in Figures 2 and 3, the images from a to h show the projection images at different gantry angles. A red circle has been drawn around the seeds to visualize the seed centers. It is noted that the seeds change position as the gantry angle changes. XVI eliminates these geometric distortions due to imager flex employing a software correction which maps out the relationship between the movement of the source and panel with gantry angle. A flex map is a correction table that looks at how the ray lines projected onto the panel shift relative to the isocenter at all projection angles. Figure 1 illustrates that the flex map values determined from the ball-bearing phantom were within 3 mm.

Table 1 shows that the ground truth and the detected seeds are very close to each other in the Sup/Inf direction for three patients. The seeds 1, 2, and 3 positional values are within 1 mm of each other, showing that the detection is accurate. The software program developed in this study can clearly display the detected seeds on each image and traces a plot of the center of the three seeds. Figure 7 is a snapshot of the software acquired while it is running to demonstrate the progression of the plot and images. Figure 8 depicts a modified version of the program illustrating the plot of each of the three seeds as each image is viewed in real time. It is interesting to note that although the plots in the Y(Sup/Inf) direction do not change much, the plots in the X/Z (AP/Lat.) direction follow a sinusoidal path as the gantry angle changes. This is again due to the rotation of the gantry around the couch while the patient is being imaged. The 2D projection of the seeds on the XVI panel as the gantry rotates causes this type of sinusoidal plot.

The detected seeds are visible at all gantry angles and do not require any prior knowledge of the position of the seeds on the projection images. However, the software requires the location of the seed positions from the TPS to assess the systematic errors. The accuracy of this method has been validated against seed positions generated from the Pinnacle TPS. This study demonstrates the feasibility of using deep learning to detect marker seeds. As a proof of concept, 1500 projection images were used to train the model, and the results demonstrate that our software could predict the seeds at most gantry angles. The effort is underway to develop a robust model by increasing the number of training datasets which is likely to increase the prediction accuracy at lateral gantry angles where seed visualization is affected by the presence of femur and other bony anatomy. This study will also be extended to sites such as the liver, lung, and kidney. A further proposed extension would be the use of an algorithmic method to calculate the centroids of seeds as opposed to the center of seed bounding boxes.

CONCLUSION

A deep learning-based detection algorithm was developed and tested on phantom and patient datasets. The model was able to predict the seed positions on 98% of the acquired projection images within 1 mm from the ground truth positions. The proposed methodology demonstrates the capability of using AI to identify and track fiducial markers on acquired 2D EPID images which would enable the clinicians to precisely deliver dose to the disease prostate volume with minimal dose spillage outside the target volume.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.