A new method for sperm characterization for infertility treatment: hypothesis testing by using combination of watershed segmentation and graph theory.

Shape and movement features of sperms are important parameters for infertility study and treatment. In this article, a new method is introduced for characterization sperms in microscopic videos. In this method, first a hypothesis framework is defined to distinguish sperms from other particles in captured video. Then decision about each hypothesis is done in following steps: Selecting some primary regions as candidates for sperms by watershed-based segmentation, pruning of some false candidates during successive frames using graph theory concept and finally confirming correct sperms by using their movement trajectories. Performance of the proposed method is evaluated on real captured images belongs to semen with high density of sperms. The obtained results show the proposed method may detect 97% of sperms in presence of 5% false detections and track 91% of moving sperms. Furthermore, it can be shown that better characterization of sperms in proposed algorithm doesn't lead to extracting more false sperms compared to some present approaches.

INTRODUCTION

It has been shown that the static and dynamic parameters of sperms may determine the chance of pregnancy.[12] Therefore, human sperm analysis has great importance for clinical study of the male infertility.[3] In recent years, the ability of analyzing sperm behavior has been provided by using microscopic imaging from human semen.[4] In this method, images which have been captured from semen specimens, are analyzed manually by an expert person. Not only tracking a large number of sperms by eye is a difficult and time-consuming procedure, but also visual problems and fatigue can affect negatively on the result.[5] Therefore, automated methods have been substituted particularly to measure important parameters of sperms. In order to obtain a good estimation of these parameters, an effective characterization scheme is required. Some major limitations make this procedure as a complex problem. The first limitation is that the location and orientation of the sperm cells simultaneously change in consecutive frames. The second limitation is the poor quality of images and finally the possibility of sperms touching each other in high-density samples.[67] Several algorithms have been developed to characterize sperms and to measure their motion parameters. In some researches,[8] several detection schemes such as split-merge or background subtraction techniques are combined with nearest neighbor method and then applied on microscopic images to characterize sperms. The performances of these methods are highly dependent to distances between sperms; therefore, they lead to considerable errors in high-density samples in which sperms are located in close proximities.

In some other researches simple algorithms based on the mean shift (MS) concept are utilized to characterize sperms. These algorithms reduce complexity and perform faster sperm tracking,[9] however, their main shortcoming is a lack of stability that leads to incomplete motion trajectories for sperms.

More sophisticated methods include various types of matching. In these methods, constant or flexible masks have been used to separate sperms from other semen particles.[1011] These approaches face some challenges such as high sensitivity to shape, size and rotation of sperms. Several types of clustering techniques have been utilized to separate sperms from other semen particles.[12] By using these techniques, trajectory of some sperms may be mistaken with each other due to sperm collisions. Therefore, clustering techniques does not lead to satisfactory characterizing of sperms. There is a class of methods that characterize sperms by using information provided by the contour of sperm head. However, this approach may not characterize sperms completely due to its weakness in extracting sperm tail.[13] In some recent researches, the optical flow (OF) algorithm is utilized to characterize sperms based on the movement of their tails.[14] This strategy causes some difficulties in detection and tracking due to fast motion of the sperm tail, the wide area of the sperm tail's movement, and its poor contrast.

In this paper, a new method for sperm characterization is introduced which is based on a combination of watershed-based segmentation and graph theory. In the first step of the proposed method, each frame of microscopic video is considered as a steady image and its probable sperms are extracted by using watershed-based segmentation. These particles are considered as “candidates.” In the second step, candidates are pruned during successive frames by using graph theory concept. Finally, a Kalman filter based algorithm is applied on remaining candidates to confirm them as sperms and make their trajectories. Using a watershed as a part of the proposed method enables it to separate neighboring sperms and to provide closed contours. Furthermore, in proposed method, the watershed algorithm is modified by using graph theory based pruning algorithm and Kalman filtering to reduce false detections and make valid motility trajectories. Despite many existing methods, the proposed algorithm doesn’t need binarization of the image. Therefore, a wide range of image information is incorporated in our proposed processing scheme. Furthermore, it distinguishes true candidates by using graph theory framework which utilizes both motion and shape characteristics of objects simultaneously. The proposed method doesn’t need primary knowledge about sperms or their paths. Furthermore, it characterizes them even with rotating trajectories.

The paper is organized as follows. In section II, the proposed algorithm has been introduced, which includes watershed-based segmentation for candidate selection, graph theory for pruning and finally trajectory making for candidate confirming. In section III the performance of the proposed method is evaluated based on real videos recorded from semen specimens. In section IV, the obtained results from experiments are compared with results of existing methods using their effective parameters. Conclusion is presented in the last section of the paper.

PROPOSED METHOD

Suppose I as a microscopic video which has been captured from a semen specimen and I as one of its frames in time slot t. This image (i.e., I) contains sperms, plasma and debris which two latter particles are called background in this article. Each pixel of I may be written as:

In above equation, I is the amplitude of a pixel in I which is located in row and column equal with l and j, respectively. Also, L, J are the image sizes. Dependence of I to background and noise (H0) or its dependence to a sperm (H1) is determined defining hypothesis testing as:

In the above equation, r, c and n show the sperm, background and noise components in I, respectively.

Candidate Selection

In order to find candidate sperms, firstly imagine I as a topographic surface which is immersed in water. Each local minimum of the topographic surface may be considered as a hole where construct a catchment basin with its surrounding low gray level neighbors. When the water starts filling all catchment basins, if two catchment basins merge as a result of further immersion, a dam that surrounds the connected immersed area of each merged catchment basin is built which represents the watershed line. Actually such watersheds may be considered as boundaries between several objects in I.

To implement this idea an efficient algorithm is presented below. Firstly the image pixels are sorted in increasing order of their gray values. Then K’ local minimums of I are extracted as some first members of the sorted list in such manner that their greatest gray level is I. Eq. 3 shows that above minimums may construct χ as a set of catchment basins . Each of these objects may be either an isolated minimum of image or a set of neighboring pixels which all of them are minimums of sorted list.[15]

Based on above procedure it may be said that all pixels of image having gray-level less than or equal to I has already been assigned to a unique catchment basin (i.e., one of χ members).

In the next step, pixels having gray-level equal to I +1 must be processed. These pixels may fall in one of the following cases. In first situation the pixel is not assigned to any existing basin. In this case it may be considered as a member of β() (i.e., union of new local minimums). In the second situation the pixel may be an extension of an existing basin if and only if at least one of its eight connected neighbors already is a member of . These pixels construct Z(χ) as a union with same size with χ which its kıth member shows the set of pixels which must be assigned to member kı of χ. Therefore by the combination of both mentioned cases each χ (for example χ) may expand to χ( as:[1516]

By repeating such strategy recursively to maximum value of sorted list, finally χ is obtained as the set of K objects (i.e., O) as:

Where χ is the set of K candidate objects which are extracted from I.

Graph Theory-based Pruning

To perform object pruning, the string λ is extracted from χ as:

In above equation, λ shows number of pixels belonging to candidate O. In the next step the members of χ are ordered due to the number of pixels belonging to each of them. Then based on the size filtering concept a new set of candidates is constructed using the F superior members of χ which their sizes are between αmax and αmax, as:

In above equations represents the f,th candidate for being a sperm in I. The above algorithm is also applied on frame t + 1 of video stream, and Fı candidates are extracted from I as:

To prune false candidates, it is necessary to assign a member of - like - to a member of - like - in such way that they could be considered as a unique sperm in two frames t and t + 1. There are several algorithms that may be used for such assignment[1718] and in this research the following method is utilized.[19]

II.2.1: Feature vectors for all members of and are extracted containing centroid coordinates, velocity, size and size rate (i.e., changes in particle size during successive frames). For instance X and X( are feature vectors extracted from and , respectively. So {X} and {X(} are feature spaces for and .

II.2.2: Each matched pairs X and X( in {X} and {X(} indicates a unique sperm. To obtain an accurate association between the elements of the two above sets, Euclidian distance d = ||X( − X|| is calculated between all members of {X} and {X(} that leads to the following distance matrix in which q = 1,2,…,F and qı = 1,2,…,Fı are indices defined similarly to f, fı.

II.2.3: The vector X( ∊ {X(} is selected and compared with X ∊ {X}. In this paper a kind of graph matching algorithms is used for this comparison as follows:

II.2.3.a: If no member of {X(} was matched to X, then X is selected as matched pair of X( and their dependence is shown by putting 1 in f element of association matrix constructed in frame t + 1 which is shown as [M].

II.2.3.b: If X had a matched pair X( ∊{X(} and if d < d, it means that X( is closer to X than X(. Consequently matching of X and X( is neglected and X is matched to X( by putting 0 and 1 in indices q and f of [M](), respectively.

II.2.3.c: If X had a matched pair X( ∊{X(}, but d ≥ d, it means that X( is closer to X than X(. Therefore indices q and f of [M]( remain unchanged as 1 and 0, respectively.

II.2.4: The mentioned (a), (b) and (c) steps are applied for all feature vectors which are members of {X} and {X(}. As result the final association matrix [M]( is obtained. Each pair of vectors in {X(} and {X} which their related member in [M]( is indicated by 1 shows a matched pair and specify a characterized sperm while others don’t indicate any valid pair (i.e., valid particle). Flowchart of pruning procedure has been shown in Figure 1.

Figure 1

Pruning procedure

Confirming Sperms by Obtaining Their Trajectories

In this stage, a Kalman-based algorithm is applied to construct meaningful trajectories. Other algorithms have been used for such purpose in some different researches.[20] The combination of the pruning and trajectory making algorithms (i.e., II.2 and II.3) may reject many objects which have been wrongly labeled as sperms by candidate selection step (i.e., II.1). The reason is that many of such candidates may not produce the feature vectors which lead to meaningful strings during successive frames or continuous trajectories for enough period of time, therefore they may omitted in pruning or trajectory making steps.

To Make Trajectories, First Suppose

Which θ contains B remained candidates in frame t + 2, after performing the prune algorithm which was mentioned in II.2. Confirming sperm trajectory is defined as finding unique X( in such way that it may be considered as the future of X(. In this paper a Kalman filter has been used for this purpose. Let ψη to be the Kalman filter which has been initially constructed by each valid sperm resulted from graph theory-based pruning step (i.e., II.2).

II.3.1: For each X( if it satisfies ψη then it is indicated as the next estimation of ψη and the filter is updated. If the number of updates exceeds a threshold γ then its associated candidate is considered as “Confirmed”.

II.3.2: If no member of θ satisfies ψη, then it is considered as lost in t + 2. Therefore its estimation is accomplished using its history temporarily. Also, if the loosed frames for a candidate are more than a threshold, then it is considered as “False” and it will be rejected.

II.3.3: Those members of θ who has not been associated to any ψη, are fed to pruning algorithm II.2, to find new candidates.

Finally combination of (2) and (11) with the procedure which has been explained in II.3.1, leads to equation bellow which determines the state of each pixel of the main video in time slot t.

Note that concluding “Do not reject H0” doesn’t necessarily mean that one of H0 or H1 is true. It only shows that there is not sufficient evidence against H0 in favour of H1 and therefore the pixel cannot be considered as a part of sperm in current time slot.

RESULTS

The proposed algorithm was applied on real data. The data set included various videos which had been obtained from microscopy of semen specimens. The videos were captured by an Orca ER Digital CCD Camera mounted on a Nikon invert microscope using a 40x zoom lens. A calibrated microscope slide was used in all of the experiments. This microscope slide was scaled per 10 μm which enabled us to estimate size and movement parameters of sperms. The complex pattern of sperms motion caused some limitations in recorded videos such as: To exit some sperms from region of interest, sperm apoptosis, and merging sperms with near distances. Using this procedure, 3480 frames of semen videos were investigated which belonged to 11 infertile men. Specifications of test scenario have been shown in Table 1.

Table 1

Specifications of test scenario

The proposed method was implemented using Matlab 2009. Additionally, three other recent algorithms were selected to implement and compare with the proposed algorithm. These alternative algorithms were: (1) Mean shift algorithm (MSA) which has been introduced in[9] and is called (MSA) for brevity in this article, (2) split and merge segmentation followed by nearest neighborhood which has been introduced in[8] and is called (SMNN) for brevity in this article and OF Algorithm which has been introduced in[14] and is called (OF) for brevity in this article. For brevity some results of the proposed and OF methods have been graphically showed in this part of article, but the complete statistics of the test results will be discussed in part IV. The captured videos were first processed using manual detection and tracking to obtain ground-truth tracks to compare the automatic methods with. Then tracked sperms were obtained by applying the proposed and other three alternative algorithms, and then the performance of each algorithm was determined by comparing of its results with manual results. Figures 2 and 3 show results which have been obtained in four different frames (15, 30, 45 and 60) by utilizing the OF and proposed methods, respectively. For example Figure 2a shows totally 63 sperms including 5 constant and 58 moving sperms in frame 15 of a test video. In this figure the OF method has extracted 56 sperms without any false detection. Figure 2b-d show 46 complete and 11 incomplete trajectories have been extracted from totally 58 moving sperms by using this algorithm. Furthermore, one trajectory has been missed. Figure 3 shows the obtained results of applying the proposed algorithms on the frames which had been shown in Figure 2. In frame 15 [Figure 3a] it is obvious that the proposed method has extracted 56 particles without false alarms. The results of frames 30, 45 and 60 (i.e. Figures 3b to c and d) show that this algorithm has extracted 53 full and 5 incomplete trajectories which shows that applying the proposed method on the same video has led to better results than OF.

Figure 2

Extracted sperms using optical flow algorithm in frames (a) 15, (b) 30, (c) 45 and (d) 60

Figure 3

Extracted sperms using proposed algorithm in frames (a) 15, (b) 30, (c) 45 and (d) 60

DISCUSSION

Real data which had been obtained from microscopy of sperms activity were analyzed. The proposed, OF, SMNN and MS methods were applied on data and the obtained results were compared with manual results using the following parameters:

Detection Rate: To estimate this parameter in each frame, the number of missed sperms were determined, then the average for all the frames was calculated and finally it were divided to total number of sperms as:

False Detection Rate: This parameter was calculated as:

Using the mentioned parameters receiver operating characteristic curves were obtained for both of the proposed and alternative methods which have been shown in Figure 4. This figure show clearly the superiority of the proposed method compared to other algorithms.

Figure 4

Receiver operating characteristic curves obtained for the proposed (solid line-blue), optical flow (dashed line-red), split and merge segmentation followed by nearest neighborhood (square line- magenta) and mean shift (dotted- black) algorithms

For better interpretation of results, P = 5% and P = 90% were considered as typical acceptable values for false detection and detection probabilities and Table 2 was constructed from these points of Figure 4. The performances of algorithms may be compared for other acceptable values of P and P in the similar way. As shown in first part of Table 2, the proposed algorithm has achieved detection rates 6%, 10%, and 20% better than OF, SMNN and MS methods versus 5% of false detection. Also, this table shows that the detection rate of the proposed algorithm reaches 90% with only 0.5% of false detections, which is 2.5%, 9.5% and 18.5% better than false alarm values which have been obtained for OF, SMNN and MS methods for the same detection rate.

Table 2

Comparing performance of algorithms in different scenarios

Track Categories

In captured videos all sperms may not be tracked because of reasons which were explained in part III. Based on those limitations, constructed trajectories were divided in three categories: “Full Trajectory” for the sperm correctly tracked along the entire video, “partial trajectory” for the sperm correctly tracked only for a portion of the video and “none trajectory” for false tracks. Table 2 shows superiority of the proposed algorithm comparing its alternatives in tracking as well as its superiority in detection. It has been shown that the proposed algorithm extracted full trajectories 11% 19% and 31% more than OF, SMNN and MS. Also, it can be shown the rate of partial trajectories extracted by the proposed algorithm has been 9%, 15% and 23% better than OF, SMNN and MS. In parallel with these better performances, the proposed algorithm has not extracted any none trajectory whereas the percent of extracted none trajectories by SMNN and MS has been 3% and 7%. The superior performance of the proposed algorithm is due to its different treatment for detection and association of sperms.

Existing methods detect sperms using image binarization by conventional thresholding methods. On the contrary our method uses watershed segmentation which is based on the gray level of the processed image. Therefore it may neglect so fewer sperms which increase the performance of algorithm. Furthermore, the proposed algorithm rejects more false particles because of utilizing graph theory framework in pruning step. This intuition is further corroborated by the obtained results mentioned before.

CONCLUSION

In this paper a new method was introduced for characterization of sperms in microscopic videos. In proposed method some particles were firstly indicated as “candidates” in each frame of microscopic video. This candidate selection was done by using watershed-based segmentation. Such a candidate selection allows us to consider the near and low contrast sperms as separated particles which makes the proposed algorithm superior from existing methods and. In the second step, the graph theory was utilized to reject some candidates who hadn’t constructed a meaningful string during successive frames. In final step, sperms were characterized from those remained candidates who had made trajectories for enough period of time.

The performance of the proposed algorithm were compared with three alternative methods (e.g. OF, SMNN and MS) using their detection-rate, false detection rate, full trajectories, partial trajectories and none trajectories. Tests were carried based on real videos containing high density sperms, so complex and close motions were recorded in captured videos. Results showed higher performance of the proposed algorithm in characterization of sperms compared to tested alternative methods. The results showed that the proposed method has detected sperms and full trajectories with accuracy of 6% and 11% respectively, better than the best of other examined algorithms. This superiority has been achieved in such way that the false detection rate of the proposed algorithm has been 2.5% better than the best of other examined algorithms. So it may be shown that better characterization of sperms by proposed algorithm not only hasn’t led to extract more false sperms and trajectories, but also, it has decreased their erroneous values too. Consequently it can be concluded that the proposed method may be used as a suitable choice for characterization of sperms and their movement parameters.

BIOGRAPHIES

Seyed Vahab Shojaedini has received his BSc in the field of Communication Engineering from Amirkabir University of Technology, Tehran, Iran in 1998. He also received his MSc and PhD in Bioelectrics from University of Tarbiat Modares, Tehran, Iran in 2001 and 2006 respectively. Since 2010 he has a position at Iranian Research Organization for Science and Technology (IROST). He is interested in Signal Processing, Image Processing, Radar and Stochastic Process.

E-mail: shojadini@irost.ir

Masuod Heydari has received his BSc in the field of Computer Engineering from Islamic Azad University, Tehran, Iran in 2004. He has been with Electrical Engineering and Information Technology Institute at Iranian Research Organization for Science and Technology (IROST) as Research Engineer. His primary research interests include Image processing and machine learning.

E-mail: ceo@vandad-bh.com