A Novel Medical Freehand Sketch 3D Model Retrieval Method by Dimensionality Reduction and Feature Vector Transformation.

To assist physicians to quickly find the required 3D model from the mass medical model, we propose a novel retrieval method, called DRFVT, which combines the characteristics of dimensionality reduction (DR) and feature vector transformation (FVT) method. The DR method reduces the dimensionality of feature vector; only the top M low frequency Discrete Fourier Transform coefficients are retained. The FVT method does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity. The experiment results demonstrate that the DRFVT method achieves more effective and efficient retrieval results than other proposed methods.

1. Introduction

The medical 3D model retrieval becomes a hot research topic due to the rapid development in clinic and teaching. The medical 3D model can not only show doctors the anatomical structure of a particular part inside the human body, but also reveal the function of human organs to a certain degree [1]. The current diagnostic techniques, especially in the cases which cannot immediately get the diagnosis, need the auxiliary diagnosis by previous similar cases. At this time, 3D model retrieval technology is applied to obtain the similar 3D model from the database. In the face of vast numbers of medical 3D models, how to find the required 3D model rapidly and extract the valuable information accurately has become an urgent issue [2].

The most popular way for retrieving 3D models is example-based paradigm [3], where the user provides an existing 3D model as query input and the retrieval method can return similar 3D models from the database. However, it is difficult for a user to have an appropriate example 3D model at hand. An alternative way is to use a 2D sketch as a query where users can describe a target 3D model by quickly drawing it. But a 2D sketch is merely a coarse and simple representation which only contains partial information of an original 3D model. Hence, it is more challenging to realize freehand retrieval [4] than example-based retrieval.

For the freehand sketch 3D model retrieval how to create efficient feature descriptors [5] is the most important part. Several freehand sketch 3D model retrieval methods have been proposed recently. Funkhouser et al. [6], Chen et al. [7], and Li et al. [8] extracted feature descriptors from the outline to describe the query sketch and 2D views of 3D models, such as the spherical harmonics descriptor, light field descriptor, and shape context features. Recently, Wang et al. [9] provided a sketch-based retrieval approach by utilizing both global feature and local feature. In the practical applications, we find that these features are not effective and efficient in evaluating the relevance between the query sketch and 3D models. There exist two main disadvantages for the current freehand sketch 3D model retrieval methods. Firstly, the traditional Fourier Transform [10] only considers describing the model and the argument of the coefficient. Compactness [11] and simplicity of feature descriptors are necessary for minimizing the storage overhead and the computing time. Therefore, it is necessary to describe the outline by a limited number of coefficients in the frequency domain. Secondly, a freehand sketch is a simple line drawing which has a high level of abstraction, inherent ambiguity, and unavoidable noise [12]. For effective retrieval, the feature descriptors should be robust to noise and invariant to transformations.

To tackle these problems, this paper proposes a novel freehand sketch 3D medical model retrieval method, which combines the characteristics of dimensionality reduction (DR) and feature vector transformation (FVT) method, called DRFVT. The DR method reduces the dimensionality of feature vector; only the top M low frequency Discrete Fourier Transform (DFT) coefficients are retained. These M coefficients are able to provide all of the abovementioned requirements and obtain good efficiency in retrieval [13]. The FVT method does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity. On this basis, the similarity between the sketch and the 3D model is calculated [14]. To evaluate our method, we test our method on the public standard data set and also compared with other leading 3D model retrieval approaches [15]. The experiments demonstrate that the DRFVT method is significantly better than any other retrieval techniques.

2. Our Method

This paper researches on how to get user's retrieval intention and the feature extraction of 2D freehand sketch. Users express their retrieval intention freely by freehand sketch. In order to realize the 3D model retrieval, this paper calculates the similarity between 2D sketches feature vector and 3D model projection view feature vector. The framework of the medical freehand sketch 3D model retrieval method is proposed, as shown in Figure 1 [16, 17].

Figure 1

The framework of the medical freehand sketch 3D model retrieval method.

The 2D freehand sketch framework contains five modules, respectively, 2D freehand sketch input module, 2D outline representation module, DR method module, FVT method module, and 2D sketch feature vector set module. The 2D freehand sketch input module would get user's retrieval intention with hand drawing or mouse drawing. The 2D outline representation module extracts the outline of sketch based on the eight-direction adaptive tracking algorithm [18]. The DR method module reduces the dimensionality of feature vector; only the top M low frequency Discrete Fourier Transform coefficients are retained. The FVT method module does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity. The 2D sketch feature vector set module contains the feature vector set generated by the DR method and the FVT method.

The medical 3D model framework contains four modules, respectively, medical 3D model set module, 2D projection view of 3D model module, feature vector extraction module, and 2D projection view feature vector set module. The medical 3D model set module contains 300 3D model files of medical organs from the Princeton Shape Benchmark [19]. The 2D projection view of 3D model module chooses and generates 2D projection views of each 3D model. The feature vector extraction module creates feature descriptors which describe the structural information and recognize interior content from 2D projection views of 3D model. The 2D projection view feature vector set module contains the feature vector set of 2D projection views of medical 3D model, which we use to compare with the 2D sketch feature vector set.

2.1. Feature Extraction of 2D Sketch

In the general sketch retrieval method, the sketches are composed of element figures, and the relationship between the element's figures is complex. For the 3D model retrieval method, the sketches are not expressed by simple element figures, because the outline of the sketch is complicated and has abundant feature information; it cannot be expressed by simple element figures. Therefore our method firstly extracts the outline of the sketch based on the eight-direction adaptive tracking algorithm and parameterizes the outline to obtain a complex discrete-time periodic signal whose period N equals the number of points. Secondly, we adopt the DR method to reduce the dimensionality of feature vector, which obtains good efficiency in retrieval. Finally, we propose that the FVT method does the transformation of the original feature vector, which uses the new feature vector to solve the sensitive noise problem [20-22].

2.1.1. 2D Outline Representation

For the sketches submitted by users, this paper extracts the outline of the sketch based on the eight-direction adaptive tracking algorithm. Let I(x, y) represent the original image [23] and C(x, y) represent the outline binary image. As shown in Figure 2, we search the outline in the 3 × 3 region with point P as centre counterclockwise.

Figure 2

The point P eight directions.

This algorithm selects the first point of I(x, y) as first_p. If there are no other points in direction 2,3, 4 of first_p counterclockwise, we save first_p into the outline binary image C(x, y). The algorithm is described as follows.

Algorithm  1

Step  1 (initialization). Set the outline binary image C(x, y) to be null. The direction array DI is assigned by DI = {0,1, 2,3, 4,5, 6,7}. Let d represent the current direction, and d = 2.

Step  2. Scan the original image I(x, y) line by line in order, from top to bottom and left to right. We obtain the starting point first_p of the outline and initialize the current point now_p with first_p.

Step  3. Add now_p to the outline binary image C(x, y). next_p represents the next adjacent point of now_p. Search next_p according to the direction array order DI[d], DI[d + 1],…, DI[d + 7]. We assume the next_p direction is DI[i]; the new search direction is d = (DI[i] + 4 + 1)mod⁡8. Then assign now_p by next_p again.

Step  4. Judge whether now_p coincides with the starting point first_p; if yes then exit; otherwise returns Step 3.

2.1.2. The DR Method Based on Fourier Transform

(1) Method Description. The outline of the sketch is closed and cyclical. We can use Fourier Transform to describe its characteristics. First of all, we consider the outline of an object as a discrete-time complex periodical signal z = 〈z 0, z 1,…, z 〉 and define , where x and y are the real coordinate values of the lth (l = 0,…, N − 1) sampled point. The signal z is then mapped to the frequency domain by the DFT [24],where R and θ are the module and the argument of the mth DFT coefficient, respectively.

(2) Invariance Requirement. In order to guarantee the translation, scaling, rotation, and starting point invariance, we have to modify the DFT coefficients accordingly. When a change is in the position, size, orientation of the object, or the initial point used to parameterize the outline, the DFT coefficients are modified [25]. In Table 1, we show how normalized DFT coefficients satisfying the invariance requirements can be obtained.

Table 1

The normalized DFT coefficients.

Invariance	Modified coefficient
Translation	Z⌢_0=0
Scale	R⌢_m=R_mR_1
Rotation	θ⌢_m=θ_m-θ_-1+θ_12
Starting point	θ⌢_m=θ_m+mθ_-1-θ_12

If one wants to achieve rotation and starting point invariance, the two corresponding modifications have to be integrated, leading to

For rotation and starting point invariance, the derivation is as follows [26-28].

We consider the original outline signals z and z′, where z′ is obtained from z by rotating each point counterclockwise by a constant factor θ 0 and by shifting the starting point by l 0 positions, z ′ = z e . The corresponding DFT coefficients areThus, θ ′ can be expressed asIn particular, when m is 1 or −1, θ 1′ and θ −1′ are

By referring to Table 1, it is obtained that

By simply performing the Inverse DFT on normalized DFT coefficients, we obtain a modified normalized signal z′ which satisfies all the requested invariance

(3) Dimensionality Selection. In order to concentrate the outline information of the sketch, we use only low frequency DFT coefficients. In particular, we keep only the M  (M ≪ N) coefficients whose frequency is closer to 0. The DR method uses the Bartolini et al. [29] spectral characteristics E(M) to determine the value of M. The energy of the signal retained by the M coefficients is defined as

The choice of a suitable value for M has to trade off the accuracy in representing the original signal with the compactness and the extraction efficiency.

2.1.3. The FVT Method

The feature vector extraction algorithm is as follows.

Let the centroid be (x 0, y 0); the distance r from the outline point to the centroid is shown as below:

Then the distance from each of M outline points to the centroid (x 0, y 0) generates the M dimension feature vector R = (r 1, r 2,…, r ).

The traditional Fourier Transform has certain sensitivity to noise, and the noise sensitivity problems will affect the similarity accuracy of feature vector. Therefore, this paper proposes the FVT method, which solves the issue of noise sensitivity. For the feature vector R = (r 1, r 2,…, r ), set r = min⁡(r 1, r 2,…, r ); then generate the new feature vector R′ = (r , r ,…, r , r 1,…, r ).

2.2. Similarity Comparison

From the user input retrieval intention, retrieving the user's satisfactory models requires comparing the feature vector distances between the input sketch and the 3D model. The similarity of models is determined by the differences between feature vectors [30-34].

In order to realize the medical freehand sketch 3D model retrieval, this paper compares the similarities of the medical 3D model based on Euclidean distance [35], where X, Y represent the feature vector of 2D sketch and 2D projection views of the 3D model, respectively:

3. Experiments and Results

We implement the medical 3D model retrieval method in C++ under Windows. The system consists of a computer with an Intel Xeon CPU E5520@2.27 GHz and 12.0 GB of RAM. On average, each 3D model takes 6.1 seconds to extract features; before the sketch retrieval we can extract and store the 2D projection view feature vector. Moreover the 2D freehand sketch takes 0.06 seconds to extract features. The freehand sketch retrieval takes 0.1 seconds in the medical 3D model retrieval system.

For performance evaluation, we select 300 3D model files of medical organs from the Princeton Shape Benchmark [18] (http://shape.cs.princeton.edu/benchmark/); each 3D model is manually classified based on 6 categories (“Body” (130 images), “Heart” (30), “Lung” (25), “Ovary” (13), “Kidney” (22), and “Liver” (18)). The 3D models do not belong to any category and were assigned to a default class (62). Table 2 shows some 3D models in the data set, along with their category.

Table 2

Sample 3D models based on 6 categories.

3D model	Category	3D model	Category	3D model	Category
	Body		Heart		Lung
	Ovary		Kidney		Liver

In this paper, we provide users' interface with hand drawing or mouse drawing, which is convenient for users to draw and modify the freehand sketches. As shown in Figure 3, the left is the tablet which is used for hand drawing or mouse drawing, and the right shows 3D model retrieval results.

Figure 3

The medical freehand sketch 3D model retrieval system.

3.1. The 2D Sketch Feature Extraction Experiments

3.1.1. Extract the Outline of Sketch

This paper extracts the outline of sketch based on the eight-direction adaptive tracking algorithm. It is assumed that when the sketch is drawn, the mouse point sequence is stored in I(x, y). After that I(x, y) is applied to generate the outline binary image C(x, y); Figure 4 shows a freehand sketch and the outline extraction.

Figure 4

The effect of the outline extraction.

3.1.2. The DR Experiments

According to the above process, we use the eight-direction adaptive tracking algorithm to parameterize N points outline. The DR method reduces the dimensionality of feature vector; only the top M low frequency DFT coefficients are retained. We consider the spectral characteristics E(M) of the data set to determine the value of M.

As an example, Figure 5 plots the value of E(M)/E(N) for the heart 3D model set. From the graph, when choosing M ∈ [16,64], the retained energy varies from 75 percent to 91 percent of the total. On the other hand, when obtaining E(M)/E(N) = 95%, M should be equal to 512. The M value will represent 512 outline points and then construct 512 vertices of graph, which is not obviously appropriate. Therefore choose the smaller M, such as M = 32; M is a power of 2; we can easily use the Fast Fourier Transform.

Figure 5

The value of E(M)/E(N) for the heart 3D model.

Experiments of the heart 3D model set show that the effectiveness of using M = 16 is much lower than M = 32, whereas increasing the value of M to 64 does not lead to further improvements. Thus, in this experiment, we will use M = 32; the result of the DR method is as shown in Figure 6. Using 32 generated outline points calculates the distance from each point to the centroid (x 0, y 0) and then generates 32D feature vector R = (r 1, r 2,…, r 32).

Figure 6

The result of the DR method for the heart 3D model.

3.1.3. The FVT Experiments

This paper considers the outline of the transformed human body, such as widening or cutting the human body. We select the M = 64 value using the DR method and calculate the distance from the outline point to the centroid . The feature vector R = (r 1, r 2,…, r 64) is shown as follows. Figure 7 shows the original human body feature vector, Figure 8 shows the widening human body feature vector, and Figure 9 shows the cutting human body feature vector.

Figure 7

The original human body feature vector.

Figure 8

The widening human body feature vector.

Figure 9

The cutting human body feature vector.

Figures 8 and 9 show that the feature vectors have certain sensitivity to noise. Therefore we transform the existing feature vector R = (r 1, r 2,…, r ) and generate new feature vector R′ = (r , r ,…, r , r 1,…, r ) to solve noise sensitive issue.

3.2. Algorithm Performance

For evaluation purposes, any 3D model in the same category of the query is considered relevant to that query, whereas all other models are considered irrelevant. To measure the retrieval effectiveness, we considered classical precision (P) and recall (R) metrics [36, 37] averaged over the set of processed queries. They are defined as follows:

3.2.1. Effect of the DR Method

In our first experiment, we compare the DR method with the one based on the extraction from the boundary of the M points having the highest curvature values, hereafter called MAXC [38]. We do not consider methods that reduce dimensionality by simply resampling the original boundary every N/M points since this easily leads to missing significant shape details. Results in Figure 10(a) clearly show that DR method consistently outperforms MAXC in precision and that it is always positive even for higher recall levels. A similar trend can be also observed on specific query, as shown in Figures 10(b) and 10(c).

Figure 10

P/R comparing DR and MAXC method.

3.2.2. Effect of the FVT Method

In order to consider the effects of the FVT method, we compare the DRFVT method with the other two methods. One method uses the DR method and discards the FVT method, hereafter called DR-NoFVT. The other method uses the MAXC and the FVT method, called MAXC-FVT. For one specific query, Figure 11 shows the P/R graph for the query “Human Body” and Figure 12 for visualization of retrieval results. The experiment results demonstrate that the FVT method is very useful, and the DRFVT method gets better precision than other proposed methods.

Figure 11

P/R graph for the query “Human Body.”

Figure 12

Results for the query “Human Body.”

4. Discussion and Conclusion

In this paper, we propose a novel medical freehand sketch 3D model retrieval method DRFVT which uses dimensionality reduction and feature vector transformation. It firstly provides a convenient interface to satisfy the user's design process and extract the outline of sketch based on eight-direction adaptive tracking algorithm. Secondly, the DR method reduces the dimensionality of feature vector; only the top M low frequency DFT coefficients are retained. These M coefficients are modified so as to achieve the invariance and then stored in the database. Finally, The FVT method is proposed, which generates a new feature vector to solve the problem of noise sensitivity. On this basis, the similarity between the sketch and the 3D model is calculated; thus the final retrieval results would be presented to the users. The experiment results demonstrate that our method achieves retrieval results more effectively and efficiently than other previously proposed methods.

The future of our work is to develop a user interaction feedback mechanism. After the users submit the query sketch, the system first provides a list of retrieved 3D models. Then, the users can refine the retrieval results by selecting the 3D models which they take as the good results. This feedback mechanism can not only provide more desirable retrieved 3D models to the user, but also enhance the user interaction just by making some easy choices. It would largely improve the effectiveness of our retrieval system [39, 40].