A new method of diagnosing constitutional types based on vocal and facial features for personalized medicine.

The aim of the present study is to develop an accurate constitution diagnostic method based solely on the individual's physical characteristics, irrespective of psychologic traits, characteristics of clinical medicine, and genetic factors. In this paper, we suggest a novel method for diagnosing constitutional types using only speech and face characteristics. Based on 514 subjects, the area under the receiver operating characteristics curve (AUC) values of classification models in age and gender groups ranged from 0.64 to 0.89. We identified significant features showing statistical differences among three constitutional types by performing statistical analysis. Also, we selected a compact and discriminative feature subset for constitution diagnosis in each age and gender group. Our method may support the direction of improved diagnosis prediction and will serve to develop a personal and automatic constitution diagnosis software for improvement of the effectiveness of prescribed medications and development of personalized medicine.

1. Introduction

Due to the development of medicine and advances of biotechnology and information technology, the midpoint of medical treatment has shifted away from common treatments of a certain disease to personalized medicine [1-8]. Consistent with this paradigm, there has been an explosion of interest in alternative oriental medicines and in a fusion of oriental and western medicines [9-16]. One of the core research areas of personalized medicine in western and oriental medicine is to understand the psychological characteristics, morphological traits, genetic characteristics, and constitution of individuals. The human constitution has been researched in western and oriental medicine for a long time. For example, Hippocrates suggested that the human constitution could be attributed to four kinds of substances (blood, phlegm, choler, and black bile) [4]. Wang classified humans into seven constitutional groups using physiological and physical status [4, 17]. Similarly, Lee classified humans into four Sasang constitutional types as TE (Teaeumin), TY (Taeyangin), SE (Soeumin), and SY (Soyangin) based on physiological, psychological, and physical characteristics [2, 11–13, 15].

Personal constitution diagnosis is important for several reasons. Firstly, people have vulnerability to particular diseases according to their individual psychological characteristics, genetic characteristics, and morphological traits. Therefore, risk factors for particular diseases can be identified according to an individual's constitution in the early stages of disease progression [18]. Secondly, drug response to prescribed medicine varies with personal constitution [11]. As such, the efficiency of prescribed medicine can be improved if we know the patient's constitution.

Many studies on Sasang constitution have been conducted. For the association of Sasang constitutions and diseases and the difference of constitutional types, many researchers introduced constitution analysis methods [1, 4, 5, 8, 12, 13, 15, 18–23]. Song et al. [22] classified TE and SE among the constitutional types using skin elasticity of the hand and proved that elasticity of the TE type was higher than that of the SE type. Their constitution diagnosis measured hand skin elasticity based on a questionnaire survey of thickness and elasticity of the skin. The limitations of the study were that experiments were performed in only TE and SE types and not in the TY and SY types, and elasticity measurements were performed only on the back of the hand. Choi et al. [18] studied the distribution of insulin resistance using multivariate logistic regression analysis and features such as age, cholesterol level, smoker/nonsmoker, diastolic blood pressure, and insulin in subjects of each constitutional type. They demonstrated that prevalence of insulin resistance differs according to Sasang constitution type and suggested that personal constitution type can act as an independent risk factor for insulin resistance. An association study between genome-wide SNP (simple nucleotide polymorphisms) profiles and Sasang constitution types for a more accurate Sasang constitution diagnosis was conducted via experiments using 353,202 SNPs from 60 DNAs by Yin et al [8]. They observed that 5,692 SNPs in TE versus SE association analysis were significantly different, 7,542 SNPs in SE versus SY were significantly different, and 4,083 SNPs in SY versus TE were significantly different. The detailed contents of Sasang constitutional medicine are described in references [2, 12], and the research on face or speech signals are described in references [24-28].

Until recently, previous studies that used face, SNPs, skin, body shape, and speech signals have focused primarily on difference analysis among constitutional types; the study of diagnosis prediction is rare. In this study, we focus on Sasang constitution diagnosis using morphological characteristics that are easily accessible to researchers and doctors.

The motivations of this study are as follows: first, how can we obtain essential and useful features that show relationships between morphological characteristics and corresponding constitutional types? Second, how does one use these features to build an efficient and accurate diagnosis model?

We make the following contributions to the field of constitution diagnosis.

Propose a readily available and novel method for an accurate and detailed constitution diagnosis using the combination of facial characteristics and speech signals in age- and gender-specific groups. Our method may support the direction of improved diagnosis prediction and will serve to develop a personal and automatic constitution diagnosis tool for improvement of the effectiveness of prescribed medications and development of personalized medicine.

Suggest discriminative and meaningful features for constitution diagnosis via statistical analysis, and identify a compact and useful feature subset in accordance with age and gender. Analysis of results will serve to create a better discriminative feature set in this field.

2. Materials and Methods

2.1. Data Preparation

Speech and facial feature extraction from 514 subjects in several hospitals and the Korea Institute of Oriental Medicine in the Republic of Korea was carried out. Constitutional types of all subjects were determined by specialists and drug responses [21]. Speech record configurations were no resonance, noise intensity from 40 to 50 dB, room temperature of 20°C ± 5°C and humidity of 40% ± 5%, Sennheiser e-835s microphone, Blaster Live 24-bit external sound card, and GoldWave recording program. Distance of the mouth of subjects and the microphone was 4–6 cm, and features were extracted using five vowels (A, E, I, O, U) and one sentence. The extracted features consisted of pitch, average ratio of pitch period, Jita (absolute Jitter), MFCC (Mel frequency cepstral coefficients) [29, 30], and so forth.

We took photographs from the side and front of the subject's face using a digital camera with a ruler (Nikon D700 with an 85 mm lens). Based on an identified feature point from a side- and front-face image, we obtained features such as distance, distance ratios, angle, and area from forehead, nose, mouth, face shape, and eye [20]. Doctors designated the feature points (Figure 1). Height and weight of subject were measured by a digital scale (LG-150; G Tech International Co., Ltd, Republic of Korea). A total of 82 features were used in this study (29 features from speech signals, 51 features from face, and weight and height features). All feature measurements were done based on self-made tool using MATLAB on Window XP. The specific content of the extracted features was described in Table 1.

Figure 1

Points used in feature extraction ((a): points and areas in frontal image; (b): points in profile image; (c): points in right eye; (d): point in left eye).

Table 1

All features used in this study and brief descriptions.

Speech	Brief description
xF0	Basic pitch of X. X is one of five vowels (A, E, I, O, and U).
xF1	Formant of 1st in 4 frequency periods of X
xF2	Formant of 2nd in 4 frequency periods of X
xJITA	Mean ratio of change in pitch period of X
xRF60_120_F240_480	(Frequency band of 60~120 Hz)/(frequency band of 240~480 Hz) of X
xRF240_480_960_1960	(Frequency band of 240~480 Hz)/(frequency band of 960~1960 Hz) of X
aRF2_F1	Relative ratio between frequencies of A (aF2/aF1)
iDF0_aF0	Difference of frequencies (iF0-aF0)
uDF0_oF0	Difference of frequencies (uF0-oF0)
xMFCC4	Mel frequency cepstral coefficients of X
SITS	Amplitude average
SISTD	Standard deviation of amplitude average
SSPD	Time to read one sentence
RSF60_120_240_480	(Power of frequency band of 60~120)/(power of frequency band of 240~480) (one sentence)
RSF240_480_960_1960	(Power of frequency band of 240~480)/(power of frequency band of 960~1960) (one sentence)
Face	Brief description
Height	Weight
Weight	Height
FD n1_ n2	Distance between point n1 and n2 in a frontal (or profile) image
FDH n1_ n2	Horizontal distance between n1 and n2 in an image
FDV n1_ n2	Vertical distance between n1 and n2 in a frontal (or profile) image
FA n1_ n2_ n3	Angle of three points n1, n2, and n3 in an image
FA n1_ n2	Angle between the line through 2 points n1 and n2 and a horizontal line
FR02_psu	FD(17,26)/FD(18,25)
FR03_psu	(FD [18,25] + FD [118,125])/FDH(33,133)
FR05_psu	FDH(33,133)/FD(43,143)
FR06_psu	FDH(33,133)/FDV(52,50)
FR08_psu	FD(43,143)/FDV(52,50)
FArea02	Area of the contour formed by the points 53,153, 133, 194, 94, 33, and 53 in an image
FArea03	Area of the contour formed by the points 94, 194, 143, 43, and 94
Fh_Cur_Max_Distan	Distance between points 7 and 77
Fh_Angle_ n1_ n2	Angle between the line through 2 points n1 and n2 and a horizontal line
Nose_Angle_ n1_ n2	Angle between the line through 2 points n1 and n2 and a horizontal line
Nose_Angle_ n1_ n2 _n3	Angle of 3 points n1, n2, and n3
SA n1_ n2	Angle between the line through 2 points n1 and n2 and a horizontal line
Fh_Cur_Max_R79_69	FD(77,9)/FD(6,9)
Nose_Area_ n1_ n2_ n3	Area of the triangle formed by 3 points n1, n2, and n3 in an image
EUL_L_el1~el7	Slope of the tangent at a point (el1~el7) in an image
EUL_R_er1~er7	Slope of the tangent at a point (er1~er7) in an image

Since the face and speech signals are influenced by age and gender [21, 31], experimental data were divided into five categories based on age and gender: Female-20 (women aged 20–29 years), Female-30 (women aged 30–49 years), Female-50 (women aged 50 years and over), Male-30 (men aged 30–49 years), and Male-50 (men aged 50 years and over). The Male-20 (men aged 20–29 years) category was excluded due to the lack of a minimum number of subjects. The numbers of each constitutional type in 5 groups are listed in Table 4 for statistical analysis.

Table 4

Statistical analysis by the one-way ANOVA test and the post-hoc Scheffe's test.

Group	Feature	Class	N	Mean	Std.	F	Sch.
Female-20	Height	TE	18	162.2	5.334	3.689∗	A, B
		SE	22	163.8	5.018		B
		SY	23	159.7	4.979		A
	Weight	TE	18	62.26	7.328	19.85∗∗∗	B
		SE	22	52.09	4.689		A
		SY	23	51.62	5.776		A
	uF2	TE	18	−0.613	0.750	4.975∗∗	A
		SE	22	−0.531	1.146		A
		SY	23	0.249	0.999		B
	aRF2_F1	TE	18	−0.414	0.912	3.772∗	A
		SE	22	0.519	1.510		A, B
		SY	23	1.225	2.644		B
	FA18_25_43	TE	18	95.28	4.945	3.520∗	B
		SE	22	91.29	4.927		A
		SY	23	92.15	4.882		A, B
	EUL_L_s6	TE	18	0.427	0.102	5.346∗∗	A
		SE	22	0.523	0.099		B
		SY	23	0.442	0.104		A
Female-30	FD43_143	TE	57	133.0	5.985	23.55∗∗∗	B
		SE	53	124.4	7.085		A
		SY	73	127.2	7.100		A
	Fh_Angle_6_7	TE	57	61.78	8.052	5.039∗∗	A
		SE	53	65.59	9.818		B
		SY	73	60.80	8.071		A, B
Female-50	Weight	TE	46	0.615	0.194	25.64∗∗∗	B
		SE	24	0.375	0.144		A
		SY	44	0.414	0.119		A
	FD43_143	TE	46	0.443	0.202	3.384∗	B
		SE	24	0.335	0.144		A
		SY	44	0.384	0.142		A, B
Male-30	Weight	TE	32	79.34	9.244	26.29∗∗∗	C
		SE	31	64.45	7.099		A
		SY	22	71.22	7.868		B
Male-50	iJITA	TE	26	0.464	1.282	3.456∗	A, B
		SE	18	1.793	4.813		B
		SY	25	−0.364	1.144		A
	oJITA	TE	26	0.099	1.023	3.558∗	A, B
		SE	18	1.762	4.564		B
		SY	25	−0.163	1.004		A
	SITS	TE	26	−0.730	1.182	5.088∗∗	A
		SE	18	0.252	0.763		B
		SY	25	−0.042	1.123		A, B
	RSF60_120_240_480	TE	26	−0.148	1.033	3.382∗	A
		SE	18	0.628	1.411		A
		SY	25	−0.217	1.025		A
	FDV81_50	TE	26	32.42	2.788	6.426∗∗	B
		SE	18	29.93	2.744		A
		SY	25	29.86	2.953		A
	FArea03	TE	26	4898	556.8	14.23∗∗∗	B
		SE	18	4135	417.1		A
		SY	25	4331	502.4		A

(N: number of samples, Std.: standard deviation, Sch.: Scheffe's test, ∗P< 0.05, ∗∗P< 0.01, and ∗∗∗P< 0.001).

2.2. Experiment Configurations

The goals of our experiment were to measure the ability to distinguish constitutional types, and to identify a more discriminative and compact feature set through feature selection. We conducted classification experiments of TE, SE, and SY constitution types with our five data sets according to the difference of age and gender, with and without feature selection. To investigate the differences of detailed performances of each feature types, speech feature set, face feature set, and hybrid feature set (combining face and speech) were used in this experiment.

We applied normalization (scale 0~1 value) to all data sets. The Wrapper approach using machine learning of LIBLINEAR [32] and the best-first search (forward) was used in feature subset selection. All experiments were performed using LIBLINEAR (L2-loss SVM dual type) in Weka software [33], and a 10-fold cross validation for a statistical evaluation of learning algorithm was performed. For optimal parameter selection (tuning), the value of the C parameter was obtained in the range of {0.01, 0.05, 0.1, 0.5, 1, 2, 4, 8, 16, 32, 64, 128, …, 16348}, and other parameters were fixed as defaults. The area under receiver operating characteristic curve (AUC) was used as a major evaluation criterion. AUC is widely used to quantify the quality of a prediction or classification model in medical science, bioinformatics, medicine statistics, and biology [34-36]. We also evaluated performance using the sensitivity, specificity, and F-measure for detailed evaluation. Statistical analyses were conducted by SPSS version 19 for Windows (SPSS Inc., Chicago, IL, USA).

3. Results and Discussion

3.1. Comparison of Experimental Results

For brief summarization of performance evaluation, the AUC values for the 5 groups with and without feature selection method are showed in Table 2.

Table 2

Comparisons using AUC values in 5 groups (Hybrid: combining face and speech data, FF: using full features, FS: using feature selection).

Method	Female-20	Female-30	Female-50	Male-30	Male-50
Hybrid-FS	0.89	0.64	0.75	0.72	0.78
Hybrid-FF	0.67	0.59	0.67	0.67	0.69
Face-FS	0.77	0.67	0.69	0.73	0.71
Face-FF	0.62	0.59	0.63	0.63	0.62
Speech-FS	0.69	0.57	0.58	0.67	0.73
Speech-FF	0.6	0.48	0.54	0.59	0.64

In experiments using full features without feature selection, the results indicated that the hybrid feature set (Hybrid-FF) performed better than the individual face and speech feature sets (Face-FF and Speech-FF), except for performance by the face feature set in the Female-30 group. The AUC values of age and gender classification models using hybrid feature set without feature selection ranged from 0.59 to 0.69%.

After application of feature subset selection, the remaining number of features was small, whereas the AUC values of constitution classification were greater than that of the full feature set in all the experiments. For example, the AUC values of classification models using hybrid feature set ranged from 0.64 to 0.89%. The values of classification models using feature selection showed improvements of 0.22 in the Female-20 group and 0.09 in Male-50.

When comparing the face (Face-FS), speech (Speech-FS), and hybrid feature sets (Hybrid-FS), the performance of Hybrid-FS method was higher than that of Face-FS and Speech-FS in the Female-20, Female-50, and Male-50 groups. However, performance of Face-FS method was higher than that of Hybrid-FS and Speech-FS in the Female-30 and Male-30 groups. Thus, it is preferable to use the Face-FS method in these two groups. The theoretical performance of the Hybrid-FS method using feature selection is better than or equal to that of Face-FS and Speech-FS, because the hybrid feature set includes all the speech and face features. However, realistically, many of the feature selection methods may not ensure better performance because a greater number of features add difficulty in building a classifier and lead to the curse of dimensionality and an NP problem [36-38]. Detailed performance evaluation of experiments using feature selection was showed in Table 3.

Table 3

Detailed performance evaluation of experiments using feature selection in all groups.

Group	Measure	Class
		TE	SE	SY
Female-20	Sensitivity	0.89	0.77	0.91
	Specificity	0.93	0.95	0.9
	F-measure	0.87	0.83	0.88
Female-30	Sensitivity	0.63	0.4	0.53
	Specificity	0.78	0.84	0.66
	F-measure	0.6	0.44	0.52
Female-50	Sensitivity	0.87	0.17	0.8
	Specificity	0.79	0.97	0.74
	F-measure	0.8	0.26	0.72
Male-30	Sensitivity	0.81	0.84	0.09
	Specificity	0.74	0.78	0.92
	F-measure	0.72	0.75	0.14
Male-50	Sensitivity	0.89	0.5	0.68
	Specificity	0.81	0.92	0.82
	F-measure	0.81	0.58	0.68

3.2. Statistical Analysis of Meaningful Features

For the statistical analysis of features obtained from feature selection in Hybrid-FS method, we carried out a one-way ANOVA test, and for the post-hoc analysis, we performed the Scheffe's multiple comparison test. The ANOVA test indicated that there was a significant difference among constitutional types. All features showing P values of <0.05 are shown in Table 4.

In this statistical analysis, we did not found features that covered a broad range of applicability in predicting constitutional types in the age- and gender-specific classification. However, we identified several features with an obvious propensity for classifications of constitutional type.

In facial feature analysis using results from the Scheffe's multiple comparison test, there was a statistically significant difference in weight among constitutional types. Weights between TE and the other types were significantly different in the Female-20 group (F = 19.85, P = 0.0000). Weights between TE and the other types in the Female-50 group were significantly different (F = 25.64, P = 0.0000). Weights among TE, SE, and SY were significantly different in the Male-30 group (F = 26.298, P = 0.0000). There was a significant difference in FD43_143 between TE and the other types in the Female-30 group (F = 23.558, P = 0.0000) and between TE and SE in the Female-50 group (F = 3.384, P = 0.0374). FArea03 between TE and the other types was significantly different in the Male-50 group (F = 14.231, P = 0.0000).

For speech features, there was a significant difference in SITS between TE and SE in the Male-50 group (F = 5.088, P = 0.0088). The uF2 value between SY and the other types was significantly different in the Female-20 group (F = 4.975, P = 0.0100), and aRF2_F1 between TE and SY was significantly different in the Female-20 group (F = 3.772, P = 0.0286).

3.3. Limitations and Future Work

Constitution diagnosis is very difficult in this area, because constitution type decisions are dependent on qualitative judgments of doctors and inspectors [22]. We think that it is possible to develop an accurate diagnosis method or standardization for constitution diagnosis after collecting more diagnosis information by doctors or inspectors.

Our experimental and statistical analysis showed an important and useful feature for better diagnosis based on differences in age and gender. Since diagnosis performances and selected features differ according to age and gender, it makes constitution diagnosis difficult in the real world. Until now, features showing an obvious propensity for constitution diagnosis are not yet sufficient, and data sets to achieve diagnoses that are more accurate are insufficient. Accordingly, more research for constitution diagnosis is needed.

In future work, we will investigate the relationship between constitution and improvement of the effectiveness of medications and explore the role of constitutional types in certain disease. We think that this is very important research in clinical medicine, because the efficiency of prescribed medicine can be improved if we know the patient's constitution. For instance, Jeong et al. [9, 10] investigated changes in cytokine production in the acute stage of SY constitution CI (cerebral infarction) patients after oral administration of Yangkyuk-Sanhwa-Tang water, and revealed that Yangkyuk-Sanhwa-Tang had a good effect on anti-inflammatory cytokines and a good CI treatment effect. The results of these studies may help to improve the effectiveness of prescribed medications in Sasang constitutional types.

4. Conclusions

This study describes a novel prediction method for constitution diagnosis as an essential prerequisite for personalized medicine or alternative medicine. We demonstrated the possibility and usefulness of constitution diagnosis using the combination of face and speech feature sets in age- and gender-specific groups, identified a compact and discriminative feature subset, and included supporting statistical analysis of significant features. Our results could be used for developing an automatic constitution diagnostic tool for improving the effectiveness of prescribed medications and could be used in the fields of speech and face recognition.