Review on the current trends in tongue diagnosis systems.

Tongue diagnosis is an essential process to noninvasively assess the condition of a patient's internal organs in traditional medicine. To obtain quantitative and objective diagnostic results, image acquisition and analysis devices called tongue diagnosis systems (TDSs) are required. These systems consist of hardware including cameras, light sources, and a ColorChecker, and software for color correction, segmentation of tongue region, and tongue classification. To improve the performance of TDSs, various types TDSs have been developed. Hyperspectral imaging TDSs have been suggested to acquire more information than a two-dimensional (2D) image with visible light waves, as it allows collection of data from multiple bands. Three-dimensional (3D) imaging TDSs have been suggested to provide 3D geometry. In the near future, mobile devices like the smart phone will offer applications for assessment of health condition using tongue images. Various technologies for the TDS have respective unique advantages and specificities according to the application and diagnostic environment, but this variation may cause inconsistent diagnoses in practical clinical applications. In this manuscript, we reviewed the current trends in TDSs for the standardization of systems. In conclusion, the standardization of TDSs can supply the general public and oriental medical doctors with convenient, prompt, and accurate information with diagnostic results for assessing the health condition.

Introduction

A tongue is an organ that reflects the physiological and clinicopathological condition of the body. Each part of the tongue is related to corresponding internal organs. Visual information is used in tongue inspection. The color, form, motion, substance, and coating of the tongue are the main factors to be considered for diagnosis. The geometric shape also helps to diagnose one's health by the observation of changes in the tongue body, such as that in the thickness, size, cracks, and teeth marks. The tongue coating, covering the tongue like moss, with characteristics such as color, degree of wetness, thickness, form, and distributed range, is the most important factor in determining disease and a patient's body condition.

Although tongue diagnosis is convenient and noninvasive, it is difficult to achieve an objective and standardized examination. Changes in inspection circumstances, such as light sources, affect the result significantly. Moreover, because the diagnosis relies on the doctor's experience and knowledge, it is hard to obtain a standardized result. Recently, many research projects have attempted to solve these problems, and various types of tongue diagnosis systems have been developed in more than five countries.2, 3, 4, 5, 6

These systems have respective unique advantages and specificities according to the application and diagnostic environment, but this variation may cause inconsistent diagnoses in practical clinical applications. In this manuscript, we summarize the development of TDSs and current technologies such as hyperspectral and 3D imaging for the standardization of TDSs

Methods

The TDS process is similar to that of general image analysis systems, as shown in Fig. 1. Image capture and image storage processes are hardware resources. The hardware resources are summarized in section 2.1. Color correction and tongue segmentation, which are necessary for automated TDSs, are image-preprocessing steps in tongue diagnosis. In the image analysis process, the tongue is classified according to the extracted diagnosis parameters, which are summarized in section 2.2. In 2.3, 2.4, the current development of new types of TDSs such as hyperspectral and 3D systems, are described.

Fig. 1

Procedure of tongue diagnosis systems (TDSs).

Hardware for tongue diagnosis systems

Most recently developed TDSs consist of three major hardware components: an image sensing module, an illumination module, and a compute and control module. The goal of these hardware components is to obtain reproducible tongue images under varying conditions.

The image-sensing module has either a charge-coupled device (CCD) or a complementary metal–oxide–semiconductor sensor, which is commonly used to acquire a digital image. Cai et al. used a commercial digital camera, which is stable, and a high-performance image-sensing module for tongue image acquisition. They first used a modified handheld color scanner with a microscopy slide on top of the tongue; however, the measurement required contact that would be difficult in a clinic environment and the hardware would require custom fitting for tongues of different sizes. Because this method was unsuitable for tongue image acquisition, they next attempted the use of a digital camera for tongue image acquisition. The tongue was photographed using a commercial digital camera (640 × 480), and a ColorChecker was embedded inside the image. Wang et al. used a CCD digital camera without distinct color distortion and with a resolution of 1024 × 768 pixels. The camera was mounted on the side of a dark chest, opposite and along a horizontal line with the face-supporting device. Jiang et al. developed a TDS using a high-quality digital camera with a 7.2 megapixel resolution. To minimize color errors, a Munsell color checker was embedded inside the image for color calibration. Jeon et al. developed a TDS using a 5 megapixel resolution CCD camera. This system was designed to provide a clear tongue image by securing a sufficient photographic distance with a surface coating mirror, as shown in Fig. 2. For superior image quality, Zhang et al. used a Sony 900E video camera, which has a new type 3CCD kernel and has relatively less distortion, as the image capture device. This camera provides an image of 720 × 576 pixels, 50 dB S/N ratio, as well as both PAL and NTSC signals. The 3CCD camera uses three separate CCDs, each taking a separate measurement of the primary colors—red, green, or blue light.

Fig. 2

Construction of tongue diagnosis system that achieves a sufficient photographic distance using a surface coating mirror.

The performance of the illumination module is critical for the acquisition of high-quality tongue images. It is uncontrollable and unstable in natural light, despite its high color temperature and satisfactory color-rendering properties. Thus, artificial light sources can be used in a semi-closed or fully closed box, avoiding interference by other unexpected light or reflections. Jeon et al. used a light-emitting diode (LED) for the illumination module. They applied a diffusing acrylic with the LED lighting to reduce specular reflection on the surface of the tongue due to saliva. Zhang et al. used two 70 W cold-light halogen lamps with 4800 K color temperature. To compensate for the high heat emission, they used an optical fiber as a waveguide and installed the light source separately from the image acquisition element. Meiling et al. used a 20 W tungsten halogen lamp in their portable device for tongue image analysis. The color temperature of the tungsten halogen lamp is only 2674 K, compared with the D65 standard light source, which has a color temperature of 6174 K. Meiling et al. therefore suggested the use of a reflection device, a bowl-shaped cover that absorbs red and infrared light and reflects blue light, which is an effective way to enhance color temperature. To reduce specular reflection, Yamamoto et al. used a combination of three illumination systems: two light sources at 45° angles, two crossed polarizers in front of the camera and the light sources, and an integrating sphere. An integrating sphere is an optical component consisting of a hollow cavity with its interior coated for highly diffuse reflectivity so that light ray incidents at any point are distributed equally throughout the inner space by multiple scattering reflections. As a result, the effects from the original direction of the light are minimized. An artificial sunlight lamp was used as the light source. The advantages and disadvantages of the hardware for tongue diagnosis systems are summarized in Table 1.

Table 1

Advantages and Disadvantage of the Hardware for Tongue Diagnosis Systems.

First author (y) ref.	Advantages	Disadvantages
Cai (2002)2	Using a ColorChecker for color correction	Low image resolution (640 × 480)No illuminator
Yamamoto (2011)3	Hyperspectral imaging for the tongue image acquisitionUsing an artificial sunlight lamp as a light sourceAdopted multiple scattering reflection technique to the light source in order to reduce specular reflectionUsing a semi-closed box to avoid other unexpected light	Low image resolution (640 × 480)No color correction method
Jeon (2008)6	A diffusing acrylic with the LED lighting to reduce specular reflectionA ColorChecker for color correctionHigh image resolution with a long focal lengthUsing a fully closed box to avoid otherunexpected light	Low color temperature of the light source
Wang (2005)8	Using several consistent color patches for color calibration to keep the consistency of the colors transmitted from camera to monitorUsing a fully closed box
Jiang (2008)9	High-quality digital camera with a 7.2 megapixel resolutionA ColorChecker for color correction	No illuminator
Zhang (2005)10	Using an optical fiber as a waveguide to compensate for the high heat emission of the light sourcePotable face contact region	No color correction method
Meiling (2008)11	Portable type system	Low color temperature of the light source
Li (2008)31	Hyperspectral imaging for the tongue image acquisition	Low image resolution (652 × 620)No color correction method
Liu (2011)34	3D image system for 3D tongue modelingHigh image resolution	No color correction method

Software for tongue diagnosis systems

Color correction

Based on the theory of oriental medicine, color information is an important feature of tongue manifestation. The purpose of color correction is to maintain consistency in the color information and minimize color distortion caused by a light source and image sensor biases. The most common method for color correction is the use of a ColorChecker, which provides a color reference to a calibration model.2, 6, 8, 9 The color checker is embedded inside the digital tongue image as shown in Fig. 3. Because the color values of the ColorChecker test cells are already known, it is possible to calibrate the colors of a distorted image through computation. Zhang et al. suggested a new ColorChecker strategy. They adopted adopt Li's regularized color clustering algorithm to produce the color codes of the new ColorChecker. The entire check-board is composed of four types of ColorCheckers: clustered color cells, Microsoft Window 24-color cells, gray color cells, and color cells clustered by Li. After preprocessing the color cells of the ColorChecker, polynomial regression-based and SVR-based calibrations were performed using the International Commission on Illumination (CIE) lab values for the color cells. As a result, the SVR-based method had a better overall performance than the polynomial method.

Fig. 3

Example of ColorChecker location in tongue image.

Segmentation of tongue region

Tongue segmentation is one of the most essential steps in automated TDSs and is a very difficult procedure owing to color variation caused by pathology of the tongue, variation in tongue shape, and interference from the lips and teeth. Low-level image processing techniques such as region growing and general edge detection, fail to segment the tongue from its surroundings. To address segmentation problems, various high-level image-processing techniques have been proposed. Active contour models, one of the most important classes of deformable shape models, are a sophisticated method for contour extraction and image interpretation. Snakes are curves defined within an image domain that move under the influence of internal forces coming from within the curve itself and external forces computed from the image data. Wu et al. adopted parametric active contours, which are a type of snake for tongue segmentation. The initial contour for the snake was acquired by watershed transformation. The watershed transformation treats an image as a surface where dark regions are low and light regions are high, finding “watershed ridge lines” between high and low regions and “catchment basins” in low regions. Zuo proposed a tongue segmentation method by combining a polar edge detector, edge filtering, edge binarization, and an active contour model. The initial boundary of the tongue was detected using the boundary characteristics, which are local minimum intensities in the radial direction. The original image was then used to construct an edge mask to filter the polar edge image. The filtered polar edge image was then binarized by a local adaptive bi-thresholding method. From the bi-level edge, the active contour model was initialized. Yu et al. proposed the addition of a color gradient to the gradient vector flow (GVF) snake. In the first step, the boundary of the tongue body was roughly detected using characteristics of the tongue body. The color gradient of the tongue image was then calculated and a homogenous region was set. Finally, the color GVF snake was applied to extract the tongue body. Normally, the gradient is defined as the first derivative of the image luminance in gray-scale images. However, Yu et al. adopted the definition of the gradient for color images in RGB color space and introduced the color gradient into the GVF snake. Pang attempted to combine the deformable template technique with the snake to build a model called the bi-elliptical deformable contour by introducing a template force to replace the internal force for tongue segmentation. The primary advantage of using a template force is that it can maintain consideration of the global shape while locally deforming the details. In addition, the template force can prevent undesirable deformation effects of traditional snakes, like shrinking and clustering. Liang et al. proposed a new tongue segmentation approach based on a combination of the features of tongue shape and the snakes correction model. They created a rough tongue contour using the intensity features of a tongue image in the hue, intensity and saturation color model, corrected the preliminary tongue contour with the features of tongue shape, and applied this result to the snake model to achieve the final result. Zhu et al. introduced a type of color tongue image fast segmentation method based on level sets, and Li et al. combined prior knowledge with the improved level set method. The contour of the tongue was initialized in the hue, saturation, and value (HSV) color space, enhancing the contrast between the tongue and its surroundings. A region-based signed pressure force function was proposed for the level set, to efficiently stop the contour at weak edges.

Segmentation of sublingual veins

Inspection of the sublingual vein can provide valuable insight into the health condition of humans. According to the theory of tongue diagnosis, the shape and breadth of the sublingual vein is the gold standard for diagnosis of portal hypertension and blood stasis. In early research, experiments were conducted on images acquired using an ordinary camera under a visible light source. However, under visible conditions, the appearances of sublingual veins of both healthy and ill humans show few differences. Yan et al. proposed a segmentation method for sublingual veins captured by a sublingual image acquisition device using infrared light. In the proposed method, a radial projection method based on a watershed algorithm was applied to acquire the entire contour of the sublingual region, and dynamic thresholding and binarization were subsequently performed. Finally, region growing was iteratively performed to trace the contours of sublingual veins. The average correct segmentation rate was up to 82.0% in a total of 105 subjects.

Tongue classification

In oriental medicine, the properties of tongue color and texture are important diagnostic characteristics; however, the development of appropriate objective features necessary for meaningful diagnosis is difficult. Recently, researchers have been developing various methods for classifying diagnostic categories from tongue features. Pang et al. proposed a computerized tongue inspection method based on quantitative features and Bayesian networks. To quantify the color features, they used the means and standard deviations of the pixel colors within the entire region of the tongue in the RGB, HSV, CIE 1976 (L*, u*, v*), and CIE 1976 (L*, a*, b*) color space. To quantify the textural features, two measures derived from the co-occurrence matrix were used to extract different textural features from tongue images. Experiments were carried out on a total of 455 inpatients affected by 13 common internal diseases and 70 healthy volunteers. The estimated prediction accuracy of the joint Bayesian network classifier was 75.8%. Gao et al. proposed a computerized tongue inspection method based on the support vector machine (SVM). The SVM method comes from the application of statistical learning theory to separate hyperplanes for binary classification. The color and texture features proposed by Pang et al. were used. Experiments were performed on a total of 665 inpatients affected by 6 common internal diseases and 103 healthy volunteers. The estimated prediction accuracy of the multi-class SVM classification was 86.6%. Hui et al. proposed the use of different machine learning techniques, including decision trees, naïve Bayes, Bayesian networks, and SVMs, to distinguish tongue classifications. They used feature form, tongue substance, and tongue coating as the basis for tongue classification. The tongue substance features include the color, location of spots, physical condition, and movement of the tongue such as stiffness, trembling, flaccidness, or protrusion. The tongue coating features include thickness, color, distribution, and texture. Hui et al. collected a total of 457 images for their experiment, and machine learning was performed. As a result, sequential minimal optimization, which is based on SVM, achieved very good performance results based on both accuracy and area under the receiver operating characteristics curve (AUC). Huang defined the features of tongue shapes using seven sub-features defined using the length, area, and angle information. To convert human judgment into classification decisions, Huang applied the analytic hierarchy process (AHP), wherein the relationship of each sub-feature to a shape is characterized on a standardized numerical scale and given a weight. A fuzzy fusion framework was then used to combine seven modules of the AHP and classify each tongue image among seven classes of tongue shapes. The accuracy of tongue shape classification on a set of 362 pre-classified tongue images was 90.3%, whereas the accuracies of k-nearest neighbor algorithm and linear discriminant analysis were 78.2% and 82.9%, respectively.

Hyperspectral imaging tongue diagnosis systems

Hyperspectral imaging collects and processes information from across the electromagnetic spectrum, dividing the light spectrum into a larger number of bands. This technique of dividing images into bands can be extended beyond the visible spectrum. Li et al. developed a hyperspectral imaging system to capture tongue images with a spectrum range of 400 nm – 800 nm and a resolution of less than 5 nm. The number of efficient pixels is 652 pixels × 620 pixels including 120 bands. A microprocessor unit was included in this system to control a walking electromotor, which determines the landscape orientation movement speed. The hyperspectral tongue imaging system captures image scenes in contiguous but narrow spectral bands over the visible wavelength range of the electromagnetic spectrum. Using this system, tongue color calibration and discrimination were performed on hyperspectral medical images. Different tongue colors are represented by corresponding reflectance spectral curves. Because different tongue colors have different reflectance spectral signatures, tongue color can be classified using a spectral angle mapper (SAM). The SAM is an automated method for directly comparing image spectra with known spectra. The colors of tongue substances were divided into six classes and the colors of tongue coatings were separated into four categories. To evaluate the performance of the color classification method, four experienced doctors were asked to label the reference samples, classify 200 tongues using the naked eye, and evaluate the experimental results. As a result, the overall rate of correctness was 85%.

By applying the hyperspectral tongue imaging system, a new sublingual vein extraction algorithm was proposed based on the hidden Markov model (HMM). The HMM is a temporal probabilistic model where the state of the process is described by a single discrete random variable. For each scene of the hyperspectral images, three single-band images at 435, 545, and 630 nm are selected to compose the corresponding pseudo color image. The HMM-based algorithm can extract sublingual veins more accurately than SAM or pixel-based sublingual vein segmentation algorithms for visible contrast sublingual images because it has strong noise rejection and because both spatial and spectral information from hyperspectral sublingual images can be used.

Yamamoto et al. developed a different type of hyperspectral tongue imaging system using a hyperspectral camera equipped with a transmission grating and an array sensor with an 8-bit 480 × 640 pixel monochrome CCD camera. The spectrum range of this system is 400 nm – 800 nm and the resolution of the spectrum is 5 nm. The optical instrument contains both a spectrometer and a scanning mechanism using an internal digital servomotor. The camera is capable of taking a hyperspectral image every 16 s, as a full-sized image. Yamamoto et al. proposed a tongue segmentation method using single-band images, which properly extracted the tongue area without coating by subtracting other areas and focusing on the spectral differences among respective facial areas.

3D-imaging tongue diagnosis systems

The surface of the tongue is made up of many bends and its curvature is complex. The feature of tongue shape is an important parameter for tongue diagnosis, but information acquired through 2D imaging is insufficient for extracting shape features such as thickness or angle of curvature. The bends of the tongue surface can also cause color distortion because the light angle on the tongue surface is not homogeneous and the curvature of the tongue surface frequently changes spontaneously, causing distortion of the color intensity in 2D tongue images. To address these problems, a 3D-imaging TDS is being developed. Liu proposed an image capture system for 3D tongue modeling using two light sources, four digital cameras, a frame for fixing the head, an adjustable laser beam matrix used for producing artificial features on the tongue surface, and a laser device for locating the tongue. The four cameras were capable of adjustable exposure times and high-speed frame rates. An affine invariant of parallelograms was used to calibrate the system. After reconstruction, a finite-element-based representation of the tongue was obtained. An example of a 3D system for tongue imaging is shown in Fig. 4.

Fig. 4

(A) Example of the input images for 3D tongue reconstruction; (B) diagram of 3D tongue acquisition model.

Discussion

With the advancement of technology, various TDS hardware systems have been developed for acquiring accurate and high-quality tongue images. The most important aspect of any tongue image acquisition module is the ability to obtain a reproducible tongue image under varying light conditions. The tongue image obtained using a commercial digital camera without an illuminator is unsuitable for tongue diagnosis. The commercial digital camera has its own internal color bias and environmental noise can severely distort color intensity. The color distortion of the image can be induced by the variation of color temperature of light sources, the inhomogeneity of light intensity and other unexpected light or reflections. The ideal solution is the isolation of the light environment and the use of a sufficient sunlight lamp. Several TDS models have been developed using isolation chambers,3, 6, 8 but this approach has not proved fully sufficient to control illumination noise. Because the curvature of the cheekbone and nose are quite different for each individual, it is difficult to perfectly isolate outside light. Thus, a practical solution is to perform color correction with a distorted image. Several color correction methods such as polynomial regression and topology resolve-map model have been proposed for medical image recently by using ColorChecker. Since the ColorChecker embedded in image provides the basic color information of the ColorChecker, the color distortion of the acquisition system can be estimated more accurately.

Conversely, different postures of the tongue can also contribute to diminished reproducibility. The curvature of the tongue causes a lack of light angle homogeneity, which can distort the color intensity of the image. So far, the postures of the tongue are not under consideration in the TDSs. However, the intensity of light and the angle of light incidence can be changed according to its postures to degenerate the color information of the tongue image. For improving reproducibility, a feedback guideline, which can help adjusting the tongue location and angle, can be practical solution. In the future, light angle correction of each pixel using a 3D imaging system is expected to improve tongue image reproducibility.

Space resolution of the acquisition module is another important characteristic of TDS hardware resources. TDSs have been recently developed using high-resolution CCD, which can acquire a much larger volume of information on the tongue surface. The CCD camera, with a resolution of over 8 megapixels with suitable lenses, is believed to acquire more detailed color information. According to the trend of TDSs, the preferred direction for future development is the implementation of an accurate color acquisition system because the color features of the tongue provide significant diagnostic information. A 3CCD camera and hyperspectral imaging are specialized for color image acquisition, and they have thus been adopted in TDSs. Li et al. and Yamamoto et al. developed hyperspectral imaging TDSs, but they did not use the ColorChecker for color correction. Because the hyperspectral imaging systems are capable of acquiring both spatial and spectral information, the color correction can be performed by normalization procedure in the spectral domain. Hyperspectral imaging is expected to be particularly useful for extracting reliable diagnostic features using images from a narrow frequency band, which include frequency area of other vision.

The tongue segmentation methods, which are based on high-level image segmentation techniques, have been improved to achieve a higher degree of accuracy, but full automation of the segmentation process poses continuing problems. In most cases, the color and the shape information of a tongue are used for tongue segmentation. The active contour model is an effective method to segment the tongue object, where the contour is determined by the sum of energies, namely gradient around the contour position, elastic force and curvature of contour, and these energies are calculated from the color information of the tongue edge. The gradient around the contour position is determined by changes of intensities. The elastic force is related with smoothness of the object contour. To improve its performance, a tongue template force or color gradient vector should be applied to the energies. It is important to improve the accuracy of not only the segmentation method, but also of the acquisition module. In the acquisition process, a slightly unfocused image can cause a segmentation error because the edge pixels of the tongue are blurred. Image resolution and focus time are also significant factors in segmentation. A user-friendly interface for segmentation checking and modification may offer a solution to segmentation error.

For the tongue classification, the clinical features, such as a color of tongue body and coating, distribution of tongue coating, teeth-mark, fissure and shape of tongue, should be quantified, and relations between the features and the technical properties, which are extracted from a tongue image, should be identified. For this purpose, a statistical analysis with massive data was performed by several researchers. Massive tongue image data have been acquired in clinical settings, and then classification techniques, such as SVM and Bayesian networks, have been proposed. Since these machine-learning techniques take advantage of the data to capture characteristics of interest of their unknown underlying probability distribution, the classification rule is changeable according to sample data for the technical properties. Therefore, it is important to identify and standardize the meaningful properties of a tongue. Tongue classification and diagnosis is a difficult process, but extensive and accurate tongue image data can be used to determine standard tongue diagnosis parameters. In the future, the performance of TDSs will continue to improve with the advancement of acquisition devices and classification techniques.

For accurate tongue diagnosis, the various types of TDSs have been developed with diverse techniques. However, the differences of TDSs can be minor effect in terms of a collaborative research or competitiveness in the clinical. Especially, an erroneous diagnosis can result from image resolution, different lighting condition and the posture of a tongue. To solve this problem, an International Standard for the functions and elements of TDS should be proposed. So far, we reviewed the various TDSs in the manuscript. The image acquisition device, light sources, input user interface, color correction, segmentation, features for the tongue classification and safety can be contents of the standard. The review of this manuscript may be expected to helpful for the standardization.

Conclusions

The TDS is an important device for improving quantification and objectivity in tongue diagnosis. Various hardware and software have been developed for TDSs, and the resulting improvement through these diverse technologies has been shown. The various technologies for tongue diagnosis systems have respective unique advantages and specificities according to application and diagnostic environment, but this variation may cause inconsistent diagnoses in practical clinical application. To improve the performance and clinical approach to use of the TDS, an international standard for the functions and elements of TDS should be proposed. In addition, the use of smart phone technology offers the possibility to apply tongue diagnosis in a homecare or mobile setting.

Conflict of interest

There is no conflict of interest.