Application of infrared thermography in computer aided diagnosis.

The invention of thermography, in the 1950s, posed a formidable problem to the research community: What is the relationship between disease and heat radiation captured with Infrared (IR) cameras? The research community responded with a continuous effort to find this crucial relationship. This effort was aided by advances in processing techniques, improved sensitivity and spatial resolution of thermal sensors. However, despite this progress fundamental issues with this imaging modality still remain. The main problem is that the link between disease and heat radiation is complex and in many cases even non-linear. Furthermore, the change in heat radiation as well as the change in radiation pattern, which indicate disease, is minute. On a technical level, this poses high requirements on image capturing and processing. On a more abstract level, these problems lead to inter-observer variability and on an even more abstract level they lead to a lack of trust in this imaging modality. In this review, we adopt the position that these problems can only be solved through a strict application of scientific principles and objective performance assessment. Computing machinery is inherently objective; this helps us to apply scientific principles in a transparent way and to assess the performance results. As a consequence, we aim to promote thermography based Computer-Aided Diagnosis (CAD) systems. Another benefit of CAD systems comes from the fact that the diagnostic accuracy is linked to the capability of the computing machinery and, in general, computers become ever more potent. We predict that a pervasive application of computers and networking technology in medicine will help us to overcome the shortcomings of any single imaging modality and this will pave the way for integrated health care systems which maximize the quality of patient care.

Accuracy

Artificial Neural Network

Area Under Curve

Breast Cancer Detection Demonstration Project

Computer-Aided Diagnosis

Discrete Fourier Transform

Complex Regional Pain Syndrome

Decision Tree

Discrete Wavelet Transform

Fractal Dimension

U.S. Food and Drug Administration

False Negative

False Positive

Fourier Spectrum

Gray Level Co-occurrence Matrix

Higher Order Spectra

Infrared

Infrared Thermography

Local Binary Pattern

Locally Linear Embedding

Law’s Texture Energy

Measurement and Procedure

National Cancer Institute

Performance Evaluation

Probabilistic Neural Network

Probability of Detection

Radial Basis Function

Receiver Operating Characteristic

Regions Of Interest

Severe Acute Respiratory Syndrome

Sensitivity

Self Organizing Map

Specificity

Support Vector Machine

True Negative

True Positive

Weighted Center Of Bispectrum

Introduction

An unexpected increase of heat indicates that something is wrong. This might be one of these universal truths which are almost to sweeping and far-reaching to be credible. In the world of mechanics, increased friction develops heat and causes wear which can led to material failure [1]. Therefore, heat pattern and indeed heat pattern changes give a good indication of machine health [2]. By a striking coincidence, diagnosing via heat pattern is not limited to mechanical systems both electronic and biological systems follow the same pattern. Especially, the human body temperature has been linked to health since the time of Hippocrates [3]. Since then, the diagnostic accuracy was linked to the instruments used for temperature measurement. Therefore, the discovery and capture of Infrared (IR) radiation from the human body, by Herschel in 1800, was a big step forward [3]. These measurements are based on the physical phenomena, that all objects, including the human body, with a temperature above absolute zero (−273 K) emit IR radiation from their surface [4]. Experiments showed that human skin emits IR radiation essentially in the range of 2–20 μm wavelengths, with an average peak at 9–10 μm. Despite this understanding, it took until 1934 before Hardy described the physiological role of IR emission from the human body. He put forward that both physiological processes and thermal properties of the skin are influenced by a wide range of factors, because the skin helps to manage the core body temperature. These factors change in the presence of disease, hence IR measurements can be used for diagnostic purposes [5]. This fundamental understanding paved the way for IR thermography as a body imaging modality for medical sciences. The first use of this new technology was reported in the year 1960, it took so long because quality equipment and technical knowhow was unavailable [6]. Since 1963 it is known that heat patterns, shown in thermographic images taken with IR cameras, provide information about physical anomalies, such as cancer, especially breast cancer, infection, eye disease, diabetes and pain [7]. However, having a qualitative statement based on general observations is something different from a quantitative statement, which expresses how useful the method is for medical diagnosis [8]. In terms of medical applications, the usefulness of a particular diagnostic method can be expressed in terms of the ability to deliver a correct diagnosis, the side effects and the cost.

Heat pattern measurement, via thermography, is fast, non-contact, non-invasive, it is even passive, i.e. no potentially harmful radiation is sent through the biological system, only the body heat is captured [9]. The body would give up this heat radiation in any case, regardless of whether or not it is measured. Passive imaging modalities for medical applications attract much interest, because they are 100% side effect free [10]. Therefore, the US clinical community viewed medical thermography as an exciting and very promising technology for a variety of diagnostic applications, in the decade that followed after the first appearance of IR body images [11]. However, despite this striking advantage over other imaging modalities and the initial enthusiasm, medical thermography has been in decline since 1980 [12]. During this time, the National Cancer Institute (NCI) started the widely cited Breast Cancer Detection Demonstration Project (BCDDP) [13]. One major finding of this project was that thermography is ineffective for breast cancer screening [14]. In retrospect, this study had serious shortcomings in the way thermography images were captured [15]. To be specific, only five out of 27 participating cancer centers had sufficiently qualified personal for taking and assessing thermography images. Gautherie et al. pointed out that a lack of technical skill and expertise for thermography image interpretation leads to a sharp decline in diagnostic accuracy [16]. The need for quality assurance, during the act of image acquisition, is further emphasized by the fact that the high false positive range of 10–30% depends on the disease and the technique used. In recent years, proper protocol, such as equipment considerations preparation of patient, examination environment, and standardization of thermal imager system have been discussed elaborately in [17], [18], [19], [20]. This lead to the postulation of image capture protocols [21], [22]. So, on an abstract level, Infrared Thermography (IRT) based diagnosis suffered from interoperator dependency and inadequate equipment. As a consequence, in 1982 the Food and Drug Administration (FDA) approved thermography only as a supplement to mammography for breast cancer screening [23]. The FDA enforced this judgment in 2011 when the organization issued a press release stating that: “…the FDA is unaware of any valid scientific evidence showing that thermography, when used alone, is effective in screening for breast cancer …” [24].

To overcome the historic prejudices against thermography, and the real problems with the technology, scientific principles have to be applied in a strict way to build trust as well as to improve the diagnostic capabilities of IRT. One of these scientific principles is transparent and objective performance assessment. Therefore, in this review we report the perforce of thermography based diagnostic systems for diabetes, infection, cancer, eye diseases and pain. Based on this review, we adopt the position that thermography image interpretation can be aided with computer preprocessing. From there it is just a small, but important step, to automate the diagnosis process as well. The importance of this step comes from the promise of progress: As computing machines get more capable, and algorithms improve, the diagnostic accuracy will increase. Another important benefit of Computer-Aided Diagnosis (CAD) is cost efficiency. Once these systems are up and running, the operation cost is low, when compared with expert human labor. Furthermore, CAD systems integrate seamlessly into digital work-flows of modern hospitals. This high level of integration and the low human interaction results in two beneficial properties: fast throughput and centralized processing. As a result, there is a real chance that future thermography based CAD systems will deliver unparalleled safety together with a high level of reliability and diagnostic accuracy.

To support our position, we have organized the paper as follows: The next section gives an overview of the algorithms used in thermography based CAD systems. Section 3 reviews to what extent these techniques were used in current and past thermography systems. In Section 4 we discuss the review results. Section 5 provides concluding remarks and an attempt to define the future direction of thermography based CAD systems.

Methodology of the CAD system

CAD plays a significant role in the analysis of IR images. Human examination of IR images is often imperfect, because it is influenced by various factors like fatigue, being careless, and sensory overload by the sheer volume of information from this medical imaging modality. Making the right diagnosis is also constrained by the limits of human visual perception. For example, optical illusions have a detrimental impact on the diagnostic accuracy [25]. Furthermore, there is a shortage of qualified radiologists [26]. These facts manifest an urgent need to develop CAD technologies. Current research aims to improve the detection accuracy of IR image processing algorithms from three perspectives: asymmetry analysis of the thermogram including automatic segmentation approaches, smart image enhancement and restoration algorithms, and feature extraction and classification [27], [28], [29], [30].

To design such CAD systems, a clear understanding of design patterns and system structure is needed. To explain the system structure, we have adopted a top down approach which introduces the individual design steps necessary to create a generic thermography based CAD system. The block diagram, shown in Fig. 1 , introduces these design steps and the logical connection between them. The diagram is divided into offline and online systems. In the offline system, all the IR images will be pre-processed and their greyscale features may be extracted using various techniques. Subsequently, the extracted greyscale features will be analyzed with statistical methods to identify which features are significant enough to be used in the classification step. The significant features, together with the ground truth of each image class, will then be used to train a classifier. The result from this offline system is a set of classifier parameters that relates the significant features to the ground truth. The online system has a pre-processing stage as well and thereafter only those greyscale features, that were found to be significant in the offline system, are computed for the testing set of images.

Fig. 1

Blockdiagram of a CAD system based on thermograms.

Pre-processing

Preprocessing deals with the non-uniformity of IR images. Another source of non-uniformity is fluctuating lighting conditions such as incandescent bulb or direct sun light. Therefore, it may be required to resize the images using techniques, such as template and interpolation methods [31]. Another source of non-uniformity is fluctuating lighting conditions. The adaptive histogram equalization method can eliminate the discontinuous background, which may result from discontinuous lighting or other inconsistencies that originate from the IR image capturing technique [32, Chapter 2]. It maps the input to the output image pixel intensity values such that the output image exhibits uniformly distributed intensities. This increases the dynamic range of the image histogram.

Feature extraction

Textures, entropies, Fourier Spectrum (FS) descriptors, Hu’s invariant moments and Higher Order Spectra (HOS) methods can be used to extract features [33]. These methods are briefly explained below.

Texture

Textural changes are a totally new paradigm for feature collection. Gray Level Cooccurrence Matrix (GLCM) [34], run length matrix [35], Fractal Dimension (FD) [36], [37], [38], Local Binary Pattern (LBP) [39], Locally Linear Embedding (LLE) [40] and Law’s Texture Energy (LTE) [41], [42] can be used to extract useful information for the differential diagnosis of normal and abnormal classes.

Entropy

Entropy measures the randomness of the pixel intensity distribution in an image. It provides insight into both extent and nature of randomness and it shows how disordered the image pixels are. There are two categories of entropy: Shannon and non-Shannon [43]. Over a range of scattering conditions, Shannon entropy has a lower dynamic range than non-Shannon entropies. This makes non-Shannon entropies more useful in estimating scatter density and regularity. Examples of non-Shannon entropies, that can be used to extract features for automated disease diagnosis, are: Renyi’s, Kapur’s and Yager’s entropy [44], [45], [46].

Hu’s invariant moments

The global characteristics of shapes, within an image, are generally represented by a set of moment functions. This set of moments provides information about the different types of geometrical image features. Hu’s invariant moments are a set of non-linear functions that were developed from regular moments. Their main advantage comes from the fact that they are invariant to translation, size alteration and rotation. Hu’s invariants describe a total of seven moment functions, which reach up to the third order [47].

Discrete Wavelet Transform (DWT)

The Discrete Wavelet Transform (DWT) captures both frequency and location of features within a thermorgraphy image. This is important, because this information helps the CAD system to identify features of interest. For example, features, such as the thermographic pattern of breast cancer, have sharp borders. These sharp borders shape high frequency DWT coefficients with high levels of location (spatial) accuracy. In contrast, larger objects with smooth edges shape low frequency coefficients with low spatial resolution [48]. The DWT coefficients are established by sending a signal through a sequence of down-sampling low- and high-pass filters [49], [50]. The low- and high-pass filter output is known as approximation and detail coefficients respectively. Mookiah et al. used a novel combination of one texture and two DWT-based features to quantify the non-linear changes in malignant and normal breast IR images [51].

Higher Order Spectra (HOS)

HOS, usually referred to as polyspectra, consist of higher order moments that have an order greater than two. Furthermore, polyspectra also include non-linear combinations of higher order moments, known as cumulants. Third and fourth order spectra are commonly referred to as bispectrum and trispectrum respectively [52]. Features, related to the third order spectra, i.e. bispectra, can be used to extract information for automated diagnosis [53], [54].

Before the extraction of greyscale features via HOS, the images need to be subjected to the Radon transform. This transform will convert 2D images into 1D signals at various angles and thereafter the calculation of greyscale features, using HOS, will begin [55].

The Fourier Transform, of the third order 1D signal correlation, will be used to find the bispectrum. The following formula defines this relationship:where, is the Fourier transform of the 1D signal and represents the expectation operation. Both bispectrum phase entropy [56] and normalized bispectrum entropies can be used as features. Various bispectrum entropies, bispctrum phase entropy, Weighted Center Of Bispectrum (WCOB), sum of logarithmic amplitudes can be evaluated from the bispectrum of the signal [57], [58].

Fourier Spectrum (FS) descriptors

To quantify changes in the outline (edges or contours) of image regions, the FS descriptor is used, because these descriptors are not invariant to scaling and translation [59]. For any edge, within an image, having coordinates , where represent the edge points of an object, the FS descriptors are denoted by the following expression:where . In Eq. (2), each point is treated as a complex number [60]. The FS descriptors are defined as the Discrete Fourier Transform (DFT) coefficients of the complex vector , which is formed by the edge points. It is expressed by:

Statistical analysis

Statistical analysis is part of the quantitative approach to knowledge. During the design of thermography based CAD systems, statistical analysis is used to summarize the data and understand the processes which generate the data. In most cases, this is done by making a mathematical model from which the data generating processes can be inferred. Furthermore, statistical analysis gives us a way to quantify how confident we are in these inferences [61], [62], [63].

Feature selection

Student’s t-test can be used to detect the clinical significance of extracted features by evaluating the p-value. This value indicates whether or not the means of two classes are statistically different [64]. A lower p-value indicates that the feature has more clinical significance [65]. To evaluate the feature performance for CAD systems, the feature which has the lowest p-value is ranked first and so on. Subsequently, the ranked features are applied one by one until the highest classification accuracy is reached.

Classification

Automated classification algorithms are a solution to the machine learning and statistics problem of finding the category or class of a new observation. Classifiers have been used in a wide range of CAD systems [66]. Supervised algorithms call for training data, which contains observations where the category membership is known, i.e. the ground truth is available. The main idea behind these classification algorithms is to extract information, which differentiates the individual categories, from the training dataset. The act of information extraction is known as learning phase. The extracted information is used in the classification phase to decide to which category or class a new observation belongs.

Support Vector Machine (SVM)

Support Vector Machine (SVM) classifiers train easily and they can generalize the training information to a wide range of classification problems. The algorithm searches for a hyperplane, which subsequently serves as a decision surface. This surface separates the datapoints, which belong to distinct classes, from each other with maximum margin. SVM are popular, because of their high classification accuracy even under adverse conditions, such as non-linear and high dimensional datasets and very few cases for training the model [66], [67].

If the data is non-linear, the boundary problems, encountered by the SVM, are non-linear as well [68], [69]. To handle such data, kernels are used to facilitate a non-linear mapping of the input data to a high-dimensional space [70]. Various kernel functions like, linear, polynomial of order 2, and 3 as well as Radial Basis Function (RBF), can be used.

Decision Tree (DT)

The Decision Tree (DT) is a simplified model that breaks down a complicated decision making process into a simple sequence of binary questions. It consists of decision nodes which extend downwards from a root node [71, Chapter 6]. An attribute is tested with each outcome in a branch at each decision node. The branches may end at another decision node or terminate at a leaf node. Nicandro et al. used this classifier to evaluate the diagnostic power of thermography in breast cancer [72].

Fuzzy Sugeno

The pattern space, in a Fuzzy Sugeno classifier, is divided into many subspaces. For each subspace, if-then type rules show the relationship between targeting the patterns and their corresponding classes. The Fuzzy Sugeno classifier discriminates unknown patterns with fuzzy inference and rejects pattern of an unknown class [73], [74]. This type of classifier was successfully used, together with a hybrid feature extraction strategy, for breast cancer detection [51].

k-Nearest Neighbour

k-Nearest Neighbour is a very simple classifier which determines the k nearest neighbors for a specific query instance by calculating the minimum distance to the training samples [75], [76]. A data point is classified by the majority of votes from these k neighbors. If k is 1, then the data point is assigned to the class of the nearest neighbor. For well designed algorithms, nearer data points give a higher contribution as compared to data points which are further apart [51].

Probabilistic Neural Network (PNN)

The Probabilistic Neural Network (PNN) classifier uses four layers of neurons to implement a kernel discriminant analysis step [77], [78]. Features, extracted from IR images, are fed from the input layer into a hidden later which computes of the Euclidean distance of the test data from the center point of the hidden neuron [51]. The RBF kernel function uses the sigma values, which were computed as part of the Euclidean distance evaluation in the hidden layer. The results from the hidden layer are fed to the pattern layer, which contains a pattern neuron for each class. Each weighted vote is added to the corresponding neuron from the hidden layer in the pattern layer. In the final layer, the largest votes are used to predict the unknown class by comparing the weighted votes, for each class, with predetermined values, which are stored in the pattern layer.

Self Organizing Map (SOM)

The Self Organizing Map (SOM) method visualizes high dimensional data by converting complex, non-linear statistical relationships into straight forward geometric relationships on a low-dimensional plain [79]. With this conversion, the algorithm compresses the data while preserving the most important topological and metric relationships of the initial data elements [80]. This creates a special kind of abstraction. These abilities are particularly important for extracting information from thermogram images [81], [82].

Artificial Neural Network (ANN)

Artificial Neural Networks (ANN) are parallel-distributed information processing structures which model the functionality of small biological neural clusters and thereby they mimic human decision making [83]. In many cases, ANN are trained with the so called backpropagation algorithm [84]. Due to its parallel nature, ANN algorithms can reach a decision after a relatively short processing time. Therefore, they are applied in hard realtime systems, such as mass fever scanning [82].

Data resampling and performance evaluation

The cross-validation method assesses a classifier with a training dataset [85]. For this technique, the entire dataset is randomly split into k equal (almost) parts. Each part contains the same ratio of samples from both classes [86]. In the first iteration (fold), data parts are used to train the classifier and the remaining part is used for testing. The iteration is repeated times using a different test set (with the remaining folds as training sets) each time. This procedure is used to develop classifiers for thermography based CAD systems [87].

In the offline system, the classification results are assessed by performance measures, such as True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN) Accuracy (A), Sensitivity (Se) and Specificity (Sp). This assessment helps to select the most appropriate classification algorithm for online system. The individual measures are briefly explained below:

TN = Number of normal data classified as normal.

FN = Number of depression data classified as normal.

TP = Number of depression data correctly classified as they are.

FP = Number of normal data classified as abnormal (depression).

A = (TP + TN)/(TP + FN + TN + FP).

Se = TP/(TP + FN).

Sp = TN/(TN + FP).

Receiver Operating Characteristic (ROC)

The diagnostic accuracy is a measure for the clinical significance of a CAD system [88]. Receiver Operating Characteristic (ROC) plots provide a pure index of accuracy by demonstrating the limits of a test’s ability to discriminate between alternative states of health over the complete spectrum of operating conditions. A wide range of studies on IR thermography used the ROC curve to measure the cutoff point, diagnostic accuracy (indicated by the area under the curve), sensitivity, and specificity [67], [89]. Despite these positive characteristics, Cook points out that ROC measures may be mediocre in assessing models that predict future risk or stratify individuals into risk categories [90].

From the algorithmic perspective, ROC is a method to evaluate the ability of a test to discriminate diseased cases from normal cases. ROC allows us to perform an objective comparison between two or more imaging modalities. ROC is a useful approach because other methods do not quantify diagnostic accuracy in sufficiently complete or meaningful way. In brief, ROC analysis is used to select the optimal cut point to dichotomize a continuous scale. The usual choice of cut points minimizes the overall number of false positive and false negative errors. The cut point may be shifted if the cost of false positives is higher than that of false negatives, or vice versa.

Probability of Detection (POD) is based on TN, FN, TP and FP. The best way to describe a confidence curve, say with 98%, is as follows: If the actual POD curve, were to be reconstructed repeatedly using the same method and data, then 98% of those constructed curves would be above the confidence curve (i.e. 2% would be below). As a consequence, we are 98% sure that the REAL POD curve is above the confidence curve. The confidence level must also consider the effects of the full matrix for POD where we have the potential for false calls.

Applications

This section discusses four prominent application areas where thermography is used for disease diagnosis. The first of these areas is diabetes diagnosis, more specifically CAD systems for diabetic foot. The next important application area is infection diagnosis. Thermography based CAD systems help to screen for fever patients in places with a high volume of people, such as airports and border crossings. Cancer, especially breast cancer in woman, is the next application area. In this area, thermography has a weaker position against the established mammography. However, this fact sparks some of the most advanced work on thermography based CAD systems. The work on eye diseases is less advanced, because it suffers from the limited availability of IR images. The CAD for pain is a very complex concept, because pain itself is very difficult to diagnose. Despite this fact, there are a number of CAD systems for this application area.

The review of the thermography systems, from these diverse application areas, focuses on the building blocks of the CAD system. This strategy ensures transferable and relevant results across the application areas. The assessment begins with Measurement and Procedure (MP). This performance parameter is composed from an integer, which indicates the measurement used, and a character, that suggests how the procedure is applied. In the measurement section (integer) we assess the work according to the following criteria:

Points of interest – One or multiple discrete measurement points on the IR image.

Regions Of Interest (ROI) or template – Predefined regions where an average temperature is obtained.

Feature obtained via image processing – A single measure which takes into account the heat distribution structure of the human skin.

Feature vector obtained via image processing – Multiple measures which are obtained by assessing the heat distribution structure. The results of the individual measures form a feature vector.

Feature vector whose elements come from diverse sources – This setup represents the most advanced form of CAD systems. For most systems the resulting feature vectors are used as input to automated classification algorithms.

The procedure part (character) of the MP criteria, establishes the way in which the studies were conducted. This is an important assessment criterion, because the false positive rate of the diagnosis depends, to a large extend, on this setup:

Comparison between groups of individuals – Most studies in this category discriminate between IR images from diseased and normal individuals.

Comparison between different symmetrical parts of the human body – This method is based on the fact that the human thermoregulatory system is substantially symmetrical [91]. The advantage of differential over group based methods is that for a symmetrical measurement there is no need to establish a general ground truth [92]. To be specific, the relevant information is extracted from the discrepancy between the temperature patterns exhibited by symmetrical parts of the human body [93].

The Performance Evaluation (PE) criteria assess the work in terms of the following criteria:

Statistical analysis – This analysis can be done with Student’s t-test which assesses whether or not two or more feature sets are statically different. Less advanced statistical analysis methods evaluate just mean and variance values.

Threshold – Thresholding is the simplest classification method. It takes only a single scalar value into account. For thermography, this single scalar value is usually the temperature measured in one point or in a region of the IR image. A temperature value above the threshold indicates disease and a value below the threshold indicates a normal sample. More advanced analysis is done using ROC and Area Under Curve (AUC) [67], [88], [94]. However, both methods are based on scalar thresholding as well.

Classifier – Automated classification algorithms represent the most advanced way to discriminate different signal classes, or in terms of CAD systems, to discriminate normal from diseased individuals. The reason for this superiority comes from the fact that classification algorithms take into account feature vectors, which represent relevant information about the image. In general, each feature vector constitutes a point in high dimensional vector spaces. This technique is much more flexible, and truer to the nature of diagnosis, 1 than simple thresholding methods. To be specific, thresholding methods are limited to discriminate points on a line, i.e. one dimensional space. In most practical scenarios, this is insufficient to achieve an acceptable classification accuracy.

The subsequent Sections describe five different application areas. Each of these sections contains a table which describes the thermography based CAD methods used in this application area. Fig. 2 shows the table header with additional information.

Fig. 2

Header structure of the result tables.

Diabetes

Thermography can be used to visualize morphological skin temperature pattern that depend on blood circulation. In ischemic conditions, where blood perfusion may be reduced, especially at the periphery of the human body and limbs (hands and feet), these temperature pattern change [95]. Diabetic foot complications are expensive and they reduce the quality of life for many patients [96]. More specifically, neuropathic foot ulceration is a major cause of morbidity in patients with diabetes and it commonly leads to prolonged hospital stays [97]. Table 1 represents the work published on CAD systems for diabetes detection. Each row details one study and this study was assessed in terms of MP, PE, A, Se and Sp. For example, the study by Peregrina et al. was done in 2013 [98]. MP = 2b indicates that they used ROI or a template to determine the temperature from an IR image and that they compare the temperature differences of symmetrical body parts. 2 The temperature results were analyzed with statistical methods. Neither threshold nor classification methods were used, therefore A, Se and Sp were not reported.

Table 1

Diabetes diagnosis performance.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Peregrina et al. [98]	2013	2b	1	–	–	–
Sivanandam et al. [94]	2013	1a	2	72	90	55
Tamaki [99]	2013	2b	1	–	60	100
Balbinot et al. [100]	2013	1b	1	–	–	–
Mori et al. [101]	2013	3b	1	–	–	–
Najafi et al. [102]	2012	2b	1	–	–	–
Barriga et al. [103]	2012	2b	1	–	–	–
Balbinot et al. [63]	2012	1b	2	–	81.3	46.2
Nagase et al. [61]	2011	1b	1	–	–	–
Bagavathiappan et al. [104]	2010	2b	1	–	–	–
Kaabouch et al. [105]	2009	3b	–	–	–	–
Lavery et al. [106]	2007	1a	1	–	–	–
Sun et al. [107]	2006	2a	1	–	–	–
Armstrong et al. [108]	2006	3b	1	–	–	–
Bharara et al. [109]	2006	1b	1	–	–	–
Marcinkowska-Gapińska and Kowal [110]	2006	2a	1	–	–	–
Sun et al. [111]	2005	2a	1	–	–	–
Armstrong et al. [112]	2003	1b	1	–	–	–
Jiang et al. [113]	2002	2a	1	–	–	–
Fujiwara et al. [114]	2000	2a	1	–	–	–
Hosaki et al. [115]	1999	2b	–	–	–
Armstrong et al. [116]	1997	1b	1	–	–	–
Benbow et al. [62]	1994	1b	1	–	–	–
Stess et al. [117]	1986	2a	1	–	–	–
Fushimi et al. [118]	1985	3b	1	–	–	–
Sandrow et al. [119]	1972	2-	–	–	–	–
Brånemark et al. [120]	1967	1a	–	–	–	–

Fever

At any one time, during the temperature regulation process, the human body establishes a temperature value as its so-called “set point” [121]. Fever happens if the hypothalamus detects pyrogens and then raises the set point. IR thermography systems are very well suited for non-contact blind mass fever scanning, because image capturing is fast, non-invasive, and it is able to reliably detect people with fever. Therefore, this technique was used to detect inflammatory abnormalities and it has the potential to become a tool for mass scanning of fever [122]. Fig. 3 shows three thermogram images captured during a fever scanning exercise. The square box in the thermograms indicates the ROI. Users need to carry out pre-screening system verification to verify and fine tune the settings by comparing the screening result with a calibrated clinical thermometer. Since the thermal imager measures the skin temperature and not the body core temperature, the threshold setting needs to take into consideration the physiological site offset and the performance characteristics of the thermal imager in order to obtain a reliable screening operation. Studies show that there is a correlation between the human skin surface temperature and the body core temperature. However, the extent of correlation depends on the changes in skin temperature. Skin temperature may be altered at different environmental and physiological conditions. Skin temperature may not have direct relationship to the infectious stages of the diseases. Table 2 details the work published on CAD systems for infection detection and fever scanning.

Fig. 3

Examples of temperature profiles using fever thermal imager with temperature reading (Reproduced with permission of SPRING Singapore. Copyright remains with SS 582: Part 2: 2013 – ‘Specification for thermal imagers for human temperature scanning’). The picture captions report the mean body temperature, measured by an aural thermometer.

Table 2

Infection results.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Sun et al. [81]	2013	5a	3	–	92.3	92.3
Singler et al. [123]	2013	2a	2	–	–	–
Chan et al. [124]	2013	2a	2	–	74	79
Romano et al. [125]	2013	2b	2	90	89	91
Priest et al. [126]	2011	2a	2	–	86	71
Nishiura and Kamiya [127]	2011	2a	2	–	72.4	81.7
Matsui et al. [128]	2009	5a	1	–	–	–
Hausfater et al. [129]	2008	2a	2	90	82	90
Chiang et al. [130]	2008	2a	2	–	40	77
Ng [131]	2007	2a	3		100	94.3
Kee and Ng [132]	2007	2a	3	96	95	85.6
Ng et al. [133]	2005	2a	3	98	–	–
Chiu et al. [134]	2005	2a	2	–	75	99.6
Ng et al. [135]	2005	2a	2	–	89.4	75.4
Ng [136]	2005	2a	2	–	90.7	75.8
Ng et al. [137]	2004	2a	2	–	85.4	95
Ng et al. [138]	2004	2a	2	–	85	95
Chan et al. [139]	2004	2a	2	–	67	96
Ng and Chong [82]	2006	2a	3	97.5	–	–
Liu et al. [140]	2004	2a	2	24	17.3	98.2
Clark and Stothers [141]	1980	1a	1	–	–	–

Cancer

The surface temperature around the area of cancerous cells is slightly higher than normal tissues and this area is seen as hot spots on thermograms [33]. This temperature increase is caused by the high metabolic activity of cancer cells [142]. IR imaging has frequently been used in clinics to detect changes in skin surface temperature associated with some superficial tumors [143]. We found 32 studies on thermography based breast cancer detection. These studies are discussed in the next section. Section 3.3.2 details thermography based skin cancer CAD systems.

Breast cancer

On a global scale, breast cancer accounts for 22.9% of all cancers (excluding non-melanoma skin cancers) in women [144]. In 2008, breast cancer caused 458,503 deaths worldwide (13.7% of cancer deaths in women) [145]. These statistics highlight the importance of breast cancer research. The research community has responded with a substantial body of work on this specific disease and IRT is at the forefront of this effort.

A breast harboring a cancer can produce an increase in IR emission causing a disparity in the thermographic skin pattern of the two breasts [146]. Therefore, in normal breast thermograms, symmetric heat patterns are observed in both breasts, but in the case of an unilateral abnormality, asymmetry is observed. Thermography is especially well suited for picking up tumors in their early stages or tumors in dense tissue and in these scenarios it outperforms other modalities, such as mammography [142]. Fig. 4 shows normal (Fig. 4a) and malignant (Fig. 4b) thermography images of the female breast. The malignant part is clearly visible in Fig. 4b. Studies have shown a high correlation between obviously abnormal thermograms and increased breast cancer risk as well as a poorer prognosis for the breast cancer patient [14], [16]. A constant irregular thermogram carries a 22 times higher risk than its regular counterpart and it is 10 times more indicative than a first-order family history of the illness as a future risk signal for breast cancer. Studies have indicated that an irregular IR image is probably the single most crucial risk marker for the presence, or future growth, of breast cancer [147]. Table 3 details the work published on CAD systems for breast cancer detection.

Fig. 4

Typical breast themograms. (The figure is reproduced from International Journal of Thermal Sciences 48 (2009) 849–859 from Elsevier Masson SAS. All rights reserved.)

Table 3

Breast cancer diagnosis results.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Kolarić et al. [148]	2013	2b	2	92	100	79
Francis and Sasikala [33]	2013	3b	3	–	88.1	85.71
EtehadTavakol et al. [55]	2013	3b	3	95	–	–
EtehadTavakol et al. [48]	2013	3b	3	86	–	–
Zore et al. [149]	2013	2b	1	–	–	–
Nicandro et al. [72]	2013	4b	3	71.88	82	37
Sella et al. [67]	2013	3b	3	–	90.9	72.5
Francis and Sasikala [150]	2013	3b	3	85.19	88.89	77.78
Acharya et al. [151]	2012	3b	3	90	76	84
Boquete et al. [152]	2012	3b	4	–	100	94.7
Zore et al. [153]	2012	b	1	–	–	–
Acharya et al. [87]	2012	3b	3	88.10	85.71	90.48
Mookiah et al. [51]	2012	3b	3	93.3	86.70	100
Kontos et al. [154]	2011	3b	2	–	25	85
Grubisic et al. [155]	2011	4b	–	–	–	–
Wishart et al. [156]	2010	3b	3	–	78	75
Wiecek et al. [157]	2010	3b	3	86.60	–	–
Schaefer et al. [142]	2009	3b	3	80	93.10	99.15
Arora et al. [158]	2008	3b	3	–	97	44
Tan et al. [89]	2007	3b	4	94.74	100	60
Qi et al. [159]	2007	3b	1	–	–	–
Ng and Kee [160]	2007	3b	3	80.95	81.2	88.2
Yang et al. [161]	2007	2a	1	–	–	–
Jakubowska et al. [162]	2003	4b	1	–	–	–
Ng et al. [163]	2002	3b	3	61.54	68.97	40
Frize et al. [164]	2002	2b	1	–	–	–
Kuruganti and Qi [165]	2002	3b	1	–	–	–
Ng et al. [166]	2001	3b	2	59	54	67
Ng et al. [167]	2001	3b	–	–	–	–
Keyserlingk et al. [168]	1998	2a	1	–	–	–
Thompson et al. [169]	1978	2b	1	–	–	–
Folberth and Heim [170]	1984	2a	1	–	–	–

Skin cancer

Flores-Sahagun et al. showed that there are significant differences in skin thermal mapping between patients suffering from basal cell carcinoma and seborrhoeic keratosis [171]. These differences come from the fact that benign skin lesions have a lower mean temperature than the surrounding healthy skin and cancerous skin mutations have a higher mean temperature [172]. IRT can be used to capture the heat patterns of the skin and the resulting images can be used for diagnosis. The sensitivity of the CAD system depends on the lesion diameter [173]. Table 4 details the work published on CAD systems for skin cancer detection.

Table 4

Skin cancer diagnosis results.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Cholewka et al. [172]	2013	2a	1	–	–	–
Garcia-Romero et al. [174]	2013	2a	1	–	–	–
Shada et al. [173]	2013	2a	2	–	95	100
Cholewka et al. [175]	2012	2a	1	–	–	–
Flores-Sahagun et al. [171]	2011	1a	1	–	–	–
Aweda et al. [176]	2010	2a	1	–	–	–
Buzug et al. [177]	2006	3a	1	–	–	–
Button et al. [59]	2004	2a	1	–	–	–

Eye

Eye thermography measures temperature changes in the vascular tissues. These temperature changes can be linked to eye diseases, such as dry eye and dry eye after eye lid reconstruction [178]. Therefore, this technique is beginning to play an important role in the field of ophthalmology [179]. Fig. 5 shows an example of thermography based dry eye detection. Table 5 details the work published on CAD systems for eye disease detection.

Fig. 5

IR images of the human eye.

Table 5

Eye disease diagnosis results.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Klamann et al. [180]	2013	2a	1	–	–	–
Arita et al. [181]	2013	2a	1	–	–	–
Purslow [182]	2013	1a	1	–	–	–
Klamann et al. [183]	2013	2a	1	–	–	–
Petznick et al. [184]	2013	2a	1	–	–	–
Gonnermann et al. [178]	2012	2a	1	–	–	–
Kottaiyan et al. [185]	2012	2a	1	–	–	–
Tan et al. [186]	2011	4a	1	–	–	–
Kamao et al. [187]	2011	2a	2	–	83	80
Tan et al. [188]	2010	4a	1	–	–	–
Tan et al. [189]	2010	3a	1	–	–	–
Tan et al. [190]	2010	2a	1	–	–	–
Acharya et al. [191]	2009	4a	1	–	–	–
Chiang et al. [192]	2006	2a	2	–	79.3	75
Purslow et al. [193]	2005	2a	2	–	–	–
Cherkas et al. [194]	2003	2a	1	–	–	–
Morgan et al. [195]	1999	1a	–	–	–	–
Mori et al. [196]	1997	1a	1	–	–	–
Morgan et al. [197]	1996	2a	1	–	–	–
Morgan et al. [198]	1995	2a	1	–	–	–

Pain and inflammation

One symptom of pain is excessive body heat, which can be measured with thermography. Such measurements are important, because in some cases it is difficult to locate the physical source of pain. For example, the Complex Regional Pain Syndrome (CRPS) develops after a minor trauma of the distal limbs. A gold standard in diagnosing CRPS has not been found yet, diagnostics are based on the patient’s medical history and correlating clinical signs [199]. IRT can help medical practitioners to detect areas of cutaneous thermal changes of neural origin and it can lead to a high degree of interexaminer agreement for assessing skin temperature differences in patients with CRPS [200], [201]. Kang et al. show that IRT can be used in the diagnosis and assessment of therapeutic results for erythromelalgia [202]. Table 6 details the work published on CAD systems for pain detection.

Table 6

Pain and inflammation diagnosis results.

Authors	Year	MP	PE	A (%)	Se (%)	Sp (%)
Jeong et al. [203]	2013	2b	–	–	–	–
Dibai Filho et al. [204]	2013	1b	1	–	–	–
Kang et al. [202]	2013	2b	–	–	–	–
Rodrigues-Bigaton et al. [88]	2013	1a	2	–	62.2	75.6
Dibai Filho et al. [205]	2013	1a	2	60	55.8	55.8
Zaproudina et al. [206]	2013	2a	1	–	–	–
Roy et al. [207]	2013	2a	1	–	–	–
Choi et al. [201]	2013	2b	1	–	–	–
Zaproudina et al. [208]	2013	2a	1	–	–	–
Frize and Ogungbemile [209]	2012	3b	3	–	96	92
Hildebrandt et al. [210]	2012	2b	–	–	–	–
Laino [211]	2012	2b	–	–	–	–
Wu et al. [212]	2009	2b	1	–	–	–
Chang et al. [213]	2008	1b	1	–	–	–
Niehof et al. [214]	2007	2a	2	–	74.3	83.9
Park et al. [215]	2007	2b	1	–	–	–
Herry and Frize [216]	2004	2a	2	–	78	83
Canavan and Gratt [217]	1995	2b	2	89	85	92
Tchou et al. [218]	1992	2a	1	–	–	–
Ben-Eliyahu [219]	1991	2a	2	–	97	90
Herrick and Herrick [220]	1987	2b	2	–	97	100

Discussion

In this paper we reviewed thermography based CAD systems. We found that over the years tremendous progress was made in terms of accuracy, sensitivity and specificity of these systems. This progress was fueled by better sensors [156], [207], diagnostic procedure [169], [221], [222], [223], more computing power and a deeper understanding of the processing algorithms [224], [225], [226], [227]. Standardization and quality assurance efforts, such as the human skin temperature atlas database, have deepened and solidified this progress [228], [229], [230], [231], [232]. However, in terms of applicability of thermography based CAD systems, there is still a mixed picture. On the one hand there are the novel applications, such as dry eye and pain indication, where thermography is the only viable imaging modality, on the other hand there are applications where thermography stands in direct competition with active imaging modalities, such as ultrasound and X-ray. When the CAD systems, for the novel applications, reach majority they will be a great help for physicians to diagnose diseases, such as pain, for which it was historically difficult to reach a conclusive diagnosis [233]. The picture is different for applications of thermography where there is direct competition from active imaging modalities. The main advantage of active imaging modalities comes from the fact that an active system knows what signal was send out and what signal was received. So, these systems can analyze the changes the human body did to these signals. Thermography does not have the benefit of knowing the signal source, because it is the body itself which radiates the signals. Unfortunately, the ways in which the body generates heat are only rudimentarily understood.

In each of the five investigated areas, thermography has a different standing when compared with other diagnostic tools. During our review, we recognized that the most advanced analysis methods were used in areas where thermography has a strong competition. There is a strong correlation between competition and system quality in the area of breast cancer screening, where thermography is recognized only as an adjunct tool to mammography [234], [235]. Mammography is accepted as the most reliable and cost-effective imaging modality, however its contribution continues to be challenged with persistent false-negative rates ranging up to 30% [236]. In light of the continued controversy surrounding established breast imaging modalities, such as mammography [237], [238], a number of new and emerging technologies are being developed for breast cancer screening and diagnosis [239]. Even back in 1973, it was established that the combination of thermography and mammography achieves better results, for diagnosing asymptomatic breast cancer, then each method individually [240]. However, the interpretation of thermograms is heavily dependent on the analysts, which may be inconsistent and error-prone [89]. Therefore, breast cancer screening with IR imaging has still a weaker position [241]. But as a consequence of this struggle, IR based breast cancer screening is the area where most of the progress was made. The progress in this field is documented by the fact that in this review 20 studies, out of 32, report a measure of diagnostic quality, i.e. either A, Se or Sp. The increase in diagnostic quality is fueled by two facts: (a) IR cameras are getting better and cheaper, (b) Computing machinery is getting ever more powerful and there are better algorithms [242], [243]. At least the later point is also true for mammography, but for this imaging modality the margins of improvement are rather slim. Therefore, we predict that over the coming years the progress of thermography will outpace the progress of mammography for breast cancer diagnosis. Furthermore, the image of thermography for breast cancer screening might be better in the general public than in the research community, because a survey shows that online advertising for thermography is effective and woman consider the uptake of alternative breast imaging services over mammography [244].

Thermography is an important tool to measure diabetes induced changes in the human body physiognomy. 16 out of the 27 studies on thermography based CAD systems for diabetes assessed temperature differences in symmetrical parts of the human body, in most cases the foot. For these different measurements, there is no need for to establish the symmetrical heat pattern for a large group of normal subjects, i.e. there is no need to establish or indeed rely on a pre-established heat pattern atlas. For contralateral asymmetry based studies normal subjects are used to establish the control group. Diagnostic progress is made by applying image processing and feature extraction algorithms which extract information from the differences in skin temperature pattern between the two feet. However, our review shows that these feature extraction algorithms and the subsequent machine classification is not applied consistently. To be specific, the PE column of Table 1 shows that no classification algorithms were used for performance evaluation.

Thermography is the imaging modality of choice for fever scanning, because this method of data acquisition has several distinct advantages, such as imaging speed, non-contact measurement and passiveness [245]. The most significant problem associated with the use of thermal imagers comes from the fact that there is no objective way to determine the body temperature of an individual [246]. The correlation of IRT temperatures with the core temperature is significant but weak [247]. The measurements do not depend on skin color, they depend on skin surface properties such as wrinkled or shiny skin, dry or sweaty skin, emissivity, reflectivity and transmissivity [248]. On the positive side, this imaging modality saves time (temperature is displayed within a few seconds) and it reduces close contacts with infected individuals [249]. However, the effectiveness in a practical setting is hard to asses. For example, during the Severe Acute Respiratory Syndrome (SARS) epidemic of 2003, thermal scanning of over 35 million international travelers entering Canada, China, Hong Kong SAR, and Singapore did not pick up a single SARS case [250]. One reason for this mediocre performance might be the simple thresholding methods which were used during the fever scanning. More advanced feature extraction algorithms and the combination of multi-sensor data, similar to the system proposed by Sun et al. [81], might improve the performance of these CAD systems.

Thermography is a novel method to measure the anterior segment temperature of a human eye. It is generally agreed that the applications of ocular surface temperature measurement can include dry eye, contact lenswear, corneal sensitivity, refractive surgery, and other ocular surface disorders. All studies, discussed in Section 3.4, were based on the comparison of IR eye images from healthy normal and diseased individuals. Most of these studies were based on a small sample size, especially the number of IR images from diseased patients was low. This lack of images is a big problem, because non-symmetric IR measurements demand a large sample size to be statistically significant.

It is very difficult to objectify pain. The difference of skin temperature in symmetrical body parts gives an objective indication that something is wrong. Therefore, thermography may be used in pain diagnosis. However, there was only limited research on CAD systems for point and inflammation diagnosis. Table 6 shows that the study on rheumatoid arthritis activity, by Frize and Ogungbemile, was the only work that used automated decision making with a classification algorithm [209]. Both sensitivity and specificity results, reported in this study, were very good. Therefore, more work is needed to support and to compete with this study, especially work with a strong methodology where the results are obtained by automated classification algorithms.

All the reviewed material focused on algorithms and systems building, thereby harvesting the advantages of modern computing machinery for medical diagnosis. However, such work can only establish maximum performance figures which are rarely achieved with deployed CAD systems [251]. In terms of systems design, theoretical and applied research can only provide evidence that a specific method, or a specific way of building a system, can serve in a practical problem solution which helps medical practitioners and patients [252], [253]. Once implemented, these practical CAD systems can only meet the promises, established through the theoretical research work, when they function according to specification [254]. Therefore, it is so important to design and develop these systems with a formal and model driven approach [255], [256].

In the review we focused on selected disease categories, thereby we neglected excellent work in related areas, such as surgery follow up [257] and dentistry [258], [259]. The work that was included in this study was reviewed with a special focus on feature extraction algorithms, statistical feature performance, classification and overall performance. This strict selection did not include important parameters, such as sample size and level of randomization. Hence, care should be taken when dealing with the reported performance values. In general, a larger sample size and more variety will lead to better or more conclusive results. Therefore, more work is needed in this area to improve the quality of thermography based CAD systems which, in the long run, re-establishes trust in this useful imaging modality.

Conclusion

Computer aided interpretation of thermogram images is of eminent importance, because the link between disease and body heat pattern is subtle and in many cases non-linear. Data mining and knowledge discovery algorithms help to improve thermogram based diagnosis in three key areas. The first of these areas is the imminent visual overload of screening professionals. Computer support reduces the diagnostic workload and the experts can focus on the hard cases and thereby increase the level of care. The second area is inter-observer variability. Relying entirely on the human brain, thermogram based diagnosis tends to be subjective and the quality varies widely. Objective methods, from the area of computer science and mathematics, can help to objectify the diagnosis and thereby reduce the inter-observer variability. The last area is concerned with the diagnosis quality. Human based diagnosis is largely based on both experience as well as mental state of the screening practitioner and to a lesser extent from equipment and diagnostic method. Hence, progress depends on the level of training and the experience in the field. For algorithm supported diagnosis progress is made in terms of hardware and software. Computing machinery will get more and more potent; this is independent of any individual application area. Progress in the software domain will come from integrating and to a lesser extent from inventing signal processing algorithms as well as managing an ever growing knowledge base. Therefore, we predict that one day CAD systems will outperform human practitioners in terms of accuracy, speed and cost.

This review describes some of the algorithms used in state of the art thermography based CAD systems. These algorithms come from the domains of pre-processing, feature extraction, statistical feature analysis, classification, and result assessment. Both statistical feature analysis and result assessment steps are very important, because the resulting performance figures foster competition between research groups and thereby drive thermography as a medical imaging modality forward. The idea of competition is most evident in areas where thermography based diagnosis is not the standard imaging modality, such as breast cancer diagnosis. The majority of researchers in this area published performance results in terms or sensitivity and specificity. The resulting competition has driven thermography based breast cancer systems to include the most advanced signal processing algorithms. Another area which uses performance measures extensively are CAD systems for influenza screening. The very act of publishing these measures builds up trust and thermography is widely seen as the most promising imaging modality for this application.

We adopt the position that CAD systems objectify thermography based diagnosis. The benefits of this objective approach are threefold. The reduction of inter-observer variability is an initial and very tangible benefit. The progress, which can be achieved with computer based systems, outpaces the progress of human observation. Finally, the performance of objective systems can be measured, which creates competition and this competition instills trust and fosters progress. This is especially important for thermography based diagnosis, because much trust was destroyed in the 1980s due to human error; lack of judgment and foremost the desire to cut corners for personal benefit. Since then, tremendous progress was made by applying scientific methods and sticking to scientific principles.

CAD systems are the future. We are living in exciting times where computers move from being a peripheral tool for administration and data storage towards the center of medical diagnosis and patient care. State of the art CAD systems mimic human diagnosis. This role model based development is important, because it guarantees a fast rate of progress during the initial phase of technology development. In layman’s terms: the apprentice mimics the master. However, there are distinct limitations of how much information even the most skilled human can process. These restrictions do not, or at least to a much lesser extent, apply to computing machinery. From this perspective, the direction for thermography diagnosis is clear: more data. This data can come from higher resolution IR scanning, but the main contribution will be existing datasets and case studies. For a human it is impossible to make sense of all available thermography images; however machines can process these images and extract relevant information. Staying with the idea of more or big data, there is no need to limit CAD systems to one data source alone. Computer based systems can make sense of data from various sources, such as thermography, X-ray, ultrasound, etc. At the end, the work is not about thermography or indeed any other imaging modality, it is about alleviating human suffering in patients and loved ones. This leaves no space for an unreasonable focus on just one imaging modality.

Conflicts of interest

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.