Remote sensing of multiple vital signs using a CMOS camera-equipped infrared thermography system and its clinical application in rapidly screening patients with suspected infectious diseases.

BACKGROUND: Infrared thermography (IRT) is used to screen febrile passengers at international airports, but it suffers from low sensitivity. This study explored the application of a combined visible and thermal image processing approach that uses a CMOS camera equipped with IRT to remotely sense multiple vital signs and screen patients with suspected infectious diseases.
METHODS: An IRT system that produced visible and thermal images was used for image acquisition. The subjects' respiration rates were measured by monitoring temperature changes around the nasal areas on thermal images; facial skin temperatures were measured simultaneously. Facial blood circulation causes tiny color changes in visible facial images that enable the determination of the heart rate. A logistic regression discriminant function predicted the likelihood of infection within 10s, based on the measured vital signs. Sixteen patients with an influenza-like illness and 22 control subjects participated in a clinical test at a clinic in Fukushima, Japan.
RESULTS: The vital-sign-based IRT screening system had a sensitivity of 87.5% and a negative predictive value of 91.7%; these values are higher than those of conventional fever-based screening approaches.
CONCLUSIONS: Multiple vital-sign-based screening efficiently detected patients with suspected infectious diseases. It offers a promising alternative to conventional fever-based screening.

Introduction

Since the outbreak of severe acute respiratory syndrome (SARS) in 2003, infrared thermography (IRT) systems have been used as border-control devices at most major international airports to screen passengers for fever. IRT remains the gold standard for border control, because it can rapidly mass-screen infected individuals without contact.1, 2, 3, 4 However, IRT measurements are influenced by several factors, including the environmental temperature and humidity, alcohol consumption, and the consumption of antipyretic medications. Body temperature, in particular, can be modified rapidly by the consumption of antipyretic drugs, which directly affects the sensitivity of IRT. Hence, fever-based screening using IRT suffers from low sensitivity. This paper describes a combined visible and thermal image processing approach that uses a complementary metal oxide semiconductor (CMOS) camera-equipped IRT system that may address this issue. These systems have already been installed at most major international airports, and they can remotely sense several vital signs, including body temperature and heart and respiration rates, thereby facilitating the rapid and accurate screening of people who are suspected of carrying infectious diseases.

The concept underlying vital-sign-based screening is based on the association between infections and inflammation. Inflammation causes elevations in body temperature and in heart and respiration rates; therefore, integrating vital sign monitoring increases screening accuracy. This concept was used in previous studies by the present author group to develop a novel infection screening radar system to mass-screen individuals. This system utilizes a multisensor fusion technique to remotely measure heart and respiration rates using a microwave radar, and the facial skin temperature is measured using IRT. The results from case–control studies that investigated seasonal influenza screening showed a detection accuracy that ranged from 81.5% to 98.0% using the heart and respiration rates and the facial skin temperature, which is higher than the detection accuracies of the conventional fever-based screening methods.6, 7, 8, 9

However, the radar system used to screen for infections incorporates expensive embedded multisensor modules, namely a microwave radar, a reflective photoplethysmography sensor, and IRT, and it requires large-scale systems. Consequently, the system is not used widely. Hence, to promote the widespread use of vital-sign-based screening, focus has been placed on systems with minimum hardware requirements to achieve a system that is more suitable for real-world settings. The most reliable solution is to enhance the functionality of the conventional IRT systems that are already installed at international airports. By incorporating the latest advances in image processing techniques, these IRT systems can acquire thermal and visible images together by integrating visible and thermal cameras.

In this study, high image and temperature resolution IRT that combines visible and thermal images was used to acquire multiple vital sign measurements from facial images using remote sensing. The benefit of this approach is that it only requires a CMOS camera that is equipped with IRT rather than a large-scale system. Technical details of the system and the evaluation of its laboratory-based performance have been described in a previous publication. Respiration rates are measured by monitoring the temperature changes around the nasal area that are associated with inspiration and expiration; the facial skin temperature can be determined easily from the thermal images simultaneously. The circulation of blood in the face causes tiny color changes that provide a visible facial image that can be used to determine the heart rate. A multiple logistic regression function is incorporated into the system to predict the possibility of infection; hence, IRT can automatically detect infected individuals based on their vital signs, which are measured in real time. This system was tested on patients with an influenza-like illness in a clinical setting to evaluate the performance of this vital-sign-based screening approach using IRT alone.

Methods and patients

Visible and thermal image processing method to remotely sense multiple vital signs

This method has been described in detail from an image-processing perspective in a previous publication.  A CMOS camera-equipped IRT system (TVS-500; NEC/AVIO Infrared Technologies Co. Ltd, Tokyo, Japan) was used; this is the same system that is used in the quarantine station at Narita International Airport in Japan.5 The IRT system integrates a CMOS camera with thermography to capture visible and thermal images, respectively (Figure 1 ).

Figure 1

Schematic representation of the visible and thermal image processing method that remotely senses multiple vital signs and the multiple logistic regression function that predicts the possibility of infection.

The visible and thermal images were recorded at a speed of 30 frames per second and at a resolution of 640 × 480 pixels. The circulation of blood in the face causes tiny color changes on facial images that are undetectable with the human eye. The CMOS camera uses this information to determine the heart rate. To measure the respiration rate from the thermal images, the temperature changes that occur around the nasal area during the inspiration of cold air from the environment and the expiration of warm air from the lungs was monitored, and the respiratory waveform was extracted from the differences in each of the thermal images. This enabled the respiration rate to be determined from the breath-to-breath intervals. The facial skin temperature was measured simultaneously using the thermal images. The image acquisition and processing programs were written in LabVIEW software (National Instruments, Texas, USA). Subjects were seated in front of the IRT system at a distance of approximately 0.5 m. The IRT system displays the ‘Infection’ or ‘Healthy’ result within 10 s using the logistic regression discriminant function, which bases the output on the heart rate, respiration rate, and facial skin temperature.

Clinical tests on patients with an influenza-like illness

This was a cross-sectional investigation that was undertaken at the Takasaka Clinic in Fukushima in Japan. The study involved 16 outpatients (11 male and five female) who visited the Takasaka Clinic with an influenza-like illness that included fever, headache, and sore throat, between January 22, 2015, and February 25, 2015. The ambient temperature (around 23.0 °C) was also monitored to ensure reproducible environmental conditions. The average axillary temperature of the patient group was 37.2 °C (range <36.2– < 39.0 °C), and their average age was 36 years. The 22 healthy control subjects (10 male and 12 female) were students and admissions staff from Tokyo Metropolitan University, Japan, none of whom had a fever, headache, or sore throat. The average axillary temperature of the control group was 36.3 °C (range <35.4– < 37.0 °C), and their average age was 35 years. In this study, abnormal vital signs were defined according to the diagnostic criteria for systemic inflammatory response syndrome (SIRS), i.e., (1) body temperature >38 °C or <36 °C, (2) heart rate >90 bpm, and (3) respiration rate >20 breaths/min. This study was approved by the Faculty of System Design Committee on Human Research at Tokyo Metropolitan University.

Logistic regression discriminant analysis to predict the possibility of infection based on the vital signs measured

To distinguish between patients with infectious influenza and healthy control subjects, logistic regression discriminant analysis was used to establish a classification model based on the three derived vital signs. Multivariable logistic regression analysis is a well-established statistical method that is used to analyze dichotomous outcomes in clinical practice; it is flexible and robust, and enables meaningful data interpretations. Moreover, logistic regression analysis is much easier to implement in a real-time classification system, and saves computation time, compared with other classification methods such as neural network computation. The logistic regression discriminant function was defined as:where p is the probability of the outcome of infection, β 0 is a constant, β 1, β 2, and β 3 are the regression coefficients corresponding to the respiration rate, heart rate, and facial skin temperature, respectively, and x 1, x 2, and x 3 are the three vital sign variables of the respiration rate, heart rate, and facial skin temperature, respectively.

Statistical analysis

The results from the logistic regression classification model were used to calculate the sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV) using a 2 × 2 contingency table. To avoid overfitting, a leave-one-out cross-validation was performed. The mean and standard deviation (SD) values of the three vital signs were calculated. The differences between the influenza patients and the healthy control subjects with respect to the three vital signs were evaluated using the Mann–Whitney U-test. A p-value of <0.05 was considered to indicate statistical significance.

Results

The classification model was established using the data that described the three vital signs from the 16 influenza virus-infected patients and the 22 healthy control subjects using multivariable logistic regression. The statistically significant model is shown below:where x 1 is the respiration rate, x 2 is the heart rate, and x 3 is the facial skin temperature. The derived logistic function, Z(x 1,x 2,x 3), was statistically significant (p <  0.05). The Z(x 1,x 2,x 3) value could be used to differentiate patients with influenza (Z ≥ 0) from healthy subjects (Z < 0). Figure 2 illustrates the discrimination results that were obtained by plotting the Z(x 1,x 2,x 3) values against the axillary temperatures of the two groups. Of the patients with influenza, 14 (red dots) are enclosed within the red ellipse and they had positive Z-values, and two patients had negative Z-values (Figure 2). The 22 healthy control subjects (blue dots) enclosed within the blue ellipse had negative Z-values and none of the healthy subjects had a positive Z-value (Figure 2). Therefore, the sensitivity, specificity, NPV, and PPV were 87.5%, 100%, 91.7%, and 100%, respectively. The fever-based screening, for which the cut-off value for the axillary temperature was set at 37.0 °C, did not detect five influenza patients (false-negative). The sensitivity of the fever-based screening was 68.7%.

Figure 2

Discrimination results represented by plotting the Z(x1, x2, x3) values against the axillary temperatures of the two groups.

Table 1 presents a more detailed comparison of the patients with influenza and the healthy control subjects. The influenza patients who had higher Z(x 1,x 2,x 3) values had more severe symptoms, namely higher body temperatures and more elevated heart and respiration rates. The classification model determined that some patients had influenza, even if they did not have a fever. The Z(x 1,x 2,x 3) values could be used to evaluate the severity of infections, and they could, therefore, support the clinical risk stratification of patients. The two influenza patients who were misclassified had negative Z-values because their vital signs were normal.

Table 1

Comparisons between the patients with influenza and the healthy control subjects.

Status	Z(x_1,x_2,x_3) value(Z ≥ 0)	x_2 Respiration rate (bpm)	x_2Heart rate (bpm)	x_3Facial skin temperature (°C)
Influenza	19.0	31	95	36.9
Influenza	18.9	24	102	37.1
Influenza	15.7	20	108	36.4
Influenza	14.1	22	91	37.1
Influenza	12.1	19	83	37.6
Influenza	10.5	21	96	36.1
Influenza	10.1	12	103	36.4
Influenza	7.6	22	86	36.1
Influenza	6.9	12	80	37.5
Influenza	6.1	14	89	36.4
Influenza	5.3	22	80	36.1
Influenza	4.8	15	84	36.4
Influenza	2.8	20	87	35.2
Influenza	1.6	25	66	36.1
Healthy	−0.4	18	72	35.9
Healthy	−0.4	18	72	35.9
Influenza	−0.4(False-negative)	18	68	36.2
Healthy	−0.9	18	80	35.2
Healthy	−1.7	18	67	36
Healthy	−2.3	31	67	34.5
Influenza	−2.7(False-negative)	20	72	35.2
Healthy	−2.7	20	65	35.7
Healthy	−2.7	28	64	35
Healthy	−2.8	24	66	35.2
Healthy	−3.8	15	67	36.7
Healthy	−4.7	22	57	35.7
Healthy	−4.9	22	63	35.7

The mean (SD) facial skin temperature of the influenza patients (36.4 (0.7) °C) was 1.1 °C higher than that of the healthy control subjects (35.3 (0.6) °C). The mean (SD) heart rate of the influenza patients (86.7 (12.1) bpm) was 21.3 bpm faster than that of the healthy control subjects (65.5 (6.8) bpm). The respiration rate did not differ significantly between the influenza patients (19.8 (5) breaths/min) and the healthy control subjects (19.5 (5) breaths/min) (Figure 3 ).

Figure 3

Mean (standard deviation) values were calculated for the three vital signs. The differences between the influenza patients and the healthy control subjects with respect to heart rate, respiration rate, and facial skin temperature were assessed. NS, not significant.

Discussion

An integrated visible and thermal image processing approach is proposed for the remote monitoring of multiple vital signs using IRT, thereby enabling the rapid screening of infection in places of mass gathering. The results of this study demonstrate that the effectiveness of IRT for the screening of infection can be greatly enhanced by measuring body temperature, as well as heart and respiration rates, using IRT without any additional sensors. The high level of accuracy of the automated IRT system has a number of clinical implications that could enable the system to be used to provide primary screening of people who may be carrying infections within emergency outpatient units or quarantine stations. Moreover, this system saves time, because considerable amounts of time are required to investigate false-positive subjects when systems have low sensitivity levels and NPVs.

This technology also opens up new opportunities for controlling the spread of infections. For example, the present study was conducted at the Takasaka Clinic in Fukushima prefecture, which is one of the three prefectures that were most affected by the 2011 earthquake and tsunami in Japan. The risk of contracting infectious diseases, particularly influenza, increased after the earthquake and tsunami in Fukushima, and healthcare professionals and medical facilities were severely affected by the disaster.15, 16 In such settings, an automated IRT system could distinguish between individuals who are and are not carrying infections, thereby alleviating the workload of healthcare professionals. Therefore, the proposed integrated visible and thermal image processing approach may be a promising pre-examination technique in disaster settings.

Limitations of the present study mostly pertain to data samples (16 patients and 22 healthy control subjects), which can be considered sufficient for evaluation by the CMOS camera-equipped IRT system for vital-sign measurement. However, the data samples are small for training in a logistic regression classification model. To refine the performance of the logistic regression classification model implemented in the IRT system, field testing with larger and completely random subject populations will be conducted in real-world settings. Moreover, to guarantee the accuracy of heart and respiration rate measurement by the IRT system, the authors are now working on the development of an automatic real-time human face tracking algorithm using visible and thermal images. Detecting and tracking human faces can significantly reduce motion artifacts, thereby extracting stable heartbeat and respiration signals. The face tracking algorithm can also be expanded to multi-person tracking, i.e., more than two human faces can be monitored simultaneously to avoid a ‘human traffic jam’ in places of mass gathering such as airports.

In summary, the feasibility of using IRT to remotely sense multiple vital signs and to rapidly and accurately screen patients who are suspected of carrying infectious diseases has been demonstrated, and it appears that this is a very promising approach that will provide an alternative to conventional fever-based screening.

Conflict of interest

The authors state that they have no conflicts of interest to declare.