Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept.

BACKGROUND: Thermal assessment of the plantar surface of the foot using spot thermometers and thermal imaging has been proven effective in diabetic foot ulcer prevention. However, with traditional cameras this is limited to single spots or a two-dimensional (2D) view of the plantar side of foot, where only 50% of the ulcers occur. To improve ulcer detection, the view has to be extended beyond 2D. Our aim is to explore for proof of concept the combination of three-dimensional (3D) models with thermal imaging for inflammation detection in diabetic foot disease.
METHOD: From eight participants with a current diabetic foot ulcer we simultaneously acquired a 3D foot model and three thermal infrared images using a high-resolution medical 3D imaging system aligned with three smartphone-based thermal infrared cameras. Using spatial transformations, we aimed to map thermal images onto the 3D model, to create the 3D visualizations. Expert clinicians assessed these for quality and face validity as +, +/-, -.
RESULTS: We could replace the texture maps (color definitions) of the 3D model with the thermal infrared images and created the first-ever 3D thermographs of the diabetic foot. We then converted these models to 3D PDF-files compatible with the hospital IT environment. Face validity was assessed as + in six and +/- in two cases.
CONCLUSIONS: We have provided a proof of concept for the creation of clinically useful 3D thermal foot images to assess the diabetic foot skin temperature in 3D in a hospital IT environment. Future developments are expected to improve the image-processing techniques to result in easier, handheld applications and driving further research.

Ulceration and infection are frequently occurring foot complications in people with diabetes and peripheral neuropathy. These complications increase morbidity and mortality.[1,2] If not treated quickly, the consequences can be devastating. Therefore, early detection of diabetic foot complications is critical.[1] However, due to loss of sensation in the feet, people often fail to notice these ulcers, while self-examination is impeded by health impairments related to diabetes and other comorbidities, like bad eyesight, limited mobility, or social impairment.[3] Self-examination by means of measuring the foot skin temperature can make the difference, as this has proven efficacy in ulcer prevention.[3-11]

Initial studies on foot skin temperature as self-assessment tool used thermal spot thermometers.[8-10] Participants of the studies were requested to measure six predefined spots on each foot and calculate the difference between both feet for each spot. In case a temperature difference greater than 2.2°C (4°F) was found on two subsequent days, participants were advised to contact their health care professional. This approach proved effective in ulcer prevention.[8-10] However, with the spot thermometer it is possible that a hotspot not occurring at the six predefined positions is missed and the method lacks usability as it requires participants to calculate these differences themselves, and do so every day.

Follow-up studies utilized thermal infrared (IR) imaging. This technique is contactless and has the potential to improve usability, as it captures the entire plantar side of both feet.[3-7,11-15] Further, a computer or dedicated application can calculate thermal differences.[16] These initiatives have not yet resulted in home monitoring systems, as initial IR cameras were too expensive. With the commercialization of cheap smartphone-based IR cameras, costs have reduced dramatically and studies for home application are within reach.[17]

However, IR cameras still have a major shortcoming. When an image is taken, this image is two-dimensional (2D). For example, if the plantar side is imaged, ulcers developing on the dorsal side will be missed. Since only half of the ulcers occur on the plantar side, the other half would go unnoticed.[18] Alternatively, one might choose to make various photos, but automatic calculation of temperature differences has only been shown possible for the plantar side, because only from the plantar side can both similar aspects of both feet be captured in one image.[16] A multiple-photo approach would also require extra effort in assessing these photos, which may not be feasible in busy daily clinical practice or home monitoring.

Three-dimensional (3D) thermography, when fully developed, could be the solution to most of the mentioned shortcomings. With a 3D model, one can view the feet beyond the plantar site in only one image. In addition, the third dimension provides extra information that may facilitate clinical assessment, and can be used for automatic evaluation of contralateral temperature differences at the plantar, medial, dorsal and lateral foot side. Some 3D thermography approaches of the feet have been proposed, but never applied in a medical setting.[19-23] With the ultimate goal in mind to create a fully developed 3D thermography system, the aim of the current article is to explore for the first time ever how 3D thermographs would be of benefit to inflammation detection in diabetic feet as proof of concept. This is investigated with an experimental setup in which 3D imaging is added to already existing 2D thermal imaging modalities.

Methods

Study Design

In this single-centre prospective cross-sectional study, a convenience sample of eight consecutive participants with diabetes mellitus who visited the multidisciplinary outpatient diabetic foot centre of expertise of Ziekenhuisgroep (Hospital Group) Twente (Almelo, the Netherlands) was included. Inclusion criteria were a current diabetic foot ulcer and ability to comply with all study measurements. To investigate 3D temperature difference assessment in diabetic foot disease, participants with a measurable temperature difference were required; this is more likely when an ulcer is present, hence our decision for this inclusion criterion. The Medical Ethical Committee Twente approved the study protocol (K17-45), and informed consent was obtained from each subject prior to the start of the study.

Materials

The 3D thermography setup comprised a high-resolution medical 3D imaging system and three thermal IR cameras. The medical 3D imaging system (Vectra XT, Canfield Imaging Systems, Fairfield, NJ, USA; Figure 1) creates 3D models using a passive photogrammetry technique. This technique is comparable to how humans see 3D with two eyes, but instead of eyes, the system uses three pairs of high-resolution cameras. To extend the view around the body the system captures images from three viewing angles with three pairs of cameras. This creates a body image (ie, patient’s feet in this study) with a combined view of 270 degrees. The IR cameras (FLIR One for Android generation 2, FLIR Systems, Wilsonville, OR, USA; Figure 1) were smartphone-based IR and color cameras with thermal resolution 160 × 120 pixels, visual (color) resolution 640 × 480 pixels, operating temperature of 0 to 35°C, scene temperature range of −20 to 120°C, focus of 15 cm to infinite, angle of view of 46°x35°, and a male micro USB connector. For the study, the IR- cameras were connected to a laptop with Linux distribution (Dell Precision 3520 i7-7700hq), not a smartphone.

Figure 1.

Imaging setup. In the middle the Vectra XT, 3D imaging system can be seen (white). On the inner side of every viewpoint of the 3D imaging system, a FLIR One IR camera was attached (blue indicator), and connected to the laptop in front of the 3D imaging system. (A) Above each viewpoint the resulting 2D images are provided. On top, framed in red, the two images from the IR camera are shown (left: color image; right: thermal IR image in grayscale). (B) Bottom image, framed in black, is the texture map from the 3D image system.

To combine the thermal images with the 3D model, each thermal camera and 3D camera pair must approximately have the same point of view. In addition, bot images must be taken at approximately the same moment. We therefore mounted the three IR cameras to the 3D imaging system, at approximately the same point of view (Figure 1A). Using a custom Linux driver and Matlab (The MathWorks, Natick, MA, USA), we simultaneously acquired two images (1 IR and 1 color) from each IR camera.

Study Procedures

After providing informed consent during regular treatment, participants were included for 3D measurements preceding their next follow-up consultation at the outpatient clinic. Participants were laid down in supine position on a treatment bench with their lower legs supported by a pillow and their bare feet over the edge of the bench. Their feet remained exposed to the environment for 5 minutes to allow equilibration of foot temperature. An image was taken with the 3D imaging system simultaneously with the images from IR cameras.

Image Processing

The 3D-images were exported from the 3D imaging system in the Wavefront .obj file format accompanied with a Material Template Library file (.mtl) and a texture map (.bmp). A texture map is an image used to define the colors of the surface (Figure 1B). An .mtl file links the texture map to the 3D model. This provided the possibility to define the colors of the surface of the 3D model with the IR-data by replacing the texture maps with the IR images. Before the IR images could replace the texture maps, they had to be spatially transformed to the dimensions of the texture maps (Figure 2). For the transformation, Matlab and Photoshop CC 2015 (Adobe Systems, San Jose, CA, USA) were used.

Figure 2.

Graphic overview of the transformation process to replace the original texture map (A) with the IR image (B). (C) The feet were cut out of the images using Photoshop and the thermal images were roughly placed on the texture map. (D) The thermal images were transformed to cover the appropriate surface of the texture map and a color map was applied. The color map ranged from 15°C (dark blue) to 41°C (dark red).

For the spatial transformation, we used a point-based projective transformation. This type of transformation is, for instance used in aerial photo projections on roadmaps. It uses at least four corresponding points; in our application, this concerned four identifiable points on the boundaries of the foot visible in both the texture map and IR image; for example, this could concern the heel, hallux, medial to MTP 1, lateral to 5, and so on. This transformation works when all selected 3D object points are lying in a 2D plane. In the frontal images of the plantar foot view this condition is met. In the images from the medial and lateral side of the foot this was poorly met, because both feet were in different 2D planes, with one foot behind the other. Therefore, each foot was separately extracted from the thermal image using photoshop and subsequently transformed in Matlab. Furthermore, besides the projective transformation a fine registration was added based on deformable registration.[24] The results were combined in a new texture map.

Face Validity

The 3D thermographs were assessed for face validity. Face validity was defined as the combined technical and clinical assessment of the 3D IR visualizations. Each image was assessed for technical shortcomings by a qualified technical physician (RFMvD), and clinically by two experienced physician assistants (both >15 years of experience in diabetic foot care). Their combined findings were translated in a face validity score of + (directly applicable in clinic), +/- (applicable, but some shortcomings), ? (unable to provide an outcome), or – (image not applicable in clinic).

Results

Study Population

Characteristics of the eight participants included are shown in Table 1. All participants had peripheral neuropathy, three participants had minor amputations and the population was predominantly male, around 65 years of age. All participants, except one, had an existing DFU, most often (5 of 8) classified as University of Texas 1A. The ulcer in one participant, case 7, healed in the period between inclusion and measurements.

Table 1.

Participant Demographics.

#	Sex	Age	DM type	UT class	Nr. ulcers	Location	Side	Amputation
1	M	61	2	2A	2	Metatarsal head 5 & heel	L	Dig. 3+4 R
2	M	66	2	2B	1	Hallux	R	Dig. 2+3 L
3	F	57	2	1A	1	Midfoot	R	−
4	F	58	2	1A	1	Hallux	R	−
5	M	89	1	1A	2	Metatarsal head 5 & 4th toe	R	−
6	M	58	2	1A	1	Midfoot	L	Forefoot L
7	M	74	2	0A	0	Heel	L	−
8	M	61	2	1A	1	Hallux	L	−

3D Thermographs

As 3D thermograph results, the image data from eight participants were successfully processed. With 3D processing software (Meshlab, Visual Computing Lab, Pisa, Italy; and/or Daz3D, DAS Productions Inc, Draper, UT, USA) the 3D models were converted to universal 3D files (.U3D), which we then managed to integrate in a PDF file using Adobe Acrobat Pro (Adobe Inc, San Jose, CA, USA). This resulted in PDF files where the feet could be rotated in 3D and assessed visually for thermal anomalies: the result for each participant, in combination with the original 3D models, can be found in the online supplementary files 1 to 8, while Table 2 shows a static image for each participant. The PDF-files were compatible with the secured hospital IT system.

Table 2.

An Overview of All the Cases With Face Validity.

#	Findings	3D thermographs	Original 3D model
1	Technical assessments: Fault lines: Both heelsBackground projection: Mainly on the dorsal right footArtefacts: —Clinical interpretation:3D thermograph:Overall increased temperature left foot.Inflammation suspected:Fifth metatarsal head (MTP5) left, anteromedial heel left.Live assessment:Suspicion confirmed with present ulcer.Face validity assessment: +
2	Technical assessments:Fault lines: Both heelsBackground projection: Both heels, minor overall boundariesArtefacts: MTP4 due to reflection on the foot.Clinical interpretation:3D thermograph:Overall increased temperature left.Inflammation suspected:Hallux Left and MTP5 right.Live assessment:Hallux left confirmed with present ulcer. MTP5 right confirmed with an ulcer presign (blister).Face validity assessment: +
3	Technical assessments:Fault lines: Both heels, right foot plantarBackground projection: Overall boundaries right foot and above the ankle, due to patient stockings. Minor, dorsal side left foot.Artefacts: —Clinical interpretation:3D thermograph:Overall increased temperature right.Inflammation suspected:Central and medial midfoot, extending to forefootLive assessment:Ulcer present on medial midfoot, interdigital ulcer presign (blister) between hallux and dig2.Face validity assessment: +/-
4	Technical assessments:Fault lines: —Background projection: Minor, dorsal side of both feetArtefacts: —Clinical interpretation:3D thermograph:No evident temperature difference is notable.Inflammation suspected:Right foot at hallux and MTP1; Left foot MTP1; however no temperature difference, so no conclusive suspicion.Live assessment:Right an ulcer at the hallux was present. Left a presign (hematoma) at MTP 1.Face validity assessment: +
5	Technical assessments:Fault lines: Both heels, medially.Background projection: Left heel, minor overall boundaries.Artefacts: —Clinical interpretation:3D thermograph:No evident temperature difference.Inflammation suspected:Right MTP5 and surrounding toes.Live assessment:Confirmed ulcer lateral at MTP5 and dorsal on dig4. Callus found lateral at MTP5 on left foot.Face validity assessment: +
6	Technical assessments:Fault lines: Minor, on right heel.Background projection: Minor, dorsal side of both feet, right toes and around the ankle.Artefacts: —Clinical interpretation:3D thermograph:Overall increased temperature left foot.Inflammation suspected:Overall left footLive assessment:Amputation wound on top of the stump.Face validity assessment: +/-
7	Technical assessments:Fault lines: Minimally on both heels.Background projection: Minor dorsally at the boundaries, mainly above the ankles.Artefacts: —Clinical interpretation:3D thermograph:Left heel and midfoot increased temperature.Inflammation suspected: Left heelLive assessment:Recently healed ulcer on the left heel.Face validity assessment: +
8	Technical assessments:Fault lines: Both heelsBackground projection: Left heel, minor boundariesArtefacts: —Clinical interpretation:3D thermograph:Inflammation suspected: Left Hallux and right medial midfoot.Live assessment:Confirmed ulcer on left hallux. Right medial midfoot was assessed as an ulcer risk area.Face validity assessment: +

To support the clinical interpretation, static images are shown. See Figure 2 for the temperature scale. The corresponding rotatable 3D PDF thermographs can be found in the supplementary files 1 to 8. The 3D view can only be enabled in an Adobe PDF viewer after accepting to trust the document. As the 3D thermography system is still an experimental setup some technical errors are present and reviewed in the technical assessment. The fault lines, dark blue background projection, and additional artefacts are highlighted. First clinical assessment the interpretation of the 3D thermographs from experience clinicians are stated with suspected locations and compared with the live assessment.

Technical assessment showed that thermal texture maps were successfully created for all eight participants. However, some minor errors were also found (Table 2). To some extent, the occluding boundaries in the 3D thermography models had a dark blue color. This was an artefact where the background color was calculated as representing the foot. Furthermore, it can be seen in all 3D thermographs that three separate texture maps were used, as the transitions were still visible in a registration fault line. The dorsal side of the feet was not yet included, because only 270 degrees around the image could be captured with the 3D imaging system. For detailed assessment of each image, see further Table 2.

Findings from clinical assessment are summarized in Table 2. Technical and clinical assessment was combined in a face validity score. Face validity was assessed as + in 6 cases and +/- in 2 cases.

Discussion

We aimed to create 3D thermographs to explore how it could benefit inflammation detection in diabetic foot disease, and succeeded in creating the first-ever 3D diabetic foot thermographs with adequate face validity. We combined IR images with a 3D model for thermal assessment beyond the plantar side of the foot in people with diabetes with foot ulceration. Within this first set of 3D thermographs, ulcer-related temperature differences were clearly visible, and three ulcer presigns and two areas at-risk could also be identified. Furthermore, a more diffuse temperature increase was visible in 6 patients, implicating more severe inflammation or infection.[7] Finally, in one patient we could identify inflammation at the location of a recently healed ulcer. Taken together, this confirms the clinical usefulness of temperature assessment, and shows proof of the concept that the 3D thermographs can create the means to go beyond the plantar side in inflammation detection in diabetic foot disease.

Besides the clinical benefits of inflammation detection for ulcer prevention at the medial, lateral and dorsal side with the generated 3D view, the 3D thermographs also have some technical advantages over 2D images when working toward an automatic evaluation tool. For instance, when the background temperature matches the foot temperature, differentiation between foot and background is difficult with 2D thermal images.[16] Using 3D, feet are automatically differentiated from the background with the additional depth information. Furthermore, in 2D images the position of the camera with respect to the feet, in terms of distance and viewing angle, is very important for temperature assessment, whereas this has less effect in 3D.[16]

The eight cases investigated generally showed accurate 3D thermographs. However, some image registration errors with the thermal texture maps still occurred. These errors were visible as dark blue background on the boundaries and fault lines, for instance visible around the heel. These registration errors resulted from substantial differences in extrinsic camera parameters (FOV, resolution, etc) between the 3D imaging system and the IR camera used in this study. Furthermore, we were limited by not being able to perfectly align physically the IR cameras with the 3D imaging system, as that would block the view in our setup. We solved this by transforming the IR image to the texture maps, but this could not rule out some minor errors. By transforming the IR image its pixels values, and with it their temperature values, are interpolated to cover the whole surface. No clinical relevant temperature deviation is expected from this process. Future integration of thermal cameras within 3D systems will overcome these registration problems, a technical solution that will be available for medical applications in the near future. Another limitation to be acknowledged is that we only included participants with an actual foot ulcer. As described in the methods, this was chosen to maximize temperature differences. It is important to note that a system for ulcer detection will have to be validated in patients at high-risk of ulceration. However, with the adequate temperature difference detection in the current patients, both for ulcers as well as for presigns, we hypothesize that detecting differences >2.2C will be feasible.

For this study, we used a medical grade high-resolution 3D imaging system to get high-end, medically validated 3D models of the feet. The benefit was that the images were of high quality and no movement errors occurred, because the images were created with just one shot. However, the 3D imaging system was limited to a 270 degrees view around the feet, which meant that the dorsal side of the feet was still missing. Furthermore, the 3D system is too big and expensive to be already implemented as home monitoring application. As this study was set up to show a proof of concept of 3D thermography, future research can now focus on technical improvements, to reduce the size of the system needed to acquire 3D thermographs of the foot, and to include the dorsal side.

The 3D thermographs show potential for DFU assessment. The smartphone-based thermal cameras are reasonably priced and have been proven valid in comparison with high-end thermal cameras.[17] Although the thermal camera used was small and affordable, our 3D camera system was big and expensive. For an affordable and handheld 3D imaging device, our suggestion would be the use of a depth sensor camera.[25] This type of camera is already used to make 3D foot models for orthopedic purposes, yet that is without integration of thermal cameras. Fusing a depth sensor camera with a thermal camera is described for industrial purposes,[25] and this is a promising road for future medical applications. Furthermore, depth sensor cameras are now being integrated in high-end smartphones, and one can expect them to become available in a broader range of future smartphones. On the long term, an IR camera add-on might already suffice for a handheld 3D thermography device.

Another area for future research is the development of automatic evaluation, to reduce the assessment time for users and clinicians and improve usability and implementation. In our opinion, the best approach for an automatic evaluation algorithm is to determine the corresponding 3D location between the feet with a mirroring and active shape analysis method. In such a method, each foot is fitted to a statistical shape model. The result consists of two feet models in which each indexed 3D point corresponds with same location on both feet. The difference between the average temperatures around these corresponding 3D points (vertex) with a radius of maximal 0.5cm would be an ideal indicator for a potential DFU, similar to the current pathological threshold used for spot thermometry at the foot skin.[8-10]

Conclusion

We have provided a proof of concept for the creation of clinically useful 3D thermographs for inflammation detection in diabetic foot disease in a hospital IT environment. This opens the door for future developments, to improve the image-processing techniques toward easier, handheld applications, thereby driving further research and clinical implementation.

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case1 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case2 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case3 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case4 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case5 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case6 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case7 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology

Click here for additional data file.

Supplemental material, vanDoremalen_et_al._Case8 for Infrared 3D Thermography for Inflammation Detection in Diabetic Foot Disease: A Proof of Concept by Rob F. M. van Doremalen, Jaap J. van Netten, Jeff G. van Baal, Miriam M. R. Vollenbroek-Hutten and Ferdinand van der Heijden in Journal of Diabetes Science and Technology