1.

INTRODUCTION

Optical surface temperature measurement is widely used, e.g., in non-destructive testing,¹ food industry,² condition monitoring,³ biological observation,⁴ and medical applications.⁵ Anomalous temperature distributions can indicate diseases like breast cancer⁶ or diabetic foot complications.⁷ Two-dimensional (2D) passive infrared thermography⁸ is an appropriate technique for measuring non-invasively, non-destructively, economically, and in real time.

However, 2D imaging techniques do not retain spatial information. By combining three-dimensional (3D) imaging systems based on structured light with thermal 2D imaging cameras, Colantonio et al.⁹ and Grubišić et al.¹⁰ obtained 3D thermograms of static objects. In order to measure dynamic scenes, real-time 3D thermography has been developed.^11–13 Frame rates up to 30 fps were achieved.

In recent years, our research group presented high-speed active 3D sensors based on array¹⁴ and GOBO (GOes Before Optics)^{15, 16} projection techniques imaging aperiodic sinusoidal patterns onto the measurement object.^{14, 17, 18} With these improvements, Heist et al.¹⁵ were able to record over 1,300 independent 3D point clouds per second. By integrating a high-speed long-wave infrared (LWIR) camera into one of our existing setups, we are now able to obtain 3D thermograms with a new multimodal five-dimensional (5D) sensor (three spatial coordinates, temperature, and time) at frame rates of up to 1 kHz at full resolution.

2.

MEASUREMENT SYSTEM

Our multimodal 5D sensor consists of two measurement principles: an active stereo vision system^19–23 for the 3D measurement and a passive infrared camera. We map the measured surface temperature onto the reconstructed 3D point cloud. The schematic setup and a photograph of our multimodal system are shown in Fig. 1.

Figure 1.

Multimodal 5D system: (a) schematic setup and (b) photograph.

For the 3D reconstruction, corresponding pixels P₁ and P₂ have to be identified in both cameras working in the visible range of light (VIS). As the object surface might appear homogeneous in VIS, we encode it by a sequence of gray values. Therefore, we use a sequence of length N of rapidly changing aperiodic sinusoidal fringe patterns projected by a high-speed VIS GOBO projector.¹⁵ The maximum of the normalized cross correlation function^{14, 24, 25} between the N temporal gray values of each pixel pair determines the corresponding points P₁ and P₂. By triangulation and taking into account the internal and external camera parameters, the 3D coordinates of the object point P are calculated from the coordinates of the pixels P₁ and P₂.

At the same time, the thermal camera records the spatial distribution of emitted radiation of the object surface. The spectral radiance depends on the surface temperature. The largest amount of radiation at room temperature is emitted at wavelengths around 10 µm. The temperatures are recorded with an LWIR camera (see Fig. 1) synchronously to the 3D data and mapped onto the previously reconstructed 3D point cloud.

Figure 2 shows a block diagram depicting the evaluation process from the image acquisition to the resulting 3D thermogram. First, we reconstruct the object 3D shape by triangulation of corresponding pixels. This 3D result is reprojected onto the thermal image such that we can assign a temperature value to each 3D point by bilinear interpolation.

Figure 2.

Block diagram depicting the evaluation process from image acquisition to resulting 3D thermogram.

The high-speed VIS cameras applied in our sensor are Photron FASTCAM SA-X2 monochrome cameras. Their maximum frame rate at full resolution of 1 Mpx is 12,500 fps. The light source of the self-built high-speed VIS projector is a Philips MSR metal-halide gas discharge lamp with about 250W optical output power. As LWIR camera, we apply a FLIR X6900sc SLS camera. In the spectral range from 7.5 to 12 µm, this camera can record up to 1,004 Hz at full resolution of 640×512 px. The integration time in the temperature range from –20 to 150 °C is very low with about 0.16 ms. The noise expressed in noise equivalent temperature difference is less than 40 mK.

3.

SENSOR CALIBRATION

The sensor calibration is intended to determine the intrinsic and extrinsic parameters of the two high-speed VIS cameras and the high-speed LWIR camera. This is required for correct 3D reconstruction and accurate mapping of the texture. In our calibration process, we follow the well-established method of bundle adjustment.^{21, 23} The camera parameters and the 3D coordinates are optimized such that the rear projection error between the projected and the measured point in the image plane is minimized.

For calibration of cameras of very different wavelength ranges, Yang et al.²⁶ split the calibration process in two sub-calibrations with two different calibration boards. In contrast to that, we developed a calibration tool with features detectable in VIS and LWIR to calibrate the three cameras in only one measurement. Our calibration tool shown in Fig. 3(a) is a special printed circuit board with a regular grid of circular rings and filled circles.

Figure 3.

(a) Photograph of calibration board, (b) VIS camera image, (c) LWIR camera image, and (d) evaluated LWIR image with detected circles and grid coordinate system.

As our calibration board has a homogeneous temperature distribution, we use the emissivity difference of the features to achieve a sufficient contrast in the LWIR images. The circles are made of electroless nickel gold (ENIG) with a very low emissivity. The plate material is a composite of epoxy resin and glass fibre fabric (FR-4) with an emissivity close to 1. Figure 3(c) shows an example LWIR camera image. The plate material appears bright, while the circles appear dark. Due to different reflective characteristics in VIS, the circles, which appear bright on a dark plate, can be detected as well (see Fig. 3(b)).

For the calibration measurement, we place our tool at various positions and orientations in the measurement volume. The evaluation starts with the detection and assignment of the circles to the grid coordinate system. The circles’ centers are determined by fitting an ellipse to the circumference of each circle. In the bundle adjustment, we consider the stereo conditions of the system and the image pixel and grid coordinates of the circles.

4.

MEASUREMENT EXAMPLES

In this section, we present example 3D thermograms of three measurements: a crushing metal tube, a basketball player, and an inflating airbag. Table 1 gives an overview of the measurement parameters for the three experiments.

Table 1.

Measurement parameters of three performed experiments

4.1

Metal tube

We measured a deformation process of a 0.5m long and 125mm diameter metal tube when hitting it with a sledge hammer. The measurement results as 3D thermograms at different times are shown in Fig. 4. The tube had a temperature of about 26 °C before the deformation process. Due to the sledge hammer impact, the tube deformed over its entire length. In the area with the highest wall bending, heat generation can be observed. This temperature increase of about 2K could be measured already at t = 50 ms. The sledge hammer impact was roughly in the tube center. However, in this direct impact area, the temperature did not change noticeably.

Figure 4.

3D thermograms of a metal tube deformation process when hitting it with a sledge hammer acquired at different times.

4.2

Basketball player

In order to measure a dribbling basketball player, we increased the size of the measurement field by adjusting the multimodal sensor. The measurement system had to be calibrated again. For three different times during the dribbling process, the measurement results are shown in Fig. 5.

Figure 5.

3D thermograms of a basketball player acquired at t = 20 ms, 258 ms, and 320 ms.

At the beginning, the basketball player controlled the ball with his hand (see Fig. 5 at t = 20 ms). We see that temperatures higher than 30 °C are mapped completely onto the hand whereas lower temperatures are mapped onto the ball surface. Thus, the mapping of the thermal data onto the 3D data works well. In the further dribbling process, the athlete moved his hand downwards and the ball separated from the hand. On the ball surface (temperatures of about 27 °C), a heat imprint of the hand (temperatures of about 31 °C) is clearly visible at t = 258 ms. The ball hit the ground and bounced off from it. At t = 320 ms, an area of increased temperature can be seen on the spot of the floor, where the ball touched the ground.

4.3

Inflating airbag

The third measurement we present in this paper is probably the most impressive one. We measured the inflation process of a front airbag. Figure 6 shows the scene at six different times. At t = 0 ms, the airbag ignition was started. The temperature of the steering wheel was about 25 °C. At t = 6 ms, an increased temperature could be observed under the cover which is pushed upwards. Until t = 20 ms, the airbag volume grew quickly. The airbag surface showed temperatures of about 30 °C. In a small area in the upper right, temperatures close to 40 °C were reached. Then, after about 50 ms, the deflation of the airbag started. As the heat diffused through the bag material, the temperature of the outer surface increased continuously until the end of the measurement. Temperatures up to 60 °C were reached. Lower temperatures make areas with thicker material, e.g., seams, visible. In general, the temperature distribution can indicate material flaws.

Figure 6.

Airbag inflation process in 3D thermograms acquired at different times.

5.

CONCLUSION

In this contribution, we presented our new multimodal 5D system for the simultaneous high-speed measurement of 3D shape and temperature. For this purpose, we further developed our high-speed 3D system by integrating a high-speed LWIR camera. The 3D data is recorded by active stereo vision whereas the 2D data is obtained by a passive thermal camera. Geometric calibration of the three cameras is carried out by using a specially designed calibration tool and bundle adjustment. This enables us to map the temperatures as texture onto the 3D data. With this system we are able to measure the three spatial dimensions and the surface temperature of rapidly changing objects.

At full resolution of 1 Mpx, the VIS cameras can record up to 12,500 frames per second. The LWIR camera has a maximum resolution of 640×512 px and reaches frame rates up to 1,000 Hz. In total, we achieve independent 3D thermograms with frame rates up to 1 kHz.

Three results of three different examples were presented, which demonstrate the broad application range of the multimodal sensor. The combination of high-speed 2D thermography with high-speed 3D sensors delivers much more information than traditional 2D thermography.