1.

Introduction

Typically, infrared (IR) cameras are composed of several optomechanical devices, electronic circuits, an IR detector, and optical lenses. Among these, the IR detector is one of the most important components in IR cameras. There are two principal methods for acquiring IR images by IR detectors: the scanned method and the focal plane array (FPA) method; in recent years, the FPA method has been more widely used. An FPA consists of a number of photodiode sets that convert thermal energy into electric signals inside the IR detector.^1,2 When an FPA is used in an IR detector, its output can have nonuniformity problems if the response characteristic of each photodiode is different. This causes fixed pattern noise (FPN) in IR images, degrading the image quality significantly.³

One of the most widely used methods for removong the nonuniformity in an FPA is a two-point nonuniformity correction (NUC), which is a representative algorithm of reference-based NUC (RBNUC). It creates two uniform reference sources by using a blackbody and calibrates the gain and offset parameters to have the same output for the same input for all photodiodes arranged in the FPA.⁴ However, assuming that the FPA output is completely corrected to have uniform output at some point, its residual nonuniformity (RNU) will become larger with the changes of the external environments of the FPA over time.⁵

Electronic boards and lens-driving motors mounted inside the IR camera constantly emit heat while the IR camera is in use. For IR cameras mounted on aircraft, external heat is constantly introduced through conduction via aerodynamic heating while flying. Since an IR camera responds to heat energy, heat generated inside the IR camera is expressed as a type of thermal noise. This internal thermal noise has the same effects as thermal stray light in an IR optical system. Although the cold shield at the front of the FPA blocks thermal stray light introduced from outside the field-of-view, the cold shield or optical path can be heated, and this will generate spatial nonuniformity in the IR images. Changes of the temperature inside the IR cameras can also change the output characteristics of the readout IC (readout integrated circuit).

As a solution for the above problem, scene-based nonuniformity correction (SBNUC),^6–11 which has the advantage of NUC without interruption of IR camera operation by utilizing characteristics of the image, has been studied. SBNUC updates the NUC parameters iteratively through computation by utilizing the image’s statistics, differences between currently inputted image frames, and the movements of the IR camera. However, existing SBNUC algorithms may not work in some applications such as IR camera-equipped aircraft at high altitudes of 45,000 feet or higher because it is very difficult to find any difference between image frames from the fixed IR camera with the small movements. In addition, since most targets displayed on the image frame are point targets, it is very difficult to perform an SBNUC with information derived from the shape and size of the targets.¹² Recently, Sozzi et al.¹³ analyzed and modeled spatial nonuniformity thoroughly by assuming that the optomechanical system and the detector effect can be separated. Although this method can mitigate the spatial nonuniformity caused by optomechanical devices, the performance of the NUC depends on the correction parameters calculated by optical analysis software (OAS). Unfortunately, in real situations, the correction parameters calculated by OAS did not sufficiently take into account the variation of detector output over time.

Thus, this study proposes an NUC method referred to as stray light compensated nonuniformity correction (SLCNUC) that can maintain uniformity of IR images despite the internal thermal stray light from the long operation of IR cameras in the above-mentioned operational environment. The proposed SLCNUC method derives compensation functions considering the thermal stray light effect between the internal temperature of the IR camera and the NUC parameters by using the least squares method.

2.

Influence of Internal Thermal Stray Light on Infrared Image Quality

The conventional observation model assumes that the output of each photodiode in the FPA is linear; this can be expressed as follows:

Fig. 1

Degradation of IR image quality due to thermal stray light inside the IR camera.

To verify thermal stray light caused by heating elements inside the IR camera, simulations were conducted using ASAP^™, an optical ray-tracing-analysis software, as shown in Fig. 2. Figure 2(a) shows the main parts of the IR camera that can generate internal thermal noise, such as motors and cooler. Figure 2(b) shows the ray tracing result of each thermal noise source. Here, the scattering characteristics of the optical equipment were modeled using the Lambertian model and the Harvey bidirectional scatter distribution function model;¹⁴ the results are shown in Table 1. The simulation showed that the maximum radiation intensity caused by the detector cooler, but arriving at the FPA, was $3.9\times {10}^{-2}$ times the emission intensity at the detector cooler, proving that the thermal noise emitted from the heating elements inside the IR camera affects the FPA.

Fig. 2

Analysis of thermal noise in an IR camera: (a) thermal noise sources inside the IR camera and (b) ray tracing results of the thermal noise sources.

Table 1

Ray-tracing simulation result of internal thermal noise.

To compare with the previous simulation result, an experiment was conducted using image frames from real IR cameras. We investigated the variation of spatial noise according to the changes of the internal temperature. The spatial noise was calculated using the standard deviation of the temporal averaged image between 300 consecutive nonuniformity-corrected image frames at every single internal temperature of the IR cameras. The results showed that the spatial noise increased linearly as a function of the internal temperature, and there were no statistically significant differences according to the detectors or optical flows. Figure 3(a) shows an example of the linear relationship between spatial noise and internal temperature variation due to the long IR camera operating time. It is observed that the spatial noise increases almost in proportion to the increase of the internal temperature, where the second-order or any higher-order of nonlinear terms can be considered as negligibly small in this increase. The increase of the spatial noise is caused by the deviation of gain and offset. In other words, as shown in Fig. 3(b), the slope of the detector output curve of the pixels affected by the internal thermal stray light is increased in proportion to the increase in the IR camera internal temperature from $T_1$ to $T_2$ and $T_3$. At the same time, since the slope of the NUC fitting line at each internal temperature changes according to the detector output curve, the gain and offset are also changed in proportion thereto. Therefore, the approximation of gain and offset with linear function of internal temperature can be regarded as the most general form for excluding high-order nonlinear term, since the spatial noise increases almost linearly with the internal temperature.

Fig. 3

Variation of spatial noise with the increase of the internal temperature of the IR camera: (a) experimental example of linear relationship between the spatial noise and the internal temperature and (b) slope changes of detector output curve and NUC fitting line with increasing internal temperature.

We can conclude that the internal thermal stray light changes the internal temperature of the IR camera, and this variation of internal temperature, in turn, causes variation in the spatial noise of the IR camera. In addition, the variation of gain and offset due to the variation of spatial noise can be approximated by linear functions of the internal temperature.

Considering both the stray-light-tracing simulation result and the results of the actual experiments in a real IR environment, the conventional observation model should be modified as follows:

Equation (2) can be rearranged and summarized as follows:

3.

Proposed Calibration Method

In this section, the proposed SLCNUC method is formulated by providing a calibration process based on the estimation of the true scene irradiance. The proposed method of true scene irradiation estimation is described in Sec. 3.1. The formulated SLCNUC method is described in Sec. 3.2.

3.1.

Estimation of True Scene Irradiance

As demonstrated in Eq. (3), the nonuniformity problem deepens over time because of the presence of the internal thermal stray light term ${\widehat{X}}_{i,j}(n)$, which changes according to the internal temperature of the IR camera. This is because the existing RBNUC algorithm creates gain and offset by estimation based on the assumption that only true scene irradiance is present.

To estimate the true scene irradiance in the existing RBNUC methods, statistical methods such as the median or mean were applied to observe the IR image data. However, since locally heated areas without uniform heating can be present in thermal stray light environments inside the IR camera, statistical values such as the median value cannot estimate the true scene value. Therefore, this study calculated the true scene irradiance of the internal thermal stray light environments based on the amount of electric charge collected in the capacitor under an IR detector standalone condition. Assuming that the expectation value of true scene irradiance is $Z$ in Eq. (4):

Eq. (4)

$$Z=g_{i,j}(T)Y_{i,j}+o_{i,j}(T).$$

In other words, the calculation of gain and offset in the SLCNUC method under the internal thermal stray light environment can be accomplished by estimating the true scene irradiance $Z$ according to the changes in target temperature and measuring the changes in observation value $Y$ according to the changes in IR camera internal temperatures.

Figure 4 shows a graph of the number of electrons collected in a capacitor as a function of the blackbody temperature detected by the IR detector. The experimental environment is as follows: the FPA well fill capacity is $6.5{\mathrm{Me}}^{-}$ and the integration time is 1 ms. The charge amount increased monotonically with the blackbody temperature. Equation (5) shows the estimation process of true scene irradiance based on the charge amount of the capacitor

Eq. (5)

$$Z=F+\left(\frac{C\times T_{\mathrm{int}}}{K}\right)K=\frac{\text{Total well fill capacitor}}{\mathrm{DR}}.$$

$F$ refers to the floor level value of the IR detector, i.e., the pixel value displayed even when the integration time is at its minimum value. $C$ refers to the amount of electron charge acquired during an integration time of 1 ms, $T_{\mathrm{int}}$ is the integration time, and DR is the dynamic range of the IR detector. In addition, $K$ refers to the true scene gain, i.e., the number of electrons required to raise a pixel by one gray level. The result of true scene irradiance estimation under the detector standalone condition using Eq. (5) is shown in Table 2. Here, 2 ms was used as the integration time $T_{\mathrm{int}}$, and the parameter values of 3726 and 21500 were assumed for $F$ and DR, respectively; these values had been supplied by the manufacturer.

Fig. 4

The number of electrons collected in a capacitor as a function of blackbody temperature.

Table 2

True scene irradiance estimation according to blackbody temperature.

Blackbody temperature (°C)	Charge (Me)	Estimated pixel value
20	1.261	6034
30	1.767	7707
40	2.435	9917
50	3.3	12,778
60	4.403	16,426
70	5.787	21,004

4.

Experiments for Performance Evaluation

To verify the performance of the proposed SLCNUC method, experiments performed with real IR images were conducted. The first experiment was a comparison of the spatial noise between the existing RBNUC and the proposed SLCNUC methods according to the IR camera’s internal temperature. The internal temperature was measured by using temperature sensors attached to the optomechanics housing.

When the IR camera’s internal temperature was set at 31°C36°C, and 45°C, and the temperature of the blackbody was set to 20°C for low temperature and 60°C for high temperature, a gain/offset table using 2-point NUC was obtained. Then, changes in spatial noise according to the IR camera’s internal temperature were observed. A Sebastian640 from the SCD Company was used as an IR detector, the wavelength was 3.6 to $4.9\mu \mathrm{m}$ for midwavelength infrared, and the resolution of the FPA was $640\times 512$. Furthermore, a heater was installed inside the IR camera to raise the internal temperature quickly.

The experimental result of Fig. 5 shows that an IR camera, to which an NUC table acquired when the IR camera internal temperature was 31°C was applied, had the lowest spatial noise at an internal temperature of 31°C, which was the same as the temperature at acquisition. However, as the internal temperature increased, spatial noise also increased rapidly. On the other hand, an IR camera with NUC parameters acquired when a camera internal temperature of 45°C was applied showed the lowest spatial noise at high temperatures. However, it showed the highest spatial noise when the internal camera temperature was low. Meanwhile, an IR camera to which the proposed SLCNUC method was applied showed that not only a low level of spatial noise can be maintained at both high and low camera internal temperatures but also that spatial noise caused by changes in internal temperatures increased linearly. This linear increase in spatial noise can be explained by an increase in the number of defective pixels or an increase in the dark current of the detector due to long operation hours of the IR camera, which can be reduced through additional IR image correction techniques, such as defective pixel replacement.

Fig. 5

Comparison of spatial noise according to IR camera internal temperature.

The second experiment was conducted to verify how pixel output can be changed according to the changes in the IR camera’s internal temperature while four arbitrary pixel coordinates were designated in the IR image frame. When the IR camera’s internal temperature was 31°C, gain and offset were acquired and used in NUC. The internal temperature was increased constantly using a heater mounted inside the IR camera. As shown in Fig. 6(a), the IR camera, with the existing RBNUC algorithm applied, had different slopes of pixel value increase according to the pixel location with the increase of the internal camera temperature. This was because the effects of thermal stray light inside the IR camera can be different according to FPA locations and the nonuniformity can deepen as the camera operation time becomes longer, with thermal stray light that has different effects at different locations. On the other hand, an IR camera with the proposed SLCNUC method applied had uniform pixel value rise slopes at all FPA locations despite the temperature changes inside the IR camera, as shown in Fig. 6(b). This result implies that even though the spatial noise increases as the internal temperature rises, the pixel values of all the coordinates are increased to the same level, so the uniformity of IR image can be maintained.

Fig. 6

Changes in pixel values at arbitrary points: (a) existing RBNUC method (b) SLBCNUC method.

The third experiment compared the performance by measuring peak signal-to-noise ratio (PSNR). To calculate the PSNR, the original image was defined as the one that is captured from the IR camera with an internal temperature of 46°C and corrected by an NUC table acquired at an internal temperature 46°C. Similarly, the other images were obtained by applying NUC tables acquired at different internal temperatures.

The experimental result showed that lower PSNR was measured when the internal temperature was cooled down from 46°C, as shown in Figs. 7(a)–7(c). Despite the internal temperature of the IR camera at 46°C, when correction parameters acquired at different internal temperatures were used, nonuniformity deepened. On the contrary, when an IR image was shot by applying the proposed SLCNUC method, PSNR was measured at 65.1 dB, which was the highest performance, as shown in Fig. 7(d). When the region marked with the red-colored square in Fig. 7 was compared, the detail of the IR image to which the proposed SLCNUC method was applied was the closest to the original image.

Fig. 7

Comparison of PSNR performance according to NUC parameter acquired internal temperature: (a) IR image using NUC parameter acquired when internal temperature was 40°C, (b) IR image using NUC parameter acquired when internal temperature was 35°C, (c) IR image using NUC parameter acquired when internal temperature was 30°C, and (d) IR image using the SLCNUC method.

5.

Conclusions

An NUC scheme named SLCNUC was proposed. Our work formulated the linear stray light compensation function to correct the local nonuniformity caused by internal thermal noise. The conventional NUC algorithms have difficulties in correcting the thermal stray light inside the IR camera because the thermal stray light is highly accumulated by means of the internal thermal noise through long operating hours and has complex effects that depend on the design of the optical system. In this study, the thermal stray light inside the IR camera was analyzed to prove that the internal thermal noise can affect the IR image quality, and a linear function was derived to compensate internal thermal noise iteratively. Experimental results on real IR images demonstrated that the proposed SLCNUC could maintain the lower level of spatial noise in spite of higher levels of thermal stray light and had better image quality with up to 18 dB PSNR improvement compared to that of the existing RBNUC method.