The prime limitation of optical sensors is the need for external sources of illumination while capturing the scene. This prevents them from recognizing objects in extreme conditions, such as insufficient illumination or severe weather (e.g., under fog or smoke). The thermal imaging sensors have been introduced to circumvent this deficiency, which acquires the image based on thermal radiation emitted by the objects. The technological advancement in thermal imaging enables the visualization of objects beyond the visible range that promotes its use in many principal applications, such as military, medical, agriculture, etc. However, hardware point of view, the cost of a thermal camera is prohibitively higher than that of an equivalent optical sensor. This led to employ software-driven approaches called super-resolution (SR) to enhance the resolution of given thermal images. We propose a deep neural network architecture referred to as “ThermISRnet” as the extension of our earlier winner architecture in the Perception Beyond the Visible Spectrum (PBVS) thermal SR challenge. We use a progressive upscaling strategy with asymmetrical residual learning in the network, which is computationally efficient for different upscaling factors such as ×2, ×3, and ×4. The proposed architecture consists of different modules for low- and high-frequency feature extraction along with upsampling blocks. The effectiveness of the proposed architecture in ThermISRnet is verified by evaluating it with different datasets. The obtained results indicate superior performance as compared to other state-of-the-art SR methods.
We address an approach called “DepthFuseNet” for the fusion of thermal and visible images using convolutional neural networks (CNN). The thermal image acquires radiating energy of the sensed objects and hence it can easily distinguish the objects from its background. However, the visible image (i.e., the image acquired within the range of visible wavelength of electromagnetic spectrum) provides a more visual context of the objects with high spatial resolution. Due to this complement nature of thermal and visible images, it is always an interest of the community to combine those two images to obtain more meaningful information from the individual source images. In DepthFuseNet method, features are extracted from given source images using CNN architecture, and they are integrated using the different fusion strategies. The auto-weighted sum fusion strategy performs better than that obtained using the other existing methods. To reduce the complexity of the architecture, we use depthwise convolution in the network. The experimental evaluation demonstrates that the proposed method exhibits salient features from the source images, and hence it performs better than the other state-of-the-art fusion methods in terms of qualitative and quantitative assessments.
Single-image super-resolution (SISR) refers to reconstructing a high-resolution image from given low-resolution observation. Recently, convolutional neural network (CNN)-based SISR methods have achieved remarkable results in terms of peak-signal-to-noise ratio and structural similarity measures. These models use pixel-wise loss functions to optimize their models, which results in blurry images. However, the generative adversarial network (GAN) has the ability to generate visually plausible solutions. The different GAN-based SISR methods obtain perceptually better SR results when compared to that with the existing CNN-based methods. However, the existing GAN-based SISR methods need a large number of training parameters in the architecture to obtain better SR performance, which makes them unsuitable for many real-world applications. We propose a computationally efficient enhanced progressive approach for SISR task using GAN, which we referred as E-ProSRGAN. In the proposed method, we introduce a novel design of residual block called enhanced parallel densely connected residual network, which helps to obtain better SR performance with less number of training parameters. The quantitative performance of the proposed E-ProSRNet (i.e., generator network of E-ProSRGAN) model is better for higher upscaling factors ×3 and ×4 for most of datasets when compared to the same obtained using different CNN-based methods whose trainable parameters are less than 7 M. In the case of upscaling factor ×2, E-ProSRNet obtains second highest structural similarity index measure values for Set5 and BSD100 datasets. The proposed E-ProSRGAN model generates SR samples with better high-frequency details and perception measures than that of the other existing GAN-based SISR methods with significant reduction in the number of training parameters for larger upscaling factor.
A pan-sharpening method using joint and dual bilateral filters (DBFs) has been proposed. This approach is based on a consistent combination of large- and small-scale features obtained from the decomposition of high spectral resolution multispectral (MS) and high spatial resolution panchromatic (PAN) images. In the decomposition process, MS and PAN images are used to extract the features using joint and DBFs, respectively. These filters accommodate the relationship between MS and PAN images and decompose them into a base layer (large-scale) and a detail layer (small-scale). Since the joint bilateral filter (JBF) preserves the edges of an auxiliary image, it is used for decomposition of MS images where different layers are estimated using the PAN image as an auxiliary image. Similarly, different layers of the PAN image are obtained from a DBF which preserves the edges of both (MS and PAN) input images. This process is further extended to multistage decomposition to obtain a bilateral image pyramid. The base and detail layers of both MS and PAN images obtained at various stages are combined using a weighted sum. Finally, the estimated weighted sum of detail layer (small-scale) of the PAN image is fused separately to the weighted base layers (large-scale) of the MS images. Performance of the proposed method is evaluated by conducting the experiments on degraded as well as undegraded datasets, captured using different satellites such as Quickbird, Ikonos-2, and Worldview-2. The noise rejection capabilities of the proposed method are also tested by conducting experiments on the noisy data. The results are compared with the widely popular methods and the recently proposed fusion approaches based on a bilateral filter. Along with qualitative evaluation, the quantitative performance of the proposed fusion technique has also been verified by estimating different measures for degraded and undegraded experiments. The experimental results and quantitative measures demonstrate that the proposed method performs better in degraded and undegraded conditions along with noisy situations when compared to other state-of-art methods.
We propose two approaches of multiresolution image fusion using multistage guided filter and difference of Gaussians (DoGs). In a multiresolution image fusion problem, the given multispectral (MS) and panchromatic (Pan) images have high spectral and high spatial resolutions, respectively. One can obtain the fused image using these two images by injecting the missing high frequency details from the Pan image into the MS image. The quality of the final fused image will then depend on the method used for high frequency details extraction and also on the technique for injecting these details into the MS image. Specifically, we have chosen the guided filter and DoGs for detail extraction since these are more versatile in applications involving feature extraction, denoising, and so on. The detail extraction process in the fusion approach using a guided filter exploits the relationship between the Pan and MS images by utilizing one of them as a guidance image while extracting details from the other. The final fused image is obtained by adding the extracted high frequency details to the corresponding MS image. This way, the spatial distortion of the MS image is reduced by consistently combining the details obtained using both MS and Pan images. In the fusion method using DoGs, the high frequency details are extracted in the first and second levels by subtracting the blurred images of the original Pan. The extracted details at both DoGs are added to the MS image to obtain the final fused image. Advantages and disadvantages of each method are discussed and the comparison of the results is shown between the two. The results are also compared with the traditional and the state-of-the-art methods using the images captured using different satellites such as Quickbird, Ikonos-2, and Worldview-2. The quantitative assessment is evaluated using the conventional measures as well as using a relatively new index, i.e., quality with no reference which does not require a reference image. The results and measures clearly show that there is promising improvement in the quality of the fused image using the proposed approaches.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.