Object tracking is challenging due to unconstrained variations in the target’s appearance and complex environmental variations. Appearance models based on a single cue are inefficient in addressing the various tracking challenges. To address this, we propose a discriminative visual tracking approach in which complementary multicue features viz. RGB cue and histogram of gradient are integrated to build an efficient appearance model. The multicue feature fusion ensures that the limitations of the individual cue are suppressed and complementary multicue information are appropriately captured in the unified feature. These unified features are robust to scale variation, rotation, and background clutter. In addition, random forest classifier not only creates clear decision boundary between the foreground fragments and the background fragments but also aids in adaptive update of the reference dictionary. This adaptive update strategy avoids the eventual drift of the tracker during illumination variation, rotation, and deformation. Extensive qualitative and quantitative analyses on benchmark OTB-2015 and VOT2017 datasets demonstrate the robustness and accuracy of the proposed tracker against seven other state-of-the-art tracker.

Infrared-visible cross-modality person re-identification (IV-ReID) is a challenging task that aims to match infrared person images with visible person images of same identity. The person images of two modalities are captured by visible cameras and infrared cameras, respectively. Due to the variation between two modalities, most existing methods tend to extract common features of different modalities by shared network. However, on account of ignoring the effect of single-modality features, the way of merely extracting common features loses part of single-modality information. To address this problem, we propose an end-to-end model, multi-complement feature network (MFN), to complement common features with single-modality features. We divide MFN into two modules, feature extracting module (FEM) and feature complementing module (FCM). At the stage of FEM, we employ a two-stream network with architecture of multiple granularities to extract single-modality features and common features. Afterward, at the stage of FCM, we utilize the characteristic of graph convolution network (GCN) to associate multiple features of different modalities. In FCM, we design a concise but effective graph structure that takes the features extracted by FEM as input of GCN. Compared with previous methods, our method reserves single-modality features and makes them work with common features. Extensive experiments implemented on two mainstream datasets of IV-ReID, SYSU-MM01, and RegDB demonstrate that our method achieves state-of-the-art performance.

Rain streaks attached to a camera may seriously affect the visibility of the background and considerably degrade image quality. We handle this problem by removing rain streaks to convert a rainy image into a clean one. This creates a problem in that the information with respect to the background of the occluded parts is close to being lost for the most part. To remove rain streaks from a single image, we employ a multi-scale aggregated generator network. Differing from previous generative adversarial networks, an enhanced generator block (EGB) is applied in the generator; this broadens the receptive field of the input features and enhances the attention to rain streaks details. In addition, both the generator and the discriminator networks contain the visual mask that learns the rain streak regions and their surroundings during training. By putting the mask forward, the generator network focuses on rain streaks and the surrounding regions and the discriminator network is capable of evaluating the local consistency of the restored regions.

Gender classification has found many useful applications in the broader domain of computer vision systems including in-cabin driver monitoring systems, human–computer interaction, video surveillance systems, crowd monitoring, data collection systems for the retail sector, and psychological analysis. In previous studies, researchers have established a gender classification system using visible spectrum images of the human face. However, there are many factors affecting the performance of these systems including illumination conditions, shadow, occlusions, and time of day. Our study is focused on evaluating the use of thermal imaging to overcome these challenges by providing a reliable means of gender classification. As thermal images lack some of the facial definition of other imaging modalities, a range of state-of-the-art deep neural networks are trained to perform the classification task. For our study, the Tufts University thermal facial image dataset was used for training. This features thermal facial images from more than 100 subjects gathered in multiple poses and multiple modalities and provided a good gender balance to support the classification task. These facial samples of both male and female subjects are used to fine-tune a number of selected state-of-the-art convolution neural networks (CNN) using transfer learning. The robustness of these networks is evaluated through cross validation on the Carl thermal dataset along with an additional set of test samples acquired in a controlled lab environment using prototype uncooled thermal cameras. Finally, a new CNN architecture, optimized for the gender classification task, GENNet, is designed and evaluated with the pretrained networks.

Semantic segmentation of three-dimensional (3D) scenes is a challenging task in 3D scene understanding. Recently, deep learning-based segmentation approaches have made significant progress. A multiscale spatial context feature learning is used in an end-to-end approach for 3D point cloud semantic segmentation. Furthermore, a local feature fusion learning block is then introduced to the hidden layers in the network to improve its feature learning capability. In addition, features learned in several different layers are fused for further improvement. Based on these strategies, end-to-end architectures are finally designed for 3D point cloud semantic segmentation. Several experiments conducted on three publicly available datasets have clearly shown the effectiveness of the proposed network.

It is very challenging to recognize the information of occluded Tibetan document pages due to the lack of digitization and their long-term storage. Multiple pages are stuck, and textual characters are occluded with each other, which causes difficulties in restoration. Due to the large size of Tibetan documents, it is impossible to separate and repair these occluded pages by professionals. Therefore, the separation of overlapping pages and restoration of occluded pages play important roles in the digitization of Tibetan documents. We extract underlying pages by show-through scanning and eliminating the text area of top pages. In order to restore the occluded underlying pages, we present a side-enhanced U-Net (SEU-Net) that attaches side feature extraction module and side classification module to the U-Net to improve the classification of textual edges. Experiments performed on the dataset of Tibetan documents restoration patches show that SEU-Net is able to classify the textual pixels in the occluded pages accurately, and both side feature extraction module and side classification module improve performance independently.

Steganography is a technique applied to ensure secure communication. It is challenged by visual observation or statistical analysis to ascertain whether a message is hidden within a cover medium. Several steganographic methods have been proposed for palette-based images. These methods maintain image quality but cannot resist some statistical and visual attacks. To overcome this problem, two parity assignments with performances similar to those of existing parity assignments are proposed. An innovative embedding process that randomly selects one parity assignment while embedding secret bits in each pixel is provided. Finally, a steganographic method using the proposed embedding process and a specified adaptive scheme is presented. Experimental results revealed that the proposed method is undetectable under some statistical and visual attacks even as it maintains image quality.

Several leading-edge applications such as pathology detection, biometric identification, and face recognition are based mainly on blob and line detection. To address this problem, Eigen value computing has been commonly employed due to its accuracy and robustness. However, Eigen value computing requires a raised computational processing, intensive memory data access, and data overlapping, which involve higher execution times. To overcome these limitations, we propose in this paper a new parallel strategy to implement Eigen value computing using a graphics processing unit (GPU). Our contributions are (1) to optimize instruction scheduling to reduce the computation time, (2) to efficiently partition processing into blocks to increase the occupancy of streaming multiprocessors, (3) to provide efficient input data splitting on shared memory to benefit from its lower access time, and (4) to propose new data management of shared memory to avoid access memory conflict and reduce memory bank accesses. Experimental results show that our proposed GPU parallel strategy for Eigen value computing achieves speedups of 27 compared with a multithreaded implementation, of 16 compared with a predefined function in the OpenCV library, and of eight compared with a predefined function in the Cublas library, all of which are performed into a quad core multi-central-processing unit platform. Next, our parallel strategy is evaluated through an Eigen value-based method for retinal thick vessel segmentation, which is essential for detecting ocular pathologies. Eigen value computing is executed in 0.017 s when using Structured Analysis of the Retina database images. Accordingly, we achieved real-time thick retinal vessel segmentation with an average execution time of about 0.039 s.

The detection of plant organs is an important research field of plant recognition area. However, due to the lack of database of plant organs, the application of convolutional neural network-based object detection on plant species is very limited. A database of plant organs for deep learning-based object detection is constructed. A huge number of plant images are clawed using specific keywords through keyword search engines such as Baidu and Google. After that, an automatic junk image cleaning method is performed to remove junk images. Finally, artificial labeling is used to delineate plant organ regions. To evaluate the quality of the database, experiments in different object detection models are implemented. Results show that the established plant organ database has good performance in plant organs positioning and classification.

Convolutional neural networks (CNN) have immensely promoted the development of face recognition (FR) technology. In order to achieve global accuracy, CNN models tend to be deeper or multiple local facial patch ensembles, leading to excessive amounts of calculation. We address these deep FR problems and propose a lightened deep learning framework under an open-set protocol to achieve a good classification effect and streamline the model itself. To this end, we improve the SphereFace that enables the deep network to learn angularly discriminative features more efficiently. First, global average pooling (GAP) is introduced to replace the original fully connected (FC) layer, which greatly reduces the storage of the model. Compared to the widely used FC layer, GAP can reduce the number of parameters and avoid overfitting. Then multilayer perceptron is added between convolution layers, which increases the ability to characterize features. These models are trained on the CASIA-WebFace dataset and evaluated on the Labeled Faces in the Wild and YTF datasets, which show the excellent performance of lightened SphereFace (L-SphereFace) in FR tasks. At the same time, computational cost is reduced in comparison with the released SphereFace model. The storage space of the model is also greatly compressed.

Subtraction techniques are used to distinguish moving objects or foregrounds that are being tracked from a static background. To prevent possible local misclassifications, the subspace spanned by low-rank features computed from a multilinear singular value decomposition (MLSVD), can be used to filter out noises or gradual changes from the background. However, as it is prohibitively expensive to compute a new MLSVD from scratch at every iteration, we propose an adaptive and efficient method for updating an MLSVD by reusing previous decompositions while tracking more accurate decomposition errors. The experimental results reveal that the proposed MLSVD update algorithm exhibits a faster execution speed and better accuracy than other MLSVD update algorithms used in background subtraction applications.

As an instance-level recognition problem, the key to effective vehicle re-identification (Re-ID) is to carefully reason the discriminative and viewpoint-invariant features of vehicle parts at high-level and low-level semantics. However, learning part-based features requires a laborious human annotation of some factors as attributes. To address this issue, we propose a region-aware multi-resolution (RAMR) Re-ID framework that can extract features from a series of local regions without extra manual annotations. Technically, the proposed method improves the discriminative ability of the local features through parallel high-to-low resolution convolutions. We also introduce a position attention module to focus on the prominent regions that can provide effective information. Given that the vehicle Re-ID performance can be affected by background clutters, we use the image obtained through foreground segmentation to extract local features. Results show that using original and foreground images can enhance the Re-ID performance compared with using either the original or foreground images alone. In other words, the original and foreground images complement each other in the vehicle Re-ID process. Finally, we aggregate the global appearance and local features to improve the system performance. Extensive experiments on two publicly available vehicle Re-ID datasets, namely, VeRi-776 and VehicleID, are conducted to validate the effectiveness of each proposed strategy. The findings indicate that the RAMR model achieves significant improvement in comparison with other state-of-the-art methods.

We propose a network structure for action recognition that is capable of extracting multi-scale temporal representations of actions. The key of the network is to combine a multi-scale temporal pooling module with a dense connection module, called multi-scale temporal pooling dense convolutional network (MTPDNet). The multi-scale temporal pooling module consists of multiple temporal scale levels. At each scale level, video frames are divided into several segments and a pooling operation is then performed on each segment to get temporal pooling information. The number of segments is set differently at different time scale levels, aiming to obtain multi-scale temporal pooling information. In addition, at each scale level, we adopt a redesigned dense connection module to learn motion representations from temporal pooling information. Finally, predictions are independently made at each scale level and the class scores of each scale level are fused to get the final prediction scores. Experimental results on two standard datasets, UCF101 and HMDB51, demonstrate that MTPDNet gets comparable or even better results among leading methods, which proves the effectiveness of the strategy combining multi-scale temporal pooling and dense connection.

Text-to-image generation can be widely applied in various fields, such as scene retrieval and computer-aided design. The existing approaches can generate realistic images from simple text descriptions, whereas rendering images from complex text descriptions is still not satisfactory for practical applications. To generate accurate high-resolution images from given complex texts, we proposed an attention-enhancing adversarial learning network (Attn-Eh ALN) based upon conditional generative adversarial networks and the attention mechanism. This model consists of an encoding module and a generative module. In the encoding module, we proposed a local attention driven encoding network that allows words in the text with different weights to enhance the semantic representation of specific object features. The attention mechanism is employed to capture more details while ensuring global information. This enables the details in the generated images to be more fine-grained. In the discriminating stage, we take multiple discriminators to distinguish the realness of the generated images, avoiding the bias caused by a single discriminator. Moreover, a semantic similarity judgment module is introduced to improve the semantic consistency between the text description and visual content. Experimental results on benchmark datasets indicate that Attn-Eh ALN generates favorable results in comparison with other state-of-the-art methods from qualitative and quantitative assessments.

Recently, deep learning has developed rapidly and made great improvements in facial expression recognition. However, deep learning has black box properties that lead to poor interpretability of the results. Differentiable programming provides a new aspect to balance the interpretability and convenience of deep learning. It provides a way to solve sparse coding problem that has solid mathematical foundations through recurrent neural network end-to-end. However, sparse representation is a traditional unsupervised learning method, and it does not effectively exploit the supervised information, which is helpful for facial expression recognition. We propose a differentiable programming algorithm that is called supervised graph-preserving sparse two neural network (SGS2Net) by exploiting both sparse and graph-preserving properties. The graph-preserving constraint may contain the class information. Therefore, the new model can be conducted as an unsupervised or a supervised way. A sparse representation of the facial images is obtained by minimizing the l₁-norm of the coefficients, and the neighborhood of the samples is preserved by retaining the graph structure. The optimization procedure is conducted by gradient descent and threshold shrinkage and implemented by a new deep network structure end-to-end. SGS2Net is applied to facial micro-expression recognition and facial expression-based pain assessment, and it enhances the recognition accuracy by more than 10% and 4%, respectively, compared to state-of-the-art. It derives the fact that graph-preserving constraint improves the discriminant property of the network greatly, and SGS2Net has solid interpretability from sparse representation and graph embedding theory.

Video super-resolution (VSR) is an image restoration task, aiming to reconstruct a high-resolution (HR) video from its down-sampled low-resolution (LR) version. Convolutional neural networks (CNNs) have been applied to VSR successfully. Explicit motion estimation and motion compensation (ME&MC) module is commonly used in the previous CNNs-based methods to better exploit input frames’ temporal similarity. We proposed a VSR network without an explicit ME&MC module. Our network makes full use of spatiotemporal information and can implicitly capture motion relations between frames. Specifically, we proposed an enhanced deep feature extraction module (EDFEM) to extract deep features from input frames. EDFEM exploits not only intra-frame spatial information but also inter-frame temporal information to enhance feature representation. Furthermore, we proposed a residual up-down block (RUDB) to fuse features. RUDB adopts up- and down-sampling layers as the residual branch. Compared to the common residual block, RUDB addresses mutual dependencies of LR and HR images. Visual and quantitative results show that our method achieves state-of-the-art performance.

A brightness preservation dehazing algorithm based on the atmospheric scattering model is proposed. We introduce a novel pixel-patch-wise transmission. It not only can keep the scene radiance depth from becoming underestimated, it also preserves image color and brightness. First, based on the correlation of saturation and brightness in hue, saturation, and intensity color space, a pixel-wise transmission is estimated. To add the local information into the transmission, then a novel patch-wise visual mask by the free-energy principle is presented. With this mask, we can optimize the transmission and atmospheric light. Since the proposed method takes account of pixel, patch, and pattern information, it can restore more natural haze-free scenes. Experimental results demonstrate that our method achieves better restorations compared with some other state-of-the-art approaches.

Natural images inevitably suffer from spatially variant blur caused by the relative motion between a camera and objects. We present an effective and efficient patch-wise edge-enhanced image regularization and a robust kernel similarity constraint to perform an accurate kernel estimation from coarse-to-fine iterations. The proposed adaptive regularization introduces a gradient magnitude penalty function into total variation to preserve and enhance salient edges while smoothing out harmful subtle structures. In addition, the similarity constraint is engaged in each patch without camera rotation effects, ensuring that the erroneous kernels can be identified by measuring the similarity among the kernels of neighbor patches and be replaced with the well-estimated ones. After obtaining accurate kernels, numerous nonblind deblurring methods can be applied to restore an image. Numerical experiments demonstrate that the proposed algorithm performs favorably without ringing artifacts and possesses high processing efficiency for natural nonuniform blurred images.

Document image binarization is the process in which pixels in a document image are classified into two groups—foreground and background. This process becomes challenging when it deals with various degradation and noise present in the images. In the recent past, it has been observed that researchers are relying on deep learning-based approaches to solve the problem of document image binarization. Of these, a group of methods considers the segmentation as a pixel-level classification problem, whereas another group considers it as an image-to-image translation problem. We have explored two popular deep learning-based architectures, one from each group, namely, U-Net and Pix2Pix, and presented a comparative assessment of their performances when applied for degraded document image binarization. In this study, no preprocessing or postprocessing methods are applied, which helps us to realize the actual strength of these architectures for the said purpose. For the performance evaluation and comparative assessment, six publicly available standard datasets, namely, document image binarization competition 2013 (DIBCO 2013), H-DIBCO 2014, H-DIBCO 2016, DIBCO 2017, H-DIBCO 2018, and DIBCO 2019, are considered. The performances of these architectures are compared with the best performing methods of the respective binarization competitions, some state-of-the-art nondeep learning-based methods, and some recently published deep learning-based methods separately. The obtained results confirm that in most of the cases U-Net outperforms the Pix2Pix model.