The guest editors introduce the Special Section on Advances in Urban Imaging and Applications.

Our study presents a probabilistic non-rigid point set registration method to deal with large and uneven deformations. Our method treats the registration as a density estimation problem. In our method, we add two key constraints to enforce landmark correspondences and preserve local neighborhood structure. We assume that the landmarks, which represent the salient points in the point sets, are given or can be detected using keypoint detectors such as scale-invariant feature transform or MeshDOG. By enforcing landmark correspondences, we preserve the overall global shape of the point set with significant deformations. Furthermore, by leveraging stochastic neighbor embedding, we incorporate constraints to preserve local neighborhood structure, which penalizes incoherent transformation within a neighborhood. Our experimental results in both 2D and 3D datasets show that our method outperforms state-of-the-art methods in a large degree of deformations. In particular, quantitative results show that the error is 29% better than the second-best result (from the state-of-the-art methods). Our analysis shows that a relatively small number of landmarks is sufficient to deal with large deformations. Finally, our study shows that our method is computationally comparable to state-of-the-art methods.

Rotation change is widespread in many computer vision applications. Robustness in the rotation is an important capability in evaluating local descriptors. We proposed a fast and rotation-robust local binary descriptor using polar location, called rotation-robust local difference binary descriptor (RLDB). The code of RLDB is generated by a binary test of average intensity, radial gradient, and tangent gradient of grid cells on multiple log-polar grids, which will save the presented local descriptors from spending extra time dealing with rotation changes. To speed up the computation, two tricks are proposed in the construction of the descriptor: one is using a lookup table mapping discrete polar coordinates with image pixel locations; the other is rebuilding the integral image in polar coordinates. Experimental results demonstrate that the presented descriptor is shown to be more robust in large rotation changes and viewpoint changes than some state-of-the-art local descriptors. At the same time, they also have comparable distinctive performance and construction speed.

Gradually crowded, complex airspace makes it necessary to identify the flight track patterns of interested targets. Existing studies on radar-based target track recognition rarely consider the impact of outliers in the acquired data, which happens very often for small air vehicles such as drones. In addition, the performance achieved with a few labeled track examples has significant room for improvement. We propose a semisupervised target track recognition algorithm based on a semisupervised generative adversarial network (SSGAN) that learns a robust model from a few labeled target track examples with the presence of outliers. Our method identifies and eliminates the outliers in the data set and fills in for the removed data. The proposed method extracts a strong recognition flight feature from the basic flight features and forms the strong recognition flight feature combination (SRFFC) by integrating the advanced flight features. The SRFFC is fed into the SSGAN model to identify target track patterns. Experiments were conducted using simulated data sets. Our results demonstrate that the proposed method achieves a highly competitive target track recognition performance in terms of accuracy, precision, and recall in comparison with the state-of-the-art methods. The minimum accuracy of our proposed method is 97%, which achieves an improvement of 15.7% compared with the state-of-the-art methods. In addition, our method exhibits great robustness with respect to the number of labeled data and choice of parameters.

Many imaging platforms, e.g., robotic devices and autonomous systems, require timely reaction to external events. The active balancing control is important to suppress rotor vibration and hence acquire stable videos and imagery. However, there is an inevitable delay in active balancing control of the high-speed, time-delay rotor systems. Eliminating this delay is crucial in real-world applications. The integral transformation term is introduced to transform the dynamic model of the time-delay rotor system into a dynamic model of the rotor system without time delay. Based on the linear-quadratic optimal control method, the influence of weight matrix is discussed. Moreover, an active balancing control strategy of the time-delay rotor is designed. The simulation model of the time-delay rotor control system is implemented with MATLAB, and a real device is fabricated to conduct both simulations and real-world experiments. Our results demonstrate that our proposed online active balancing control method effectively suppresses the vibration of the time-delay rotor system under different time-delay. The convergence time is short, and the system is agile.

In the context of facial expression recognition (FER), a lot of methods focus on feature extraction of the whole facial image, but sometimes it is equally important to concentrate on local area features that embody stronger emotions. This paper proposes an FER method based on the facial part attention mechanism (FPA). We adopt the attention mechanism to extract emotional rich local area features, which are complementary to the whole facial features for better FER. First, this paper proposes a cluster-based facial landmarks selection method to select facial landmarks that have commonality and best reflect local area emotions. Then, we design an FPA convolutional neural network, which consists of two parts. The first part is the object network, which is used to extract the whole facial features; the second part is the part attention network, and the attention component is used to extract the local part features. Finally, the two parts are merged to train the final classifier. The method was tested on the Real-World Affective Faces database and the recognition accuracy of 87.26% was obtained, which proved the effectiveness of the method.

Nowadays, with the growing maturity of action capture technology and the increasing requirements of the game and film industry for action and shooting level, action capture technology has become more and more widely applied. In addition, virtual shooting is widely applied to many TV programs and film production. For example, the movie “The Lion King” released in 2019 was completed using virtual shooting technology. For example, the orangutan king Caesar in the movie “Rise of the Planet of the Apes” was completed by actor Andy Serkis through motion capture and expression capture. In the process of research and use, the synchronization of virtual camera motion attitude has been difficult to solve, and its accuracy needs to be improved. Furthermore, either the traditional single optical motion capture or the inertial motion capture can better tackle the multiplayer motion capture. In order to solve this problem, based on the Unity3D platform, the position of camera and virtual role is accurately located by optical positioning technology, and then inertial attitude sensing technology is used to obtain the attitude of multiple targets, achieving the integration of the optical positioning and the inertial attitude sensor. As a result, the real-time virtual shooting of multiplayer motion capture can be realized.

We introduced an efficient technique to suppress speckle noise in medical ultrasound images while maintaining low computational complexity. The medical ultrasound image is divided into overlapping subimages, and Lanczos decomposition is then applied to the average Hermitian covariance matrix of all subimages. The resultant orthonormal vectors are used for filtering speckle noise through orthogonal and oblique projections, i.e., by projecting noisy signal onto the signal subspace. After sorting the orthonormal vectors, an orthogonal projection matrix is formed by selecting the first K vectors contributing to the signal, whereas an oblique projection matrix is formed by selecting the first K vectors contributing to the signal and the last K vectors contributing to the noise. The procedure of Lanczos is also followed with singular value decomposition (SVD). Schemes are applied to real ultrasound images and two types of speckle noise simulations: fine and rough speckle noise. Numerical and visual results depict that proposed technique has outperformed various popular benchmark schemes, i.e., Frost, Lee, probabilistic nonlocal means, geometric nonlinear diffusion filter, guided speckle reducing bilateral filter, and SVD, while maintaining a competitive computational complexity. Lanczos-based scheme showed a relatively lagging performance in terms of resolution, which was always outshone by a leading performance in terms of key measures such as signal-to-noise ratio (SNR), peak signal-to-noise ratio (PSNR), feature similarity, and mean structural similarity. Orthogonal and oblique projections were found to perform the same except when speckle noise is fine, where insignificant differences were found in terms of SNR, PSNR, and resolution. Lanczos-based scheme tends to offer a better estimation of orthonormal vectors than SVD, consequently a better speckle noise suppression. Furthermore, it offers an efficient tuning parameter per block/subimage size to treat various speckle noise patterns.

With the rapid development of deep learning, iris recognition methods based on deep learning are constantly being proposed. These methods generally consist of iris segmentation and normalization to more accurately locate iris regions and reduce the impact of iris feature changes caused by pupil expansion. We propose a non-segmentation (NS) iris recognition framework based on a deep learning classification model, which directly takes a raw image as input for feature extraction and recognition without performing iris segmentation and normalization. This method outperforms other methods. We proposed a non-segmentation network (NSNet). NSNet is a convolutional neural network (CNN) based on an attention mechanism that enhances the robustness and accuracy of the network by reusing features and assigning channel attention values. In addition, while ensuring the advanced performance of the network, it uses only 31 convolutional layers to complete iris feature extraction and recognition tasks in the NS iris recognition framework. Since the deep learning classification model cannot recognize that the category of an image is an untrained image category, we proposed a dual-threshold iris framework. In the proposed framework, all untrained image categories are classified as impostor classes, and the first threshold set in the proposed framework can effectively prevent impostors and eliminate inferior images. The proposed framework is suitable for the pursuit of more accurate, more robust, and safer private iris recognition scenarios. We conduct experiments with uniform parameter settings on four publicly available databases and shows that even in the challenging situation where a raw image is used as the input image, the proposed method is still the most advanced algorithm. Moreover, a series of ablation experiments were conducted to further confirm that the proposed framework has a higher accuracy, robustness, and generalizability through verification and discussion.

The vulnerability of face recognition-based unlocking systems to spoofing attacks is a serious problem. With the development of science and technology, face pictures on the Internet are particularly easy to obtain and recreate. A face spoofing attack happens when someone fakes a face and tries to pass the verification of the face recognition system, which seriously threatens the security of users’ information. We propose a method based on quantifying the difference between neighborhood pixels around the center of a local region, namely, weighted neighborhood difference quantization local binary pattern (LBP), against print attacks, and video-replay attacks. The proposed method quantifies the differences between neighborhood pixels without using the center one and uses a linear weighting scheme to improve the discriminant capability of the traditional LBP. The combination of the proposed algorithm and the spatial pyramid further improves the performance of face spoofing detection. We also conducted a lot of experiments in different color spaces to illustrate the role of color in face spoofing detection. The improved method achieves better results in three challenging face anti-spoofing databases, CASIA FASD, Replay-Attack, and Replay-Mobile, respectively.

The feedback mechanism method of simulating the biological vision system has not been widely used in deep learning dehazing algorithms. To alleviate the difficulty of feature interaction, we combine the feedback mechanism with dense skip connections to fuse features of different levels in a dehazing network. Inspired by the feedback network in which previous network layers can have access to rich information processed by the following network layers, we propose an end-to-end dense feedback network (DFBDehazeNet) for single image dehazing that implements the feedback mechanism using hidden states of constrained RNN. The low-level hazy feature information can be continuously corrected by the high-level feature information obtained from the dense feedback block via the recurrent feedback connection. The top-down feedback mechanism is adopted in DFBDehazeNet to refine the low-level hazy feature information, thereby achieving a powerful image restoration effect. The ablation experiment proves that the iterative structure of DFBDehazeNet and the projection unit play an important role in removing haze from images. The experimental results show that the results of image haze removal are superior to the great majority of existing methods both qualitatively and quantitatively.

Palmprint region of interest (ROI) extraction under unconstrained conditions is an unavoidable problem in realizing palmprint recognition. However, the diversity of palm size, posture, illumination, and background undoubtedly poses a great challenge. Therefore, in response to this problem, we collected about 30,000 unconstrained palmprint images from 100 people using five mobile phones with the flash on and off and then manually annotated the 14 key points of each image, which constitutes the most challenging palmprint database at present, namely XJTU-UP. In addition, we propose a method for palmprint ROI extraction. First, palm detection is performed to remove irrelevant background, then detect the key points, and finally establish a coordinate system based on the obtained points to extract the ROI. In the process of key points detection, an auxiliary network and data imbalance functionality are introduced to improve the accuracy. Finally, the experimental results on the XJTU-UP database show that the recognition accuracy has a maximum increase of 2.16% and the true acceptance rate is improved by up to 20.56% when the false acceptance rate is 0.01% compared to suboptimal method.

In recent years, to save bandwidth and storage space, images are usually compressed to reduce data volume, which leads to the loss of image details and affects the super-resolution (SR) performance. SR of compressed images is a key technique for addressing this problem. We propose the compressed-image super-resolution using enhanced attention network (CISREAN). First, the task is divided into two subtasks: image decompression and SR. Each subprocess introduces an enhanced residual module (ERM) with an attention mechanism. The ERM consists of several wide-activation residual blocks (WARBs) and an attention unit called the cascading residual attention (CRA) block. A WARB achieves better reconstruction than traditional image processing with the same computational complexity, and the CRA extracts more useful information in feature mapping even with fewer channels. This makes the ERM light and effective. Next, initial features are selected for wider inception and less blocking by overlapping extractions from the compressed image during decompression by convolutional layers. After completing both subprocesses, an end-to-end network is trained; it reduces compression artifacts, performing SR simultaneously. Extensive experiments on JPEG images with different quality factors show that CISREAN provides state-of-the-art performance based on objective metrics and subjective visual quality.

Supervised single-image super-resolution (SISR) reconstruction models are trained with both low-resolution images (I^LR) and their corresponding high-resolution images (I^HR). During the training process, I^LR are obtained by performing a bicubic downscaling on their I^HR counterparts. This means that the model learns an inverted version of the bicubic downscaling, resulting in less realistic images that are limited to specific conditions. Generating realistic textures is non-trivial. The obtained details are either blurred or not reminiscent of the usually observed textures. SISR reconstruction with faithful ground-truth texture and no external information remains an issue, especially when the degradation model is not defined. We propose an unsupervised internal learning method of a small convolutional neural network (CNN) using the internal image statistics. We use the power of deep generative models to capture latent representation of patches within the test image across two scales and train a downscaling CNN D_w to learn how to downscale the image by matching these latent distributions. D_w constitutes the downscaling operation with the correct image-specific degradation and is subsequently used in the generation of the training dataset. Obtained results show the effectiveness of our image-degradation estimation method in extracting inner-image statistics for a better super-resolution perceptual reconstruction.

Visual object tracking, which aims to estimate the position of an arbitrary target in a video sequence automatically, has drawn great attention in recent years. Many efforts have been made regarding this topic. The Siamese network, with a balanced accuracy and speed, has achieved great success. The Siamese network consists of two branches: one for the target image and the other for the search image. The position with the maximum score in the similarity map between the target and the search images indicates the place of the target image in the search image. Current Siamese trackers treat the features of different channels and spatial locations equally. However, the features of different channels and spatial locations may represent different semantic information. We propose a channel and spatial (CS) attention-based Siamese network for visual object tracking. A CS attention mechanism is inserted into the feature extractor to enhance the semantic feature learning. The experimental results show that the proposed network significantly improves the performance of the baseline tracker and is one of the top-ranked trackers among all tested state-of-the-art trackers on the most widely used visual object tracking datasets.

Compared with the improvement of model structure in the field of object detection, the imbalanced problems in the training process have received less attention, but it is also one of the main reasons affecting its performance. We mainly analyze the imbalanced problems that occur in different stages of the network training process and propose a more balanced network called Balanced-RetinaNet, which has three improvements. First, in the feature extraction stage, a multi-scale feature balanced module is proposed to settle the problem in terms of imbalanced feature distribution, that is, the high-level feature lacks spatial information and while the low-level feature lacks semantic information; then, in the object regression stage, an interval-based regression loss is proposed to solve the problem of inaccurate localization caused by the different contributions of different samples to the regression loss; finally, in the object classification stage, an adaptive focal loss is proposed to solve the problem of classification errors caused by the loss of a large number of negative samples overwhelming the overall classification loss. Experiments have proved that by solving the imbalanced problems, the detection accuracy has been significantly improved on the MS COCO dataset.

Nowadays, many RGB-D saliency detection models have been proposed, and they are conducted based on the pixel-level depth data obtained by active depth sensors or dense stereo matching algorithms in advance. However, the pixel-level depth maps may have a negative impact on saliency ranking, especially when they are inconsistent or invalid within the object area. It can be found that people tend to observe an object as a whole and ignore the details of the target surface. Inspired by this characteristic, we propose a salient object detection method via joint perception of region-level spatial distribution and color contrast. First, a twice segmentation strategy based on multifeature fusion is used to compute the region-level information. Then, the region-level spatial distribution maps are constructed instead of pixel-level depth maps, which is helpful for avoiding the interference of dense depth information. To improve the integrity of object detection, color saliency maps are also computed based on regional segmentation information. After that, we adopt a fusion strategy to achieve the effective complementarity of the two kinds of information. Two optimization strategies are employed to further improve the results of saliency ranking. Experimental results on two benchmark datasets demonstrate that the proposed method has better performance than most of the state-of-the-art methods, and it also shows competitive capability compared with deep learning-based methods.

Purpose: Recent research has been producing an important effort in encoding of digital image content. Most of the adopted paradigms only focus on local features and lack of information about location and relationships between them.

Approach: To fill this gap, we propose a framework built on three cornerstones. First, the adoption of attributed relational scale-invariant feature transform regions graph, for image representation. Second, the application of a graph embedding model, to work in a simplified vector space, is performed. Finally, fast graph convolutional networks address classification task on a graph-based dataset representation.

Results: The framework is evaluated on state of art object recognition datasets with uniform background.

Conclusions: A wide experimental phase is performed through a comparison to well-known competitors.

The ArUco marker is one of the most popular squared fiducial markers using for precise location acquisition during autonomous unmanned aerial vehicle (UAV) landings. This paper presents a novel method to detect, recognize, and extract the location points of single ArUco marker based on convolutional neural networks (CNN). YOLOv3 and YOLOv4 networks are applied for end-to-end detection and recognition of ArUco markers under occlusion. A custom lightweight network is employed to increase the processing speed. The bounding box regression mechanism of the YOLO algorithm is modified to locate four corners of each ArUco marker and classify markers irrespective of the orientation. The deep-learning method achieves a high mean average precision exceeding 0.9 in the coverless test set and over 0.4 under corner coverage, whereas traditional algorithm fails under the occlusion condition. The custom lightweight network notably speeds up the prediction process with acceptable decline in performance. The proposed bounding box regression mechanism can locate marker corners with less than 3% average distance error for each corner without coverage and less than 8% average distance error under corner occlusion.

Recently, deep learning has become a rapidly developing tool in the field of image fusion. An innovative image fusion method for fusing infrared images and visible-light images is proposed. The backbone network is an autoencoder. Different from previous autoencoders, the information extraction capability of the encoder is enhanced, and the ability to select the most effective channels in the decoder is optimized. First, the features of the source image are extracted during the encoding process. Then, a new effective fusion strategy is designed to fuse these features. Finally, the fused image is reconstructed by the decoder. Compared with the existing fusion methods, the proposed algorithm achieves state-of-the-art performance in both objective evaluation and visual quality.

Local features are widely applied in person re-identification (ReID) because of the rich fine-grained information. However, there are two problems existing in the local feature methods: the local regions are not accurate and the contextual information between different regions is not considered. To solve these problems, we propose a person ReID method based on attention clustering and long short-term memory (LSTM) network. First, in the feature extraction stage, an attention mechanism is utilized to suppress the background noise and a clustering operation is performed to extract accurate local features of human body parts. Second, we propose to regard the human body parts from head to foot as a sequence and utilize LSTM to take into account the contextual information between human body parts. Through the above two strategies, accurate local features of human body parts with contextual information can be extracted. In addition, we introduce the vector approximation-file index for fast ReID. Experiments on three benchmark datasets demonstrate the effectiveness of our method.

The high-efficiency video coding (HEVC) standard can reduce the bit rate of compression video. However, the compression efficiency of HEVC is still challenging current internet bandwidth. This problem can be solved by reducing the number of video frames encoded by HEVC and reconstructing the reduced frames that rely on frame rate up conversion (FRUC). Unfortunately, most existing FRUC methods need to re-estimate the motion information, a step that will lead to a huge increase of video decoder complexity. An improved HEVC video compression algorithm using low-complexity FRUC is proposed based on reusing the motion vector (MV) predicted by HEVC encoding procedure to directly reconstruct the skipped frames. First, an adaptive frame-skipping scheme is used to maximize reducing the number of frames that will encode by HEVC. Second, decoder extracts the MV and residual from the received bit stream. Then an MV reliability checking and rectifying method is proposed according to adaptive residual energy analyzing. Finally, a motion-compensation method and efficient occluded area processing algorithm are proposed to reconstruct skipped frames. These methods all directly utilize the rectified MVs. Experimental results show that due to the reduced frames and decoder that does not need to re-estimate MV, the proposed method can not only significantly decrease the time of all codec procedure when comparing HEVC and other FRUC-based video compression methods but also achieve 9.7022% to 66.0030% reduction in Bjøntegaard delta bit rate at low bit rates when compared to HEVC standard. In addition, the compression of the proposed method is also better than other state-of-the-art methods.

A convolutional capsule network that characterizes small objects previously has shown remarkable performance in small object classification. We extend the idea of capsules to the image denoising task and combine it with the generative adversarial network to develop a generative adversarial capsule network (DeCapsGAN). Both the generator and discriminator adopt the capsule network architecture. The convolutional capsule network is used to capture richer image features. We introduce deconvolution into the generator and propose a convolutional–deconvolutional capsule block. Skip connections are beneficial to transfer image features to deeper networks. A pretrained residual network (ResNet) is implemented as a feature extractor that captures features from the denoised image and reference image to measure the difference in perceptual information in the feature space. The performance of the proposed model is evaluated on the image with synthetic noise (Gaussian noise and mixed Gaussian with impulse noise) and real noise. Extensive experiments show that our model achieves excellent denoising performance in terms of both visual quality and quantitative metrics.

Action recognition makes significant contributions to sports video analysis, especially for athletes’ training evaluations. For sports video analysis, the action information is mainly conveyed by human body parts’ temporal movement, and each of the parts has a unique importance to the action representation. Aiming to involve this point in action recognition, we propose a part-attention spatio-temporal graph convolutional network (PSGCN) to exploit the dynamic spatio-temporal information in a sports video; it learns the importance of different parts to emphasize the contribution on the task of action recognition. Specifically, PSGCN first divides the human body into six parts and extracts their convolutional neural network (CNN) features, as well as concatenating the global feature of the whole frame; it then utilizes a cross-part and cross-frame graph building module to formulate the graph correlation of the parts from different frames. Inspired by the larger temporal variation of the same part containing more action information, we further propose a part-attention (PA) learning module to estimate the importance of each part, which can strengthen the graph correlation to support a PA graph. Finally, PSGCN conducts a graph convolutional network on the learned PA spatio-temporal graph with the learned part CNN features, which can obtain the action representation for the given sports video. In addition, the whole network is optimized by two losses of PA and action classification. To perform the superiority of PSGCN, we carry out extensive experiments of our model compared with several state-of-the-art methods on widely used action recognition datasets, especially for sports action. The results reflect the advantages of the proposed PSGCN on sports video analysis.

We propose an optimization framework for interactive image segmentation (IIS) that operates in bilateral space to achieve robust object extraction and instant visual feedback. More specifically, we first resample an input image using a regular bilateral grid with a resolution that is typically coarser than the input image to reduce the complexity of subsequent IIS tasks. We then design a Markov random field energy on the vertices of the bilateral grid that can be solved efficiently using a standard graph cut label assignment. To achieve this, we introduce reliable color models to distinguish the foreground and background despite the presence of extremely difficult cases and a higher-order potential to encourage spatial consistency in segmentation. We conduct comprehensive experiments on three standard interactive segmentation datasets, MSRA 10K, IIS, and PASCAL VOC 2012 segmentation validation set. The results show that the proposed method achieves competitive performance compared with state-of-the-art methods while making the current system efficient in terms of speed.

With the development of feature extraction technique, one image object can be represented by multiple heterogeneous features from different views that locate in high-dimensional space. Multiple features can reflect various characteristics of the same object; they contain compatible and complementary information among each other, integrating them together used in the special image processing application that can obtain better performance. However, facing these multi-view features, most dimensionality reduction methods fail to completely achieve the desired effect. Therefore, how to construct an uniform low-dimensional embedding subspace, which exploits useful information from multi-view features is still an important and urgent issue to be solved. So, we propose an innovative fusion dimension reduction method named tensor dispersion-based multi-view feature embedding (TDMvFE). TDMvFE reconstructs a feature subspace of each object by utilizing its k nearest neighbors, which preserves the underlying neighborhood structure of the original manifold in the lower dimensional mapping space. The new method fully exploits the channel correlations and spatial complementarities from the multi-view features by tensor dispersion analysis model. Furthermore, the method constructs an optimization model and derives an iterative procedure to generate the unified low dimensional embedding. Various evaluations based on the applications of image classification and retrieval demonstrate the effectiveness of our proposed method on multi-view feature fusion dimension reduction.

We proposed a motion-perceptive deformable alignment network that introduces a pre-computed optical flow to improve the motion perception of the deformable alignment process. The pre-computed flows share the burden of the learned offsets for motion estimation while the flexibility of the deformable convolutional network is maintained. In addition, a motion-adaptive pyramid structure in which the features are aligned in multi-scale levels and then merged based on the motion strength among input frames is proposed. With the above structures, an innovative space-time super-resolution (STSR) network is constructed with an improved motion compensation ability. STSR aims to restore a high-resolution and high-frame-rate sequence from its corresponding low-resolution and low-frame-rate version. The proposed STSR network is trained with a Vimeo-90K dataset, and tests are conducted on the Vimeo-90K, densely-annotated video segmentation (DAVIS), and realistic and diverse scenes (REDS) datasets. Performance is evaluated using the peak signal-to-noise ratio and structural similarity index of the entire restored frame on the Y channel. Extensive experiments demonstrate the superiority of the proposed network among both the one- and two-stage STSR methods, its improved alignment ability, and its significantly improved interpolated frame synthesis.

Facial expression recognition (FER) has applications in many scenarios, making it a valuable research direction. However, due to the uncertainty of real-world images, their recognition accuracy is not satisfying. To deal with this problem, we propose a two attention modules-based network that uses two different attention modules to extract features. The main classifier uses the self-attention module (SAM), and the auxiliary classifier uses the channel enhancement module. At the same time, the classification results of the two classifiers are constrained by variance regularization to suppress uncertainty. In addition, we use knowledge distillation to further optimize the predicted results using the mutual information between the teacher network and the student network. Our model achieves an optimal accuracy rate of 88.75% in the RAF-DB dataset and 80.59% in the FERPlus dataset, which is occluded with the lower half being visible.

Removing spatially variant motion blur from a blurry image is a challenging problem as image blur can be complicated and difficult to model accurately. Recent progress in deep neural networks suggests that kernel-free single image deblurring can be achieved, but questions about deblurring performance persist. To improve performance, we proposed a deep convolutional neural network to restore a sharp image from a noisy/blurry image pair captured in quick succession. Two neural network structures, Deblur Long Short-Term Memory (LSTM) and DeblurMerger, are presented to fuse the pair of images in either sequential or parallel manner. To boost the training, gradient loss, adversarial loss, and spectral normalization are leveraged. The training dataset that consists of pairs of noisy/blurry images and the corresponding ground truth sharp image is synthesized based on the benchmark dataset GOPRO. We evaluated the trained networks on a variety of synthetic datasets and real image pairs. The results demonstrate that the proposed approach outperforms the state-of-the-art methods both qualitatively and quantitatively. DeblurLSTM achieves the best debluring performance, while DeblurMerger achieves nearly the same result but with significantly less computation time.

Underexposed images are usually low in brightness and contrast, which degrade the performance of many computer vision algorithms. To solve the problem of overexposing areas that tend to be normal while recovering dark areas in low-light image enhancement tasks, we propose an image-to-patch enhancement model and design a lightweight convolutional neural network called PatchNet. Specifically, the new enhancement model indirectly enhances the network by introducing a patch image, which preserves the incremental information from the low-light image to the normal image. The incremental information is fused with the input image to recover the dark areas while protecting the normal areas of the image. Extensive experiments on real datasets demonstrate the advantages of our method over state-of-the-art methods in subjective feeling and objective evaluation. Our method has achieved better results in restoring details and the adjustment of brightness. By comparing to other methods, our method is more efficient.

Nowadays, deep learning is widely used to detect interest points and extract the corresponding descriptors and achieved suitable results for many applications of computer vision, such as image matching, three-dimensional reconstruction, simultaneous localization, and mapping. We propose an approach for interest point detection and descriptor extraction using pyramid convolution and circle loss, which is named as PC-SuperPoint. We utilize pyramid convolutions in the backbone network, which includes convolution kernels of different scales for multiscale feature extraction. The following well-designed networks are able to capture the local and global information from the obtained backbone feature maps. In addition, circle loss, which enhances weight attributes for each pair of descriptors, is also applied to improve the convergence speed in the training phase. Experiments on the HPatches dataset and KITTI dataset achieve promising results, which reveal the effectiveness of the proposed method.

Action recognition has wide applications in fields such as human–computer interaction, virtual reality, and robotics. Since human actions can be represented as a sequence of skeleton graphs, approaches based on graph neural networks (GNNs) have attracted considerable attention in the research action recognition. Recent studies have demonstrated the effectiveness of two-stream GNNs in which discriminative features for action recognition are extracted from both the joint stream and the bone stream. Each stream is generated by GNNs that support message passing along fixed connections between vertices. However, existing two-stream approaches have two limitations: no interaction is allowed between the two streams and temporary contacts between joints or bones cannot be modeled. To address these issues, we propose the interactive two-stream graph neural network, which employs a joint–bone communication block to accelerate the interaction between the joint stream and the bone stream. Furthermore, an adaptive strategy is introduced to enable dynamic connections between vertices. Extensive experiments on three large-scale datasets have demonstrated the effectiveness of our proposed method.

Single-image dehazing is a critical problem since haze existence degrades the quality of images and hinders most advanced computer vision tasks. Early methods solve this problem via the atmospheric scattering model, which estimate the intermediate parameters and then recover a clear image by low-level priors or learning on synthetic datasets. However, these model-based methods do not hold in various scenes. Recently, many learning-based methods have directly recovered dehazed images from inputs, but these methods fail to deal with dense haze and always lead to color distortion. To solve this problem, we build a recurrent grid network with an attention mechanism, named RGNAM. Specifically, we propose a recurrent feature extraction block, which repeats a local residual structure to enhance feature representation and adopts a spatial attention module to focus on dense haze. To alleviate color distortion, we extract local features (e.g., structures and edges) and global features (e.g., colors and textures) from a grid network and propose a feature fusion module combining trainable weights and channel attention mechanisms to merge these complementary features effectively. We train our model with smooth L1 loss and structural similarity loss. The experimental results demonstrate that our proposed RGNAM surpasses previous state-of-the-art single-image dehazing methods on both synthetic and real haze datasets.

Electro-optical (EO) imaging sensors are employed for object detection, recognition, identification, and tracking. The observation range of EO systems is defined as the maximum distance at which a target of interest can be seen and perceived. The performance of these systems deteriorates under degraded weather conditions, which severely affects the image quality and thus the observation range of EO surveillance systems. An improved image quality results in an enhanced observation range. Image quality can be improved by applying suitable digital image enhancement techniques by considering target/background signature knowledge. Under bad weather conditions, these digital techniques become ineffective due to the presence of scattered background signals. To reduce the effect of scattering, optical methods are also used to mitigate degradation caused by the atmosphere. It is observed that the optical method alone cannot produce the desired result. Observation range enhancement in surveillance system is achieved using the proposed hybrid method, which utilizes the optimal combination of optical and digital techniques in real-time. The proposed method not only improves human visual perception but also reveals more image content for better image analysis. The performance of the proposed range enhancement technique is compared with the predicted observation range model. The processed images using the hybrid method show more than 30% observation range enhancement under bad weather conditions. In addition to the subjective evaluation, qualitative and quantitative evaluations of image quality achieved through the proposed method are also presented. It is observed that the proposed hybrid method significantly improves the observation range and image quality of EO sensors under degraded weather in real time.

In the past few years, deep learning-based image inpainting has made significant progress. However, many existing methods do not take into account the rationality of the structure and the fineness of the texture, which leads to the scattered structure or excessive smoothness of the repaired image. To solve this problem, we propose a two-stage image inpainting model composed of structure generation network and texture generation network. The structure generation network focuses on the structure and color domain and uses the damaged structure map extracted from the mask image to reasonably fill the mask area to generate a complete structure map. The texture generation network uses the repaired structure map to guide the refinement process. We train the two-stage network on the public datasets Places2, CelebA, and Paris StreetView, and the experimental results show the superiority of our method over the previous methods.

Convolutional neural networks (CNNs) have achieved success in optical flow estimation using labeled datasets, but they fail to build an internal representation to fast adapt to the specific task. On the other hand, limited by the lack of ground truth, existing CNNs-based methods suffer from high noise sensitivity and inferior generalization performance. We integrate the meta-learning technique with optical flow estimation, which can learn internal features to search optimal initial state parameters of the network. Meanwhile, we devise an enhanced network termed MetaFlow to further improve performance. MetaFlow extracts per-pixel features, builds correlation volumes for all pairs of pixels, and iteratively updates optical flow through optical flow predictor using meta-learning. In addition, we propose a meta-transfer pretraining approach to obtain initial network weights, which can efficiently avoid network overfitting. Empirical experiments on MPI Sintel and KITTI benchmarks have shown that the proposed MetaFlow achieves the state-of-the-art results and performs outstanding in challenging scenarios such as textureless regions and abrupt motions.

Blurry image discrimination is a challenging and critical problem in computer vision. It is useful for image restoration, object recognition, and other image applications. In previous studies, researchers proposed a discrimination method based on hand-extracted features or deep learning. However, these methods are either pure data driven by deep learning or over-simplified assumptions on prior knowledge. As a result, a discrimination method is proposed for distinguishing sharp images and blurry images based on a fusion network. The proposed method can automatically discriminate and detect blur without performing image restoration or blur kernel function estimation. Actually, the blur and the noise are extracted by the improved VGG16 network and texture noise extraction algorithm, respectively. Then the fusion network integrates the advantages of deep learning and hand-extracted features, and achieves ultimate high-accuracy discrimination results. Rigorous experiments performed on own dataset and other popular datasets with a number of blurry images and sharp images, including RealBlur dataset, BSD-B dataset, and GoPro dataset. The results show that the proposed method outperforms with an accuracy of 98% on our own dataset and 94.8% on the other dataset, which satisfies the requirements of the image applications. Similarly, we have compared our method with state-of-the-art methods to show its robustness and generalization ability.

Patch-based approaches such as 3D block matching and non-local Bayes are widely accepted filters for removing Gaussian noise from single-frame images. We propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single-reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatiotemporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets, we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.

Deep learning has made substantial progress in crowd density estimation, but there are still some problems in existing methods, such as large population density, background interference, and scale change, which makes it difficult to count people. To solve the above problems, we proposed a crowd counting method based on a cross column fusion attention mechanism. First, the first ten layers of VGG16 with good migration ability and feature extraction ability are used as the front-end network to preliminarily extract human head features. Then, a cross column fusion attention module is designed. In this module, feature maps are fused across columns to make the network contain richer deep and shallow features. At the same time, to alleviate the background interference, the attention mechanism is used to guide the network to focus on the head position in the picture, and different weights are assigned to different positions according to the attention score map, so as to highlight the crowd and weaken the background, and finally get a high-quality density map. In addition, a shallow convolution module is designed as another branch. The output feature map of the shallow convolution module and the output feature map of the attention module of cross column fusion are fused to solve the problem of scale change effectively. Finally, in the last layer of the network, the convolution layer of 1  ×  1 is used to replace the full connection layer, and fewer network parameters are used to reduce the calculation and the population density map is regressed. The experimental results show that the mean absolute error and mean square error of the proposed algorithm are significantly reduced compared with the comparison algorithm.

Generative adversarial network (GAN) has made great progress in image generation and reconstruction, but it may cause the mode collapse problem in practice. We proposed the dual discriminator weighted generative adversarial network (D2WGAN), whose objective function weights the Kullback–Leibler divergence (KL divergence) and the reverse KL divergence and uses the complementary characteristics of these two divergences to make the generated models more diverse. Moreover, we proved the theoretical conditional optimality of the D2WGAN to show that the generator can learn the real data distribution. Finally, we conduct experiments on a large amount of synthetic data and real-world datasets (e.g., MNIST and CIFAR-10). The results show that, compared with the traditional dual discriminator generative adversarial network and GAN, the proposed D2WGAN can process multiple mode data and generate better sample diversities.

The deployment of large-scale deep neural networks on field programmable gate array (FPGA) platforms is severely hindered by the high requirements on computational resources and off-chip data bandwidth. Traditional nonstructured sparsity algorithms can efficiently reduce the nonzero weights of neural network models. However, the nonstructured sparse connections across channels also degrade the degree of computational parallelism and consequently seriously deteriorate the performance of the FPGA accelerator. We propose an FPGA accelerator by exploring the kernel structured sparsity and hybrid arithmetic computation for the convolutional neural network (CNN). On the one hand, we introduce a hardware-friendly kernel pruning method to reduce the number of arithmetic operations of the CNN model. Our proposed method maintains high accuracy (achieving a less than 0.32% accuracy loss) and achieves a high degree of parallelism. On the other hand, we design a specific hybrid arithmetic computation for the FPGA accelerator to speed up the performance of the pruned CNN model. The FPGA accelerator consists of only 64 sets of hybrid 8-bit and 16-bit floating-point units for the convolution operation. Experiments on VGGNet16 demonstrate that the proposed FPGA accelerator achieves a state-of-the-art 5  ×   convolution operation reduction and a 3  ×   parameter compression. The proposed FPGA accelerator is able to perform at 13.2 FPS, and the corresponding energy efficiency can be boosted up to 1.9  image  /  J.

Iris is one of the most well-known biometrics; it is a nonintrusive and contactless authentication technique with high accuracy, enhanced security, and unique distinctiveness. However, its dependence on image quality and its frontal image acquisition requirement limit its recognition performance and hinder its potential use in standoff applications. Standoff biometric systems require a less controlled environment than traditional systems, so their captured images will likely be nonideal, including off-angle. We present convolutional neural network (CNN)-based deep learning frameworks to improve the recognition performance of iris, ocular, and periocular biometric modalities for off-angle images. Our contribution is fourfold: first, the performances of popular AlexNet, GoogLeNet, and ResNet50 architectures are presented for off-angle biometrics. Second, we study the effect of the gaze angle difference between training and testing images on iris, ocular, and periocular recognitions. Third, we investigate the network behavior for untrained gaze angles and the information fusion capability of CNN networks at multiple off-angle images. Finally, deep learning-based results are compared with a traditional iris recognition algorithm using the gallery approach. Our results with off-angle images ranging from −50  deg to 50 deg in gaze angle show that the proposed methods improve the recognition performance of iris, ocular, and periocular recognition.

Seatbelt detection is an important topic in intelligent transport systems. The accuracy of seatbelt detection in traffic video surveillance is affected by many factors, such as complex road environments, lighting, weather, direction of the camera, etc., which bring difficulties for traditional image processing methods. We propose a seatbelt detection method based on classification using convolutional neural networks. Dense residual blocks are used to avoid gradient dispersion and information loss during the network training. A two-level attention mechanism is also introduced to further improve the network performance by simultaneously modeling the correlation among different feature channels using channel attention mechanism, and correlation among different pixels within the feature map using pixel attention mechanism. To make the whole system complete, we added the driver area localization module, which is accomplished using the YOLOV3 model. Small images concentrated on the driver location, which are obtained through the localization operation from larger images, are passed into the classification network for seatbelt detection. We compared the proposed method with several other methods. Comparative results show that the proposed method has higher accuracy and is more robust for seatbelt detection for complex environments.