1.

INTRODUCTION

Esophageal cancer is the sixth leading cause of cancer-related mortality and the eighth most common cancer worldwide[1]. It affects more than 500 000 people worldwide and the incidence is rapidly increasing [2]. To date, barium esophagram is a global test for patients with dysphagia because it can simultaneously detect morphologic and functional abnormalities in the pharynx and esophagus [3]. In addition, due to its inexpensive and noninvasive feature as well as widely available imaging procedures, barium esophagram has a priority than other technologies such as endoscopy for clinical diagnostic selection [4]. Although esophagoscopy is the gold standard for esophageal cancer diagnosis, it is invasive and expensive [5]. Therefore, barium esophagram would be a cost-effective approach that can serve as an important initial diagnostic test for patients with dysphagia [6,7]. Previous study suggested that barium esophagram is a sensitive modality to diagnose esophageal cancer [8]. However, inter-observer inconsistent interpretations are inevitable because of manual examination through radiologists and diagnosis by the conventional visual assessment. Furthermore, multi-directional barium esophagrams often consume a lot of time for interpretation.

Van et al presented a novel algorithm for automatic detection of early cancerous tissue in High Definition (HD) endoscopy[9]. The algorithm achieves a result with recall of 0.95 and precision of 0.75 in 38 lesions by calculating local color and texture features based on the original and Gabor-filtered image. Van et al presented an algorithm which employed specific texture, color filters, and machine learning to detect early neoplastic lesions in Barrett ’ s esophagus[10]. Based on 100 images from 44 patients with Barrett’s esophagus, the system achieved a sensitivity and specificity of both 0.83 at image-level and a sensitivity and specificity of 0.86 and 0.87 at patient-level. In the above literature, the esophageal cancer CAD system is mainly studied for esophagoscopy. Zhang et al. designed and developed an automatic deep learning system to detect esophageal cancer [5]. This study marked the first use of deep learning methods to detect esophageal cancer on barium esophagography. While the system showed remarkable results in terms of recall rate, it faced challenges in processing barium meal esophagography with irregular, complex morphological characteristics, particularly in distinguishing between the structure of a narrow benign esophagus and malignant esophageal carcinoma.

Nowadays, Deep learning algorithms, in particular Convolutional Neural Networks, have rapidly become a methodology of choice for analyzing medical images. Girshick R proposed an object detection method R-CNN based on the combination of RoI (Region of Interest) and CNN (Convolutional Neural Network)[11]. This algorithm is the first masterpiece of an object detection algorithm based on deep learning. R-CNN first generates 2000 category-independent RoIs containing suspicious objects from the input image and then inputs these 2000 RoIs into the CNN model to extract fixed-length feature vectors from each RoI. These feature vectors are used as the input of the SVM and are got by SVM the probability that the RoI belongs to a certain category of object. Although R-CNN implements an object detection algorithm based on deep learning, the parameter of convolution layers cannot be shared during the feature vector extraction of RoIs. Girshick R was inspired by SPP-NET[12] and proposed the Fast Region-based Convolutional Neural Network Fast R-CNN[13]. This algorithm first proposed the RoIPooling layer which makes the parameters of convolution layers used for RoI feature vector extraction share. However, Selective Search algorithm which runs on the CPU is still used in the extraction of RoIs and it leads to the slow speed of the algorithm. Ren Shaoqing proposed the object detection network Faster R-CNN[14] which used Region Proposal Network (RPN) to detect region proposals. The main steps of object detection include extracting Region proposals from images, extracting features from Region proposals, and classification and regression of the bounding box, all of which are integrated into the network. Therefore, once Faster R-CNN is proposed, it shows obvious advantages in medical image detection.

Figure 1 shows the main challenges of detecting esophageal cancer in images of the barium esophagram. Given this and inspired by other researcher’s work above, we propose a deep learning system to identify positive patient cases and detect esophageal cancer. The system mainly consists of two networks, Faster R-CNN detection network with Deformable FPN (FRDF) and images of the barium esophagram classification network with attention module (BEAM).

Figure 1.

a~f are examples of images of the barium esophagram. The red rectangle indicates the gold standard for the lesion labeled by the radiologist. The main challenges include high brightness value in the lesion (a), different scales of lesions (b, d, and e), and irregular deformation of the esophagus (a~f). g, h, and i are lesions of esophageal cancer. Challenges include the presence of a large-scale background area within lesions.

The layout of this paper is organized as follows: the dataset and proposed research methodology are given in Section 2. The experimental settings, parameter optimization, performance metrics, results analysis, and ablation study are shown in Section 3. A brief discussion is provided in Section 4, and the conclusion and future work are presented in Section 5.

2.

METHODS

2.3

Detection network

As shown in Figure 2, the FRDF (Faster R-CNN detection network with Deformable FPN) is mainly composed of Deformable ResNet50 Backbone Network, Feature Pyramid Network, Region Proposal Network, MultiScale-RoIAlign Layer, and Fast R-CNN Predictor Network.

Figure 2.

The proposed FRDF architecture, contains Deformable ResNet50 Backbone Network, Feature Pyramid Network, Region Proposal Network, MultiScale-RoIAlign Layer, and Fast R-CNN Predictor Network.

The images of the barium esophagram are input to the Deformable ResNet50 Backbone Network to extract features. Feature Pyramid Network fuses features of different layers. Region Proposal Network generates region proposals by these features. For region proposals of different sizes, the MultiScale-RoIAlign Layer maps their feature vectors to the same size. Finally, the feature vectors of these region proposals are input into the Fast R-CNN Predictor Network to output RoIs and corresponding confidence. Each improved networks are introduced as follows.

2.3.1

Deformable ResNet50 Backbone Network

In FRDF, the backbone network is proposed to extract features. The feature extraction ability of ResNet50 is improved by a large number of convolution layers. However, ResNet50 is still defective for extracting features of images because convolution layers can’t adapt to esophageal deformation due to its fixed geometric structures. Therefore, a Deformable ResNet50 Backbone Network is proposed as shown in Figure 3a. It replaces the residual bottleneck block in the last convolutional group of ResNet50 with Deformable residual bottleneck block shown in Figure 3b. 3*3 Deformable Convolutional Network in Deformable residual bottleneck block as shown in Figure 3c.

Figure 3.

(a): Deformable ResNet50 Backbone Network architecture. (b): Deformable residual bottleneck block. (c): 3*3 Deformable Convolutional Network in deformable residual bottleneck block. The upper branch learns sampling point offset paths to adjust the sampling point location and the lower branch learns weights of offset sampling points to weight modulate the offset sampling points.

2.3.2

Feature Pyramid Network

Feature Pyramid Network (FPN) as shown in Figure 4. It merges convolution layers to enhance the semantic features. As a result, the features of convolution layers in the last convolutional group (Deformable_conv5.x in Figure 4) are fused to Conv2.x, Conv3.x and Conv4.x convolutional groups. So these convolutional groups also contain features after adaptive sampling with Deformable_conv5.x. Replacing bottleneck residual block with Deformable residual bottleneck block in these groups is redundant. Therefore, this work only replaces the residual bottleneck block with Deformable residual bottleneck block in Conv5.x group.

Figure 4.

Deformable ResNet50 Backbone Network+FPN

Specifically, the feature maps in the last convolution layer of each convolutional group are changed channels through the 1×1 convolution layer. Then feature map with a lower resolution is upsampled (nearest neighbor upsampling) and merged into a feature map with a higher resolution through element addition. Finally, the 3×3 convolution layer is used to generate the feature prediction layer (P3, P2, P1, R0). P4 is generated through MaxPooling based on P3. The feature prediction layer R0 generates minimal anchors and these anchors will generate minimal region proposals by RPN. However, no minimal lesions exist in the images and these useless region proposals will significantly increase the computation costs. So R0 is only used to complete feature mapping in some region proposals instead of using as a predictive feature layer.

2.3.3

MultiScale-RoIAlign Layer

The purpose of the MultiScale-RoIAlign Layer as shown in Figure 5 is to map feature vectors between feature maps and region proposals and then scale these feature vectors to the same size.

Figure 5.

MultiScale-RoIAlign Layer

The mapped region proposal in the feature map is divided into 7*7 bins. In this work, we find that the best performance can be obtained by setting four sampling points. So the bin is divided into 2*2 regions on average and no rounding is performed. Bilinear interpolation is used for calculating the central point value of these regions. Finally, the mean of these four central points is taken as the feature value of this bin. Similar processing is done for other bins and these feature values of bins are concatenated as the fixed feature vector for this region proposal.

2.4

Barium esophagram classification network with attention module

In the FRDF, feature vectors used for locating RoIs and classifying RoIs are extracted by the Deformable ResNet50 Backbone Network. However, Deformable ResNet50 Backbone Network is deep and suitable for locating RoIs rather than classifying RoIs. Therefore, this paper uses an independent classification network to classify RoIs.

The state-of-the-art classification networks (MobileNetV3 [15], EfficentNetV2 [16], etc.) are usually deep, complex, and suitable for classifying natural images. We propose a novel classification network to classify RoIs as shown in Figure 6. The network is mainly composed of three blocks, one attention module, and two linear layers. Block1, block2, and block3 are respectively composed of two 3*3*64, two 3*3*128, and three 3*3*256 convolutional layers. These blocks extract features from RoIs, and then these features are enhanced by the attention module. Finally, these enhanced features are input into the two linear layers and output classification results.

Figure 6.

The proposed classification network BEAM.

The attention module CBAM consists of channel and spatial attention modules. The channel attention module first performs average-pooling and max-pooling operations on feature map F to extract its channel information then generates the corresponding channel attention map through a shared multi-layer perceptron (MLP). Finally, the two channel attention maps are added and the sigmoid activation function is used to generate the channel attention Mc.

The spatial attention module firstly carries out average-pooling and max-pooling on the input F along the channel dimension and then uses 7*7 convolutional layers to extract spatial context information. Finally, the sigmoid activation function is used to generate spatial attention Ms.

The CBAM combines the channel attention module and the spatial attention module in order. Its height, width, and channels of outputs are the same as the input F.

3.

RESULTS

For the detection network, 5279 images with annotated esophageal cancer lesions from 160 patients are retrieved as the training set, and 1166 images with annotated esophageal cancer lesions from 40 patients are retrieved as the testing set. The data details of detection and classification networks are shown in Table 1 and Table 2, respectively.

Table 1.

Data details of detection network

Table 2.

Data details of classification network

This research is implemented with Pytorch and GPU NIVIDA 2080 Ti with 11 GB onboard memory. The proposed classification network first trains 15 epochs on the ImageNet2012 and fine-tunes on the RoIs dataset for 50 epochs. The optimizer is Adam optimizer (learning rate=0.005, betas= (0.9, 0.999), weight decay=0). At the same time, “Step Learning Rate” is used to adjust the learning rate. Considering classification network is small and no large fluctuation in early training, “Warm up” training strategy is not used in the classification network.

4.

DISCUSSION

The high-resolution images of the barium esophagram need a deep network to extract features to better locate RoIs. However, the low-resolution RoIs need a shallow network to extract features to complete more accurate classification. The depth of the network should be proportional to the size of the dataset [17]. However, the FRDF extracts the features by the same Deformable ResNet50 Backbone Network to locate RoIs and classify RoIs. Therefore, this paper proposes a deep learning system containing a detection network as well as an independent classification network. The regression subnetwork of detection network is used to locate RoIs and classification subnetwork of detection network is used to eliminate RoIs of low confidence. Filtered RoIs are input to an independent classification network to classify RoIs. On this basis, a majority voting method is used to identify positive cases.

For detection network, we introduce the ResNet50+FPN module to better predict small lesions. At the same time, Deformable Convolutional Network is introduced into the ResNet50 backbone network to sample the deformed esophagus area adaptively. To avoid the loss of feature vectors of region proposals in the mapping process, the feature vectors of region proposals are extracted by the RoIAlign that no rounding operation is involved in the mapping process.

For classification network, we introduce CBAM to enhance the significance of the esophageal region. At the same time, compared with EfficientNetV2 and MobileNetV3, our classification network parameter amount is greatly reduced, and the performance is greatly improved.

Finally, our proposed algorithm is evaluated comprehensively at patient-level. All 40 of 40 patients with esophageal cancer and 45 of 53 patients without esophageal cancer are correctly classified. In these 1166 images of 40 patients with esophageal cancer cases, we evaluate the detection performance of the FRDF. Its AP metric is 63.30 which is 6.59 higher than that of Faster R-CNN of 56.71. The experiment shows that our proposed algorithm performs promising results on the barium esophagram dataset.

5.

CONCLUSION

The proposed detection network FRDF and an independent classification network BEAM are more suitable for detecting esophageal cancer in the barium esophagram. FRDF detects RoIs and BEAM classifies these RoIs. The proposed method may also be used for other diseases that occur in the esophagus. Nowadays, Transformer is one of the hot spots of CV research [18-20], and its excellent modeling ability makes it perform well in CV tasks. But Transformer training often requires a lot of data. Therefore, its effectiveness in the application of medical datasets remains to be studied. In the future, we will try to combine Transformer with CNN and propose a method that is more suitable for medical imaging datasets to get better performance.