2.1

Overview

As shown in Figure 1, the network architecture takes polyp images as input. It extracts multi-scale long-range dependency features laterally from the PVT encoder, generating feature maps X_j ∈ ℝ^{C_i×H′×W′} (j ∈ {1,2,3,4})with the spatial resolution of H / 4 × W / 4, H / 8 × W / 8, H / 16 × W / 16, H / 32 × W / 32, respectively, where C_i ∈{64,128,320,512}. These features are progressively aggregated across stages via SFM to capture richer semantic information, resulting in the fused features Γ₁ and the output polyp prediction mask M₁. The purpose of FEM is to use channel and spatial attention to enhance local detail information for better polyp localization and to obtain enhanced low-level feature Γ₂. Meanwhile, GAM is employed to align high-level and low-level features Γ₁ and Γ₂, outputting the polyp prediction mask M₂. Finally, we optimize the model by using Loss₁ and Loss₂.

Figure 1.

Illustration of CFA-PVT architecture

2.2

Stage bridging module

As shown in Figure 1, the network uses the Pyramid Vision Transformer PVTv2[9] as the encoder. PVTv2 is a variant of PVT[8]. Given an input image I ∈ ℝ^C×H×W, we first adjust the channels of the high-level features X₂, X₃, and X₄ from three stages to 32 through 1×1 convolutions. Subsequently, X₄ is upsampled by 2x and 4x, followed by a sequence of 1x1-3x3-1x1 residual convolutions, then concatenated with X₂ and X₃, respectively. These feature maps are then simply fused. After the above operations, the final feature map Γ₁ is obtained. Furthermore, by using CFM to integrate features, we aim to exploit local and global contextual information fully. This top-down multi-stage bridging feature fusion approach effectively guides the network to explore missing parts and details of objects.

2.3

Feature enhancement module

Typically, low-level features contain rich details such as textures and edges, but since they are close to the input, they also include a lot of redundant information. Inspired by[12], as shown in Figure 2, we use channel and spatial attention mechanisms to enhance global and local information interaction, refine the features, and filter out irrelevant features. This allows the model to focus more on important channels and spatial locations, thereby accurately capturing the details of polyps.

Figure 2.

Illustration of feature enhancement module

Subsequently, the channel information of the feature maps is aggregated using average pooling, max pooling, and soft pooling[13]. Each aggregated feature map undergoes channel attention enhancement individually through a shared MLP network. The formula for channel attention map (denoted as M_c) can be obtained by:

where σ(·) is sigmoid function and η denotes the MLP operation. P_avg (·), P_max (·), P_soft (·) represent avgpool, maxpool, and softpool, respectively. Next, concatenate the values from the three pooling operations along the channel dimension. Finally, apply a 7x7 convolutional layer followed by a sigmoid function to output a spatial attention map M_s.

where f^7×7 denotes the convolution operation with a filter size of 7x7, while , , represent the maximum value, average value, and exponentially weighted value obtained along the channel dimension.

2.4

Global adaptive module

This module takes two inputs: low-level features Γ₂ from FEM and high-level features Γ₁ from CFM, which are channel-wise concatenated. Inspired by[14], the module simultaneously captures spatial non-local contextual features and global correlation features across different channels. According to Figure 3, attention weights are initially obtained by applying a 1x1 convolution W_k and softmax function. Subsequently, feature linear transformation is achieved with a 1x1 convolution W_v that includes a reduction ratio, significantly reducing the parameter count. This is followed by a sequence of (layer normalization + ReLU) layers, and finally, features are fused through addition.

Figure 3.

Illustration of global adaptive module

2.5

Loss function

The total loss consists of the main loss Loss₂, which predicts the final segmentation result, and the Loss₁ supervises local information, trained jointly. This paper employs a combined loss comprising weighted binary cross-entropy loss and the weighted intersection over union loss . The total loss can be represented as:

3.1

Implementation details

The experimental environment is established on the Pytorch framework, using PVTv2[9] as the encoder backbone network and the Adam optimizer, with an initial learning rate set to 1×10^-4. All experiments are conducted on an NVIDIA A6000 GPU (48 GB VRAM) with a batch size of 16. All experiments are trained for 100 epochs, with all input images set to 352×352 and utilizing a multi-scale training strategy of {0.75, 1, 1.25}.

We use three prominent polyp datasets, Kvasir-SEG[15], CVC-ClinicDB[16]^, and CVC-ColonDB[17], to evaluate the model’s performance. Table 1 summarizes the details of these datasets and their respective data splits. Notably, ColonDB is an unseen dataset used exclusively for evaluation purposes.

Table 1.

Details of the datasets, where ‘n/a’ indicates that the dataset is not available.

To quantitatively evaluate the performance of our proposed method, we employed four widely used metrics, including mean Dice (mDice), mean Intersection over Union (mIoU), weighted F-measure , and S-measure (S_α)[18] as accuracy evaluation indicators.

3.2

Comparisons with State-of-the-Art Methods

To comprehensively evaluate the effectiveness of our model, we compared it against four SOTA methods, including U-Net[3], UNet++[4], PraNet[19], and Polyp-PVT[10]. All of these predictions were directly provided by the authors. We used the Kvasir dataset to assess the model’s learning capability and the CVC-ColonDB dataset to evaluate its generalization ability. The experimental findings reveal that our proposed approach exhibited the highest performance levels across all evaluated metrics.

The quantitative analysis results are listed in Table 2. Specifically, it attained a mDice score of 0.924, an mIoU score of 0.872, a score of 0.917, and a S_α score of 0.930 on the Kvasir dataset. On the ClonoDB dataset, we achieved a mDice score of 0.812, an mIoU score of 0.733, a of 0.796, and a S_α of 0.869. As shown in Table 2, CFA-PVT achieved more significant improvements over the SOTA methods on both the Kvasir and ClonoDB datasets. This demonstrates that our model has strong learning and generalization capabilities.

Table 2.

Segmentation results of different methods. The best result is highlighted in bold and underlined fonts

Figure 4 depicts qualitative analysis results, illustrating the segmentation outcomes of our network and other methods across various polyp datasets. Polyp segmentation poses challenges, particularly with polyps of varying shapes and sizes, including those obscured by folds in the colon. Nevertheless, CFA-PVT demonstrates commendable segmentation performance, producing prediction maps that closely approximate the ground truth and exhibit relative completeness.

Figure 4.

Visualizations on Kvasir (top three rows) and ClonoDB (bottom three rows).

3.3

Ablation study

To further explore the impact of each component on the outcomes, we tested each component of CFA-PVT on the Kvasir dataset in this section, aiming for a deeper understanding of our model. PVTv2 is used as the baseline for ablation experiments, with components gradually removed, and the experimental results are listed in Table 3.

Table 3.

Ablation study results on Kvasir

After removing the CFM, the performance of the model dropped significantly. The mDice decreased from 0.924 to 0.911, and the mIoU decreased from 0.872 to 0.850, indicating that the CFM effectively perceives the overall region and extracts rich semantic information. Removing the FEM resulted in a 0.4% and 0.3% reduction in mDice and mIoU metrics, respectively. This demonstrates that it focuses selectively on crucial channel responses while suppressing irrelevant and redundant information. Removing the GAM caused a 0.7% and 0.6% reduction in mDice and mIoU metrics, respectively, validating the GAM ability of the model for global polyp localization. The above results fully demonstrate the effective contribution of these modules to the final prediction.