SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

In plain words

Salient object detection asks a model to find the thing in a picture that a person would look at first, and to draw its outline. It gets harder when the light is poor or the background is busy. A camera that also records depth (how far each point is) or heat (a thermal image) gives extra clues that a plain colour picture lacks.

SwinNet takes the colour picture and the second picture, and passes each through its own copy of a Swin Transformer. This is a vision model that looks at the image in small windows and shifts those windows from layer to layer, so distant parts of the image can still exchange information. At each stage the two streams are first aligned with a shared attention map, then each is rescaled by its own channel weights. A separate edge branch, built from the shallower depth features, supplies outline information to a decoder that merges the stages back to full image size. The same design is used for depth pictures and for thermal pictures.

TL;DR

SwinNet predicts a salient-object mask from an RGB image plus a depth image (RGB-D) or a thermal image (RGB-T). It uses two Swin-B encoders, a per-level alignment and re-calibration module, and an edge-guided decoder §3.1 fig. 1. The RGB-T text says every metric is best, but the table has a higher F-measure for CGFNet on VT1000 §4.3.2 tab. 2. The released code differs from the equations in several places, and it has no LICENSE file (see Paper vs code).

Problem

Single-image saliency models can miss objects in poor light or cluttered scenes. The authors argue that depth or thermal data add useful cues in those cases §1. Fusing two modalities is the hard part, because the second stream must be combined without losing the first §1.

CNN backbones gather information from local neighbourhoods and lose spatial detail through pooling, so they handle long-range context poorly §1. Swin Transformer limits self-attention to local windows and shifts the window grid between blocks so that neighbouring windows exchange information §1. Shallow decoder features carry detail but also background noise, so the outline must be guided separately §1.

Assumptions stated in the paper: a second, spatially paired modality (depth or thermal) is available at test time §3.1 §4.2. All inputs are resized to 384 × 384 §4.2.

Method

Overview. Two Swin-B streams produce hierarchical features for the RGB image and the second modality. Each level is aligned and re-calibrated, a depth-based edge feature is built, a decoder aggregates the fused levels from the deepest up, and two heads predict the saliency map and the edge map §3.1 fig. 1. The paper describes RGB-D in detail and says RGB-T is the same §3.1.

1. Encoders. Two Swin Transformers, one per modality, each with the Swin-B configuration §3.2. The patch embedding turns the colour stream into raw RGB values per patch, and the depth stream into three copies of the depth value §3.2. Each stream yields four levels, written ST_i^c and ST_i^d for i = 1 to 4 §3.2. The paper does not say which Swin stage outputs form these four levels (see Pitfalls).
2. Spatial alignment (per level i). A common spatial attention map is computed from the element-wise product of the two streams eq. 1. Its definition is a channel-wise max, then a 3 × 3 convolution, then a sigmoid eq. 2. Both streams are multiplied by this map eq. 3.
3. Channel re-calibration. Each aligned stream gets its own channel attention, a global max pool, a 1 × 1 convolution and a sigmoid eq. 4 eq. 5. Each channel attention map multiplies the original, unaligned feature of its stream eq. 6.
4. Edge feature. The three shallowest depth levels are each reduced by a 1 × 1 convolution, upsampled to the resolution of the first, and concatenated eq. 7. A 3 × 3 convolution, batch normalisation and ReLU give a block whose channel attention gates the feature, with a residual connection eq. 8 §3.4.
5. Pairwise fusion. For each level, the two modality features are combined by a sum and by an element-wise product, and the two results are concatenated eq. 9.
6. Decoder. Starting from the deepest level, each fused level is added to a 3 × 3 convolution of the upsampled decoder feature from the next deeper level. The process repeats toward level 1 eq. 10 §3.5.
7. Heads. The shallowest decoder feature is concatenated with the edge feature eq. 11. The saliency map is a 3 × 3 convolution followed by 4× upsampling eq. 15. The edge map is a 3 × 3 convolution on the edge feature, followed by 4× upsampling eq. 13.
8. Loss. The total is the edge loss plus the saliency loss eq. 12. Both are binary cross-entropy over pixels, with the edge pixels and the salient pixels separated from background eq. 14 eq. 16 §3.6. The edge ground truth is the Canny edge map of the saliency ground truth §3.6.1.

The pseudocode below is written from the equations in §3, and checked against the official code. Where the code differs, the card follows the paper and the difference is listed in Paper vs code.

# SwinNet forward pass, one pair of inputs  [fig. 1] [§3.1]
# Input:  rgb, depth: (B, 3, 384, 384); depth copied to 3 channels  [§3.2]
# Output: sal_logits, edge_logits: (B, 1, 384, 384)

def forward(rgb, depth):
    # Two Swin-B encoders, one per modality  [§3.2]
    # Index 1 is the shallowest level, 4 the deepest (inferred from eq. 10, 13)
    rgb_lv = swin_rgb(rgb)
    dep_lv = swin_dep(depth)
    # Align and re-calibrate each level  [§3.3] [eq. 1] to [eq. 6]
    for i in 1, 2, 3, 4:
        f_c[i], f_d[i] = align_recalibrate(rgb_lv[i], dep_lv[i])
    # Edge feature from the three shallowest depth levels  [§3.4] [eq. 7, 8]
    edge_feat = edge_branch(dep_lv[1], dep_lv[2], dep_lv[3])
    # Pairwise fusion of the two modalities per level  [§3.5] [eq. 9]
    for i in 1, 2, 3, 4:
        fused[i] = concat(f_d[i] + f_c[i], f_d[i] * f_c[i])
    # Top-down aggregation, deepest level first  [§3.5] [eq. 10]
    agg[4] = fused[4]
    for i in 3, 2, 1:
        agg[i] = fused[i] + conv3x3(upsample2(agg[i + 1]))
    # Combine with the edge feature  [§3.5] [eq. 11]
    f_s = concat(edge_feat, agg[1])
    # Saliency and edge heads  [§3.5] [eq. 13] [eq. 15]
    sal_logits = upsample4(conv3x3(f_s))
    edge_logits = upsample4(conv3x3(edge_feat))
    return sal_logits, edge_logits


# Spatial alignment and channel re-calibration at one level  [§3.3]
def align_recalibrate(f_rgb, f_dep):
    # Common spatial map from the element-wise product  [eq. 1]
    sa = spatial_attention(f_rgb * f_dep)          # (B, 1, H, W)
    # Both streams are weighted by the shared map  [eq. 3]
    a_rgb = sa * f_rgb
    a_dep = sa * f_dep
    # Channel weights from the aligned features  [eq. 4] [eq. 5]
    ca_rgb = channel_attention(a_rgb)              # (B, C, 1, 1)
    ca_dep = channel_attention(a_dep)              # (B, C, 1, 1)
    # Re-calibrate the original, unaligned features  [eq. 6]
    return ca_rgb * f_rgb, ca_dep * f_dep


# Spatial attention: channel-wise max, 3x3 conv, sigmoid  [eq. 2]
def spatial_attention(x):
    pooled = max_over_channels(x)                  # (B, 1, H, W)
    return sigmoid(conv3x3(pooled))


# Channel attention: global max pool, 1x1 conv, sigmoid  [eq. 5]
def channel_attention(x):
    pooled = global_max_pool(x)                    # (B, C, 1, 1)
    return sigmoid(conv1x1(pooled))


# Edge feature from the shallow depth levels  [eq. 7] [eq. 8]
def edge_branch(d1, d2, d3):
    # Reduce, upsample to level 1 resolution, and concatenate  [eq. 7]
    f_e = concat(conv1x1(d1), upsample2(conv1x1(d2)),
                 upsample4(conv1x1(d3)))
    # Channel attention on a BN-ReLU block, plus residual  [eq. 8]
    return f_e * channel_attention(bconv3x3(f_e)) + f_e


# Training loss: edge term plus saliency term  [§3.6] [eq. 12]
def loss(sal_logits, edge_logits, gt, edge_gt):
    # Binary cross-entropy with logits, per pixel  [eq. 14] [eq. 16]
    loss_edge = bce_with_logits(edge_logits, edge_gt)
    loss_sal = bce_with_logits(sal_logits, gt)
    return loss_edge + loss_sal

Equations, as numbered in the compiled paper §3.3 §3.4 §3.5 §3.6:

SA_i = SA(ST_i^c \times ST_i^d) eq. 1

SA(x) = \mathrm{Sigmoid}(\mathrm{Conv}_3(\mathrm{CGMP}(x))) eq. 2

ST1_i^c = SA_i \times ST_i^c,\quad ST1_i^d = SA_i \times ST_i^d eq. 3

CA_i^c = CA(ST1_i^c),\quad CA_i^d = CA(ST1_i^d) eq. 4

CA(x) = \mathrm{Sigmoid}(\mathrm{Conv}_1(\mathrm{GMP}(x))) eq. 5

F_i^c = CA_i^c \times ST_i^c,\quad F_i^d = CA_i^d \times ST_i^d eq. 6

F_e = \mathrm{Concat}(\mathrm{Conv}_1(ST_1^d), Up_2(\mathrm{Conv}_1(ST_2^d)), Up_4(\mathrm{Conv}_1(ST_3^d))) eq. 7

F'_e = F_e \times CA(BConv(F_e)) + F_e eq. 8

F_i = \mathrm{Concat}((F_i^d + F_i^c), (F_i^d \times F_i^c)) eq. 9

FF_i = \begin{cases} F_i + \mathrm{Conv}_3(Up_2(FF_{i+1})), & i = 1, 2, 3 \\ F_i, & i = 4 \end{cases} eq. 10

F_s = \mathrm{Concat}(F'_e, FF_1) eq. 11

L = L_e(S_e) + L_s(S) eq. 12

S_e = Up_4(\mathrm{Conv}_3(F'_e)) eq. 13

L_e(S_e) = -\sum_{j \in Z_+} \log Pr(y_j = 1 \mid S_e) - \sum_{j \in Z_-} \log Pr(y_j = 0 \mid S_e) eq. 14

S = Up_4(\mathrm{Conv}_3(F_s)) eq. 15

L_s(S) = -\sum_{j \in Y_+} \log Pr(y_j = 1 \mid S) - \sum_{j \in Y_-} \log Pr(y_j = 0 \mid S) eq. 16

Here CGMP is global max pooling along the channel axis, GMP is global max pooling over space, Up_x is x-times upsampling, and BConv is a 3 × 3 convolution with batch normalisation and ReLU §3.3 §3.4 §3.5. Z+ and Z- are edge and non-edge pixels, and Y+ and Y- are salient and non-salient pixels §3.6.

Hyperparameters

Values from the paper, with the official code's value where it differs. The code is at the commit in the front matter.

Results

Author-reported. Setup: RGB-D training uses 2,185 pairs (1,485 from NJU2K and 700 from NLPR), or 2,985 pairs when DUT is included, and the rest are for testing §4.1.1. RGB-T training uses 2,500 pairs from VT5000 §4.1.1. Metrics are S-measure, adaptive F-measure, adaptive E-measure and MAE §4.1.2. Baselines are listed in §4.3. The RGB-D table below shows VST only, the transformer baseline the paper singles out §4.3.1. The full comparison is in tab. 1.

RGB-D, SwinNet against VST tab. 1:

RGB-T, SwinNet against CGFNet, the last baseline column tab. 2. The best baseline differs by metric, so the full table should be used for ranking:

Backbone ablation, S-measure, RGB-D tab. 3. Each row replaces the Swin-B backbone §4.4.1:

Other ablations, all RGB-D, reported by the paper tab. 4 tab. 5 tab. 6. Replacing the alignment module with the depth-enhanced module of BBS-Net gives NLPR S .936 against .941 for the full model tab. 4. Removing edge guidance gives NLPR S .938 against .941 tab. 5. On NLPR, RGB alone gives S .932 and depth alone .896 tab. 6. On STERE, RGB alone (S .923) is above RGB+depth (S .919), and the paper attributes this to low-quality depth maps §4.4.4 tab. 6.

Complexity tab. 7 §4.4.5: SwinNet has 198.7 M parameters and about 124.3 GFLOPs. Removing the alignment module changes the parameter count by 0.4 M (198.3 M), and removing the edge-guided decoder gives 173.6 M and 88.9 GFLOPs. The paper says most of the cost is in the two Swin backbones.

Paper vs code

Repository. The abstract links github.com/liuzywen/SwinNet. The owner matches the local part of the first author's e-mail address in the author block, so it is treated as the authors' own. The code in the clone has no LICENSE file, so nothing from it is quoted, and it is cited by path and line only. The clone is at the commit in the front matter. The README points to the journal version (IEEE TCSVT, 2022) and to a revision PDF in the repository, which this card did not read. The card compares against the arXiv v1 text.

Discrepancies, found by reading models/Swin_Transformer.py, data.py, SwinNet_train.py, SwinNet_test.py, utils.py, options.py and gen_edge.py:

• Batch size. Paper: 3 §4.2. Code default: 8 options.py.
• Epoch count. Paper: convergence within 200 epochs §4.2. Code default: 300, with a loop of 299 epochs options.py SwinNet_train.py.
• Edge feature. Paper: a residual channel-attention block on a 3 × 3 BN-ReLU of the concatenated feature, with no extra convolutions eq. 7 eq. 8. Code: a 3 × 3 convolution and ReLU after each reduction, then a residual channel-attention block over the 96-channel concatenation, then a 3 × 3 classifier to one channel Swin_Transformer.py. The 32-channel edge feature is then a 3 × 3 BN-ReLU of that one-channel map Swin_Transformer.py Swin_Transformer.py. Its residual sits inside the block (res += x), not in the form of eq. 8.
• Channel attention. Paper: one 1 × 1 convolution eq. 5. Code: two 1 × 1 convolutions with reduction 16 and a ReLU between them Swin_Transformer.py.
• Spatial attention. Paper: 3 × 3 convolution eq. 2. Code: 7 × 7 by default, which fuse_enhance uses Swin_Transformer.py Swin_Transformer.py. Two further spatial-attention modules are built for the RGB and depth streams but are not called in forward Swin_Transformer.py Swin_Transformer.py.
• Alignment. No discrepancy: the code multiplies the unaligned input by the channel weight, as eq. 6 says Swin_Transformer.py.
• Decoder aggregation. Paper: each level adds a 3 × 3 convolution of the upsampled deeper decoder feature eq. 10. Code: each level adds the deeper decoder output to its own fused feature first, then upsamples and applies a 3 × 3 convolution, batch normalisation and ReLU Swin_Transformer.py. The deepest level is also convolved, from 2048 to 1024 channels Swin_Transformer.py Swin_Transformer.py.
• Output heads. Paper: convolution then 4× upsampling eq. 13 eq. 15. Code: the saliency head applies a ReLU to the concatenation, upsamples 4×, then convolves Swin_Transformer.py. The edge head upsamples then convolves Swin_Transformer.py.
• Loss reduction. Paper: sums over pixels eq. 14 eq. 16. Code: BCEWithLogitsLoss with its default mean reduction for both terms SwinNet_train.py SwinNet_train.py. A weighted variant, bce2d_new, is defined but commented out SwinNet_train.py SwinNet_train.py.
• Edge ground truth. Paper: Canny edges, no thresholds §3.6.1. Code: a separate script computes Canny(30, 100) on the GT files, and training does not run it gen_edge.py. The edge files are read as they are data.py.
• Depth input. The code feeds depth unnormalised, one channel copied to three, and its comment says normalised depth would improve results data.py data.py SwinNet_train.py. RGB is ImageNet-normalised data.py.
• Augmentation. Paper: flips, rotations and border clipping §4.2. Code adds colour jitter data.py data.py and salt-and-pepper noise on the GT, about 0.15% of pixels set to 0 or 255 data.py data.py.
• Gradient clipping. Not stated in the paper. Code clamps each gradient element to ±0.5 utils.py options.py.
• Checkpoint selection. Not stated in the paper. Code saves a checkpoint every 5 epochs SwinNet_train.py and keeps the one with lowest MAE on the validation root given by --test_*_root, whose default is ./RGBD_train/val/ SwinNet_train.py options.py.
• Test-time normalisation. Not stated in the paper. Code min-max normalises each output map SwinNet_test.py.
• Unused stage. forward_features returns five levels: the patch embedding, the outputs of the first three stages after patch merging, and the normalised output of the fourth stage Swin_Transformer.py. SwinNet.forward uses only the first four entries Swin_Transformer.py. So the fourth stage is computed but unused, and its parameters receive no gradient. This is read from the code and was not run.
• RGB-T. The repository has training and test code only for RGB-D. The commented-out block in the test script reuses the same model with a T (thermal) folder in place of depth SwinNet_test.py. The RGB-T weights are linked from the README [README]. The RGB-T schedule is the same as the RGB-D schedule in the paper §4.2.

Pitfalls

• Feature levels. The paper names four levels ST_1 to ST_4 but does not say which Swin outputs they are §3.2. The code uses the patch embedding (128 channels, stride 4) and the outputs of the first three stages after patch merging (256, 512 and 1024 channels, strides 8, 16 and 32) Swin_Transformer.py Swin_Transformer.py. An implementation that takes the four stage outputs will not match the released weights.
• Unused stage. The fourth Swin stage and the final norm are dead code in the released model (see Paper vs code). The parameter count of 198.7 M includes them tab. 7 §4.4.5. Whether they count depends on the implementation.
• Reproduction target. Where the text and code differ, decide first which one you are reproducing: the text eq. 2 eq. 5 eq. 8 eq. 10, or the released weights Swin_Transformer.py. Mixing the two gives a model that matches neither.
• Loss scale. Paper sums; code means. Both terms are weighted equally, so the balance between edge and saliency pixels differs from the formula as written eq. 14 eq. 16 SwinNet_train.py.
• Soft targets. Canny output is 0 or 255, and the GT and edge images are resized with transforms.Resize at its default interpolation, which is bilinear for PIL images. The targets are therefore soft at boundaries and not re-binarised data.py. The paper does not say whether the targets are binarised after resizing.
• Random GT noise. The salt-and-pepper noise on the training GT changes the labels on about 0.15% of pixels in every training crop data.py. It is absent from the paper §4.2.
• Schedule off-by-one. The first decay applies to epoch 100 of a counter that starts at 1 utils.py SwinNet_train.py. An implementation whose counter starts at 0 decays one epoch later than the code does.
• Model selection. The paper does not say how the reported checkpoint was chosen. The code picks by MAE on the validation folder given on the command line SwinNet_train.py. If that folder is the benchmark test set, reported results are optimistic. The card cannot tell which set the authors used.
• Claim versus table. The RGB-T text says all metrics are best §4.3.2. Table 2 has VT1000 F-measure at .896 for SwinNet against .906 for CGFNet tab. 2.
• Compute. About 10 FPS is reported with I/O and preprocessing included, on hardware not stated for that figure §4.4.5. Training used one RTX 2080 Ti for about 26 hours §4.2.
• Split. The paper does not state the test split for VT821 and VT1000, only that the rest of VT5000 is for testing §4.1.1.

Lineage

• Builds on: Swin Transformer for the encoder, with its shifted-window design §1 §3.2. The edge idea follows the paper's reading that depth contrast marks object contours §3.4 (no card). Visual Saliency Transformer is the closest transformer SOD baseline §2.1 tab. 1 (no card).
• Compared against: D3Net for RGB-D tab. 1; VST, MMNet, CGFNet, MIDD, ECFFNet and others tab. 1 tab. 2 (cards for these were not checked).
• Superseded by: no entry recorded in this wiki.