RGBT Salient Object Detection: A Large-scale Dataset and Benchmark

In plain words

Salient object detection means finding the thing in a picture that a person would look at first. Most systems use ordinary colour photos, and those fail when it is dark or when the object looks like its background. A thermal camera records heat instead of light, so it can still show the object in the dark. The authors argue that the field lacks a large set of matched colour and thermal images with careful outlines, which is what deep networks need.

They build that set, called VT5000. Its colour and thermal images line up without manual alignment, every object has a hand-made outline, and each picture is tagged with the kinds of difficulty it contains, such as small objects, low light or clutter.

They also give a baseline network. It runs one feature branch per camera, uses attention to pick useful parts of each branch, adds the two branches level by level, and then builds the final map with a pyramid-style module and an aggregation decoder. A boundary term in the loss pushes the outline to be sharper. In the paper's comparisons it scores best on F-measure, though not on every number.

TL;DR

• VT5000: 5000 aligned RGB-thermal pairs with pixel masks and 11 challenge tags. The paper trains on 2500 of them §3 §5.1.
• ADFNet: two VGG16 streams with CBAM on each of five levels, summed level by level, then a PPM and FAM decoder, trained with cross entropy plus a Laplacian edge loss §4.1-§4.6.
• Training as written: Adam, 25 epochs, learning rate 1e-4 with weight decay 5e-4, inputs resized to 400x400 §5.1. The released code differs in several places (see Paper vs code).
• Author-reported: max F 0.863 and MAE 0.049 on the VT5000 test split for the full model tab. 4. ADFNet has the best F-measure on all three test sets against the ten compared methods §5.2 fig. 8.

Problem

Most salient object work uses RGB alone, and it degrades in dark scenes, cluttered backgrounds, and when object and background share a colour §1. A thermal infrared camera records heat radiated from the object surface, so it adds information that RGB lacks §1. The paper says RGBT methods were limited by small or weak data, because deep networks need more training data than the existing sets provide: VT821 has misalignment and black padding from manual alignment, and VT1000 has many easy samples §1 §3.4. Earlier RGBT methods were traditional, merged the two modalities without a learned fusion, and left boundaries blurred §1.

Assumptions: the two images of a pair must be spatially aligned. VT5000 uses two cameras with the same imaging parameters so that no manual alignment is needed §3.1. The paper lists alignment-free RGBT detection as future work §8.

Method

The paper contributes a dataset (Section 3) and a baseline network, ADFNet (Section 4). ADFNet has two VGG16 streams, one for RGB and one for thermal. Each stream's five levels are refined by a convolutional block attention module (CBAM) §4.2 fig. 6. The two streams are summed after every level §4.3 eq. 3. The deepest fused map is passed to a pyramid pooling module (PPM) §4.4, and feature aggregation modules (FAM) decode the levels from coarse to fine, each with the global guidance §4.5. The loss is cross entropy plus an edge term on Laplacian-filtered maps §4.6.

Dataset construction §3

1. Capture with two FLIR cameras (T640 and T610), one thermal and one CCD, with the same imaging parameters §3.1.
2. More than 5500 pairs were collected, and 5500 were selected to be as different from each other as possible. Six viewers marked the salient objects they saw first; 5000 pairs with consistent selections were kept §3.2.
3. Pixel masks were drawn by hand in Adobe Photoshop §3.2.
4. Each pair is tagged with any of eleven difficulties §3.3:
   • BSO and SSO: the object covers more than 0.26 or less than 0.05 of the image's pixels;
   • MSO: more than one salient object;
   • LI: captured on cloudy days or at night;
   • CB: the object sits far from the image centre;
   • CIB: the object is cut by the image border;
   • SA: the object has a colour close to its surroundings;
   • TC: the object has a temperature close to its surroundings;
   • IC: the surroundings are complex or cluttered;
   • OF: the whole image is blurred;
   • BW: captured in rain or bad weather. Each pair also has per-modality tags for poor RGB or poor thermal quality §3.3.
5. Split: 2500 VT5000 pairs for training; the remaining 2500 VT5000 pairs, VT821 and VT1000 for testing §5.1. The paper does not say which 2500 pairs are for training.

ADFNet §4

Overview: the backbone is VGG16 initialised from pre-trained weights. Each stream gives five feature levels at 1, 1/2, 1/4, 1/8 and 1/16 of the input size §4.1 §4.2.

Attention (CBAM, applied to each branch at each level) §4.2: channel attention pools each map with average and max pooling, passes both through shared 1x1 convolutions, sums them and applies a sigmoid eq. 1. Spatial attention then pools across channels with average and max, applies one convolution and a sigmoid eq. 2.

M^{C_R}_{i} = \big(\sigma(\mathrm{Conv}(\mathrm{AvgPool}(X^{R}_{i})) + \mathrm{Conv}(\mathrm{MaxPool}(X^{R}_{i})))\big) * X^{R}_{i} eq. 1

M^{S_R}_{i} = \big(\sigma(f^{k*k}([\mathrm{AvgPool}(M^{C}_{i}), \mathrm{MaxPool}(M^{C}_{i})]))\big) * M^{C_R}_{i} eq. 2

Multi-level fusion §4.3: the first level is the sum of the two attended branches. Each later level is a convolution block on the previous fused map, plus the two attended branches at that level.

F_{i} = \begin{cases} M^{S_R}_{i} + M^{S_T}_{i}, & i = 1 \\ \mathrm{Conv}(F_{i-1}) + M^{S_R}_{i} + M^{S_T}_{i}, & i = 2, 3, 4, 5 \end{cases} eq. 3

Pyramid pooling §4.4: the PPM takes the deepest fused map and uses four branches: the identity, global average pooling, and adaptive average pooling to 3x3 and 5x5. Its output is upsampled and passed to every level as global guidance §4.4 §4.5. How the four branches are combined is not stated.

Feature aggregation §4.5: each FAM has four branches, one identity and three average-pooling paths whose outputs are upsampled and merged, followed by a 3x3 convolution. Each FAM takes the coarser decoder output, doubled in size, the fused feature of its level and the guidance for that level §4.5.

Loss §4.6

Cross entropy over pixels §4.6.1 eq. 4 and an edge loss §4.6.2 eq. 5 eq. 6 eq. 7. The total is the unweighted sum eq. 8.

L_{C} = \sum_{i=0}^{\mathrm{size}(Y)} \big(Y_i \log P_i + (1 - Y_i)\log(1 - P_i)\big) eq. 4

\Delta f = \frac{\partial^{2} f}{\partial x^{2}} + \frac{\partial^{2} f}{\partial y^{2}} eq. 5

\Delta \tilde{f} = \mathrm{abs}(\tanh(\mathrm{conv}(f, K_{\mathrm{laplace}}))) eq. 6

L_{E} = -\sum_{i=0}^{\mathrm{size}(Y)} \big(\Delta Y_i \log \Delta P_i + (1 - \Delta Y_i)\log(1 - \Delta P_i)\big) eq. 7

L_{S} = L_{C} + L_{E} eq. 8

Here Y is the ground-truth mask, P the predicted saliency map, and the edge maps are the Laplacian responses of Y and P. The Laplacian kernel K_laplace is not given §4.6.2.

Forward pass, as described in the text. The output head is not described in the paper text; see Paper vs code.

# ADFNet forward  [§4.1] [§4.2] [§4.3] [§4.4] [§4.5]
# Input:  rgb, thermal (B, 3, H, W)
# Output: saliency map (B, 1, H, W)

def adfnet(rgb, thermal):
    # Five levels per stream, CBAM on each level  [§4.1] [§4.2]
    rgb_levels = vgg16_with_cbam(rgb)
    thermal_levels = vgg16_with_cbam(thermal)

    # Level 1: sum of the two streams  [§4.3] [eq. 3]
    fused = rgb_levels[0] + thermal_levels[0]
    fused_levels = []
    # Levels 2 to 5: conv block, then add both streams  [§4.3] [eq. 3]
    for level in range(1, 5):
        fused = conv_block[level](fused)
        fused = fused + rgb_levels[level] + thermal_levels[level]
        fused_levels.append(fused)

    # Global guidance from the deepest fused map  [§4.4]
    guidance = pyramid_pooling(fused_levels[-1])

    # Decoder: start at 1/16, move to 1/2  [§4.5]
    # guidance_at[level]: guidance resized to that level  [§4.4] [§4.5]
    merged = fused_levels[-1] + guidance
    for level in range(2, -1, -1):
        merged = feature_aggregation[level](
            merged, fused_levels[level], guidance_at[level]
        )
    # Last aggregation step, no skip input  [count and form: see Paper vs code]
    merged = final_aggregation(merged)

    # Output head, then upsample to the input size  [§5.1] (head: not stated)
    return upsample(output_head(merged), input_size)

# Training loss  [§4.6] [eq. 4] [eq. 5] [eq. 6] [eq. 7] [eq. 8]
# Input:  logits (B, 1, H, W), mask (B, 1, H, W) with values 0 or 1
# Output: scalar loss

def training_loss(logits, mask):
    # Predicted saliency probabilities  [§4.6.1]
    probs = sigmoid(logits)
    # Pixel cross entropy, summed over pixels  [§4.6.1] [eq. 4]
    # Printed without a minus sign; standard sign used here
    loss_ce = -sum(mask * log(probs) + (1 - mask) * log(1 - probs))
    # Edge maps: Laplacian, absolute value, tanh  [§4.6.2] [eq. 6]
    edge_mask = abs(tanh(laplace(mask)))
    edge_probs = abs(tanh(laplace(probs)))
    # Cross entropy on the edge maps  [§4.6.2] [eq. 7]
    loss_edge = -sum(
        edge_mask * log(edge_probs)
        + (1 - edge_mask) * log(1 - edge_probs)
    )
    # Unweighted total  [§4.6.2] [eq. 8]
    return loss_ce + loss_edge

Hyperparameters

Paper settings are in Section 5.1; the paper has no appendix. "not stated" means neither the paper text nor its LaTeX source gives a value.

Results

Author-reported. Setup: ADFNet with VGG16 backbone, trained on VT5000 training pairs; baselines were run by the authors with their published code on the same training and test splits, with no post-processing §5.2. A single run is reported §5.1.

Dataset statistics tab. 1: the diagonal gives the number of VT5000 pairs with each challenge: BSO 1746, CB 1176, CIB 1134, IC 1096, LI 535, MSO 491, OF 326, SSO 340, SA 401, TC 858, BW 222, RGB 401, T 702.

Baselines tab. 2:

F-measure per challenge on VT5000 tab. 3: the paper marks ADFNet best per row; it ties CPD on CIB (0.860).

Ablation on VT5000 test tab. 4:

Time cost tab. 5. Times are seconds for one training epoch on 2500 pairs and for the 2500-pair test pass:

The paper's per-sample figures (about 0.095 s for CBAM, 0.079 s for PPM and 0.051 s for FAM in training) match these totals divided by 2500 §6.

RGBD methods on RGBT data tab. 6. The text says these methods were trained on the VT5000 "testing set", which is probably a typo (see Pitfalls):

RGBD datasets, ADFNet trained on NJU2K and NLPR tab. 7:

Text claims (author-reported, ADFNet minus the other method):

• Ablation: adding CBAM alone gives +2.1 F and -0.5 MAE over the base network; edge loss alone lowers F (0.825 against 0.836) §6 tab. 4.
• Against PoolNet, the baseline it extends, F is higher by 5.5 to 9.9 points across the three test sets §5.2. The percentages in the text match the absolute differences in tab. 3 (for SA and SSO).
• Against CPD: VT821 +1.8 F and -0.2 MAE; VT1000 +0.9 F and -0.3 MAE; VT5000 +1.6 F and +0.1 MAE §5.2.
• Against PiCANet: VT821 +1.5 F and +0.1 MAE; VT1000 +5.9 F and -2.6 MAE; VT5000 +7.5 F and -1.8 MAE §5.2.
• Against EGNet on VT5000: +2.4 F; the MAE sentence is ambiguous (Pitfalls) §5.2.
• Against the RGBD methods on RGBT data, ADFNet is best on VT5000 for both F and MAE tab. 6. On VT821 it is behind S2MA on F (0.804 against 0.812). On VT1000 it ties S2MA on F (0.923) and is behind S2MA and AFNet on MAE.
• The 10 baselines' F and MAE numbers on the three RGBT test sets are only printed inside fig. 8 and are not reproduced here.

Paper vs code

Code: official repository linked from the abstract, path Code/ADF/ (commit a38d411). The repository has no LICENSE file, so the code is cited by path and line only. The repository ships training and test scripts, but not the VT5000 data, the VGG16 weights or trained models; those are on a Baidu Pan link in the README [Readme.md].

Discrepancies, each with both citations:

1. Learning-rate decay. Paper: 1e-4, reduced to 1e-5 after epoch 20 §5.1. Code: lr_decay_epoch = [], so no decay happens config.py. The decay branch that would multiply the rate by 0.1 exists [code: Code/ADF/solver.py#L76-L78] but is never reached.
2. Loss weighting. Paper: total = cross entropy + edge, unweighted §4.6.2 eq. 8. Code: 0.7 times cross entropy plus 0.3 times edge loss [code: Code/ADF/solver.py#L57].
3. Edge operator. Paper: absolute value of tanh of the Laplacian §4.6.2 eq. 6. Code: absolute value of the Laplacian divided by 9, with no tanh LaplaceEdge.py. The tanh version exists in the file but is commented out LaplaceEdge.py. The code's Laplacian is the 8-neighbour kernel LaplaceEdge.py. The paper does not state its kernel.
4. Input resolution. Paper: resize to 400 x 400 §5.1. Code: no resize; the full image is read and normalised dataset.py. At test time the output is upsampled to the input size [code: Code/ADF/APMFnet.py#L170].
5. Batch size. Paper: not stated §5.1. Code: batch 1 with gradient accumulation over 10 steps, so the effective batch is 10 [code: Code/ADF/config.py#L20] config.py [code: Code/ADF/solver.py#L63-L67].
6. Mean subtraction. Paper: not mentioned. Code: subtracts a per-channel mean for each modality dataset.py [code: Code/ADF/dataset.py#L71-L73].
7. Decoder skip features. Paper: F_i in eq. 3 includes the two streams' attention outputs at level i. Code: the decoder receives the convolution output before the two streams are added at that level [code: Code/ADF/APMFnet.py#L63-L67]. Only the PPM input uses the post-addition map APMFnet.py APMFnet.py.
8. Spatial attention. Paper: one k x k convolution then sigmoid eq. 2. Code: two 3x3 convolutions with ReLU between, then sigmoid [code: Code/ADF/vgg.py#L9-L14]. Channel attention: paper two 1x1 convolutions eq. 1; code uses a bottleneck to half the channels with ReLU [code: Code/ADF/vgg.py#L19-L23].
9. PPM and FAM internals. Paper describes branches only. Code adds a 1x1 convolution per PPM branch, concatenates, then a 3x3 convolution with ReLU APMFnet.py; the guidance is projected with a 3x3 convolution per level APMFnet.py. Each FAM pooling path has a 3x3 convolution APMFnet.py and the merged output is fused with another 3x3 convolution [code: Code/ADF/APMFnet.py#L111].
10. Output head. Paper: not described in the text. Code: 1x1 convolution to one channel APMFnet.py. The model returns logits; the sigmoid is applied in the loss and at test time [code: Code/ADF/solver.py#L51] solver.py.
11. Initialisation. Paper: random except the VGG16 backbone §5.1. Code: all convolutions are drawn from N(0, 0.01) APMFnet.py and then the VGG16 weights are loaded into both streams and into the fusion network's blocks 2 to 5 solver.py.
12. Evaluation thresholds. Paper: 0 to 255 §5.1. Code: 100 thresholds in [0, 1] get_PRF_mat.py. Max F is taken from the curve of per-image mean precision and recall [code: Code/ADF/get_PRF_mat.py#L84-L90].
13. Evaluation normalisation and binarisation. The code min-max normalises each map and each ground truth to [0, 1] get_PRF_mat.py get_PRF_mat.py get_PRF_mat.py. Ground truth is binarised at 0.5 for PR and for MAE [code: Code/ADF/get_PRF_mat.py#L12] get_PRF_mat.py. Eq. 10 does not binarise the ground truth eq. 10.
14. Output format. Test writes 255 * sigmoid(pred) as PNG [code: Code/ADF/solver.py#L99-L101].
15. Seed. Paper: fixed seeds, value not stated §5.1. Code: seed 1185 for random, numpy and torch run.py.

No appendix exists in the paper's source; all settings are in Section 5.1.

Pitfalls

• Loss sign. Eq. 4 is printed without the minus sign that cross entropy needs eq. 4. The code uses the standard negative sum solver.py. The summation index also starts at i = 0 and N is defined but never used.
• Sum versus mean. The code sums the loss over pixels and divides only by the batch and accumulation counts solver.py [code: Code/ADF/solver.py#L57]. A mean over pixels would change the gradient scale by the pixel count. Keep the sum if you use lr = 1e-4 as in the code.
• Edge values. Without tanh (the code), the edge map is bounded by 8/9 LaplaceEdge.py, so the binary cross entropy stays in its valid range. With tanh (paper eq. 6), values lie in [0, 1] by construction.
• Unstated details an implementer must choose: batch size, Adam betas, flip probability, channel-attention width, spatial-attention kernel, Laplacian kernel, PPM combination, output head, and the VGG16 weight source. Each choice above is the code's, not the paper's.
• Which 2500 pairs train the model, and whether the released data matches the paper's split, are not stated. The repository contains no split list [Readme.md].
• Dataset terms. The paper says the dataset will be released for "free academic usage" §1. The repository has no LICENSE, and the data is distributed through Baidu Pan links [Readme.md]. The paper's CC BY 4.0 license covers the paper, not the dataset. Do not redistribute the data, and do not treat it as licensed for commercial use.
• Single run. The paper reports one run and says repeated runs are not bit-identical because of floating-point error §5.1. No variance across runs is reported.
• Inconsistent numbers. The full model is 0.049 MAE in tab. 4 and 0.048 MAE for ADFNet on VT5000 in tab. 6; the F-measure is 0.863 in both.
• Baseline name. The same method is called SGDL in Section 2, Table 2 and the saliency-map caption §2 tab. 2 fig. 9, and SCGL in Table 3 and the PR caption tab. 3 fig. 8.
• "Basic network". The ablation calls the no-CBAM, no-edge base "PoolNet" §6, but Section 4 describes a two-stream network. The paper does not show that the base is the PoolNet model rather than a version of ADFNet.
• RGBD training sentence. Section 7.1 says the RGBD methods were trained on "our VT5000 testing set" §7.1. The rest of the paper trains on the training split only, so this is probably a typo for the training set. Check before relying on tab. 6.
• EGNet MAE sentence. Section 5.2 says EGNet's MAE is "0.1% lower than our method" on VT1000 §5.2. The sentence can be read two ways, so the MAE comparison with EGNet is unclear.
• Percent units. The text gives gains as percentages §5.2. For SA and SSO against PoolNet they equal absolute F-measure differences in tab. 3, so read them as points. The paper does not say which unit it uses elsewhere.
• Figure-only numbers. The F and MAE values for the ten baselines on VT821 and VT1000 appear only inside fig. 8.
• Cost. Relative to the row without CBAM, the full model's training epoch is 982.65 s against 744.63 s (about 32 percent more), and its test pass is 383.30 s against 229.01 s (about 67 percent more) tab. 5. The paper gives no timing hardware beyond the Titan Xp used for training §5.1.
• Evaluation script. PR_Curve and MAE_Value resize a 2-D map with curGT.shape[2], which raises an error when the map and ground truth differ in size get_PRF_mat.py get_PRF_mat.py. Resize the prediction to the ground-truth size before evaluating.

Lineage

• Builds on: VGG16 for the backbone §4.1; CBAM for attention §4.2; PoolNet (arXiv 1904.09569) for the PPM and FAM modules §4.4 §4.5. The ablation adds CBAM to PoolNet §6.
• Prior RGBT data and methods: VT821 with MTMR §2.1; VT1000 with SGDL (arXiv 1905.06741) §2.1.
• Compared against: RAS, PiCANet, R3Net, MTMR, SGDL, PFA, CPD, PoolNet, BASNet, EGNet tab. 2. EGNet has a card: EGNet. RGBD methods DMRA, MMCI, AFNet, A2dele and S2MA on RGBT data tab. 6.
• Superseded by: not stated in the wiki.