Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

In plain words

Salient object detection asks a model to find the object in a photo that people would look at first, and to draw its outline. Most such models only see the colour image. Phones now also record a depth map (how far each pixel is from the camera), which could help, but a depth map from a phone can be messy, and using a bad one can make the result worse than ignoring it.

The paper's model, D3Net, runs three small networks side by side: one looks only at the colour image, one looks at colour plus depth, and one looks only at depth. At test time it compares the answer from colour-plus-depth with the answer from depth alone. If the two agree, the depth map is probably good, and the colour-plus-depth answer is used. If they disagree, the depth map is probably bad, and the colour-only answer is used instead.

The authors also collected a new set of photos of people taken with a phone, and benchmarked many earlier models on it and on several older datasets. The point is to test these methods on everyday phone photos, not only on the datasets the field already used.

TL;DR

• D3Net is a simple baseline for RGB-D salient object detection, built from three VGG-16 feature-pyramid streams and a test-time gate on the depth map §4.
• The gate (the depth depurator unit) keeps the RGB-D prediction only when it matches the depth-only prediction, and otherwise uses the RGB prediction §4.2 eq. 3.
• The paper adds the SIP dataset of phone-captured images with people §3 and a benchmark of 18 models on seven datasets §5.1 tab. 4.
• Author-reported: D3Net has the best S-measure on six of seven test sets; on LFSD, CPFP is ahead tab. 4.
• Code is released, but the S-measure and E-measure toolbox is not in the repository, and the training data must be fetched separately.

Problem

Most salient object detection models use only RGB images and ignore depth, which is available on modern phones §1. The paper argues that depth maps are often low quality: in low-quality maps the scene is cluttered, while good maps show a clear object with closed boundaries and a double-peaked depth histogram §4.2 fig. 7. The fusion therefore has to tolerate bad depth, not just use good depth. The paper also says that a large, common benchmark for the field has long been missing [abstract].

Method

Overview

D3Net has two parts that form a nested structure: a three-stream feature learning module (FLM) and a depth depurator unit (DDU) §4 fig. 5. The three streams are RgbNet (RGB input), RgbdNet (RGB and depth stacked as four channels) and DepthNet (depth only) §4.1 RgbdNet.py DepthNet.py. Each stream runs on a 224 by 224 resized image §4.1.

Feature learning module (each stream)

The backbone is VGG-16 with five convolution stages, giving feature maps at 64, 128, 256, 512 and 512 channels at decreasing resolution §4.1. The paper adds a sixth, coarser level built with two 3 by 3 convolutions §4.1. The features are merged top-down, from coarse to fine: the coarse map is upsampled, the finer map is reduced by a 1 by 1 convolution, and the two are concatenated §4.1 fig. 6.

# Single stream: VGG-16 levels plus a coarser sixth level  [§4.1]
# Input:  image (B, C, 224, 224), C = 3 (RGB), 4 (RGB-D) or 1 (depth)
# Output: feature map (B, 32, 224, 224)

def stream(image):
    # Five VGG-16 stages: 64x224x224, 128x112x112, 256x56x56,
    # 512x28x28 and 512x14x14  [§4.1]
    levels = vgg16_stages(image)
    # Sixth, coarser level from the last stage (7x7, 32 channels)  [§4.1]
    # Pooling and convolution sizes: [code: model/RgbdNet.py#L47-L53]
    levels.append(sixth_level(levels[4]))
    # Merge from the coarsest level down to the finest  [§4.1] [fig. 6]
    merged = levels[5]
    for level in [4, 3, 2, 1, 0]:
        merged = decode(merged, levels[level])
    return merged

# Merge one coarse map into the next finer one  [§4.1] [fig. 6]
# Input:  coarse (B, 32, h, w), fine (B, C, H, W) with H = 2h
# Output: (B, 32, H, W)

def decode(coarse, fine):
    # Upsample the coarse map to the finer size. The paper says nearest
    # neighbour; the code uses bilinear with align_corners=True  [§4.1]
    # [code: model/RgbdNet.py#L36]
    upsampled = upsample(coarse, size=fine.size)
    # Reduce the finer map to 32 channels with a 1x1 convolution and ReLU
    # (the paper mentions the 1x1 reduction; the ReLU is code only)
    # [§4.1] [code: model/RgbdNet.py#L22-L26]
    reduced = relu(conv1x1(fine, out_channels=32))
    # Concatenate the two maps (64 channels)  [§4.1] [fig. 6]
    merged = concatenate(upsampled, reduced)
    # Two 3x3 convolutions with ReLU: not described in the paper; code only
    # [code: model/RgbdNet.py#L27-L32]
    merged = relu(conv3x3(merged, out_channels=32))
    return relu(conv3x3(merged, out_channels=32))

# Prediction head: one saliency map per stream  [code: model/RgbdNet.py#L77-L95]
# Input:  features (B, 32, H, W); output: saliency (B, 1, H, W) in [0, 1]
# The paper does not describe this head; the code is the only source.

def predict(features):
    # Two 3x3 convolutions with ReLU  [code: model/RgbdNet.py#L80-L85]
    hidden = relu(conv3x3(features, out_channels=32))
    hidden = relu(conv3x3(hidden, out_channels=32))
    # One-channel 1x1 convolution and sigmoid  [code: model/RgbdNet.py#L86-L89]
    return sigmoid(conv1x1(hidden, out_channels=1))

Depth depurator unit (test time only)

During training the three streams are not combined. At test time the DDU takes the three saliency maps S_rgb, S_rgbd and S_depth, and outputs the RGB-D map if its mean absolute difference from the depth-only map is at most a threshold t, otherwise the RGB map §4.2 eq. 1 eq. 2 eq. 3. The paper sets t = 0.15 after trying several values §4.3.

# Depth depurator unit at test time  [§4.2] [eq. 2] [eq. 3]
# Input:  s_rgb, s_rgbd, s_depth: saliency maps (H, W) in [0, 1]
# Output: final map (H, W)

def depth_depurator(s_rgb, s_rgbd, s_depth):
    # Mean absolute difference between the two maps  [§4.2] [eq. 2]
    # Paper: per-image distance delta with threshold t = 0.15  [§4.3]
    distance = mean(abs(s_rgbd - s_depth))
    # Gate: 1 keeps the RGB-D map, 0 keeps the RGB map  [§4.2] [eq. 3]
    # Paper uses distance <= t; the code uses strict < [code: eval.py#L76-L77]
    keep_rgbd = 1.0 if distance < 0.15 else 0.0
    return keep_rgbd * s_rgbd + (1.0 - keep_rgbd) * s_rgb

Training

Each stream is trained on its own with a pixel-wise binary cross entropy averaged over all pixels §4.3 eq. 4. The training set is 1,485 NJU2K and 700 NLPR image pairs, following the settings of CPFP §4.3. Augmentation is a horizontal flip §4.3. The code trains each stream in a separate run and never trains the three together train.py [README: lines 85-87].

# Train one stream  [§4.3] [eq. 4]
# Input:  loader yields (images, gt) with gt in {0, 1}^(224x224)
# Each stream is trained in its own run  [code: train.py#L38]

def train_epoch(stream_model, loader, optimizer):
    for images, gt in loader:
        # Predicted saliency map in [0, 1]  [code: model/RgbdNet.py#L86-L89]
        saliency = stream_model(images)
        # Binary cross entropy, mean over all pixels  [eq. 4]
        # [code: model/RgbdNet.py#L139-L149]
        loss = binary_cross_entropy_mean(saliency, gt)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Hyperparameters

Results

Author-reported. Setup: seven test sets (NJU2K test, NLPR test, STERE, DES, SSD, LFSD, SIP), 18 models, each scored on S-measure, max F-measure, max E-measure and MAE §5.1 §5.2 tab. 4. D3Net is trained on NJU2K and NLPR training pairs only §4.3. The table below gives D3Net and CPFP (the best published method in the paper's comparison §5.5) for all 28 cells; the other 16 models are in tab. 4 and are not copied here.

The paper's ablation on S-measure compares the three streams and the DDU bounds. "Lower bound" and "Upper bound" take, per image, the worst and best of the RGB and RGB-D maps, so they are oracle values, not methods §7.1 tab. 5.

Source: tab. 5. The paper's text notes that the RgbdNet stream alone beats the RGB stream on most sets, and that the DDU helps on STERE, DES, NJU2K and NLPR, but not on SSD §7.1.

Paper vs code

Official repository: https://github.com/DengPingFan/D3NetBenchmark. The paper links it in the abstract, so it counts as the authors' code. The commit is the head of the default branch at clone time; the README says it was updated on 2022-06-09 [README: line 112]. The repository has no LICENSE file, so its code is not licensed for reuse. This card cites code by path and line but does not quote it.

• Joint training: the abstract says the parts are "learned jointly" [abstract]. The code trains RgbNet, RgbdNet and DepthNet in three separate runs, and no joint loss exists train.py [README: lines 85-87]. Only the DDU is applied at test time, and it is applied after the three nets are trained.
• Upsampling: the paper says nearest-neighbour upsampling §4.1. The code uses bilinear interpolation with align_corners=True RgbdNet.py.
• Depth input: the paper gives no channel count for depth. The code feeds RgbdNet four channels and DepthNet a one-channel depth map repeated to three channels RgbdNet.py DepthNet.py.
• DDU comparison: the paper computes the distance on maps resized to 224 x 224 §4.2. The code resizes the three maps to the ground-truth size first and computes the mean absolute difference there eval.py.
• DDU boundary: the paper's equation uses distance at most t eq. 2. The code uses strict less-than eval.py.
• Checkpoint selection: the code keeps the checkpoint with the lowest MAE on the NJU2K test split, NJU2K_TEST, which is also one of the reported test sets train.py train.py utils.py. The paper does not say how the checkpoint was chosen §4.3.
• F-measure: the code uses beta squared 0.3 utils.py. The paper does not state beta.
• Learning-rate schedule and Adam betas: the paper does not state either. The code uses a constant rate and betas (0.9, 0.99) train.py utils.py.
• Readout head and decoder convolutions: not described in the paper; present in the code RgbdNet.py RgbdNet.py.
• Multi-level outputs: the paper describes a feature pyramid at 11 levels §4.1. The code keeps only the finest decoder output for the prediction RgbdNet.py.
• Anchors: the paper says it defines anchors on the levels §4.1. The code has no anchor mechanism, and the paper does not explain what the anchors are.
• Metric toolbox: the S-measure and E-measure code is not in the repository. The README links a separate toolbox [README: line 112]. The repository computes only MAE and max F-measure utils.py.
• Dataset counts: the README reports 17 methods, 91K images, and a book-cover application at 20 fps [README: lines 28-29]. The paper's text reports 18 models and about 97K images §5, and its abstract reports 32 models and 65 fps.
• Training data: the repository refers to dataset folders that are not included. The training list is on the authors' website, which the paper cites §4.3.

No discrepancies found in the loss, the input resolution, the optimizer family, the learning rate, the batch size, the epoch count, or the horizontal flip augmentation.

Pitfalls

• Joint training: the paper's central claim is that the parts are learned jointly, but the released code trains them separately. An agent that follows the paper's wording will build a different system from the one that produced the numbers.
• Depth source for SIP: the contributions say the depth maps are captured by a real smartphone §1.1, while the dataset section says the depth map is automatically estimated by the phone §3.2. Whichever is true, SIP depth is not a dedicated sensor's output, and the DDU's premise (depth quality varies) depends on that depth.
• Threshold choice: t = 0.15 was picked after trying six values, and the paper does not say on which data §4.3. How the gate behaves on other data is not reported.
• Test-set selection: the released training code picks the checkpoint on the NJU2K test split train.py. Reported NJU2K numbers may therefore be optimistic. Use a separate validation split if you need unbiased numbers.
• Metric reproduction: S-measure and E-measure values cannot be reproduced from the repository, because the toolbox is distributed separately [README: line 112]. Max F-measure uses beta squared 0.3 in the code.
• Speed: the inference script runs all three streams for every image eval.py. The paper's 0.015 s per image does not say whether all three streams are counted §5.1.
• "Exceeds any prior contender": the abstract says D3Net exceeds all prior work on all five metrics [abstract]. In the table, CPFP has a higher S-measure, max F-measure, max E-measure and lower MAE than D3Net on LFSD tab. 4. The text also reports margins of 1.0 to 5.8 percent over CPFP on six datasets and 1.4 percent on SIP §5.5. The absolute differences on SIP are 0.010 (S), 0.010 (max F), 0.006 (max E) and 0.001 (MAE) tab. 4. The paper does not say whether 1.4 percent is relative or what it measures.
• Aspect ratio: every image is resized to a square 224 x 224 input §4.1 train.py. Predictions are resized back with bilinear interpolation eval.py. The paper does not say how outputs are resized for scoring.
• Memory: three VGG-16 streams run at test time, and the paper says so as a limitation §7.2. The paper suggests MobileNet V2 or ESPNet V2 as lighter backbones but does not test them.
• Pretrained weights: training needs the ImageNet VGG-16 file, which the repository does not contain RgbdNet.py [README: line 83].
• Data size: the SIP dataset is small compared with most RGB salient object detection datasets §7.2. The 929 images in the text are the count to use, not the 1K in the abstract §3.1 §1.1.
• The paper's "anchors" wording §4.1 has no counterpart in the code. Implement the decoder as in the code, or ask the authors.

Lineage

• Builds on: feature pyramid networks, which the paper follows as a template §4.1 (not on the wiki); VGG-16 as backbone §4.1 (not on the wiki); CPFP's training settings §4.3 (not on the wiki).
• Image selection follows Fan et al. 2018 §3.1 (not on the wiki).
• Compared against: 17 other methods, including CPFP, TANet, PCF and MMCI tab. 4; CPFP is named the best published method at the time §5.5.
• Superseded by: not stated in the paper or repository.