EGNet: Edge Guidance Network for Salient Object Detection

In plain words

A salient-object model marks the most noticeable thing in a photo. Many such models produce blobby masks, because the coarse, deep features that say "this is the object" lose the fine outline of the object.

EGNet adds a second job for the network. Besides predicting the object, it also predicts where the object's outline is. The outline is learned from low-level, detailed features, and it is helped by the global sense of where the object is. The outline features are then mixed into each scale of the object prediction, so each scale gets a hint about where the edges are.

The four predictions from the different scales are then combined into one map, and that map is the output. The authors report that this gives sharper outlines and better object placement than earlier methods, and that it does so without a separate refinement step afterwards.

TL;DR

• Two paths share one backbone: a top-down salient-object path over side features C3 to C6, and a salient-edge path built from C2 plus top-level location §3.2 eq. 2 eq. 3.
• The edge features F_E are added to each object-scale feature, the object features are predicted again at four scales, and the four maps are fused into the output §3.3 eq. 7 eq. 9.
• Training uses deep supervision on every side output and a cross-entropy edge loss. The paper gives a learning rate, decay schedule and epoch count §4.1, but does not name the optimizer.
• Author-reported: max F-measure 0.893 on DUTS-TE with ResNet and 0.880 with VGG; the paper states it beats the methods it compares with and uses no CRF or superpixel post-processing tab. 2 §4.4.

Problem

Salient object detection gives a per-pixel map of the visually distinctive object. CNN methods that use only high-level features locate objects well but blur the boundaries; the paper says the boundary is not modelled explicitly, and that the complementarity between salient edges and salient objects has not been used §1 §3.1. Earlier fixes either fuse more scales or add post-processing such as CRF or superpixels, which the paper says are slow §1.

Method

Overview §3. The network has three steps: (1) build multi-resolution object features with a top-down U-Net-style path; (2) build salient-edge features from the Conv2-2 side features plus top-level location; (3) guide each object scale with the edge features, predict at each scale, and fuse the four predictions §3.2 §3.3.

Backbone §3.2. The network is independent of the backbone. The paper uses VGG, truncating its last three fully connected layers, and attaches a side path to the last pooling layer, giving six side features Conv1-2 to Conv6-3. Conv1-2 is dropped (too close to the input and too small a receptive field), leaving C2 to C6 §3.2. The ResNet variant is named only as "ResNet" in the paper §4.1.

Salient-object path §3.2.1. Each side path C^(i) runs through three convolution-plus-ReLU layers, called T in the paper and listed per side in Table 1 tab. 1. The path is top-down: the feature at side i is the enhanced sum of C^(i) and the upsampled feature from side i+1 eq. 3. The top side has no input from above eq. 3. A deep-supervision head D converts each enhanced feature to one channel tab. 1.

Salient-edge path §3.2.2. The edge branch starts from C2, which the paper chooses because Conv2-2 keeps edge information better §3.2. The top feature F̂^(6) is passed through a convolution (Trans) and ReLU, upsampled to C2's size, and added to C2 eq. 2. The sum is enhanced with the same T design, giving the salient-edge features F_E. A separate edge loss supervises F_E eq. 4. The paper says Trans is a convolution that changes the channel count and does not state its kernel size §3.2.2.

Object supervision §3.2.2. Each object side output i = 3 to 6 is supervised with a cross-entropy loss on the salient-region pixels and the non-salient pixels eq. 5. The total loss of this stage is the edge loss plus these four losses eq. 6.

One-to-one guidance §3.3. Four sub-side paths are added, one per object scale i = 3 to 6. Each adds F_E to the upsampled object feature F̂^(i) eq. 7, then runs its own enhancement T' and one-channel head D'. Each sub-side output has its own loss eq. 8. The four sub-side maps are fused by a weighted sum with weights β_i, and the fused map has its own loss eq. 9. The paper does not state the β_i values or how they are learned §3.3.

Inference §4.1. Only the fused prediction map is used as the output saliency map. The paper's edge map is not used at test time §4.1.

# EGNet forward pass  [§3.2] [§3.3]
# Input:  image (3, H, W)
# Output: edge_map (1, H, W); saliency_map (1, H, W)

# up_t(x, ref) = bilinear_resize(ReLU(Trans(x)), size of ref)  [§3.2.2] [eq. 2]
# enhance(x) = three conv + ReLU layers T                 [§3.2.1] [tab. 1]

def egnet_forward(image):
    # 1. Backbone side features; the Conv1-2 side path is dropped  [§3.2]
    c2, c3, c4, c5, c6 = backbone(image)    # C2: edge source; C3..C6: object

    # 2. Top-down salient-object path  [§3.2.1] [eq. 3]
    f6 = enhance(c6)
    f5 = enhance(c5 + up_t(f6, c5))
    f4 = enhance(c4 + up_t(f5, c4))
    f3 = enhance(c3 + up_t(f4, c3))

    # 3. Salient-edge path: local C2 plus top-level location  [§3.2.2] [eq. 2]
    c2_bar = c2 + up_t(f6, c2)
    f_edge = enhance(c2_bar)                 # salient edge features F_E
    edge_map = upsample(head(f_edge), image) # head = D, 3x3 conv to 1 channel

    # 4. One-to-one guidance: each object scale receives F_E  [§3.3] [eq. 7]
    guided_maps = []
    for f_i in (f3, f4, f5, f6):
        g_i = up_t(f_i, f_edge) + f_edge
        g_hat = enhance_prime(g_i)           # T', see Paper vs code
        guided_maps.append(upsample(head_prime(g_hat), image))

    # 5. Fuse the four guided maps; beta_i is not stated  [§3.3] [eq. 9]
    saliency_map = sum(beta[i] * guided_maps[i] for i in range(4))
    return edge_map, saliency_map

# Training loss  [eq. 4] [eq. 5] [eq. 6] [eq. 8] [eq. 9] [eq. 10]
# Inputs: edge_gt (H, W) in {0, 1}; sal_gt (H, W) in {0, 1}
# bce_sum sums the per-pixel cross-entropy over all pixels, as in the equations
# All loss weights are 1  [§4.1]

def training_loss(edge_map, side_maps, guided_maps, fused, edge_gt, sal_gt):
    # 1. Edge loss on F_E: positives Z+, negatives Z-  [eq. 4]
    loss = bce_sum(edge_map, edge_gt)
    # 2. Object side outputs i = 3..6  [eq. 5] [eq. 6]
    for side in side_maps:
        loss = loss + bce_sum(side, sal_gt)
    # 3. Each guided sub-side output i = 3..6  [eq. 8]
    for guided in guided_maps:
        loss = loss + bce_sum(guided, sal_gt)
    # 4. Fused map supervised with the object ground truth  [eq. 9]
    loss = loss + bce_sum(fused, sal_gt)
    return loss                          # L_t = L + L'  [eq. 10]

Key equations, transcribed from the source:

\bar{C}^{(2)} = C^{(2)} + \mathit{Up}\big(\phi(\mathit{Trans}(\hat{F}^{(6)};\theta)); C^{(2)}\big) eq. 2

\hat{F}^{(i)} = f\big(C^{(i)} + \mathit{UpT}(\hat{F}^{(i+1)};\theta, C^{(i)});\, W_T^{(i)}\big) eq. 3

\mathcal{L}^{(2)}(F_E; W_D^{(2)}) = -\sum_{j\in Z_+}\log Pr(y_j=1 \mid F_E) - \sum_{j\in Z_-}\log Pr(y_j=0 \mid F_E) eq. 4

\mathcal{L}^{(i)}(\hat{F}^{(i)}; W_D^{(i)}) = -\sum_{j\in Y_+}\log Pr(y_j=1 \mid \hat{F}^{(i)}) - \sum_{j\in Y_-}\log Pr(y_j=0 \mid \hat{F}^{(i)}), \quad i\in[3,6] eq. 5

\mathbb{L} = \mathcal{L}^{(2)}(F_E; W_D^{(2)}) + \sum_{i=3}^{6}\mathcal{L}^{(i)}(\hat{F}^{(i)}; W_D^{(i)}) eq. 6

G^{(i)} = \mathit{UpT}(\hat{F}^{(i)};\theta, F_E) + F_E, \quad i\in[3,6] eq. 7

\mathcal{L}'_f(\hat{G}; W_{D'}) = \sigma\Big(Y, \sum_{i=3}^{6}\beta_i\, f(\hat{G}^{(i)}; W_{D'}^{(i)})\Big) eq. 9

\mathbb{L}' = \mathcal{L}'_f + \sum_{i=3}^{6}\mathcal{L}^{(i)'}(\hat{G}^{(i)}; W_{D'}^{(i)}), \qquad \mathbb{L}_t = \mathbb{L} + \mathbb{L}' eq. 10

Eq. 8 has the same form as eq. 5, with the sub-side maps Ĝ^(i) in place of F̂^(i) §3.3.

Hyperparameters

Table 1 gives the enhancement and head layers for each side output. Both columns list kernel size, padding and channels for each of the three convolutions T1, T2, T3, and the head D (converts to one channel) tab. 1.

Caption of Table 1: T denotes the feature enhance module, each with three convolutions and three ReLU layers; D is the transition to one channel tab. 1.

Results

Author-reported. Setup: trained on DUTS-TR; tested on ECSSD, PASCAL-S, DUT-OMRON, HKU-IS, SOD and DUTS-TE; metrics are max F-measure (β² = 0.3), MAE and S-measure §4.1 §4.2. Baselines are 15 methods §4.4. Methods marked * use pre- or post-processing tab. 2. The table's "-" marks a dataset a method was trained on, so DHS is not evaluated on DUT-OMRON tab. 2. The source has a typo ("0852") in the RFCN ECSSD S-measure, read here as 0.852.

Max F-measure tab. 2:

MAE, lower is better tab. 2:

S-measure, higher is better tab. 2:

Author statements §4.4: the ResNet model improves on the strongest prior method (PiCANet*) by 1.9 points of average F-measure over six datasets; the text reports larger gains on SOD and DUTS. The paper's text says the numbers are reached "without any pre-processing and post-processing" §4.4.

Ablation, trained on DUTS-TR, evaluated on SOD and DUTS-TE §4.3 tab. 3:

Row 3 minus row 1 gives the F-measure gain of 3.1 (SOD) and 2.4 (DUTS-TE) points that the text reports §4.3.2. Edge comparison against NLDF's edges, author-computed §4.3.3 tab. 4:

No runtime or memory numbers appear in the active text. The inference-time table is commented out in the source §4.4.

Paper vs code

Code: official repository https://github.com/JXingZhao/EGNet, cloned at commit 5e1b933eb022d64c3fdfbe15070fb32b625ea864. The paper's abstract and author note give mmcheng.net/egnet, whose page links this repository (abstract; project page) README.md. The repository has no LICENSE file, so the code licence is unverified; no code is quoted here.

• Edge loss is class-balanced in the code, plain in the paper. Paper eq. 4 is an unweighted cross-entropy eq. 4. The code uses a weighted BCE where edge pixels get the fraction of negatives and background pixels get 1.1 times the fraction of positives solver.py solver.py. The comment at solver.py says the opposite of the code.
• Optimizer. The paper gives momentum 0.9 and weight decay 5e-4 but no optimizer §4.1. The code uses Adam with weight decay 5e-4 and creates a new Adam after each LR decay solver.py solver.py. Momentum is defined but not used solver.py.
• Schedule. The paper: 24 epochs, one decay after 15 §4.1. The code: default 30 epochs run.py, decays at epochs 15 and 24 solver.py solver.py.
• Default backbone. The code selects ResNet solver.py; the learning-rate comment gives 2e-5 for VGG solver.py. The paper reports 5e-5 for both §4.1. A VGG model is built in the same file model.py, but the paper gives no VGG settings beyond the shared ones.
• Trans runs only when channels drop. Eq. 3 applies Trans at every step eq. 3. The code applies its 1x1 transition only when the lower feature has fewer channels than the upper one, and adds the upper feature directly otherwise model.py.
• Guidance fusion is feature-level. Eq. 9 sums per-scale prediction maps with weights β_i eq. 9. The code adds the four guided features with a ReLU after each addition, then applies a 5x5 and a 3x3 convolution to get one map model.py model.py. The four per-scale maps are still supervised separately model.py solver.py.
• Guidance channel count. Table 1 gives 256 to 512 channels for the object side paths S3 to S6 tab. 1. The one-to-one module uses 128 channels for all four T' and heads model.py model.py model.py. Whether T' follows Table 1 is not stated.
• ResNet adaptation. The paper does not describe how ResNet features are mapped to the same channel counts. The code uses 1x1 conv + ReLU layers to 128 to 512 channels model.py model.py, and the ResNet depth is 50 in the code model.py, not stated in the paper.
• Initialisation. Paper: truncated normal, σ = 0.01 §4.1. Code: plain normal, σ = 0.01, zero biases, applied to every convolution before the pretrained backbone is loaded model.py solver.py solver.py.
• Backbone weights. The paper does not name its pretrained checkpoints. The code expects local files at hard-coded author-machine paths run.py.
• Augmentation. The paper states none §4.1. The code uses random horizontal flips only and no resize or crop; test images are kept at native size dataset.py dataset.py dataset.py.
• Upsampling. The code uses bilinear interpolation with align_corners=True model.py. The paper does not state the mode.
• Edge labels. The paper does not say how edge labels are made. The released script takes the gradient of the binary mask and marks non-zero gradient pixels sal2edge.m sal2edge.m. As released, its loop runs over only the first ten images sal2edge.m.
• Output. Test-time output is the sigmoid of the fused map, saved at native size solver.py solver.py. This matches §4.1.
• Trans kernel. The paper leaves Trans's kernel size open §3.2.2. The code uses 1x1 model.py model.py.

No discrepancy found in: the three-layer T design and the deep-supervision head D model.py model.py; the choice of F̂^(6) as the edge source model.py; batch size one with gradient accumulation over ten run.py solver.py.

Pitfalls

• Ambiguous optimizer and schedule. Momentum and weight decay are given without an optimizer. An implementer following only the paper will not know whether to use SGD with momentum or Adam. The released code uses Adam, so its numbers depend on that choice §4.1 solver.py.
• Unstated fusion weights. β_i in eq. 9 are not given §3.3. The code replaces the weighted sum with a feature-level fusion, so eq. 9 alone does not reproduce it model.py.
• Loss normalisation. Eq. 4 is written as a sum over pixels, not a mean. The code divides summed losses by (10 × batch size) solver.py solver.py. The paper does not state its normalisation, so the loss scale relative to the learning rate is unverified.
• Class-balanced edge loss in the code. The code's edge weights differ from eq. 4, and its comment gives the wrong direction solver.py. The paper does not say whether its reported results used the balanced loss or the plain one of eq. 4 §4.3.
• Edge-label generation not in the paper. Labels depend on the released script's gradient rule and its 10-image loop sal2edge.m. A faithful implementation needs its own edge-label procedure.
• Input details missing from the paper. Input size, mean subtraction and augmentation are not stated §4.1. The code uses native size and BGR mean subtraction dataset.py and a horizontal flip only.
• Hidden cost claim. The paper says the top-down path adds "no additional time and space consumption" §4.3.2, but no timing or memory numbers appear in the active text, and the timing table is commented out §4.4.
• Ablation caption typo. Table 3's caption names "edge_TDLF"; the rows use edge_TDLP tab. 3.
• Ablation baseline. "B" is described only as the U-Net architecture with PSFEM-style multi-scale fusion, with the side path S2 removed §4.3.1.
• Licence. No LICENSE file in the repository, so no code is quoted in this card; it paraphrases the code only.

Lineage

• Builds on: U-Net-style top-down fusion §3.2 (named as [ronneberger2015u] in the source; no card in this wiki); DSS-style short connections from HED §3.2 (no card); NLDF's IoU-based edge penalty, which the paper compares against in its ablation §4.3.3 (no card).
• Compared against: 15 methods, listed in §4.4 and tab. 2.
• Superseded by: not stated in this wiki.