Reverse attention

Overview

Reverse attention takes a prediction that is already partly right, such as a coarse mask or a fixation map, and flips it. Where the prediction is confident the flipped map is near zero. Where it is missing the object, the flipped map is high. Multiplying features by this map, or adding it as guidance, pushes the network to look at the parts the prediction has not covered [arxiv:2102.10274 §4.3] 2103.04011 §3.2.

The shared step is the reversal, 1 − sigmoid(prediction). The variants differ in how often the reversal is applied, which feature it guides, and whether the guidance is accumulated across stages. The survey that classifies attention methods puts reverse attention under "arithmetic" scores, which are built from simple operations on other scores or features, such as reversing [arxiv:2204.07756 §2.1.4].

Variants

One reverse step on a fixation map

Lv et al. predict a fixation map F that marks the region which makes a camouflaged object visible. They take 1 − F as a reverse attention and multiply it with a backbone feature before the mask decoder, so the mask branch works on the regions the fixation branch did not select 2103.04011 §3.2. The method is trained jointly with a fixation loss and a segmentation loss, with weight λ = 1 2103.04011 §3.2. Card: Simultaneously Localize, Segment and Rank the Camouflaged Objects.

Iterated reverse guidance in a coarse-to-fine cascade

Fan et al. build SINet-V2 with a neighbor connection decoder that makes a coarse map C6, then three stages that refine it. At each stage the coarser map is resampled to that level, passed through a sigmoid and reversed. The reversed map guides a group-reversal attention (GRA) block 2102.10274 §4.3 eq. 2. Each stage has three GRA blocks with channel group sizes 32, 8 and 1 2102.10274 §4.3 §5.3. The network outputs four maps and supervises all of them with a weighted IoU and BCE loss, with no extra labels beyond the masks 2102.10274 §4.4.1. Card: Concealed Object Detection.

Arithmetic form in the survey

The survey lists reverse attention among arithmetic score methods and writes a different formula for it: a sigmoid of 1 divided by (ReLU of the convolution output plus 0.125), minus 4. It says the constants are for convergence and gives no further reason 2204.07756 §2.1.4. This is not the 1 − sigmoid form used by the two method papers above.

When to use which

Implementation sketch

Two forms, in the common notation. Both use the reversal and differ in where it is applied.

# Single reverse step on a backbone feature  [arxiv:2103.04011 §3.2]
# feature: (B, C, h, w); fixation_logits: (B, 1, h, w)

def guide_once(feature, fixation_logits):
    # Reverse the fixation map: 1 - F  [arxiv:2103.04011 §3.2]
    reverse = 1 - sigmoid(fixation_logits)
    # Multiply the reversed map with the feature  [arxiv:2103.04011 §3.2]
    return feature * reverse

# One reverse-guided refinement stage  [arxiv:2102.10274 §4.3] [eq. 2]
# feature: (B, C, h, w); coarse_logits: (B, 1, h, w) from the stage above

def refine_stage(feature, coarse_logits):
    # Resample the coarser map to this resolution  [arxiv:2102.10274 §4.3]
    guide = resample(coarse_logits, feature)
    # Reverse once: 1 - sigmoid  [arxiv:2102.10274 §4.3] [eq. 2]
    reverse = 1 - sigmoid(guide)
    # Group-reversal blocks with group sizes 32, 8, 1  [arxiv:2102.10274 §4.3]
    for group_size in [32, 8, 1]:
        # Mix the reversed map into channel groups  [arxiv:2102.10274 §4.3]
        feature = feature + block(feature, reverse, group_size)
    return feature

The second sketch reverses once and reuses that map in every block, as the paper defines it. The released code accumulates the guidance across blocks instead (see Contradictions). The sketches omit the residual head and the loss.

Contradictions

• Operation. The survey gives a sigmoid of 1 over (ReLU of the convolution plus 0.125), minus 4 2204.07756 §2.1.4. The two method papers use 1 − sigmoid of the coarser or fixation map 2103.04011 §3.2 2102.10274 §4.3 eq. 2. These are different functions under the same name.
• Which feature is guided. The paper says the reversed fixation map is multiplied with the first backbone feature s1 2103.04011 §3.2. The card for that paper records that the released code guides the stage-2 feature with a maximum of a smoothed and a raw reversed map, so the paper's wording does not match the code. See its Paper vs code section.
• Accumulation across blocks. The paper defines the guidance of block i+1 as the first reversed map plus a group term, so every block uses the same reversed map 2102.10274 §4.3 eq. 3 eq. 5. The released code updates the guidance in place, so each block sees the previous block's output. The card for that paper records this under Paper vs code.
• Size of the gain. The authors report the reverse-guidance ablation as improving CAMO scores 2102.10274 §5.3 tab. 6. On COD10K, the row without reverse has a slightly higher F-measure than the full model 2102.10274 tab. 6. Neither paper reports significance testing.

Papers

• Simultaneously Localize, Segment and Rank the Camouflaged Objects: one reverse step on a fixation map, feeding a mask branch.
• Concealed Object Detection: SINet-V2, a coarse-to-fine cascade with reverse guidance and GRA blocks.
• Visual Attention Methods in Deep Learning: An In-Depth Survey: classifies reverse attention as an arithmetic attention score.