Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation

In plain words

To train a system that colours every pixel of a photo by object, you normally need someone to outline each object. Cheaper labels exist: you only need to know which objects appear somewhere in the picture. A standard trick turns such a classifier into rough pixel masks by asking which parts of the image made it decide "dog" or "sofa". Those rough masks are then used as training targets for a proper segmentation model.

The trouble is that the classifier was trained to say "yes" to every object present, so the masks for one object often spill onto a neighbouring object that usually shows up with it, and they can miss parts of an object that the classifier never needed. This paper's fix is to add a second, separate training stage. For each object in the picture, the rough mask is used to keep only the features that belong to that object. A second small classifier then has to tell these isolated pieces apart, which forces the features to separate the objects. The masks are then read out again from the improved model. The method is cheap to bolt onto existing mask generators, and the authors show gains on two standard benchmarks.

TL;DR

ReCAM takes a converged multi-label classifier trained with binary cross-entropy (BCE), keeps its FC layer (FC1), and adds a second FC layer (FC2) trained with softmax cross-entropy (SCE) on single-class features. Each single-class feature is the backbone feature map multiplied by that class's normalised CAM and then pooled §4.1 eq. 5–7. Training minimises L_BCE + λ·L_SCE eq. 8; new CAMs from FC1 and FC2 weights are the seeds [eq. 9]. Author-reported on PASCAL VOC 2012 seeds: CAM 48.8 → ReCAM 54.8 mIoU; with IRN refinement 66.3 → 70.9 tab. 2. Use it if you generate pseudo masks from CAMs for weakly-supervised segmentation with image-level labels; the official code is MIT-licensed and PyTorch-based, and the two-stage training is cheap compared with AdvCAM.

Problem

With image-level labels only, the pseudo masks come from class activation maps (CAM) of a multi-label classifier §3 §1. The authors trace the poor masks to the BCE loss: its sigmoid outputs are not mutually exclusive, so a pixel can be active for several co-occurring classes (false positive pixels), and the activation of one class can be split with others (false negative pixels) §1. Their toy experiment on single-label MS COCO images reports that a softmax (SCE) model gives better CAMs than a BCE model with the same classifier quality on the 80-class setting, and a clearer gap on 5 hoofed animal classes §1 fig. 1.

Directly replacing BCE with SCE is not valid for multi-label images, since the class probabilities of an image are not independent §1 §5.2. The method therefore keeps BCE and only adds SCE on isolated features. Assumptions: image-level labels only; the backbone is ImageNet pre-trained ResNet-50 §5.1.

Method

Stage 1: CAM model. Train a ResNet-50 backbone with global average pooling (GAP) and FC1 under BCE §3 eq. 1 eq. 4. The paper states that FC1 is trained "with the same setting as in" IRN §5.1. This is the converged model that ReCAM then reactivates §4.1.

Stage 2: ReCAM training. For each training image x with multi-hot label y:

1. Compute the feature map f(x) before GAP and the CAM of each class k §4.1 eq. 2. The CAM is ReLU of the FC1 response, divided by its maximum, so it lies in [0, 1].
2. For each class k that is present, form a single-label feature f_k by multiplying every channel of f(x) by the CAM M_k (element-wise), then pool it §4.1 eq. 5 eq. 6.
3. Feed each f_k to FC2 and apply SCE with the class label k. The loss averages over the positive classes of the image, normalised by the number of positive labels §4.1 eq. 7.
4. Minimise L_ReCAM = L_BCE + λ·L_SCE with gradients updating the backbone §4.1 eq. 8. FC1 keeps receiving BCE gradients, because FC1 is needed to produce the updated masks §4.1.

Seed extraction. Once trained, compute ReCAM for each image as in eq. 2, but with the classification weights w'' §4.1 eq. 9. The paper allows four choices for w'': FC1 weights w, FC2 weights w', their element-wise sum, or their element-wise product. The product is reported for VOC and the FC1 weights for MS COCO tab. 1 §5.2.

Optional refinement (not part of ReCAM itself, used as a comparison and in combination):

• AdvCAM: adversarial climbing on the image, with a regularisation mask built from the ReCAM (not max-normalised) §4.1 eq. 10. Its hyperparameters are taken from the original AdvCAM paper §5.1.
• IRN: an inter-pixel relation network estimates class boundaries, and a random walk over a transition matrix refines the masks §4.1 eq. 11. IRN's training details are omitted in this paper §4.1.

Segmentation. The pseudo masks train a fully supervised segmenter with the standard per-pixel cross-entropy §3 eq. 3. Models: DeepLabV2 and DeepLabV3+ with ResNet-101, and UperNet with Swin Transformer §3 §5.1. The paper notes that the method itself was not applied to transformer classifiers §3 footnote.

Why SCE on isolated features. The softmax gradient on a logit is σ_softmax(z) − y, whereas the sigmoid BCE gradient is (σ_sigmoid(z) − y)/K eq. 12. In the two-class case the paper argues that SCE moves the positive and the negative logit together when they are confused, while BCE moves them one at a time §4.2 eq. 13 fig. 4. The authors also monitor the gradient magnitudes empirically on a 5-hoofed-animal toy problem §4.2.

Training step (the algorithm in the supplement is the same two-stage procedure app. H; the step below is the stage-2 loss with citations):

# One stage-2 training step  [§4.1] [eq. 1] [eq. 2] [eq. 5-8]
# Input:  images (B, 3, H, W); labels (B, K) multi-hot
# Output: scalar loss (BCE + lam * SCE)

def stage2_loss(images, labels, backbone, fc1, fc2, lam):
    # Backbone feature map before pooling  [§4.1]
    features = backbone(images)                    # (B, C, h, w)
    # Pooled FC1 logits for the BCE term  [eq. 4]
    logits = fc1(global_average_pool(features))    # (B, K)
    loss_bce = bce(logits, labels)                 # [eq. 1]
    # CAM of each class, divided by its per-image maximum  [eq. 2]
    masks = relu(conv2d(features, fc1.weight))     # (B, K, h, w)
    masks = masks / max_per_class(masks)
    # Mask-weighted features, pooled per class  [eq. 5] [eq. 6]
    class_features = mean_hw(masks * features)     # (B, K, C)
    # SCE on positive classes only, normalised per image  [eq. 7]
    loss_sce = sce_on_positive_classes(fc2(class_features), labels)
    return loss_bce + lam * loss_sce               # [eq. 8]

Equations (numbering as in the compiled paper):

\mathcal{L}_{bce}=-\frac{1}{K} \sum_{k=1}^{K} y[k] \log \sigma(z[k])+(1-y[k]) \log [1-\sigma(z[k])] eq. 1

\operatorname{CAM}_k(\bm{x})=\frac{\operatorname{ReLU}(\bm{A}_k)}{\max(\operatorname{ReLU}(\bm{A}_k))}, \quad \bm{A}_k=\mathbf{w}_{k}^{\top} f(\bm{x}) eq. 2

\mathcal{L}_{ss}=-\frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} \frac{1}{K+1} \sum_{k=1}^{K+1} y_{i,j}[k] \log \frac{\exp (z_{i,j}[k])}{\sum_{k} \exp (z_{i,j}[k])} eq. 3

\bm{z} = \operatorname{FC}_1(\operatorname{GAP}(f(\bm{x}))) eq. 4

f_k^c(\bm{x}) = \bm{M}_k \otimes f^c(\bm{x}) eq. 5

\bm{z}_{k}' = \operatorname{FC}_2(\operatorname{GAP}(f_k(\bm{x}))) eq. 6

\mathcal{L}_{sce}=-\frac{1}{\sum_{i=1}^{K} \bm{y}[i]} \sum_{k=1}^{K} \bm{y}[k] \log \frac{\exp (\bm{z}_{k}'[k])}{\sum_{j} \exp (\bm{z}_{k}'[j])} eq. 7

\mathcal{L}_{ReCAM}= \mathcal{L}_{bce} + \lambda \mathcal{L}_{sce} eq. 8

\operatorname{ReCAM}_k(\bm{x})=\frac{\operatorname{ReLU}(\bm{A}_k)}{\max(\operatorname{ReLU}(\bm{A}_k))}, \quad \bm{A}_k={\mathbf{w}_{k}''}^{\top} f(\bm{x}) eq. 9

\bm{x}^{t} = \bm{x}^{t-1}+\xi \nabla_{\bm{x}^{t-1}} \mathcal{L}_{adv}, \quad \mathcal{L}_{adv} = \bm{y}^{t-1}[k]-\sum_{j \in K \backslash k} \bm{y}^{t-1}[j] - \mu \left\|\mathcal{M} \otimes\left|\operatorname{ReCAM}_{k}(\bm{x}^{t-1})-\operatorname{ReCAM}_{k}(\bm{x}^{0})\right|\right\|_{1} eq. 10

\operatorname{vec}(\bm{M}'_k)=\mathbf{T}^{t} \cdot \operatorname{vec}\left(\operatorname{ReCAM}_k(\bm{x}) \otimes(1-\mathcal{B})\right) eq. 11

\nabla_{\bm{z}} \mathcal{L}_{bce} = \frac{\sigma_{sig}(\bm{z})-\bm{y}}{K}, \quad \nabla_{\bm{z}} \mathcal{L}_{sce} = \sigma_{sof}(\bm{z})-\bm{y} eq. 12

Two-class (K = 2) form, with positive class p and negative class q eq. 13:

• ∇{z_p} L_bce = −1 / (2 + 2e^{z_p}); ∇{z_q} L_bce = 1 / (2 + 2e^{−z_q})
• ∇{z_p} L_sce = −1 / (1 + e^{z_p − z_q}); ∇{z_q} L_sce = 1 / (1 + e^{z_p − z_q})

Notation used above: M_k is the CAM of class k eq. 2; ⊗ is element-wise multiplication eq. 5; ξ and μ are AdvCAM hyperparameters eq. 10; T is the IRN transition matrix and B the boundary map eq. 11. The paper's ReCAM mask uses the max-normalised form of eq. 9; for AdvCAM refinement it uses the ReCAM without max normalisation §4.1.

Hyperparameters

Results

Author-reported. Setup: PASCAL VOC 2012 with the 10,582-image augmented training set §5.1; MS COCO 2014 (80k train, 40k val) §5.1; only image-level labels used during training §5.1; ResNet-50 for seeds §5.1; segmentation with DeepLabV2 (ResNet-101) or UperNet-Swin §5.1. Baselines: CAM [cam], IRN [irn], AdvCAM [anti].

Ablation on the FC layers and combination rules tab. 1 — mIoU (%), VOC / MS COCO:

"rp." marks the variant the paper reports for each dataset tab. 1.

ReCAM as seeds, and as a plug-in tab. 2 — mIoU (%) and computing time (ut; 0.7 h per unit on VOC, 5.4 h on MS COCO; "Time" covers training and generating masks for all training images). The MS COCO rows marked * are the authors' re-implementation.

The table's "ReCAM" column for the AdvCAM rows is AdvCAM applied on ReCAM seeds, not a plain ReCAM number; the text compares the values differently §5.2 tab. 2.

Segmentation mIoU (%) tab. 3 — seed masks CAM or ReCAM, refinement in the row title; the row "IRN" is IRN refinement and "AdvCAM + IRN" both:

VOC DeepLabV2 test values for the CAM columns are 55.0 (ResNet-50), 64.8 (IRN), 57.9 (AdvCAM), 68.0 (AdvCAM + IRN). VOC UperNet test: 49.6, 67.7, 56.2, 70.4 (CAM) and 55.3, 71.5, 57.4, 71.7 (ReCAM). The paper reports DeepLabV3+ results in the supplement app. B.

Plug-in comparison with other refinement methods, DeepLabV2 on VOC tab. 4 — mIoU (%), val / test. Left block: no saliency; right block: with saliency modules (* uses DeepLabV2 pre-trained on MS COCO).

Source: tab. 4. The paper's claim: "any CAM variant" can take the plug-in §5.2 tab. 4. Figures: qualitative examples fig. 5; the sensitivity to λ on VOC fig. 6; learning-rate sensitivity and MS COCO λ sweep in the supplement app. E app. F.

Paper vs code

Repository: https://github.com/zhaozhengChen/ReCAM at commit 2bdbcd2ed89d800b3340fc4021692a636a1db32d, MIT License, Copyright (c) 2021 Zhaozheng Chen [LICENSE]. The README links the arXiv paper and gives the paper's author list in its citation block.

Matches the paper:

• BCE stage-2 term: multi-label soft-margin loss averages over classes and the batch, which is eq. 1 with 1/K train_recam.py eq. 1.
• Mask-pooled features: the CAM is max-normalised per image and class, then multiplied with the feature map and averaged over space; this equals GAP of eq. 5 resnet50_cam.py eq. 2 eq. 5.
• SCE over positive classes, averaged per image then over the batch resnet50_cam.py train_recam.py eq. 7. The paper normalises by the number of positives; the code averages per image, which is the same quantity for each image.
• Loss weighting: λ = 1 for VOC run_sample.py, λ = 0.1 for MS COCO run_sample_coco.py; both match the paper §5.1.
• Combination rule: VOC seeds use w ⊗ w' make_recam.py resnet50_cam.py tab. 1; MS COCO seeds use the FC1 weights alone make_recam.py tab. 1.
• Stage-2 length and LR: 4 epochs, initial LR 5e-4 run_sample.py §5.1.
• FC2 initialisation: random, per the algorithm app. H train_recam.py.

Differs from the paper, or is not in the paper:

• FC1 and backbone learning rate: 0.1× the stage-2 LR [code: step/train_recam.py#L71-L73]. Paper gives no LR for these layers.
• Polynomial LR decay exponent: the code uses the SGD momentum value 0.9 as the exponent torchutils.py. The paper does not state an exponent.
• Optimiser: SGD with momentum 0.9 torchutils.py. The paper does not state the optimiser for stage 2.
• Stage-2 initialisation text vs algorithm: the supplement says the two FC layers start from the converged BCE baseline rather than from scratch app. F, while the algorithm initialises FC2 randomly app. H. The code loads only the stage-1 weights and creates FC2 with default initialisation train_recam.py. The code follows the algorithm.
• Multi-scale seeds (1.0, 0.5, 1.5, 2.0) are a code default [code: run_sample.py#L44]; the paper does not state a seed scale set.
• Weight decay 1e-4, batch size 16 (VOC) and 6 (MS COCO), and VOC crop and flip augmentation: all in code only run_sample.py run_sample_coco.py train_recam.py.
• AdvCAM refinement is not in this repository (the file list contains IRN training but no AdvCAM code). The authors cite AdvCAM for those hyperparameters §4.1 §5.1.

No discrepancy was found in the loss definitions or the λ/epoch/LR values; the items above are the places where the code fills gaps the paper leaves open.

Pitfalls

• Two-stage training is required. Stage 2 starts from a converged BCE model; training FC2 from scratch with the combined loss is not what was evaluated train_recam.py app. F.
• FC2 initialisation is ambiguous in the paper. The algorithm says random, the supplement says pre-trained BCE weights app. H app. F. The code uses random FC2 and the stage-1 weights for FC1. Plan on random FC2 unless you have a reason to test otherwise.
• Gradients through the mask. The code does not detach the CAM before it multiplies the features, so the SCE gradient also reaches the FC1 weights used to build the mask resnet50_cam.py. The paper says the gradients of L_sce update "the model including the backbone" §4.1 and is silent on the mask. A detached variant has not been tested here.
• Weight choice is dataset-dependent. The paper picks w ⊗ w' for VOC and w for MS COCO, and states the FC2 may overfit on MS COCO's noisier features §5.2 tab. 1. Table 1 marks the chosen variants with "rp."; the paper does not say which split they were chosen on.
• Learning rate sensitivity. The paper reports large learning rates make CAM training unstable with NaN loss, while ReCAM is insensitive app. F. This sensitivity is for stage 2 starting from a converged model.
• Max normalisation changes the refinement input. AdvCAM uses ReCAM without max normalisation §4.1, while the seed mask is max-normalised eq. 9. Mixing these up changes the regularisation term in eq. 10.
• Computing cost. On VOC the seed generation takes 1.9 ut vs 1.0 ut for CAM tab. 2. Adding IRN is about 4.5× the vanilla ReCAM cost and AdvCAM about 160× §5.2. The AdvCAM-plus-ReCAM variant is therefore much more expensive than ReCAM with IRN.
• Batch sizes and augmentation are not in the paper's ReCAM text. The VOC (16) and MS COCO (6) values are code defaults run_sample.py run_sample_coco.py. Do not assume they are the paper's values without checking the supplement.
• Segmentation hyperparameters are per backbone and dataset. Refer to app. I and §5.1 rather than assuming one schedule.
• Results are author-reported. The λ value and the w / w' combination were chosen by looking at results §5.2 tab. 1; the paper does not say on which split the choice was made.
• No code for the refinement methods in this repository. AdvCAM is cited for hyperparameters but its code is not in the repository (see Paper vs code).
• Stage 1 (BCE) details are only "same as IRN", which in turn depends on the IRN paper; the card does not restate them.

Lineage

• Builds on: CAM (Zhou et al., "Learning deep features for discriminative localization", CVPR) [cam], which ReCAM takes as its starting model §1 §4.1. Refinement options used with it: IRN (Ahn et al., CVPR) [irn] and AdvCAM (Lee et al., CVPR) [anti] §4.1.
• Compared against: IRN, AdvCAM, OOA, ICD, SEAM, SC-CAM, BES, CONTA, RIB, DSRG, SGAN, NSROM, EDAM and EPS tab. 4 §5.2.
• Superseded by: not stated in this card's sources.