Paper · 2022-03-17 · cs.CV
Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation
RCA adds a regional memory bank to image-label segmentation (OAA++ and EPS baselines). Semantic contrast pulls each pseudo-region toward same-class memory entries, with region mixup. Semantic aggregation feeds k-means prototypes of the memory into the CAM head. Author-reported: VOC 2012 val mIoU 70.6 for RCA+OAA++ (VGG16), +2.9 over its baseline, and 72.2 for RCA+EPS (ResNet38 classifier).
Weakly-supervised learningSemantic segmentationContrastive learningMemory-augmented neural networkweakly-supervisedsemantic-segmentationcontrastive-learningmemory-bank
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Segmentation models need a label for every pixel, which is expensive. This paper works with a cheaper kind of label: each training image only says which object classes appear in it, with no outlines. A common way to learn from such labels is to find the rough area of each object in each image and then use those rough areas as pretend pixel labels.
The method's idea is that a single image sees only a few objects, but the whole training set sees many examples of each class. The model keeps a store of object-region summaries gathered from all training images. For each object it sees, it is pulled toward summaries of the same class and pushed away from other classes. It also looks up the most typical summaries in the store and mixes that context into its own features. Both steps are meant to make the rough object areas more complete, which gives better pretend labels and a better final segmentation model.
TL;DR
RCA is a training add-on for image-label segmentation. It keeps a memory of object-region embeddings per class, applies a contrastive loss against that memory with a mixup of regions, and enriches features with k-means prototypes of the memory. The author-reported gains are about 1 to 4 points, reported on PASCAL VOC 2012 and COCO 2014 with VGG16 or ResNet backbones. The code covers only the classifier and the pseudo-label step; it has no segmentation or evaluation code, and it has no license.
Problem
With only image-level labels, a classifier's class activation map (CAM) tends to cover only the most discriminative part of an object. Earlier fixes expand each image's map on its own, or compare two or four images at a time. The paper argues that these methods see too little of the dataset to learn a full picture of how each object class looks, and that pixel-wise cross-image modelling is too costly and noisy §1 §2.
Method
The base model follows the usual weakly supervised pipeline: a backbone gives a dense feature map, a class-wise head gives maps, and maps are pooled for the image-level loss §3.1 eq. 1. RCA adds three parts on top of it. Each part is described below in the order the paper gives §3.2.
1. Pseudo-regions and the memory bank §3.2.1 §3.2.2. For each class present in an image, the paper pools the dense features over the pixels whose intermediate activation is above that map's mean. This gives one embedding per present class eq. 2. Each class has a memory of such embeddings. When a class is present and its image-level score is above a threshold ν, the memory entry is updated as an exponential moving average with momentum γ eq. 3 §3.2.2.
2. Semantic contrast with region mixup §3.2.3. A region embedding is pulled toward memory entries of its own class and pushed away from entries of other classes, using a cosine-similarity InfoNCE-style loss with temperature τ eq. 4. To cope with noisy weak labels, each region is also mixed with a region from another image, with a weight ω drawn from a symmetric Beta distribution eq. 5. The loss on the mixed embedding is the ω-weighted sum of the losses against the two source classes eq. 6.
3. Semantic aggregation over prototypes §3.2.4. For each class, k-means clusters the memory into K centroids, the prototypes. All prototypes are stacked, and each feature location computes a softmax affinity to them eq. 7. The prototypes are combined with that affinity eq. 8, and the result is concatenated with the original feature map, doubling the channels eq. 9.
4. Final CAM and loss §3.2.5 §3.3. A second class-wise head reads the enriched features and produces the final maps O eq. 10. The training loss sums over images three terms: the region mixup contrast, a cross-entropy on the GAP of the intermediate maps, and a cross-entropy on the GAP of the final maps eq. 11. The paper states that the memory is dropped at inference and only the prototypes are kept §3.3.
5. Pseudo-labels and segmentation §4.1. After training, the final maps O become foreground seeds. A saliency map supplies background cues, and the two are combined. A DeepLab-v2 segmentation network is trained on these pseudo-masks with its default settings, and dense CRF is applied as post-processing to refine boundaries §4.1. The paper does not specify the combination rules.
# Forward pass and loss for one mini-batch [§3.2] [eq. 11]
# Input: images (B, 3, H, W); labels (B, L) multi-hot
# Output: scalar training loss
def rca_loss(images, labels, memory, prototypes):
# Dense embedding F from the backbone [§3.1] [eq. 1]
feats = backbone(images) # (B, D, H, W)
# Intermediate class maps P and their pooled scores [eq. 1]
coarse_maps = coarse_cam_head(feats) # (B, L, H, W)
coarse_scores = pool(coarse_maps) # (B, L)
# Concatenate F with the prototype-attended features F' [eq. 7-9]
attended = aggregate(feats, prototypes) # (B, D, H, W)
enriched = concat_channels(feats, attended) # (B, 2D, H, W)
# Final class maps O and their pooled scores [eq. 10]
final_maps = final_cam_head(enriched) # (B, L, H, W)
final_scores = pool(final_maps) # (B, L)
# Region contrast with mixup, for every present class [§3.2.3]
contrast = contrast_loss(feats, coarse_maps, labels, memory)
# Weighted sum of the three terms [eq. 11]
return contrast + bce(coarse_scores, labels) + bce(final_scores, labels)# Region-mixup contrast for one image and one present class [§3.2.3]
# Mixing weight omega is Beta(beta, beta) [eq. 5]
def region_contrast(region, other, other_label, memory, label):
# Pooled embedding of the partner region in another image [eq. 2]
weight = beta_sample(8, 8) # omega [§4.1]
# Mixed embedding of the two regions [eq. 5]
mixed = weight * region + (1 - weight) * other
# Contrast against each memory class, weighted as in eq. 6 [eq. 4] [eq. 6]
loss_same = nce(mixed, memory, label)
loss_other = nce(mixed, memory, other_label)
return weight * loss_same + (1 - weight) * loss_other# Prototype attention used by semantic aggregation [§3.2.4] [eq. 7-8]
# prototypes: (L * K, D), with K centroids per class
def aggregate(feats, prototypes):
# Flatten the spatial grid into rows # (B, H*W, D)
flat = flatten_spatial(feats)
# Softmax affinity between locations and prototypes [eq. 7]
affinity = softmax(flat @ transpose(prototypes), dim=-1)
# Weighted sum of prototypes at each location [eq. 8]
return reshape_spatial(affinity @ prototypes)Hyperparameters
Values stated in the paper are cited to it. Values only in the code are marked so. "Not stated" means neither the paper nor the code gives a value.
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbones | VGG16, ResNet38 | §3.3 §4.1 | Only the VGG16 path is in the code rca.py |
| Input size | 368 x 368 | train.py | Not stated in the paper; square resize LoadData.py |
| Batch size | 8 | §4.1 | Matches train.py |
| Epochs | 30 | §4.1 | Matches train.py |
| Optimiser | SGD, momentum 0.9 | §4.1 | Code adds Nesterov train.py |
| Weight decay | 5e-4 | §4.1 | Matches train.py |
| Learning rate, backbone | 1e-3 | §4.1 | Code gives 2e-3 to backbone biases train.py |
| Learning rate, other parts | 1e-2 | §4.1 | Code gives 1e-2 to the head weights and 2e-2 to head biases; see Paper vs code |
| LR schedule | x0.1 every five epochs | §4.1 | Code decays at epochs 5 and 10 only train.py my_optim.py |
| Warm-up | first epoch, alpha1 = 0 | §4.1 | Code uses the first two epochs train.py |
| Momentum gamma | 0.99 | §4.1 §4.2 | Best of the sweep in the uncaptioned gamma table in §4.2 |
| Threshold nu | 0.7 | §4.1 | Code applies it to a sigmoid score rca.py rca.py |
| Beta shape beta | 8 | §4.1 | Used as Beta(8, 8) rca.py |
| Loss weight alpha1 | 0.01 | §4.1 eq. 11 | Code uses 0.01 on the summed contrast term train.py |
| Loss weight alpha2 | 0.4 | §4.1 eq. 11 | Code uses 0.4 on the intermediate CE train.py |
| Prototypes per class K | 10 | §4.1 §4.2 | Best in the K sweep (uncaptioned table in §4.2) |
| Memory per class | VOC adaptive; COCO 500 | §4.1 | Code uses 500 for all datasets rca.py |
| Temperature tau | not stated | §3.2.3 | Code uses 0.07 rca.py |
| Dense CRF | not stated | §4.1 | Code uses weights from imutils.py |
| Hardware | 4 x V100 train, 1 x RTX2080Ti test | §4.1 | Code requires CUDA train.py |
Results
Author-reported. Setup: PASCAL VOC 2012 (train 1,464, val 1,449, test 1,456, plus 10,582 extra training images §4.1), COCO 2014 (80K train, 40K val §4.1). Metric: mIoU. VOC test scores come from the official server §4.1.
Ablation on VOC 2012 train and val, VGG16 classifier tab. 1. Pseudo-label mIoU on train, segmentation mIoU on val.
| Variant | Pseudo-label | Segmentation |
|---|---|---|
| OAA+ (baseline, reported) | not given | 65.2 |
| OAA++ (own baseline) | 68.2 | 67.7 |
| OAA++ with RSC | 69.5 | 69.3 |
| OAA++ with RSA | 68.5 | 68.6 |
| OAA++ with RSC and RSA | 71.4 | 70.6 |
Pseudo-label quality on VOC 2012 train tab. 2.
| Base method | Classifier | Base mIoU | With RCA | Gain |
|---|---|---|---|---|
| OAA++ | VGG16 | 68.2 | 71.4 | +3.2 |
| OAA++ | ResNet38 | 69.4 | 73.2 | +3.8 |
| EPS | ResNet38 | 71.4 | 74.1 | +2.7 |
Segmentation on VOC 2012 tab. 3. The table also lists about 20 other methods. Only the rows that pair with RCA are shown here.
| Method | Classifier | val mIoU | test mIoU |
|---|---|---|---|
| OAA++ | VGG16 | 67.7 | 67.4 |
| RCA + OAA++ | VGG16 | 70.6 (+2.9) | 71.0 (+3.6) |
| OAA++ | ResNet38 | 68.1 | 68.2 |
| RCA + OAA++ | ResNet38 | 71.1 (+3.0) | 71.6 (+3.4) |
| EPS | ResNet38 | 70.9 | 70.8 |
| RCA + EPS | ResNet38 | 72.2 (+1.3) | 72.8 (+2.0) |
Segmentation on COCO 2014 val, VGG16 tab. 4.
| Method | mIoU | With RCA |
|---|---|---|
| OAA+ | 24.6 | 26.7 (+2.1) |
| EPS | 35.7 | 36.8 (+1.1) |
Diagnostics in §4.2, reported as text and uncaptioned tables. Removing region mixup lowers the mIoU by 0.8 (71.4 to 70.6) §4.2. Momentum gamma gives 69.9 at 0, 70.9 at 0.5, 71.2 at 0.8 and 0.9, 71.4 at 0.99, and 70.9 at 0.999. The paper says the sweep is on pseudo-label accuracy §4.2. K gives 70.4 at 1, 71.4 at 10, 71.1 at 20 and 50, 71.3 at 100, and 70.0 with all memory entries. Memory size gives 70.8 at 100 entries per class, 71.2 at 500, and 71.4 with all entries. The paper does not name the metric of the K and memory sweeps. These tables have no captions, so they are cited by section.
Paper vs code
Code: official repository at commit d410a96472580ff980bf89ea250b2c82aadca795.
It has no LICENSE file, so no code is quoted here. Checked: the README, all
Python files in the repository and data/voc/train_cls.txt. The code covers
the VGG16 classifier, training, and pseudo-label generation. Searched: the
paper's author line links the repository, so it was used without further search.
- Memory query and store differ. The paper pools the same dense feature for queries and memory eq. 1 eq. 2. The code stores the backbone output rca.py rca.py, but queries the output of a separate head rca.py rca.py.
- Aggregation is not as written. The paper's affinity uses a softmax and concatenates 2D channels eq. 7 eq. 9. The code adds learned Q, K and V projections rca.py, scales the affinity by 20/10 with no softmax rca.py, and fuses back to 512 channels with a 1x1 convolution rca.py rca.py. The prototypes are detached rca.py rca.py.
- CAM heads are not 1x1. The paper says each head is a 1x1 convolution §3.3. The code uses three 3x3 convolutions plus a 1x1 for the intermediate head rca.py, and one such stack per class for the final head rca.py.
- Mask uses a different map. The paper thresholds the raw activation map of each class at its mean eq. 2. The code applies a softmax across classes first rca.py rca.py, then thresholds at the mean rca.py rca.py.
- Enqueue score differs. The paper's score is the un-normalised GAP of the intermediate maps P §3.1. The code uses the sigmoid of the GAP of the final maps O rca.py rca.py, against the same threshold of 0.7 rca.py.
- Mixup partner is not constrained. The paper requires the partner region to come from another image and a different category §3.2.3. The code draws a random permutation of the batch for each region, which can return the same image rca.py. The partner label is drawn at random from that image's labels, with no check against the region's class rca.py.
- Loss normalisation differs. The paper sums all three terms over images eq. 11. The code averages the two cross-entropy terms, with a multilabel soft-margin loss train.py train.py, and sums the contrast term over the batch rca.py.
- Warm-up length. The paper uses one warm-up epoch §4.1. The code uses two train.py.
- Learning-rate schedule. The paper reduces the rate every five epochs §4.1. The code reduces it at epochs 5 and 10 only train.py my_optim.py.
- Learning rates of new layers. The paper gives 1e-2 to the other components §4.1. The code gives 10x only to parameters whose names contain "extra". The Q, K, V, aggregation and fusion layers fall in the backbone group, which uses 1e-3 rca.py train.py.
- Memory size. The paper uses an adaptive memory on VOC §4.1. The code fixes 500 entries per class for all data rca.py.
- Memory timing. The paper says the memory is updated during backward propagation §3.2.2. The code updates it at the end of the forward pass rca.py, under no_grad rca.py.
- Inference path. The paper drops the memory at inference §3.3. The repository has no such path. The forward pass always needs image labels rca.py rca.py, so it cannot run label-free.
- Pseudo-labels come from training-time maps. The paper uses the final maps after training §4.1. The code writes the maps during training, keeps a running maximum over epochs for images whose score is at least 0.1, and writes them unconditionally in the last epoch if no file exists rca.py.
- Pseudo-label rules are not in the paper. The code uses a background exponent of 4, 10 CRF iterations, a conflict level of 0.9, a background threshold of 32 and a foreground threshold of 0.8, and drops masks with less than 1% foreground gen_labels.py gen_labels.py gen_labels.py.
- Initialisation and prototypes. Memory and prototypes start as N(0, 1) noise rca.py. The prototypes are first computed at the start of epoch 1, so epoch 0 uses noise rca.py. The paper states neither point.
- Augmentation. The paper does not state any. The code resizes to a square 368 x 368, applies colour jitter, and does not shuffle the training list LoadData.py transforms.py.
- Only VGG16 is implemented. The paper reports ResNet38 §4.1. The code builds only the dilated VGG16 variant rca.py rca.py rca.py.
- Missing from the repository. The segmentation network, the OAA++ and EPS
integrations, any evaluation code, the COCO pipeline, and
data/voc/val_cls.txt, whichtrain.pyrequires train.py LoadData.py. The saliency maps are expected but not supplied gen_labels.py. - Unused or dead code. The
--thresholdargument is parsed and never used train.py. The momentum-encoder update refers to modules that are never created rca.py.
Pitfalls
- Random memory initialisation persists. Each slot starts as Gaussian noise and each update keeps 99% of the old value. A slot that is updated n times keeps a factor of 0.99^n of its start, so the noise fades slowly. The magnitudes were not measured. Rare classes are most affected. Treat early memory and prototypes with caution, or initialise to zero and test the effect.
- Epoch 0 has random prototypes. Clustering starts only at epoch 1, so early aggregation uses noise.
- Pure Python loops. The contrast loss loops over images, classes and regions, and compares each region with all memory entries. Expect slow training, and check the cost before scaling to COCO.
- Labels are needed in the forward pass. The model cannot run inference on images without labels. Plan a separate label-free path if you need one.
- Unstated values. The paper leaves out tau, the CRF settings, the saliency source, the k-means initialisation and seed, and the augmentation. Take them from the code, and say which you did.
- The code is unlicensed. There is no LICENSE file, so do not copy code from the repository. Write your own implementation from this card and the paper.
- Memory and prototypes after training. The paper keeps only the prototypes at inference §3.3. The code does not implement that, so the memory is part of the saved checkpoint.
- Results are author-reported. The OAA++ baseline is the authors' own construction §4.1. The paper does not say whether the EPS numbers were rerun. Reproduce the baseline before you compare.
Lineage
- Builds on: OAA+ (ICCV 2019) and EPS (CVPR 2021), the two baselines the paper extends tab. 3. Memory banks and momentum updates, cited in §2 and §3.2.2.
- Compared against: the WSSS methods listed in tab. 3 and tab. 4.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation” by Tianfei Zhou, Meijie Zhang, Fang Zhao, Jianwu Li (arXiv:2203.09653), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/maeve07/RCA at commit d410a9647258 (None: the repository has no LICENSE file). Any quoted code is shown under that license.