Concealed Object Detection

In plain words

Some animals and objects are built to blend into their background, so well that a person needs a few seconds to spot them. Detectors trained on ordinary photos struggle with them, because the usual cues (a clear edge, a colour that stands out) are missing.

The paper borrows an idea from how predators hunt. First the predator searches for a rough area where prey might be. Then it looks closer and works out the exact outline. The network does the same: one part finds a rough blob, and a chain of refinement steps sharpens its outline, each step paying special attention to the parts the previous step had not yet covered.

The authors also release a new dataset of 10,000 images of concealed objects, labelled in more detail than earlier collections, and public code. The method beats twelve earlier networks on the test sets they used.

TL;DR

Fan, Ji, Cheng and Shao define concealed object detection (COD) as predicting a binary pixel mask for objects that blend into their surroundings §2. They build the COD10K dataset §3 and SINet-V2, a search-and-identification network built from a Res2Net-50 encoder, texture-enhanced modules, a neighbor connection decoder and group-reversal attention blocks §4. Results are author-reported and on the authors' own test split. The code is public under Apache-2.0, and the training schedule in code differs from the paper in several small ways (see Paper vs code).

Problem

Concealed objects share texture and colour with the background, so low-level edge cues and saliency cues mislead detectors §1 §2. Earlier COD datasets were small: CHAMELEON has 76 images and CAMO has 2,500 §2. The authors say the lack of a large benchmark has held back the field §1.

Method

Task. Each pixel receives a label, 1 for concealed object and 0 otherwise. COD is class-agnostic §2.

Dataset (COD10K). 10,000 images: 5,066 concealed, 3,000 background images collected as salient images from Flickr to reduce selection bias, and 1,934 non-concealed scenes §3.1. The concealed images fall into 69 sub-classes, with 9 non-concealed sub-classes §3.1. Annotations are hierarchical: category, bounding box, attributes and object or instance masks §3.2. The images carry seven challenge attributes (multiple objects, big object, small object, out-of-view, occlusion, shape complexity, indefinable boundaries) fig. 7. Object-level labels cover 5,069 objects and 5,930 instances §3.2. Each image was labelled to matting level, which the authors say takes about 60 minutes per image §3.2. The split is 6,000 for training and 4,000 for testing §3.3 tab. 1 (but see Pitfalls for a conflicting test count).

Network overview. SINet-V2 has two phases, search and identification, in a coarse-to-fine cascade §4.1 fig. 13.

1. Feature extraction. Res2Net-50 gives five feature levels, with spatial resolution H/2^k for k = 1 to 5. Only the top three levels (k = 3, 4, 5) are used, because low-level features are expensive and add little §4.2.
2. Texture enhanced module (TEM). Each level passes through a TEM. It has four residual branches with dilation rates 1, 3, 5 and 7. Each branch first reduces the channels to 32 with a 1×1 convolution, then uses an asymmetric pair of (2i−1)×1 and 1×(2i−1) kernels, and a dilated 3×3 convolution when i > 1. The four branches are concatenated and reduced to C = 32 channels by a 3×3 convolution, and a shortcut is added before a ReLU §4.2 fig. 13.
3. Neighbor connection decoder (NCD). It fuses the three TEM outputs, each step upsampling the coarser level and multiplying it with the finer one §4.2 eq. 1. The output is a coarse map C6 at stride 8.
4. Reverse guidance. For each stage k, the coarser prediction is resampled to level k, passed through a sigmoid, then reversed (1 − x). The reversed map marks the regions the coarser stage has not yet explained §4.3 eq. 2.
5. Group-reversal attention (GRA). Each stage has three GRA blocks. A block splits the feature channels into groups and places the reversed guidance after each group. This is the group guidance operation (GGO) §4.3 eq. 3. A 3×3 convolution with batch normalisation then turns the result into a residual update of the features eq. 4. The guidance receives a single-channel residual update eq. 5. Group sizes are 32, 8 and 1 channels for the three blocks §4.3 §5.3.
6. Output. Each stage outputs the refined guidance plus the upsampled coarser prediction eq. 6. The network produces four maps, C6, C5, C4 and C3, and supervises all four §4.4.1.

Loss. Each of the four maps (C6, C5, C4, C3) receives a loss against the ground truth. The loss is the sum of a weighted IoU loss and a weighted binary cross entropy. The weights increase on hard pixels, meaning pixels near boundaries §4.4.1. The paper defines the losses by reference to earlier work and gives no constants.

Training. Adam, batch size 36, learning rate 1e-4, divided by 10 every 50 epochs, for 100 epochs. Training takes about 4 hours on one TITAN RTX GPU §4.4.2. Training data is the combined CAMO and COD10K training sets, without extra data §5.1, footnote.

Inference. Each image is resized to 352×352. The network runs once with no post-processing, and the prediction is upsampled to the original size §4.4.2. The authors report about 45 fps on one GPU, excluding I/O §4.4.2.

# SINet-V2 forward pass  [§4.1] [fig. 13]
# image: (B, 3, 352, 352); returns four logit maps (C6, C5, C4, C3)

def forward(image):
    # Res2Net-50 features at strides 8, 16, 32  [§4.2]
    feat_3, feat_4, feat_5 = res2net_levels(image)
    # 1x1 reduce to C = 32 and four dilated branches  [§4.2]
    tem_3 = texture_enhanced(feat_3)     # (B, 32, 44, 44)
    tem_4 = texture_enhanced(feat_4)     # (B, 32, 22, 22)
    tem_5 = texture_enhanced(feat_5)     # (B, 32, 11, 11)
    # Coarse location map C6 from the neighbor connection decoder  [eq. 1]
    map_6 = neighbor_connection_decoder(tem_5, tem_4, tem_3)
    # Stage 5: resample C6 by x1/4 to level 5  [§4.3] [eq. 2]
    guidance_5 = downsample(map_6, factor=4)
    map_5 = gra_stage(tem_5, guidance_5) + guidance_5
    # Stage 4: resample C5 by x2 to level 4  [§4.3] [eq. 2]
    guidance_4 = upsample(map_5, factor=2)
    map_4 = gra_stage(tem_4, guidance_4) + guidance_4
    # Stage 3: resample C4 by x2 to level 3  [§4.3] [eq. 2]
    guidance_3 = upsample(map_4, factor=2)
    map_3 = gra_stage(tem_3, guidance_3) + guidance_3
    return map_6, map_5, map_4, map_3

# One GRA stage: reverse once, then three GRA blocks  [§4.3]
# features: (B, C, h, w); guidance: (B, 1, h, w) logits from the stage above

def gra_stage(features, guidance):
    # Reverse guidance r_1 = 1 - sigmoid(guidance)  [§4.3] [eq. 2]
    reverse_1 = 1 - sigmoid(guidance)
    residual = reverse_1
    # Group sizes 32, 8, 1 channels for blocks 1, 2, 3  [§4.3] [§5.3]
    for group_size in [32, 8, 1]:
        # Interleave r_1 after each channel group  [§4.3] [eq. 3]
        mixed = group_guidance(features, reverse_1, group_size)
        # Residual feature update: 3x3 conv + batch norm  [§4.3] [eq. 4]
        features = features + conv3x3_bn(mixed)
        # Residual guidance: r_1 plus a single-channel update  [eq. 5]
        residual = reverse_1 + conv3x3_to_one(features)
    # Refined residual prediction; the caller adds the upsampled coarser map
    # [§4.3] [eq. 6]
    return residual

# Training schedule  [§4.4.2]
# images are resized to 352 x 352 and augmented  [code: utils/data_val.py#L115]

def train(model, loader):
    # Adam with learning rate 1e-4  [§4.4.2]
    optimizer = adam(model.parameters(), lr=1e-4)
    for epoch in range(1, 101):
        # Divide the learning rate by 10 every 50 epochs  [§4.4.2]
        lr = 1e-4 * 0.1 ** (epoch // 50)
        for images, masks in loader:    # batch size 36  [§4.4.2]
            maps = model(images)
            # Sum of IoU and BCE losses over all four maps  [§4.4.1]
            loss = sum(structure_loss(m, masks) for m in maps)
            optimizer.step()
    return model

Hyperparameters

Results

Author-reported. Setup: three test sets (CHAMELEON 76 images, CAMO test, COD10K test), 12 baselines trained on the combined CAMO and COD10K training sets, no post-processing for SINet-V2 §5.1 §5.2. Metrics: S-measure (Sα, higher is better), mean E-measure (Eφ, higher), weighted F-measure (Fβʷ, higher) and mean absolute error (M, lower) §5.1.

Headline: SINet-V2 is best on every metric and every test set in tab. 2. On COD10K test it reaches Sα 0.815 against 0.789 for PraNet, the next best baseline in Sα tab. 2. The paper's own conference version (SINet_cvpr) is also listed.

Table 2: main results, tab. 2

ANet-SRM was evaluated on CAMO only tab. 2.

Table 3: COD10K by super-class, tab. 3. Per the table caption, all methods are trained on the same data as the conference version. The image counts are as printed in the table. Each cell is Sα / Eφ / Fβʷ / M.

Per-sub-class results are in tab. 4 (S-measure only). They are not reproduced here. The authors name Grouse as the easiest sub-class and LeafySeaDragon as the hardest §5.2.

Generalisation, tab. 5. Structure-measure after cross-dataset training. The authors train SINet_cvpr, not SINet-V2, on 800 images and test on 200 §5.2. Their scores are not comparable with tab. 2.

Ablation, tab. 6. Each row retrains with the same hyperparameters §5.3. PD is the partial decoder, NCD the neighbor connection decoder, Sy and Asy symmetric and asymmetric convolution in the TEM, Reverse the reverse pattern per GRA block, Group the group sizes per GRA block. Values are Sα / Eφ / Fβʷ / M.

What the ablation supports, as the authors state it §5.3: the NCD raises CAMO Eφ from 0.869 to 0.882 tab. 6, #1 and #OUR; the asymmetric TEM raises CAMO Eφ from 0.866 to 0.882 tab. 6, #4 and #OUR; reversing only the first block works best; and the progressive group sizes {32;8;1} generalise better than the uniform alternatives.

Qualitative results are described in words only §5.2 fig. 16. The authors claim better results than the conference version under different lighting, appearance changes and indefinable boundaries fig. 16.

Paper vs code

Code: GewelsJI/SINet-V2 at commit 5b9844c (HEAD when cloned). The repo says it is the PyTorch implementation of the journal version README.md. The GitHub owner is Ge-Peng Ji's account, listed as an author in the README README.md. Licence: Apache-2.0 LICENSE.

• TEM shortcut. Paper: identity shortcut §4.2. Code: the shortcut is a 1×1 conv-bn projection Network_Res2Net_GRA_NCD.py added before the ReLU Network_Res2Net_GRA_NCD.py.
• TEM first branch. Paper: four branches, the first with kernel 1 and dilation 1 §4.2. Code: branch 0 is a single 1×1 conv-bn with no 3×3 conv Network_Res2Net_GRA_NCD.py. Branches 1 to 3 match the paper's asymmetric (1,3)/(3,1), (1,5)/(5,1), (1,7)/(7,1) pattern with dilated 3×3 convs Network_Res2Net_GRA_NCD.py.
• Activations. BasicConv2d defines a ReLU but its forward returns before it is applied Network_Res2Net_GRA_NCD.py. Only the TEM output applies ReLU Network_Res2Net_GRA_NCD.py.
• GRA normalisation. Paper: 3×3 conv followed by batch norm §4.3. Code: 3×3 conv followed by ReLU, no batch norm Network_Res2Net_GRA_NCD.py.
• GRA guidance accumulation. Paper: r_{i+1} = r_1 + g[p_{i+1}], and the GGO uses r_1 in every block §4.3 eq. 3 eq. 5. Code: the guidance y is updated in place, y = y + score(x), and the next block uses the updated y Network_Res2Net_GRA_NCD.py Network_Res2Net_GRA_NCD.py. Only the first block's guidance is reversed, as the paper says Network_Res2Net_GRA_NCD.py.
• NCD beyond Eq. 1. Paper gives Eq. 1 for the fusion step and calls the decoder a modified UNet-like one §4.2 eq. 1. Code adds concatenation and convolution stages after the fusion, then a 3×3 conv and a 1×1 output conv Network_Res2Net_GRA_NCD.py. The NCD upsampling uses align_corners=True Network_Res2Net_GRA_NCD.py, while the rest of the network uses the default Network_Res2Net_GRA_NCD.py.
• Group sizes. Paper: group sizes 32, 8, 1 with m = C/g groups §4.3 §5.3. Code: GRA(channel, subchannel) uses channel // subchannel chunks, giving 1, 4 and 32 chunks, which matches Network_Res2Net_GRA_NCD.py.
• Loss. Paper: weighted IoU plus weighted BCE, weights unspecified §4.4.1. Code: weights 1 + 5·|avgpool31(G) − G|, per-image normalised, BCE on logits and IoU on sigmoid outputs MyTrain_Val.py. The four-map sum matches §4.4.1 MyTrain_Val.py.
• Epochs. Paper: 100 epochs §4.4.2. Code: range(1, 100), so 99 epochs at default settings MyTrain_Val.py.
• Learning-rate schedule. Paper: divide by 10 every 50 epochs §4.4.2. Code: factor 0.1^(epoch // 50) with epoch counted from 1, so epochs 1 to 49 use 1e-4 and epochs 50 to 99 use 1e-5 utils.py MyTrain_Val.py.
• Gradient clipping. Not in paper. Code clamps each gradient element to ±0.5 MyTrain_Val.py utils.py.
• Augmentation. Not in paper. Code applies horizontal flip, random crop, rotation, colour jitter and pepper noise data_val.py.
• Model selection. Paper: silent. Code: keeps the checkpoint with lowest MAE on --val_root, defaulting to the CAMO test set MyTrain_Val.py MyTrain_Val.py.
• Test-time map. Paper: "final prediction" without naming the map §4.4.2. Code: uses the finest map C3, applies sigmoid, and min-max rescales each map to [0, 1] before saving MyTesting.py.
• Ablation variants. Only the full network is in the repo Network_Res2Net_GRA_NCD.py. The variants in tab. 6 cannot be checked against code.
• Evaluation. Repo ships MATLAB scripts in eval/ main.m. They were not read in detail for this card.

No other discrepancies found after checking the encoder, the TEM branch convolutions, the reverse operation, the NCD equation and the four-map loss.

Pitfalls

• Conflicting test-set size. Section 3.3 and tab. 1 give 4,000 test images §3.3 tab. 1, but the COD10K test results section says 2,026 test images §5.2. The paper does not explain the gap.
• Conflicting super-class count. Section 3.1 lists 10 super-classes §3.1. Section 3.2 says five concealed super-classes §3.2, and Section 3.3 says five again §3.3. tab. 3 shows four. Its image counts sum to 2,011, not 2,026 tab. 3.
• Model selection on a reported test set. The code picks the best epoch by MAE on CAMO test by default, and CAMO test is one of the reported benchmarks MyTrain_Val.py MyTrain_Val.py. Use a separate validation split when reproducing, or report that the selection leaks.
• Label noise in augmentation. The training transform applies pepper noise to the ground-truth mask: about 0.15% of its pixels, at random positions, are set to 0 or 255 each time the sample is loaded data_val.py data_val.py. The paper does not mention it.
• Crop and rotation change labels. Random crop, rotation and flip are applied to image and mask together data_val.py, so masks stay aligned. Rotation fills corners with zeros data_val.py.
• Weight constants are in code only. The IoU and BCE weighting uses a 31-pixel average pool and factor 5 MyTrain_Val.py. The paper gives no constants.
• Upsampling modes differ. The NCD uses align-corners bilinear and the rest of the network does not Network_Res2Net_GRA_NCD.py Network_Res2Net_GRA_NCD.py. Reproduce both.
• Resize ignores aspect ratio. Inputs are forced to 352 × 352 with no letterboxing, in training and at test time data_val.py data_val.py. Objects are distorted in non-square images.
• Generalisation numbers are not SINet-V2. tab. 5 uses the conference model and a small 800/200 split §5.2. Do not compare it with tab. 2.
• Reverse guidance is implemented as a running sum. Following the code instead of the paper changes the guidance each GRA block sees [see Paper vs code].
• Speed and cost. The 45 fps figure is on a single GPU and excludes I/O §4.4.2. Training takes about 4 hours on one TITAN RTX GPU §4.4.2. Ablation rows use the same schedule.
• Results are author-reported with no variance. The tables give one number per cell, with no spread across runs. Small gaps in the ablation, such as #1 vs #OUR on CHAMELEON Eφ (0.940 vs 0.942), are not tested for significance.

Lineage

• Builds on: the authors' CVPR 2020 SINet, cited as fan2020camouflaged §1. SINet-V2 adds the NCD and GRA blocks and extends the dataset and benchmark §1. No card for the conference version.
• Compared against: twelve baselines §5.1, including ANet-SRM, whose card is Anabranch Network and whose results here are CAMO-only tab. 2.
• Superseded by: none recorded in this wiki.