Simultaneously Localize, Segment and Rank the Camouflaged Objects

In plain words

Some animals are hard to see because their colours and texture blend into the background. Finding them is the camouflaged object detection problem. Earlier models were trained only on "animal or not animal" masks, which say where the object is but not how well it hides.

This paper adds two more questions. The first is which part of the animal catches the eye first, such as a head or a bright pattern. The second is how hard the animal is to spot. The authors answered the second question by recording how long people took to notice each animal with an eye tracker, then turning those times into three difficulty levels.

One network does all three jobs. A shared backbone feeds a branch that finds the eye-catching region, and that region steers a second branch that draws the full outline of the animal. A third branch, which is not in the released code, puts each animal into a difficulty level. The authors report better numbers than earlier camouflage models and a new test set of about 4,000 images.

TL;DR

• Joint model (ResNet-50 backbone) that predicts a fixation map, a camouflaged-object mask and a camouflage rank §3.2, §3.3.
• Ranks come from median eye-tracker detection delays of six observers on 2,000 training and 280 test images §3.1.
• Author-reported gains on COD10K, CAMO, CHAMELEON and the new NC4K set tab. 1, tab. 4.
• The released code trains the fixation and mask branches only; the ranking branch and the rank dataset builder are absent train.py.

Problem

Binary camouflage masks say where an object is but not how well it blends in, so models cannot explain why some objects are hard to find §1. The authors also argue that detectability depends on a few discriminative parts rather than the whole object §1. Existing camouflage datasets have no difficulty or discriminative-region labels, and few test images: the other public test sets have fewer than 300 images each, while COD10K has 2,026 test images §1 §2.

Method

The paper has three parts: a new labelled dataset §3.1, a joint fixation-and-mask network §3.2, and a ranking branch §3.3.

Dataset (CAM-FR) §3.1 fig. 2.

1. Eye-tracker measurements (SMI RED250, 250 Hz, 60–80 cm viewing distance) were collected for images taken from CAMO and COD10K §3.1.
2. Six observers viewed each image. For observer j and instance i, the observation time is the median of the fixation times on that instance, measured from the observer's start time eq. 1.
3. Per-instance time is the median of the six per-observer values, using the median definition in eq. 2 §3.1.
4. If more than half of the observers never fixated an instance, it is a hard sample and its search time is set to 1 after normalization. Otherwise, the missing observers are dropped and the median uses the rest §3.1.
5. Instances are assigned rank 1 (hardest), 2 (median), 3 (easiest), and rank 0 marks background §3.1. The text does not state the cut points between the three ranks.

Joint localization and segmentation §3.2 fig. 3.

1. A ResNet-50 backbone yields four stages, s1 to s4 §3.2.
2. A Fixation Decoder predicts the discriminative-region map. Each stage is reduced to 32 channels by a 3×3 convolution. The decoder uses dual residual attention (position and channel attention, from DANet) and an ASPP-type module §3.2 fig. 3.
3. The fixation prediction F gives a reverse attention 1 − F. The paper multiplies this with the backbone feature s1 to form guided features s_i^r for all four stages §3.2.
4. A Camouflage Decoder with the same structure as the fixation decoder predicts the segmentation map S from the guided features §3.2.
5. Loss: binary cross-entropy for the fixation term, a pixel-position-aware loss (from F3Net) for the segmentation term, combined as L_fc = L_f + λ·L_c with λ = 1 §3.2 eq. 3.

Ranking §3.3 fig. 2 fig. 4.

1. The ranking model follows Mask R-CNN: a ResNet-50 backbone with an FPN, an RPN, RoIAlign, then a classification-and-regression head that predicts the rank and a mask head that predicts a binary mask for each non-background instance §3.3 fig. 2.
2. Training loss: L = L_rpn + L_rank + L_mask eq. 4. L_mask is applied only where the predicted rank is not 0 §3.3.
3. Ranks are progressive, so the rank loss is weighted by a 4×4 similarity prior S_p. The text defines S_p(m, n) as the penalty for predicting rank n as rank m, but its worked example reads "predicted rank 2, ground truth 0" as S_p(2, 0) = 0.4 §3.3 fig. 4. The full prior matrix is only shown as a figure, and the weight for correct predictions is not given in the text.

Evaluation metrics §4.1.

• Binary COD: MAE, mean F-measure, mean E-measure and S-measure §4.1.
• Fixation: SIM, CC, EMD, KLD, NSS, AUC_Judd, AUC_Borji, shuffled AUC §4.1.
• Ranking: r_MAE, a pixel-wise rank error that penalises an "easiest" sample less when it is predicted as "median" than as "hardest" eq. 5 §4.1.

Equations (numbered in source order; the paper's own labels are the same).

Observation time of observer j on instance i eq. 1:

\triangle t_{ij} = \mathrm{median}(\boldsymbol{\delta t_{ij}}), \qquad \boldsymbol{\delta t_{ij}} = \{ t_{ij}^{k} - t_{j}^{0} \}_{k=1}^{K}

where K is the number of fixation points on instance i, t_j^0 is observer j's start time and t_ij^k is the time of the k-th fixation on instance i §3.1. The final instance delay, not numbered in the paper, is \triangle t_i = \mathrm{median}_j(\triangle t_{ij}) §3.1.

Median of a sorted list x of length n eq. 2:

\mathrm{median}(\boldsymbol{x}) = \begin{cases} x_{(n+1)/2}, & n/2 \neq 0 \\ \dfrac{x_{\lfloor n/2 \rfloor} + x_{\lfloor n/2 \rfloor + 1}}{2}, & n/2 = 0 \end{cases}

Joint objective eq. 3:

\mathcal{L}_{fc} = \mathcal{L}_{f} + \lambda \mathcal{L}_{c}, \qquad \lambda = 1

Ranking training loss eq. 4:

\mathcal{L} = \mathcal{L}_{rpn} + \mathcal{L}_{rank} + \mathcal{L}_{mask}

Rank error eq. 5:

r_{MAE} = \frac{\sum_{i=1}^{w} \sum_{j=1}^{h} |r_{ij} - \hat{r}_{ij}|}{N}

where N is the number of pixels, w and h the image width and height, r_ij and r̂_ij the ground-truth and predicted ranks (0 background, 1 hardest, 2 median, 3 easiest) §4.1.

Pseudocode, joint model (the paper's Method with the released code's reverse-attention placement; see Paper vs code).

# Joint fixation and camouflage forward pass  [§3.2] [fig. 3]
# Input:  image (B, 3, H, W)
# Output: fixation, coarse, refined logits (B, 1, H, W)

def forward(image):
    # Four ResNet-50 stages  [§3.2]
    # [code: joint_cod_col/model/ResNet_models.py#L270-L277]
    stage1, stage2, stage3, stage4 = resnet50_stages(image)
    # Fixation decoder: discriminative-region logits  [§3.2]
    fixation = fixation_decoder(stage1, stage2, stage3, stage4)
    # Camouflage decoder on unguided features (code only)
    # [code: joint_cod_col/model/ResNet_models.py#L280]
    coarse = camouflage_decoder(stage1, stage2, stage3, stage4)
    # Reverse attention 1 - F at stage-2 resolution  [§3.2]
    # [code: joint_cod_col/model/ResNet_models.py#L282]
    reverse = 1 - sigmoid(resize_half(fixation))
    # Code guides stage 2 with a max of smoothed and raw reverse mask
    # [code: joint_cod_col/model/HolisticAttention.py#L34-L38]
    smoothed = normalize(gaussian_blur_31(reverse))
    guided_stage2 = stage2 * maximum(smoothed, reverse)
    # Second-branch stages 3 and 4 on the guided features
    # [code: joint_cod_col/model/ResNet_models.py#L283-L284]
    guided_stage3 = second_stage3(guided_stage2)
    guided_stage4 = second_stage4(guided_stage3)
    # Camouflage decoder on guided features  [§3.2]
    # [code: joint_cod_col/model/ResNet_models.py#L285]
    refined = camouflage_decoder(
        stage1, guided_stage2, guided_stage3, guided_stage4)
    return fixation, coarse, refined

Pseudocode, training loss (released code's terms; the paper gives BCE for the fixation term).

# Pixel-position-aware loss  [§3.2]
# [code: joint_cod_col/train.py#L48-L57]
def structure_loss(logits, mask):
    # Boundary weight 1 + 5 * |avgpool_31(mask) - mask|
    # [code: joint_cod_col/train.py#L49]
    weight = 1 + 5 * abs(avg_pool_31(mask) - mask)
    # Weighted BCE, normalised by the weight sum per image
    # [code: joint_cod_col/train.py#L50-L51]
    wbce = sum(weight * bce(logits, mask)) / sum(weight)
    prob = sigmoid(logits)
    # Weighted IoU term, sums over H and W per image
    # [code: joint_cod_col/train.py#L52-L56]
    inter = sum(prob * mask * weight)
    union = sum((prob + mask) * weight)
    wiou = 1 - (inter + 1) / (union - inter + 1)
    return mean(wbce + wiou)

# Joint training loss, every term weighted 1  [§3.2] [eq. 3]
# [code: joint_cod_col/train.py#L136-L139]
def training_loss(fixation, coarse, refined, fix_gt, mask_gt):
    # Fixation term: MSE on sigmoid output; paper says BCE
    # [code: joint_cod_col/train.py#L136]
    fixation_loss = mse(sigmoid(fixation), fix_gt)
    # One camouflage term per decoder output
    # [code: joint_cod_col/train.py#L137-L138]
    coarse_loss = structure_loss(coarse, mask_gt)
    refined_loss = structure_loss(refined, mask_gt)
    return fixation_loss + coarse_loss + refined_loss

Pseudocode, rank loss (paper only; no ranking code in the repository).

# Rank loss with similarity prior  [§3.3] [fig. 4]
# Index order of similarity_prior is ambiguous; see Pitfalls
def weighted_rank_loss(rank_logits, rank_gt):
    # Per-instance loss before averaging  [§3.3]
    loss = cross_entropy(rank_logits, rank_gt, reduction="none")
    # Penalty for the predicted rank against the true rank  [§3.3]
    penalty = similarity_prior[predicted_rank, rank_gt]
    return mean(penalty * loss)

Hyperparameters

Results

Author-reported throughout. Setup is given per table.

Joint model trained on CAM-FR (2,000 images), tested on benchmark sets tab. 1. Competing SOD models are retrained on the same data.

Discriminative-region localization (fixation), trained on CAM-FR tab. 2.

Camouflage ranking, trained on CAM-FR tab. 3.

Joint model trained on COD10K (3,040) + CAMO (1,000) = 4,040 images, tested on benchmark sets and NC4K tab. 4. Competing methods are salient-object-detection models retrained on the COD dataset §4.1. "Ours_cod_full" is the single camouflaged-object-detection model trained on this set §4.1.

Ablation on the CAM-FR ranking test set tab. 5. "‡" means the value is not produced by that single-task model.

Figures: fig. 5 visualizes fixation, camouflage and rank predictions on easy, medium and hard examples; the paper describes the colour coding (blue = hardest) in its caption. fig. 1 shows the Binary, Fixation and Ranking annotations for the same image.

Paper vs code

Code: https://github.com/JingZhang617/COD-Rank-Localize-and-Segment at 3a50e565ae6e0be9db00402fa918a31fe19a1d8a. The repository has no LICENSE file. The paper's footnote links it §1; Jing Zhang is an author.

What the code contains: the joint fixation-and-camouflage model and its training and test scripts in joint_cod_col/, plus a second model and script in cod/. Nothing implements the ranking branch, the rank loss, or the rank dataset builder: no file mentions rank, Mask R-CNN, RPN or FPN.

1. Learning rate. Paper: 5e-5 with Adam §4.1. Code default: 2.5e-5 train.py.
2. Training length. Paper: 10k iterations §4.1. Code: 50 epochs of 2,000 images, batch 10, three scales per batch, which is three optimizer steps per batch train.py. Derived: 200 batches × 3 × 50 = 30,000 steps. Whether the paper's "iteration" is one optimizer step is not stated.
3. Input size. Paper: 352×352 §4.1. Code: base 480 with random scales 0.75, 1 and 1.25, giving 352, 480 and 608 after rounding to multiples of 32 train.py. Test size 480 test.py.
4. LR schedule. Paper: not stated. Code: adjust_lr multiplies the current learning rate by 0.9^(epoch // 40) after every epoch utils.py train.py. Because the update is *=, the factor applies again on every epoch from 40 to 50. Derived final LR: 2.5e-5 × 0.9^11 ≈ 7.8e-6.
5. Fixation loss. Paper: binary cross-entropy §3.2. Code: MSE on the sigmoid output train.py. The code's BCE object (CE, L44) is unused.
6. Loss terms. Paper: L_f + λ·L_c with λ = 1 for one camouflage term §3.2 eq. 3. Code: fixation MSE + structure loss on both the coarse camouflage output and the refined one, each with weight 1 train.py. The coarse output is not described in the paper.
7. Reverse attention. Paper: multiply 1 − F with the backbone feature s1 §3.2. Code: the HA module multiplies stage-2 features x2 by the element-wise maximum of the normalized 31×31 Gaussian-smoothed (1 − F) and (1 − F) ResNet_models.py HolisticAttention.py. The refined path then recomputes stages 3 and 4 with separate second-branch layers ResNet_models.py.
8. Decoder attention. Paper: dual residual attention (position + channel attention from DANet) in the decoders §3.2 fig. 3. Code: the decoders use RCAB channel-attention blocks ResNet_models.py ResNet_models.py. The PAM (position attention) class is defined at ResNet_models.py but is not instantiated in that file. The separate cod/ model does instantiate PAM modules ResNet_models.py.
9. Multi-scale context module. Paper: DenseASPP fig. 3. Code: a Classifier_Module that sums parallel dilated 3×3 convolutions, with dilations 3, 6, 12, 18 in the decoder blocks and 6, 12, 18, 24 in the output heads ResNet_models.py ResNet_models.py ResNet_models.py.
10. Decoder naming. In code, cod_dec is the fixation decoder and sal_dec is the camouflage decoder ResNet_models.py. The same sal_dec produces both the coarse and the refined camouflage map ResNet_models.py ResNet_models.py.
11. Ranking branch. Paper: Mask R-CNN-style ranking model with rank loss and mask loss §3.3 eq. 4. Code: none.
12. Augmentation. Paper: not stated. Code: horizontal flip, random crop with a 30-pixel border, rotation of ±15° applied with probability 0.2, colour jitter, and salt-and-pepper noise on the ground-truth mask (0.15% of pixels) data.py data.py data.py data.py data.py data.py.
13. Normalization. Paper: not stated. Code: ImageNet mean and standard deviation data.py.
14. Which script made which table. Not stated. The repository does not say which script produced the reported Table 4 "Ours_cod_full" numbers.
15. Fixation maps. The code reads precomputed fixation PNGs from a Fix/ folder data.py. The paper does not state how eye-tracker fixations are rendered into maps.
16. Dataset and checkpoint. The README links external dataset files and says the released dataset and model outputs are hosted on Google Drive [README.md#L6-L51]. The repository itself contains no data.

Consistent: ResNet-50 backbone, 32-channel reduction, Adam, λ = 1 between the fixation and camouflage terms, and the 2,000 / 280 CAM-FR split README.md.

Pitfalls

• The released code cannot reproduce the ranking results. Tables 3 and 5 (ranking rows) and the ranking branch of Table 1 need the ranking model, which is absent. The rank dataset builder is also absent, so implementers must write the ranking labels themselves.
• Rank cut points are not stated. The paper gives the three ranks and their order but not how continuous detection delays are split into ranks 1–3 §3.1.
• Normalization before the hard-sample rule is not stated. The rule that sets a hard sample to 1 "after normalization" does not say what normalization is used §3.1. "More than half of the observers" with six observers means four or more ignoring the instance; the text does not say so explicitly.
• MAE for ranking is not defined. Table 3 and Table 5 report MAE next to r_MAE, but only r_MAE is defined eq. 5 §4.1. The MAE definition is not stated.
• r_MAE uses all pixels. Eq. 5 divides by the total pixel count N, so background pixels (rank 0) count in the average eq. 5.
• Table 2 and Table 5 disagree on EMD for the same joint model. Table 2 reports EMD 3.361 and Table 5 reports 3.398 for "Ours" fixation. The text does not say whether the two use the same test set.
• Ranking-only vs joint. Table 5 shows the ranking-only model with a lower MAE (0.046) than the joint model (0.049), yet the text calls them "similar". Treat the joint model's ranking advantage as unproven from this table.
• Similarity-prior indices. The text defines S_p(m, n) as the penalty for predicting n as m, but the worked example S_p(2, 0) uses predicted 2 and truth 0 §3.3. The two readings are transposes of each other, and the matrix is only shown as a figure fig. 4. The diagonal (correct-prediction) weight is not stated.
• Table 4 competitors may not be retrained. The text says competing SOD models were retrained §4.1, and SINet is said to be the only code-available camouflage model. It does not say whether SINet was retrained or taken from its release.
• Compounding learning-rate decay. The code's decay compounds from epoch 40 on, which is probably not the intent of the decay_epoch argument. An implementer copying the schedule gets roughly 7.8e-6 at the end, not 2.5e-5 × 0.9 = 2.25e-5 (derived).
• Tables are not comparable across settings. Table 1 and Table 5 use CAM-FR training; Table 4 uses the standard 4,040-image training set. The NC4K and COD10K numbers from different tables must not be compared directly.
• Reverse attention location. Paper and code differ on which feature is guided (s1 vs stage-2 x2 with a max-combined mask). An implementation that follows only the paper will not match the released model.
• Licensing of the data. The repository does not state a licence for the CAM-FR or NC4K images. NC4K images come from the internet §1. This card does not use or redistribute any dataset.

Lineage

• Builds on: Mask R-CNN, FPN and RPN for the ranking branch §3.3 fig. 2; DANet dual attention for position and channel attention §3.2 fig. 3; DenseASPP as the context module fig. 3; a reverse-attention scheme similar to CPD §3.2; the F3Net pixel-position-aware loss §3.2; CAM-style discriminative-region localization §2.
• Compared against: SINet as the only deep camouflage baseline with code §4.1 tab. 4; salient-object-detection models SCRN, CSNet, UCNet, BASNet, PiCANet, CPD, PoolNet, F3Net, ITSD, NLDF, EGNet and SSAL tab. 1 tab. 4; ranking baselines SOLOv2, MS-RCNN and RSDNet tab. 3.
• Superseded by: not stated in the paper.