Anabranch Network for Camouflaged Object Segmentation

In plain words

Some animals and soldiers are built to look like their background, so a standard saliency model, which looks for the thing that stands out, tends to miss them. Most segmentation models also assume the object they look for is always there.

The paper makes two contributions. The first is a new collection of images with camouflaged objects, each annotated pixel by pixel, which gives the field a benchmark it lacked. The second is a network with two parts. One part draws a map of where the object might be. The other part answers a yes-or-no question: is there camouflage in this picture at all? The yes-or-no answer scales the map, so the aim is that a picture without camouflage yields no segmentation.

The authors attach this design to four existing saliency networks and report higher scores than those networks alone, on their own test sets.

TL;DR

Le, Nguyen, Nie, Tran and Sugimoto introduce the CAMO dataset of camouflaged objects and the Anabranch Network (ANet). ANet pairs any fully convolutional segmenter with an image-level camouflage classifier, and multiplies the classifier's probability into the segmentation map §4.2. The results are author-reported, the test sets are small (250 images each), and the paper's own code release does not contain the network definitions needed to check the classifier.

Problem

Camouflaged objects hide their texture in the background, so features that help detect salient objects work against them §1 §2.2. Before this paper there was no public dataset with both training and test data for camouflaged object segmentation §2.1 §3.1. Existing saliency segmenters assume a salient object is always present, while a camouflaged object may be absent from an image §2.2 §4.1. Such a method must therefore first decide whether camouflage is present §4.1.

Method

Dataset. The CAMO dataset has 1,250 images, each with at least one camouflaged object, drawn from 3,000 collected images after removing low resolution and duplicates §3.2. It covers natural camouflage (animals) and artificial camouflage (people such as soldiers and body paint) §3.3 fig. 3. Three annotators labelled every image with a custom interactive tool §3.2. The images are annotated for six challenge attributes: object appearance, background clutter, shape complexity, small object (camouflaged area under 0.1 of the image), occlusion and distraction. About 8% of images have multiple objects §3.3 fig. 4.

CAMO-COCO adds 1,250 non-camouflaged images from MS-COCO, each with an all-zero mask §5.1 fig. 6. Each set is split 80/20 at random, giving 1,000 training and 250 test images per set §5.1 tab. 2.

Network. ANet has two streams that take the same image §4.2 fig. 5.

• Segmentation stream. Any end-to-end FCN that maps an image to a map. The authors use four saliency FCNs: DHS, DSS, SRM and WSS §5.3.
• Classification stream. Three fully connected layers on top of the FCN's convolutional features. The first two each have a ReLU and 50% dropout, and the last is a softmax over two classes (camouflaged or not), trained with cross-entropy §4.2 tab. 1. The hidden width conflicts between the text (4096) and the table (2048); see Pitfalls.
• Fusion. The classifier's probability, a single scalar, multiplies every pixel of the segmentation map §4.2.

Training. The paper trains in two steps. First the segmentation stream is fine-tuned on CAMO. Then the segmentation parameters are fixed and the classification stream is trained on CAMO-COCO §5.3. A joint variant sums the two losses eq. 2 and trains both streams on CAMO-COCO for 10k iterations. The authors report that joint training does not help consistently §5.5.2 tab. 6.

Evaluation. Three metrics: F-measure with β² = 0.3 eq. 1, intersection over union (IoU) and mean absolute error (MAE) §5.2. Two binarisation settings are used. Where every image is assumed to contain camouflage, the threshold is the map's mean plus its standard deviation. Where presence is not guaranteed, the threshold is 0.5. MAE uses the raw map §5.2. Comparison baselines are each FCN retrained on CAMO or CAMO-COCO with similar hyperparameters §5.3.

F_\beta = \frac{(1+\beta^2)\,\mathrm{Precision} \times \mathrm{Recall}}{\beta^2 \times \mathrm{Precision} + \mathrm{Recall}} eq. 1

\mathcal{L} = \mathcal{L}_{\rm seg}(\hat{y}_{\rm seg}, y_{\rm seg}) + \mathcal{L}_{\rm cls}(\hat{y}_{\rm cls}, y_{\rm cls}) eq. 2

Forward pass, as the text describes it:

# Anabranch network forward pass  [§4.2] [fig. 5]
# Input:  image (H, W, 3)
# Output: camouflage map (H, W), scaled by the classifier

def anet_forward(image):
    # Segmentation stream: any FCN returning a map at image size  [§4.2]
    seg_map = segmentation_fcn(image)                   # (H, W)
    # Classification stream: fully connected layers on FCN features  [§4.2]
    # Which conv layer is tapped, and how it is flattened: not stated
    features = conv_features(image)                     # not stated
    hidden_1 = dropout(relu(fc_1(features)), rate=0.5)  # width 4096 or 2048
    hidden_2 = dropout(relu(fc_2(hidden_1)), rate=0.5)  # width 4096 or 2048
    # Two-way softmax; index of the camouflaged class: not stated  [tab. 1]
    class_probs = softmax(fc_3(hidden_2))               # (2,)
    camo_prob = class_probs[camouflaged_index]          # scalar
    # Fusion: multiply the scalar into every pixel of the map  [§4.2]
    return camo_prob * seg_map

Two-step training, as run in the reported experiments:

# Two-step training  [§5.3]
# Both steps: SGD, momentum 0.9, weight decay 0.0005, batch size 2  [§5.3]

def train_anet(segmentation_fcn, classifier, camo_set, camo_coco_set):
    # Step 1: fine-tune the segmentation stream on CAMO only  [§5.3]
    # Learning rate 1e-4 for 10 epochs, stated as 5k iterations  [§5.3]
    fine_tune(segmentation_fcn, camo_set, lr=1e-4, iterations=5000)
    # Step 2: fix the segmentation parameters  [§5.3]
    freeze(segmentation_fcn)
    # Learning rate 1e-6 for 3 epochs, stated as 3k iterations  [§5.3]
    train(classifier, camo_coco_set, lr=1e-6, iterations=3000)

Binarisation for scoring:

# Binarise a camouflage map before scoring  [§5.2]
# MAE is computed on the raw map instead  [§5.2]

def binarize(camo_map, always_present):
    # Context 1: every image is assumed to contain camouflage  [§5.2]
    if always_present:
        # Threshold = mean + standard deviation of the map  [§5.2]
        threshold = camo_map.mean() + camo_map.std()
    # Context 2: presence is not guaranteed  [§5.2]
    else:
        # Fixed threshold  [§5.2]
        threshold = 0.5
    # Strict or non-strict comparison: not stated
    return camo_map > threshold

Hyperparameters

Results

Author-reported. Test sets: CAMO has 250 test images, CAMO-COCO has 500 (250 camouflaged, 250 not). Each block gives MAE (lower is better), then F-beta and IoU (higher is better) under the adaptive and the fixed threshold tab. 3.

FT = fine-tuned. Bold rows are ANet. The table's colour coding of first and second places is not reproduced here.

Classification accuracy on CAMO-COCO, % tab. 4:

Average wall-clock time per image, milliseconds tab. 5:

Joint training, from tab. 6 (adaptive threshold for CAMO and CAMO-COCO shown; fixed-threshold columns are in the paper):

Headline reading (author-reported): among the baseline rows, ANet-SRM and ANet-WSS have the highest F-measure on CAMO-COCO (0.826), and ANet-SRM has the highest IoU (0.732) tab. 3. On CAMO, the paper says ANet is expected to be comparable with the FCN fine-tuned on CAMO, and on CAMO-COCO it can beat that FCN §5.4.2. The authors also report a t-test at the 90% level, which they say favours ANet on CAMO-COCO §5.4.2. Adding the classifier raises the per-image time by 3 to 10 ms in the table tab. 5.

Paper vs code

The repository is ltnghia/ANet, linked from the authors' project page §1. Its owner handle matches the project page's ltnghia path, which belongs to the first author. It contains a Caffe tree with three custom data layers, but not the network definitions, solver files, the fusion step, the classifier loss, or the dataset lists. Things the paper describes that cannot be checked in the code:

• Classifier widths (text 4096, table 2048) §4.2 tab. 1.
• The fusion (multiplication) and how the classifier output enters it.
• Learning-rate schedule, iteration counts, and the input size.

What the data layers do camourflage_data2_layer.cpp:

• Each list line gives an image, a segmentation file, an integer label and an optional saliency file camourflage_data2_layer.cpp. The paper does not describe a saliency-map input. Whether it is used in the reported experiments is not stated.
• The image-level label is copied into the classifier output blob camourflage_data2_layer.cpp.
• Random horizontal flip happens with probability 0.5 when the transform enables mirroring data_transformer.cpp. This matches the paper's random flip §5.3.
• The layer refuses a mean file camourflage_data2_layer.cpp. The paper does not mention mean subtraction, so there is no conflict.
• The three data layers (camourflage_data2, _data3, _data4) have the same logic and differ only in class names and one debugging block camourflage_data3_layer.cpp. Which layer each stream uses is not stated.

The README says "The code is used for academic purpose only", and the repository has no top-level LICENSE file. The bundled Caffe copy has its own LICENSE. No code is quoted in this card; all code references are pointers.

Pitfalls

• Classifier width conflict. Text says 4096 and 4096 hidden units; Table 1 says 2048 and 2048 §4.2 tab. 1. Neither is confirmed by the code here.
• Classifier input. The paper does not say which convolution layer feeds the classifier, how the feature map is reduced to a vector, or which softmax index is "camouflaged".
• Fusion point. The paper does not say whether the multiplication uses the segmentation map before or after its output activation, nor whether the classifier output is thresholded first §4.2.
• Empty ground truth. Images in CAMO-COCO have all-zero masks. F-measure and IoU are undefined when both the prediction and the mask are empty. The paper does not state a convention. A commented-out passage in the LaTeX source proposed setting one pixel in both maps; it is not in the published text, so do not assume it.
• Averaging. The paper does not say whether F-measure and IoU are computed per image and averaged or pooled over the dataset.
• Threshold details. The adaptive threshold uses the map's mean plus its standard deviation, but the map's value range (0 to 1 or 0 to 255) is not stated. The comparison operator is not stated.
• Statistical test. The t-test at the 90% level is named, but the test variant and pairing are not stated §5.4.2. The claim that ANet "significantly outperforms" rests on it.
• Small test sets. CAMO has 250 test images. Differences of a few hundredths of F-measure may be within noise, and the random split's seed is not stated §5.1.
• Split release. The paper says the CAMO-COCO split "will be available" §5.1. Reproduce with your own random 80/20 split and report it.
• Input size and crop. Images are rescaled to "the resolution of the specific FCN" §5.3. The sizes are not given in the text. No crop size is stated.
• Pre-trained weights. The paper cites MSRA, DUT-OMRON and DUTS as sources of the pre-trained saliency models §5.4, but does not say which checkpoint each FCN uses.
• Joint training is mixed. Joint training lowers DHS and WSS and raises DSS and SRM §5.5.2 tab. 6. The authors say joint training may not help and leave it for future work §5.5.2.
• Classifier uses frozen features. Step 2 trains the classifier while the segmentation parameters stay fixed §5.3, so the classifier's features come from a stream that is not updated.
• Dataset licence. This card's CC BY 4.0 licence covers the paper, not the CAMO images or annotations. The paper does not state a licence for them, and the images were collected from the Internet by keyword search §3.2. Check the dataset's own terms before redistributing images.
• Figure and table numbers are taken from LaTeX source order, not from the compiled PDF. Check them against the PDF before citing numbers in print.

Lineage

• Builds on: the saliency FCNs DHS, DSS, SRM and WSS, used unchanged as the segmentation stream §5.3. Two-stream networks are reviewed in §2.3, but the paper's streams differ in architecture and task.
• Compared against: each FCN fine-tuned on CAMO or CAMO-COCO tab. 3. For classification, SVM with bag-of-words, AlexNet and VGG-16 tab. 4.
• Superseded by: not stated.