Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers

In plain words

Masking one object is hard when another object stands in front of it. The mask head sees one blob with one outline and cannot tell which edge belongs to the front object and which edge belongs to the object behind it.

BCNet treats the crop around each detected object as two stacked layers. The top layer works out what stands in front, including where that object's edge lies. The bottom layer works out the object behind, and it receives the top layer's answer as a hint, so it can draw its outline through the hidden area instead of stopping at the front object's edge. Both layers are trained together, and the design plugs into existing detectors.

TL;DR

• BCNet is a mask head for detect-then-segment models. The main experiments use FCOS, and a Faster R-CNN variant is also reported §3.1 §4.2.
• Each detected region is split into two graph layers. The occluder layer predicts the front object's masks and contours. The occludee layer, guided by it, predicts the partly occluded object §3.2.
• Each graph layer is a non-local (dot-product) graph convolution over the 14×14 region feature §3.2 §4.2.
• The paper reports 39.8 mask AP on COCO test-dev with Faster R-CNN and ResNet-101 tab. 8. Its ablation shows larger gains on the overlap-heavy COCO-OCC subset than on COCO tab. 1.
• The official code is an MIT-licensed detectron2 fork. It differs from the paper in several numerical and architectural details, listed under Paper vs code.

Problem

A two-stage mask head regresses each instance mask from one region feature with a single fully convolutional branch. The region is treated as one layer §1 fig. 3. When objects overlap, the front object's pixels and outline end up inside the region, and the mask head cannot separate the front outline from the true boundary of the object behind it. The paper reports that many errors on COCO come from overlapping objects, especially of the same class §1.

Amodal segmentation methods predict complete masks, including hidden parts, but they regress only one occluded target per region and do not model the interaction with the occluder §1 §2.

Method

Overview §3.1 fig. 4

The model has three parts: a backbone with an FPN for region features; a detection head that proposes boxes, with FCOS as the default detector and the statement that any fully supervised detector can be used §3.1; and the BCNet mask head that segments each proposed box from its cropped region feature §3.1.

Inside the mask head, the region feature first goes through the occlusion branch. This branch is a first graph layer followed by two convolution layers (an FCN). Its output is added back to the region feature by a residual connection. The second graph layer then segments the occludee from this occlusion-aware feature §3.1.

Graph convolution block §3.2 eq. 1 eq. 2 eq. 3

Each graph layer takes a feature X of size N×K, where N = H×W is the number of pixels in the region and each pixel is one graph node §3.2. The layer computes Z = σ(A X W_g) + X eq. 1. The output dimension K' equals K in the paper's setting §3.2. The nonlinearity σ is described as layer normalization followed by ReLU §3.2.

The adjacency matrix A is built from pairwise dot products of two learned 1×1 projections, θ(x_i) and φ(x_j), passed through a softmax eq. 2 eq. 3. The paper follows the non-local operator of Wang et al. for this block §3.2.

Bilayer occluder-occludee structure §3.2 eq. 4 eq. 5 eq. 6

The two graph layers are written as follows §3.2:

• First (occluder) layer on the region feature X_roi: Z⁰ = σ(A⁰ X_roi W⁰_g) + X_roi eq. 6.
• Fusion: X_f = Z⁰ W⁰_f + X_roi, where W⁰_f are the weights of the FCN eq. 5.
• Second (occludee) layer on the fused feature: Z¹ = σ(A¹ X_f W¹_g) + X_f eq. 4.

The paper motivates this as making the occluder explicit, so that the overlap between the two layers can mark an occlusion boundary §3.2.

The occluder branch predicts a mask and a boundary from its features. It uses a 3×3 convolution, one GCN layer, one FCN layer, an upsampling layer and a 1×1 convolution §3.2. The occludee branch uses the second graph layer, and its boundary and mask losses use the occludee targets §3.3.

Occluder and occludee outputs §3.2 §3.3

• Occluder boundary loss: a binary cross-entropy against the occluder boundary, which is "readily computed from mask annotations" §3.2 eq. 7.
• Occluder mask loss: a binary cross-entropy against the occluder mask annotation §3.2 eq. 8.
• Occludee boundary and mask losses have the same form, with the occludee targets §3.3 eq. 11.

The paper does not define the boundary width. The code uses a two-pixel band (see Pitfalls).

Total loss §3.3 eq. 9 eq. 10 eq. 11 eq. 12

The training loss is L = λ₁ L_Detect + L_Occluder + L_Occludee eq. 9. Here L_Occluder = λ₂ L′_Occ-B + λ₃ L′_Occ-S eq. 10, L_Occludee = λ₄ L_Occ-B + λ₅ L_Occ-S eq. 11, and L_Detect = L_Reg + L_Centerness + L_Class is the FCOS detection loss eq. 12. The weights are λ₁…λ₅ = {1, 0.5, 0.25, 0.5, 1.0}, tuned on a validation set §3.3.

Pseudocode: bilayer mask head

# Bilayer mask head for one region  [§3.2] [eq. 4-6]
# Input:  roi_feature (B, C, 14, 14), C = 256  [§4.2]
# Output: occludee and occluder masks and boundaries, each (B, 28, 28) logits

def bilayer_mask_head(roi_feature):
    # Occluder branch: two 3x3 convs  [§3.2]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L371-L373]
    occluder = run_convs(occluder_convs[:2], roi_feature)
    # First graph layer models the occluder  [§3.2] [eq. 6]
    occluder = graph_block(occluder, occluder_graph)
    # Two more convs form the FCN  [§3.1]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L374-L400]
    occluder = run_convs(occluder_convs[2:], occluder)
    # Occlusion-aware feature: add the region feature back  [§3.2] [eq. 5]
    occlusion_aware = roi_feature + occluder
    # Occludee branch: two convs, second graph layer, two convs  [§3.2] [eq. 4]
    occludee = run_convs(occludee_convs[:2], occlusion_aware)
    occludee = graph_block(occludee, occludee_graph)
    occludee = run_convs(occludee_convs[2:], occludee)
    # Each output: 2x2 transposed conv (x2 upsampling), ReLU, 1x1 predictor
    # [code: detectron2/modeling/roi_heads/mask_head.py#L293-L317]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L440-L450]
    occludee_mask = mask_head_out(occludee)
    occludee_boundary = boundary_head_out(occludee)
    occluder_mask = mask_bo_head_out(occluder)
    occluder_boundary = boundary_bo_head_out(occluder)
    return occludee_mask, occludee_boundary, occluder_mask, occluder_boundary

Pseudocode: graph block

# Non-local graph block over N = H * W nodes  [§3.2] [eq. 1-3]
# Input: features (B, C, H, W)
# Output: features of the same shape, with a residual connection

def graph_block(features, layer):
    # One graph node per region pixel  [§3.2]
    batch, channels, height, width = features.shape
    # Queries and keys from the 1x1 projections theta and phi  [eq. 3]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L378-L382]
    queries = layer.theta(features).reshape(batch, channels, -1)
    keys = layer.phi(features).reshape(batch, channels, -1)
    # A value projection, not written in the paper
    # [code: detectron2/modeling/roi_heads/mask_head.py#L384-L386]
    values = layer.value(features).reshape(batch, channels, -1)
    # Scaled dot products between all node pairs  [eq. 2]
    # The 1/sqrt(C) scale is in the code only
    # [code: detectron2/modeling/roi_heads/mask_head.py#L336]
    affinity = matmul(transpose(queries), keys) / sqrt(channels)
    # Row-wise softmax over the keys gives the adjacency A  [eq. 2]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L389]
    adjacency = softmax(affinity, axis=-1)
    # Aggregate neighbour values with A  [eq. 1]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L391-L393]
    propagated = matmul(adjacency, transpose(values))
    # Back to a feature map
    # [code: detectron2/modeling/roi_heads/mask_head.py#L393-L395]
    propagated = transpose(propagated)
    propagated = reshape(propagated, (batch, channels, height, width))
    # Output projection, then BatchNorm; the paper says LayerNorm and ReLU
    # [code: detectron2/modeling/roi_heads/mask_head.py#L397-L398]
    propagated = batch_norm(layer.output(propagated))
    # Residual connection  [eq. 1]
    # [code: detectron2/modeling/roi_heads/mask_head.py#L400]
    return features + propagated

Pseudocode: loss

# Total training loss with the paper's weights  [§3.3] [eq. 9-12]
# The released code uses different weights; see Paper vs code.

def total_loss(detect, occluder, occludee):
    # FCOS detection loss: regression, centerness, class  [eq. 12]
    detect_loss = detect.reg + detect.centerness + detect.cls
    # Occluder: boundary weight 0.5, mask weight 0.25  [eq. 10]
    occluder_loss = 0.5 * occluder.boundary + 0.25 * occluder.mask
    # Occludee: boundary weight 0.5, mask weight 1.0  [eq. 11]
    occludee_loss = 0.5 * occludee.boundary + 1.0 * occludee.mask
    # Multi-task sum  [eq. 9]
    return 1.0 * detect_loss + occluder_loss + occludee_loss

Pseudocode: boundary target

# Boundary target from a binary region mask  [§3.2]
# The paper gives no width; the code uses a two-pixel band.
# [code: detectron2/layers/boundary.py#L12-L20]

def boundary_target(mask):
    # Grow the mask by one pixel
    # [code: detectron2/layers/boundary.py#L15]
    outer = dilate(mask, pixels=1)
    # Shrink the mask by one pixel
    # [code: detectron2/layers/boundary.py#L18]
    inner = erode(mask, pixels=1)
    # The pixels where the two differ form the band
    # [code: detectron2/layers/boundary.py#L19]
    return outer != inner

Training §3.3

Losses are computed on positive proposals only fcos.py. The paper says the first graph layer is trained with occlusion cases making up 50% of the sampled proposals, to balance the small share of partially occluded objects in COCO §3.3. The optimizer, schedule and input sizes are in the Hyperparameters table.

For the synthetic occlusion dataset, the paper creates 100k images with a uniform class distribution across the 80 COCO categories. Each image has complete contours for both the occluder and the occludee §4.1. Its generation process is described in a supplementary file, which is not part of the arXiv source used here.

Inference §3.3

The mask head runs on up to 50 high-scoring boxes from the FCOS detector §3.3. The occludee mask is the output used for evaluation mask_head.py.

Hyperparameters

Values as stated in the paper, with the released code's value where the paper is silent or differs. "Not stated" means the paper does not give the value.

Results

All numbers are author-reported. Setups are given in each table's caption and text.

Ablation: explicit occluder modelling (ResNet-50-FPN, FCOS, COCO and COCO-OCC) tab. 1

Ablation: occludee guidance tab. 2

Ablation: bilayer vs single layer, GCN vs FCN tab. 3

Detector comparison tab. 4

COCOA (amodal), AP_all tab. 5

KINS (amodal) tab. 6

COCO-OCC split tab. 7

COCO test-dev, single model, trained on train2017 tab. 8

Text claims, as stated in the paper: joint occluder modelling raises COCO AP from 32.65 to 33.43 and COCO-OCC AP from 29.04 to 30.37 §4.2. Bilayer GCN raises COCO-OCC AP from 29.63 to 30.68 over a single GCN §4.2. The COCO-OCC result with synthetic data is 32.89 AP §4.3 tab. 7.

Paper vs code

Code: github.com/lkeab/BCNet at commit d6580e8a2a0b5e71c0ae6913ed0340c101d35723 (MIT). The repository is a detectron2 fork built on FCOS. The mask head is in detectron2/modeling/roi_heads/mask_head.py. I read the mask head, the FCOS forward and loss, the default config and the FCOS configs. I did not run training or evaluation.

1. Convolutions around the first graph layer. The paper says the occluder module is one 3×3 conv, then one GCN, then one FCN §3.2, and its overview says the FCN has two convs §3.1. The code's default is NUM_CONV: 4, with two convs, the GCN, then two convs, in each branch Base-FCOS.yaml mask_head.py mask_head.py.
2. Normalization and nonlinearity in the graph layer. The paper uses layer normalization and ReLU §3.2. The code uses BatchNorm2d (eps 1e-4) and no ReLU mask_head.py mask_head.py.
3. Scaling of the affinity. The code divides the dot products by sqrt(C) mask_head.py mask_head.py. Eq. 2 has no scale.
4. Projections. The code has four 1×1 convs per graph layer (query, key, value, output), all C→C with no channel reduction, and no bias mask_head.py. The paper names θ, φ and W_g only.
5. Zero initialization. A code comment says the BatchNorm "blocker" layers should be zero-initialized, but they are not in the init list and keep the default initialization mask_head.py mask_head.py. The paper says nothing about this.
6. Detection loss weight. The paper uses λ₁ = 1 for L_Detect §3.3. The code multiplies the FCOS regression loss by 1.25 and leaves the centerness and class losses at 1 fcos.py.
7. Boundary loss counted twice. The boundary loss is JointLoss(BceLoss, BceLoss), which sums two identical BCE terms mask_head.py joint_loss.py other_losses.py. The code then multiplies it by 0.5 fcos.py. So the effective weight on one BCE is 1.0, not the paper's 0.5. The occluder boundary has the same pattern mask_head.py.
8. Other loss weights match. Occluder mask 0.25, occludee mask 1.0, and the two boundary weights of 0.5 match fcos.py §3.3.
9. Warm-up. The paper says "constant" warm-up §3.3. The code default is linear defaults.py.
10. Optimizer details not in the paper. Momentum 0.9, weight decay 1e-4, LR steps at 60K and 80K with factor 0.1 come from the code defaults.py Base-FCOS.yaml.
11. Boxes at inference. The paper says up to 50 boxes §3.3. The post-processor's cap is TEST.DETECTIONS_PER_IMAGE, defaulting to 100 defaults.py inference_fcos.py. None of the three configs I read overrides it.
12. Leaderboard schedule. The paper gives 90K iterations §3.3. The released leaderboard configs use 180K (2×) or 270K (3×) iterations with other steps Base-FCOS-test-2x.yaml Base-FCOS-test-3x.yaml. The paper does not say which schedule its leaderboard numbers used.
13. Faster R-CNN path looks inconsistent. The Faster R-CNN branch calls mask_rcnn_loss(mask_logits, proposals) with two arguments roi_heads.py roi_heads.py. The function imported there is the five-argument BCNet version roi_heads.py mask_head.py. I did not run it, but the call as written does not match the definition. The Faster R-CNN results in tab. 4, tab. 7 and tab. 8 cannot be checked against this code without a fix.
14. Occluder annotations come from outside this repo. The loss reads a bg_object_segmentation field from each annotation detection_utils.py detection_utils.py. The repository README asks users to download converted mask annotations. The repo does not generate them.
15. Text vs table, FCOS gain. The text says FCOS gains 2.23 AP on COCO-OCC §4.2. The table gives 30.68 − 28.43 = 2.25 tab. 4. The Faster R-CNN gain of 2.04 matches tab. 4.
16. Text vs table, leaderboard margins. The text says BCNet beats PANet and Mask Scoring R-CNN by 1.5 AP with Faster R-CNN §4.3. The 1.5 margin matches the ResNet-101 Mask Scoring R-CNN row. The PANet margin on ResNet-50 is 1.8 tab. 8.
17. Not in the code read. The 50% occlusion sampling §3.3 and the synthetic dataset generator are not in the parts I read.

Searched for code: the abstract links github.com/lkeab/BCNet, and the owner matches an author, so this is the official code.

Pitfalls

• Occluder definition. The paper does not say which annotated object is an occluder or how its mask is made §3.2. The code reads it from an external per-instance field, so the definition lives in that annotation file.
• Boundary width and resolution. The paper gives no width. The code makes a two-pixel band with 1-pixel dilation and erosion. The band is computed per instance, on the CPU, on the 28×28 cropped mask, with numpy and scikit-image every training step mask_head.py. Expect this to be slow. The weight map returned alongside it is never used.
• Double-counted boundary loss. A reimplementation that follows the paper's formula will train the boundary head at half the strength of the released code (see Paper vs code, item 7).
• Loss weights were tuned on a validation set, which the paper does not name §3.3.
• Output of the second layer. The occludee mask is the one used for evaluation mask_head.py. The code still runs the occluder heads at test time, which costs compute.
• Overlap-specific data. The COCO-OCC split uses 1,005 validation images with box overlap ≥ 0.2 §4.1. The released split script hard-codes a local path process_dataset_occ.py.
• Synthetic data. The "+ Synthetic" rows tab. 7 depend on a 100k-image dataset whose generation is in a supplement not included in the arXiv source. This card cannot reproduce them.
• Numbers depend on the schedule. Ablations use 90K iterations; leaderboard numbers may use longer schedules (see Paper vs code, item 12).
• Citation numbers. Equation, table and figure numbers were counted from the LaTeX source. The compiled PDF was not checked. Table numbers: 1–4 ablations, 5–7 amodal and COCO-OCC, 8 COCO test-dev. Figure numbers: 1 teaser, 2 COCO comparison, 3 mask-head comparison, 4 architecture, 5 COCOA, 6 KINS, 7 qualitative results.
• Figures. No figures are reproduced here. The architecture is in fig. 4, and the mask head comparison is in fig. 3.

Lineage

• Builds on: Mask R-CNN for the two-stage mask-head design §1 §2; FCOS as the default detector §3.1; Faster R-CNN as the second detector §4.2; the non-local operator of Wang et al. for the graph block §3.2; and graph convolution networks, cited as the general idea §3.2.
• Compared against: Mask R-CNN, PANet, Mask Scoring R-CNN, CenterMask, HTC, TensorMask, ShapeMask, BlendMask, AmodalMask, AmodalMRCNN, ORCNN and ASN tab. 5-tab. 8.
• Superseded by: not known from this wiki.