CBNet: A Composite Backbone Network Architecture for Object Detection

In plain words

An object detector is only as good as the image features its backbone produces, and the usual way to get better features is a bigger network, which costs more to train and to run. This paper asks whether you can get more out of a backbone you already have, such as a ResNet or Swin model pretrained on ImageNet.

The answer it proposes is to run several copies of the same pretrained backbone side by side, in a chain. Each copy passes its intermediate features forward to the next copy, so the last copy sees a mix of what the earlier copies computed. Only the last copy's features go to the detector's head. The earlier copies also get their own detection branch during training, so they learn useful features too. Those extra heads are thrown away at test time.

The point is that the copies reuse weights you already have, so the paper does not need a new large pretraining run. The cost is that the chain still runs every copy at test time, so the network is not free to run.

TL;DR

• CBNet groups K identical pretrained backbones (K ≥ 2), linked by composite connections that pass each stage's output to the next backbone's stages §3.1.
• The main variant, CBNetV2, uses dense higher-level composition, auxiliary supervision on the assisting backbones, and a pruning option §3.5.
• It is author-reported to beat the same backbone alone by about 3.4 to 3.5 AP on Faster R-CNN, and it is claimed to beat wider and deeper backbones at similar cost §4.3.1.
• The paper reports 59.4 box AP and 51.6 mask AP on COCO test-dev (single scale) with Swin-L and HTC in 12 epochs §4.2.2.
• Relevant if you already have a pretrained backbone and want more accuracy without new ImageNet pretraining. Check the compute cost: the composite connections add FLOPs.

Problem

Detectors gain accuracy from stronger backbones, but designing a new backbone takes expertise and many trials, and pretraining a large one on ImageNet is expensive. Training a detector from scratch avoids pretraining but needs more compute and skill §1. The paper targets the case of reusing existing open-source pretrained backbones under the pretrain-then-fine-tune recipe, without additional pretraining of the composite §1.

Method

Architecture of CBNet

CBNet has K identical backbones, K ≥ 2. The case K = n is written CB-Backbone-Kn and the suffix is dropped for K = 2 §3.1. The last backbone B_K is the lead. The others B_1 … B_{K-1} are assisting §3.1 fig. 1.

Each backbone has L stages, usually L = 5. A standard network computes the stage output x^l = F^l(x^{l-1}) eq. 1. In CBNet, each stage of backbone k also receives a composed copy of the previous backbone's features eq. 2:

x_k^l = F_k^l\big(x_k^{l-1} + g^{l-1}(\boldsymbol{x_{k-1}})\big), \quad l \ge 2,\; k = 2, \ldots, K eq. 2

Here g^{l-1} is the composite connection. It takes all stage outputs of backbone k−1 and returns a tensor with the same size as x_k^{l-1} §3.1. The stage-1 outputs x_1^1 … x_K^1 are weight-shared §3.1. For detection, only the lead's stage outputs x_K^i (i = 2 … L) go to the neck and head. The assisting outputs go only to the next backbone §3.1. The assisting backbones can be any pretrained model, such as ResNet, ResNeXt, Res2Net or Swin, and each can be initialised from the same single pretrained checkpoint §3.1.

Composite strategies

The paper defines five ways to write g §3.2. U is up-sampling and D is down-sampling. w is defined as a 1×1 convolution plus batch normalisation only for SLC; the other strategies reuse the symbol without defining it §3.2.1.

• Same-level (SLC): g^l(x) = w(x^l), l ≥ 2 §3.2.1. Uses only the matching stage.
• Adjacent higher level (AHLC): g^l(x) = U(w(x^{l+1})), l ≥ 1 §3.2.2. This is the composition in the earlier CBNet paper §3.5.
• Adjacent lower level (ALLC): g^l(x) = D(w(x^{l-1})), l ≥ 2 §3.2.3.
• Dense higher level (DHLC): g^l(x) = Σ_{i=l+1}^{L} U(w_i(x^i)), l ≥ 1 §3.2.4 eq. 6. Sums every higher-level stage of the previous backbone.
• Fully connected (FCC): sums resized features from all stages of the previous backbone, using D when the source stage is deeper than the target and U when it is shallower §3.2.5 eq. 7.

The recommended choice is DHLC, which the paper picks for its computational simplicity §4.6.1. Its proposed form is:

g^l(x) = \sum_{i=l+1}^{L} U\big(w_i(x^i)\big), \quad l \ge 1 eq. 6

Index convention: eq. 6 writes g^l for the input of stage l+1, so the sum starts at l+1. In the released code the sum for the input of each stage includes that stage's own level (see Paper vs code).

Auxiliary supervision

Assisting backbones also get a detection branch during training. The paper applies the same neck and head weights to each backbone's output, so the two branches share parameters §3.3 fig. 4. The total training loss is:

\mathcal{L} = \mathcal{L}_{\mathrm{Lead}} + \sum_{i=1}^{K-1} \lambda_i \cdot \mathcal{L}_{\mathrm{Assist}}^{i} eq. 8

λ_i weights the i-th assisting backbone §3.3. At inference the auxiliary branch is dropped and only the lead's features are used, so the paper says it does not change inference speed §3.3. The assisting backbones still run, because the composite connections need their features (see Pitfalls).

Pruning

To cut cost, the paper prunes stages in backbones 2 … K instead of composing them in full §3.4. It names the configurations s_i, where i is the number of stages kept in backbones 2 … K: stages j with 6 − i ≤ j ≤ 5 are kept. Pruned stages are filled with the features of the same stage from the first backbone §3.4 fig. 5. The paper's default for CBNetV2 is s_3 §4.6.5. The paper argues pruning costs little accuracy because the pruned stage's weights are fixed during detector training. It says s_3 has about 10 % fewer FLOPs than s_4 with a 0.1 AP drop §4.6.3, but this comparison appears only in a figure fig. 6.

Detection network

CBNet attaches to any detector without changing its head. The lead backbone's outputs go into an FPN and the detection head §3.5. CBNetV1 uses AHLC only. CBNetV2 uses DHLC, auxiliary supervision and the s_3 pruning §3.5. CBNetV2 is the default CBNet in the experiments unless stated otherwise §3.5.

Pseudocode

The forward pass below uses DHLC with stage indices 1 to L; stage 1 is the stem. It follows eq. 2 and eq. 6 and omits pruning and auxiliary heads.

# CBNet forward pass with DHLC, no pruning  [§3.1] [eq. 2] [eq. 6]
# stages[l]: output of stage l, l = 1..L (stage 1 is the stem)
# Input:  image (B, 3, H, W); K identical backbones
# Output: lead-backbone stages 2..L for the neck

def cbnet_forward(image, backbones, projections):
    # The first (assisting) backbone runs on the image  [§3.1]
    prev = run_stages(backbones[0], image)
    for k in range(1, K):
        # Stage 1 is weight-shared, so it is reused, not recomputed  [§3.1]
        cur = {1: prev[1]}
        for l in range(2, L + 1):
            # Add the composed higher-level features to the stage input  [eq. 2]
            feed = dense_feed(prev, projections[k], l)  # (B, C_l, H_l, W_l)
            cur[l] = backbones[k].stage(l, cur[l - 1] + feed)
        prev = cur
    # Only the lead's stages go to the neck and head  [§3.5]
    return [prev[l] for l in range(2, L + 1)]


def dense_feed(prev, projection, l):
    # Sum over stages l..L of the previous backbone  [§3.2.4] [eq. 6]
    total = 0
    for i in range(l, L + 1):
        # Project to the channel count of the target stage  [§3.2.4]
        # w_i is a 1x1 conv (+ BN in the ResNet code)  [code: cbnet.py#L156-L160]
        reduced = projection[i](prev[i])
        # Resize to the target stage's spatial size  [§3.2.4]
        total = total + upsample(reduced, size_of_stage(l - 1))
    return total

Hyperparameters

The paper gives the settings below. The released code is the MMDetection fork at the commit in the frontmatter. Stated-only values are marked as such.

Results

All numbers are author-reported. Each table is cited to the paper. Setups differ across tables, so compare within a table only.

Test-dev, SOTA detectors tab. 1

The text gives 52.8 for the anchor-free row; the table says 52.7 (see Pitfalls) §4.2.1.

Instance segmentation and detection with Swin tab. 2

* multi-scale testing; † extra unlabeled data for pretraining tab. 2 §4.2.2.

Generality, same detector with other backbones tab. 3 tab. 4

Faster R-CNN, box AP. The input size is not stated for these tables; §4.1.2 sets 800×500 for ablations and simple comparisons §4.3.1.

tab. 4 (text: CB-ResNet50 is about 1.7 above ResNet101 §4.3.1).

Swin Transformer tab. 5

Cascade Mask R-CNN §4.3.2. The table does not state the split. Gains are against the same backbone row.

Other backbones, mAP tab. 6

Mainstream detectors, ResNet50 vs CB-ResNet50 tab. 7

YOLOX and Joint-DetNAS tab. 8

DetectoRS comparison, Faster R-CNN 1× tab. 9

Deformable convolution tab. 10

Faster R-CNN, box AP, without and with DCN §4.5.1.

Model ensemble with PRAE tab. 11

Single-model best AP and PRAE ensemble AP, Faster R-CNN §4.5.2.

Composite strategies, Faster R-CNN CB-ResNet50 tab. 12 §4.6.1

Ablation setup (800×500).

AHLC vs DHLC on other backbones tab. 13 §4.6.1

Auxiliary loss weights tab. 14 §4.6.2

CBNetV1 vs CBNetV2, Faster R-CNN ResNet50 tab. 15 §4.6.5

Identical vs mixed backbones tab. 16 §4.6.6

Detector and schedule are not stated in the table or the text. Other rows include backbones of the same family with mixed depth.

Other results in the text only: a 2.3 to 2.7 AP gain with DCN §4.5.1; an ensemble gain of 0.8 to 1.7 AP with PRAE §4.5.2; and the s_3 vs s_4 FLOPs saving §4.6.3 fig. 6.

Paper vs code

Code: mmdet/ fork of MMDetection at code_commit, licence Apache-2.0 LICENSE. Backbones and composition are in mmdet/models/backbones/cbnet.py, the loss weights in mmdet/models/detectors/two_stage.py, and the neck in mmdet/models/necks/cbnet_fpn.py.

• Two backbones only. The paper allows K ≥ 2 §3.1. Both CNN and Swin wrappers hard-code range(2), so K = 2 is the only value built cbnet.py cbnet.py. The K = 3 results tab. 14 cannot be reproduced with this code, and no K = 3 config exists in configs/cbnet.
• Only DHLC. The paper defines five strategies §3.2. The code implements only the dense sum: each stage's input adds upsampled, projected stages from the previous backbone cbnet.py. SLC, AHLC, ALLC and FCC are not in the code, and CBNetV1 (AHLC) has no config. So the composite rows of tab. 12, tab. 13 and the V1 rows of tab. 15 are not reproducible from this repo.
• Which stages are summed. For the input of stage l, the code sums the previous backbone's outputs from stage l to stage L, including the same level cbnet.py cbnet.py. This matches eq. 6 once its index shift is applied. Implementers should not shift the index twice.
• Projection layer. The paper says w_i is a 1×1 conv and does not mention normalisation for DHLC §3.2.4. The CNN code uses a 1×1 conv followed by BatchNorm cbnet.py. The Swin code uses a 1×1 conv with no normalisation when the channel counts differ, and an identity when they match cbnet.py.
• Up-sampling. The paper calls U "up-sampling" without a method §3.2. The code uses nearest-neighbour interpolation cbnet.py cbnet.py.
• Zero initialisation. The composite layers start at zero: the last BatchNorm of each CNN projection is zero-initialised cbnet.py, and all Swin projections are zero-initialised cbnet.py. The paper does not state this.
• Pruning on CNN. With cb_del_stages=1, the stem and stage 2 are deleted in backbones 2 … K cbnet.py faster_rcnn_cbv2d1_r50_fpn_1x_coco.py. In the paper's stage numbering this is s_3, which matches the default in §3.4.
• Pruning on Swin. The Swin default, cb_del_stages=1 cbnet.py, deletes only the patch embedding in backbones 2 … K. The layer loop range(0, self.del_stages - 1) cbnet.py removes nothing at this value. In the paper's indexing this is s_4 (stage 1 pruned, stages 2 to 5 kept). The paper does not say which pruning it used for Swin.
• Auxiliary loss weights. The code default is [0.5] + [1]*(K-1) two_stage.py. Index 0 is the first (assisting) backbone, and the last is the lead. Each weight multiplies that backbone's RPN and RoI losses two_stage.py two_stage.py two_stage.py. This matches tab. 14 for K = 2 and K = 3.
• Shared heads. The neck and heads are shared across backbones, as the paper says §3.3 two_stage.py cbnet_fpn.py. At inference only the lead's FPN output is used cbnet_fpn.py.
• BatchNorm. Projection BatchNorms are set to eval mode during training cbnet.py. The paper does not mention this.
• Frozen stages. The Faster R-CNN base config freezes stage 1 and sets BatchNorm to eval [code: configs/base/models/faster_rcnn_r50_fpn.py#L9-L11]. The paper does not state either setting.
• Ablation resolution. The paper uses 800×500 §4.1.2. The repo's base dataset config is 1333×800 [code: configs/base/datasets/coco_detection.py#L9], and no 800×500 ablation config is included.
• Missing configs. No ATSS config exists in configs/cbnet for tab. 1. The Cascade config is 20 epochs; the paper's 32-epoch SWA run is not in the repo, and the word "swa" does not appear in configs or mmdet.
• Soft-NMS. The paper says "threshold 0.001" §4.1.2. The code sets score_thr=0.001 with soft_nms cascade_rcnn_cbv2d1_r2_101_mdconv_fpn_20e_fp16_ms400-1400_coco.py. The paper does not say whether it means a score or an IoU threshold.
• Consistent items. The 1x schedule (SGD 0.02, decay at 8 and 11, 12 epochs) [code: configs/base/schedules/schedule_1x.py#L2-L11] and the Swin 3x schedule (decay at 27 and 33, 36 epochs) mask_rcnn_swin_small_patch4_window7_mstrain_480-800_adamw_3x_coco.py match the paper. The SOTA multi-scale range and 1600×1400 test size match too cascade_rcnn_cbv2d1_r2_101_mdconv_fpn_20e_fp16_ms400-1400_coco.py cascade_rcnn_cbv2d1_r2_101_mdconv_fpn_20e_fp16_ms400-1400_coco.py.

Pitfalls

• Cost is not zero. The assisting backbones run at inference, because the lead depends on their features. The paper's claim that auxiliary supervision adds no inference cost is true for the heads only §3.3. DHLC raises the ResNet50 model from 90 G to 127 G FLOPs and from 30.0 to 21.4 FPS in the authors' measurement tab. 12. Budget for roughly K times the backbone compute.
• Parameter count and speed comparisons. CB-ResNet50 is compared with deeper single backbones at similar FLOPs. The authors report FPS on one V100 §4.1.2 and say the comparison is at "comparable" cost §4.3.1. These are author-reported, measured with their hardware and code.
• Pruning explanation. The paper says pruned stages cost no accuracy because their weights are fixed during detector training §4.6.3. The code removes those layers outright for backbones 2 … K cbnet.py. The stated reason is not spelled out in the paper, so treat the claim as author-reported.
• Swin pruning is unclear. The Swin default removes only the patch embedding (see Paper vs code). Do not assume the CNN s_3 result applies to Swin.
• Weight sharing of stage 1. The paper calls stage-1 outputs weight-shared §3.1. The code reuses the first backbone's stem output and does not compute it again cbnet.py cbnet.py. The same applies to pruned stages.
• Numbers that disagree within the paper.
  • Anchor-free result: the text says 52.8 AP §4.2.1; Table 1 says 52.7 tab. 1.
  • DCN gains over the other rows: the text says +3.7 over ResNeXt50-32x4d-DCN and +1.3 over ResNeXt101-64x4d-DCN §4.5.1; Table 10 gives +3.9 and +1.5 tab. 10.
  • Shorter schedules: the text says 16× and 12× shorter than EfficientDet and YOLOv4 §4.2.1; the table lists 600 and 450 epochs against 32, which is about 18.8× and 14.1× tab. 1.
  • MobileNetV2 gain: the text says +3.1 §4.3.3; Table 6 gives 22.2 to 25.4, which is +3.2 tab. 6.
  • Aux-supervision gain: the text says "2.1 over the baseline" for "rows 1 and 6" §4.6.5. Table 15 gives 34.6 to 38.1 (+3.5) for rows 1 and 6, and +2.1 only against CBNetV1 (row 2, 36.0).
  • Identical backbones: CB-Res2Net50 is said to beat every mixed pair §4.6.6, but Res2Net101-C-ResNet50 is 41.2, the same as CB-Res2Net50 tab. 16.
  • Pruning FLOPs and the backbone-count curve are given only in figures §4.6.3 §4.6.4 fig. 6; no table backs them.
• Choice of K. The K = 3 rows tab. 14 need a code change (see Paper vs code). The K = 4 and 5 results exist only as a figure §4.6.4 fig. 6.
• Composite search is expensive. The paper's grid search trained 512 detectors for 205 GPU days §4.6.1. The searched strategy matched DHLC at 37.3 AP with 0.6 M fewer parameters, so it did not beat it §4.6.1. Do not expect a search-based variant to win without that budget.
• Index conventions. eq. 6 and the stage numbering are easy to shift by one. Check against the code cited above before porting.
• Projection choice. Normalisation, zero-initialisation and the interpolation method are not stated in the paper, and each changes training. Use the code's defaults, or state the change.
• Head sharing. The aux and lead heads share all weights §3.3. Separate heads would be a different model from the one the paper reports.
• Missing details. Frozen stages, warm-up, batch-norm mode and the ATSS and SWA training recipes are not stated in the paper. The code uses MMDetection defaults for most of them.
• Endorsement. This page is a compilation. The authors do not endorse it.

Lineage

• Builds on: CBNet (AAAI 2020, cited as [LiuWWLZTL20] in the paper), the preliminary version. The paper says CBNetV2 changes the composition strategy, adds auxiliary supervision and the pruning option §3.5. Its arXiv id is not in the source.
• Evaluated with: Swin Transformer [/papers/2103.14030] (the backbone for the Swin results §4.3.2).
• Compared against: DetectoRS §4.4, Joint-DetNAS §4.4, and the PRAE ensemble §4.5.2.
• Superseded by: not known from this wiki.