wikillm

Paper · 2024-01-07 · cs.CV

Bilateral Reference for High-Resolution Dichotomous Image Segmentation

BiRefNet segments thin structures in high-resolution images. A Swin encoder locates the object; the decoder then restores detail from original-resolution image tiles (inward reference) and from gradient maps that gate its features (outward reference). Trained with BCE, IoU, SSIM and CE losses. Author-reported state of the art on DIS, HRSOD and COD benchmarks.

Salient object detectionDeep supervisionSegmentation evaluation (scoring foreground, saliency and segmentation masks)Hierarchical vision transformerssegmentationdichotomous-image-segmentationhigh-resolutionsalient-object-detectioncamouflaged-object-detectionvision-transformer

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Some pictures are large and full of thin, delicate parts: hair, cracks, slender branches. Cutting out the exact outline of an object there is hard. Models that shrink the picture to save memory lose these details, and models that look at the picture only coarsely tend to miss them.

The authors split the job in two. A localisation part looks at the whole image and finds roughly where the object is. A reconstruction part then rebuilds a sharp outline. It gets two extra hints at each step. One is the original, full-resolution image, cut into pieces that match the current working size, so no fine detail is thrown away. The other is a map of where the picture changes sharply, which points the model at fine edges.

The paper also collects training tricks for these large inputs: longer training, supervision at several intermediate outputs to speed up learning, and a final fine-tune on region-level losses.

TL;DR

  • BiRefNet is an image-to-mask network for high-resolution dichotomous image segmentation: it separates an object from its background at 1024 × 1024 with fine outlines §1 §4.3.
  • It has a localisation module (Swin encoder, a squeezed bottleneck with ASPP) and a reconstruction module built from bilateral reference blocks §3.1.
  • The inward reference feeds original-resolution image tiles into each decoder stage; the outward reference predicts input gradient maps and uses them as attention §3.4.
  • Author-reported: best results on the DIS5K test sets, and better results than task-specific models on HRSOD and COD tab. 4 tab. 5 tab. 6.
  • The official MIT-licensed code differs from the paper in several defaults. See Paper vs code before reimplementing.

Problem

Dichotomous image segmentation (DIS) asks for a binary mask of a target object, including its slender and fine structures, in high-resolution images §1. The paper argues that existing DIS methods, which use intermediate supervision, a frequency prior, or a split-and-merge strategy, still miss very fine features §1 §2.1. Methods that feed a resized copy of the image into the decoder lose detail; the paper instead keeps the original-scale image as a reference §1 §2.1. Training on high-resolution data is costly, which motivates the training strategies in §3.6. The paper trains on the DIS5K training set (DIS-TR) and tests on DIS-TE1 to DIS-TE4 and DIS-VD §4.1. The method also transfers to high-resolution salient object detection and to concealed object detection §4.5.

Method

Overview

The network has two modules: a localisation module (LM) that produces a coarse, semantic estimate, and a reconstruction module (RM) that refines it using bilateral reference (BiRef) blocks §3.1 fig. 3. Compared with earlier designs that feed the image pyramid or scaled images into the encoder, the original image is used as an inward reference at the decoder, and gradient maps are used as an outward reference §3.1 fig. 2.

Localisation module (LM)

  • A Swin Transformer encoder gives features at four stages, at 1/4, 1/8, 1/16 and 1/32 of the input, written F¹ᵉ to F³ᵉ and Fᵉ §3.2.
  • The first three stages reach the decoder through 1 × 1 lateral convolutions. The final encoder block combines the stages, first stacking then concatenating them, to form Fᵉ §3.2.
  • Fᵉ feeds a classification branch: global average pooling and a fully connected layer over C = 219 classes, used as an auxiliary semantic signal §3.2 §4.3.
  • The bottleneck squeezes Fᵉ into Fᵈ and applies ASPP for multi-context fusion §3.2.

Reconstruction module (RM)

  • Each BiRef block uses a reconstruction block (RB) in place of a residual block. The RB runs deformable convolutions with receptive fields 1 × 1, 3 × 3 and 7 × 7 plus an adaptive average-pooling branch, concatenates the results, and applies a 1 × 1 convolution and batch normalisation to give F^d' §3.3 fig. 4.
  • The first BiRef block predicts a coarse map. Later blocks upsample the previous decoder feature and add the lateral feature from the LM, F^d+ = Upsample(F^d + F^l) §3.3.
  • Each block also produces an intermediate map M_i (i = 3, 2, 1), trained with multi-stage supervision. The last decoder feature passes through a 1 × 1 convolution to give the final map M at input resolution §3.3.

Bilateral reference (BiRef) fig. 4

  • Inward reference (InRef). The input image is cropped into patches whose size equals the spatial size of the stage's decoder feature. The patches are stacked onto F^d+ and fed into the RM. This is done at every stage, so the image is never resized to match the decoder §3.4.
  • Outward reference (OutRef). Gradient maps of the input image are computed as G^gt §3.4. A feature branch F^θ predicts a gradient map Ĝ_i, which is supervised against a masked target G^m_i. A convolution and a sigmoid give an attention map A^G, which multiplies F^d' to form the block output §3.4.
  • Masking. The intermediate map M_i is dilated with a morphological operation and used as a mask on G^gt, so gradients outside the predicted region are removed §3.4. The dilation size is not stated.

Objective §3.5 eq. 1

The total loss is a weighted sum of a pixel term (BCE), a region term (IoU), a boundary term (SSIM) and a semantic term (CE). The weights λ1 to λ4 are 30, 0.5, 10 and 5, chosen to put the terms on the same scale at the start of training §3.5.

L = \lambda_1 L_{BCE} + \lambda_2 L_{IoU} + \lambda_3 L_{SSIM} + \lambda_4 L_{CE} eq. 1

L_{BCE} = -\sum_{(i,j)}\big[G(i,j)\log M(i,j) + (1-G(i,j))\log(1-M(i,j))\big] eq. 2

L_{IoU} = 1 - \frac{\sum M\,G}{\sum (M + G - M\,G)} eq. 3

L_{SSIM} = 1 - \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)} eq. 4

L_{CE} = -\sum_{c=1}^{N} y_{o,c}\log p_{o,c} eq. 5

In eq. 2, G is the ground truth and M is the binarised prediction §3.5. Eq. 3 sums over pixels of one image. Eq. 4 compares N × N patches of the ground truth and the prediction; N and the constant C are not stated §3.5. Eq. 5 is the cross-entropy over the N classes, with y the one-hot label and p the predicted probability §3.5.

Training strategies for DIS §3.6 tab. 1 tab. 3

  • Long training helps fine detail. Rough structure and edges converge at about 200 epochs on DIS5K; fine-detail metrics keep improving to 400 epochs §3.6 tab. 1.
  • Multi-stage supervision speeds up learning. The paper reports that 200 epochs with it match 400 epochs without it §3.6 tab. 1.
  • Fine-tuning with region-level losses improves binarisation and the metrics closest to practical use §3.6 tab. 3.
  • Context feature fusion and an image-pyramid input on the backbone give a small general gain §3.6 tab. 3.

Pseudocode: forward pass

Pseudocode
# BiRefNet forward pass  [§3.1] [§3.2] [§3.3] [§3.4]
# Input:  images (B, 3, 1024, 1024), normalised
# Output: final map and three intermediate maps, resized to input size

def forward(images):
    # Encoder: stages at strides 4, 8, 16, 32  [§3.2]
    stage_feats = encoder(images)
    # Localisation: squeeze the deepest stage, then ASPP  [§3.2]
    decoded = aspp(squeeze(stage_feats[3]))
    # Gradient map of the input image: outward-reference target  [§3.4]
    grad_gt = gradient_map(images)
    maps = []
    grad_preds = []
    grad_targets = []
    # Three BiRef blocks, coarse to fine  [§3.3]
    for stage in (2, 1, 0):
        # Upsample, then add the lateral feature from the LM  [§3.3]
        decoded = upsample(decoded) + lateral(stage_feats[stage])
        # Inward reference: image tiles the size of the decoder feature  [§3.4]
        tiles = tile_like(images, decoded)     # (B, 3 * n_tiles, h, w)
        feats = reconstruction_block(concat(decoded, tiles))  # [§3.3]
        # Outward reference: predict a gradient map, then gate features  [§3.4]
        grad_pred = gradient_head(feats)       # (B, 1, h, w)
        gate = sigmoid(gate_head(feats))       # (B, 1, h, w)
        feats = feats * gate                   # (B, C, h, w)
        # Intermediate map for multi-stage supervision  [§3.3]
        map_i = map_head(feats)                # (B, 1, h, w)
        # Gradient target: input gradients masked by the dilated map  [§3.4]
        mask = dilate(map_i)                   # dilation size not stated
        grad_target = resize(grad_gt, mask.shape) * mask
        maps.append(map_i)
        grad_preds.append(grad_pred)
        grad_targets.append(grad_target)
        decoded = feats
    # Final map from the last decoder feature  [§3.3]
    final = conv1x1(decoded)                   # resized to input size
    return final, maps, grad_preds, grad_targets

Pseudocode: objective

Pseudocode
# Training objective  [§3.5] [eq. 1]
# Weights: BCE 30, IoU 0.5, SSIM 10, CE 5  [§3.5]

def objective(maps, final, grad_preds, grad_targets, gt, logits, label):
    loss = 0.0
    # Pixel, region and boundary losses on every output map  [§3.5]
    for pred in maps + [final]:
        prob = sigmoid(pred)                   # resize to gt size first
        loss += 30.0 * bce(prob, gt)           # pixel term  [eq. 2]
        loss += 0.5 * iou_loss(prob, gt)       # region term  [eq. 3]
        loss += 10.0 * ssim_loss(prob, gt)     # boundary term  [eq. 4]
    # Semantic term from the 219-class head  [§3.5] [eq. 5]
    loss += 5.0 * cross_entropy(logits, label)
    # Gradient supervision: described in §3.4, not in eq. 1; code weight 1.0
    for pred, target in zip(grad_preds, grad_targets):
        loss += bce(sigmoid(pred), sigmoid(target))
    return loss

Hyperparameters

NameValueWhereNotes
Input size1024 × 1024 for training and testing§4.3Predictions are resized to the ground-truth size with bilinear interpolation §4.3 inference.py
BackboneSwin Transformer large (default); Swin-B, Swin-T, PVTv2-b2 variants§4.3 tab. 4 tab. 7Code default swin_v1_l config.py
Classes219 (names of the DIS-TR labels)§3.2 §4.3Code list has 219 entries dataset.py
OptimiserAdam§4.3Code uses AdamW, weight decay 0.01 config.py train.py
Initial learning rate1e-4 for DIS; 1e-5 for HRSOD and COD§4.3Code scales by √(batch/4) config.py
Learning-rate schedulenot stated§4.3Code never decays within the run config.py train.py
Epochs600 DIS; 150 HRSOD; 150 COD§4.3Code script: 500 DIS, 150 COD, 150 HRSOD train.sh
Fine-tuningLast 20 epochs with the IoU loss§4.3Code: last 40 DIS epochs, BCE off, IoU halved config.py train.py
Batch size4 per GPU, eight NVIDIA A100§4.3Code default is 8 per device config.py
Loss weightsBCE 30, IoU 0.5, SSIM 10, CE 5§3.5 eq. 1Code defaults match for BCE, IoU, SSIM config.py
CE weight5§3.5Code ce weight 5.0 config.py, but inactive by default config.py
Gradient supervision weightnot stated§3.4Code adds it with weight 1.0 train.py
Mask dilationnot stated§3.4Code applies no dilation birefnet.py
Deformable receptive fields1 × 1, 3 × 3, 7 × 7 plus pooling§3.3 fig. 4Code branches [1, 3, 7] aspp.py
AugmentationHorizontal flip only§4.3Code applies four operations config.py image_proc.py
Multi-stage outputsThree intermediate maps plus the final map§3.3Trained with the same objective loss.py
S-measure α0.5§4.2metrics.py
F-measure β²0.3§4.2metrics.py
Relaxed HCE tolerance γ5§4.2
Precisionnot statedCode default bf16 config.py
Random seednot statedCode seed 7 config.py

Results

Author-reported. Setup: trained on DIS-TR; evaluated on DIS-TE1 to DIS-TE4 and DIS-VD at 1024 × 1024 §4.1 §4.3. Baseline numbers are taken from the UDUN paper, as the authors state, not re-run §4.5 tab. 4. Metrics: F-measure, weighted F-measure, MAE (M), S-measure (Sm), mean E-measure (Em) and relaxed HCE §4.2. The DIS columns are in that order.

DIS5K (DIS-TE1 to DIS-TE4 aggregate, 2,000 images; DIS-VD, 470 images) tab. 4

MethodTE FβxTE FβωTE M↓TE SmTE EmTE HCE↓VD FβxVD FβωVD M↓VD SmVD EmVD HCE↓
BASNet.744.664.092.786.8141007.737.656.094.781.8091132
U²Net.771.676.082.799.8251042.753.656.089.785.8091139
HRNet.743.658.087.781.8401432.726.641.095.767.8241560
PGNet.809.746.063.830.8851173.798.733.067.824.8791326
IS-Net.799.726.070.819.8581016.791.717.074.813.8561116
FP-DIS.831.770.047.847.8951165.823.763.062.843.8911309
UDUN.831.772.057.844.892977.823.763.059.838.8921097
BiRefNet (Swin-L).896.858.035.901.934916.891.854.038.898.931989
BiRefNet (Swin-B).891.855.036.898.933954.881.844.039.890.9251029
BiRefNet (Swin-T).866.822.045.877.916980.862.819.045.874.9171070
BiRefNet (PVTv2-b2).878.838.041.886.925969.868.827.044.880.9191073

Per-subset numbers for DIS-TE1 to DIS-TE4 are in tab. 4. The paper's text describes Swin-L as its default and reports a 6.8% average Sm gain on DIS over earlier models §1 §4.5.

HRSOD and low-resolution SOD (Sm and M only) tab. 5

MethodTraining setsDAVIS-S Sm / MHRSOD-TE Sm / MUHRSD-TE Sm / MDUTS-TE Sm / MDUT-OMRON Sm / M
PGNet2, 3.954 / .010.938 / .020.935 / .026.859 / .038.786 / .058
BiRefNet1, 2, 3.975 / .006.962 / .013.957 / .016.944 / .018.882 / .038

Training sets: 1 = DUTS, 2 = HRSOD, 3 = UHRSD tab. 5. The paper reports a 2.0% average Sm gain over the previous best HRSOD model §4.5.

COD (CAMO, COD10K, NC4K) tab. 6

MethodCAMO SmFβωFβmEmExMCOD10K SmFβωFβmEmExMNC4K SmFβωFβmEmExM
FSPNet.856.799.830.899.928.050.851.735.769.895.930.026.879.816.843.915.937.035
HitNet.849.809.831.906.910.055.871.806.823.935.938.023.875.834.853.926.929.037
BiRefNet.904.890.904.954.959.030.913.874.888.960.967.014.914.894.909.953.960.023

The paper reports a 5.6% average Sm gain over the previous best COD models §4.5.

Ablation of the modules, DIS-VD tab. 2

RMInRefOutRefFβxFβωM↓SmEmHCE↓
.837.785.056.845.8871204
✓.855.831.048.865.8951167
✓.848.825.050.857.9031152
✓✓.869.834.041.886.9121093
✓✓.863.831.042.891.9181106
✓✓.861.839.044.881.9111114
✓✓✓.889.851.038.900.9241065

Practical training tricks, DIS-VD tab. 3. CFF is context feature fusion, IPT is the image-pyramid input, and RLFT is region-level fine-tuning.

CFFIPTRLFTFβxFβωM↓SmEmHCE↓
.889.851.038.900.9241065
✓.893.856.038.904.9281054
✓.895.857.037.904.9271051
✓.890.861.036.899.9321043
✓✓✓.897.863.036.905.9371039

Epochs and multi-stage supervision (MSS), DIS-VD tab. 1

MSSEpochsFβxFβωM↓SmEmHCE↓
200.875.848.041.886.9141207
400.897.863.036.905.9371039
✓200.892.858.037.901.9321043

Efficiency tab. 7. Measured on one NVIDIA A100 and an AMD EPYC 7J13 CPU §4.5. The column header reads "#Params (MB)" as printed in the paper. The text says the compiled Swin-L variant is 13% faster §4.5, but Table 7 gives 83.3 ms against 78.3 ms, about 6%.

ModelRuntime (ms)#Params (MB, as printed)MACs (G)DIS-TE HCEDIS-TE Fβω
BiRefNet Swin-L83.32151143916.858
BiRefNet Swin-L, compiled78.32151143916.858
BiRefNet Swin-B61.4101561954.855
BiRefNet Swin-T40.939231980.822
BiRefNet PVTv2-b247.835195969.838
BiRefNet PVTv2-b136.623147978.817
BiRefNet PVTv2-b032.911891013.806
IS-Net16.0441601016.726
UDUN (ResNet-50)33.525142977.772

Paper vs code

Official code: https://github.com/ZhengPeng7/BiRefNet, linked from the abstract, owned by the first author's account. Commit f7079632d681ecf12161fcea8905682934e59e51, MIT License. I read the training, loss, model and evaluation code. The differences below are what an implementer will hit first.

  • Epochs. Paper: 600 for DIS §4.3. The training script uses 500 for DIS5K train.sh.
  • Batch size and learning rate. Paper: 4 per GPU with lr 1e-4 for DIS §4.3. Code default: batch 8 per device config.py. Code sets lr = base × √(batch/4) config.py, so the paper's values are reproduced only at batch 4. The code comment says "linearly"; the formula is a square root.
  • Optimiser. Paper: Adam §4.3. Code: AdamW with weight decay 0.01 config.py train.py.
  • Fine-tuning. Paper: last 20 epochs with the IoU loss §4.3. Code: last 40 epochs for DIS5K config.py. In those epochs the code sets the BCE weight to zero, halves the IoU weight and keeps SSIM train.py. This is not IoU-only.
  • Augmentation. Paper: horizontal flip only §4.3. Code: flip, colour enhancement, rotation with probability 0.2 by an integer angle in [−15, 15), and salt-and-pepper noise config.py image_proc.py image_proc.py image_proc.py. Random crop is listed but disabled.
  • Classification and CE. Paper: CE with weight 5 on a 219-class head §3.2 §3.5. Code: the head exists only when auxiliary_classification is set, and it is False by default config.py birefnet.py. With no head, the class term is 0 train.py.
  • Gradient supervision in the loss. Paper: eq. 1 has no gradient term §3.5. Code: adds a BCE term on the gradient maps with weight 1.0 train.py train.py. The code also applies a sigmoid to the gradient target, which the paper does not mention.
  • Gradient target mask. Paper: a dilated, binarised intermediate map masks the gradient map §3.4. Code: the raw, undilated, unthresholded logit map multiplies the gradient map, and no .detach() is applied, so the gradient loss also updates the map head birefnet.py train.py. The code's own comment mentions dilation, but none is applied.
  • BCE. Paper: sum over pixels, on binarised predictions §3.5 eq. 2. Code: nn.BCELoss averages over pixels, applied to continuous sigmoid outputs loss.py loss.py.
  • IoU. Paper: eq. 3 for one image §3.5. Code: the per-image losses are summed over the batch, not averaged; the averaging line is commented out loss.py.
  • SSIM. Paper: 1 − SSIM on N × N patches, N and C not stated §3.5 eq. 4. Code: an 11 × 11 Gaussian window with σ = 1.5, C1 = 0.01², C2 = 0.03², and loss = 1 − (1 + SSIM)/2 loss.py loss.py loss.py.
  • Loss summation. Paper: one term per map, not described for several maps §3.5. Code: the full objective is added for each of the four outputs (three intermediate maps and the final map) loss.py.
  • Bottleneck. Paper: ASPP in the bottleneck §3.2. Code default: a 3 × 3 squeeze block config.py birefnet.py. The ASPP with deformable branches sits inside each decoder block, with a dropout of 0.5 that the paper does not mention config.py aspp.py.
  • Decoder block. Paper: each baseline decoder block is two residual blocks §4.4. Code default: a plain two-convolution block with batch norm config.py decoder_blocks.py.
  • Inward reference. Paper: patches are stacked onto the feature §3.4. Code: the patch branch is two 3 × 3 convolutions with no activation between them birefnet.py. At the coarsest stage there are 1024 tiles, so the branch takes 3072 input channels birefnet.py birefnet.py.
  • Default architecture extras. The code defaults add three encoder context-fusion inputs and a concatenated half-resolution image pyramid config.py birefnet.py birefnet.py. Table 3 says the final model uses context fusion and the pyramid input tab. 3, so these are consistent.
  • HRSOD training data. Paper's best row trains on DUTS, HRSOD and UHRSD tab. 5 §4.1. Code default is HRSOD plus UHRSD only config.py.
  • Mixed precision and LR decay. Not stated in the paper. Code: bf16 mixed precision by default config.py, and no learning-rate decay within training config.py.

Checked and consistent: 219 classes dataset.py; Swin-L default backbone config.py; the λ values in config for BCE, IoU and SSIM config.py; the S-measure α of 0.5 and F-measure β² of 0.3 metrics.py metrics.py; and bilinear resizing of predictions back to the ground-truth size inference.py.

Pitfalls

  • Objective. Eq. 1 is incomplete. A faithful implementation of the paper's text includes the gradient supervision that the code applies with weight 1.0. The paper gives no weight for it.
  • Reductions change the loss scale. The paper's BCE sums over pixels. The code averages it, and the IoU term is summed over the batch. Keep the λ values with the reductions the code uses, or retune them.
  • Auxiliary CE is off by default. A run using the default config trains without the semantic term that the paper lists in eq. 1.
  • Undilated, raw-logit mask. The gradient targets depend on the exact mask. Without the dilation and thresholding the paper describes, the gradient target includes negative logits before the sigmoid. The dilation size is not stated.
  • Inputs are squashed to a square. Every image is rescaled into a square 1024 × 1024 frame for both training and inference §4.3 inference.py. Non-square inputs are distorted, and the output is resized back to the original shape.
  • Memory at the coarsest stage. The inward reference at stride 32 builds 1024 tiles, each 32 × 32, so the first patch branch takes 3072 channels birefnet.py. Expect a large memory cost on that path.
  • Latency. The paper's text says the largest model runs at more than 10 FPS §4.5. Table 7 gives 83.3 ms for Swin-L on one A100, which is about 12 FPS tab. 7. Measure on your own hardware.
  • Mixed precision and seeds. The paper does not state precision, seed, or LR schedule. The code uses bf16 and seed 7 config.py config.py.
  • Baselines are not re-run. The competitor numbers come from the UDUN paper §4.5 tab. 4. Compare them with care, as numbers from another run.
  • Table and figure numbers. The numbers here follow the order of \caption commands in the LaTeX source. The rendered PDF may place floats differently. I did not check the PDF, so confirm a number against the rendered paper before citing it.
  • Ablation text. The text's relative gains match the table rows: RM alone gives about 2.2% Fβx over the baseline, InRef and OutRef together without RM about 2.9%, and the full model about 6.2% §4.4 tab. 2.
  • Efficiency units. "#Params (MB)" in Table 7 is the header as printed, and the unit is probably not megabytes. Do not use it for memory budgeting without checking tab. 7.

Lineage

  • Builds on: a Swin Transformer encoder §3.2; an FPN-style top-down decoder §3.3; ASPP from DeepLab v3 §3.2; deformable convolutions §3.3; a hybrid BCE, IoU and SSIM loss in the style of BASNet §3.5; and the DIS5K dataset and IS-Net work of Qin et al. §2.1.
  • Compared against: IS-Net, UDUN, FP-DIS (DIS); PGNet, HRSOD and DHQ (HRSOD and SOD); and FSPNet, HitNet, ZoomNet and SINetv2 (COD) tab. 4 tab. 5 tab. 6.
  • Superseded by: not known from the content of this wiki.

Source and license

Adapted from “Bilateral Reference for High-Resolution Dichotomous Image Segmentation” by Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, Nicu Sebe (arXiv:2401.03407), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ZhengPeng7/BiRefNet at commit f7079632d681 (MIT). Any quoted code is shown under that license.