Side Adapter Network for Open-Vocabulary Semantic Segmentation

In plain words

Open-vocabulary segmentation means labelling every pixel of a photo with a name you choose at test time. CLIP is good at naming a whole picture, but its features are not aimed at single regions.

This paper keeps CLIP frozen and adds a small companion network. The companion network proposes candidate regions as masks. It also produces a per-region hint that CLIP's last layers use as an attention spotlight, so CLIP looks at just that region when it names it. The companion reads CLIP's earlier features, which lets it stay small. Training updates both parts together, so the proposals learn to suit what CLIP can recognise.

The result has few trainable parameters, is quick at test time, and scores well on several standard benchmarks. It is relevant to anyone who wants to adapt a strong image-text model to dense prediction without fine-tuning the whole model.

TL;DR

• SAN is an end-to-end open-vocabulary segmenter: a frozen CLIP ViT plus a light side ViT, trained on COCO-Stuff §4.1 §3.
• The side network outputs mask proposals and per-head attention biases; the biases steer CLIP's last blocks to classify each proposal §3.
• Reported gains over two-stage and fine-tuning baselines, with 8.4M trainable parameters and 15.2 FPS on one GPU at 640² tab. 2 tab. 3.
• Author-reported. The paper's "ensemble" rows need a model that is not in the public code.
• The code is MIT-licensed and mostly matches the paper; the main gaps are the inference formula, the background class, and the default recognition attention (see Paper vs code).

Problem

CLIP is trained with image-level contrastive learning, so its representation lacks the pixel-level recognition that segmentation needs §1. Two common routes have costs:

• Fine-tuning CLIP on segmentation data. Segmentation datasets are much smaller than CLIP's pre-training data, so open-vocabulary ability is often compromised §1.
• Two-stage pipelines. A separate mask generator proposes masked crops, and CLIP classifies each crop. The generator is independent of CLIP, the mask generator is heavy, and CLIP must run once per crop §1 §2.

CLIP's released ViT models are built for low-resolution input (for example 224²). Applied directly to high-resolution images, they perform poorly §1.

The evaluation protocol trains on COCO-Stuff and tests on five other datasets, so the reported numbers assume that setup §4.1.

Method

SAN treats segmentation as region recognition §1 §3. A side network is attached to a frozen CLIP ViT, and it has two outputs:

• Mask proposals, the inner product of pixel and query embeddings eq. 1 §3.
• Attention biases, the same kind of product with one bias per CLIP head eq. 2 §3.

The biases are added to the attention of CLIP's last blocks. Shadow [CLS] tokens, called [SLS] tokens and copied from the [CLS] token, read the visual tokens under this bias eq. 3 fig. 4. Their output is compared with CLIP text embeddings of the class names §3. The segmentation map combines the mask proposals and the class predictions eq. 4.

Key design choices:

• Decoupled heads. Mask proposals and recognition come from separate branches. Queries are shared between them §3 fig. 5.
• Single forward. Shallow CLIP blocks feed the side network through feature fusion, and the deeper blocks do recognition, so CLIP runs once per image §3 §4.4.
• Asymmetric input. CLIP sees a downscaled image, and the side network sees the full 640² input §4.2 §4.4.
• End-to-end training. The classification loss passes through the frozen CLIP's recognition blocks into the side network, so the proposals become CLIP-aware §3 §4.4 fig. 5.
• Positional embedding. The CLIP positional embedding is fine-tuned by default; the other CLIP weights stay frozen §4.4 visual.py.

Side network §3 §4.2: the image is split into 16×16 patches and linearly embedded. N = 100 learnable query tokens are prepended. An absolute position embedding is added in every block and shared across layers. The side network has 8 blocks, width 240 and 6 heads.

Mask decoder §3: query and pixel features are each projected to 256 dimensions by a 3-layer MLP. The mask logits are their inner product. The attention bias uses the same pattern, with a 3-layer MLP that outputs one 256-dimensional map per CLIP head.

Feature fusion §3: CLIP features from chosen layers are reshaped to maps, passed through a 1×1 convolution and resized, then added to the side network's features at the matching depth. For ViT-B/16 the fused CLIP layers are {stem, 3, 6, 9} and they feed side layers {stem, 1, 2, 3}. The paper fuses the first 9 CLIP blocks for ViT-B/16 and the first 18 for ViT-L/14 §4.2 tab. 5.

Equations, reproduced from the paper:

\mathbf{M} = \mathbf{V}_{\texttt{mask}} \mathbf{Q}_{\texttt{mask}}^{\mathrm{T}} eq. 1

\mathbf{B} = \mathbf{V}_{\texttt{attn}} \mathbf{Q}_{\texttt{attn}}^{\mathrm{T}} eq. 2

\mathbf{X}_\texttt{[SLS]}^{l+1} = \texttt{softmax}(\mathbf{Q}_\texttt{[SLS]}^l \mathbf{K}_\texttt{visual}^l + \mathbf{B}_k) \mathbf{V}_\texttt{[SLS]}^l eq. 3

\mathbf{S} = \mathbf{M} \times \mathbf{P}^\mathrm{T} eq. 4

L_{\texttt{seg}}=\lambda_1 L_{\texttt{mask\_dice}} + \lambda_2 L_{\texttt{mask\_bce}} + \lambda_3 L_{\texttt{cls}} eq. 5, with λ1 = 5.0, λ2 = 5.0, λ3 = 2.0 §3

Symbols as defined in the paper: M has shape H/16 × W/16 × N, B has shape H/16 × W/16 × K × N with K the CLIP head count, and P has shape C × N §3.

The forward pass, in pseudocode. It follows the paper's description and cites the code that implements each step:

# SAN forward pass for one image batch  [§3]
# Input:  image (B, 3, H, W), pixels in 0..255
# Output: mask_logits (B, N, C+1), mask_preds (B, N, H/16, W/16)

def san_forward(image, class_embeds):
    # 1. Normalise with CLIP's mean and std, then pad
    #    [code: san/model/san.py#L165-L167]
    image = (image - clip_mean) / clip_std
    # 2. CLIP gets a downscaled copy: factor 0.5 (B/16), 0.7 (L/14)
    #    [§4.2] [code: san/model/san.py#L169-L172]
    clip_image = resize(image, factor=clip_resolution)
    # 3. Frozen CLIP runs up to the last fused block and keeps the
    #    tokens of every layer, including [CLS]  [§3]
    #    [code: san/model/clip_utils/visual.py#L57-L91]
    clip_feats = clip_front(clip_image)
    # 4. Side network: N learnable queries plus patch tokens of the
    #    640^2 image; CLIP features are fused in at fixed depths  [§3]
    #    [code: san/model/side_adapter/side_adapter.py#L189-L234]
    pixel_feats, query_feats = side_network(image, clip_feats)
    # 5. Mask proposals: M = V_mask Q_mask^T  [eq. 1]
    #    [code: san/model/side_adapter/side_adapter.py#L78]
    mask_preds = einsum("bchw,bqc->bqhw", pixel_feats, query_feats)
    # 6. One attention bias per CLIP head and recognition block  [eq. 2]
    #    [code: san/model/side_adapter/side_adapter.py#L80-L85]
    attn_bias = einsum("bkchw,bqc->bkqhw", attn_feats, query_feats)
    # 7. [SLS] tokens start as copies of [CLS]; biased attention
    #    reads the visual tokens in the last blocks  [eq. 3]
    #    [code: san/model/clip_utils/visual.py#L173-L178]
    sls_embeds = clip_back(clip_feats, attn_bias)
    # 8. Cosine similarity with class text embeddings, scaled by CLIP's
    #    logit scale  [code: san/model/san.py#L182-L185]
    mask_logits = clip_scale * cosine(sls_embeds, class_embeds)
    return mask_logits, mask_preds

Inference turns the mask proposals and class logits into a per-pixel map. The paper writes this as the product in eq. 4. The code adds a softmax and a sigmoid, as shown below:

# Semantic map from mask proposals and class logits  [eq. 4, as coded]
# Input:  mask_logits (B, N, C+1), mask_preds (B, N, H, W)
# Output: semseg (B, C, H, W)

def semantic_inference(mask_logits, mask_preds):
    # 1. Softmax over the C classes plus background, then drop the
    #    background column  [code: san/model/san.py#L269-L271]
    class_probs = softmax(mask_logits, dim=-1)[..., :-1]
    # 2. Mask logits become soft masks with a sigmoid
    #    [code: san/model/san.py#L271]
    soft_masks = sigmoid(mask_preds)
    # 3. Sum over queries: semseg[c] = sum_n p(c|n) * m_n  [eq. 4]
    #    [code: san/model/san.py#L272]
    return einsum("bnc,bnhw->bchw", class_probs, soft_masks)

Training loss. The paper follows Mask2Former's practice §3. The code matches each query to one ground-truth mask with the Hungarian algorithm, then computes the loss on sampled points:

# Training loss for one batch  [§3] [eq. 5]
# Matching cost uses class 2.0, BCE 5.0 and dice 5.0 (code defaults)
# Input:  per-query logits and masks, per-image targets
# Output: scalar loss

def training_loss(outputs, targets):
    # 1. One-to-one Hungarian matching of queries to ground-truth masks,
    #    using sampled points  [code: san/model/matcher.py#L108-L156]
    matched = hungarian_match(outputs, targets)
    # 2. Classification: cross-entropy over C+1 classes; unmatched
    #    queries are background with weight 0.1
    #    [code: san/model/criterion.py#L114-L115]
    #    [code: san/model/criterion.py#L142-L145]
    loss_cls = cross_entropy(outputs.logits, matched, weights)
    # 3. Mask BCE and dice on 112*112 sampled points, not full masks
    #    [code: san/model/criterion.py#L169-L194]
    loss_bce, loss_dice = point_losses(outputs.masks, targets, matched)
    # 4. Weighted sum, normalised by the number of ground-truth masks
    #    in the batch  [eq. 5]  [code: san/model/criterion.py#L237-L243]
    loss = 2.0 * loss_cls + 5.0 * loss_bce + 5.0 * loss_dice
    # 5. Deep supervision: same losses on the output of side block 7
    #    [code: san/model/san.py#L192-L204]
    return loss + aux_loss(outputs.aux)

Text prompts app. A.3: 14 prompt templates are applied to each class name, the text embeddings are averaged, and the average is L2-normalised tab. 13. The code uses these templates utils.py and averages them before normalising classifier.py.

Hyperparameters

Values marked "paper" are stated in the paper. "Code" entries are from the official repository, and some are not in the paper.

Results

Author-reported. Setup: models trained on COCO-Stuff and evaluated on five benchmarks, mIoU, mean ± std over 5 trials §4.1 tab. 2. Speed numbers come from one Titan Xp GPU, 640 × 640 input, single image per inference §4.3.

Segmentation (mIoU)

The table also lists Group-ViT and LSeg+/OpenSeg rows, not copied here. The text's gains of +1.8 mIoU (ViT-B/16) and +2.3 (ViT-L/14), averaged over five datasets, match the table when SAN without ensemble is compared with OvSeg §4.3 tab. 2. Recomputed from the table: +1.84 and +2.26.

Efficiency

Setting: ViT-B/16 CLIP, 640 × 640 input. "*" marks baselines re-implemented by the authors because no official code existed when the paper was written tab. 3.

Ablations (ADE-150 mIoU, ViT-B/16 unless noted)

The default is the ViT-B/16 CLIP with an 8-layer side network of width 240 and 6 heads §4.4.

Figure 7 (text only, no numbers): raising the learning rate of CLIP lowers ADE-847 performance, while COCO-Stuff performance rises app. B fig. 7.

Paper vs code

Code: MendelXu/SAN at commit e19262d1a23992fc2cef934d6756b89ff50fe5c3, MIT License (Copyright 2023 MendelXu). The paper links this repository. Its owner matches the first author's project page (mendelxu.github.io/SAN, given in the LaTeX source). Hyperparameters that match the paper include the side network size, fusion map, layer split, loss weights, optimiser, learning rate, weight decay, batch size, iterations, crop size, and the 14 prompt templates.

Differences, each with both citations:

• Inference formula. The paper writes the map as the product of mask proposals and class predictions eq. 4. The code applies a softmax over C+1 classes, drops the background column, applies a sigmoid to the masks, and then sums over queries san.py.
• Background class. The paper's class matrix has C rows §3. The code adds a learnable background embedding, giving C+1 columns classifier.py, and uses a no-object weight of 0.1 in the cross-entropy config.py criterion.py. The paper says nothing about either.
• Recognition attention. The paper describes updating [SLS] tokens by cross-attention, and its cost formula assumes that §3. The default config sets REC_CROSS_ATTN = False config.py, which uses the concatenated, masked self-attention path visual.py and builds its mask at visual.py.
• Bias downsampling. The paper says the attention biases are resized §3. The default is max-pooling config.py attn_helper.py visual.py.
• Bias scaling. The code applies a learnable scalar affine transform to the biases (RESCALE_ATTN_BIAS = True) config.py side_adapter.py side_adapter.py. Equation 2 has no such scale eq. 2.
• Fusion layer. The paper describes a 1×1 convolution and a resize §3. The code applies a LayerNorm before the 1×1 convolution, then bilinear resize and addition layers.py.
• Training resize range. The paper says short side in [320, 1024] §4.2. The config samples 320 to 960 Base-coco-stuff-164K-171.yaml.
• Augmentation. The code also uses random flip and SSD colour augmentation Base-coco-stuff-164K-171.yaml mask_former_semantic_dataset_mapper.py. The paper names resizing and cropping only §4.2.
• Gradient clipping. Not in the paper. The config key is CLIP_VALUE, but the code calls clip_grad_norm_ with max-norm 0.01 over all parameters Base-coco-stuff-164K-171.yaml train_net.py.
• Deep supervision. The code trains an auxiliary head on side block 7 config.py san.py. The paper does not mention it.
• Ensemble. The paper's footnote says SAN is ensembled with a model fine-tuned on COCO-Stuff, and the "SAN ensemble" rows are not the default setting §4.3 footnote tab. 2. No ensemble code is in the repository.
• Mixed precision and padding. The code enables AMP Base-coco-stuff-164K-171.yaml and pads images to a multiple of 32 config.py san.py. The paper does not state either.
• Typo in the paper. Section 4.2 says "ViT-L/16" where the model is ViT-L/14 §4.2 san_clip_vit_large_res4_coco.yaml.
• Label-set similarity. The Hausdorff-based analysis in tab. 1 has no script among the repository's Python files, so it cannot be checked against code.

Pitfalls

• Inference is not the paper's formula. Implementing eq. 4 literally (product of raw mask logits and raw class scores) will not match the reported numbers. Use the softmax, sigmoid and background drop in the pseudocode above san.py.
• Matching and loss are sampled. Losses and matching use random points, not full-resolution masks: 112² points per image, with oversampling 3.0 and importance ratio 0.75 config.py criterion.py. Matching draws new random points on each call matcher.py, so matches are not deterministic across calls.
• Dice smoothing. Dice adds 1 to numerator and denominator criterion.py. Mask BCE is averaged over points, not summed criterion.py.
• Masking uses −100, not −∞. In the concatenated path, forbidden attention entries are the additive value −100 visual.py. After softmax they are close to zero but not exactly zero, so do not assume exact masking when porting to another attention kernel.
• Costs in the default path. Because the default recognition path is full masked self-attention, the paper's cross-attention cost analysis does not apply to the default configuration config.py. Recognition backpropagates through CLIP's last 3 (B/16) or 6 (L/14) blocks at 320² or 448² inputs. The paper does not report the memory cost.
• Token counts. From the stated sizes: CLIP sees (320/16)² = 400 tokens for B/16 at 0.5 scale, while the side network sees (640/16)² = 1600 tokens. Fused CLIP features are bilinearly upsampled to the side grid layers.py.
• Positional embeddings at other sizes. The side network's positional embedding is trained on the 640² grid and is bicubically resized when the grid differs side_adapter.py. CLIP's positional embedding is resized the same way visual.py. The paper does not say how the evaluation sizes were chosen, though the code evaluates at short side 640.
• Initialisation. The side network is randomly initialised and not pretrained config.py. Results depend on COCO-Stuff training alone, 60K iterations at batch 32.
• Learnable background. The background vector is trained at the base learning rate, not scaled by the CLIP multiplier, and is initialised with std D^-0.5 classifier.py train_net.py. Dropping it changes the output channel count from C+1 to C.
• Text embedding normalisation. Templates are averaged before normalisation, and the normalised class embeddings are scaled by CLIP's logit scale classifier.py san.py.
• Paper numbers are internally inconsistent. Section 4.4 says multi-layer fusion adds +0.8 mIoU over single-layer fusion. Table 4 gives 27.8 for {stem,3,6,9} against 27.1 for the best single layer, a gap of +0.7 §4.4 tab. 4.
• Ensemble column. Table 2 marks OvSeg and SimSeg "yes" and SAN "no" in the ensemble column. The text's averages compare SAN without ensemble against that table §4.3 tab. 2. The two sides are not matched on this axis, so quote the ensemble column next to any gain.
• Speed numbers. FPS and GFLOPs were measured on a Titan Xp with PyTorch 1.9.0 and CUDA 11.3 §4.3. Baselines marked * are re-implementations tab. 3.
• Small gaps. Ablations average 2 trials and system results 5 §4.1. The paper itself notes that a 0.2 mIoU gap "could arise from randomness" §4.4 footnote; treat differences of that size as noise.

Lineage

• Builds on: Side-Tuning and its variants, a side network attached to a pre-trained model §2; no card, named only. CLIP §1; no card, named only.
• Compared against: OvSeg, which fine-tunes CLIP with mask adaptation [/papers/2210.04150]; SimSeg, a two-stage frozen-CLIP baseline §2 tab. 2; MaskCLIP §2 tab. 3; no cards for SimSeg or MaskCLIP.
• Also in Table 2: GroupViT [/papers/2202.11094].