Deformable convolution

Overview

A standard convolution reads its kernel taps from a fixed grid around each output position. A deformable convolution adds a learned offset to each tap, so the grid can stretch or shift to follow the content. The output at a position is a weighted sum over these shifted sampling points 2107.02963 eq. 2. Each paper gets its offsets from a prediction layer, and they differ in what that layer sees and what else it predicts.

The operator comes from deformable ConvNets. The DDOD paper attributes it to Dai et al. (2017) 2107.02963 §4.2, and the InternImage paper cites DCNv2 and DCNv1 2211.05778 §1 2211.05778 §3.1. The BasicVSR++ paper credits its deformable alignment to EDVR and TDAN 2104.13371 §1. None of these works has a card here, so this page describes only what the five cards report.

The five cards place the operator in different roles: inside the head of a one-stage detector, inside the attention of a detection head, as the building block of a whole backbone, as the alignment step of a video model, and as a 1D attention step in a temporal action detector. Compare the variants before picking one.

Variants

Deformable first layer in each detection-head branch

The DDOD detector replaces the first convolution of the classification tower and of the regression tower with a deformable convolution. The other convolutions stay regular, so the head keeps the same number of convolutions 2107.02963 §4.2. The authors argue that the two tasks want different spatial evidence: classification should cover semantic regions and regression should follow object contours. They support this with a Grad-CAM visualisation, which they treat as a qualitative check 2107.02963 §3.2 fig. 2. Card: Disentangle Your Dense Object Detector.

Reported ablation, with the rest of DDOD off: on ATSS, the Spatial component (the deformable layer) reaches 40.6 AP against 39.4 for the baseline. All three components together reach 41.6 2107.02963 tab. 4.

Layer position. The authors tried the deformable layer at four positions in the head. The first conv of each branch gave the best AP, 40.6, against 40.3, 40.1 and 40.1 for the others 2107.02963 §5.4.2 2107.02963 tab. 6. The differences are at most 0.5 AP, from a single run per row, and no variance is reported.

Deformable sampling inside an attention head

Dynamic Head replaces a detector's head with stacked blocks that apply three attentions over a feature pyramid viewed as levels, positions and channels. The spatial-aware attention is the deformable one. For each pyramid level and each of K sampling points, the head predicts a learned offset and an importance scalar. Features are read at the shifted positions, weighted by a learned weight per level and point, summed, and divided by the number of levels [arxiv:2106.08322 §3.2] 2106.08322 eq. 4. The paper does not state K, and it says the offsets and scalars come from the median-level feature 2106.08322 §3.2. Card: Dynamic Head.

Reported ablation on ResNet-50, 1x, val2017: the spatial-aware attention alone reaches 41.4 AP against 39.0 for the baseline. All three attentions together reach 42.6 2106.08322 tab. 1.

Deformable operator as a backbone block (DCNv3)

InternImage starts from DCNv2 and changes three things so the operator can be trained from scratch at scale 2211.05778 §3.1 2211.05778 eq. 2:

1. Shared projection. DCNv2 has a separate projection per sampling point. DCNv3 uses one projection shared by all points, and predicts the per-point modulation separately.
2. Groups. The channels are split into G groups, and each group has its own sampling offsets and modulation weights 2211.05778 §3.1.
3. Softmax over points. DCNv2 uses a sigmoid per point. DCNv3 uses a softmax over the K points of each group, so the weights of a group sum to 1 2211.05778 §3.1.

The operator sits in a block that resembles a transformer block: layer norm, DCNv3, a feed-forward network with GELU, and residual connections 2211.05778 §3.2. Model size is set by four hyper-parameters, and the largest variant has about 1B parameters 2211.05778 tab. 1. Card: InternImage.

Ablations on the T variant, ImageNet-1K top-1 and COCO Mask R-CNN box AP at 1x 2211.05778 tab. 6:

The paper attributes the sigmoid drop to unstable gradients 2211.05778 §3.1. At the H scale, sharing the projection saves 42.0% of parameters and 84.2% of GPU memory 2211.05778 §4.3. Replacing the 3x3 kernel with 5x5 or 7x7 gave 83.6 and 82.8 top-1 against 83.5 for 3x3, with more parameters and FLOPs 2211.05778 app. B.

Flow-guided deformable alignment in video super-resolution (BasicVSR++)

BasicVSR++ uses deformable convolution to align neighbouring frames in a recurrent video super-resolution model. Optical flow sets the base of each offset, and a small network predicts a residue that corrects the flow. The network also predicts a modulation mask for each sampling point. The deformable convolution then samples the unwarped earlier states with the combined offsets and masks 2104.13371 eq. 6 2104.13371 eq. 7. The authors say a flow base is easier to train than offsets learned from scratch 2104.13371 §3.2.

Their no-flow ablation supports that claim: without flow the model reaches 27.44 dB and its training collapses, against 32.39 dB for the full model 2104.13371 tab. 4. The dataset of that table is not named, so the numbers are not tied to a benchmark in the paper.

Reported result, REDS4 (BI, x4), author-reported: 32.39 dB with 7.3M parameters and 77 ms, against 31.42 dB with 6.3M parameters and 63 ms for BasicVSR 2104.13371 tab. 1. The 0.82 dB gain in the abstract belongs to the smaller variant, BasicVSR++ (S); the full model is 0.97 dB above BasicVSR on the same set 2104.13371 tab. 1 2104.13371 tab. 2. Card: BasicVSR++.

Temporal deformable attention in a video action detector (TadTR)

TadTR applies the same sampling idea along one axis, video time, inside a Transformer rather than a convolution. In each encoder layer, every query predicts K sampling offsets and K attention logits per head. It reads the encoder memory by linear interpolation at the shifted positions and returns a softmax-weighted sum for each head 2106.10271 §III-A [arxiv:2106.10271 eq. 1] 2106.10271 eq. 2. In the encoder the reference point is the frame's own position. In the decoder it is the predicted centre of the action being refined 2106.10271 §III-A. The default is K = 4 points per head and 8 heads 2106.10271 §IV-A. The card traces the mechanism to Deformable DETR (arXiv:2010.04159), which has no card here.

Ablation, default setting, average mAP on HACS: replacing this attention with dense attention gives 13.58 against 30.83, at 564.5 against 100.5 MFLOPs 2106.10271 tab. 4. The paper's ablations on the other two datasets show the same direction, with a smaller gap on ActivityNet 2106.10271 tab. 4.

Reported headline, author-reported: self-contained THUMOS14 average mAP of 56.7 for TadTR with an instance filter and focal loss, and 49.3 for the default setting 2106.10271 tab. 1. Card: End-to-end Temporal Action Detection with Transformer.

When to use which

Implementation sketch

The variants share one core: sample the input at shifted positions and sum the weighted values. They differ in where the offsets and weights come from.

# Deformable sampling core, shared by all variants  [arxiv:2107.02963 eq. 2]
# Input:  feature map x (C, H, W); K taps on a regular grid g_k
# Output: y (C, H, W)

def deform_sample(x, offsets, weights, grid):
    # offsets: (H, W, K, 2) learned per output position  [arxiv:2107.02963 §4.2]
    # weights: (H, W, K) per tap  [arxiv:2211.05778 eq. 2]
    y = zeros_like(x)
    for p0 in positions(H, W):
        for k in range(K):
            # Read the input at the shifted location, bilinear between cells
            q = p0 + grid[k] + offsets[p0, k]           # (2,)
            v = bilinear(x, q)                          # (C,)
            y[p0] += weights[p0, k] * v
    return y

Where the variants differ:

• DDOD uses this layer as the first conv of each head branch. The rest of the head stays regular 2107.02963 §4.2. The offset predictor is not described in the paper.
• Dynamic Head predicts the offsets and an importance scalar per point, and the weights come from a learned per-level weight 2106.08322 eq. 4.
• DCNv3 shares one projection across points, groups the channels, and normalises the weights with a softmax over the K points 2211.05778 eq. 2.
• BasicVSR++ sets each offset to the optical flow plus a learned residue, and adds a modulation mask. It samples the unwarped earlier frame states, not the warped ones 2104.13371 eq. 6 2104.13371 eq. 7.
• TadTR samples along time only. Each query predicts K offsets and K logits per head, and reads the encoder memory by linear interpolation. The weights are a softmax over the sampled points 2106.10271 §III-A [arxiv:2106.10271 eq. 1] 2106.10271 eq. 2.

The InternImage card notes that its PyTorch reference is for testing only and that the paper does not say which implementation its speed numbers use 2211.05778 app. D. Its offset normalisation and padding choices are code-only details; see the card's Pitfalls. The BasicVSR++ card records further code-only details, including a bound on the residue, which the paper's equation does not state 2104.13371 eq. 6; see that card's Pitfalls.

Contradictions

No direct contradictions between the cards, but the gains are not comparable. DDOD reports the deformable layer at 39.4 to 40.6 AP on ATSS [arxiv:2107.02963 tab. 4]. Dynamic Head reports its spatial attention at 39.0 to 41.4 AP on its own ResNet-50 1x baseline 2106.08322 tab. 1. BasicVSR++ reports 31.42 to 32.39 dB on REDS4 for a video task 2104.13371 tab. 1. The baselines, tasks and heads differ, so these gains should not be compared or added. TadTR reports average mAP on action segments, 49.3 to 56.7 on THUMOS14 and 30.83 on HACS with the default setting 2106.10271 tab. 1 2106.10271 tab. 2. Those use temporal-segment mAP, not the COCO-style AP above, so they should not be set against each other.

Within one paper: the Dynamic Head abstract says the head adds no computational overhead, but its table reports 21.50 extra GFLOPs at six blocks 2106.08322 tab. 2.

Papers

• Disentangle Your Dense Object Detector: replaces the first conv of each detection-head branch with a deformable convolution, and reports the layer-position ablation.
• Dynamic Head: uses deformable sampling with learned offsets and importance scalars as the spatial-aware attention in a stacked detection head.
• InternImage: uses DCNv3, a deformable operator with shared projection, groups and softmax weights, as the core of a CNN backbone.
• BasicVSR++: uses flow-guided deformable alignment, with optical flow as the base offset and a learned residue, in a recurrent video super-resolution model.
• End-to-end Temporal Action Detection with Transformer: uses temporal deformable attention, with learned offsets over video time, in a DETR-style decoder for action segments.