--- kind: concept slug: deformable-convolution title: Deformable convolution summary: >- A convolution whose sampling points move by learned offsets, so each output reads content-dependent positions. Five cards use it differently: a deformable first layer in detector heads, deformable sampling in 2D and 1D attention, a backbone operator (DCNv3), and flow-guided video alignment. Gains are author-reported and not comparable across papers. tags: [convolution, object-detection, sampling, backbone, video-super-resolution, temporal-action-detection] papers: ["2104.13371", "2106.08322", "2106.10271", "2107.02963", "2211.05778"] related: [convolutional-neural-network, vision-transformer, attention-mechanism, object-detection, label-assignment, multi-scale-feature-fusion, image-super-resolution, recurrent-neural-network, set-prediction, object-queries] status: draft updated_at: 2026-11-01 --- ## Overview A standard convolution reads its kernel taps from a fixed grid around each output position. A deformable convolution adds a learned offset to each tap, so the grid can stretch or shift to follow the content. The output at a position is a weighted sum over these shifted sampling points [arxiv:2107.02963 eq. 2]. Each paper gets its offsets from a prediction layer, and they differ in what that layer sees and what else it predicts. The operator comes from deformable ConvNets. The DDOD paper attributes it to Dai et al. (2017) [arxiv:2107.02963 §4.2], and the InternImage paper cites DCNv2 and DCNv1 [arxiv:2211.05778 §1] [arxiv:2211.05778 §3.1]. The BasicVSR++ paper credits its deformable alignment to EDVR and TDAN [arxiv:2104.13371 §1]. None of these works has a card here, so this page describes only what the five cards report. The five cards place the operator in different roles: inside the head of a one-stage detector, inside the attention of a detection head, as the building block of a whole backbone, as the alignment step of a video model, and as a 1D attention step in a temporal action detector. Compare the variants before picking one. ## Variants ### Deformable first layer in each detection-head branch The DDOD detector replaces the first convolution of the classification tower and of the regression tower with a deformable convolution. The other convolutions stay regular, so the head keeps the same number of convolutions [arxiv:2107.02963 §4.2]. The authors argue that the two tasks want different spatial evidence: classification should cover semantic regions and regression should follow object contours. They support this with a Grad-CAM visualisation, which they treat as a qualitative check [arxiv:2107.02963 §3.2] [fig. 2]. Card: [Disentangle Your Dense Object Detector](/papers/2107.02963). Reported ablation, with the rest of DDOD off: on ATSS, the Spatial component (the deformable layer) reaches 40.6 AP against 39.4 for the baseline. All three components together reach 41.6 [arxiv:2107.02963 tab. 4]. **Layer position.** The authors tried the deformable layer at four positions in the head. The first conv of each branch gave the best AP, 40.6, against 40.3, 40.1 and 40.1 for the others [arxiv:2107.02963 §5.4.2] [arxiv:2107.02963 tab. 6]. The differences are at most 0.5 AP, from a single run per row, and no variance is reported. ### Deformable sampling inside an attention head Dynamic Head replaces a detector's head with stacked blocks that apply three attentions over a feature pyramid viewed as levels, positions and channels. The spatial-aware attention is the deformable one. For each pyramid level and each of K sampling points, the head predicts a learned offset and an importance scalar. Features are read at the shifted positions, weighted by a learned weight per level and point, summed, and divided by the number of levels [arxiv:2106.08322 §3.2] [arxiv:2106.08322 eq. 4]. The paper does not state K, and it says the offsets and scalars come from the median-level feature [arxiv:2106.08322 §3.2]. Card: [Dynamic Head](/papers/2106.08322). Reported ablation on ResNet-50, 1x, val2017: the spatial-aware attention alone reaches 41.4 AP against 39.0 for the baseline. All three attentions together reach 42.6 [arxiv:2106.08322 tab. 1]. ### Deformable operator as a backbone block (DCNv3) InternImage starts from DCNv2 and changes three things so the operator can be trained from scratch at scale [arxiv:2211.05778 §3.1] [arxiv:2211.05778 eq. 2]: 1. **Shared projection.** DCNv2 has a separate projection per sampling point. DCNv3 uses one projection shared by all points, and predicts the per-point modulation separately. 2. **Groups.** The channels are split into G groups, and each group has its own sampling offsets and modulation weights [arxiv:2211.05778 §3.1]. 3. **Softmax over points.** DCNv2 uses a sigmoid per point. DCNv3 uses a softmax over the K points of each group, so the weights of a group sum to 1 [arxiv:2211.05778 §3.1]. The operator sits in a block that resembles a transformer block: layer norm, DCNv3, a feed-forward network with GELU, and residual connections [arxiv:2211.05778 §3.2]. Model size is set by four hyper-parameters, and the largest variant has about 1B parameters [arxiv:2211.05778 tab. 1]. Card: [InternImage](/papers/2211.05778). Ablations on the T variant, ImageNet-1K top-1 and COCO Mask R-CNN box AP at 1x [arxiv:2211.05778 tab. 6]: | Change from final DCNv3 | Top-1 | Box AP | |---|---|---| | Final DCNv3 (shared weights, groups, softmax) | 83.5 | 47.2 | | Separate projection per point | 83.6 | 47.4 | | No groups | 82.3 | 43.8 | | Sigmoid instead of softmax | 65.7 | 38.7 | The paper attributes the sigmoid drop to unstable gradients [arxiv:2211.05778 §3.1]. At the H scale, sharing the projection saves 42.0% of parameters and 84.2% of GPU memory [arxiv:2211.05778 §4.3]. Replacing the 3x3 kernel with 5x5 or 7x7 gave 83.6 and 82.8 top-1 against 83.5 for 3x3, with more parameters and FLOPs [arxiv:2211.05778 app. B]. ### Flow-guided deformable alignment in video super-resolution (BasicVSR++) BasicVSR++ uses deformable convolution to align neighbouring frames in a recurrent video super-resolution model. Optical flow sets the base of each offset, and a small network predicts a residue that corrects the flow. The network also predicts a modulation mask for each sampling point. The deformable convolution then samples the unwarped earlier states with the combined offsets and masks [arxiv:2104.13371 eq. 6] [arxiv:2104.13371 eq. 7]. The authors say a flow base is easier to train than offsets learned from scratch [arxiv:2104.13371 §3.2]. Their no-flow ablation supports that claim: without flow the model reaches 27.44 dB and its training collapses, against 32.39 dB for the full model [arxiv:2104.13371 tab. 4]. The dataset of that table is not named, so the numbers are not tied to a benchmark in the paper. Reported result, REDS4 (BI, x4), author-reported: 32.39 dB with 7.3M parameters and 77 ms, against 31.42 dB with 6.3M parameters and 63 ms for BasicVSR [arxiv:2104.13371 tab. 1]. The 0.82 dB gain in the abstract belongs to the smaller variant, BasicVSR++ (S); the full model is 0.97 dB above BasicVSR on the same set [arxiv:2104.13371 tab. 1] [arxiv:2104.13371 tab. 2]. Card: [BasicVSR++](/papers/2104.13371). ### Temporal deformable attention in a video action detector (TadTR) TadTR applies the same sampling idea along one axis, video time, inside a Transformer rather than a convolution. In each encoder layer, every query predicts K sampling offsets and K attention logits per head. It reads the encoder memory by linear interpolation at the shifted positions and returns a softmax-weighted sum for each head [arxiv:2106.10271 §III-A] [arxiv:2106.10271 eq. 1] [arxiv:2106.10271 eq. 2]. In the encoder the reference point is the frame's own position. In the decoder it is the predicted centre of the action being refined [arxiv:2106.10271 §III-A]. The default is K = 4 points per head and 8 heads [arxiv:2106.10271 §IV-A]. The card traces the mechanism to Deformable DETR (arXiv:2010.04159), which has no card here. Ablation, default setting, average mAP on HACS: replacing this attention with dense attention gives 13.58 against 30.83, at 564.5 against 100.5 MFLOPs [arxiv:2106.10271 tab. 4]. The paper's ablations on the other two datasets show the same direction, with a smaller gap on ActivityNet [arxiv:2106.10271 tab. 4]. Reported headline, author-reported: self-contained THUMOS14 average mAP of 56.7 for TadTR with an instance filter and focal loss, and 49.3 for the default setting [arxiv:2106.10271 tab. 1]. Card: [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271). ## When to use which | Situation | Use | Why | |---|---|---| | One-stage detector, and only the head can change | Deformable first conv in each branch (DDOD) | Smallest change to an existing head. Author-reported +1.2 AP on ATSS from that layer alone [arxiv:2107.02963 tab. 4] | | Head must handle mixed scales, shapes and targets, and extra compute is acceptable | Deformable spatial attention in a stacked attention head (Dynamic Head) | Spatial module alone: 41.4 AP against 39.0 on ResNet-50 1x [arxiv:2106.08322 tab. 1]. Six blocks add 21.50 GFLOPs on a 254.39 baseline [arxiv:2106.08322 tab. 2] | | Building a CNN backbone that must scale, and a custom CUDA operator is acceptable | DCNv3 as the block operator (InternImage) | Author-reported top-1 of 83.5 for T against 82.1 for ConvNeXt-T at the same input size [arxiv:2211.05778 tab. 2] | | Recurrent video restoration where neighbouring frames must be aligned, and an optical-flow model is available | Flow-guided deformable alignment (BasicVSR++) | Flow gives a stable base for the offsets. The no-flow variant collapsed in training [arxiv:2104.13371 tab. 4]. Cost at a 180 x 320 LR input: 7.3M parameters and 77 ms, on a GPU the paper does not name [arxiv:2104.13371 tab. 1] | | Limited training data | Not supported by the cards' results | At 1% and 10% of ImageNet-1K, the T variant reaches 5.9% and 56.0% top-1, against 12.2% and 57.5% for ResNet-50 [arxiv:2211.05778 app. E] | | Need a larger kernel for a bigger receptive field | Keep 3x3 and rely on the learned offsets | 5x5 and 7x7 kernels did not improve top-1 over 3x3 [arxiv:2211.05778 app. B] | | Localising actions in long untrimmed video, with a one-to-one set predictor and no NMS | Temporal deformable attention in a DETR-style decoder (TadTR) | Local sampling instead of dense attention. On HACS, dense attention gives 13.58 average mAP against 30.83, at about 5.6 times the MFLOPs [arxiv:2106.10271 tab. 4]. Needs features from a frozen video encoder [arxiv:2106.10271 §III] | | Other detector families | Dynamic Head has the broadest evidence: gains on five detectors | Dynamic Head reports gains of 1.2 to 3.2 AP on Faster R-CNN, RetinaNet, ATSS, FCOS and RepPoints [arxiv:2106.08322 tab. 3]. DDOD reports ATSS, RetinaNet and FCOS [arxiv:2107.02963 tab. 3] | ## Implementation sketch The variants share one core: sample the input at shifted positions and sum the weighted values. They differ in where the offsets and weights come from. ```pseudocode # Deformable sampling core, shared by all variants [arxiv:2107.02963 eq. 2] # Input: feature map x (C, H, W); K taps on a regular grid g_k # Output: y (C, H, W) def deform_sample(x, offsets, weights, grid): # offsets: (H, W, K, 2) learned per output position [arxiv:2107.02963 §4.2] # weights: (H, W, K) per tap [arxiv:2211.05778 eq. 2] y = zeros_like(x) for p0 in positions(H, W): for k in range(K): # Read the input at the shifted location, bilinear between cells q = p0 + grid[k] + offsets[p0, k] # (2,) v = bilinear(x, q) # (C,) y[p0] += weights[p0, k] * v return y ``` Where the variants differ: - **DDOD** uses this layer as the first conv of each head branch. The rest of the head stays regular [arxiv:2107.02963 §4.2]. The offset predictor is not described in the paper. - **Dynamic Head** predicts the offsets and an importance scalar per point, and the weights come from a learned per-level weight [arxiv:2106.08322 eq. 4]. - **DCNv3** shares one projection across points, groups the channels, and normalises the weights with a softmax over the K points [arxiv:2211.05778 eq. 2]. - **BasicVSR++** sets each offset to the optical flow plus a learned residue, and adds a modulation mask. It samples the unwarped earlier frame states, not the warped ones [arxiv:2104.13371 eq. 6] [arxiv:2104.13371 eq. 7]. - **TadTR** samples along time only. Each query predicts K offsets and K logits per head, and reads the encoder memory by linear interpolation. The weights are a softmax over the sampled points [arxiv:2106.10271 §III-A] [arxiv:2106.10271 eq. 1] [arxiv:2106.10271 eq. 2]. The InternImage card notes that its PyTorch reference is for testing only and that the paper does not say which implementation its speed numbers use [arxiv:2211.05778 app. D]. Its offset normalisation and padding choices are code-only details; see the card's Pitfalls. The BasicVSR++ card records further code-only details, including a bound on the residue, which the paper's equation does not state [arxiv:2104.13371 eq. 6]; see that card's Pitfalls. ## Contradictions No direct contradictions between the cards, but the gains are not comparable. DDOD reports the deformable layer at 39.4 to 40.6 AP on ATSS [arxiv:2107.02963 tab. 4]. Dynamic Head reports its spatial attention at 39.0 to 41.4 AP on its own ResNet-50 1x baseline [arxiv:2106.08322 tab. 1]. BasicVSR++ reports 31.42 to 32.39 dB on REDS4 for a video task [arxiv:2104.13371 tab. 1]. The baselines, tasks and heads differ, so these gains should not be compared or added. TadTR reports average mAP on action segments, 49.3 to 56.7 on THUMOS14 and 30.83 on HACS with the default setting [arxiv:2106.10271 tab. 1] [arxiv:2106.10271 tab. 2]. Those use temporal-segment mAP, not the COCO-style AP above, so they should not be set against each other. Within one paper: the Dynamic Head abstract says the head adds no computational overhead, but its table reports 21.50 extra GFLOPs at six blocks [arxiv:2106.08322 tab. 2]. ## Papers - [Disentangle Your Dense Object Detector](/papers/2107.02963): replaces the first conv of each detection-head branch with a deformable convolution, and reports the layer-position ablation. - [Dynamic Head](/papers/2106.08322): uses deformable sampling with learned offsets and importance scalars as the spatial-aware attention in a stacked detection head. - [InternImage](/papers/2211.05778): uses DCNv3, a deformable operator with shared projection, groups and softmax weights, as the core of a CNN backbone. - [BasicVSR++](/papers/2104.13371): uses flow-guided deformable alignment, with optical flow as the base offset and a learned residue, in a recurrent video super-resolution model. - [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271): uses temporal deformable attention, with learned offsets over video time, in a DETR-style decoder for action segments.