wikillm

Paper · 2022-02-20 · cs.CV

Dynamic Spatial Propagation Network for Depth Completion

DySPN refines a depth map by spatial propagation whose affinity weights are recomputed by attention at every step, instead of staying fixed as in CSPN. A diffusion suppression term stops over-smoothing early, and three cheaper neighbourhood variants are proposed. Author-reported: 709.12 mm RMSE on the KITTI depth completion benchmark (Deformable DySPN) and 0.090 m on NYU Depth v2.

Depth completionSpatial propagation networkDynamic convolutionAttention mechanismdepth-completionspatial-propagationattention

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Depth sensors such as LiDAR return distances only for some pixels of an image, so the depth map has many holes. Depth completion fills those holes, usually with the help of the ordinary colour image. A common trick is to let each pixel copy information from its neighbours, step after step, where the copying strength is a "how related are these two pixels" weight. Earlier versions of this trick keep those weights fixed for all steps. This paper lets the weights change at each step, so that far-away pixels help at the start to fill large holes and close pixels take over near edges later. It also adds a brake that stops the smoothing once the depth map is good enough, so objects do not blur into their surroundings. The result is competitive accuracy with fewer refinement steps and cheaper variants.

TL;DR

DySPN is a spatial propagation network for depth completion whose affinity matrix is adaptive: each of its six refinement steps uses its own attention maps §3.3 §3.4. A diffusion suppression term lets propagation converge early and avoid over-smoothing §3.2. Its author-reported KITTI benchmark result is 709.12 mm RMSE for the Deformable variant tab. 2. On NYU Depth v2 the 7×7 model reaches 0.090 m RMSE tab. 1. Relevant if you work on depth refinement, affinity-based propagation, or LiDAR-guided completion.

Problem

Sparse depth input (LiDAR or time-of-flight) leaves large empty regions §1. Affinity-based SPNs such as CSPN compute one affinity matrix that stays fixed during all propagation steps. With a 7×7 kernel most of the affinity weight sits in the outer ring, so the matrix cannot express fine pixel-to-neighbour relations, and more iterations are needed §1 fig. 1 §2. Running propagation to a fixed number of steps can also over-smooth dense regions §3.2.

Method

Non-linear propagation model (NLPM). Depth is a vector V^t. Each step multiplies it by a transformation G^t whose off-diagonal entries are affinity weights and whose diagonal row sums make each row total one eq. 1 §3.1. Unlike the linear model, G^t changes with t. The paper writes it as a diffusion process with a time-varying Laplacian, using a global attention matrix K^t eq. 2 eq. 3 eq. 4 §3.1. The full global form costs O(m²n²N) for an m×n image and N steps, which is impractical §3.3.

Diffusion suppression (DS). The paper defines a diagonal matrix D̄^t that holds, for each pixel, the affinity of the refined depth between steps t and t+1 eq. 5 §3.2. When diffusion has converged, K^t is zero and the step leaves V unchanged eq. 6 §3.2. In the implemented propagation this appears as the k = 0 attention term, which is the centre pixel's own weight eq. 7 fig. 4 §4.4.

Neighbourhood decoupling (the practical model). Instead of attending to every pixel, the neighbourhood is split into parts by distance k. Each part gets one attention map per step. Attention π_k^t is a sigmoid output §3.3. The one-step update for pixel (i, j) is:

  • the sum over distance parts k ≥ 1 of attention times the affinity times the neighbour's depth, divided by S';
  • plus the centre term π_0 times the current depth, divided by S';
  • plus (1 − S/S') times the initial depth h⁰ eq. 7.

S uses signed weights and S' uses absolute weights, both including π_0 eq. 8. The normalised form is the paper's way of keeping the update stable §3.3. Neighbourhood definitions:

  • 7×7 DySPN: all pixels in the 7×7 window, grouped into rings k = 1, 2, 3 (8k pixels each) plus the centre, so four attention maps per step eq. 9 fig. 3 §3.3.
  • Dilated DySPN: two 3×3 neighbourhoods, one with spacing 1 and one with spacing 3, given by offsets in {−2k+1, 0, 2k−1} eq. 10 fig. 3 §3.3. This reading of the formula is ours; the text only describes the dilation fig. 3 caption.
  • Deformable DySPN: a fixed 3×3 ring for k = 1, plus learned offsets f_φ for k ∈ {2, 3}, with real-valued positions eq. 11 fig. 3 §3.3.

Network. A U-Net-like encoder–decoder with a ResNet-34 backbone is used for fair comparison with other SPNs §3.4. Its head outputs in parallel: the attention maps π_k^t, the initial depth h⁰, the affinity matrix A, and a confidence map for the sparse input §3.4. The confidence map is not used in any equation in the paper §3.4.

Loss. A weighted sum of an L1 term and an L2 term on the refined depth h^N, each averaged over pixels with valid ground truth, with α = β = 1 and ρ ∈ {1, 2} eq. 12 §3.5.

Pseudocode
# One DySPN refinement, N = 6 steps  [§3.3] [§4.1]
# Inputs: depth0 (B, H, W) = h^0  [§3.4]
#         weights (B, H, W, K): affinity w per neighbour  [§3.4]
#         attention (N, B, H, W, 1 + K): sigmoid maps pi  [§3.3]
# Output: depth (B, H, W) = h^N

def refine(depth0, weights, attention, steps=6):
    # Start from the initial depth map  [§3.1]
    depth = depth0
    for t in range(steps):
        # Attention maps of step t; index 0 is the DS term  [eq. 7]
        pi = attention[t]
        # Normalisers: signed S and absolute S'  [eq. 8]
        s = pi[0] + sum(pi[k] * weights[k] for k in parts)
        s_abs = pi[0] + sum(pi[k] * abs(weights[k]) for k in parts)
        # Centre term plus the weighted neighbour sum  [eq. 7]
        update = pi[0] * depth + sum(
            pi[k] * weights[k] * neighbour(depth, k) for k in parts
        )
        # Divide by S' and pull back towards h^0  [eq. 7]
        depth = update / s_abs + (1 - s / s_abs) * depth0
    return depth
Pseudocode
# Training loss  [eq. 12]  [§3.5]
# Inputs: pred (B, H, W) = h^N;  gt (B, H, W)

def loss(pred, gt, alpha=1.0, beta=1.0):
    # Valid mask: ground-truth depth is positive  [eq. 12]
    valid = gt > 0
    # Number of valid pixels S  [eq. 12]
    count = valid.sum()
    # Mean absolute error on valid pixels, rho = 1  [eq. 12]
    l1 = (abs(pred - gt) * valid).sum() / count
    # Mean squared error on valid pixels, rho = 2  [eq. 12]
    l2 = ((pred - gt) ** 2 * valid).sum() / count
    return alpha * l1 + beta * l2

Hyperparameters

NameValueWhereNotes
OptimiserAdam, β1 = 0.9, β2 = 0.999§4.1
Learning rate, phase 10.001 for the first 30 epochs§4.1
Learning rate, phase 20.002 for a further 10 epochs§4.1Text says the rate "reduces to" 0.002, which is larger than 0.001. Direction and schedule not stated; see Pitfalls.
RegularisationStochastic depth, no weight decay§4.1 tab. 3Final ablation row uses stochastic depth alone. Stochastic-depth rate: not stated.
Batch size, NYU Depth v224§4.1Per GPU or total: not stated.
Batch size, KITTI DC8§4.1
Depth samples per image, NYU500, sampled at random from ground truth§4.1Sampling at test time: not stated.
AugmentationHorizontal flip, colour jitter§4.1Parameters: not stated.
Hardware4 × NVIDIA RTX 2080 Ti§4.1
Epochs, totalNot stated; 30 + 10 implied§4.1
Propagation steps N6 (all DySPN variants)§1 §4.5 tab. 4CSPN baseline: 12 in ablation tab. 3
Kernel7×7 (full); dilated and deformable use 3×3 blocks§3.3 fig. 3
Loss weights α, β1, 1eq. 12
Loss exponent ρ{1, 2}eq. 12Term-by-term use per ρ is inferred from the definition.
NLPM / DS attentionSigmoid (final); softmax tested§4.6 tab. 3Sigmoid and softmax gave similar results §4.6.
Weight decay (tested)5×10⁵, 10⁵, 5×10⁶ as printedtab. 3Signs missing in source; see Pitfalls.
NYU input sizeResized to 320×240, centre-cropped to 304×228§4.2
NYU train / test50K-image subset of the official train split; 654 test images§4.2
KITTI DC splitAbout 90K train, 1K val, 1K test pairs§4.2Train pairs top-cropped by 100 px, then centre-cropped to 1216×256 §4.2.
KITTI sparse input densityLess than 6% valid pixels§4.2Ground truth about 14% density §4.2.

Results

Author-reported. Setups are given per table.

NYU Depth v2, test set tab. 1 (depth in metres; higher δ is better):

MethodRMSE (m)RELδ1.25δ1.25²δ1.25³
CSPN0.1170.01699.299.9100.0
NLSPN0.0920.01299.699.9100.0
7×7 DySPN0.0900.01299.699.9100.0
Dilated DySPN0.0910.01299.699.9100.0
Deformable DySPN0.0900.01299.699.9100.0

KITTI DC, benchmark submission tab. 2 (mm, and 1/km for inverse metrics; lower is better; Deformable DySPN is the submitted model §4.5):

MethodRMSEMAEiRMSEiMAE
CSPN1019.64279.462.931.15
TWISE840.20195.582.080.82
DSPN766.74220.362.471.03
DeepLiDAR758.38226.502.561.15
ACMNet744.91206.092.080.90
CSPN++743.69209.282.070.90
NLSPN741.68199.591.990.84
GuideNet736.24218.832.250.99
FCFRNet735.81217.152.200.98
PENet730.08210.552.170.94
Deformable DySPN709.12192.711.880.82

The text reports "nearly 21 mm" RMSE gain and first place on all four metrics at the time of submission §4.5. These are not verifiable from the paper.

KITTI DC validation, ablation tab. 3 (RMSE and MAE in mm):

ConfigurationIter.SigmoidSoftmaxDSWDStoch. depthRMSEMAE
7×7 CSPN12789.6201.4
7×7 CSPN6811.0207.3
7×7 DySPN12✓768.1196.4
7×7 DySPN6✓769.0195.9
7×7 DySPN6✓770.0197.1
+ DS6✓✓763.9195.5
+ DS, WD 5×10⁵6✓✓5×10⁵753.2196.5
+ DS, WD 10⁵6✓✓10⁵752.9194.6
+ DS, WD 5×10⁶6✓✓5×10⁶755.8195.2
+ DS, WD 10⁵, stoch. depth6✓✓10⁵✓747.0193.2
+ DS, stoch. depth only6✓✓✓745.8192.5

Runtime and accuracy, KITTI DC validation tab. 4 (hardware and input size not stated):

ModuleIter.Time (ms)RMSE (mm)MAE (mm)
NLSPN1855--
PENet (C2)1215757.2209.0
7×7 CSPN1270789.6201.4
7×7 DySPN638745.8192.5
Dilated DySPN66.9748.5193.8
Deformable DySPN610.3739.4191.4

Paper vs code

No official code.

Searched for an official repository in this order: the paper text and arXiv metadata (no link to code); GitHub repository search for "DySPN" and "dyspn depth completion", and for "spatial propagation depth completion". The two DySPN repositories found are third-party: S-Torin/DySPN, described as a "non-official PyTorch implementation", and Kyakaka/DySPN, described as an "implement of" the paper. Neither is used. Searched GitHub users for the first author's name: no matches. No organisation search, because none of the authors' affiliations are given in the source.

Pitfalls

  • Learning-rate schedule is ambiguous. The text says the rate "reduces" from 0.001 to 0.002 after 30 epochs. 0.002 is higher, so the value, direction and schedule shape are unclear. Treat the phase-2 rate as not stated and check against a training log before you rely on it §4.1.
  • Weight-decay values lack their minus signs. Table 3 prints 5×10⁵, 10⁵ and 5×10⁶. Read as written these are very large values; the intended values are probably 5×10⁻⁵, 10⁻⁵ and 5×10⁻⁶, but the paper does not say so tab. 3.
  • Diffusion suppression is not defined without the DS term. Eq. 7 uses π_0 as the DS attention, but the paper does not say what π_0 is when DS is switched off (zero, one, or removed). The ablation implies that the DS-free model is still a DySPN tab. 3 §4.6.
  • The attention and affinity heads are under-specified. The paper does not say how the affinity matrix A is normalised or signed before it reaches eq. 7. The use of |w| in S' implies that w can be negative eq. 8. Where the attention maps come from in the network head (channel layout, activation per map) is not stated beyond "sigmoid" §3.4.
  • Division by S' has no stated guard. If every π and w in a pixel's neighbourhood is zero, S' is zero. The paper does not handle this case eq. 7 eq. 8.
  • Deformable offsets. The paper says f_φ predicts real-valued offsets for k ∈ {2, 3} eq. 11, but not how it is parameterised, how it is trained, or how the depth is sampled at fractional positions (for example bilinear). Those choices change results and are not stated.
  • Dilated formula reading. The spacing in eq. 10 is {−2k+1, 0, 2k−1}. For k = 1 this gives a unit 3×3 block and for k = 2 a block with spacing 3, which agrees with the caption of fig. 3. Whether k also takes other values, and how many variants there are in code, is not stated.
  • δ metric formula looks like a typo. The printed definition is max(h/ĥ − ĥ/h) < τ, a difference. The standard metric uses the maximum of the ratio and its inverse, max(h/ĥ, ĥ/h) < τ. Implement the standard form; the paper's printed form may be a typesetting error. We could not confirm this against code, because none is available §4.3.
  • Loss averaging. The normalisation S is written for one depth map. Whether the loss is averaged per image or over the whole batch is not stated eq. 12.
  • Confidence head unused. The network predicts a confidence map for the sparse input, but no loss or propagation term uses it in the paper §3.4. Implementations must decide how to use it; the paper does not say.
  • Test-time sparse sampling. Training samples 500 valid depth pixels per NYU image §4.1. How the test input is sampled, and the number of points, is not stated.
  • Benchmark claims are time-stamped. "Ranks 1st on KITTI DC at the time of submission" and "nearly 21 mm" are author-reported claims for a benchmark that changes. Use the table values with their date and setup.
  • Validation vs test. The ablation is on the KITTI validation split tab. 3 tab. 4. Only the benchmark row tab. 2 is a test-server result. The NYU table reports the labelled test set §4.2.
  • Runtime hardware. Table 4 gives per-module times but not the GPU, batch size, or input resolution. Compare only within the table.
  • Table numbering. The source labels its tables with internal names that do not match print order. The printed numbers used on this card are: NYU (tab. 1), KITTI test (tab. 2), ablation (tab. 3), runtime (tab. 4). These follow the order in which the tables appear, which is how the document numbers them.
  • Attention-map cost. The network predicts one attention map per distance part and per step: four per step for the 7×7 model, so 24 over six steps §3.3 eq. 7. The paper does not report the head's parameter count or memory use. The main benchmark claim is for the Deformable variant, while the cleanest ablation is for the 7×7 model tab. 2 tab. 3.

Lineage

  • Builds on: CSPN (linear, fixed-affinity propagation; the ablation baseline) §2 tab. 3. Also SPN, the first CNN-learned affinity propagation §2; NLSPN (non-local neighbours) §2; CSPN++ (adaptive kernel sizes) §2.
  • Draws on dynamic filters, which generate weights from the input, such as DFN, CondConv and DynamicConv §2.
  • Compared against: CSPN, CSPN++, NLSPN, DSPN, PENet, DeepLiDAR, ACMNet, GuideNet, FCFRNet and TWISE tab. 1 tab. 2.
  • Superseded by: not stated in the source.

Source and license

Adapted from “Dynamic Spatial Propagation Network for Depth Completion” by Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, Hua Yang (arXiv:2202.09769), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.