Robust Consistent Video Depth Estimation

In plain words

Single-image depth networks give a plausible depth map for each frame, but each frame is guessed on its own, so the depth does not agree from one frame to the next and the picture flickers. The known fix needs the camera's exact path through the scene, and that path is hard to get from shaky, blurry footage with moving objects.

This method starts from the network's guesses and tunes them. It stretches each frame's depth with a smooth, flexible warp, and it solves for the camera path at the same time, so no poses have to be supplied. A last filter averages each pixel with matching pixels in nearby frames, which brings back fine detail. The authors report that it works on casual videos and beats earlier methods on a standard synthetic benchmark.

TL;DR

Input is only a colour video. A pretrained depth CNN gives per-frame depth, and optical flow links pixels between frame pairs. The method jointly fits camera poses, intrinsics and a per-frame spline warp of the depth, then runs a geometry-aware temporal filter. On Sintel it reports lower depth and pose errors than DeepV2D. Use it when SfM poses are unreliable; the main known flaw is residual wobble in the aligned depth §5.3.

Problem

Dense per-frame depth is useful for stabilisation, AR and VR, but casual videos are hard: noise, shake, motion blur, rolling shutter, dynamic objects and poorly textured regions §1. Multi-view SfM and MVS often fail to register all frames or produce outlier poses on such video §1. Hybrid methods that combine a learned depth prior with geometry (CVD, Yoon et al.) assume precise poses as input, so they fail when pose estimation fails [§1, §2]. The method targets static-scene geometry; moving people, animals and vehicles are masked out of the geometric terms §4.1.

Method

Overview §3, fig. 2: the depth network runs per frame, then poses and a flexible depth alignment are optimized jointly, then a geometry-aware filter resolves fine detail §3, fig. 2.

Pre-processing §4.1: sample frame pairs at growing temporal spacings eq. 1, compute dense optical flow with a forward-backward consistency mask, and compute a dynamic-object mask per frame. The method reuses the CVD preprocessing steps and does not run COLMAP §4.1.

Depth model §4.2: a pixel p of frame i is lifted to 3D with the per-frame scale s_i and the CNN depth d_i eq. 2. It is projected into frame j with the frame's rotation, translation and intrinsics eq. 3. The reprojection loss compares the 3D point reached by geometry with the point reached by optical flow eq. 4, eq. 5.

Pose optimization §4.3: CVD's loss can be re-used with the depth fixed and the camera parameters free. The paper notes that this works only when the depth is precise, because depth misalignment becomes noise in the poses §4.3.

Alternating depth and pose fine-tuning §4.4: depth is trained by SGD and poses by a global solver, in alternation. The paper reports that starting from imprecise depth yields jittery poses and no convergence §4.4, fig. 3d.

Flexible deformation §4.5: the per-frame scale s_i is replaced by a spatially varying bilinear spline over a grid of handles, s_i^k eq. 6. This removes large low-frequency misalignments, so poses come out smoother and the method no longer alternates pose and depth fine-tuning §4.5, fig. 3e.

Geometry-aware filter §4.6: each output depth is a weighted sum over a 3x3 neighbourhood and frames i-4 to i+4 eq. 7. Each sample is reprojected into frame i along chained flow, and its weight drops with depth ratio across edges, with lambda_f = 3 eq. 8. The printed equation has no normalising denominator; see Pitfalls.

Losses §4.7: the reprojection similarity loss is the sum of an image-space term eq. 14 and a depth-ratio term eq. 16 eq. 17. The paper rejects a plain Euclidean loss, which shrinks the scene, and a disparity loss, which pushes the scale up eq. 13, eq. 15. The deformation smoothness penalty sums squared differences between neighbouring handles, weighted by a per-handle dynamic-fraction term eq. 9, eq. 10. A focal-length prior pulls each frame's focal parameter toward a fixed value eq. 12.

# Joint pose and depth alignment with flexible deformation  [§4.5] [§4.7]
# Input:  depth maps (T, H, W); flow-matched pixel pairs per frame pair
# Output: camera parameters per frame; spline handles s_i^k per frame

def align_video(depths, pairs, dynamic_masks):
    # Start with one handle per frame; normalise scale  [§4.7]
    handles = init_grid(depths, rows=1, cols=1)
    for grid in coarse_to_fine_grids():  # 1x1 up to 17x10  [§4.7]
        # Sample matches at least 10 px apart; skip dynamic pixels  [§4.7]
        matches = sample_matches(pairs, min_distance=10,
                                 exclude=dynamic_masks)
        # Minimise to convergence before the next grid  [§4.7]
        poses, handles = minimise(
            reprojection_loss(matches, poses, handles),  # [eq. 4] [eq. 17]
            deformation_penalty(handles, dynamic_masks),  # [eq. 9] [eq. 10]
            focal_prior(poses),                           # [eq. 12]
        )
        handles = split_grid(handles, grid)  # subdivide the grid  [§4.7]
    return poses, handles

# Geometry-aware temporal depth filter  [§4.6] [eq. 7] [eq. 8]
# Input:  aligned depth for all frames; flow chains; tau = 4
# Output: filtered depth for frame i

def filter_depth(i, pixel):
    weighted_sum = 0.0
    weight_total = 0.0
    for j in range(i - 4, i + 5):  # window of 2 * tau + 1 frames  [§4.6]
        for q in neighbourhood_3x3(pixel):  # [§4.6]
            # Follow the chained flow from frame i into frame j  [§4.6]
            q_in_j = chained_flow(i, j, q)
            # Depth of q as seen from frame i, after reprojection  [eq. 7]
            z = reprojected_z(i, j, q_in_j)
            # Edge-aware weight from the depth ratio, lambda_f = 3  [eq. 8]
            w = exp(-3.0 * depth_ratio(i, j, q, q_in_j))
            weighted_sum += z * w
            weight_total += w
    # Normalise; the printed eq. 7 has no denominator  [eq. 7]
    return weighted_sum / weight_total

Hyperparameters

Results

Author-reported on MPI Sintel §5.1, using the Clean and Final versions. Sintel has 23 sequences with ground-truth depth and poses. COLMAP fails on 11 of 23 sequences, and CVD inherits that failure, so averaged metrics are not comparable across methods §5.2. The table below is the only table in the main text. Abs Rel, Sq Rel and RMSE are depth errors; delta metrics are accuracy; ATE and RPE are pose errors tab. 1.

Sintel Clean tab. 1, top:

Sintel Final tab. 1, bottom:

Headline numbers, full method vs DeepV2D tab. 1:

• Clean: Abs Rel 0.377 vs 0.526; ATE 0.1843 m vs 0.9526 m.
• Final: Abs Rel 0.419 vs 0.526; ATE 0.1831 m vs 0.9192 m.
• Depth fine-tuning gives the lowest ATE in both settings (0.1656 Clean, 0.1594 Final) but a worse depth error than the full method tab. 1.
• Fig. 5 plots sorted per-pair RPE errors; COLMAP and CVD appear with partial data because they fail on many sequences fig. 5.

Paper vs code

Official code: facebookresearch/robust_cvd, linked from the paper's project page. Its LICENSE file is CC BY-NC 4.0, so no code is quoted and the discrepancies below cite paths and lines only. The paper's Sintel numbers cannot be checked from the repository: the README points to a Colab notebook that is not in the repo, and the repo has no Sintel evaluation script.

• Reprojection loss. Paper: image term plus depth-ratio term §4.7, eq. 17. Code default: a disparity-difference loss, the variant the paper rejects params.py, PoseOptimizer.cpp. The ratio loss is selectable PoseOptimizer.cpp. The repository does not say which one produced Table 1.
• Dynamic-aware deformation weights. Paper: w = lambda_1 + lambda_2 times the dynamic fraction, with lambda_1 = 0.1, lambda_2 = 10 §4.7, eq. 10. Code: the adaptive option exists but defaults to 0, so it is off [code: lib/PoseOptimizer.h#L74], params.py. The base weight is 1.0 in the normalisation pass and 0.1 in the pose steps, with the annealing off by default PoseOptimizer.h, [code: lib/PoseOptimizer.cpp#L834-L845]. The paper gives no base weight.
• Focal prior. Paper: u-hat = 0.35, about 40 degrees §4.7, eq. 12. Code default: 0.34615, documented as about 38.2 degrees for an iPhone 7 portrait photo PoseOptimizer.h. Its weight defaults to 1.0 PoseOptimizer.h.
• Depth filter spatial window. Paper: 3x3 neighbourhood §4.6. Code: the spatial radius defaults to 0, and the pipeline's filter call sets only the frame radius Processor.h, [code: pose_optimization.py#L318-L321], so the effective window is 1x1 [code: lib/Processor.cpp#L449-L455].
• Filter weight form. Paper: exp(-lambda_f (ratio - 1)) eq. 8. Code: exp(-3 times the ratio), normalised by the weight sum [code: lib/Processor.cpp#L552-L557], Processor.cpp. The difference is a constant factor that cancels in the normalisation.
• Dynamic classes. Paper: person, animal, vehicle §4.1. Code: COCO class indices 0 to 7 and 13 to 22 dynamic_mask_generation.py. Its own docstring names a different set that includes boat and giraffe. Under the usual COCO index order, index 13 is bench, so the code's list is not the paper's set.
• Dynamic exclusion margin. Paper: exclude matches inside the mask §4.7. Code: an 8-pixel distance is set in the Mask mode [code: pose_optimization.py#L170-L172]. The paper states no margin.
• Scale normalisation. Paper: first-frame median depth = 1, same scale for all other frames §4.7. Code: a normalisation flag defaults to true [code: lib/PoseOptimizer.h#L103]. The scale constraint loops over every frame in the range it is given PoseOptimizer.cpp. Whether that is the first frame only depends on the caller, which I did not confirm.
• Frame pairs. Paper: pairs at spacings k = 1, 2, 4, ... eq. 1. Code: a hierarchical pair sampler with two-way pairs yields the same pairs for powers of two frame_sampling.py, video.py. Which mode the Colab run uses is not in the repository.
• Consistent: grid schedule (17 by 10, four steps) [code: lib/PoseOptimizer.h#L58, #L81-L83]; RAFT weights and 1024 px flow size [code: flow.py#L18], flow.py; 10 px match spacing [code: lib/Processor.h#L75]; filter window 4 params.py.
• Fine-tuning defaults. Paper: no values in the compiled text §4.4. Code: batch size 2 and 10 epochs depth_fine_tuning.py.
• Solver iterations: paper not stated; code default 1000 [code: lib/PoseOptimizer.h#L56].
• Mask detector threshold: paper not stated; code default 0.5 [code: dynamic_mask_generation.py#L83-L86].

Pitfalls

• The filter sum in eq. 7 has no denominator. Implement it as a normalised weighted mean, or the depth scales with the weight sum §4.6, eq. 7.
• Flow chaining across the filter window is described only in words. How masks break a chain, and which direction to chain, is not stated §4.6.
• Pose initialisation (the starting R_i, t_i and u_i) is not stated in the compiled text §4.3-§4.7.
• Scale rule conflicts. Eq. 6 gives each frame its own handles s_i^k, but §4.7 says the same scale is used for all other frames. The text does not say how a shared scale and per-frame handles combine §4.5, §4.7.
• No combined objective or regulariser weights appear in the compiled text. The source has a final-objective equation, but it sits in a block that the compiled paper leaves out §4.7.
• Intrinsics: only the focal parameter is free. The principal-point convention and the coordinate normalisation in the image-space loss are not stated §4.7, eq. 11, eq. 14.
• Disparity loss is biased toward large scene scale, while the ratio loss has no such bias. Using the disparity loss changes the answer §4.7, eq. 15-16.
• Pose estimation is non-convex and sensitive to local minima. The paper reduces that sensitivity by starting from a 1x1 grid and subdividing §4.7.
• Denser pair constraints blow up solver memory and slow it down. This is a stated limitation of the global formulation §5.3.
• Residual wobble remains in most aligned depth maps, and the authors point to pixel-based deformation as the fix §5.3.
• Evaluation scaling: depth uses median scaling and drops values above 80 m. Pose is scaled and aligned by minimising ATE, so ATE is not a metric error. Whether the alignment is a similarity transform, and whether median scaling is per frame or per sequence, is not stated §5.1-§5.2, tab. 1 caption.
• The text compares against MiDaS-v2, but Table 1 has no MiDaS-v2 row. Its "aligned MiDaS" row is an ablation of the authors' own method, not MiDaS itself §5.2, tab. 1.
• COLMAP and CVD fail on part of Sintel, so the averages are not comparable §5.2.
• Runtime is not in the main text. A runtime analysis exists only in a supplement that the main source does not include.
• Eq. 14 and eq. 15 use the projected coordinates a_xy/a_z and 1/a_z. Their scale depends on the intrinsics convention, which the paper does not define in full §4.7.

Lineage

• Builds on: Consistent Video Depth Estimation (Luo et al., 2020), which fine-tunes a single-image CNN against reprojection error with known poses. This paper removes the pose input and replaces the test-time fine-tuning with joint pose and deformation optimization §1, §4.2.
• Single-image depth prior: MiDaS v2 (Lasinger et al.) §2, §5.1.
• Compared against: COLMAP, DeepV2D, CVD and MiDaS v2 §5.1.