Feature-metric Loss for Self-supervised Learning of Depth and Egomotion

In plain words

Teaching a network to estimate depth and camera motion from ordinary video, without depth sensors, works by checking whether each frame can be rebuilt from its neighbours. Pixel colours are the thing being compared. The trouble is that a plain flat wall has almost the same colour everywhere, so a wrong guess about depth still looks almost right and gives the optimiser little to go on.

This paper compares features instead of colours. A second network, an auto-encoder, learns a description of each pixel from the image alone, and the rebuilding check is made on those descriptions. Two extra rules keep the descriptions useful: one makes neighbouring pixels differ enough to be told apart, and one keeps the change between neighbours smooth so the optimiser can slide towards the right answer. The depth and motion networks are trained with this loss added to the usual colour loss.

The method is tested on the KITTI driving dataset, where it reports better depth and odometry numbers than the self-supervised methods it is compared with.

TL;DR

• Self-supervised depth and egomotion training usually uses a photometric loss, which has flat gradients in textureless areas §3.2.
• The paper replaces the pixel comparison with a comparison of features from a FeatureNet trained by single-view reconstruction with two regularisers §3.3.
• Reported KITTI results are the best among self-supervised methods, and odometry on sequences 09 and 10 beats the listed deep baselines tab. 2 tab. 4.
• Most of the depth gain at the top of the table depends on online refinement and binocular training tab. 2.
• No official code was found (see Paper vs code).

Problem

Photometric loss is a weak signal for depth and pose. A small photometric error does not prove that depth and pose are right. In textureless regions many pixels share one value, and the gradient with respect to depth and egomotion is near zero there §3.2 eq. 4 eq. 5. The paper also notes that the image gradient can change direction locally, which makes the update direction inconsistent §3.2. Smoothness penalties on depth help only over a short range and tend to blur object boundaries §1.

Assumptions: static scene, Lambertian surfaces, no occlusion in both views §3.2. Occlusion is handled by taking a minimum over several source frames §3.4 eq. 18.

Method

Geometry. Pinhole projection and back-projection use intrinsics (f_x, f_y, c_x, c_y) §3.1 eq. 1 eq. 2. The warp maps a pixel to the other frame using depth and a rigid motion G in SE(3) §3.1 eq. 3.

Cross-view reconstruction. Target pixels are sampled from a source image at warped positions §3.2 eq. 4. The paper's cross-view loss sums a per-pixel loss over pixels §3.2 eq. 5. The proposed version replaces the image values with feature values from the same warp §3.2 eq. 8.

Single-view reconstruction (FeatureNet). An auto-encoder reconstructs its input image. Its loss has three terms §3.3 eq. 9:

• Reconstruction, an L1 difference between input and output §3.3 eq. 10.
• Discriminative term, the negative L1 norm of the first-order feature gradient, weighted by exp(-|∇I|) so that low-texture pixels count more §3.3 eq. 11 eq. 12.
• Convergent term, the L1 norm of the second-order feature derivative §3.3 eq. 13.

The two regularisers are applied to the largest feature map of the encoder, which has 64 channels §3.5.

Total loss. The paper states L_total = L_s + L_{s→t} §3.4 eq. 14. The cross-view term is the feature-metric loss plus the photometric loss §3.4 eq. 15. The feature-metric loss is an L1 difference of warped and target features §3.4 eq. 16. The photometric loss is 0.15 times an L1 difference plus 0.85 times (1 - SSIM)/2 §3.4 eq. 17.

Occlusion. The cross-view loss takes the minimum over the source set V §3.4 eq. 18. V holds the previous and next frames for monocular video. For calibrated binocular video, the opposite stereo frame is added §3.4.

# Feature-metric training objective  [§3.2] [§3.3] [§3.4]
# Input:  target (B, 3, 320, 1024); sources: list of (B, 3, 320, 1024)
# Output: scalar loss

def training_loss(target, sources, alpha, beta):
    # FeatureNet single-view loss: reconstruction plus two regularisers
    # applied to the largest 64-channel encoder feature  [§3.3] [eq. 9]
    feats, recon = feature_net(target)
    loss_single = (
        l1(target, recon)                             # [eq. 10]
        + alpha * discriminative(feats, target)       # [eq. 11] [eq. 12]
        + beta * convergent(feats)                    # [eq. 13]
    )

    # Depth and pose drive the warp for every source frame  [§3.4]
    depth = depth_net(target)                         # [§3.4]
    cross = []
    for source in sources:
        pose = pose_net(target, source)               # axis-angle  [§3.5]
        src_feats = feature_encoder(source)           # [§3.3]
        # Feature-metric plus photometric term for this source  [eq. 15]
        loss_fm = l1(warp(src_feats, depth, pose), feats)   # [eq. 16]
        loss_ph = photometric(warp(source, depth, pose), target)  # [eq. 17]
        cross.append(loss_fm + loss_ph)

    # Per-pixel minimum over sources handles occlusion  [eq. 18]
    loss_cross = minimum_over_sources(cross)

    # Total objective  [eq. 14]
    return loss_single + loss_cross

Notes on the pseudocode: alpha and beta are 1e-3 §3.3. The paper does not state how the L1 terms are reduced (sum or mean), and the feature resolution at which the warp is applied is not stated.

Hyperparameters

Results

Author-reported. Setup: KITTI, Eigen split with 697 test images §4.1. Tables 2 and 4 report the paper's results and the baselines it lists. Abs Rel and Sq Rel are lower-is-better; delta metrics are higher-is-better tab. 1 tab. 2.

Monocular depth, selected rows tab. 2:

Codes from the tab. 2 caption: M monocular video, MS binocular video,

• online refinement, † post-processing. Supervised baselines Dorn and BTS are listed in the same table.

Odometry, KITTI sequences 09 and 10 tab. 4:

Cross-view loss ablation tab. 3a:

Single-view loss ablation, with odometry tab. 3b. The training setting is not stated in the caption:

Qualitative comparison with Monodepth2 is in fig. 2. The feature visualisations are in fig. 3 tab. 3.

Paper vs code

No official code. The paper and abstract contain no code link. The arXiv metadata has no comments field that names a repository. The paper links no project page. A GitHub repository search for the title words returned one hit, sconlyshootery/FeatDepth (MIT), whose README calls itself the official code. Its owner is not one of the authors and the page does not show a link to them, so the repository is not treated as the authors' own and is not cited. The DeepMotionAIResearch organisation (the authors' affiliation) lists only DenseASPP and DenseMatchingBenchmark. A web search was not run because the permission was not granted.

Pitfalls

• Odometry training set conflicts within the paper. §4.1 says the model is trained on sequences 00 to 08. §4.2 says the model was trained on KITTI raw data and applied to sequences 09 and 10. An implementer has to pick one, and the odometry numbers depend on which.
• Headline δ1 comparison. The abstract's 0.885 to 0.925 uses a baseline row (DepthHint, MS†) that includes post-processing, and the 0.925 result uses online refinement and binocular training tab. 2. The plain monocular model reports δ1 0.893 tab. 2.
• Joint or separate training of FeatureNet. §3.4 says FeatureNet is supervised by the single-view loss and that its features define the cross-view loss. The text does not say whether the cross-view gradient updates FeatureNet. The conclusion calls the framework end-to-end trainable §5. Both readings are consistent with the text.
• Gradient definitions. ∇¹ is defined as ∂x + ∂y and ∇² as ∂xx + 2∂xy + ∂yy §3.3. These are not the usual norm or Laplacian. Whether the derivatives are applied per channel, and how they are discretised, is not stated. A signed sum of partials can cancel, so the implementation choice matters.
• Feature-channel norm. Eqs. 11 to 13 use an L1 norm of the feature gradients, but the reduction across the 64 channels is not stated.
• Discriminative sign. The term enters the loss with a minus sign so that feature gradients are pushed up §3.3 eq. 11. The paper reports that the discriminative term alone can give a zig-zag feature landscape §4.3.
• Convergent term is unweighted. The image-weighted version of the second term appears only in commented-out text. The paper's active equation uses the plain L1 norm §3.3 eq. 13.
• Learning-rate schedule. The starting learning rate of the warm-up is not stated. The phrase "halved twices" is read as two halvings, at epochs 20 and 30 §3.5. The resulting step size after epoch 30 is inferred, not stated.
• Batch size. Batch size 2 on 8 GPUs can mean 2 per GPU or 2 in total §3.5. This doubles or divides the effective batch.
• Depth output. The network output is a sigmoid map §3.5. The mapping to metric depth, and any clamping, is not stated in the active text.
• Occlusion minimum. Eq. 18 takes the minimum of L_{s→t} over sources, but the text does not say whether the minimum is taken before or after the feature and photometric terms are summed. Eq. 15 sums them per source, so the summed form is read here.
• Scale ambiguity in odometry. For methods that suffer from scale ambiguity the paper uses one global scale per sequence §4.2. Whether this paper's reported numbers use that alignment is not stated.
• Ablation coverage. No row tests the feature-metric loss without the photometric loss. Every row of tab. 3a includes L_ph.
• Equation numbers. Equation numbers here were counted from the LaTeX source in order, because the compiled PDF in the source archive is the venue template and not the paper. If a number does not match the rendered page, cite the section.

Lineage

• Builds on: photometric loss and its SSIM term from Monodepth §3.4; occlusion handling by taking the minimum over source frames from Monodepth2 §3.4; online refinement from struct2depth and GLNet §3.5; auto-encoder reconstruction §2.
• Compared against: Monodepth2, DepthHint, DFR, SC-SfMLearner, NeuralBundler, DNC, LEGO, and other self-supervised and supervised methods tab. 2 tab. 4.
• Superseded by: not stated in this wiki.