HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation

In plain words

Estimating depth from a single photo is useful for robots and cars, but normally needs a depth sensor for training. This paper trains the network from ordinary video instead: it checks that neighbouring frames still line up after the predicted depth is used to shift pixels around.

The authors found that feeding in bigger images does not help the usual models much. The network predicts at low resolution and the result is stretched up with simple interpolation, which smears the edges where depth changes fast. Their fix is a decoder that keeps detailed high-resolution features and mixes in meaning from deeper layers, plus a small attention-style block for merging those features. They also train a much smaller model to copy a larger one, and that small model still does well at high resolution.

TL;DR

HR-Depth is a self-supervised monocular depth network with a redesigned U-Net decoder: dense skip connections give the high-resolution features more semantic content, and an fSE block fuses them. On KITTI it reports the best Abs Rel among the self-supervised methods it compares with, at 640x192 and at higher resolutions. Lite-HR-Depth, a MobileNetV3 model trained by distillation, reaches Monodepth2-level or better accuracy at high resolution with 3.1M parameters. All numbers are author-reported. The released code is evaluation-only.

Problem

Self-supervised depth networks are usually trained at low resolution, because high resolution costs memory §1. When the same models are run at higher resolution, their error hardly moves: Monodepth2 scores Abs Rel 0.115 at both 640x192 and 1024x320 tab. 1. The authors attribute this to bilinear up-sampling of a low-resolution prediction, whose error is largest where the depth gradient is large, such as object boundaries §3.2 fig. 2. Better high-resolution results therefore need more accurate depth in those large-gradient regions §3.2.

Lightweight alternatives have a different problem: they lose accuracy when they shrink, and the authors cite PyDNet at 1.9M parameters as an example §2.3 tab. 3.

Method

Setup. The system has two networks. The depth network (DepthNet) predicts a disparity map from the target frame. The pose network (PoseNet) predicts the camera motion to each of two neighbouring frames, so the target can be warped into those frames and compared §3.1. Training sequences are three consecutive frames, I(t-1), I(t) and I(t+1) §4.2. The Monodepth2 pose design is reused: ResNet-18 with the first convolution widened from 3 to 6 input channels, outputting six numbers, three translations and three Euler angles §4.2.

Encoders. HR-Depth uses ResNet-18 as encoder §4.2. Lite-HR-Depth uses MobileNetV3 §4.2 §3.5.

Loss. The photometric error mixes an SSIM term and an L1 term with weight alpha = 0.85 §4.2 eq. 1. It is taken per pixel as the minimum over the two source frames, following Monodepth2, to handle occlusion §3.1 eq. 2. An edge-aware smoothness term regularises low-texture areas §3.1 eq. 3. The final loss sums the two terms over the output scales, with smoothness weight lambda = 1e-3 §4.2 eq. 4.

r(I_t, I_{t'}) = \frac{\alpha}{2}\,(1 - \mathrm{SSIM}(I_t, I_{t'})) + (1 - \alpha)\,\lVert I_t - I_{t'} \rVert_1 eq. 1

L_{re} = \min_{t'} r(I_t, I_{t'}) eq. 2

L_{smooth} = |\delta_x D_t|\, e^{-|\delta_x I_t|} + |\delta_y D_t|\, e^{-|\delta_y I_t|} eq. 3

L_{final} = \frac{1}{s} \sum_{i}^{s} \left( L^{i}_{re} + \lambda\, L^{i}_{smooth} \right) eq. 4

Here I_t' is the source frame warped into the target view, s is the number of scales, and the paper's text uses four disparity outputs in the loss §3.1 §4.2.

Decoder: dense skip connections. The U-Net skip connections are replaced by a grid of nodes §3.3 fig. 3. Encoder node x^e_i is the output of encoder level i. Intermediate node x_{i,j} aggregates the encoder node, the earlier nodes on the same row and one node from below. Decoder node x^d_i is an fSE fusion node on the decoder path fig. 3. The paper writes the grid as follows §3.3:

x^{e}_{i} = \begin{cases} \mathcal{E}(I), & i = 1 \\ \mathcal{E}(x^{e}_{i-1}), & i > 1 \end{cases} eq. 5

x_{i,j} = \begin{cases} \mathcal{F}\big(\big[x^{e}_{i}, [x_{i,k}]_{k=0}^{j-1}\big],\ \mathcal{U}(x_{i+1,j-1})\big), & j = 1 \\ \mathcal{F}\big(\big[x^{e}_{i}, [x_{i,k}]_{k=0}^{j-1}\big],\ \mathcal{U}(x^{e}_{i+1})\big), & j > 1 \end{cases} eq. 6

x^{d}_{i} = \begin{cases} \mathcal{D}\big(\mathcal{U}(x^{d}_{i+1})\big), & j = 0 \\ \mathcal{D}\big(\big[x^{e}_{i}, [x_{i,k}]_{k=0}^{j-1}\big],\ \mathcal{U}(x^{d}_{i+1})\big), & j > 0 \end{cases} eq. 7

Here E is a residual feature-extraction block, F is a convolution plus activation, D is a fusion block, U is a convolution followed by bilinear up-sampling, and [.] is concatenation §3.3. Equation 7 tests j although x^d_i carries only i (see Pitfalls).

Decoder: fSE block. A plain 3x3 fusion convolution has C_in x C_out x k^2 + C_out parameters §3.4 eq. 8. The fSE block replaces the fusion on the decoder nodes with channel re-weighting. It pools each channel globally, passes the result through two fully connected layers with a sigmoid, multiplies the features by the resulting weights, then fuses the channels with a 1x1 convolution §3.4 fig. 3. Its parameter count is (2/r) C_in^2 + (C_in + 1) C_out, with reduction ratio r = 4 in all experiments §3.4 eq. 9.

C_{in} \times C_{out} \times k^{2} + C_{out} eq. 8

\frac{2}{r} \times C_{in}^{2} + (C_{in} + 1) \times C_{out} eq. 9

Lite-HR-Depth. A MobileNetV3 encoder and reduced decoder nodes give 3.1M parameters, 2.82M of them in the encoder §3.5. A large network is first trained self-supervised to give a teacher T. A student S is then trained to match the teacher's disparity with the loss below §3.5 eq. 10. The paper does not say which teacher model was used, nor the norm.

L_{sup} = \lVert d_{\mathcal{T}} - d_{\mathcal{S}} \rVert eq. 10

Output. Each decoder output is a 3x3 convolution followed by a sigmoid fig. 3 HR_Depth_Decoder.py. At test time only the finest output is used, resized to ground-truth resolution with bilinear interpolation §4.2 evaluate_depth.py. Predictions are then scaled per image by the median ratio to LiDAR ground truth, and capped at 80 m tab. 2 caption evaluate_depth.py.

The sigmoid output is mapped to depth by the code as follows. The paper does not state the depth range it uses, and the code docstring points to an "additional considerations" section that is not in this paper's source layers.py.

# Sigmoid output to depth  [code: layers.py#L16-L25]
# Evaluation calls it with min 0.1 and max 100.0  [code: evaluate_depth.py#L94]
def disp_to_depth(sigmoid_out, min_depth, max_depth):
    # Inverse depth at the two ends of the range  [code: layers.py#L21-L22]
    min_disp = 1 / max_depth
    max_disp = 1 / min_depth
    # Linear map onto that inverse-depth range  [code: layers.py#L23]
    scaled_disp = min_disp + (max_disp - min_disp) * sigmoid_out
    return 1 / scaled_disp

The decoder grid can be written as the released code does it. Node names give the row (encoder level) first and the column second.

# HR-Depth decoder: grid of nodes X[row][col], 5 encoder levels  [§3.3] [fig. 3]
# encoder[i] is level i, finest (i = 0) to coarsest (i = 4)
# [code: networks/HR_Depth_Decoder.py#L95-L101]
# Input: encoder, a list of 5 feature maps
# Output: disparity maps at scales 0 to 3, each in (0, 1)

# Visit order, as in the released code  [code: HR_Depth_Decoder.py#L18]
ORDER = ["01", "11", "21", "31", "02", "12", "22", "03", "13", "04"]
# Nodes whose fusion is an fSE block  [code: HR_Depth_Decoder.py#L19]
ATTENTION_NODES = ["31", "22", "13", "04"]

def decode_disparities(encoder):
    # Column 0 holds the encoder outputs  [§3.3] [eq. 5]
    grid = {(i, 0): encoder[i] for i in range(5)}
    for name in ORDER:
        row = int(name[0])
        col = int(name[1])
        # Earlier nodes on this row, kept for dense skips  [§3.3] [eq. 6]
        left = [grid[(row, k)] for k in range(col)]
        # Node one level down and one column left: convolve  [eq. 6]
        below = conv_0[row + 1, col - 1](grid[(row + 1, col - 1)])
        if name in ATTENTION_NODES:
            # fSE up-samples its first input itself, then re-weights channels  
            # [code: layers.py#L324-L326] [§3.4] [eq. 9]
            grid[(row, col)] = fse[name](below, left)
        else:
            # Up-sample, then concatenate with the left nodes  [eq. 6]
            up = upsample(below)
            # For col > 1 a 1x1 convolution follows, not in eq. 6  
            # [code: HR_Depth_Decoder.py#L57-L63]
            fused = concat([up] + left)
            if col > 1:
                fused = conv_1x1[name](fused)
            grid[(row, col)] = conv_1[row + 1, col - 1](fused)
    # Scale 0: extra convolution and up-sampling on X04
    # [code: networks/HR_Depth_Decoder.py#L114-L116]
    x = conv_0[0, 4](grid[(0, 4)])
    x = conv_1[0, 4](upsample(x))
    disp = [None] * 4
    disp[0] = sigmoid(disp_conv[0](x))
    # Scales 1 to 3 read nodes X04, X13 and X22
    # [code: networks/HR_Depth_Decoder.py#L117-L120]
    disp[1] = sigmoid(disp_conv[1](grid[(0, 4)]))
    disp[2] = sigmoid(disp_conv[2](grid[(1, 3)]))
    disp[3] = sigmoid(disp_conv[3](grid[(2, 2)]))
    return disp

The loss at each scale is written as the paper's equations 1 to 4 imply it. Resizing images to each scale is not stated.

# Photometric error between the target and one warped source  [§3.1] [eq. 1]
# alpha = 0.85  [§4.2]
def photometric_error(target, warped, alpha):
    # SSIM dissimilarity, weighted alpha / 2  [eq. 1]
    ssim_term = (alpha / 2) * (1 - ssim(target, warped))
    # Pixel-wise L1 error, weighted 1 - alpha  [eq. 1]
    l1_term = (1 - alpha) * abs(target - warped)
    return ssim_term + l1_term

# Per-pixel minimum over the two source frames  [§3.1] [eq. 2]
def reprojection_loss(target, warped_prev, warped_next, alpha):
    # Keep the smaller error at each pixel, which handles occlusion  [eq. 2]
    error_prev = photometric_error(target, warped_prev, alpha)
    error_next = photometric_error(target, warped_next, alpha)
    return minimum(error_prev, error_next)

# Edge-aware smoothness of a disparity map  [§3.1] [eq. 3]
def smoothness_loss(disparity, target):
    # Disparity gradients in x and y, damped where the image has edges  [eq. 3]
    grad_x = abs(dx(disparity)) * exp(-abs(dx(target)))
    grad_y = abs(dy(disparity)) * exp(-abs(dy(target)))
    # Reduction over pixels and channels is not stated in the paper  [§3.1]
    return grad_x + grad_y

# Final loss, averaged over the output scales  [§3.1] [eq. 4]
# lambda = 1e-3  [§4.2]
def total_loss(disparities, targets, sources, alpha, lam):
    losses = []
    for scale in range(4):
        # Images at this scale: the resize method is not stated  [§3.1]
        target = targets[scale]
        warped_prev, warped_next = sources[scale]
        # Photometric term plus weighted smoothness at this scale  [eq. 4]
        scale_loss = reprojection_loss(target, warped_prev, warped_next, alpha)
        scale_loss = scale_loss + lam * smoothness_loss(
            disparities[scale], target)
        losses.append(scale_loss)
    return mean(losses)

Distillation for the lightweight model, in the same form:

# Student-teacher loss for Lite-HR-Depth  [§3.5] [eq. 10]
def distill_loss(teacher_disp, student_disp):
    # Match the student's disparity to the teacher's  [eq. 10]
    # Norm type is not stated in the paper  [§3.5]
    return norm(teacher_disp - student_disp)

Hyperparameters

Results

Author-reported on KITTI, Eigen split, depths up to 80 m, with median scaling against LiDAR at test time tab. 2 caption. "M" means monocular video only, "MS" means monocular plus stereo, "K" is KITTI, "CS+K" is CityScapes pretraining then KITTI. Full Table 2 follows, copied from the paper.

tab. 2

Lite-HR-Depth against Monodepth2 and PyDNet tab. 3. "T" means trained with a teacher network.

Ablation on KITTI at 640x192, monocular training, Eigen split. Baseline is Monodepth2 tab. 4.

tab. 4

Table 1 gives the resolution check that motivates the paper: Monodepth2 at 640x192 gives Abs rel 0.115 and d < 1.25 of 0.877; at 1024x320 it gives 0.115 and 0.879 tab. 1.

Headline claims from the text: HR-Depth is said to outperform all self-supervised methods compared §4.3. The lowest Abs rel in Table 2 is 0.101, from the MS variant at 1024x320 tab. 2. The 640x192 monocular model gains over Monodepth2 are small in Abs rel (0.109 against 0.115) tab. 2. Lite-HR-Depth with teacher at 1024x320 and 1280x384 is better than Monodepth2 at 1024x320 on Abs rel, with 3.1M parameters against 14.84M tab. 3 §3.5. At 640x192 without teacher, Lite-HR-Depth is slightly worse than Monodepth2 on Abs rel (0.116 against 0.115) tab. 3.

Paper vs code

Checked against raw/code/2012.07356 at code_commit 1804ddd. The repository holds evaluation code, the network definitions and pretrained model links. It has no training script, so the loss, optimiser and schedule can only be compared with the paper and the README.

• fSE reduction ratio. Paper sets r = 4 §3.4 eq. 9. Code sets reduction = 16 in the fSE module layers.py. The FC layers are four times smaller in the code than the parameter formula in the paper implies.
• fSE internals. The paper says two fully connected layers followed by a sigmoid §3.4. The code puts a ReLU between the two FC layers, with no bias on either layers.py, and a ReLU after the 1x1 convolution layers.py layers.py. Neither ReLU is in the paper.
• Up-sampling inside the decoder. The paper defines the up-sampling block U as convolution plus bilinear interpolation §3.3 eq. 6. The code's upsample uses nearest-neighbour mode layers.py, and the decoder calls it for every dense connection HR_Depth_Decoder.py. Bilinear interpolation appears only at test time, when the output is resized evaluate_depth.py.
• Dense skip wiring for j > 1. Equation 6, second case, feeds the encoder node x^e_(i+1) into the grid §3.3 eq. 6. The code feeds the intermediate node one column left on the row below, X[i+1, j-1] HR_Depth_Decoder.py. The code's wiring is the only concrete one available here.
• Extra 1x1 convolution in the decoder. The code applies a 1x1 convolution before the fusion block whenever col > 1 for non-fSE nodes HR_Depth_Decoder.py. Equation 6 has no such step.
• Activation in fusion blocks. The paper says only "an activation function" §3.3. The code uses ELU in every ConvBlock layers.py.
• Decoder widths. Not stated in the paper. The code uses channel counts 16 to 256 for ResNet-18 and 4 to 80 for MobileNetV3 HR_Depth_Decoder.py.
• Learning rate. The paper gives 1e-3 §4.2. The README says the paper's value was written wrong and should be 1e-4 README.md.
• Training code. The README says the training code is not yet released, and that the codes are modified from Monodepth2 README.md. Loss, optimiser, schedule and teacher training cannot be compared with code.
• SSIM window. The code computes SSIM with 3x3 average pooling and constants 0.01^2 and 0.03^2 layers.py. The paper does not state a window size §3.1.
• Image-gradient term in smoothness. The released smoothness function averages the image gradient over colour channels before the exponential layers.py. The paper's equation 3 does not say how channels are combined §3.1 eq. 3.
• Evaluation protocol. The paper refers to the Eigen metrics §4.3. The code adds an Eigen crop, a 1e-3 to 80 m validity mask, depth clamping to the same range and a per-image median scale evaluate_depth.py evaluate_depth.py. The error averages are taken over per-image results evaluate_depth.py.
• Depth range for sigmoid output. The code converts sigmoid output with a depth range of 0.1 to 100 at test time evaluate_depth.py. The paper does not state the range §3.2.
• Legacy decoder. networks/depth_decoder.py is a Monodepth2 decoder and reads self.attention_method, which is never set depth_decoder.py. It is not used by the evaluation script, which uses HRDepthDecoder evaluate_depth.py. Do not use it.
• Pretrained models. The README links pretrained weights for four configurations, and the Table 2 and Table 3 numbers match them README.md.

Pitfalls

• No section numbers in the PDF. The paper sets secnumdepth to 0, so no section numbers are printed §1 (source). Citations here count sections in order: §1 Introduction, §2 Related work (2.1 supervised, 2.2 self-supervised, 2.3 lightweight), §3 HR-Depth network (3.1 to 3.5), §4 Experiments (4.1 to 4.5), §5 Conclusion.
• Equation numbers come from the LaTeX source, which has no labels. Equations 5 to 7 are the three rows of one aligned block, numbered in order. Check the PDF before quoting an equation number.
• Figure reference mismatch. Section 4.5 says "Figure 4 shows" the intermediate feature maps §4.5. The feature-map figure is Figure 5 fig. 5. Figure 4 is the qualitative comparison fig. 4.
• Index base. The paper counts encoder levels from i = 1 eq. 5. The code's X_00 through X_40 are levels 0 to 4 HR_Depth_Decoder.py. Shift the paper's index by one when you implement.
• Decoder equation 7 notation. The decoder node x^d_i carries only i, yet its two cases test j eq. 7. The code treats decoder nodes as the diagonal grid nodes X[3,1], X[2,2], X[1,3] and X[0,4] with fSE HR_Depth_Decoder.py.
• fSE parameter count depends on r. The paper's 14.62M (Table 4) and its fSE formula use r = 4 §3.4 tab. 4. The code's r = 16 gives smaller FC layers. Neither can be checked from this release.
• Table 2 duplicate rows. The Monodepth2 MS rows at 640x192 and 1024x320 have identical numbers in every column tab. 2. This may be a copy error in the paper. Not verifiable from this source.
• Decoder widths are in a supplement that is not part of this source. Use the code values with a note that they are code-only.
• Teacher not identified. The Lite distillation teacher, its resolution and its training are not stated §3.5. The released Lite weights are labelled "K_T" at 1280x384 README.md.
• Distillation norm. The norm in equation 10 is not stated §3.5 eq. 10.
• Smoothness reduction. The paper gives no reduction over pixels and channels §3.1 eq. 3. The code's version averages layers.py. Pick one and state it.
• Per-scale images. The paper uses four scales in the loss §4.2 but does not say how images are resized for each scale §3.1.
• Eigen crop and validity mask are needed to reproduce Table 2 numbers and are only in the evaluation code evaluate_depth.py.
• Headline numbers are not like-for-like. Several rows use PackNet-SfM at a different resolution (1280x384 against HR-Depth 1024x320), and the MS and CS+K variants come from different supervision and pretraining. Compare rows with the same supervision and resolution.
• Lite at 640x192 is not better than Monodepth2 on Abs rel (0.116 against 0.115), so the claim "beats Monodepth2" holds at 1024x320 and 1280x384 only tab. 3.
• Pretraining for ResNet-18 is not stated in the paper. The released evaluation builds the encoder without pretrained weights and then loads the saved checkpoint evaluate_depth.py.

Lineage

• Builds on: Monodepth2 (Godard et al., 2019, cited as godard2019digging) for the minimum-over-sources loss §3.1 and the PoseNet design §4.2; U-Net skip connections §1; dense skip connections in the nested U-Net style of Zhou et al. (2018) §3.3; squeeze-and-excitation (Hu et al., 2019) §3.4; MobileNetV3 (Howard et al., 2019) §4.2; distillation (Hinton et al., 2015) §3.5. These are named in the paper and have no cards here.
• Compared against: Monodepth2, PackNet-SfM, Struct2Depth, Vid2Depth, SfMLearner and Zhou et al. tab. 2; PyDNet tab. 3.
• Superseded by: not stated in this source.