Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation

In plain words

Training a depth network from video is hard because nobody supplies the correct depth. Instead, the network guesses a depth for each frame, uses that guess and the estimated camera motion to warp a neighbouring frame into the current view, and is scored on how well the pixels match. Weak spots show up as blurry edges and an incomplete overall layout of the scene.

This paper extends a known model of this kind with two attention blocks that work across the channels of the internal feature maps. The first lets each channel collect information from the channels that respond to other parts of the picture, which helps the network grasp the overall scene structure. It adds no trainable weights. The second re-weights channels where the decoder combines fine detail from early layers with coarse context from later layers, so object edges stay sharper.

The authors report more accurate and sharper depth than the methods they compare against on the KITTI benchmark. These are their own numbers.

TL;DR

CADepth-Net is a self-supervised monocular depth network built on Monodepth2. A structure perception module applies channel self-attention to the encoder output and adds no parameters §3.3 tab. 2. Detail emphasis modules re-weight channels at each decoder scale §3.4. The authors report AbsRel 0.105 at 640x192 on KITTI Eigen with a ResNet50 encoder, and state-of-the-art KITTI results §4.2. No official code was found. Some setups are unclear: what "MS" training means and which backbone the Monodepth2 row in Table 1 uses (see Pitfalls).

Problem

Supervised monocular depth needs dense ground truth, which is expensive to collect from LiDAR §1. Self-supervised methods instead use geometric constraints from monocular video or stereo pairs §1. The authors argue that prior work learns scene structure only implicitly, which leaves the overall layout incomplete in complex scenes §1. They also argue that U-Net decoders that fuse features by concatenation and a plain convolution lose detail and produce blurry artefacts at depth discontinuities §1. The model is built on Monodepth2 §2.2, which is not a card in this wiki.

Method

The method has two parts: a standard self-supervised training objective §3.1, and two channel-attention modules inserted into a U-Net-style depth network §3.2.

Self-supervised training §3.1

A depth network predicts a depth map D_t from the target frame I_t. A pose network predicts the relative camera pose T_{t→t'} for each neighbouring frame t' in {t-1, t+1}. The neighbouring frames are warped into the target view with the predicted depth, the pose and the intrinsics K, using bilinear sampling §3.1. The loss is a per-pixel minimum photometric error over the two source frames eq. 1. The photometric error mixes SSIM and L1 with weight alpha eq. 2.

L_{p} = \min_{t'} pe\left(I_{t}, I_{t' \rightarrow t}\right) eq. 1

pe = \frac{\alpha}{2}\left(1 - \mathrm{SSIM}\left(I_{t}, I_{t' \rightarrow t}\right)\right) + (1 - \alpha)\left\|I_{t} - I_{t' \rightarrow t}\right\|_{1} eq. 2

I_{t' \rightarrow t} = I_{t'}\left\langle \mathrm{proj}\left(D_{t}, T_{t \rightarrow t'}, K\right)\right\rangle eq. 3

Auto-masking handles cases where the static-scene assumption breaks, such as a stationary camera or moving objects §3.1. It drops pixels for which the unwarped source already matches the target better than the warped one. The mask is an Iverson bracket computed on the forward pass eq. 4.

\mu = \left[\min_{t'} pe\left(I_{t}, I_{t' \rightarrow t}\right) < \min_{t'} pe\left(I_{t}, I_{t'}\right)\right] eq. 4

An edge-aware smoothness term regularises texture-less regions §3.1. It uses the mean-normalised inverse depth d*_t = d_t / mean(d_t), which the paper says discourages the depth from shrinking §3.1.

L_{s} = \left|\partial_{x} d_{t}^{*}\right| e^{-\left|\partial_{x} I_{t}\right|} + \left|\partial_{y} d_{t}^{*}\right| e^{-\left|\partial_{y} I_{t}\right|} eq. 5

The total loss averages, over S scales, the masked photometric loss plus the weighted smoothness loss §3.1. The number S is not stated.

L = \frac{1}{S} \sum_{i}^{S} \left(\mu L_{p}^{i} + \lambda L_{s}^{i}\right) eq. 6

Network §3.2 tab. 4 app. §1

• Encoder: ResNet50 with ImageNet weights §4.1 app. §1. The structure perception module takes the encoder output at 1/32 resolution, 2048 channels tab. 4.
• Decoder: five upsampling stages (upconv5 to upconv1). Each stage applies a detail emphasis module (dem5 to dem1) to the upsampled features and, except at the highest resolution, an encoder skip feature. An ELU convolution (iconv) follows tab. 4. The paper still applies its detail emphasis module at the highest resolution, where there is no skip connection app. §1.
• Outputs: sigmoid disparity heads at four scales (disp4 to disp1) tab. 4. The paper upsamples the predicted inverse depths to the input resolution with nearest-neighbour interpolation and computes the loss at that resolution §3.2.
• Depth conversion: D = 1 / (a*sigma + b), where sigma is the sigmoid output. The constants a and b are chosen so that D lies between 0.1 and 100 units §4.1. The values of a and b are not stated.
• Pose network: ResNet50 with six input channels, so two adjacent frames go in together. Output is a 6-DoF pose (translation and Euler angles) scaled by 0.01 §4.1. For the 1024x320 and 1280x384 settings, the pose encoder is a ResNet18 trained at 640x192 to save memory app. §1.

Structure perception module (SPM) §3.3 fig. 3

The module is a self-attention over channels. It adds no learned parameters tab. 2.

Given the encoder feature map F with shape (C, H, W), flatten it to (C, N), where N = H*W §3.3. The channel similarity matrix S is F times its transpose, and each entry is the dot product of two channel maps eq. 7.

S_{ij} = F_{i} \cdot F_{j}^{T} eq. 7

The module converts similarity into discrimination with a subtraction eq. 8. Channels that respond to different regions get higher scores. The attention map is a softmax over j of the discrimination matrix eq. 9. The output adds the attention-weighted sum of all channels back to the input eq. 10.

D_{ij} = \max_{i}(S) - S_{ij} eq. 8

A_{ij} = \frac{\exp(D_{ij})}{\sum_{j=1}^{C} \exp(D_{ij})} eq. 9

E_{i} = \sum_{j=1}^{C} \left(A_{ij} F_{j}\right) + F_{i} eq. 10

# Structure perception module  [§3.3] [eq. 7-10]
# Input:  features (B, C, H, W) from the encoder  [app. §1]
# Output: features (B, C, H, W), no new parameters  [tab. 2]

def structure_perception(features):
    # Flatten each channel to its N = H*W pixel values  [§3.3]
    flat = reshape(features, (B, C, H * W))            # (B, C, N)
    # Channel-to-channel similarity S = F F^T  [eq. 7]
    similarity = bmm(flat, transpose(flat))            # (B, C, C)
    # Discrimination: subtract similarity from a max  [eq. 8]
    # The max is read as per-row or global; see Pitfalls.
    discrimination = max_of(similarity) - similarity
    # Attention: softmax over the second channel index  [eq. 9]
    attention = softmax(discrimination, dim=-1)        # (B, C, C)
    # Channel mixing, then residual add of the input  [eq. 10]
    mixed = bmm(attention, flat)                       # (B, C, N)
    return reshape(mixed, (B, C, H, W)) + features

Detail emphasis module (DEM) §3.4 fig. 4

The module fuses a skip feature L with a decoder feature H and re-weights the channels. Its output keeps the channel count of the concatenated input, as the DEM channel counts in tab. 4 show.

The two inputs are concatenated, then a convolution, batch normalisation and ReLU produce U eq. 11. The paper writes the convolution as "3x3 or 1x1" and does not say which one is used where. A global average pool of U feeds two 1x1 convolutions, the first followed by ReLU (the paper defines sigma as ReLU in eq. 11) and the second followed by a sigmoid. The result is a channel weight vector V eq. 12. The output adds U back to V times U eq. 13.

U = \sigma\left(BN\left(W_{1} \otimes f(L, H)\right)\right) eq. 11

V = \delta\left(W_{2} \otimes \sigma\left(W_{3} \otimes \left(\frac{1}{H \times W} \sum_{i=1}^{H} \sum_{j=1}^{W} U_{i,j}\right)\right)\right) eq. 12

O = V \odot U + U eq. 13

# Detail emphasis module  [§3.4] [eq. 11-13]
# Input:  skip (B, C_l, H, W), decoder (B, C_h, H, W)
# Output: (B, C_l + C_h, H, W)

def detail_emphasis(skip, decoder):
    # Concatenate the skip and decoder features  [§3.4]
    fused = concat([skip, decoder], channel_axis)
    # Convolution, batch norm, ReLU  [eq. 11]
    features = relu(batch_norm(conv(fused)))           # U
    # Squeeze spatial dimensions to one vector per channel  [eq. 12]
    pooled = mean(features, dim=(H, W))                # (B, C)
    # Two 1x1 convolutions, ReLU between, sigmoid at the end  [eq. 12]
    weights = sigmoid(conv_2(relu(conv_3(pooled))))    # V in [0, 1]
    # Re-weight channels and keep a residual path  [eq. 13]
    return weights * features + features               # O

Hyperparameters

Architecture table tab. 4, as printed in the supplementary material. Here k is kernel size, s stride, chns output channels, res downscaling factor, and activation the non-linearity. The source marks upsampling with an arrow; it is written as up(...) here.

Results

Author-reported. Setup: KITTI 2015 with the Eigen split, distances up to 80 m, per-image median scaling, trained on KITTI only tab. 1 §4.2. Make3D: trained on KITTI, tested on a 2:1 centre crop with median scaling §4.3 tab. 3.

KITTI Eigen, selected rows tab. 1. Lower is better for the first four columns, higher for the last three. Full table with all baselines in the source.

The text states that CADepth-Net outperforms the existing self-supervised approaches on all metrics §4.2. In the 640x192 M group, the CADepth-Net row is bold in every column tab. 1. The "MS" label is not defined in the captions (see Pitfalls).

Ablation on KITTI Eigen, 640x192, monocular training tab. 2. Backbone, modules, parameters and inference time are as printed; time is on an RTX 3090.

KITTI improved ground truth, selected rows tab. 5 app. §2. Uses the improved depth maps for 652 of the 697 Eigen test frames. The text states that CADepth-Net outperforms all the compared methods on all metrics app. §2.

The table also lists Johnston et al. (640x192, M) and a Monodepth2 MS row; both are omitted here tab. 5.

KITTI online benchmark, selected rows tab. 6 app. §3. Trained on a new split of 72,084 training, 6,060 validation and 500 test examples. Median scaling is unavailable here, so the scale factor comes from 2,000 KITTI training samples with ground truth app. §3.

Make3D (trained on KITTI, monocular) tab. 3 §4.3. Lower is better.

Backbone comparison, KITTI Eigen tab. 7 app. §4. All trained with the same settings.

Qualitatively, the paper shows sharper depth on thin structures such as road signs and poles fig. 1 fig. 5, and better depth on distant objects with the SPM than without app. §4. The SPM visualisations show channel maps that pick up distant regions and the vanishing point fig. 6. The DEM visualisations show high-weight channels that highlight object boundaries fig. 7.

Paper vs code

No official code.

The paper and its bibliography link no repository or project page. A GitHub repository search for "CADepth-Net", "CADepth" and "cadepth-net in:readme" returned two candidates, neither accepted:

• One repository under a personal account that is not an author. Its README calls it the official implementation and lists the four authors, but the paper does not link it and the owner is not an author or the authors' lab. Its license is listed as MIT. It was not cloned and its code was not read.
• A third-party reproduction of Monodepth2 that bundles CADepth-Net, with a license that is not clearly identified. Not used.

No code was read, so none of the paper's hyperparameters or equations were checked against an implementation.

Pitfalls

• Max in discrimination eq. 8. The paper writes max_i(S) without saying whether the maximum is per row, per column or global. A global maximum is a constant across j, so the softmax over j in eq. 9 cancels it and the operation reduces to softmax of minus S. A per-row or per-column maximum changes the attention. Pick one and check the result against the figures in fig. 6.
• Mixing direction eq. 10. The text says the matrix product uses the transpose of A §3.3, but eq. 10 writes A_ij F_j. The two differ unless A is symmetric, which a softmax output generally is not. The printed equation and the text disagree and the code was not read, so this is unresolved.
• Similarity sign. Channels with high similarity get low discrimination, so they get low attention weight. This is the design, but it is easy to invert when implementing from the equations alone.
• No scale on the SPM residual. eq. 10 adds the attention output to the input with no learned weight. Do not add one unless the code does.
• DEM internals. The DEM reduction ratio in the two 1x1 convolutions is not stated. The first convolution's kernel size (3x3 or 1x1) is not stated per scale. The middle activation is inferred from the definition of sigma as ReLU eq. 11. The channel width of U is inferred from the DEM channel counts in tab. 4; the paper does not state it.
• Monodepth2 row in Table 1. The 640x192 M Monodepth2 row (0.115 / 0.903 / 4.863) matches the ResNet18 baseline in the ablation tab. 2, not the ResNet50 baseline (0.110 / 0.831 / 4.642). The ResNet50 row is commented out in the source. Compare the ablation with the ResNet50 baseline when judging the gain. The Table 1 caption does not say which backbone it uses.
• "MS" training. The captions define only S (stereo) and M (monocular). The method describes only monocular training with the pose network §3.1. The MS rows are therefore not described by the paper; do not assume a stereo term unless the authors say so.
• Parameter count. The text says 59M parameters for the full ResNet50 model §4.6; the table says 58.34M tab. 2. Use the table value.
• Disparity resolution column. disp4, disp3 and disp2 in tab. 4 list res 1, although their inputs (iconv4, iconv3, iconv2) are at res 8, 4 and 2. The paper says the predicted disparities are upsampled to the input resolution before the loss §3.2. The column probably gives the size after upsampling, but the paper does not say so.
• Depth conversion constants. The constants a and b in D = 1/(a*sigma + b) are not stated. Keep the 0.1 to 100 clamp and check the output scale.
• Loss normalisation. The paper does not state whether the L1 term is averaged or summed over channels and pixels, nor how the per-scale losses are weighted beyond the 1/S average eq. 6.
• Evaluation scaling. All KITTI Eigen and Make3D numbers use median scaling against ground truth §4.2 §4.3. The online benchmark cannot, and uses a scale fitted on 2,000 training samples app. §3. Results with different scaling are not comparable.
• Cost. The full model uses 58.34M parameters and takes 28.41 ms per frame on an RTX 3090 at 640x192 with ResNet50 tab. 2. The authors describe this as real-time, which is their claim §4.6.
• Unstated training details. Weight decay, warm-up, data augmentation, number of loss scales and the shape of the learning-rate schedule beyond "decay to 1e-5 after 15 epochs" are not stated §4.1.
• Claims. "State-of-the-art" is the authors' claim, measured against the baselines in their own tables §4.2. The paper evaluates only on KITTI and Make3D §4.

Lineage

• Builds on: Monodepth2 (Godard et al., ICCV 2019), which the paper extends with its two modules §2.2. Not a card in this wiki; no arXiv id is in the paper's bibliography.
• Related attention work: spatial self-attention for monocular depth by Johnston et al. (2020). The paper contrasts its channel-wise design with that spatial approach §2.3.
• Compared against: the KITTI Eigen table includes SfMLearner, GeoNet, DDVO, Struct2depth, SIGNet, SGDepth, Monodepth2, PackNet-SfM, HR-Depth, DF-Net, CC, FeatDepth, DepthHints, DualNet, EPC++, TrianFlow and Johnston et al. tab. 1. The other tables add further baselines, including Karsch, Liu, Laina and Vid2Depth tab. 3 tab. 5 tab. 6.
• Superseded by: not stated in this wiki.