Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

In plain words

A self-driving car has to answer several questions about its surroundings at the same time: which pixels are road or car, how far away things are, which laser points belong to which object, and what the scene looks like from above. Usually each question gets its own model, and running all of them is expensive. This paper trains one network that answers all four in a single pass.

The network takes input from several sensors: regular cameras, event cameras that only report pixels whose brightness changes, and a spinning laser scanner (LiDAR). The laser points are turned into a small stack of images, one per height band, so the network can tell low things such as road from tall things such as buildings. Each input has its own part of the network, and the outputs are joined back to those inputs with shortcut links that carry fine detail.

Training several tasks together is hard because their error numbers live on different scales, so one task can take over. The authors adapt an existing method, GradNorm, that adjusts each task's weight during training so the tasks learn at similar speeds. Their version changes the weights only once per epoch to save computation.

The authors report that the combined network does better than a set of separate models on three of the four test sets, while using far fewer parameters and running faster. All of these numbers are their own.

TL;DR

One network with per-input encoders, per-output decoders and two shared bottlenecks predicts four driving perception outputs in one forward pass: depth and semantic segmentation for four views, LiDAR segmentation, and a bird's-eye-view (BEV) map §1 fig. 1. Inputs are RGB, dynamic vision sensor (DVS) and a 15-layer LiDAR height stack §4.3. Loss weights are adapted once per epoch with a modified GradNorm §3.3. Author-reported, the combined network has a lower total metric than the separate single-task baselines together on three of four test sets, but not on set C tab. 2. The code is MIT-licensed and differs from the paper in several places; see Paper vs code.

Problem

Training one model per perception task is costly and inefficient §1. Training several tasks together is not automatically better: tasks can conflict in their gradients, and the training can focus on one task and degrade the others §1. The authors also note that loss weights are tedious to tune by hand §1.

Single sensors fail in some conditions. An RGB camera may capture too little in poor light, so DVS, radar or LiDAR are used as alternatives §1. Earlier BEV methods used other models to produce semantic maps first, so the pipeline did not run in one pass §2.2. Point-based and view-based LiDAR encodings are costly, so a fixed-size tensor is needed §2.2.

The data are three CARLA simulator datasets and one real-world dataset (nuScenes-lidarseg), which is used with a substituted sensor layout and ground truth derived from its LiDAR points §4.1 §4.2.

Method

The network is an encoder-decoder with a specific encoder for each input and a specific decoder for each output §3.1 fig. 2. Its inputs are four RGB views (front, left, right, rear), four DVS views and one top-view LiDAR tensor, all at 128 × 128 §4.3 §4.1. Its outputs are depth for each of the four views, semantic segmentation for each of the four views, a top-view LiDAR segmentation, and a top-view BEV map §3.1.

Each RGB encoder feeds the segmentation decoder of its own view, and each DVS encoder feeds the depth decoder of its own view, by concatenation at matching resolution §3.1. The LiDAR encoder feeds the LiDAR segmentation decoder, and that output feeds the BEV decoder §3.1. The network has two bottlenecks, each a convolution block §3.1. The first fuses the RGB, DVS and LiDAR features and drives the segmentation, depth and LiDAR segmentation decoders. The second stores what those tasks extracted and drives the BEV decoder §3.1. The authors describe the second bottleneck as taking "extracted information" from the other tasks §3.1; the code feeds it re-encoded output maps instead (see Paper vs code).

The convolution block is two 3 × 3 convolutions, each followed by batch normalization and ReLU §3.1. Encoders downsample by 2 × 2 max pooling and decoders upsample by 2 × 2 bilinear interpolation §3.1. Depth uses a ReLU output; segmentation and BEV use sigmoid outputs §3.1. Dropout with p = 0.5 is applied to the blocks at the centre of the network §3.1.

# Forward pass of the multi-task network  [§3.1] [fig. 2]
# rgb:   4 views (F, L, Ri, R), each (3, 128, 128)
# dvs:   4 views, each (2, 128, 128)
# lidar: top view, (15, 128, 128)

def forward(rgb, dvs, lidar):
    # One encoder per input and view  [§3.1]
    rgb_maps = [encode(view) for view in rgb]
    dvs_maps = [encode(view) for view in dvs]
    lidar_map = encode(lidar)
    # Bottleneck 1 fuses RGB, DVS and LiDAR features  [§3.1]
    shared_1 = bottleneck_1(concat(rgb_maps, dvs_maps, lidar_map))
    # Task decoders, with skips to their own input encoders  [§3.1]
    depth = depth_decoder(shared_1, dvs_maps)     # 4 views, ReLU
    seg = seg_decoder(shared_1, rgb_maps)         # 4 views, sigmoid
    lidar_seg = lidar_decoder(shared_1, lidar_map)  # top, sigmoid
    # Bottleneck 2 stores what the three tasks extracted  [§3.1]
    shared_2 = bottleneck_2(encode_outputs(depth, seg, lidar_seg))
    # BEV decoder takes the LiDAR segmentation output  [§3.1]
    bev = bev_decoder(shared_2, encode(lidar_seg))  # top, sigmoid
    return depth, seg, lidar_seg, bev

Training uses a modified GradNorm (MGN) for the loss weights §3.3 alg. 1. The total loss is the weighted sum of the task losses, and a small weight-decay term is added eq. 4 eq. 9. The task losses are Huber for depth and a combined binary cross-entropy and Dice loss for the three segmentation-style tasks §3.2 eq. 1 eq. 2 eq. 3.

GradNorm balances the training rate of the tasks by pushing each task's gradient norm toward a target §2.3 §3.3. The authors pick one parameter tensor per bottleneck: the first layer of bottleneck 1 for depth, segmentation and LiDAR segmentation, and the first layer of bottleneck 2 for BEV §3.3 eq. 10. They update the loss weights only at the last step of each epoch, not at every step, to reduce computation §3.3.

# Training with modified GradNorm (MGN)  [alg. 1]
# Start: network weights W (Kaiming init), w_i(0) = 1 for all tasks i
# Input: batches of 6 samples for each training step

def train_epoch(batches, model, loss_weights):
    for step, batch in enumerate(batches):
        # Forward pass gives the per-task losses L_i(t)  [§3.2]
        preds = model(batch.inputs)
        losses = task_losses(preds, batch.targets)
        # Total loss as a weighted sum over the tasks  [eq. 9]
        total = sum(loss_weights[i] * losses[i] for i in tasks)
        if step == 0:
            # Initial losses L_i(0), taken at the first step of the epoch
            # [alg. 1] [§3.3]
            initial_losses = detach(losses)
        if step == last_step:
            # Only the last step of the epoch updates the loss weights
            # [alg. 1] [§3.3]
            loss_weights = update_loss_weights(
                losses, initial_losses, loss_weights, model)
        # Standard backward pass and step on the network weights  [alg. 1]
        total.backward()
        optimizer.step()
    return loss_weights

The loss-weight update itself uses the gradient norms of the weighted task losses on the chosen layers, a loss-ratio measure of training progress, and a gradient-norm loss that is minimised with SGD.

# Loss-weight update at the last step of an epoch  [alg. 1] [§3.3]
# shared(i): the bottleneck layer chosen for task i  [eq. 10]

def update_loss_weights(losses, initial_losses, loss_weights, model):
    # Norm of the gradient of w_i * L_i on that layer  [eq. 11]
    grad_norms = [
        l2_norm(gradient(loss_weights[i] * losses[i], shared(i)))
        for i in tasks
    ]
    # Average norm over the T = 4 tasks  [eq. 12]
    mean_grad_norm = sum(grad_norms) / T
    # Loss ratio: how far each task loss has fallen  [eq. 13]
    loss_ratios = [losses[i] / initial_losses[i] for i in tasks]
    # Relative inverse training rate  [eq. 14]
    rates = [ratio / mean(loss_ratios) for ratio in loss_ratios]
    # Target gradient norm, with asymmetry alpha = 1.5  [eq. 16]
    targets = [mean_grad_norm * rate ** 1.5 for rate in rates]
    # Gradient-norm loss, summed over tasks  [eq. 15]
    grad_loss = sum(abs(targets[i] - grad_norms[i]) for i in tasks)
    # SGD step on the loss weights, learning rate 0.1  [§3.3] [§4.4]
    loss_weights = sgd_step(loss_weights, gradient(grad_loss, loss_weights))
    # Renormalise so the weights sum to T = 4  [eq. 17]
    return [w * T / sum(loss_weights) for w in loss_weights]

LiDAR points are projected into a fixed 15-channel tensor. Fourteen channels hold points by height, from −2 m to 11 m, and the last channel holds a flat top-view projection of all points §4.3 eq. 19 eq. 20 eq. 21 eq. 22 fig. 6. DVS events are stored as two binary channels, one per polarity §4.3 eq. 18.

Equations

The paper's equations, reproduced in full because an implementer needs them. Symbols are defined in the source: ŷ is the prediction, y the target, N the number of elements, V the number of views and T the number of tasks.

Depth loss, with δ = 0.5 §3.2:

\mathcal{L}_{DE} = \frac{1}{V} \sum_{i = 1}^{V} \frac{1}{N} \sum_{j = 1}^{N} z_{ij} eq. 1

z_{ij} = \begin{cases} 0.5\,(\hat{y}_{ij} - y_{ij})^2 & \text{if } |\hat{y}_{ij} - y_{ij}| < \delta \\ \delta\,(|\hat{y}_{ij} - y_{ij}| - 0.5\delta) & \text{otherwise} \end{cases} eq. 2

Segmentation-style losses, for SS, LS and BEV. As printed in the paper, the binary cross-entropy has no leading minus sign; read it as the standard loss (see Pitfalls):

\mathcal{L}_{\{SS,LS,BEVP\}} = \frac{1}{V} \sum_{i = 1}^{V} \Big( \frac{1}{N} \sum_{j = 1}^{N} y_{ij} \log(\hat{y}_{ij}) + (1 - y_{ij}) \log(1 - \hat{y}_{ij}) \Big) + \Big( 1 - \frac{2 |\hat{y}_i \cap y_i|}{|\hat{y}_i| + |y_i|} \Big) eq. 3

Total training loss, with λ = 0.0001 §3.2:

\mathcal{L}_{total} = \lambda \Sigma \mathcal{W}^2 + \sum_{i = 1}^{T} w_i \mathcal{L}_i eq. 4

Depth metric. The paper prints the inner sum index as i; it should be j:

MAE_{DE} = \frac{1}{V} \sum_{i = 1}^{V} \frac{1}{N} \sum_{j = 1}^{N} |\hat{y}_{ij} - y_{ij}| eq. 5

Segmentation metric. The paper prints the denominator as |ŷ_i| ∪ |y_i|; the intended term is the union |ŷ_i ∪ y_i|:

IoU_{\{SS,LS,BEVP\}} = \frac{1}{V} \sum_{i = 1}^{V} \frac{|\hat{y}_i \cap y_i|}{|\hat{y}_i| \cup |y_i|} eq. 6

Total metric, used for model selection and learning-rate decisions §3.2:

TM = MAE_{DE} + (1 - IoU_{SS}) + (1 - IoU_{LS}) + (1 - IoU_{BEVP}) eq. 7

Metric variance across the four tasks, with T = 4 §3.2:

MV = \frac{1}{T} \sum_{i = 1}^{T} \Big( M_i - \frac{TM}{T} \Big)^2 eq. 8

Weighted total loss at step t §3.3:

\mathcal{L}(t) = \sum_{i = 1}^{T} w_i(t) \mathcal{L}_i(t) eq. 9

Shared weight subsets: W_1 is the first layer of bottleneck 1 and W_2 the first layer of bottleneck 2 §3.3:

W(s) = \{W_1(s), W_2(s)\} \subset \mathcal{W}(s) eq. 10

Gradient norm of the weighted task loss on the chosen layer §3.3:

G_W^{(i)}(s) = \left\lVert \nabla_W \big( w_i(s) \mathcal{L}_i(s) \big) \right\rVert_2 eq. 11

\overline{G}_W(s) = \frac{1}{T} \sum_{i = 1}^{T} G_W^{(i)}(s) eq. 12

Loss ratio, the last step over the first step §3.3:

\bar{\mathcal{L}}_i(s) = \frac{\mathcal{L}_i(s)}{\mathcal{L}_i(0)} eq. 13

Relative inverse training rate §3.3:

r_i(s) = \frac{\bar{\mathcal{L}}_i(s)}{\frac{1}{T} \sum_{i = 1}^{T} \bar{\mathcal{L}}_i(s)} eq. 14

Gradient-norm loss for the loss weights §3.3:

\mathcal{L}_{MGN}(s) = \sum_{i = 1}^{T} \left\lvert \mathcal{G}_W^{(i)}(s) - G_W^{(i)}(s) \right\rvert eq. 15

Target gradient norm, with asymmetry α = 1.5 §3.3:

\mathcal{G}_W^{(i)}(s) = \overline{G}_W(s)\, r_i(s)^{\alpha} eq. 16

Renormalisation so that the loss weights sum to T §3.3:

w_{i_{new}}(s) = \frac{w_i(s)}{\sum_{i = 1}^{T} w_i(s)} T eq. 17

DVS tensor. The pixel is 1 when the event sensor records a polarity at that location, otherwise 0 §4.3:

I_{DVS_{ij}} = \begin{cases} 1 & \text{if there is polarization record in } A_{DVS_{xy}} \\ 0 & \text{otherwise} \end{cases} eq. 18

Top-view LiDAR tensor, using the cosine of the incidence angle as the value §4.3:

I_{LID_{ij}} = \begin{cases} \cos(\theta) & \text{if there is a point record in } A_{LID_{xy}} \\ 0 & \text{otherwise} \end{cases} eq. 19

Mapping of LiDAR x and y to pixel indices, over −32 m to 32 m, with rounding brackets as printed §4.3:

I_{LID_i} = \left\lfloor \frac{A_{LID_x} - (-32)}{32 - (-32)} \times 127 \right\rceil eq. 20

I_{LID_j} = \left\lfloor \frac{A_{LID_y} - (-32)}{32 - (-32)} \times 127 \right\rceil eq. 21

Mapping of LiDAR height to the 15 channels, over −2 m to 11 m §4.3:

I_{LID_k} = \left\lfloor \frac{A_{LID_z} - (-2)}{11 - (-2)} \times 13 \right\rceil eq. 22

Algorithm

The modified GradNorm algorithm is given as Algorithm 1 alg. 1. The pseudocode above follows its steps. The paper's text says that per epoch, only two training steps enter the MGN computation, the first and the last §3.3.

Hyperparameters

Values in the paper and its code. "Not stated" means the paper does not give a value.

Datasets

Data generation settings for the CARLA sets tab. 1:

Results

Author-reported. Setup: four datasets (three CARLA sets A, B and C, and nuScenes-lidarseg), evaluated on each test set §5 tab. 2. The baselines are a combination of single-task models (Chen et al. for BEV, PolarNet for LiDAR segmentation) and a multi-task GradNorm model for depth and segmentation, each retrained by the authors §5.4. The "1L" variant uses a single LiDAR channel, "15L" uses the 15-channel stack, and "+MGN" adds the adaptive loss weights §5.1 §5.2.

Test-set results from tab. 2. Bold is as printed. The ± values are the paper's uncertainty, which is a variance (see Pitfalls). The total metric (TM) of the combined baselines sits on the PolarNet row.

† Averaged over the four views, each by an independent GradNorm model. * No DVS input, because nuScenes-lidarseg has none tab. 2.

Model size and speed from tab. 3. Inference speed is on a GTX 1080 Ti with batch size 1 tab. 3. For the simulated datasets the FPS is averaged over A, B and C.

† Model size and parameters of GradNorm are multiplied by 4 for the four per-view models. * No DVS encoders tab. 3.

Headline claims, as the authors state them §5.4 §6:

• On set A, B and nuScenes-lidarseg, 15L+MGN has a lower total metric (1.393, 1.086, 1.069) than the combined baselines (1.430, 1.160, 1.124). On set C the combined baselines are better (0.976 against 0.999) tab. 2.
• The paper says its models have less than 2% of the combined baselines' parameters §5.4 tab. 3. The table gives 2521504 against 136728752 on the simulated sets.
• The best LiDAR segmentation (IoU_LS) belongs to PolarNet on all four datasets §5.4 tab. 2.
• Adaptive loss weighting lowers TM on all test sets against 15L and has a smaller metric variance on all simulated sets §5.2 tab. 2. The 15L model keeps a lower depth MAE and a higher SS IoU than 15L+MGN on every test set, which the authors describe as a trade-off between the two groups of tasks §5.2 tab. 2.
• Adding the 15-layer LiDAR stack lowers TM against the one-layer variant on all test sets §5.1 tab. 2. The paper attributes the LiDAR segmentation gain to the direct skip from the LiDAR encoder, and the depth and segmentation gains to fusion at the first bottleneck §5.1.
• Learned loss weights order the tasks, at convergence, as depth highest, then LiDAR segmentation, then segmentation, then BEV lowest on the simulated sets. On nuScenes, LiDAR segmentation and segmentation swap §5.3 fig. 9.
• On the qualitative examples, 15L+MGN gives a clearer BEV projection of a car behind the ego vehicle on a rainy night and a better road map on a sunny day. The 15L variant segments overall views better, per the authors §5.2 fig. 7 fig. 8. The authors also report that the combined baselines miss surrounding vehicles on a rainy night §5.4 fig. 7.

Paper vs code

Paths are relative to the repository at code_commit, MIT-licensed, by the first author LICENSE. The paper and the code agree on most optimiser settings (SGD, momentum 0.9, learning rate 0.1, batch size 6, the 0.5-factor plateau schedule with patience 4, the 25-epoch stop, and α = 1.5). The differences below are the ones that change what an implementation computes.

• Depth loss scale. Eq. 2 gives 0.5 (error)² inside δ = 0.5. The code calls smooth_l1_loss(beta=0.5), which returns twice that value at every error size, so the depth term is twice eq. 2 function.py train.py eq. 2.
• Weight decay. Eq. 4 adds λΣW² with λ = 0.0001. The code passes weight_decay=0.0001 to SGD, which adds λW to each gradient: the same as a penalty of (λ/2)ΣW². It is also applied to batch-norm weights and biases, which the paper does not mention train.py eq. 4.
• Segmentation loss. Eq. 3 averages the Dice term outside the sum over views; the code adds one BCE and one Dice term per view and averages them over views train.py function.py. The code uses the standard positive BCE and a Dice with smoothing 1e-7 over the whole batch function.py eq. 3.
• IoU metric. Eq. 6 has no threshold. The code binarises prediction and target at 0.5 and computes one intersection over union over the whole batch and all channels function.py eq. 6.
• Loss-weight gradient reaches the loss weights from the total loss. The code calls total_loss.backward(retain_graph=True) after zeroing the loss-weight gradients, then Lgrad.backward(). Because each task loss is multiplied by its weight, the weight gradients include ∂L_total/∂w_i as well as the gradient of eq. 15 train.py train.py eq. 15.
• Loss-weight gradient reaches the network weights. The gradient norms are computed with create_graph=True, so Lgrad.backward() also writes into the network-weight gradients, which the optimiser applies at the same step. Algorithm 1 updates only the loss weights in this step train.py train.py train.py alg. 1.
• Where L_i(0) comes from. The code resets L_i(0) at the first mini-batch of every epoch, and takes L_i(s) from the last mini-batch of that epoch train.py train.py. The paper's text speaks of "the first epoch" as well as the first step §3.3.
• Timing of the loss-weight update. The paper's algorithm updates the weights before the backward pass at t = s. The code renormalises the weights after the epoch's validation, so the new weights act from the next epoch train.py alg. 1.
• Second stage of the network. The code feeds the outputs of the SS, DE and LS heads (sigmoid or ReLU maps) through a second encoder, and that encoder feeds bottleneck 2 and the BEV decoder's skip inputs. The paper describes this as "extracted information from those tasks" arch.py arch.py §3.1.
• Nuscenes-lidarseg class count. The paper uses 32 classes for SS, LS and BEV on nuScenes §4.2. The code's nuScenes variant E1 has 23 outputs for SS and LS and 9 for BEV arch.py arch.py. The shipped configs select E0 train.py predict.py.
• Data preprocessing is not in the repository. The DVS and LiDAR projections (eq. 18 to 22) are not in the code. The data loader reads precomputed *_128.npy files for LiDAR, segmentation and BEV, and DVS as three-channel images with channel 1 removed util.py util.py util.py. The dataset itself is not in the repository; the README points to an external download README.md.
• Channel widths. The paper does not state them. The code uses 16, 32, 64 and 128 arch.py.
• Output range. The paper says the depth output is in a normalised range of 0 to 1 §3.1. The code has a ReLU only, with no upper bound arch.py arch.py.
• Metric variance. The MV in eq. 8 is not computed in the repository. predict.py computes np.var over per-batch test scores, which is the uncertainty described in the table note predict.py eq. 8 tab. 2.
• Loss-weight order in the config comment. The comment reads "DE, SS, BEVP, LS" but the code orders them DE, SS, LS, BEVP. All initial values are 1, so the effect is nil today train.py train.py.
• Parameter indices. The code selects the bottleneck layers by position in the parameter list, [168, 530], with a #check BOTTLENECK comment. By counting the registration order of E0 (not executed here), these are the first convolution weights of each bottleneck. Any change to the layer order silently changes them train.py train.py.
• Matching settings. Learning rate 0.1, halving with factor 0.5 and patience 4, the minimum learning rates, the 25-epoch stop, and α = 1.5 all match the paper train.py train.py train.py.

Pitfalls

• Depth loss is twice eq. 2. smooth_l1_loss(beta=0.5) is not the paper's formula. If you implement eq. 2 literally, the depth term is half of what the code trains with, and the relative weights of the tasks shift at the start.
• Typos in the printed equations. Eq. 3 has no leading minus on the BCE. Eq. 5 uses i for the inner sum; the index should be j. Eq. 6 writes the union as |ŷ_i| ∪ |y_i| where it should be |ŷ_i ∪ y_i|. Read these as the intended formulas.
• Rounding in LiDAR mapping. Eq. 20 to 22 print ⌊ ⌉ brackets. The code is not in the repository, so it is unknown whether the paper rounds or floors. The paper also says that when two points land in the same voxel, the one with the highest z is kept.
• The DVS mapping is not documented. Eq. 18 describes two binary channels, one per polarity. The loader reads three-channel images and drops channel 1 util.py. How polarity maps to the remaining channels is not stated.
• L_i(0) is a single mini-batch. The initial and final losses in the ratio of eq. 13 are one batch of six samples each, and the code uses the first batch of each epoch. The ratio is noisy, and an implementation that uses a running average would not match.
• No clamp on loss weights. Eq. 17 renormalises to T = 4. The code only multiplies by T / sum, with no lower or upper bound on any weight train.py. A weight can therefore go negative if the gradient step drives it there; the paper does not say what to do in that case.
• The depth metric is in normalised units. The depth target is normalised to 0 to 1 and the paper does not give the scale in metres, so MAE_DE cannot be read as a distance §4.3.
• The "±" values are variances. predict.py computes np.var over per-batch scores predict.py, so the ± values in Table 2 are variances, not standard errors. Treat them as spread indicators only.
• Metric averaging hides batch and view structure. Segmentation IoU is pooled over the batch and then averaged over views for SS. Depth MAE is averaged over views. The paper gives no per-class IoU.
• Numerical precision. The network is run in float64 train.py while loss weights are float32 tensors train.py. The paper does not state precision.
• Dropout placement. The paper says dropout is applied "at the center of the architecture" §3.1. The code applies it after the blocks at 1/4 and 1/8 resolution (the conv2_* and conv3_* blocks), in both encoder stages and in the decoders that follow the bottlenecks arch.py arch.py.
• nuScenes ground truth is derived. The paper builds segmentation and depth targets from the LiDAR points by filling gaps with neighbouring values, and substitutes the front-left and front-right cameras for left and right §4.2. The official test split is excluded, so the nuScenes numbers come from a held-out part of trainval §4.2.
• The baselines are the authors' retrainings. The BEV baseline has a modified output layer, the LiDAR baseline is trained with its own code's configuration, and the segmentation baseline is a VGG16 SegNet trained per view §5.4. Their numbers are not the original papers' numbers.
• Speed and size are measured one way. FPS uses batch size 1 on a GTX 1080 Ti and averages three simulated sets tab. 3. Speed on other hardware is not stated.
• Set C is the exception. The combined baselines win on set C, and the authors attribute this mainly to PolarNet's LiDAR segmentation score §5.4.

Lineage

• Builds on: GradNorm (Chen et al., ICML) for the loss-weight method, changed to update once per epoch, to use the first layer of each bottleneck as the shared subset, and to use a 1.5 asymmetry §2.3 §3.3. Not in this wiki.
• Builds on: the encoder-decoder with skip connections of U-Net §3.1; the per-task decoder style of Lv et al. §2.1; the intermediate fusion of Nie et al. §2.1; the multi-view segmentation to BEV of Reiher et al. §2.2; the LiDAR height encoding of Yang et al. (PIXOR) and the top-view LiDAR encoding of Imad et al. §2.2.
• Compared against: Chen et al. (IROS) for BEV, PolarNet (CVPR) for LiDAR segmentation, and a VGG16 SegNet version of GradNorm for depth and segmentation §5.4 tab. 2.
• Superseded by: not known.