Robo3D: Towards Robust and Reliable 3D Perception against Corruptions

In plain words

Self-driving cars and robots often find objects and label scenery with a laser scanner (LiDAR). Real scanners get dirty, fog and snow scatter the beams, and sensors can interfere with each other, so the data a deployed model sees is often worse than the clean data it was tested on. Robo3D is a test suite that simulates eight of these problems on public driving datasets, at three strengths each, and measures how much accuracy a model keeps.

The authors found that many recent models do well on clean data but lose more accuracy under these problems than a simple baseline does. To make models sturdier they add two things. The first is a voxel grid whose cell size is randomly varied rather than fixed. The second is a fine-tuning step that shows the model a thinned-out copy of each scan and asks the full and thinned versions to agree. Both help robustness in their tests, at a small cost in clean accuracy.

TL;DR

Robo3D is a robustness benchmark for LiDAR 3D object detection and semantic segmentation under eight simulated corruptions at three severities, built on the validation sets of KITTI, SemanticKITTI, nuScenes and Waymo Open. The authors report that many recent models have higher corruption error than a MinkUNet baseline despite better clean scores. They propose a flexible voxelisation and a density-insensitive fine-tuning framework, which lower mean corruption error on SemanticKITTI-C in their ablation. Use the benchmark's protocol and corruption scripts if you need a robustness test for LiDAR models; the proposed training method is not in the released code.

Problem

Models trained and tested on cleaned driving datasets are assumed to hold up when deployed, but that is not checked. The paper argues that natural corruptions from weather, external disturbances and sensor failure are common in deployment and are not covered by standard benchmarks §1. Earlier robustness work covered indoor point clouds or CAD objects, camera-based corruptions, or single tasks with few corruption types §2 §1. The paper targets outdoor LiDAR, where it notes that camera-oriented corruption types are unsuitable §2.

Method

Benchmark. Robo3D defines eight corruption types in three groups: severe weather (fog, wet ground, snow), external disturbances (motion blur, beam missing) and internal sensor failure (crosstalk, incomplete echo, cross-sensor) fig. 1 §3.1. Each corruption maps a LiDAR point with coordinates and intensity to a corrupted point or to nothing (removal). Corruptions follow physical models where one exists and engineering rules otherwise §3.1.

Corruption rules. Notation: a point is \mathbf{p} with coordinates (p^x, p^y, p^z) and intensity p^i; r is its range.

• Fog. Attenuated intensity p^{i}_{\text{hard}} = p^i e^{-2\alpha r} eq. 1. Soft-target response p^{i}_{\text{soft}} = p^i (r^2/\beta_0)\,\beta_{\text{bs}}\,p^i_{\text{tmp}} eq. 2. The corrupted point takes the soft branch when p^{i}_{\text{soft}} > p^{i}_{\text{hard}}, and the hard branch otherwise eq. 3.
• Wet ground. Ground points get a modified intensity \hat p^i and are kept only if it exceeds a noise floor i_n. Non-ground points are unchanged eq. 4 §3.1.
• Snow. Modelled like fog, with snow particles sampled in 2-D and the snowfall rate r_s setting their number §3.1.
• Motion blur. Each coordinate gets a translation offset drawn from a zero-mean Gaussian with standard deviation \sigma_t; intensity is unchanged eq. 5.
• Beam missing. Points on m randomly sampled beams are removed eq. 6.
• Crosstalk. A fraction k_t of points gets a random offset drawn from a zero-mean Gaussian with standard deviation \sigma_c, over all four features eq. 7.
• Incomplete echo. A fraction k_e of points belonging to vehicle, bicycle and motorcycle classes is removed, along with their labels. Ground-truth boxes are kept eq. 8 §3.1.
• Cross-sensor. Points on m beams are dropped and 50% of the remaining points are sub-sampled eq. 9 § Additional Implementation Detail (Cross-Sensor Simulation).

Corruption sets. The corruptions are applied to the validation sets of KITTI, SemanticKITTI, nuScenes and Waymo Open, giving SemanticKITTI-C, KITTI-C, nuScenes-C and WOD-C. These contain 97,704, 90,456, 144,456 and 143,424 point clouds, each with eight corruptions at three severities §3.2.

Metrics. Corruption error per corruption type i sums the error over the three severities and divides by the same sum for a baseline model eq. 10. The baseline is CenterPoint for detection and MinkUNet for segmentation §3.3. Mean corruption error (mCE) averages over the N = 8 types. Resilience rate \text{RR}_i divides the summed accuracy over the three severities by three times the clean accuracy eq. 11; mean RR (mRR) averages over the types. Accuracy is mIoU for segmentation, and AP, NDS or APH(L2) for detection §3.3.

Flexible voxelisation. Sparse convolution maps each point to a voxel index by flooring its coordinates divided by a fixed voxel size per axis eq. 12 §5. The paper replaces the fixed size with a size offset per axis, drawn from a continuous uniform distribution with interval \gamma §5.

Density-insensitive training. A teacher network sees the full scan and a student network sees a copy in which a ratio \beta of points is randomly masked §5 fig. 5. Both are initialised from the same baseline model. Each branch gets the task loss against its own targets eq. 13. The completion term compares the teacher's output on the full scan with the interpolated student output on the masked scan eq. 14. The confirmation term compares the sub-sampled teacher output with the student output eq. 15. Both use squared L2 distance. The total loss is the two task losses plus \alpha_1 times completion plus \alpha_2 times confirmation §5. For detection the two consistency terms are applied to BEV feature maps; for segmentation they are applied to logits § Additional Experimental Result (Density-Insensitive Training).

# Density-insensitive fine-tuning, one sample  [§5] [eq. 13-15]
# Input:  points (N, 4) with x, y, z, intensity; targets
# Output: scalar loss to minimise

def density_insensitive_loss(points, targets, beta, alpha1, alpha2):
    # Make the sparser view by masking a ratio beta of the points  [§5]
    partial_points = random_mask(points, beta)
    # Build the masked targets; the construction is not stated  [§5]
    partial_targets = masked_targets(targets)
    # Full view to the teacher, partial view to the student  [§5]
    teacher_out = teacher(points)
    student_out = student(partial_points)
    # Task loss on each view against its own targets  [eq. 13]
    loss_full = task_loss(targets, teacher_out)
    loss_part = task_loss(partial_targets, student_out)
    # Completion: teacher output vs interpolated student output  [eq. 14]
    loss_part2full = squared_l2(
        teacher_out,
        interpolate(student_out),
    )
    # Confirmation: sub-sampled teacher output vs student output  [eq. 15]
    loss_full2part = squared_l2(
        subsample(teacher_out),
        student_out,
    )
    # Weighted sum of the four terms  [§5]
    total = loss_full + loss_part
    total = total + alpha1 * loss_part2full + alpha2 * loss_full2part
    return total

# Crosstalk corruption  [§3.1] [eq. 7]
# Input:  points (N, 4); ratio k_t; sigma_c (spread)
# Output: corrupted points (N, 4)

def crosstalk(points, ratio, sigma_c):
    # Choose round(ratio * N) distinct points at random  [§3.1]
    count = int(ratio * points.shape[0])
    chosen = sample_without_replacement(points.shape[0], count)
    # Add zero-mean Gaussian offsets to all four features  [eq. 7]
    # sigma_c is not stated in the paper; the code uses 3.0 times unit noise
    # [code: create/semantickitti_c/crosstalk/gen_crosstalk.py#L21-L26]
    noise = normal(0.0, sigma_c, size=(count, 4))
    points[chosen] = points[chosen] + noise
    return points

# Robustness metrics  [§3.3] [eq. 10] [eq. 11]
# acc[i][l]: accuracy for corruption i at severity l in {1, 2, 3}
# base_acc: same layout for the baseline; acc_clean: clean accuracy

def corruption_error(acc, base_acc, i):
    # Summed error relative to the baseline's summed error  [eq. 10]
    model_err = sum(1 - acc[i][l] for l in (1, 2, 3))
    base_err = sum(1 - base_acc[i][l] for l in (1, 2, 3))
    return model_err / base_err

def resilience_rate(acc, acc_clean, i):
    # Mean accuracy over severities, relative to clean  [eq. 11]
    return sum(acc[i][l] for l in (1, 2, 3)) / (3 * acc_clean)

def mean_corruption_error(acc, base_acc):
    # Average over the N = 8 corruption types  [eq. 10]
    return mean(corruption_error(acc, base_acc, i) for i in range(8))

Hyperparameters

Corruption parameters are from the appendix. Where the paper's list does not say which value belongs to which severity or dataset, that is noted in the Notes column and in Pitfalls.

Results

Author-reported throughout. Setup: SemanticKITTI-C for the ablation §5; MinkUNet as the segmentation baseline tab. 7; 34 detection and segmentation models benchmarked on six corruption sets §4.1 fig. 2.

Ablation, SemanticKITTI-C tab. 7

The text reports that the flexible voxel alone reduces mCE by 2.6% on the in-context baseline and by 1.5% on the out-of-context baseline §5. Both additions slightly lower clean accuracy §5.

Density-insensitive training per task tab. 26 (appendix table, numbered in LaTeX order)

Benchmark observations. Selected SemanticKITTI-C corruption errors tab. 1, where a value above 100 means less robust than MinkUNet18:

The authors' main findings (all qualitative, from figures and tables):

• Recent segmentors with strong clean scores are often less robust than the baseline; many have mCE above 100 §4.2 tab. 1.
• On detection, models with higher NDS become less resilient to corruption §4.2 fig. 2.
• Models trained on 64-beam data (KITTI, WOD) are less robust to motion blur and snow than models trained on the sparser nuScenes data §4.2.
• Range-view and BEV projection methods have much higher corruption error than raw-point, sparse-voxel and point-voxel fusion methods across nearly all corruptions §4.2 fig. 3.
• Detectors are more robust to motion blur and crosstalk. Segmentors are more steady under fog, wet ground and snow §4.2 fig. 2.
• Out-of-context augmentations (Mix3D, LaserMix, PolarMix) lower corruption error by large margins, except for wet ground, where the loss of ground points limits scene mixing §4.2 fig. 4.
• For segmentors, a larger voxel favours motion blur and a finer voxel favours fog, snow and crosstalk §4.2 fig. 4.

Paper vs code

The authors' repository (ldkong1205/Robo3D, the first author's account) at commit 481a3b8634b2d291b84736cab1b546ed266efafa contains the corruption generators and the evaluation scripts. It does not contain the density-insensitive training or the flexible voxelisation. The README's TODO list names only corruption scripts, corruption-set downloads and evaluation scripts README.md. A keyword search for the loss and voxel terms found no training code. The method in §5 therefore cannot be checked against code.

The repository is licensed CC BY-NC-SA 4.0 by its README, which refers to a docs/LICENSE.md that is not in the checkout README.md. It is cited here only, and nothing is quoted from it.

Corruption generators compared with the paper:

• Motion blur. The paper gives one Gaussian offset per axis for the whole scan eq. 5. The code adds that offset, and also a per-point jitter with spread 0.1\,\sigma_t in x and y and 0.05\,\sigma_t in z, clipped to \pm 3\sigma_t gen_motion_blur.py. The per-point jitter is not in the paper.
• Beam missing. The paper says "randomly sample" m beams eq. 6. The code samples beams with np.random.choice and the default replacement setting, so the same beam can be drawn twice and fewer than m distinct beams are dropped gen_beam_missing.py.
• Beam counts and crosstalk ratios. The code uses 16 beams for light, 32 for moderate and 48 for heavy gen_cross_sensor.py, and 0.006 / 0.008 / 0.01 for light / moderate / heavy crosstalk gen_crosstalk.py. The paper's lists for these are per dataset, so the mapping differs.
• Crosstalk spread. The paper does not give sigma_c. The code uses unit-normal noise scaled by 3.0 on all four features gen_crosstalk.py.
• Cross-sensor. The code drops fixed beams at a regular spacing, with beam indices from np.arange(1, 64, 4) for 16 beams gen_cross_sensor.py, and keeps every second remaining point gen_cross_sensor.py. This matches the paper's 50% sub-sampling.
• Snow. The precompute list holds rates 0.5, 1.0, 2.0, 2.5, 1.5 precompute.py. The paper lists 0.5, 1.0, 2.5. The light script also sets a terminal velocity of 2.0 snow.sh, which the paper does not state.
• Wet ground. The light script uses water height 0.0002 m, which matches 0.2 mm, and noise floor 0.3 wet_ground.sh. The paper gives no noise floor.
• Fog. The code draws alpha from the same six values as the paper fog_simulation.py. The light script uses beta 0.008 fog.sh.
• Incomplete echo. The light default ratio is 0.75 and the comments give 0.75 / 0.85 / 0.95 for light / moderate / heavy gen_incomplete_echo.py gen_incomplete_echo.py.

Pitfalls

• Missing fog coordinates. The fog equation writes the corrupted point as (\hat p^x, \hat p^y, \hat p^z, p^i_{\text{soft}}), but the hat coordinates are never defined eq. 3 § Additional Implementation Detail (Fog Simulation). An implementer has to choose them; the paper does not say how.
• Beta symbol. The main text writes the back-scattering coefficient as \beta_{\text{bs}} and the appendix as \beta, with \beta_0 for differential reflectivity in both. The severity values 0.008, 0.05 and 0.2 are the back-scattering values § Additional Implementation Detail (Fog Simulation).
• Per-dataset or per-severity lists. For beam missing, cross-sensor, crosstalk and incomplete echo, the paper's three values can be read as datasets (with "respectively") or as light / moderate / heavy. The released code uses severities for SemanticKITTI-C. Use the code's mapping and check other datasets against the paper's wording.
• Unstated values. The crosstalk spread sigma_c, the wet-ground noise floor i_n, the snow terminal velocity and the units of the snow rate are not stated in the paper. The code supplies some of them (see Paper vs code), and those are not confirmed by the paper.
• Ambiguous "interval". The flexible voxel offset is drawn "with an interval gamma" §5. The paper does not say whether that means a range of plus or minus gamma or a width of gamma. The voxel size is written as l \pm \text{dv}, which suggests plus or minus, but the sampling per point, per scan or per epoch is not stated.
• Teacher and student. The notation \mathcal{G}_\theta^{\text{tea}} and \mathcal{G}_\theta^{\text{stu}} suggests shared weights, but the text only says both are initialised from the same baseline. Whether gradients flow through the teacher, and whether it is an exponential moving average, is not stated §5.
• Undefined operators. The masked targets \tilde y, the \texttt{interp} operator and the \texttt{subsample} operator are not defined eq. 13 eq. 14 eq. 15. For detection the consistency terms use BEV maps and for segmentation they use logits § Additional Experimental Result (Density-Insensitive Training).
• Loss weights. The text gives weights 50 and 100 for the two consistency terms in that order. The mapping is inferred from the order, not stated explicitly §5.
• Mask ratio. Two values are used (0.4 for ICA, 0.6 for OCA). The text describes a trade-off: a higher ratio removes more information, but 0.3 to 0.6 gives the lowest mCE §5 fig. 6.
• Clean accuracy cost. Both additions lower clean accuracy slightly, and the authors suggest this reflects less overfitting to the training data §5.
• Metric comparability. The baseline for mCE differs by task (CenterPoint or MinkUNet), so do not compare mCE between a detector and a segmentor §3.3.
• Retrained numbers. Where a model used extra tricks such as test-time augmentation or ensembles, the authors retrained it under standard settings. Reported numbers may differ from the original papers §4.1.
• Model count. The text says 34 models; the SemanticKITTI-C table caption says 22 segmentors §4.1 tab. 1. Use per-table rows rather than the total.
• Corruption cost. The four sets hold 90,000 to 144,000 point clouds each. Fog and snow also need semantic labels or ground-plane estimates (labels or RANSAC) § Additional Implementation Detail (Wet Ground Simulation).

Lineage

• Builds on: MinkUNet (the segmentation baseline, cited as [2019Minkowski]) and CenterPoint (the detection baseline) §3.3.
• Uses masking-based representation ideas: masked autoencoders is among the works §5 names as inspiration for the random masking.
• Compared against the out-of-context augmentations Mix3D, LaserMix and PolarMix §4.1.
• Superseded by: not known from content already in this wiki.