Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving

In plain words

Labelling every point in a LiDAR scan by hand is slow and expensive, so this paper asks how to train a driving scene segmenter from a few labelled scans plus many unlabelled ones. A LiDAR scan comes from several laser beams, each at its own height angle, and the things around a car sit on predictable bands of those angles. The method cuts two scans into bands by beam angle and swaps the bands between them. The model is then trained to give consistent labels on the original and the swapped scans, and a slower-moving copy of the model supplies labels for the unlabelled scans.

The second part, LaserMix++, uses the camera as well. Pixels from a photo that lines up with the LiDAR points add texture information. Two further signals pull the LiDAR features toward what image models and text descriptions of the classes say about the same points. The authors report that this needs far fewer labels than fully supervised training.

For an engineer, the useful parts are the band-swapping recipe and the list of loss terms. The released code implements only the first part, so the camera parts have to be built from the equations in the paper.

TL;DR

• Semi-supervised LiDAR semantic segmentation. LaserMix swaps laser-beam bands between two scans; LaserMix++ adds three camera-related parts §1 §4.
• Author-reported gain of LaserMix++ over LaserMix: 2% to 3% on average across settings §5.3. With Cylinder3D at 1% labels the gain is 3.1 to 5.6 mIoU across the three datasets tab. 2.
• Authors claim comparable accuracy to fully supervised methods with 2× to 5× fewer annotations §5.3.
• The linked repository is the LaserMix code (Apache-2.0). The camera parts of LaserMix++ are not in it, and its LaserMix++ result rows are empty README.md README.md.
• Several values in the released configs differ from the paper's text, including the number of bands, loss weights, pseudo-label threshold and optimizer in one config. See Paper vs code.

Problem

Dense LiDAR labels are costly, which limits fully supervised training in driving §1. Semi-supervised methods were built for images. The authors argue they underperform on LiDAR because LiDAR points carry spatial structure that image methods do not use §1 §2. They also argue that the other sensors on a driving car, such as cameras, are underused for LiDAR semi-supervision §1.

The paper assumes labelled and unlabelled scans come from the same sensor setup, and that camera and LiDAR are calibrated and synchronised §4.1. It evaluates on nuScenes, SemanticKITTI and ScribbleKITTI at 1%, 10%, 20% and 50% labelled data §5.1.

Method

1. Spatial prior and objective §3.1

The paper treats the goal as making predictions confident and consistent inside predefined spatial areas, whatever lies outside those areas. It writes the training objective as the supervised likelihood minus a weighted entropy term over the areas eq. 5:

C(\theta) = L(\theta) - \lambda\,\hat{H}(Y_{crop} \mid X_{crop}, A) eq. 5

The empirical entropy is estimated by padding the data outside each area and marginalising over that outside data eq. 3 eq. 4. The paper states that setting the area to the whole scan reduces the objective to classic entropy minimisation §3.1. The value of λ is not given.

2. Laser-beam partition §3.2

Each point gets an inclination angle from its Cartesian coordinates eq. 6:

\phi_i = \arctan\left(\frac{p^z_i}{\sqrt{(p^x_i)^2 + (p^y_i)^2}}\right) eq. 6

For m areas, m+1 inclination edges are spaced evenly between the minimum and maximum inclination in the dataset. Area a_i holds points with inclination in [φ_{i−1}, φ_i) §3.2. The paper states that the angle range differs between LiDAR sensors and gives no numeric range §3.2. Figure 2 shows the partition fig. 2.

3. LaserMix §3.2

Two scans x1 and x2 are mixed by taking the odd-indexed areas from one scan and the even-indexed areas from the other, so that each area's neighbours come from the other scan §3.2 eq. 7:

\tilde{x}_1 = x_1^{a_1} \cup x_2^{a_2} \cup x_1^{a_3} \cup \cdots, \qquad \tilde{x}_2 = x_2^{a_1} \cup x_1^{a_2} \cup x_2^{a_3} \cup \cdots eq. 7

Labels are mixed in the same way §3.2. The paper says LaserMix works on raw points, so it applies to range view, bird's-eye view, voxels and fusion backbones alike §3.2.

4. Consistency framework §3.3

• A student network and a teacher network share the architecture. The teacher's weights are an exponential moving average of the student's, and the teacher is used at inference §3.3.
• Each batch holds equal amounts of labelled and unlabelled data §3.3.
• The teacher predicts pseudo-labels for the unlabelled scans. Only predictions above a confidence threshold T are kept §3.3 §5.4.
• Each unlabelled scan is mixed with a randomly chosen labelled scan, and the student predicts the mixed scan §3.3.
• Losses. The supervised term is a point-wise cross-entropy against ground truth eq. 8. The mean-teacher term is a squared ℓ2 difference between student and teacher predictions eq. 9. The total is L = L_sup + λ_mix L_mix + λ_mt L_mt §3.3.

\mathcal{L}_{ce} = \frac{1}{|x|}\sum_{i=1}^{|x|}\text{CrossEntropy}\big(y^{(i)}, \mathcal{G}^{(i)}_\theta(x)\big) eq. 8

\mathcal{L}_{mt} = \frac{1}{|x|}\sum_{i=1}^{|x|}\big\|\mathcal{G}^{s,(i)}_\theta(x) - \mathcal{G}^{t,(i)}_\theta(x)\big\|_2^2 eq. 9

Figure 3 shows the dual-branch setup fig. 3.

5. LaserMix++: camera correspondences §4.1

A LiDAR point projects to a pixel using the camera intrinsics and the LiDAR-to-camera transform eq. 10:

[p^u, p^v, 1]^{T} = \frac{1}{p^z} \times \Gamma_{K} \times \Gamma_{c \leftarrow l} \times [p^x, p^y, p^z]^{T} eq. 10

A mask M is 1 where the point lands on a valid pixel and 0 otherwise. Masked entries are zero-padded §4.1.

6. LaserMix++: multi-modal LaserMix §4.2

The same band swap is applied to (point, pixel) pairs, so each point keeps its image feature. The number of areas is drawn from 2 to 6 §4.2 §5.2 eq. 11. The formal definition in eq. 11 has the same form as eq. 7, applied to pairs.

7. LaserMix++: camera-to-LiDAR feature distillation §4.3

• The student backbone's point features pass through a projection layer to match the image feature size eq. 12.
• A pre-trained image segmentation backbone gives the image features eq. 12.
• Features are aligned through the correspondence mask, and the paper uses residual connections so that point geometry is kept §4.3.
• The loss is averaged over valid points eq. 13:

\mathcal{L}_{c2l} = \frac{1}{\sum_{i}\mathcal{M}^{(i)}} \sum_{i} \mathcal{M}^{(i)} \left(1 - \langle F_{p}^{(i)}, \tilde{F}_{img}^{(i)} \rangle \right) eq. 13

8. LaserMix++: language-driven knowledge guidance §4.4

• A CLIP text encoder embeds the class names. An open-vocabulary segmentation head combines image features with those text embeddings eq. 14.
• A cosine-similarity loss aligns text and image features §4.4. The text gives no equation or weight for this step.
• The masked loss between the student's LiDAR outputs and the text-aligned image features is eq. 15:

\mathcal{L}_{lkg} = \frac{1}{\sum_{i}\mathcal{M}^{(i)}} \sum_{i} \mathcal{M}^{(i)} \left(1 - \langle F'^{(i)}_{p}, \tilde{F}'^{(i)}_{img} \rangle \right) eq. 15

9. Full LaserMix++ objective §4.5

\mathcal{L} = \mathcal{L}_{sup} + \lambda_{mix}\mathcal{L}_{mix} + \lambda_{mt}\mathcal{L}_{mt} + \lambda_{c2l}\mathcal{L}_{c2l} + \lambda_{lkg}\mathcal{L}_{lkg} eq. 16

Algorithm sketch: LaserMix band swap

The paper gives no algorithm box. This sketch follows §3.2 and eq. 7. The released code computes the same bands with fixed angles from the config lasermix.py.

# LaserMix: swap laser-beam bands between two scans  [§3.2] [eq. 7]
# Input:  scan_a, scan_b: points (N, 4) with per-point labels (N,)
# Output: mixed_1, mixed_2: points and labels, built from the same bands

def laser_mix(scan_a, scan_b, num_bands):
    # Inclination of each point from its x, y, z  [eq. 6]
    pitch_a = inclination(scan_a.points)                # (N_a,) radians
    pitch_b = inclination(scan_b.points)                # (N_b,) radians
    # m+1 edges, evenly spaced over the dataset's min..max  [§3.2]
    edges = linspace(pitch_min, pitch_max, num_bands + 1)
    mixed_1 = []
    mixed_2 = []
    for band in range(num_bands):
        # Band j holds points with pitch in [edges[j], edges[j+1])  [§3.2]
        low = edges[band]
        high = edges[band + 1]
        # Even bands (a1, a3, ...) come from scan_a in mixed_1  [eq. 7]
        if band % 2 == 0:
            first, second = scan_a, scan_b
        else:
            first, second = scan_b, scan_a
        # The other scan's band goes into mixed_2 at the same position  [eq. 7]
        mixed_1.append(crop(first, low, high))
        mixed_2.append(crop(second, low, high))
    return concat(mixed_1), concat(mixed_2)

Algorithm sketch: LaserMix++ training step

This sketch combines eq. 8 to eq. 15 into one step. The released code has no camera branch, so this is written from the paper alone.

# LaserMix++ training step, written from the paper  [§4.5] [eq. 16]
# labeled: points, image pixels, labels
# unlabeled: points, image pixels

def training_step(labeled, unlabeled, student, teacher, image_model):
    # Supervised cross-entropy on ground-truth labels  [§3.3] [eq. 8]
    logits_sup = student(labeled.points)
    loss_sup = cross_entropy(logits_sup, labeled.labels)
    # Teacher pseudo-labels, kept only above threshold T  [§3.3] [§5.4]
    pseudo = teacher_pseudo_labels(unlabeled.points, threshold=T)
    # Multi-modal LaserMix on (point, pixel) pairs  [§4.2] [eq. 11]
    mixed = multimodal_laser_mix(labeled, unlabeled, pseudo)
    loss_mix = cross_entropy(student(mixed.points), mixed.labels)
    # Mean-teacher consistency on the softmax outputs  [§3.3] [eq. 9]
    loss_mt = squared_error(softmax(student(unlabeled.points)),
                            softmax(teacher(unlabeled.points)))
    # Distill camera features onto points with a valid pixel  [§4.3] [eq. 13]
    feat_s = project_head(student.backbone(labeled.points))
    loss_c2l = masked_cosine_distance(feat_s, image_model(labeled.pixels),
                                      mask)
    # Language-guided term against text-aligned image features  [§4.4] [eq. 15]
    loss_lkg = masked_cosine_distance(student_features, text_aligned_pixels,
                                      mask)
    # Weighted sum; weights are under Hyperparameters  [§4.5] [eq. 16]
    return (loss_sup + lambda_mix * loss_mix + lambda_mt * loss_mt
            + lambda_c2l * loss_c2l + lambda_lkg * loss_lkg)

Hyperparameters

Values marked "paper" come from the main text, and values marked "code" come from the released LaserMix configs. Where they differ, both are given.

Results

Author-reported. Setup: LiDAR semantic segmentation, mIoU, Cylinder3D backbone, labelled fractions as in §5.1. The baselines are supervised-only training and LaserMix tab. 2.

Notes on the table:

• On ScribbleKITTI at 10%, 20% and 50%, the LiM3D method reports 60.3, 60.5 and 60.9 mIoU, above LaserMix++ at 56.7, 57.6 and 59.8. The paper attributes LiM3D's numbers to test-time ensembles that make its protocol different §5.3 tab. 2.
• Ablation with the FIDNet range-view backbone, SemanticKITTI at 10%: the mean-teacher baseline reaches 53.1, LaserMix 60.1, and the full LaserMix++ 61.9. On nuScenes at 10% the figures are 60.4, 68.2 and 69.8 tab. 5.
• Robustness on Robo3D, lower mCE is better and higher mRR is better. On SemanticKITTI-C with MinkUNet: no mixing gives mCE 100.0 and mRR 81.9; LaserMix gives 83.3 and 87.0; LaserMix++ gives 80.9 and 88.1 tab. 3. On WOD-C with CenterPoint: LaserMix 100.5 mCE, LaserMix++ 98.2 tab. 3.
• nuScenes with pre-trained and fine-tuned representations, 1% labels: supervised-only 30.3, LaserMix 48.4, LaserMix++ 49.9 mIoU tab. 4. The paper does not describe the pre-training recipe in the text read.
• Cityscapes, with the mean-teacher backbone at 1/16 labels: 66.1 mIoU baseline, 68.7 with the proposed regularisation tab. 6.
• Full supervision with LaserMix used as augmentation: SemanticKITTI with Cylinder3D rises from 63.7 to 64.7 mIoU tab. 9. The paper reports gains of 0.4 to 2.4 mIoU across backbones §5.4.

Paper vs code

The linked repository is ldkong1205/LaserMix at commit a4277292d9d25fba399bfb1cec5d2757331c4975. It is an MMDetection3D fork under Apache-2.0 LICENSE. I cloned it and searched inside it. It holds the LaserMix code only.

• Scope. The camera parts of LaserMix++ (multi-modal mixing, camera-to-LiDAR distillation, language guidance) are absent. The README leaves the LaserMix++ rows of its results tables empty README.md README.md. Eqs. 10 to 16 can only be implemented from the paper.
• Loss terms. The paper's objective is entropy-based eq. 5. The released losses are supervised cross-entropy and Lovász, a mean-teacher MSE on softmax outputs and a mixing loss. There is no entropy term lasermix.py.
• Supervised loss. Eq. 8 is cross-entropy only. The Cylinder3D head adds Lovász with weight 3.0 on SemanticKITTI lasermix_cy3d_semi_semantickitti_10.py and weight 2 on nuScenes lasermix_cy3d_semi_nuscenes_10.py. The mixing loss goes through the same head semi_base.py, so Lovász also applies to mixed scans.
• Number of areas. The paper samples 2 to 6 §5.2. The SemanticKITTI config samples 4 to 8 lasermix_cy3d_semi_semantickitti_10.py. The nuScenes config and the MinkUNet config sample 3 to 6 lasermix_cy3d_semi_nuscenes_10.py lasermix_minku18_semi_semantickitti_10.py.
• Band edges. The paper derives edges from the dataset's minimum and maximum inclination §3.2. The code uses fixed angles from the config, −25° to 3° on SemanticKITTI and −30° to 10° on nuScenes lasermix.py. The code's interval test is (pitch > start) & (pitch <= end) lasermix.py. The paper writes [φ_{i−1}, φ_i).
• Range-view path. The FIDNet range-view results in the paper tab. 2 tab. 5 cannot be reproduced from the repository. The range path splits the beam axis into equal index ranges, not inclination bands lasermix.py. It runs only when the batch has an imgs key lasermix.py, and no shipped config provides one. No FIDNet config is in the repository.
• λ_mix. The paper sets λ_mix to 2.0 §5.2. Every released LaserMix config sets the mixing weight to 1 lasermix_cy3d_semi_semantickitti_10.py.
• λ_mt. The paper sets 250 §5.2. The SemanticKITTI config matches at 250 lasermix_cy3d_semi_semantickitti_10.py. The nuScenes config uses 500 lasermix_cy3d_semi_nuscenes_10.py and the MinkUNet config 200 lasermix_minku18_semi_semantickitti_10.py.
• Threshold. The paper uses about 0.9 §5.4. The SemanticKITTI config uses 0.9 lasermix_cy3d_semi_semantickitti_10.py and the nuScenes config 0.95 lasermix_cy3d_semi_nuscenes_10.py.
• Optimizer. The paper says all baselines use AdamW at 0.008 §5.2. The SemanticKITTI and nuScenes configs do lasermix_cy3d_semi_semantickitti_10.py. The MinkUNet config uses SGD at 0.24 lasermix_minku18_semi_semantickitti_10.py.
• Learning-rate schedule. The config sets the optimizer lr to 0.008 and OneCycleLR eta_max to 0.001 lasermix_cy3d_semi_semantickitti_10.py. Which value sets the peak depends on how mmengine's OneCycleLR treats eta_max. I did not check that in the mmengine source, so do not assume the peak is 0.008.
• Batch size. The paper says 2 per GPU on eight A100s §5.2. The config uses 4 per source lasermix_cy3d_semi_semantickitti_10.py. Its comment gives 45000 iterations for 8×2 or 4×4 lasermix_cy3d_semi_semantickitti_10.py.
• Momentum. The paper speaks of EMA decay rates from 0.9 to 0.99 §5.4. The config passes momentum 0.01 lasermix_cy3d_semi_semantickitti_10.py. The update in _update_ema_variables is teacher ← (1 − m)·teacher + m·student semi_base.py, so the teacher keeps 0.99 of its weights each step. The call site is in the external mmdet hook, which is not in the clone.
• Mean-teacher loss. The code applies softmax to both networks' logits before the MSE lasermix.py. Eq. 9 is written on the network outputs without a softmax.
• Pairing. The paper mixes each unlabelled scan with a random labelled scan §3.3. The code pairs labelled and unlabelled scans at the same batch index lasermix.py.
• Split files. The configs name split files such as semantickitti_infos_train.10.pkl. They are not in the clone. The only split generator I found is for nuScenes generate_nus_semi_pkl.py.
• Matches. The voxel grid [240, 180, 20] matches the paper §5.2 lasermix_cy3d_semi_semantickitti_10.py. Inference uses the teacher by default §3.3 semi_base.py.

Pitfalls

• LaserMix++ has no code. Eqs. 10 to 16 are the only specification of the camera parts. The text does not name the image segmentation model, says nothing on whether the image branch is frozen, and gives no equation or weight for the text-alignment step §4.3 §4.4.
• Inner product versus cosine. Eqs. 13 and 15 are written with an inner product, but the text calls them cosine distances §4.3 §4.4. The paper does not say whether the features are normalised first.
• Points below the threshold. Section 3.3 does not say what happens to pseudo-labels below T. The code labels them with an ignore index, 19 on SemanticKITTI and 16 on nuScenes semi_base.py lasermix_cy3d_semi_nuscenes_10.py.
• Objective versus implementation. The paper's motivation is entropy minimisation §3.1. The implemented objective has no entropy term (see Paper vs code). The value of λ in eq. 5 is not given.
• Compute. The paper says memory doubles per batch and training speed is unaffected §3.2. The released step runs the teacher twice (labelled and unlabelled scans) and the student three times (labelled, unlabelled, mixed) lasermix.py. I did not measure memory or speed.
• Threshold dependence. The paper says the best threshold "varies across datasets" §5.4 fig. 6c. The configs confirm this (0.9 and 0.95).
• Mixing unlabelled scans only. The paper reports 68.2 falling to 66.9 mIoU §5.4. The 68.2 matches the LaserMix nuScenes 10% FIDNet value in tab. 2, but the text does not name the setting.
• Azimuth partitions. The text says azimuth splits give no gain §5.4. In tab. 7, azimuth-only splits score 60.9 to 61.5 against a baseline of 60.4, a small gain. The table's best cell is (2α, 5φ) at 66.7, just above the inclination-only (1α, 4φ) at 66.5. The table caption does not name a dataset.
• Pre-training comparison. The text says LaserMix++ "combined with Seal" reaches 77.9 from 74.7 §5.3. The table's 74.7 is the supervised-only row, while Seal alone reaches 75.6 at full supervision tab. 4. The LaserMix++ row does not name Seal.
• Single runs. The text reports no seeds or variance.
• Missing appendix. The text refers readers to an appendix §5.2. The source has none, so iterations, augmentation and weight decay come only from the code.
• Cityscapes. The table says "w/ Ours" without naming LaserMix or LaserMix++, and the text does not say how masks or mixing apply to images §5.4 tab. 6.
• Comparison with LiM3D. LiM3D's numbers use test-time ensembles, so the comparison is not like for like §5.3.

Lineage

• Builds on: LaserMix (Kong et al., CVPR 2023; no card in this wiki and no arXiv id in the paper's bibliography). Mean Teacher (NeurIPS 2017) for the teacher-student framework §3.3.
• Uses or compares against: MixUp, CutMix and CutOut for mixing baselines §5.4; CPS and CBST for semi-supervised baselines §5.2; CLIP for text embeddings §4.4; Seal for pre-training comparison tab. 4; Lim3D and GPC for LiDAR baselines §5.3; Robo3D for robustness §5.1.
• Superseded by: none in the wiki.