OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving

In plain words

Self-driving cars use LiDAR, a laser sensor that returns a cloud of 3D points. Real LiDAR data is costly to collect and label, so researchers generate synthetic scans with diffusion models, the same family of generators used for pictures. Earlier LiDAR generators make a whole scene at once. The paper argues that they handle the nearby objects that matter most for driving poorly.

OLiDM works in two stages. First it generates single objects, such as a car, from a text description and a 3D box that gives the object's position and heading. Then it places those objects into a scene, so the scene is built around the objects the user asked for. An extra training term groups the scene's features by object category, so that object regions are learned in their own space and do not blur into the background.

The authors report better scores than earlier scene generators on the KITTI-360 dataset, better object quality, and gains when generated objects are added to training data for 3D detectors. These are the authors' own results, not independent checks.

TL;DR

OLiDM generates LiDAR at two levels: foreground objects first, then the scene around them §3.2. Each object comes from a point-cloud diffusion model conditioned on a CLIP text embedding and a Fourier-embedded 3D box §3.2 eq. 1-3. The scene is a range-image diffusion model with a ControlNet-style controller that receives the objects §3.2 eq. 7-8. An extra per-category loss (OSA) is added §3.3 eq. 9. Author-reported: FPD 7.60 on KITTI-360 against 25.1 for UltraLiDAR tab. 1. The linked repository has only the project page, so no training code can be checked.

Problem

Object-level LiDAR is hard to generate. Real objects, unlike CAD models, are partial, sparse and depend on their distance from the ego-vehicle §3.2. The authors also note that foreground points are typically under 10% of a scene's points, so a scene-level model is biased toward background §3.2. Foreground and background also differ in geometry, so a single range-image model mixes them §3.3. Their argument is that object quality and controllability are missing in earlier scene-level generators §1 §4.2. The method assumes object-level annotations exist: a category, a caption and a 3D box for each training object §3.2 app. A.

Method

The model has two diffusion stages and one auxiliary loss. No algorithm is numbered in the paper, so the pseudocode below follows the text and cites it.

1. Object-level diffusion §3.2. A DDPM-style noising process is applied to object point clouds. The noisy points are voxelised and embedded with 3D convolutions §3.2. The condition has two parts. The text T is a template ("An object from class , ") with the captions produced by BLIP-2 §3.2 app. A. The text is embedded by a CLIP text encoder and summarised by its [CLS] vector eq. 1 §3.2. The box B = [x_c, y_c, z_c, w, l, h, r] is Fourier-embedded eq. 2 and the two embeddings are combined by linear layers eq. 3.
2. Object denoiser §3.2 eq. 4-6 app. A. A cross-attention layer lets the voxel tokens query the condition plus the embedded timestep eq. 4. The attention output is added back to the voxel tokens eq. 5. A feed-forward head predicts the object noise, trained with a mean-squared error eq. 6. The appendix says the denoiser follows DiT-3D with 12 layers app. A.
3. Object sampling §3.2. At generation time the user sets the condition. Boxes may be placed by uniform or random sampling on the road, and only the category description needs to be given.
4. Scene-level diffusion §3.2 eq. 7-8. Object points are projected into an object range image I^o. The scene is a range image with height 64 for KITTI-360 and 32 for nuScenes app. A §3.2. A scene controller, a copy of the scene denoiser's architecture app. A, combines zero-convolution features of I^o and the noisy scene I_t eq. 7. The scene denoiser is an 8-layer U-Net that receives these features app. A eq. 8.
5. Object Semantic Alignment (OSA) §3.3 eq. 9 fig. 3. For each object category, a binary mask marks the range-image pixels of that category. The range image is split into one channel per category. The split tensor is noised and the same noise-prediction loss is applied eq. 9. The paper adds this loss to the scene loss §3.3. The weighting is not stated.

Object denoiser training loss:

\mathcal{L}_{\text{object}} = \mathbb{E}_{P^o_0, \mathbf{c}, \epsilon^o, t}\left[ \left\| \epsilon^o - \epsilon^o_{\theta}(P^o_t, t, \mathbf{c}) \right\|^2 \right] eq. 6

Condition from text and box:

[\mathbf{f}^T_{\text{CLS}}, \mathbf{f}^T_0, \cdots, \mathbf{f}^T_L] = F_{\text{text}}(T) eq. 1

\mathbf{f}^B = F_{\text{box}}(B) = F_{\text{fe}}([x_c, y_c, z_c, w, l, h, r]) eq. 2

\mathbf{c} = F_{\text{com}}([\mathbf{f}^B, \mathbf{f}^T]) eq. 3

Cross-attention in the object denoiser, as printed:

\mathbf{q} = \text{linear}(\mathbf{V}_t), \quad \mathbf{k} = \text{linear}(\mathbf{c} + \mathbf{f}^t), \quad \mathbf{v} = \text{linear}(\mathbf{c} + \mathbf{f}^t) eq. 4

\hat{\mathbf{V}}_t = \mathbf{V}_t + \mathbf{v}^{\mathsf{T}} \cdot \text{softmax}(\mathbf{k} \cdot \mathbf{q}^{\mathsf{T}}) eq. 5

Scene controller features and scene and OSA losses:

\mathbf{h}^t_c = \text{zero}(\text{conv}(I^o)) + \text{zero}(\text{conv}(I_t)) eq. 7

\mathcal{L}_{\text{object-scene}} = \mathbb{E}_{I^o, I_0, \epsilon^s, t}\left[ \left\| \epsilon^s - \epsilon^s_{\theta}(I_t, t, \epsilon_{\phi}(I^o)) \right\|^2 \right] eq. 8

\mathcal{L}_{\text{osa}} = \mathbb{E}_{\hat{I}_0, \hat{\epsilon}^s, t}\left[ \left\| \hat{\epsilon}^s - \epsilon^s_{\theta}(\hat{I}_t, t) \right\|^2 \right] eq. 9

Pseudocode of one object-level training step:

# Object-level training step  [§3.2] [eq. 1-3] [eq. 6]
# Input:  object points (B, N, 3); captions; boxes (B, 7)
# Output: scalar loss

def object_training_step(points, captions, boxes):
    # Text: CLIP token features; keep the [CLS] vector  [eq. 1]
    token_feats = clip_text_encoder(captions)        # (B, L+1, d)
    text_feat = token_feats[:, 0]                    # (B, d)

    # Box: Fourier embedding of the 7 box values  [eq. 2] [app. A]
    box_feat = fourier_embed(boxes)                  # (B, d_box)

    # Fuse text and box into one condition vector  [eq. 3]
    cond = linear(concat(box_feat, text_feat))       # (B, d_c)

    # Forward diffusion with a sampled timestep  [§3.2]
    t = sample_timestep()
    noise = sample_gaussian(points.shape)
    noisy = forward_diffuse(points, t, noise)        # (B, N, 3)

    # Voxelise the noisy points and embed them  [§3.2] [app. A]
    voxel_tokens = voxel_embed(noisy, voxel_size=32)

    # Cross-attention with condition plus timestep  [eq. 4] [eq. 5]
    # The printed form is ambiguous; see Pitfalls.
    hidden = voxel_tokens + cross_attend(voxel_tokens, cond, t)

    # Predict the noise with the 12-layer denoiser  [app. A]
    pred = object_denoiser(hidden)                   # (B, N, 3)

    # Mean squared error against the true noise  [eq. 6]
    return mse(pred, noise)

Hyperparameters

Results

Author-reported. Setup: KITTI-360 for generation, with R2DM as the main baseline §4.1; nuScenes for detection augmentation §4.4. No standard deviations or seeds are reported.

Scene-level generation on KITTI-360 tab. 1 §4.2:

Foreground-object quality on KITTI-360 tab. 2 §4.2:

Sparse-to-dense completion on KITTI-360, 16 input beams to 64 output beams (25% of beams) tab. 3 §4.3:

Augmenting 3D detectors on nuScenes, mAP and NDS tab. 4 §4.4:

Computational cost, KITTI-360 app. C (the table has no hardware or generated-sample count):

Headline claims as written: FPD 17.5 better than UltraLiDAR §4.2; semantic IoU 57.47 points better than LiDARGen [abstract]; mAP +2.4 and NDS +1.9 on nuScenes [abstract].

Paper vs code

The abstract and the paper link https://yanty123.github.io/OLiDM. That project page links the repository https://github.com/yanty123/OLiDM. The repository holds only the project-website files and a README. It contains no model, training, sampling or configuration code, and no LICENSE file. So none of the hyperparameters or algorithm steps above can be checked against code. The repository's README says the website is CC BY-SA 4.0, which does not cover any code. Nothing else was searched: the repository was the official link.

Inconsistencies inside the paper, which the code cannot resolve:

• Dataset size: 55k objects are sampled per dataset app. A, but the object denoiser trains on 50,000 samples app. A.
• Scene conditioning: eq. 7 feeds the noisy scene I_t into the controller features, while eq. 8 writes the controller input as only I^o eq. 7-8.
• Abstract vs table: "a 57.47% increase in semantic IoU" over LiDARGen is 79.93 - 22.46 = 57.47 percentage points tab. 3. The relative increase is about 256%.
• Detection gains: the mAP gain of +2.4 is the mean of the baseline gains (+1.9 for PointPillars, +2.9 for CenterPoint) tab. 4. The NDS gain of +1.9 does not match: the table gives +2.8 and +2.7 over baseline. Both figures are points, not percent.
• GT-Aug is labelled "Real" for PointPillars but "Synthetic" for CenterPoint tab. 4.
• The text says OLiDM improves detection by 2.7 over GT-Aug §4.4. That is the CenterPoint mAP difference (61.9 - 59.2); for PointPillars it is 1.6.
• The text uses "9.06" as the KITTI-360 reference box count §4.2. No table reports it.
• The supplement is cited as "Section 1.2" for the metrics §4.1. In the compiled paper the supplement is app. A.

Pitfalls

• Eq. 5 is ambiguous. As printed, \mathbf{v}^{\mathsf{T}} \cdot \text{softmax}(\mathbf{k} \cdot \mathbf{q}^{\mathsf{T}}) has no stated transposes, scaling, number of heads, or output projection. The shapes do not define an attention operation on their own. An implementer has to choose the arrangement; the paper does not say which.
• Eq. 7 and eq. 8 disagree on the controller's input, as noted above. The paper does not say which version was trained.
• OSA channel layout. \hat{I} is said to have c channels, but I has two channels (distance and intensity) §3.3. The product I \odot M_i and the layer that accepts c channels are not defined. The number c of categories is not listed.
• Loss weighting between eq. 8 and eq. 9 is not stated §3.3.
• Sampling is not described beyond "iteratively" §3.2: no step count, sampler, guidance scale or conditioning dropout is given.
• Object denoiser and text encoder. Whether CLIP is frozen is not stated. The text template and captioning prompt are described only in words.
• Table 1 scaling. The caption says MMD is multiplied by 10^4 and JSD by 10. The printed values carry that scaling, and the raw values are not given. Undo the scaling before comparing with other papers.
• Table 2 direction. The #Box column has no arrow. The text says values near 9.06 are better §4.2; this reading is not confirmed by a table.
• Hidden costs. The 3.5 h inference time in app. C is for an unstated number of samples and an unstated GPU. Scene generation runs 300k training steps at batch 16 app. A, but the scene hardware is not stated.
• Augmentation places generated objects uniformly on the ground plane to avoid oversampling near the ego-vehicle app. E. The number of objects added and the placement rules for detection training are not stated.
• Range image width is undefined for OLiDM. Only the R2DM baseline's width is given, as 1024 §4.1.
• Figures are described only. No image is used on this page. The visual evidence is in fig. 1 (motivation), fig. 2 (pipeline), fig. 3 (OSA), fig. 4 (scene comparison), fig. 5 (completion) and fig. 6 (control).

Lineage

• Builds on: DDPM for the diffusion process §2 §3.2; DiT-3D for the object denoiser design app. A; ControlNet's zero convolution for the scene controller §3.2; LiDARGen's range-image representation for the scene §3.2; R2DM for range-image diffusion §2; CLIP for text §3.2; the NeRF Fourier embedder for boxes §3.2; BLIP-2 for captions §3.2.
• Compared against: LiDARGen, R2DM and UltraLiDAR for scene generation tab. 1 §4.2; LiDARVAE and Proj.GAN tab. 1; GT-Aug and LiDAR-Aug for detection augmentation tab. 4.
• Superseded by: not known from this wiki.