Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

In plain words

A self-driving car needs to know which of the things around it are moving right now. A parked car and a driving car look alike to a laser scanner, and a semantic label such as "car" does not tell them apart. This paper labels each laser return as either moving or not moving.

The trick is to compare the current scan with the scans just before it. Once the car's own motion is compensated, things that did not move line up, and things that did move show up as differences. Those differences are stored as an extra picture next to the current laser image, and a standard image-segmentation network learns to read them.

The authors use existing networks without changing their design. On their own benchmark the best reported setup uses one extra picture from the past, plus semantic labels, and reaches a score well above the simpler baselines. The numbers come from the authors and from their benchmark only.

TL;DR

• Moving object segmentation (MOS) labels each LiDAR point as moving or static, using only the current and past scans, so it can run online §1 §3.2.
• The input is the range image of the current scan plus normalised range residuals against past scans, transformed with odometry §3.3.
• Any range-projection segmentation network can be used without architecture changes; the paper uses SalsaNext, RangeNet++ and MINet §3.4.
• Author-reported: IoU 52.0 with one residual image and 62.5 with eight residuals plus semantics on the test split tab. 2. Official code ships SalsaNext and RangeNet++ variants with one residual data_preparing.yaml.
• Runs faster than a 10 Hz sensor with SalsaNext: about 51 ms per scan including odometry §4.4.

Problem

Semantic LiDAR segmentation labels what an object is (car, person, road) but not whether it is moving §1 §2. Map-based removal of moving points needs a pre-built, cleaned map and so cannot run online §2. Point-based networks that use sequences are reported as heavy and slow for driving §2. The paper targets online, frame-rate MOS on a rotating LiDAR, with the motion cue taken from the previous scans §1 §3.2.

Method

Overview fig. 2. For the current scan S_0, past scans S_1 … S_N are moved into the current frame with the relative poses from a SLAM or odometry system. Each is projected to a range image. The per-pixel residual against the current range image is computed and concatenated with the current range image as input to a fully convolutional segmentation network, trained with two labels (moving, static) fig. 2.

Step 1, range projection §3.1 eq. 1. Each point (x, y, z) is mapped to spherical and then to image coordinates (u, v) on an H × W grid. Here f is the sum of the upper and lower vertical field of view, and r is the point range eq. 1. The pixel stores x, y, z, range r and remission e, so the image has five channels §3.1.

Step 2, ego-motion compensation §3.2 eq. 2. With estimated relative poses T between consecutive scans, scan S_k is expressed in the frame of scan S_l as S^{k→l}, using the chained transform T^l_k eq. 2.

Step 3, residual image §3.3 eq. 3. The transformed past scan is projected into the current image grid. For each pixel, the residual is the absolute range difference divided by the current range eq. 3. Pixels without a valid measurement get zero. The paper notes that residuals show moving objects as large displacements, but that a moving object can produce two patterns and slow objects give weak residuals, so residuals alone perform poorly §3.3.

Step 4, fusion §3.3 §4.1. The residual images are appended as extra channels to the current range image. The paper writes the per-pixel vector as (x, y, z, r, e, residuals) in §3.3 and as 5 + N input channels in §4.1 §3.3 §4.1. The code uses 5 + N SalsaNext.py.

Step 5, segmentation network §3.4. The architecture is not changed. The paper uses SalsaNext, RangeNet++ and MINet, and retrains each with binary labels §3.4. The loss is the one of the original method, applied to the two classes §3.4. After segmentation, a GPU k-nearest-neighbour search over the point cloud removes artefacts of the range projection §3.4.

# Per-pixel moving / static prediction for scan S_0  [§3.1] [§3.4]
def moving_object_mask(current_scan, past_scans, poses, net):
    # Project the current scan to a range image  [§3.1] [eq. 1]
    range_image = project(current_scan)            # (H, W, 5): x, y, z, r, e
    residuals = []
    for k in range(1, len(past_scans) + 1):
        # Move past scan S_k into the frame of S_0  [§3.2] [eq. 2]
        moved_scan = transform(past_scans[k], poses[k])
        # Project it onto the same image grid  [§3.3] [eq. 1]
        past_range = project(moved_scan)[..., 3]   # (H, W) range channel
        # Normalised absolute range difference; zero where invalid  [§3.3] [eq. 3]
        residuals.append(residual(range_image[..., 3], past_range))
    # Append residuals as channels next to the current image  [§3.3] [§4.1]
    inputs = concat(range_image, stack(residuals))   # (H, W, 5 + N)
    # Unchanged range-image segmentation network, two outputs  [§3.4]
    logits = net(inputs)                             # (H, W, 2)
    return argmax(logits)                            # 1 = moving, 0 = static

Labels and benchmark §3.5. The benchmark is derived from SemanticKITTI. All classes are merged into moving (moving vehicles and humans) and static (every other class) §3.5. Training uses sequences 00–07 and 09–10; sequence 08 is the validation set; sequences 11–21 are the hidden test set §4 §3.5. The metric is the IoU of the moving class eq. 4.

Evaluation of the residual variants §4.1. Three inputs are compared on sequence 08 (validation): the current scan only, the current scan plus the previous frame (2 × 5 channels), and the current scan plus N residual images (5 + N channels) §4.1 tab. 1.

Hyperparameters

The paper says each network keeps "its specific training hyperparameters" and refers to the original papers §4. The values below are therefore from the official code, not stated in the paper, unless marked.

Results

Author-reported. Benchmark: SemanticKITTI-based MOS benchmark §3.5. Metric: IoU of the moving class eq. 4.

Table 1, residual inputs on the validation split (sequence 08) tab. 1:

Table 2, comparison on the test split (sequences 11–21) tab. 2:

Ablations §4.1 fig. 4: with SalsaNext, one residual image gives the largest gain, and more residuals help further with diminishing returns beyond N = 8 §4.1. Adding noise to the poses lowers IoU; beyond 20 noise units the network reportedly ignores the residuals and the score stops falling §4.1.

Odometry §4.3.1 tab. 3: the MOS output is used as a mask before SuMa. Relative errors are averaged over 100–800 m trajectories (rotation in degrees per 100 m / translation in %) tab. 3:

Runtime §4.4: on an Intel i7-8700 (3.2 GHz) with one Nvidia Quadro P6000: about 10 ms for odometry and residual generation; network inference 75 ms (RangeNet++), 42 ms (SalsaNext) and 21 ms (MINet). Including odometry, the paper reports about 51 ms per scan with SalsaNext, against a 10 Hz sensor §4.4.

Qualitative fig. 5 fig. 6: on an intersection with many moving and static objects, the method separates slow-moving cars that other methods miss §4. Mapping fig. 7: removing predicted moving points from aggregated maps removes trails of moving objects §4.3.2.

Paper vs code

Official repository: https://github.com/PRBonn/LiDAR-MOS, linked from the paper §1, footnote. The owner is the authors' lab (PRBonn, University of Bonn), and the LICENSE file is MIT LICENSE. Checked at commit 4f2e96f46f5ac267cf00e404ec94514f91ebfbf1.

• Training residuals and poses. The paper says odometry comes from SemanticKITTI, estimated with SuMa §4. The residual generator reads the KITTI poses.txt file gen_residual_images.py, and the data README describes that file as ground truth README.md. The code shows no SLAM-pose path, and the paper does not say which poses were used for the training residuals. Treat the pose source as unresolved.
• Valid pixels. The paper computes residuals for pixels with a measurement §3.3. The code additionally requires both the current range and the transformed range to lie within 2 m and 50 m gen_residual_images.py.
• Residual channel scale. The five image channels are standardised, but the residual channel is appended after that and is not standardised parser.py.
• Frame zero. The code writes zero residuals for the first N frames gen_residual_images.py. The paper does not say what happens at the start of a sequence.
• Channel count. §3.3 lists N−1 residuals, §4.1 gives 5 + N channels. The code uses 5 + N (residual channels equal n_input_scans) SalsaNext.py.
• Released configs. The repo's MOS configs use n_input_scans: 1 salsanext_mos.yml salsanext_mos.yml. The N = 8 model is listed as a download README.md but no N = 8 config is in the repo.
• MINet. The paper evaluates MINet §3.4 tab. 1. The repo contains SalsaNext and RangeNet++ variants only. No MINet MOS code was found in raw/code/2105.08971 (the paper's MINet numbers cannot be reproduced from the repo).
• kNN. The paper mentions kNN but gives no parameters §3.4. The code enables it with k = 5, search 5, σ = 1.0, cutoff 1.0 salsanext_mos.yml.
• Split and epochs. Matches the paper: 150 epochs, validation on sequence 08 §4 semantic-kitti-mos.yaml.

Pitfalls

• The residual count is ambiguous in the paper. §3.3 writes N−1 residual terms, §4.1 says 5 + N channels. Follow the code's 5 + N and state N explicitly in any implementation.
• Lag and file naming. The code names residual files by lag (residual_images_<k>). The config comment says the suffix must equal num_last_n, so keep the two in step data_preparing.yaml data_preparing.yaml.
• Relative vs absolute residual. The paper's residual is relative to the current range eq. 3. The code has a flag for absolute metres; check which one a checkpoint was trained on data_preparing.yaml.
• Pose error hurts. The ablation shows lower IoU with noisy poses §4.1 fig. 4. The residual depends on odometry accuracy, which the paper does not characterise beyond that test.
• Slow objects and ambiguous residuals. The paper says residuals can be weak for slow objects and show two patterns for one object §3.3. The reported gains are on the authors' benchmark; the paper says the method can separate slow moving objects from static ones §4 but gives no number for them.
• Labels are per scan. Moving status comes from the SemanticKITTI annotations merged into two classes §3.5. Unlabelled points (class 0) are ignored in loss and metric semantic-kitti-mos.yaml README.md.
• Hidden test set. Test results need submission to an external benchmark server; the repo's README says the old server was closed and gives a new link README.md. Test numbers cannot be re-run locally.
• Computing cost. The paper's timings are for one GPU and one CPU §4.4. Odometry is an extra dependency, and the 51 ms figure includes it §4.4.
• Comparison fairness. Several baselines are the authors' own adaptations of methods built for other tasks (for example scene flow with a threshold chosen on validation) §4.2. Their numbers are not a clean comparison.

Lineage

• Builds on: SalsaNext (Cortinhal et al.), RangeNet++ (Milioto et al.) and MINet (Li et al.), used unchanged apart from input and labels §3.4. Residual idea inspired by Wang et al. (frame differences for action recognition) §3.3.
• Odometry: SuMa, with SuMa++ as the semantic baseline §4.3.1.
• Compared against: SalsaNext (moveable classes and retrained), Yoon et al.'s residual heuristic (Residual + RG), FlowNet3D (SceneFlow), SqSequence and KPConv §4.2 tab. 2.
• Superseded by: not recorded in this wiki.