Semantic scene completion

Overview

Semantic scene completion asks for a dense 3D map of a scene. Each cell is empty, belongs to a semantic class, or is not known, and the answer covers space the sensor never observed. Both papers here use the same invariant: an unseen cell is not free space. JS3C-Net leaves unseen cells to its completion decoder 2012.03762 §3.3. GEM-Occ keeps a region's old answer when an object hides it, because not seeing a cell is not evidence that it is empty 2607.05543 §3.1, §3.2. A query that finds no occupied or free support returns unknown, and such cells are excluded from the evaluation 2607.05543 app. B.5.

The two papers differ in the sensor and in the time scale. JS3C-Net works on one LiDAR sweep at a time and uses completion as a training aid 2012.03762 §3.1. GEM-Occ takes posed RGB frames and keeps a memory that persists across rooms and revisits 2607.05543 §3.2, §3.3.

Variants

Joint segmentation with an auxiliary completion branch (LiDAR)

A sparse-voxel U-Net labels each point of a single sweep. A completion decoder is trained with it on a fixed scene grid of 256 × 256 × 32 cells at 0.2 m, and a point-voxel interaction module passes information between the two 2012.03762 §3.2–§3.4. The grid's targets are built from 70 merged consecutive sweeps 2012.03762 §4.1. At inference only the segmentation output is used 2012.03762 §3.6. Card: Sparse Single Sweep LiDAR Point Cloud Segmentation.

On the SemanticKITTI test submission, the authors report a semantic scene-completion mIoU of 23.8 over 19 classes, against 17.7 for TS3D³, and a completion IoU of 56.6 against 50.6 2012.03762 tab. 3. On the validation split, removing the joint training drops the full model's semantic completion mIoU from 24.0 to 19.4 2012.03762 tab. 4.

Online semantic occupancy memory from posed RGB

Each pixel with a valid depth becomes one occupied Gaussian with a semantic distribution and a confidence. The free space in front of each visible surface is recorded along the camera rays 2607.05543 §3.1, eq. 1–2. New evidence is matched to existing memory elements, fused with a confidence-weighted average, and the occupancy log-odds are updated by a surface term and a free-space term 2607.05543 §3.2, eq. 4–5. The memory is split into submaps joined in a building-level graph, and periodic maintenance merges or prunes elements 2607.05543 §3.3. Card: GEM-Occ.

On the HIOcc benchmark the authors report these numbers, all with 11 occupied classes plus free space:

• single RGB view: 61.37 IoU and 57.76 mIoU, against 60.69 and 55.08 for GPOcc 2607.05543 tab. 3;
• room-level causal sequences: 56.79 IoU and 46.20 mIoU 2607.05543 tab. 4;
• building-level panoramic sequences: 53.8 IoU and 46.7 mIoU, with 0.81 MB/m² of memory against 1.36 for flat Gaussian memory 2607.05543 tab. 5.

When to use which

Implementation sketch

Language-agnostic pseudocode of the common core. The LiDAR variant adds a second decoder that is used only in training. The memory variant replaces the encoder with posed per-pixel evidence and skips the decoder.

# Shared core: per-cell semantic loss with unknown targets masked out
# Input:  features (N, C_in); target (N,) in 0..C-1, -1 = unknown
# Output: scalar training loss

def train_step(features, target, aux_target=None):
    # Encoder over sparse voxels or per-pixel evidence  [arxiv:2012.03762 §3.2]
    #                                                    [arxiv:2607.05543 §3.1]
    z = encoder(features)
    # One logit per semantic class for each cell or point  [arxiv:2012.03762 §3.2]
    logits = semantic_head(z)                                  # (N, C)
    # Unknown cells carry no occupancy supervision  [arxiv:2607.05543 §3.4]
    loss = masked_cross_entropy(logits, target, ignore=-1)
    # Auxiliary completion over a dense grid, used in training only
    # [arxiv:2012.03762 §3.3] [arxiv:2012.03762 §3.6]
    if aux_target is not None:
        grid_logits = dense_decoder(z)                         # (X, Y, Z, C + 1)
        loss = loss + completion_loss(grid_logits, aux_target)
    return loss

Contradictions

• Different splits, one paper. The SemanticKITTI test value for the completion mIoU is 23.8 2012.03762 tab. 3, and the validation value for the full model is 24.0 2012.03762 tab. 4. They come from different splits, so they are not a contradiction, but neither should be read as the other.
• Paper against released checkpoint. The paper's validation numbers are higher than the values recorded in the file name of the released checkpoint. The details are in the card's Paper vs code section 2012.03762.
• Internal inconsistency in the memory paper. The ablation caption says local metrics come from single frames. Yet removing the confidence weighting, the hierarchy, or the pruning lowers local mIoU from 57.76 to 52.02, 52.41 and 52.48 2607.05543 tab. 6. The paper does not explain this.
• Not comparable. The LiDAR results use outdoor SemanticKITTI with 19 classes 2012.03762 tab. 3. The memory results use indoor HIOcc with 11 occupied classes plus free space 2607.05543 §4. The numbers of the two papers cannot be ranked against each other.

Papers

• JS3C-Net: a LiDAR segmentation network trained jointly with a completion decoder, which is dropped at inference.
• GEM-Occ: an RGB-based semantic occupancy memory with free-space rays, confidence-weighted fusion and a hierarchical map, plus the HIOcc benchmark.