--- kind: concept slug: semantic-scene-completion title: Semantic scene completion summary: >- Predict a dense 3D grid in which every cell is empty, a semantic class, or unknown, including cells the sensor never saw. Covers a LiDAR variant that trains completion as an auxiliary task and drops it at inference, and an online camera variant that fuses posed RGB into a persistent occupancy memory. Gives the trade-offs each reports, and warns that the two papers' numbers are not comparable. tags: [scene-completion, occupancy-prediction, lidar, point-cloud, semantic-segmentation, autonomous-driving, indoor-scenes, gaussian-splatting, online-mapping, embodied-ai] papers: ["2012.03762", "2607.05543"] related: [semantic-segmentation, gaussian-splatting, simultaneous-localization-and-mapping, lidar-inertial-odometry, bev-representation, 3d-scene-graph] status: draft updated_at: 2026-11-01 --- ## Overview Semantic scene completion asks for a dense 3D map of a scene. Each cell is empty, belongs to a semantic class, or is not known, and the answer covers space the sensor never observed. Both papers here use the same invariant: an unseen cell is not free space. JS3C-Net leaves unseen cells to its completion decoder [arxiv:2012.03762 §3.3]. GEM-Occ keeps a region's old answer when an object hides it, because not seeing a cell is not evidence that it is empty [arxiv:2607.05543 §3.1, §3.2]. A query that finds no occupied or free support returns unknown, and such cells are excluded from the evaluation [arxiv:2607.05543 app. B.5]. The two papers differ in the sensor and in the time scale. JS3C-Net works on one LiDAR sweep at a time and uses completion as a training aid [arxiv:2012.03762 §3.1]. GEM-Occ takes posed RGB frames and keeps a memory that persists across rooms and revisits [arxiv:2607.05543 §3.2, §3.3]. ## Variants ### Joint segmentation with an auxiliary completion branch (LiDAR) A sparse-voxel U-Net labels each point of a single sweep. A completion decoder is trained with it on a fixed scene grid of 256 × 256 × 32 cells at 0.2 m, and a point-voxel interaction module passes information between the two [arxiv:2012.03762 §3.2–§3.4]. The grid's targets are built from 70 merged consecutive sweeps [arxiv:2012.03762 §4.1]. At inference only the segmentation output is used [arxiv:2012.03762 §3.6]. Card: [Sparse Single Sweep LiDAR Point Cloud Segmentation](/papers/2012.03762). On the SemanticKITTI test submission, the authors report a semantic scene-completion mIoU of 23.8 over 19 classes, against 17.7 for TS3D³, and a completion IoU of 56.6 against 50.6 [arxiv:2012.03762 tab. 3]. On the validation split, removing the joint training drops the full model's semantic completion mIoU from 24.0 to 19.4 [arxiv:2012.03762 tab. 4]. ### Online semantic occupancy memory from posed RGB Each pixel with a valid depth becomes one occupied Gaussian with a semantic distribution and a confidence. The free space in front of each visible surface is recorded along the camera rays [arxiv:2607.05543 §3.1, eq. 1–2]. New evidence is matched to existing memory elements, fused with a confidence-weighted average, and the occupancy log-odds are updated by a surface term and a free-space term [arxiv:2607.05543 §3.2, eq. 4–5]. The memory is split into submaps joined in a building-level graph, and periodic maintenance merges or prunes elements [arxiv:2607.05543 §3.3]. Card: [GEM-Occ](/papers/2607.05543). On the HIOcc benchmark the authors report these numbers, all with 11 occupied classes plus free space: - single RGB view: 61.37 IoU and 57.76 mIoU, against 60.69 and 55.08 for GPOcc [arxiv:2607.05543 tab. 3]; - room-level causal sequences: 56.79 IoU and 46.20 mIoU [arxiv:2607.05543 tab. 4]; - building-level panoramic sequences: 53.8 IoU and 46.7 mIoU, with 0.81 MB/m² of memory against 1.36 for flat Gaussian memory [arxiv:2607.05543 tab. 5]. ## When to use which | Situation | Use | Why | |---|---|---| | Single LiDAR sweep at run time, with sequences available for training | Joint segmentation with auxiliary completion | Only the segmentation branch runs at inference [arxiv:2012.03762 §3.6]; training needs merged-sweep targets [arxiv:2012.03762 §4.1] | | Posed RGB stream, robot leaves and returns to rooms | Online occupancy memory | The memory keeps what was seen earlier and corrects it on revisit [arxiv:2607.05543 §3.3, app. B.4] | | Memory size matters at building scale | Confidence-weighted Gaussian memory with hierarchy | 0.81 MB/m² against 1.36 for flat Gaussian memory [arxiv:2607.05543 tab. 5] | | Need a fast online pipeline | Plan for a few FPS | The authors report 5.2 FPS for the full online pipeline, hardware not stated [arxiv:2607.05543 app. A.2] | | Scene is not static, or poses are unknown | Check before choosing; neither paper tests this | GEM-Occ assumes static indoor scenes with supplied poses and does not test pose drift [arxiv:2607.05543 app. A.3]. The LiDAR paper kept moving objects in training, and removing them changed validation segmentation by 0.2 points [arxiv:2012.03762 app. B.2] | | Single RGB view only, no memory | GEM-Occ local prediction | The single-view numbers are the best of the compared methods, by 0.68 IoU over GPOcc in a single run [arxiv:2607.05543 tab. 3] | ## Implementation sketch Language-agnostic pseudocode of the common core. The LiDAR variant adds a second decoder that is used only in training. The memory variant replaces the encoder with posed per-pixel evidence and skips the decoder. ```pseudocode # Shared core: per-cell semantic loss with unknown targets masked out # Input: features (N, C_in); target (N,) in 0..C-1, -1 = unknown # Output: scalar training loss def train_step(features, target, aux_target=None): # Encoder over sparse voxels or per-pixel evidence [arxiv:2012.03762 §3.2] # [arxiv:2607.05543 §3.1] z = encoder(features) # One logit per semantic class for each cell or point [arxiv:2012.03762 §3.2] logits = semantic_head(z) # (N, C) # Unknown cells carry no occupancy supervision [arxiv:2607.05543 §3.4] loss = masked_cross_entropy(logits, target, ignore=-1) # Auxiliary completion over a dense grid, used in training only # [arxiv:2012.03762 §3.3] [arxiv:2012.03762 §3.6] if aux_target is not None: grid_logits = dense_decoder(z) # (X, Y, Z, C + 1) loss = loss + completion_loss(grid_logits, aux_target) return loss ``` ## Contradictions - **Different splits, one paper.** The SemanticKITTI test value for the completion mIoU is 23.8 [arxiv:2012.03762 tab. 3], and the validation value for the full model is 24.0 [arxiv:2012.03762 tab. 4]. They come from different splits, so they are not a contradiction, but neither should be read as the other. - **Paper against released checkpoint.** The paper's validation numbers are higher than the values recorded in the file name of the released checkpoint. The details are in the card's Paper vs code section [arxiv:2012.03762]. - **Internal inconsistency in the memory paper.** The ablation caption says local metrics come from single frames. Yet removing the confidence weighting, the hierarchy, or the pruning lowers local mIoU from 57.76 to 52.02, 52.41 and 52.48 [arxiv:2607.05543 tab. 6]. The paper does not explain this. - **Not comparable.** The LiDAR results use outdoor SemanticKITTI with 19 classes [arxiv:2012.03762 tab. 3]. The memory results use indoor HIOcc with 11 occupied classes plus free space [arxiv:2607.05543 §4]. The numbers of the two papers cannot be ranked against each other. ## Papers - [JS3C-Net](/papers/2012.03762): a LiDAR segmentation network trained jointly with a completion decoder, which is dropped at inference. - [GEM-Occ](/papers/2607.05543): an RGB-based semantic occupancy memory with free-space rays, confidence-weighted fusion and a hierarchical map, plus the HIOcc benchmark.