---
kind: concept
slug: semantic-occupancy-mapping
title: Semantic occupancy mapping
summary: >-
  Keeps a 3D memory of which space is occupied, free or unknown, with a semantic class for occupied points, updated as posed views arrive. One card so far: GEM-Occ turns RGB predictions into semantic Gaussians plus free-space rays, fused into submaps. Author-reported on HIOcc: 61.37 local IoU, 0.81 MB/m² at building scale. Code not yet released.
tags: [occupancy-mapping, semantic-mapping, online-mapping, embodied-ai, indoor-scenes]
papers: ["2607.05543"]
related: [gaussian-splatting, simultaneous-localization-and-mapping, surfel-mapping, semantic-segmentation, bev-representation, 3d-scene-graph]
status: draft
updated_at: 2026-11-01
---

## Overview

Semantic occupancy mapping keeps a 3D memory of a scene. Each query point gets
one of three answers: occupied with a semantic class, free, or unknown. An
embodied agent needs this memory to survive when it leaves a region and comes
back, and to revise old estimates when a new view disagrees with them
[arxiv:2607.05543 §1].

Single views are partial and uncertain, so the memory stores a probability and
a class distribution per element, not one fixed label [arxiv:2607.05543 §3.2].
The rule the method keeps is that free-space evidence is added only along each
camera ray up to the first predicted surface. Anything beyond that hit gets no
free-space evidence, so an area hidden behind an object keeps its earlier answer
[arxiv:2607.05543 §3.1] [arxiv:2607.05543 app. B.2].

## Variants

### Surface and ray evidence in a hierarchical memory (GEM-Occ)

Each posed observation yields two kinds of evidence. Each valid pixel with a
predicted surface gives one occupied semantic Gaussian. Its centre is the
unprojected depth point in the world frame, its covariance is aligned to the
viewing ray, and its opacity comes from a learned evidence adapter
[arxiv:2607.05543 §3.1, eq. 1–2]. The visible segment of each ray in front of
the hit gives free-space evidence [arxiv:2607.05543 §3.1].

Evidence is matched to nearby memory elements by squared Mahalanobis distance
under a gate. Matched elements are fused with confidence weights, and the
history is discounted by a factor λ [arxiv:2607.05543 app. B.3] [arxiv:2607.05543 §3.2,
eq. 4]. Each element keeps a log-odds occupancy, which rises with surface
matches and falls with overlap against visible free space
[arxiv:2607.05543 §3.2, eq. 5]. Memory is kept as a local cache, spatial
submaps and a building-level connectivity graph, and periodic maintenance merges
close elements and prunes weak ones [arxiv:2607.05543 §3.3, eq. 6]. Card:
[GEM-Occ](/papers/2607.05543).

Reported results, all author-reported and from single runs:

- Local prediction from one RGB view on HIOcc: 61.37 IoU and 57.76 mIoU, against
  60.69 and 55.08 for GPOcc, the next-best row in the paper's table
  [arxiv:2607.05543 tab. 3].
- Room-level online mapping: 56.79 IoU and 46.20 mIoU, against 52.94 and 44.81
  for GPOcc [arxiv:2607.05543 tab. 4].
- Building level on Matterport3D: 53.8 IoU and 46.7 mIoU at 0.81 MB/m² and 33.8 ms
  per dense query. Flat Gaussian memory, the paper's own baseline, gives 50.4 IoU
  at 1.36 MB/m² and 49.5 ms [arxiv:2607.05543 tab. 5].
- Ablations at room level remove one part at a time. Removing the semantic
  Gaussian evidence lowers local mIoU from 57.76 to 47.12. Removing the
  free-space ray evidence lowers room IoU from 56.79 to 52.74. Removing the
  hierarchy raises memory to 1.43 MB/m², and removing pruning and merging raises
  it to 1.78 MB/m² [arxiv:2607.05543 tab. 6].

The paper's benchmark, HIOcc, is built from ScanNet, ScanNet++ and Matterport3D.
It has three tracks: local, room and building [arxiv:2607.05543 §4] [arxiv:2607.05543 §5].
The paper does not state a licence for HIOcc. Each source dataset has its own
terms, so check them before any commercial use of the data
[arxiv:2607.05543 §4].

### Other designs

The card set has no other design yet. The other methods in the paper's tables
are baselines only. No card exists for them, so this page does not describe how
they work.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Posed RGB frames, static indoor scene, memory must persist across revisits | Surface and ray evidence with hierarchy (GEM-Occ) | It is the only design with a card, and it has the best reported local, room and building scores on HIOcc [arxiv:2607.05543 tab. 3–5] |
| Building-scale memory budget is the binding limit | Hierarchy and pruning (GEM-Occ) | Reported 0.81 MB/m² against 1.36 for flat Gaussian memory. The ablation without pruning reaches 1.78 [arxiv:2607.05543 tab. 5–6] |
| Memory is the only constraint and accuracy matters less | Per-frame fusion, a paper baseline | Lowest reported memory at 0.42 MB/m², but building IoU is 38.6 [arxiv:2607.05543 tab. 5] |
| Camera poses are estimated or drift | Not covered | The paper supplies poses and does not evaluate SLAM drift [arxiv:2607.05543 app. A.3]. Building IoU drops to 50.14 at 0.1 m translation error and to 45.76 at 0.2 m [arxiv:2607.05543 tab. 9] |
| Depth is noisy | Expect a drop | Building IoU falls to 39.22 at 10% relative depth noise [arxiv:2607.05543 tab. 9] |
| Dynamic scenes, outdoor scenes, or an open label set | Not covered | The paper assumes static indoor scenes and a fixed set of 11 occupied classes plus free space [arxiv:2607.05543 app. A.3] [arxiv:2607.05543 §4] |
| Real-time budget on unknown hardware | Measure first | Evidence extraction runs at 8.7 FPS and the full online pipeline at 5.2 FPS. The hardware is not stated [arxiv:2607.05543 app. A.2] |

## Implementation sketch

The paper gives no training or fusion constants, and its code is not released
yet. The steps below follow the text. Values marked "not stated" in the card
must be chosen and tuned on a validation split [arxiv:2607.05543 §3.2].

```pseudocode
# Evidence from one posed observation  [§3.1] [eq. 1]
def observation_evidence(frame, pose):
    # Per-pixel features from the visual encoder  [§3.1]
    F = encoder(frame)
    # Depth, semantic logits and confidence from the heads  [§3.1]
    D, S, Q = heads(F)
    surfaces, free = [], []
    for u in valid_pixels(D):
        # One occupied semantic Gaussian per valid surface pixel  [eq. 1]
        surfaces.append(surface_gaussian(u, D, S, Q, F, pose))
        # Free-space segment from the camera up to, not past, the hit  [§3.1]
        free.append(ray_segment_before_hit(u, D, pose))
    return surfaces, free
```

```pseudocode
# Causal update of the memory for one observation  [§3.2] [app. B.3]
def update_memory(memory, surfaces, free, lam, d_occ, d_free, tau_m):
    for g in surfaces:
        # Nearest element under the Mahalanobis gate, or None  [app. B.3]
        e = nearest_within_gate(g, memory, tau_m)
        if e is None:
            # Unmatched evidence starts a new element  [§3.2]
            memory.append(new_element(g))
        else:
            # Discount history by lam, then fuse with confidence  [eq. 4]
            fuse(e, g, lam)
            # Surface term: log-odds rise by eta times d_occ  [eq. 5]
            e.log_odds += g.eta * d_occ
    for e in memory:
        # Free-space term applies even without a surface match  [app. B.3]
        e.log_odds -= overlap_with_free(e, free) * d_free
```

The paper does not state the order of the surface and free-space terms, and
it does not clamp the log-odds. The order above is one reading.

```pseudocode
# Readout at a query point x  [§3.4] [eq. 7–8] [app. B.5]
def label_at(x, elements, free_cache, tau_occ, eps):
    # Weight each nearby element by its occupancy and its spatial response  [eq. 7]
    w = [sigmoid(e.log_odds) * gaussian_response(e, x) for e in elements]
    # Occupancy from the product of complements  [eq. 7]
    p_occ = 1.0 - prod(1.0 - wi for wi in w)
    if p_occ > tau_occ:
        # Class is the arg-max of the weighted class mix  [eq. 8]
        return argmax(weighted_mix(elements, w, eps))
    if free_cache.contains(x):
        return FREE
    # No occupied or free support: unknown, excluded from evaluation
    return UNKNOWN
```

## Contradictions

None known between papers, since only one card is in the set. Within the one
paper, the ablation caption calls the local metrics single-frame. Yet removing
the confidence weighting, the hierarchy, or pruning lowers local mIoU from 57.76
to 52.02, 52.41 and 52.48 [arxiv:2607.05543 tab. 6]. The paper does not explain
this. Its revisit-consistency values of 88.6 (room ablation) and 86.2 (building
comparison) come from different settings and should not be compared
[arxiv:2607.05543 tab. 5] [arxiv:2607.05543 tab. 6].

## Papers

- [GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory](/papers/2607.05543): surface and free-space evidence fused into a hierarchical semantic occupancy memory, with the HIOcc benchmark. The only card so far.