GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory

In plain words

A robot that walks through a house needs a memory of which space is solid, which is empty and which it has not seen yet. A camera gives a partial and uncertain picture from each spot, and the robot forgets an earlier view once it walks away. This paper builds that memory from a stream of camera frames, so the map keeps what the robot saw in earlier rooms and updates it when the robot comes back.

The core idea is to store the map as many small, soft blobs. Each new frame adds blobs where it sees surfaces, and marks the space in front of those surfaces as empty. Where a new view sees empty space where the map said "solid", the map is corrected. Where something is hidden behind another object, the map keeps its old answer, because not seeing something is not proof that it is gone. Blobs that agree are merged, and the map is split into regions so that answering "what is at this point?" stays fast as the building grows.

The authors also build a benchmark, called HIOcc, from three indoor scan datasets. It asks models to predict the same kind of semantic map at the scale of one view, one room and a whole building. On these tests, the method scores higher than the other methods they compare with, and it uses less memory per square metre at building scale than the flat Gaussian memory the authors compare against.

TL;DR

GEM-Occ builds a semantic occupancy memory from posed RGB observations. Each surface prediction becomes an occupied semantic Gaussian, and each visible segment before the surface becomes free-space evidence §3.1, eq. 1–2; app. B.2. Memory elements are fused with confidence weights, and free space lowers occupancy while occlusion does not §3.2, eq. 4–5. The authors report 61.37 IoU and 57.76 mIoU on local prediction, which beats GPOcc by 0.68 and 2.68 points tab. 3. The linked repository, at the recorded commit, has no implementation yet, so no training or inference detail can be checked against code README.md.

Problem

Embodied agents need a semantic occupancy memory that survives when the agent leaves a region and returns to it §1. Each observation is incomplete and uncertain, so the memory has to revise old estimates §1. The authors name two limits in earlier work: EmbodiedOcc initialises Gaussians uniformly over a predefined scene extent, so its initial cost grows with the scene volume §1. GPOcc merges Gaussians from each frame into a global map without any training step app. D.

The paper's own assumptions are: static indoor scenes, supplied camera poses and calibration, and a fixed semantic label set. It does not evaluate pose estimation or SLAM drift correction app. A.3, "Scope and limitations".

Method

Setting. The input is posed RGB observations with calibration. One time step is either a perspective frame or a panorama with its calibrated sub-views, all expressed in one world frame. Depth and semantic labels are used only for supervision and benchmark targets. RGB-only inference uses predicted geometry. Memory updates use only observations up to the current step app. B.1.

Step 1: observation evidence §3.1. A VGGT-based encoder, adapted to perspective and panoramic inputs, gives features F_t for each observation §3.1; app. B.2. Heads predict geometry D_t, semantic logits S_t and confidence Q_t §3.1. Each valid surface pixel u yields one occupied semantic Gaussian. Its centre is the predicted point mapped into the world frame. Its opacity comes from the evidence adapter, its semantics are the softmax of S_t(u), and its confidence is Q_t(u) §3.1, eq. 1.

# Occupied semantic Gaussian for one valid pixel u  [§3.1] [eq. 1]
# Input:  features F_t, geometry D_t, logits S_t, confidence Q_t, pose
# Output: Gaussian g = (mu, Sigma, alpha, p, eta)
def surface_evidence(u, F_t, D_t, S_t, Q_t, pose):
    # Centre: unprojected depth point, moved to world frame  [§3.1]
    mu = pose.apply(unproject(u, D_t[u]))
    # Covariance from pixel footprint and depth uncertainty  [§3.1] [eq. 2]
    Sigma = ray_aligned_covariance(u, D_t, pose)
    # Opacity predicted by the Gaussian evidence adapter from F_t(u)  [§3.1]
    alpha = evidence_adapter(F_t[u])
    # Semantic distribution and confidence at pixel u  [§3.1]
    p = softmax(S_t[u])
    eta = Q_t[u]
    return (mu, Sigma, alpha, p, eta)

The covariance is aligned to the viewing ray: its longitudinal axis follows the world-frame ray and its two transverse axes follow the projected pixel footprint §3.1, eq. 2; app. B.2.

g_i^t = (\mu_i^t,\Sigma_i^t,\alpha_i^t,p_i^t,\eta_i^t) eq. 1

\Sigma_i^t = R_i^t\,\mathrm{diag}\!\left(\sigma_{\perp,i}^{2},\sigma_{\perp,i}^{2},\sigma_{\parallel,i}^{2}\right)(R_i^t)^\top eq. 2

Free-space evidence is recorded along each valid ray from the camera up to, but not including, the predicted surface hit. Anything beyond the hit gets no free-space evidence §3.1; app. B.2.

Step 2: association app. B.3. Each evidence Gaussian is compared with nearby memory elements by squared Mahalanobis distance, using the memory covariance. The closest element with distance below \tau_m receives it. If none passes the gate, the evidence starts a new memory element with its own centre, covariance and semantic distribution app. B.3, §3.2. The distance is

d_M(g_i^t,G_j^{t-1}) = (\mu_i^t-\mu_j^{t-1})^\top(\Sigma_j^{t-1})^{-1}(\mu_i^t-\mu_j^{t-1}) app. B.3

Memory elements carry log-odds \ell, semantics p, an accumulated weight w and a count n of supporting observations §3.2, eq. 3.

G_j^t = (\mu_j^t,\Sigma_j^t,\ell_j^t,p_j^t,w_j^t,n_j^t) eq. 3

Step 3: confidence-weighted fusion §3.2. For a matched pair, the weight is discounted by \lambda \in [0,1] and the new confidence is added. The centre and the semantic distribution are averaged with these weights §3.2, eq. 4. Covariance is merged by a weighted moment rule that includes the displacement of each centre from the fused centre app. B.3.

\begin{aligned} \bar{w}_j^t &= \lambda\, w_j^{t-1} + \eta_i^t, \\ \mu_j^t &= \frac{\lambda\, w_j^{t-1}\mu_j^{t-1} + \eta_i^t\mu_i^t}{\bar{w}_j^t}, \\ p_j^t &= \frac{\lambda\, w_j^{t-1}p_j^{t-1} + \eta_i^t p_i^t}{\bar{w}_j^t}. \end{aligned} eq. 4

The new weight is w_j^t = \bar{w}_j^t §3.2. The covariance update in the appendix is

\Sigma_j^t=\frac{1}{\bar{w}_j^t}\Big[\lambda w_j^{t-1}\big(\Sigma_j^{t-1}+\delta_j^{t-1}(\delta_j^{t-1})^\top\big) + \eta_i^t\big(\Sigma_i^t+\delta_i^t(\delta_i^t)^\top\big)\Big] app. B.3

with \delta_j^{t-1}=\mu_j^{t-1}-\mu_j^t and \delta_i^t=\mu_i^t-\mu_j^t app. B.3.

When several evidence Gaussians from one observation match the same element, they are first merged into one aggregate with summed confidence. The historical discount and the free-space correction are each applied once per element per observation app. B.3.

Step 4: occupancy update §3.2. Log-odds change by a surface term and a free-space term:

\ell_j^t = \ell_j^{t-1} + \eta_i^t\,\Delta\ell_{\mathrm{occ}} - \beta_j^t\,\Delta\ell_{\mathrm{free}} eq. 5

Here \Delta\ell_{\mathrm{occ}}, \Delta\ell_{\mathrm{free}} > 0 are increments, and \beta_j^t \in [0,1] is the overlap of the element's support with the visible free-space segments of the current observation §3.2. The surface term is zero when nothing matches §3.2. The overlap is a normalised Gaussian-weighted count over sample points x \in \mathcal{S}_j^t inside the element's support app. B.3:

\beta_j^t = \frac{\sum_{x\in\mathcal{S}_j^t}\kappa_j^t(x)\,\mathbf{1}[x\in\mathcal{F}_t]}{\sum_{x\in\mathcal{S}_j^t}\kappa_j^t(x)} app. B.3

with \beta_j^t = 0 when the sample set is empty app. B.3. An element can receive a free-space correction without a surface match. If neither term applies, its log-odds stay unchanged app. B.3. Visible free segments are also kept in a sparse cache \mathcal{R}^{\mathrm{free}}_t for later queries §3.2.

The full per-observation update, as the text describes it:

# Causal memory update for one posed observation  [§3.2] [app. B.3]
# Input:  evidence Gaussians; memory elements; visible free segments
# Output: updated memory; sparse free-space cache R_free
def update_memory(evidence, memory, free_segments, free_cache, params):
    # Gate each evidence Gaussian to its closest element below tau_m  [app. B.3]
    groups = {}
    for g in evidence:
        element = nearest_within_gate(g, memory, params.tau_m)
        if element is None:
            # No match: the evidence starts a new memory element  [§3.2]
            memory.append(new_element(g))
        else:
            groups.setdefault(element, []).append(g)
    for element, group in groups.items():
        # Merge the group first: summed confidence, weighted centre  [app. B.3]
        aggregate = merge_evidence(group)
        # Discount history by lambda, then fuse the aggregate  [§3.2, eq. 4]
        fuse(element, aggregate, params.lam)
        # Surface term of the occupancy update  [§3.2, eq. 5]
        element.log_odds += aggregate.eta * params.delta_occ
    for element in memory:
        # Free-space term: overlap with visible free segments  [app. B.3]
        beta = free_overlap(element, free_segments)
        # Applies even without a surface match  [app. B.3]
        element.log_odds -= beta * params.delta_free
    # Keep visible free segments for later occupancy queries  [§3.2]
    free_cache.add(free_segments)
    return memory, free_cache

The paper does not give the order between the surface term and the free-space term, or any clamp on the log-odds. The pseudocode above puts the surface term first, which is an assumption.

Step 5: hierarchy and maintenance §3.3; app. B.4. Memory is organised into a local cache, spatial submaps and a building-level connectivity graph. Submaps are created as new regions are observed, and each observation is assigned to submaps by its camera pose. Graph nodes are submaps, and edges follow the observation-graph connectivity §3.3. Revisits update the stored Gaussians through the same fusion rules app. B.4. Maintenance runs periodically inside active submaps. It merges nearby elements whose covariance supports overlap and whose semantic distributions are close under a symmetric KL divergence, and it prunes elements whose occupancy stays low, whose semantic entropy stays high, or that free space keeps contradicting §3.3, eq. 6; app. B.4:

D_{\mathrm{SKL}}(p_j^t,p_k^t)=D_{\mathrm{KL}}(p_j^t\|p_k^t)+D_{\mathrm{KL}}(p_k^t\|p_j^t) < \tau_{\mathrm{sem}} eq. 6

Merged parameters use the fusion rule of step 3 §3.3.

Step 6: occupancy queries §3.4; app. B.5. For a query point x, let \mathcal{N}(x) be the nearby memory elements, and let \pi_j = \mathrm{sigmoid}(\ell_j). Each element contributes a Gaussian response \kappa_j(x). Occupancy and semantics are computed from these responses §3.4, eq. 7–8:

\kappa_j(x) = \exp\!\left[-\tfrac{1}{2}(x-\mu_j)^\top\Sigma_j^{-1}(x-\mu_j)\right],\qquad P_{\mathrm{occ}}(x) = 1-\prod_{j\in\mathcal{N}(x)}\left(1-\pi_j\,\kappa_j(x)\right) eq. 7

P_{\mathrm{sem}}(x) = \frac{\sum_{j\in\mathcal{N}(x)}\pi_j\,\kappa_j(x)\,p_j}{\sum_{j\in\mathcal{N}(x)}\pi_j\,\kappa_j(x)+\epsilon} eq. 8

The label is decided as follows. If P_{\mathrm{occ}}(x) > \tau_{\mathrm{occ}}, the label is the arg-max class of P_{\mathrm{sem}}. Otherwise it is free if x is supported by the free-space cache, and unknown if not §3.4; app. B.5.

# Occupancy and semantic readout at a query point x  [§3.4] [app. B.5]
# Input:  nearby elements N(x); free-space cache; thresholds
# Output: class id, 0 (free) or None (unknown)
def query_label(x, elements, free_cache, tau_occ, eps):
    # Gaussian weights: occupancy probability times spatial response  [eq. 7]
    weights = []
    for e in elements:
        weights.append(sigmoid(e.log_odds) * kappa(e, x))
    # Occupancy: one minus the product of the complements  [eq. 7]
    p_occ = 1.0 - product(1.0 - w for w in weights)
    # Semantics: same weights, normalised, with eps for stability  [eq. 8]
    numerator = sum(w * e.p for w, e in zip(weights, elements))
    p_sem = numerator / (sum(weights) + eps)
    if p_occ > tau_occ:
        # Occupied evidence takes precedence over cached free space  [app. B.5]
        return argmax(p_sem)
    if free_cache.contains(x):
        # Free only where observed free-space support exists  [app. B.5]
        return 0
    # No occupied or free support: unknown, excluded from evaluation  [app. B.5]
    return None

Step 7: training §3.4; app. B.5, B.6. Training unrolls memory fusion over several observations. It supervises the fused-memory queries together with local predictions, which are made by splatting the incoming evidence Gaussians with their opacities §3.4; app. B.5. Four terms apply to both outputs: occupancy, semantics, geometry and a ray term that penalises occupancy along observed free-space segments §3.4, eq. 9. Unknown targets are excluded from occupancy supervision §3.4. Only the visual prediction network and the evidence adapter are learned. Fusion and query rules add no learned parameters app. B.5.

\mathcal{L} = \lambda_{\mathrm{occ}}\mathcal{L}_{\mathrm{occ}} + \lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}} + \lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}} + \lambda_{\mathrm{ray}}\mathcal{L}_{\mathrm{ray}} eq. 9

Benchmark: HIOcc §4; app. C. HIOcc is built from ScanNet, ScanNet++ and Matterport3D §4. Each benchmark item is one viewpoint: either a posed perspective image, or a panorama together with its calibrated sub-views app. C.1. Targets are sparse occupied-voxel entries with a semantic label. The 11 occupied classes follow Occ-ScanNet, and free space is a twelfth label §4; app. C.1. Free space comes from observed empty ray segments, and unknown locations fall outside the valid target support and are masked out app. C.1. Perspective targets use frustum and depth-consistency filtering. Panoramic targets use first-hit filtering from the panorama centre §4; app. C.2. The three tracks are local prediction (single RGB observation), room-level online mapping (causal perspective sequences) and building-level mapping (connected panoramic viewpoints) §4, §5.

Hyperparameters

The paper gives almost no numerical hyperparameters for training or memory fusion. Most of them are deferred to the supplementary material, and the supplementary material does not state them either. Each row below says what is stated. Anything marked "not stated" must be chosen by the implementer and tuned on a validation split.

Results

Author-reported. Setup: HIOcc, 11 occupied classes plus free space, evaluated on the valid mask §4, §5; app. C.4. Local uses a single posed RGB observation on the ScanNet and ScanNet++ perspective splits. Room-level uses causal perspective sequences on the same splits. Building-level uses Matterport3D panoramic sequences, with the predictor trained on Matterport3D §5.

Local prediction, single observation tab. 3. Local mIoU is the mean over the 11 classes. Column abbreviations: Ceil. = ceiling, Win. = window, Furn. = furniture, Obj. = objects.

Text: +0.68 IoU and +2.68 mIoU over GPOcc; the largest per-class gain is on ceiling, +20.49 §5, tab. 3.

Room-level online mapping tab. 4. Causal perspective sequences; metrics computed after online fusion.

Text: +3.85 IoU and +1.39 mIoU over GPOcc; +5.89 mIoU over EmbodiedOcc §5, tab. 4.

Building-level mapping, Matterport3D tab. 5. All variants use the same predictor and supervision; they differ only in the memory design.

Text: against flat Gaussian memory, memory per m² falls by 40.4% and dense-query latency by 31.7% §5, tab. 5. Against GPOcc on the external comparison, GEM-Occ is +3.55 IoU and +2.89 mIoU at building level app. A.1, tab. 7.

Annotation consistency, HIOcc construction tab. 2. 2D–3D semantic-consistency IoU, not a model score. Columns: complete geometry reference (CompleteScanNet), mesh voxels, cropping, frustum filter, ray/first-hit depth filter, multiview semantic validation.

Ablations tab. 6. Local metrics are single-frame; room metrics are after causal fusion; revisit and memory are room-level.

Supplementary, external baselines app. A.1, tab. 7. ^\dagger = adapted to the shared protocol by the authors.

Supplementary, runtime app. A.2, tab. 8. Evidence extraction 8.7 FPS. Complete online pipeline (extraction, fusion, submap and graph maintenance, occupancy query) 5.2 FPS. Hardware not stated.

Supplementary, robustness app. A.3, tab. 9. Building level, default IoU 53.80 / mIoU 46.70.

Supplementary, perspective target grids app. C.2, tab. 10.

Dataset size §4, tab. 1. HIOcc: over 2,000 scenes, about 300,000 observations, about 142,600 m² in total, from ScanNet, ScanNet++ and Matterport3D (about 46.3k, 31.6k and 64.7k m² per the table row, without an explicit per-source mapping) tab. 1.

Figures, described in words. fig. 1 (teaser) shows HIOcc's hierarchy from local views to buildings and the three method tracks. fig. 2 shows the pipeline: evidence adapter, causal memory updates, confidence-weighted fusion, free-space revision, and readout at three scales. fig. 3 shows the annotation pipeline: category mapping, voxelisation, and viewpoint cropping and filtering. fig. 4 shows local predictions, where GEM-Occ has fewer missing and fragmented regions than the baselines. fig. 5 shows the map growing over connected panoramic viewpoints. Appendix app. A.4 adds two more building sequences, and app. C.1 compares HIOcc targets with Occ-ScanNet.

Paper vs code

The paper's title block links github.com/zhuhu00/GEM-Occ, and the owner zhuhu00 matches the authors' project-page domain zhuhu00.top § title block. Only this repository was checked, because the paper linked it and no other search was needed. The repository at commit 5ca9104845c6b14f55c709b37fb364f5745bb63b holds only a README and an MIT LICENSE file, with no source code, configs or scripts. The README says the implementation, checkpoints and dataset, training and evaluation scripts are "planned for release" README.md. The paper's reproducibility statement also says the code "will be released" [reproducibility statement, after §6].

So no paper value can be checked against code yet. The hyperparameters listed as "not stated" above cannot be filled from this repository.

Pitfalls

• No implementation to compare against. The code is planned, not released README.md. Every value marked "not stated" (the association gate, \lambda, the log-odds increments, \tau_{\mathrm{occ}}, \tau_{\mathrm{sem}}, \epsilon, the loss weights, the optimiser and the schedule) has to be chosen by the implementer §3, app. B.6. Tune them on a held-out split, not on the reported benchmark.
• Local metrics move in the ablation, though they are single-frame. The ablation caption says local metrics come from single-frame predictions tab. 6. Yet switching off the confidence weighting, the hierarchy, or pruning/merging lowers local mIoU from 57.76 to 52.02, 52.41 and 52.48 tab. 6. The paper does not explain this. The text does not say how each ablation was trained or evaluated.
• Two revisit-consistency numbers. The full model reports 88.6 in the room-level ablation tab. 6 and 86.2 in the building comparison tab. 5. These are different settings. Do not mix them.
• Revisit consistency and progress AUC have no formulas. The text describes both in words §4; app. C.4. The sampling of exploration progress and the matching of revisited regions are not stated.
• Where depth uncertainty comes from is not stated. \sigma_{\parallel} "reflects depth uncertainty" §3.1. But the prediction heads listed are D_t, S_t and Q_t only §3.1. The source of the uncertainty and the scale rule for \sigma_{\perp} and \sigma_{\parallel} are not stated.
• Covariance conditioning. The association gate uses (\Sigma_j^{t-1})^{-1} app. B.3 and the query uses \Sigma_j^{-1} eq. 7. The paper does not discuss regularisation. A near-singular covariance (for example a very small \sigma_{\parallel}) would make these inverses unstable (inference from the formulas, not stated in the paper).
• Free space is only the visible part before the hit. Beyond the first hit, the memory keeps its old state §3.1; app. B.2. An unknown point and a free point differ only through the sparse cache \mathcal{R}^{\mathrm{free}}_t app. B.5, so a sparse cache leads to more "unknown" answers. The paper does not report how often that happens.
• Log-odds have no stated clamp, and the order of the surface term and the free-space term is not stated §3.2; app. B.3. The pseudocode above is one reading.
• Occupied readout precedence. If an occupied query and cached free space overlap, occupied wins app. B.5. That is stated, but it is a design choice the paper does not justify.
• Panorama sub-view count and per-track grid resolution are not stated app. B.1, app. C.2.
• Building-level results depend on the Matterport3D-trained predictor. Training on perspective data and testing on panoramas is not evaluated app. A.3. Neither is any other dataset.
• External baselines are adapted by the authors. The † rows in tab. 7 follow the shared protocol, but the adaptation is not described app. A.1.
• Runtime hardware is not stated, and the query workload behind "dense-query latency" is not described app. A.2.
• Single runs. The paper reports no variance across seeds. The gap over GPOcc on local IoU is small, 0.68 points tab. 3.
• Annotation consistency is a proxy. The 56.5 → 74.9 gain is 2D–3D semantic agreement, not accuracy against ground truth §4, app. C.3.
• Dataset terms. HIOcc is on Hugging Face with sign-in and access conditions README.md. The paper does not state the license of HIOcc. Its sources, ScanNet, ScanNet++ and Matterport3D, have their own terms that the paper does not discuss. Check those terms before using the data commercially. This card's CC BY 4.0 covers the paper text only.
• Version. This card follows arXiv v2 (updated 2026-10-05), from the LaTeX source fetched by wikillm fetch.

Lineage

• Builds on: VGGT, a visual geometry transformer, used as the encoder after adaptation §3.1; app. B.2; wang2025vggt.
• Compared against: MonoScene, ISO, SplatSSC, EmbodiedOcc, EmbodiedOcc++, GPOcc (local, room and building), SplicingOcc (room), LegoOcc and FreeOcc (supplementary) tab. 3, tab. 4, tab. 7.
• Related design: flat Gaussian memory, pointmap fusion and per-frame fusion are the building-level baselines inside the paper tab. 5.
• Superseded by: not stated.