---
kind: concept
slug: 3d-scene-graph
title: 3D scene graph
summary: >-
  A 3D scene graph stores a scene as nodes at several levels of meaning, joined by spatial or temporal relations. Two routes are covered: a robot pipeline that builds the layers from stereo or RGB-D and IMU data, with the upper layers parsed after the run (Kimera), and a single-image pipeline that lifts a photo to 3D boxes and turns them into spatial QA for a vision-language model (SpatialRGPT).
tags: [scene-graph, 3d-mapping, slam, spatial-reasoning, vision-language-model, path-planning]
papers: ["2101.06894", "2406.01584"]
related: [scene-graph-generation, simultaneous-localization-and-mapping, visual-inertial-odometry, monocular-depth-estimation, multimodal-language-model, visual-grounding]
status: draft
updated_at: 2026-10-31
---

## Overview

A 3D scene graph describes a space as a set of nodes joined by relations. The
nodes carry 3D attributes such as a pose, a shape or a bounding box, and each
node has a unique ID. Edges record how two nodes relate, for example which
room an agent is in at a given time [arxiv:2101.06894 §2]. The graph can be
flat, with objects and their pairwise relations only, or layered, with
several levels of meaning stacked on top of each other [arxiv:2101.06894 §2]
[arxiv:2406.01584 §3.1].

Two families are covered here, and they use the name for different jobs.

- **Robot mapping.** The graph is built from the sensors of a moving robot so
  that a planner can reason over it. Kimera is the reference design
  [arxiv:2101.06894 §2] [arxiv:2101.06894 §3].
- **Image understanding.** The graph is built from one photo and turned into
  question-answer pairs, so that a vision-language model learns spatial
  relations such as left, above and metric distance [arxiv:2406.01584 §3.1]
  [arxiv:2406.01584 §3.2].

Relations between 2-D detections in a single image, without metric 3D boxes,
belong to [scene graph generation](/concepts/scene-graph-generation).

## Variants

### Layered robot map

Kimera defines a five-layer graph for a single-storey indoor building: a
metric-semantic mesh, objects and agents, places and structures, rooms, and
one building node [arxiv:2101.06894 §2]. The bottom layer is built in real
time from stereo or RGB-D images and IMU data [arxiv:2101.06894 §3]. Agents
are updated during the run. Objects, places, structures and rooms are parsed
after the run ends [arxiv:2101.06894 §3]. Places come from a topological graph
on the ESDF, each object links to its nearest place, and rooms come from an
ESDF slice taken 0.3 m below the ceiling [arxiv:2101.06894 §3.7]. Moving
humans get their own pose graphs and are used for dynamic masking
[arxiv:2101.06894 §3.5]. Card: [Kimera: from SLAM to Spatial Perception with
3D Dynamic Scene Graphs](/papers/2101.06894).

The author-reported results are on simulated and EuRoC data. Room parsing
reaches 99% precision and 99% recall in an office and 68% and 61% in an
apartment [arxiv:2101.06894 §4.7]. Building a DSG for the largest scene, a
subway, takes about 30 minutes [arxiv:2101.06894 §4.8].

### Single-image spatial graph

For each photo, the pipeline keeps images that look like ordinary indoor or
outdoor photos [arxiv:2406.01584 app. F.1], finds the object classes, and
grounds each to a box and then a mask [arxiv:2406.01584 §3.1]. It estimates
the camera intrinsics and the metric depth, levels the ground with pitch and
roll, and fits one axis-aligned 3D box per object [arxiv:2406.01584 §3.1]
[arxiv:2406.01584 app. F.3]. Edges then store left/right, above/below,
front/behind and metric direction and distance [arxiv:2406.01584 §3.1].
Template questions and an LLM-written set are generated from the graph
[arxiv:2406.01584 §3.2]. The released dataset has 1M images, 5M regions and
about 8.7M questions in total [arxiv:2406.01584 §3.2]. Card: [SpatialRGPT:
Grounded Spatial Reasoning in Vision Language Models](/papers/2406.01584).

The model takes region masks or boxes plus an optional relative-depth map
[arxiv:2406.01584 §3.3]. On the authors' SpatialRGPT-Bench, the 7B model
scores 91.78 average qualitative accuracy against 58.14 for GPT-4V
[arxiv:2406.01584 tab. 1]. Its direct-distance success is 41.2% against 29.7%
[arxiv:2406.01584 tab. 1]. Width and height success are below GPT-4V, while
the errors are lower [arxiv:2406.01584 tab. 1]. The judge is GPT-4
[arxiv:2406.01584 app. J].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Robot with stereo or RGB-D plus IMU that must plan over the map | Layered robot map | The bottom layer runs online. Planning on the high layers took 0.34 s to 1.24 s, against 420 s to 21,389 s on the ESDF, in the paper's office test [arxiv:2101.06894 tab. 9] |
| Map needed while the robot is still moving, with rooms and objects | Layered robot map, with care | Objects, places and rooms are parsed after the run [arxiv:2101.06894 §3]. The paper lists incremental construction as future work [arxiv:2101.06894 §8] |
| Only photos, no depth sensor and no trajectory | Single-image spatial graph | Depth and intrinsics come from estimators, so no SLAM is needed [arxiv:2406.01584 §3.1] |
| A vision-language model must answer side or distance questions | Single-image spatial graph as training data | Distance and direction success rose over GPT-4V on the authors' benchmark [arxiv:2406.01584 tab. 1] |
| People move through the scene | Kimera's human graphs and dynamic masking | With 60 humans and ground-truth poses, mesh RMSE fell from 0.19 m to 0.06 m [arxiv:2101.06894 tab. 5] |
| Only 2-D relations between detections are needed | [Scene graph generation](/concepts/scene-graph-generation) | No metric 3D is needed |
| Limited budget for generating training data | Plan for the LLM cost | The LLM questions were written by Llama3-70B [arxiv:2406.01584 §3.2]. Scene-graph building took about one day on 8 GPUs for 1.7M images [arxiv:2406.01584 app. I] |

Spatial training slightly lowered some general VQA scores. LLaVA-Bench fell
from 75.9 to 71.5 for the 3B model [arxiv:2406.01584 tab. 2]. Keep that in
mind when the model must also do open-ended chat.

## Implementation sketch

Both variants reduce to the same core: nodes with a layer tag and 3D
attributes, and typed edges. They differ in where the nodes come from.

```pseudocode
# Common core: nodes with a layer and 3D attributes, plus typed edges
# [arxiv:2101.06894 §2]

def add_node(graph, layer, attrs):
    # Every node gets a unique ID and its spatial attributes  [arxiv:2101.06894 §2]
    node_id = graph.next_id()
    graph.add_node(node_id, layer=layer, **attrs)
    return node_id

def add_edge(graph, a, b, relation, time=None):
    # Edges are pairwise, optionally time-stamped  [arxiv:2101.06894 §2]
    graph.add_edge(a, b, relation=relation, time=time)
```

```pseudocode
# Variant A, robot map: online bottom layer, parsed upper layers
# [arxiv:2101.06894 §3] [arxiv:2101.06894 §3.7]

def build_dsg_from_run(frames, imu):
    # Live part: pose and mesh, updated per keyframe  [arxiv:2101.06894 §3]
    mesh = run_online_mapping(frames, imu)
    # After the run: parse the upper layers from the mesh  [arxiv:2101.06894 §3]
    esdf = build_esdf(mesh)
    places = topological_graph(esdf)
    rooms = rooms_from_slice(esdf, height_below_ceiling=0.3)  # metres
    return link_objects_to_places(places, rooms, objects_from(mesh))
```

```pseudocode
# Variant B, one photo: lift to 3D, then write relations  [arxiv:2406.01584 §3.1]

def scene_graph_from_photo(image):
    # Skip images that are not ordinary photos  [arxiv:2406.01584 app. F.1]
    if not is_ordinary_photo(image):
        return None
    # Ground each class to a box, then refine to a mask  [arxiv:2406.01584 §3.1]
    regions = ground_and_segment(image)
    # Intrinsics first, then metric depth that uses the focal length  [arxiv:2406.01584 §3.1]
    K = estimate_intrinsics(image)
    depth = metric_depth(image, focal=K.fx)
    # Level the ground with pitch and roll, then fit boxes  [arxiv:2406.01584 app. F.3]
    boxes = fit_boxes(lift(regions, depth, K, level_ground(image)))
    # Pairwise left/right, above/below, front/behind and metric edges  [arxiv:2406.01584 §3.1]
    return relations(boxes)
```

## Contradictions

- **Across the two papers:** none known. They measure different things: map
  building for planning in Kimera, and question-answer accuracy for a
  vision-language model in SpatialRGPT.
- **Within Kimera:** the Kimera-Semantics voxel size is 0.05 m in the
  clustering section [arxiv:2101.06894 §3.6]. The timing section gives 0.1 m
  as the default on the TX2 [arxiv:2101.06894 §4.9]. The paper does not say
  which setting produced its accuracy tables [arxiv:2101.06894 tab. 6].
- **Within Kimera:** the text says the multi-frame mesh is two orders of
  magnitude faster than the global mesh [arxiv:2101.06894 §4.3]. The reported
  times give a factor closer to seven [arxiv:2101.06894 §3.3] [arxiv:2101.06894 §4.8].
- **Within SpatialRGPT:** the text says only the depth connector trains in the
  second stage, with the language model frozen [arxiv:2406.01584 §3.4]. The
  released scripts also tune the language model in that stage. See the
  [card's paper-vs-code section](/papers/2406.01584).

## Papers

- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](/papers/2101.06894): a five-layer scene graph built online from stereo or RGB-D and IMU data, with layers above the mesh parsed after the run.
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](/papers/2406.01584): a single-image pipeline that lifts photos to 3D boxes and relations, and a region-prompted VLM trained on the resulting questions.