3D scene graph

Overview

A 3D scene graph describes a space as a set of nodes joined by relations. The nodes carry 3D attributes such as a pose, a shape or a bounding box, and each node has a unique ID. Edges record how two nodes relate, for example which room an agent is in at a given time 2101.06894 §2. The graph can be flat, with objects and their pairwise relations only, or layered, with several levels of meaning stacked on top of each other 2101.06894 §2 2406.01584 §3.1.

Two families are covered here, and they use the name for different jobs.

• Robot mapping. The graph is built from the sensors of a moving robot so that a planner can reason over it. Kimera is the reference design 2101.06894 §2 2101.06894 §3.
• Image understanding. The graph is built from one photo and turned into question-answer pairs, so that a vision-language model learns spatial relations such as left, above and metric distance 2406.01584 §3.1 2406.01584 §3.2.

Relations between 2-D detections in a single image, without metric 3D boxes, belong to scene graph generation.

Variants

Layered robot map

Kimera defines a five-layer graph for a single-storey indoor building: a metric-semantic mesh, objects and agents, places and structures, rooms, and one building node 2101.06894 §2. The bottom layer is built in real time from stereo or RGB-D images and IMU data 2101.06894 §3. Agents are updated during the run. Objects, places, structures and rooms are parsed after the run ends 2101.06894 §3. Places come from a topological graph on the ESDF, each object links to its nearest place, and rooms come from an ESDF slice taken 0.3 m below the ceiling 2101.06894 §3.7. Moving humans get their own pose graphs and are used for dynamic masking 2101.06894 §3.5. Card: Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs.

The author-reported results are on simulated and EuRoC data. Room parsing reaches 99% precision and 99% recall in an office and 68% and 61% in an apartment 2101.06894 §4.7. Building a DSG for the largest scene, a subway, takes about 30 minutes 2101.06894 §4.8.

Single-image spatial graph

For each photo, the pipeline keeps images that look like ordinary indoor or outdoor photos 2406.01584 app. F.1, finds the object classes, and grounds each to a box and then a mask 2406.01584 §3.1. It estimates the camera intrinsics and the metric depth, levels the ground with pitch and roll, and fits one axis-aligned 3D box per object 2406.01584 §3.1 2406.01584 app. F.3. Edges then store left/right, above/below, front/behind and metric direction and distance 2406.01584 §3.1. Template questions and an LLM-written set are generated from the graph 2406.01584 §3.2. The released dataset has 1M images, 5M regions and about 8.7M questions in total 2406.01584 §3.2. Card: SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models.

The model takes region masks or boxes plus an optional relative-depth map 2406.01584 §3.3. On the authors' SpatialRGPT-Bench, the 7B model scores 91.78 average qualitative accuracy against 58.14 for GPT-4V 2406.01584 tab. 1. Its direct-distance success is 41.2% against 29.7% 2406.01584 tab. 1. Width and height success are below GPT-4V, while the errors are lower 2406.01584 tab. 1. The judge is GPT-4 2406.01584 app. J.

When to use which

Spatial training slightly lowered some general VQA scores. LLaVA-Bench fell from 75.9 to 71.5 for the 3B model 2406.01584 tab. 2. Keep that in mind when the model must also do open-ended chat.

Implementation sketch

Both variants reduce to the same core: nodes with a layer tag and 3D attributes, and typed edges. They differ in where the nodes come from.

# Common core: nodes with a layer and 3D attributes, plus typed edges
# [arxiv:2101.06894 §2]

def add_node(graph, layer, attrs):
    # Every node gets a unique ID and its spatial attributes  [arxiv:2101.06894 §2]
    node_id = graph.next_id()
    graph.add_node(node_id, layer=layer, **attrs)
    return node_id

def add_edge(graph, a, b, relation, time=None):
    # Edges are pairwise, optionally time-stamped  [arxiv:2101.06894 §2]
    graph.add_edge(a, b, relation=relation, time=time)

# Variant A, robot map: online bottom layer, parsed upper layers
# [arxiv:2101.06894 §3] [arxiv:2101.06894 §3.7]

def build_dsg_from_run(frames, imu):
    # Live part: pose and mesh, updated per keyframe  [arxiv:2101.06894 §3]
    mesh = run_online_mapping(frames, imu)
    # After the run: parse the upper layers from the mesh  [arxiv:2101.06894 §3]
    esdf = build_esdf(mesh)
    places = topological_graph(esdf)
    rooms = rooms_from_slice(esdf, height_below_ceiling=0.3)  # metres
    return link_objects_to_places(places, rooms, objects_from(mesh))

# Variant B, one photo: lift to 3D, then write relations  [arxiv:2406.01584 §3.1]

def scene_graph_from_photo(image):
    # Skip images that are not ordinary photos  [arxiv:2406.01584 app. F.1]
    if not is_ordinary_photo(image):
        return None
    # Ground each class to a box, then refine to a mask  [arxiv:2406.01584 §3.1]
    regions = ground_and_segment(image)
    # Intrinsics first, then metric depth that uses the focal length  [arxiv:2406.01584 §3.1]
    K = estimate_intrinsics(image)
    depth = metric_depth(image, focal=K.fx)
    # Level the ground with pitch and roll, then fit boxes  [arxiv:2406.01584 app. F.3]
    boxes = fit_boxes(lift(regions, depth, K, level_ground(image)))
    # Pairwise left/right, above/below, front/behind and metric edges  [arxiv:2406.01584 §3.1]
    return relations(boxes)

Contradictions

• Across the two papers: none known. They measure different things: map building for planning in Kimera, and question-answer accuracy for a vision-language model in SpatialRGPT.
• Within Kimera: the Kimera-Semantics voxel size is 0.05 m in the clustering section 2101.06894 §3.6. The timing section gives 0.1 m as the default on the TX2 2101.06894 §4.9. The paper does not say which setting produced its accuracy tables 2101.06894 tab. 6.
• Within Kimera: the text says the multi-frame mesh is two orders of magnitude faster than the global mesh 2101.06894 §4.3. The reported times give a factor closer to seven 2101.06894 §3.3 2101.06894 §4.8.
• Within SpatialRGPT: the text says only the depth connector trains in the second stage, with the language model frozen 2406.01584 §3.4. The released scripts also tune the language model in that stage. See the card's paper-vs-code section.

Papers

• Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs: a five-layer scene graph built online from stereo or RGB-D and IMU data, with layers above the mesh parsed after the run.
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models: a single-image pipeline that lifts photos to 3D boxes and relations, and a region-prompted VLM trained on the resulting questions.