RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

In plain words

Vision-language models can describe a scene, but they often get the layout wrong: where one object sits relative to another, or whether there is room for a new one. Robots need exactly that. RoboSpatial is a large set of questions built automatically from existing 3D scans of rooms and tabletops. Its questions ask where an object can go, whether it fits, and whether one object is left of, above or in front of another. Each question can be read from the camera's view, from a fixed world frame, or from the object's own facing direction. Because the answers come from the 3D boxes rather than from human labels, the set can be made very large.

Models fine-tuned on this set answer questions from the authors' held-out split much better than before. In a small real-robot test, the score of one model rose clearly, while another barely changed. All numbers here are the authors' own.

TL;DR

RoboSpatial is a spatial question-answering dataset built automatically from 3D indoor and tabletop scans: about 3M question-answer pairs over 1M images and 5k scans §1 tab. 2. It covers three relation types, each read in ego, world or object frames §3.1. Fine-tuning 2D and 3D VLMs on it raises accuracy on the authors' held-out split, for example LLaVA-NeXT from 30.3 to 60.5 tab. 3. Use it when you need training data for spatial relations. The official generator is non-commercial, so a commercial build has to re-implement the pipeline from this card.

Problem

General-purpose VLM training sets lack reference-frame understanding, meaning whether a relation such as "left" is meant from the camera, the world or the object §1. Existing spatial VLMs learn from internet images with labels made by perception models, which can fail to transfer to robot cameras §1. Pointing models predict free-space points but do not model whether an object fits or which frame applies §1. The paper's working hypothesis is that scarce suitable training data is a main limit for VLMs in robotics §1.

Method

Relation types. The dataset has three types §3.1:

• Context: points in empty space that satisfy a directional relation to an anchor object. The answer is a list of 2D image coordinates §3.1 §3.2.3.
• Compatibility: whether a virtual box of the target's size fits at a candidate location. The answer is True or False §3.2.2.
• Configuration: whether a directional relation holds between two objects. The answer is True or False §3.2.1.

Directions are left, right, above, below, front and behind §3.2.1. Distance is excluded because it is hard to normalise across scales and tasks §3.1 app. B.2. Counting is excluded as outside the scope of spatial reasoning app. B.2.

Inputs. The pipeline takes RGB images, camera intrinsics and extrinsics, and oriented 3D bounding boxes with semantic labels from a scene dataset §3.2. The authors take boxes and embodied images from EmbodiedScan and apply the pipeline to ScanNet, Matterport3D and 3RScan (indoor) and HOPE and GraspNet-1B (tabletop) §4.1. The point-cloud and mesh data are not needed §3.2.1.

Stage 1: 3D relation extraction §3.2.1.

1. Each oriented box gives a centre and a heading vector. The heading is the object's front direction §3.2.1.
2. Relations are computed separately in three frames: world (dataset axes), ego (camera pose) and object (the anchor's heading) §3.2.1.
3. Vertical relations in the world frame use global z app. B.4.
4. Only visible object pairs that appear uniquely in the image are used §3.2.1.
5. Each label is True or False §3.2.1.

# Stage 1: label one configuration question  [§3.2.1]
# Input:  anchor, target: oriented 3D boxes (center, size, heading)
#         camera: extrinsics (world to camera)  [§3.2.1]
#         relation: left, right, above, below, front or behind
#         frame: ego, world or object
# Output: True or False

def configuration_label(anchor, target, camera, relation, frame):
    # Only visible pairs that appear uniquely in the image are used  [§3.2.1]
    # Express the offset target - anchor in the chosen frame  [§3.2.1]
    # ego: camera pose; world: dataset axes; object: anchor heading
    offset = offset_in_frame(anchor, target, camera, frame)
    # Vertical relations in the world frame use global z  [app. B.4]
    # The exact direction-region test is not stated  [§3.2.1]
    return offset_matches_relation(offset, relation, frame)

Stage 2: 2D point and region sampling §3.2.2.

1. Build a top-down occupancy map from the scene's 3D boxes §3.2.2 fig. 4.
2. For context questions, sample free 3D points in the requested direction from the anchor §3.2.2.
3. Drop points whose camera ray hits an occupied box before reaching the point §3.2.2.
4. Project the remaining points into the image with the intrinsics §3.2.2.
5. For compatibility, place a virtual box of the target's size on the ground plane. Allow translation and in-plane rotation. Accept if it hits no scene box and keeps a 10 cm margin along each axis §3.2.2 app. B.4.

# Stage 2: spatial context points  [§3.2.2]
# Input:  scene boxes, anchor box, camera (intrinsics, extrinsics), frame
# Output: list of (x, y) image coordinates

def context_points(scene, anchor, camera, relation, frame):
    # Top-down occupancy map of the boxes  [§3.2.2] [app. B.4]
    occupancy = top_down_occupancy(scene.boxes)
    # GPT-4o also picks support surfaces; its use is not detailed  [app. B.4]
    # Sample free-space 3D points in the relation direction  [§3.2.2]
    # Sample count and distribution are not stated
    candidates = sample_free_points(occupancy, anchor, relation, frame)
    points_2d = []
    for point in candidates:
        # Drop points whose camera ray hits an occupied box first  [§3.2.2]
        if ray_hits_box_before(camera.center, point, scene.boxes):
            continue
        # Project with the camera intrinsics  [§3.2.2]
        points_2d.append(project(point, camera))
    return points_2d

# Spatial compatibility  [§3.2.2] [app. B.4]
# Input:  target box (size), candidate ground-plane location, scene boxes
# Output: True or False

def compatible(target, location, scene_boxes):
    # Try translations and in-plane rotations of the target box  [app. B.4]
    # The number of poses tried is not stated
    for pose in candidate_poses(target, location):
        virtual = place_box(target, pose)
        # The placed box must not intersect any scene box  [§3.2.2]
        if intersects_any(virtual, scene_boxes):
            continue
        # Keep a 10 cm margin along each axis  [§3.2.2]
        # Whether the margin pads the box or measures the gap is not stated
        if has_margin(virtual, scene_boxes, 0.10):
            return True
    return False

Question-answer generation §3.2.3. Each question is built from a fixed template, "{target} {relation} {anchor} {ref. frame}". The authors use deterministic templates so that models learn grounding rather than language priors §3.2.3. The paper does not list the template wording beyond that format.

Object grounding auxiliary data §3.2.3 app. B.3. For each image, object descriptions such as "the chair" are paired with 2D boxes projected from the 3D boxes. The auxiliary set has 100k grounding QA pairs. It is added during training but does not count toward the spatial result §4.1.1 app. B.3.

Scale and splits tab. 2 tab. 7. The main text gives about 3M QA pairs. Training uses 900,000 relations, sampled equally across datasets app. B.1. The validation split is drawn from scans never seen in training §4.1.2.

Training and evaluation §4.1.1 §4.1.2 §4.1.4.

• Trained: VILA-1.5-8B and LLaVA-NeXT-8B, plus SpaceLLaVA-13B and RoboPoint-13B as specialised models; 3D-LLM and LEO as 3D models.
• Zero-shot only: Molmo-7B and GPT-4o.
• Metrics: accuracy on yes/no questions. For coordinate answers, a prediction counts only if it falls within the hull spanned by reference points that come from the scene's geometry. The authors call this strict §4.1.2.

Hyperparameters

Results

The author-reported numbers are below. "+RoboSpatial" is fine-tuned on the dataset. The paper's own tables are copied in full. Accuracy is percent.

Table 3 (validation split, RoboSpatial-Val) tab. 3. Columns are indoor and tabletop scores for each relation type, then averages.

Table 8 (same split, per reference frame) tab. 8. Frame scores, with the same averages as Table 3.

Table 4 (out-of-domain) tab. 4. RoboSpatial-Home (350 questions, configuration, context and compatibility), BLINK spatial accuracy, and SpatialBench position accuracy.

Table 5 (cross-dataset generalisation) tab. 5. Column headers as printed: "Indoor ↓ Tabletop" and "Tabletop ↓ Indoor". The arrow direction is not explained, so which split was used for training is unclear (see Pitfalls).

Table 6 (real robot, success rate) tab. 6. Success over the robot task suite (over 200 model queries in total §4.3).

Table 7 (full dataset statistics, indoor and tabletop) tab. 7. Scans or scenes, images and QA counts per relation type.

Table 2 (summary) tab. 2. Indoor train 4,916 scans, 883k images, 2.7M QA; indoor val 40 scans, 1k images, 3k QA; tabletop train 190 scenes, 76k images, 220k QA; tabletop val 77 scenes, 355 images, 3k QA. The totals match the rows of Table 7.

Table 9 (data scaling, LLaVA-NeXT, average accuracy on RoboSpatial-Val) tab. 9:

Table 10 (general multimodal benchmarks, LLaVA-NeXT) tab. 10. Commonsense check.

Table 11 (grounding ablation, LLaVA-NeXT, average on RoboSpatial-Val) tab. 11:

Headline numbers, author-reported. On the held-out split, every model gains from fine-tuning, with totals from 30.3 to 60.5 for LLaVA-NeXT and from 42.8 to 71.9 for LEO tab. 3. Out of domain, SpaceLLaVA's compatibility score on the Home set rises from 41.0 to 72.4 tab. 4. On the robot suite, LLaVA-NeXT rises from 23.7% to 52.6%, while RoboPoint barely moves (44.7% to 46.2%) tab. 6.

Paper vs code

No code comparison was made. The official repository is NVlabs/RoboSpatial, linked from the project page at https://chanh.ee/RoboSpatial/. Its LICENSE is the NVIDIA Source Code License, whose use limitation (LICENSE §3.3) allows only research or evaluation use. wikillm publishes with advertising, so this card is written from the paper alone and does not read or cite the repository's code. The paper's own statements are the only source here. The repository is the best place to resolve the open questions in Pitfalls, for anyone with a research licence.

Pitfalls

• Direction rule is not given. The paper defines directions by a "directional region" relative to the heading vector §3.2.1, but the exact cut-off is not stated. Only the vertical relations in the world frame are said to use global z app. B.4. An implementer must choose a rule and test it against a few hand-checked scenes.
• Clearance wording is ambiguous. "At least a 10 cm margin along each axis" §3.2.2 could mean the placed box is padded, or that the gap to neighbours is measured. The paper does not say. The pose search size is also not stated app. B.4.
• Sampling is underspecified. Context points are "randomly sampled" in free space §3.2.2, but the number of samples, the sampling distribution and the free-space threshold are not given.
• GPT-4o is a hidden dependency. The appendix says GPT-4o chooses candidate support objects app. B.4. The prompt, the model version used for the data and the object list are not given, so the pipeline is not reproducible without re-doing that step.
• The data size in the abstract is not the training size. The intro and abstract describe about 3M relations §1. Training used 900,000 relations app. B.1. Table 9 marks 900K as the default and gives 60.5 for that row tab. 9, which matches LLaVA-NeXT + RoboSpatial in Table 3 tab. 3.
• Table 3 rows include the grounding data. The grounding auxiliary set is used in training §4.1.1. Table 11 separates the effects: 51.8 for RoboSpatial alone and 60.5 for both tab. 11. Expect a lower score if you drop the grounding data.
• Fine-tuning hyperparameters are not in the paper. The paper says each 2D model was fine-tuned with "default hyperparameters" from its own repository app. C.2. Learning rate, batch size, optimiser and schedule are not stated. The 3D models use 20 epochs (3D-LLM) and 10 epochs (LEO), with the best validation checkpoint kept app. C.2.
• The coordinate metric is strict. A point just outside the convex hull counts as wrong, so the authors call the scores conservative §4.1.2. The reference point set's construction is not described beyond "derived from scene geometry".
• Table 5 direction is unclear. The section says models are trained on one environment type and evaluated on the other §4.1.3. The arrows in the headers ("Indoor ↓ Tabletop", "Tabletop ↓ Indoor") do not say which is training and which is test tab. 5. Confirm the direction before using these numbers.
• The 3D comparison is confounded. 3D-LLM and LEO were pretrained on RGB-D indoor scans, some overlapping the source datasets, which the authors say may bias them §4.2. The paper calls the 2D versus 3D comparison inconclusive.
• Commonsense claim is slightly overstated. The text says trained models maintain or slightly improve performance on every general benchmark app. D.3. Table 10 shows MME cognition falling from 305.4 to 293.2 tab. 10.
• Figures are described, not reproduced. Figure 2 shows in-domain and out-of-domain examples fig. 2; Figure 3 shows the robot setup and outputs fig. 3. The paper says some robot cases were worse after RoboSpatial training, especially for RoboPoint app. D.5.
• Robot results are small and modular. The VLM outputs a point; SAM 2 finds the object for picking; placement uses depth at the point app. D.5. The authors report no cuRobo failures. Per-model query counts are not given §4.3. The paper says a 2-pixel image shift can mean a 5 to 10 cm error §4.3, which is a stated observation, not a measured sweep.
• Source data has its own terms. Training needs ScanNet, Matterport3D, 3RScan, HOPE, GraspNet-1B and EmbodiedScan, each under its own licence. The acknowledgements point to the Matterport terms of use. This card does not check those terms or the licence of the released dataset.
• Scope limits. Containment (whether an object fits inside another) is not covered app. A. Humans and animals are not in the scenes app. A.

Lineage

• Builds on: EmbodiedScan as the source of 3D boxes and embodied images §4.1. The scene datasets are ScanNet, Matterport3D, 3RScan, HOPE and GraspNet-1B §4.1.
• Fine-tunes: VILA-1.5-8B, LLaVA-NeXT-8B, SpaceLLaVA (community SpatialVLM), RoboPoint-13B, 3D-LLM and LEO §4.1.1 app. C.2.
• Compared against: Molmo-7B and GPT-4o as zero-shot baselines §4.1.1. BLINK-Spatial and SpatialBench for out-of-domain tests §4.1.4. SpatialRGPT is excluded because it needs external masks §4.1.1.