SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

In plain words

Vision-language models can look at a picture and answer questions about it, but they are weak at spatial questions: which object is to the left, which is in front, or roughly how far apart two things are. This paper's model, SpatialRGPT, lets the user point at a specific object, with a box or a mask, and ask about it. That makes "the cup on the left" a concrete region instead of a phrase the model has to guess.

Training data is the hard part, and the authors build it automatically. For each photo they find the objects, estimate how far away they are and how the camera is tilted, and place each object in a simple 3D map of boxes. From that map they generate questions and answers about sizes, left/right, above/below and distances. The model also gets an optional depth map as an extra input, through a small add-on module, so it can use depth when it is available and still work without it.

The authors report that the model beats GPT-4V on their own spatial benchmark and that it can also serve as a reward annotator for a robot task.

TL;DR

SpatialRGPT is a LLaMA2-7B vision-language model with region prompts and a depth plugin, trained on 1M OpenImages photos converted into 3D scene graphs and then into QA pairs §3.1 §3.2. The authors report 91.78 average qualitative accuracy on SpatialRGPT-Bench against 58.14 for GPT-4V tab. 1. The code is Apache-2.0 and differs from the paper in several training details (see Paper vs code). Results are author-reported and judged by GPT-4.

Problem

Standard VLMs process the whole image globally and cannot be asked about one object instance reliably; the paper calls this the region problem §1. Spatial relations such as direction and distance also cannot be recovered from RGB pixels alone, so the paper argues 3D or depth input is needed §1. Prior spatial VQA data (SpatialVLM) is not public, and no public VLM benchmark measures metric distance or size between objects §4.1. Prior work also relies on language references to objects, which is ambiguous when several similar objects are present §1.

Method

The method has two parts: a data pipeline that turns single images into 3D scene graphs and then QA, and a model that takes region masks or boxes plus an optional depth map §3.

3.1 Scene graph from a single image §3.1

1. Filter images. A CLIP-based zero-shot classifier labels each image with four positive labels (DSLR or iPhone photos of indoor or outdoor scenes) and six negative labels (close-up of an object, product shot, painting, collage, GUI screenshot, text). On OpenImages, 700K of 1.7M samples were removed app. F.1.
2. Find the object classes with an open-vocabulary image tagging model, ground each class to a 2D box with GroundingDINO, and refine each box to a mask with a segmentation model (SAM-HQ) §3.1.
3. Estimate the camera intrinsics with WildCamera, and metric depth with Metric3Dv2 that takes the focal length as input §3.1 app. F.2.
4. Estimate pitch and roll with Perspective Fields. Build a rotation from them and rotate each point cloud into a canonical frame where the ground is level §3.1 app. F.3. The paper says this avoids surface segmentation and RANSAC used in SpatialVLM §3.1.
5. Denoise each object's points: remove statistical outliers, voxel-downsample with a scale-dependent voxel size, run DBSCAN, and drop the object if fewer than ten points remain app. F.3.
6. Build one axis-aligned 3D box per object. Its width and height in metres are node attributes §3.1.
7. Write the edges: relative relations (left, right, above, below, behind, front, wide, thin, tall, short, big, small) and metric relations (direction, direct, horizontal and vertical distance) §3.1.

3.2 Scene graph to training QA §3.2

1. Template QA: each attribute and relation becomes a question with several phrasings and true and false answer variants, with objects named as Region [X] §3.2 app. F.4.
2. LLM QA: a text description of the scene graph, with region tags, is given to Llama3-70B, which writes complex spatial questions and answers §3.2 app. F.5.
3. The resulting Open Spatial Dataset (OSD) contains 1M images, 5M regions with boxes and masks, 8M template QAs and 700K LLM QAs §3.2.

3.3 Model §3.3 app. G

The model uses a visual encoder, a region-feature extractor, linear connectors and a LLaMA2-7B language model §3.3.

• The RGB image is encoded once. Two deconvolution layers upscale the feature map, and mask pooling averages the features inside each region mask app. G.2.
• Relative depth from an off-the-shelf model (Depth Anything) is encoded by the same visual encoder. A separate depth-to-language connector projects it, and its weights are initialized from the RGB connector and trained on spatial QA only §3.3.
• Region prompts are placeholders <region> and <depth> in the text. Each one is replaced by the region's RGB embedding and depth embedding. All tokens are interleaved and fed to the LLM §3.3.

# Build LLM input embeddings for one image and its regions  [§3.3] [app. G]
# Input:  image (3, H, W); depth (3, H, W) or None; masks (M, H, W)
# Output: interleaved token embeddings for the LLM

def build_llm_inputs(image, depth, masks, text_tokens):
    # 1. Encode the RGB image into patch features  [app. G.1]
    rgb_features = visual_encoder(image)                  # (N, HW, C)
    # 2. Upscale the features with two deconvolution layers  [app. G.2]
    local_features = refine_deconv(rgb_features)
    # 3. Average the features inside each region mask  [app. G.2]
    region_rgb = mask_average(local_features, masks)      # (M, C)
    # 4. Project RGB region features into the language space  [§3.3]
    region_rgb = rgb_connector(region_rgb)
    if depth is not None:
        # 5. Encode the depth map with the same visual encoder  [§3.3]
        depth_features = visual_encoder(depth)
        # 6. Average depth features per mask, then project them  [§3.3]
        region_depth = depth_connector(mask_average(depth_features, masks))
    # 7. Replace each <region> and <depth> placeholder  [§3.3]
    # 8. Interleave image, region and depth tokens with the text  [§3.3]
    return interleave(image_tokens, region_rgb, region_depth, text_tokens)

3.4 Training and inference §3.4 app. H

Training has three stages: connector alignment, visual-language pretraining, and visual instruction tuning, following VILA §3.4. In stage 1 the RGB connector is trained on CC3M captions. In stage 2 the paper says the RGB connector and language model are frozen and only the depth connector is trained, on OSD §3.4. Stage 3 fine-tunes on an instruction mixture that includes OSD, the LLaVA instruction data and region-level data §3.4 app. H.1. Region inputs are sampled randomly as box or mask during training. At inference, masks are used when available; otherwise the box is converted to a mask with SAM §3.4.

Hyperparameters

Items marked "not stated" are absent from the paper. Code values come from the official repository at code_commit and are not the paper's values.

Compute detail from app. I: filtering 4 h; calibration and depth 4 h each; detection and segmentation 8 h; LLM QA synthesis 12 h with Llama3-70B. Stage 1 (connector initialization) takes 4 h, visual-language pretraining 30 h on 16 nodes. Depth-connector pretraining takes 4 h and instruction tuning 12 h, both on 2 nodes.

Results

Author-reported. Setup: SpatialRGPT-Bench, 657 qualitative and 749 quantitative VQA pairs from Omni3D cuboids across 88 classes §4.1. Baselines: GPT-4 (blind, class name prepended), GPT-4V and LLaVA-v1.6-34B, region-aware VLMs (GPT-4V+SoM, LLaVA+SoM, KOSMOS-2, RegionVILA-7B) §4.1 tab. 1. Judge: GPT-4 app. J.

SpatialRGPT-Bench tab. 1. Qualitative rows are accuracy in %. Quantitative rows are success rate in % / absolute relative error.

Region-level classification (COCO-2017 val, ground-truth boxes, mAP / accuracy %) tab. 3: RegionGPT-7B 70.0 / 80.6; SpatialRGPT-7B 69.7 / 79.9; SpatialRGPT-VILA-1.5-3B 72.5 / 82.5; SpatialRGPT-VILA-1.5-8B 72.9 / 82.9.

BLINK relative depth (accuracy %) tab. 4: GPT-4V-Turbo 66.9; GPT-4o 64.5; LLaVA-v1.6-34B 64.5; SpatialRGPT-7B 82.3; SpatialRGPT-VILA-1.5-8B 87.9.

General VQA (VILA-1.5-3B vs SpatialRGPT-VILA-1.5-3B) tab. 2: VQAv2 80.4 → 81.1; GQA 61.5 → 62.3; MMVet 35.4 → 38.2; MMBench 63.4 → 65.6; but LLaVA-Bench 75.9 → 71.5, MME 1442 → 1424, POPE 85.9 → 85.5, MMMU (val) 33.3 → 33.0.

Ablations: augmenting the benchmark's questions with GPT-4 keeps SpatialRGPT-7B ahead of GPT-4V-Turbo (90.0 vs 60.5 qualitative average) app. A. Adding width and height data helps size questions and slightly hurts horizontal and vertical distance questions app. B. Axis-aligned boxes have lower width and height MSE against Omni3D than PCA-oriented boxes (8.27 vs 17.09 and 2.35 vs 4.83) app. C. Box inputs score about 0.3 points lower on average than mask inputs (91.47 vs 91.78) app. D.

Paper vs code

Code: AnjieCheng/SpatialRGPT at 16715d4f1419997da18926c6ce574802d1eb3a37, Apache-2.0, linked from the project page and owned by the first author. Shell scripts cover LLaMA2-7B, LLaMA3-8B and Sheared-3B. The discrepancies below are for the LLaMA2-7B scripts unless stated.

• Visual encoder. The paper says CLIP-L at 336 px app. G.1. The scripts use google/siglip-so400m-patch14-384 1_mm_align.sh 2_pretrain.sh.
• Image connector. The paper says linear connectors app. G.3. The scripts pass --mm_projector mlp_downsample 1_mm_align.sh. The region-level RGB and depth projectors are nn.Linear base_extractor.py.
• Depth connector initialization. The paper says the depth connector is initialized from the RGB connector §3.3. The depth projector is constructed as a fresh nn.Linear base_extractor.py. No code that copies RGB weights into it was found in llava/.
• Depth refinement. The paper says depth has a separate feature refinement module app. G.2. The code applies feature_refinement only to the RGB features llava_arch.py. Depth features go to the region extractor without refinement llava_arch.py.
• Stage 2 freezing. The paper says only the depth connector trains in stage 2, with RGB connector and LLM frozen §3.4. The script sets --tune_language_model True and --tune_mm_projector True 2_pretrain.sh.
• Stage 3 vision tower. The paper does not say whether the encoder is tuned. The stage-3 script sets --tune_vision_tower True 3_sft.sh.
• Learning rates and batch sizes. Paper: depth-connector pretraining 1e-4 at batch 32; SFT 5e-5 at batch 16 app. H.2. Code: stage 2 uses 5e-5 at batch 16 2_pretrain.sh; stage 3 uses 2e-5 at batch 16 with gradient accumulation 2 3_sft.sh 3_sft.sh.
• Stage 1 data. The paper says CC3M §3.4. The script uses the mixture named llava_1_5_mm_align 1_mm_align.sh. Its contents were not checked.
• Judge model. The paper says GPT-4 app. J. The evaluation script calls gpt-4-turbo evaluate_spatial_with_gpt4.py.
• Success criterion. Paper: within ±25% §4.1. Code: between 0.75× and 1.25× the answer evaluate_spatial_with_gpt4.py. These agree; both are relative to the ground truth.
• Global image tokens (code only). When regions are enabled, the global image features are adaptive-pooled to 27×27 before the image projector base_extractor.py llava_arch.py. The paper does not describe this.
• Calibration. The paper uses Perspective Fields §3.1. The default config does too v2.py. A GeoCalib variant is also in the code v2_geocalib.py pointcloud.py.
• Voxel and denoising. The paper's listing matches the code pointcloud.py. The config value downsample_voxel_size = 0.025 v2.py is not what the listing uses; the config value was not traced further.

Other paper claims not confirmed by the code: the paper says SpatialRGPT beats the region baselines on classification §4.2. In tab. 3 RegionGPT-7B is higher than SpatialRGPT-7B on both mAP and accuracy; only the VILA variants are higher.

No discrepancies found in: metric depth model (Metric3Dv2 ViT-giant2) metric3d_v2.py, WildCamera intrinsics pointcloud.py, the denoising parameters app. F.3, and the LLaMA2-7B language model.

Pitfalls

• Stage 2 is not what the paper says. The code trains the LLM and projector in stage 2. An implementer following the paper's frozen-LLM depth pretraining will get a different model; follow the code or the paper, and say which.
• Depth connector start point. Random initialization (code) versus RGB initialization (paper) changes early training. Neither run is reported.
• Depth path differs from RGB. Depth skips the refinement module in code. The paper's G.2 description would add a second refinement module.
• Abstract name. The abstract and the bench name differ in spelling ("SpatialRGBT-Bench" in the abstract, "SpatialRGPT-Bench" in the body).
• Width and height success is below GPT-4V. SpatialRGPT-7B has lower success for width (49.6 vs 51.1) and height (57.9 vs 68.4) tab. 1. Its error is lower. The text's claim of better quantitative performance holds for error, not every success rate.
• Errors use answered questions only. Relative error excludes responses without a number §4.1. Success rate counts all questions. Compare both columns.
• BLINK and the 20-point claim. The text says over 20% gain over GPT-4V-Turbo §4.2. That holds for the 8B model (87.9 vs 66.9). The 7B model is 82.3 (+15.4).
• Judge dependence. Scores come from GPT-4 judging answers and converting units app. J. The judge model is not pinned in the paper; the code uses gpt-4-turbo. Rerunning with another judge may move the numbers.
• Axis-aligned boxes. Object width and height come from axis-aligned boxes after canonicalization. Labels can be wrong for partly elevated objects §5 app. K. Relations in the QA inherit this error.
• Canonicalization uses pitch and roll only. The paper states this app. F.3. Yaw is not used.
• Metric scale needs the right intrinsics. Depth is metric only with the calibrated focal length; the paper uses WildCamera for it §3.1.
• Filter labels. The CLIP negative labels were set for OpenImages by the authors app. F.1. Other sources will need new labels.
• Dataset and benchmark licenses. The paper's licenses appendix lists OpenImages as Apache-2.0 and cites Unsplash photos app. M. This card does not check the OSD or benchmark dataset license; the dataset was not fetched.
• Scene-graph data uses an LLM. The LLM-written QAs come from Llama3-70B §3.2. Reproducing them needs that model and its prompts app. F.5.

Lineage

• Builds on: RegionGPT for region features and the region-level SFT blend §3.3 app. H.1; VILA for the three-stage training recipe §3.4; Depth Anything for relative depth §3.3. Not linked to wiki cards.
• Compared against: GPT-4, GPT-4V, LLaVA-v1.6-34B, KOSMOS-2, RegionVILA-7B and SpatialVLM as the benchmark's predecessor §4.1 tab. 1.
• Superseded by: not stated in this card's sources.