--- kind: concept slug: visual-grounding title: Visual grounding summary: >- Tying words to a place a system can act on. Shikra writes boxes as text (87.01 on RefCOCO val). TransVG regresses one box (81.02). GLaMM attaches a mask to each phrase. LM-Nav ranks map nodes. ScanQA ranks 3D proposals. GeoChat writes aerial boxes. Words2Contact picks a contact point. SpatialRGPT takes regions as input. RoboSpatial trains on free-space points. Figures are author-reported. tags: [vision-language, grounding, referring-expression, segmentation, multimodal-llm, navigation, 3d-scene-understanding, remote-sensing, robotics, contact-planning, spatial-reasoning, region-grounding] papers: ["2306.15195", "2104.08541", "2311.03356", "2207.04429", "2112.10482", "2311.15826", "2407.14229", "2406.01584", "2411.16537"] related: [multimodal-language-model, visual-instruction-tuning, object-detection, referring-segmentation, vision-language-pretraining, zero-shot-learning, topological-map-navigation, multimodal-fusion, low-rank-adaptation, visual-question-answering-evaluation, task-decomposition, human-in-the-loop, constrained-decoding, 3d-scene-graph, monocular-depth-estimation, scene-graph-generation] status: draft updated_at: 2026-10-31 --- ## Overview Visual grounding asks a model or system to tie words to a place it can point at. The carded work uses five forms of place that a system outputs. A multimodal LLM can write the location as a box or point in the image [arxiv:2306.15195 §5.2]. A transformer can regress a box directly from a learned token [arxiv:2104.08541 §3.2]. A chat model can attach a mask to each object phrase it names [arxiv:2311.03356 §3.1]. A system can choose which of several candidate places a phrase matches, such as stored images at the nodes of a map [arxiv:2207.04429 §4.3]. Or it can compute a single pixel point from a segmentation heatmap or from arithmetic over detector boxes, then let a person correct it [arxiv:2407.14229 §3.3]. Two more cards use a place differently. SpatialRGPT takes a box or mask from the user as input and reasons about that region, so the model names no place itself [arxiv:2406.01584 §3.3]. RoboSpatial trains models to give free-space points that satisfy a relation to an anchor object [arxiv:2411.16537 §3.1]. A model that only captions an image can do none of these, because its answers carry no coordinates, masks or choice of place [arxiv:2306.15195 §1]. The shared core is that the place must be something the system can act on: a set of coordinates in the output text, a regressed box, a mask, a computed point, or a candidate that a scorer ranks [arxiv:2306.15195 §5.2] [arxiv:2104.08541 §3.2] [arxiv:2311.03356 §3.1] [arxiv:2207.04429 §4.3] [arxiv:2407.14229 §3.3]. Shikra argues that coordinates can be ordinary text, not special tokens [arxiv:2306.15195 §2] [arxiv:2306.15195 §5.2]. The scope is limited to the nine carded papers. Other grounding designs, such as dedicated coordinate vocabularies, detection heads, or the two-stage proposal pipelines that TransVG compares against, are named in the related work but are not described here [arxiv:2306.15195 §2] [arxiv:2104.08541 §1]. ## Variants ### Coordinates as text in a multimodal LLM The image goes through a frozen CLIP ViT-L/14 encoder and one trainable linear layer into 256 visual tokens. Boxes are written as `[x_min,y_min,x_max,y_max]` and points as `[x_center,y_center]`, normalised by image size and kept to three decimals. The language model reads and writes these like any other words [arxiv:2306.15195 §5.1] [arxiv:2306.15195 §5.2]. Card: [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](/papers/2306.15195). Training runs in two stages on public vision-language data. The second stage raises the sampling ratio of instruction data and of 5,922 generated referential-dialogue pairs to 50% [arxiv:2306.15195 §5.3] [arxiv:2306.15195 §5.4]. The generated pairs come from GPT-4, which received image captions and the boxes of the mentioned objects, not the images [arxiv:2306.15195 §5.3.2]. Reported results, all author-reported: - Standard referring-expression comprehension (REC) on RefCOCO val: 87.01 for Shikra-7B and 87.83 for Shikra-13B. The caption does not define the metric [arxiv:2306.15195 tab. 3]. - Visual-7W "which box" point questions: 85.33 [arxiv:2306.15195 tab. 4]. - A toy comparison with numbers versus extra coordinate tokens: numbers score higher on every split tested [arxiv:2306.15195 tab. 2]. ### Box regression from a learned token A ResNet backbone turns the image into visual tokens, which pass through six transformer encoder layers. The phrase goes through the first 12 layers of BERT-base. One learnable [REG] token joins the two sequences, and six fusion layers run over the joint sequence, so every token can attend to every other [arxiv:2104.08541 §3.2]. The output at [REG] goes through a small MLP to four numbers, (x, y, w, h). No candidate boxes or anchors are used [arxiv:2104.08541 §1] [arxiv:2104.08541 §3.2]. Training uses smooth-L1 plus a GIoU term with weight 1 [arxiv:2104.08541 §3.3]. Card: [TransVG: End-to-End Visual Grounding with Transformers](/papers/2104.08541). Reported results, author-reported, as top-1 accuracy at IoU 0.5: - RefCOCO val: 80.32 with ResNet-50 and 81.02 with ResNet-101. RefCOCO testB: 78.12 and 78.35 [arxiv:2104.08541 tab. 2]. - ReferItGame test: 69.76 and 70.73. Flickr30K test: 78.47 and 79.10 [arxiv:2104.08541 tab. 1]. - The learned starting vector for [REG] scores highest on RefCOCO val. The five alternatives tested reach at most 79.12 [arxiv:2104.08541 tab. 3]. ### Masks for each phrase in a chat model A CLIP image encoder and a 7B LLM write the answer in text [arxiv:2311.03356 §3.1] [arxiv:2311.03356 app. A.2]. When the model names an object, it writes a `` token after it. The last-layer state at that token goes through a projection into a SAM-style pixel decoder, which outputs one mask for it [arxiv:2311.03356 §3.1]. Answers follow the grounded-conversation format, in which each grounded phrase is wrapped in `

` tags and followed by `` [arxiv:2311.03356 §3.2]. The user can also mark boxes, which become region inputs [arxiv:2311.03356 §3.1]. The image encoders stay frozen, the LLM is tuned with LoRA, and the region encoder, projections and pixel decoder are trained [arxiv:2311.03356 app. A.2]. Card: [GLaMM: Pixel Grounding Large Multimodal Model](/papers/2311.03356). ### Boxes as text with task tokens for aerial images GeoChat is a LLaVA-1.5-style chat model tuned on remote-sensing images. It writes a box as text: four coordinates normalised to [0, 100] and a rotation angle θ measured from the lower edge. Region boxes in the prompt use the same form [arxiv:2311.15826 §3.1]. Three task tokens select the behaviour. `[grounding]` describes the image and gives boxes for the objects it names, `[identify]` describes the object inside a given box, and `[refer]` locates the object named in a `

` phrase [arxiv:2311.15826 §3.1] [arxiv:2311.15826 tab. 1]. The CLIP encoder and the adaptor stay frozen, and LoRA trains the LLM [arxiv:2311.15826 §3.2]. Card: [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](/papers/2311.15826). The training pairs are generated from object annotations in aerial detection sets. A text-only language model writes the questions and answers from those annotations, without seeing the images [arxiv:2311.15826 §4]. Reported results, author-reported: - On the authors' own grounding benchmark, accuracy at IoU 0.5 is 10.6 overall, against 7.6 for MiniGPT-v2 [arxiv:2311.15826 tab. 7]. The benchmark uses the same pipeline as the training data, so these numbers are in-distribution [arxiv:2311.15826 §5.4]. Reported results, author-reported: - Grounded conversation generation, test split: METEOR 15.8, CIDEr 43.5, mask AP50 29.2, mean IoU 65.6 and recall 40.8. Trained on the paper's GranD data and finetuned on GranD_f. Excluding the 1K human-annotated images lowers every metric [arxiv:2311.03356 tab. 3]. - Referring-expression segmentation on refCOCO val: 79.5. The caption calls the table "qualitative" and does not name the metric [arxiv:2311.03356 tab. 4]. ### Scoring stored places with CLIP and a route search A language model first turns a free-form instruction into an ordered list of landmark phrases. A frozen CLIP then scores each phrase against the stored images at every node of a topological map, and the scores are normalised over the nodes. A graph search picks the walk that visits the landmarks in order while penalising travel distance [arxiv:2207.04429 §4.2] [arxiv:2207.04429 §4.3] [arxiv:2207.04429 §4.4]. Nothing here outputs a box or point. The output is a node on the map, and the robot navigates there with a separate model [arxiv:2207.04429 §3]. Card: [LM-Nav](/papers/2207.04429). The authors use the model with no fine-tuning and no language-labelled robot data [arxiv:2207.04429 §1]. The official code differs from the paper in how CLIP similarities are used. It takes raw cosine similarity with no softmax, and the best image at each node stands for that node [arxiv:2207.04429 §4.3] (see Contradictions). Reported results, author-reported on a Clearpath Jackal in outdoor suburban environments: - 0.8 net success on 20 instructions in each of two environments, with 0 and 0.1 disengagements per experiment [arxiv:2207.04429 tab. 1]. - Landmark detection by CLIP, on 100 images and 30 landmarks, succeeds when one of the top three predictions matches. CLIP scores 0.87, against 0.38 for ViLD and 0.07 for Faster-RCNN [arxiv:2207.04429 tab. 3]. - The authors name landmark recognition as the main failure point, for example a fire hydrant [arxiv:2207.04429 §5.2]. ### Answer and object box in a 3D scan ScanQA answers a free-form question about an indoor 3D scan and points to the objects the question refers to [arxiv:2112.10482 §3.1]. VoteNet on a PointNet++ backbone proposes 256 object boxes from the scan, and a GloVe and LSTM encoder reads the question. Transformer layers let question words and proposals attend to each other [arxiv:2112.10482 §4]. A localization head scores each proposal against the question with cross-entropy. An answer head, an object-class head and VoteNet's detection loss train in the same run [arxiv:2112.10482 §4]. Card: [ScanQA: 3D Question Answering for Spatial Scene Understanding](/papers/2112.10482). The card records the official code as CC BY-NC-SA 3.0, which is non-commercial, so this page quotes no code from it. The card also lists several differences between the paper and the code, including the localization target. See the [ScanQA card](/papers/2112.10482), paper-vs-code section. Reported results, author-reported: - Exact match at 1 on the test split with objects: 23.45. The best 2D baseline, RandomImage+MCAN, scores 22.31 [arxiv:2112.10482 tab. 3]. On the split without objects, EM@1 is 20.90 [arxiv:2112.10482 tab. 3]. - In the component ablation on the split with objects, adding the localization head to the answer and object-class heads raises EM@1 from 18.31 to 23.45 [arxiv:2112.10482 tab. 4]. ### Pixel point from a heatmap or box arithmetic, with human correction A humanoid receives a spoken instruction and must pick the point on its camera image where a hand or foot should make support contact. An LLM first sorts the instruction into one of three steps: predict a point, correct it, or confirm it [arxiv:2407.14229 §3.2]. For a named object, a language-grounded segmenter produces a heatmap, and the point is its maximum [arxiv:2407.14229 §3.3]. For a place defined relative to objects, an open-set detector returns boxes. The LLM writes an arithmetic expression for x and one for y over the box values, and separate code evaluates them. The paper reports that this beat having the LLM compute the number itself by about 10% in preliminary experiments [arxiv:2407.14229 §3.3]. A correction such as "a bit to the right" can name no object, and the operator confirms the final point before the controller moves a hand there [arxiv:2407.14229 §3.4] [arxiv:2407.14229 §3.5]. Card: [Words2Contact](/papers/2407.14229). Reported results, author-reported, on 78 image, prompt and mask tuples at 1280 × 720 [arxiv:2407.14229 §4.1]: - Median overall success is 0.61 for GPT-3.5 with GroundingDINO and CLIPSeg, against 0.13 for random points. Its absolute and relative medians are 0.71 and 0.50 [arxiv:2407.14229 tab. 1]. - A pilot with 11 volunteers gives a median distance of 21 px at the fifth prompt, against an 18 px target radius [arxiv:2407.14229 §4.2]. - On the Talos robot, four reaching settings all succeeded. The text gives no trial count [arxiv:2407.14229 §4.3]. ### Region prompts as input to a spatial VLM SpatialRGPT is a LLaMA2-7B vision-language model. The user supplies the region as a box or mask, and the model answers about it. The image is encoded once, the features are upscaled, and each region's features are averaged over its mask. Those features replace a `` placeholder in the text [arxiv:2406.01584 §3.3] [arxiv:2406.01584 app. G.2]. An optional relative depth map, from an off-the-shelf model, goes through the same encoder and a separate connector [arxiv:2406.01584 §3.3]. No coordinate is written, so this variant does not output a place; it ties a named region to the answer. Training data is 1M OpenImages photos turned into 3D scene graphs, then into template and LLM-written questions [arxiv:2406.01584 §3.1] [arxiv:2406.01584 §3.2]. Training has three stages [arxiv:2406.01584 §3.4]. Card: [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](/papers/2406.01584). Reported results, author-reported, with answers judged by GPT-4: - On SpatialRGPT-Bench, the 7B model's qualitative average is 91.78, against 58.14 for GPT-4V [arxiv:2406.01584 tab. 1]. - BLINK relative depth: 82.3 for the 7B model, against 66.9 for GPT-4V-Turbo [arxiv:2406.01584 tab. 4]. - On the general VQA check, the 3B model gains on some benchmarks and loses on others. LLaVA-Bench falls from 75.9 to 71.5 [arxiv:2406.01584 tab. 2]. ### Spatial relations and free-space points as training data RoboSpatial is a generator of spatial questions, not a new output head. It labels relations between oriented 3D boxes from indoor and tabletop scans, in ego, world and object frames [arxiv:2411.16537 §3.1] [arxiv:2411.16537 §3.2.1]. Three question types come from those labels. Context questions ask for a list of 2D image points in free space that satisfy a direction to an anchor. Compatibility questions ask whether a virtual box of the target's size fits, and configuration questions ask whether a relation holds between two objects. Both of the last two are True or False [arxiv:2411.16537 §3.1] [arxiv:2411.16537 §3.2.2]. Training uses 900,000 relations, not the 3M full set [arxiv:2411.16537 app. B.1]. Card: [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](/papers/2411.16537). Reported results, author-reported, on the held-out split of scans: - LLaVA-NeXT goes from 30.3 to 60.5 in total accuracy after fine-tuning. Its indoor context score goes from 12.5 to 41.3 [arxiv:2411.16537 tab. 3]. - On the authors' real-robot task suite, LLaVA-NeXT goes from 23.7% to 52.6%, and RoboPoint from 44.7% to 46.2% [arxiv:2411.16537 tab. 6]. - For LLaVA-NeXT, the average accuracy rises with the number of relations used: 38.1 at 100K, 60.5 at the 900K default and 72.4 at the full 3M [arxiv:2411.16537 tab. 9]. The official generator and its repository carry a non-commercial licence. This variant takes nothing from them; see the RoboSpatial card, which states the licence in its header. ## When to use which The table compares the carded designs with the alternatives their papers report. Each row states what the cited numbers support. | Situation | Use | Why | |---|---|---| | You want a box or point in one image, from an existing multimodal LLM with no new vocabulary, detector or position encoder to build | Coordinates as text | The paper adds none of these and gets 87.01 on RefCOCO val with a 7B model [arxiv:2306.15195 §5.1] [arxiv:2306.15195 tab. 3] | | You are choosing how to encode coordinates in a new model | Plain numbers over new coordinate tokens | In the toy setting of [arxiv:2306.15195 tab. 2], numbers win on all eight splits, but the authors note they need more tokens per box [arxiv:2306.15195 §6.2] | | Many objects per image, and output length or latency matters | Budget for the extra tokens | Numbers need more tokens per box than a single coordinate token would. The paper notes this cost for dense scenes [arxiv:2306.15195 §6.2] | | You need one box per phrase, no LLM at inference, and you can train on a grounding dataset | Box regression from a learned token | No candidates or anchors, and one forward pass [arxiv:2104.08541 §3.2]. Batch size is 64 and hardware is not stated [arxiv:2104.08541 §4.2]. Reported 81.02 on RefCOCO val with ResNet-101 [arxiv:2104.08541 tab. 2] | | You need a mask per named object inside a conversation | Masks through `` tokens | The only design on this page that outputs pixel masks in a chat answer [arxiv:2311.03356 §3.1]. Finetuning used 8 A100-40GB GPUs [arxiv:2311.03356 app. A.3]. The finetuning set GranD_f repurposes Flickr30k, RefCOCOg and PSG data, so check upstream terms before commercial use [arxiv:2311.03356 §4.5] [arxiv:2311.03356 app. A.4] | | Benchmark accuracy on RefCOCO is the main goal, and a fine-tuned specialist is acceptable | Specialist models | The specialist rows reach 90.56 to 92.64 on RefCOCO val, above Shikra's 87.01 [arxiv:2306.15195 tab. 3]. TransVG's 81.02 on the same split is below Shikra's figure [arxiv:2104.08541 tab. 2] | | Dense detection or segmentation | Not Shikra | The authors list dense detection and segmentation as unsuitable for their model [arxiv:2306.15195 §7]. For masks inside a chat answer, see the chat-model variant [arxiv:2311.03356 §3.1] | | The answer is one of a set of known places (a map, stored views), and no box or point is needed | Score candidates with CLIP, then search | The paper needs no fine-tuning or language-labelled robot data and reports 0.8 net success on 20 outdoor instructions [arxiv:2207.04429 §1] [arxiv:2207.04429 tab. 1]. The numbers come from one robot and 20 instructions | | Your landmarks are rare or hard to recognise | Expect failures; check detection first | CLIP's 0.87 detection rate is the best of the three tested, but the authors' own test set is small and the metric is top-3 [arxiv:2207.04429 tab. 3]. Fire hydrants were a failure case [arxiv:2207.04429 §5.2] | | The scene is a 3D indoor scan, and the answer must name the object it refers to | Proposal scoring with an answer head (ScanQA) | The only 3D design on this page. Its 2D baseline reaches 22.31 EM@1 with objects [arxiv:2112.10482 tab. 3], and its localization head raises EM@1 from 18.31 to 23.45 in the ablation [arxiv:2112.10482 tab. 4]. Its code is non-commercial (see the variant) | | The scene is aerial or satellite imagery, and you need a box in a chat answer | Boxes as text with task tokens (GeoChat) | The only remote-sensing design on this page. Its authors' benchmark gives 10.6 acc@0.5 against 7.6 for MiniGPT-v2 [arxiv:2311.15826 tab. 7], but that benchmark is in-distribution with the training data [arxiv:2311.15826 §5.4] | | The output is one contact point on a robot's camera image, chosen from a spoken instruction, and a person can correct it before the robot moves | Heatmap or box arithmetic, with a correction loop (Words2Contact) | The paper reports a median overall success of 0.61 on its 78-sample set, against 0.13 for random points [arxiv:2407.14229 tab. 1]. The set is small and from one robot, and the paper does not define success beyond the acceptable-contact masks [arxiv:2407.14229 §4.1] | | You need a model to answer about a region the user marks, such as a left/right or distance question about one object, and you can supply the box or mask at inference | Region prompts as input (SpatialRGPT) | The 7B model scores 91.78 on its own qualitative benchmark, against 58.14 for GPT-4V, judged by GPT-4 [arxiv:2406.01584 tab. 1]. Its width and height success rates are below GPT-4V's, so do not assume a lead on every relation [arxiv:2406.01584 tab. 1] | | You need a model to answer relation or fit questions (left, right, above, front, whether an object fits) and you can build or license 3D scans to generate training data | Spatial relation data (RoboSpatial) | Fine-tuning lifts LLaVA-NeXT from 30.3 to 60.5 on the authors' held-out split [arxiv:2411.16537 tab. 3]. The source scans each have their own licence [arxiv:2411.16537 §4.1], and the official generator is non-commercial (see the RoboSpatial card) | ## Implementation sketch Five sketches, covering five of the nine variants. They share no code, only the idea that the output is text coordinates, a regressed box, a mask, a computed point or a ranked candidate. The first also serves GeoChat's text boxes, the fourth is the same scoring idea as ScanQA's localization head (see Notes), and the fifth is the point step of Words2Contact. The first sketch covers coordinates as text. It follows the format in [arxiv:2306.15195 §5.2] and the normalisation described there. Padding and other preprocessing details that the paper does not state are left out. ```pseudocode # Write a box as text for the prompt or the target [arxiv:2306.15195 §5.2] def box_to_text(box_px, width, height): # Normalise by image size, clamp to [0, 1], keep three decimals x1, y1, x2, y2 = box_px norm = [clamp(x1 / width), clamp(y1 / height), clamp(x2 / width), clamp(y2 / height)] return "[" + ",".join(f"{v:.3f}" for v in norm) + "]" # Read a box back from generated text [arxiv:2306.15195 §5.2] def text_to_box(text, width, height): # Expects four comma-separated numbers in brackets x1, y1, x2, y2 = parse_numbers(text) return (x1 * width, y1 * height, x2 * width, y2 * height) ``` The second sketch covers box regression. The order of the three token groups and the loss weight are the paper's [arxiv:2104.08541 eq. 4] [arxiv:2104.08541 §3.3]. ```pseudocode # Visual, language, then one learned [REG] token [arxiv:2104.08541 eq. 4] def transvg_box(visual_tokens, word_tokens, reg_token, fusion_layers, head): joint = concat(visual_tokens, word_tokens, reg_token) # Every token attends to every other token [arxiv:2104.08541 §3.2] for layer in fusion_layers: joint = layer(joint) # Four numbers (x, y, w, h) from the [REG] output [arxiv:2104.08541 §3.2] return head(joint[-1]) # Smooth-L1 plus GIoU, weight 1 [arxiv:2104.08541 eq. 5] def transvg_loss(pred_box, gt_box): return smooth_l1(pred_box, gt_box) + 1.0 * giou_loss(pred_box, gt_box) ``` The third sketch covers masks from `` tokens. It follows the paper's description of the pixel path [arxiv:2311.03356 §3.1]. ```pseudocode # One mask per token the LLM emits [arxiv:2311.03356 §3.1] def seg_masks(hidden_states, seg_positions, sam_features, projector, decoder): # Last-layer state at each position [arxiv:2311.03356 §3.1] seg_states = hidden_states[seg_positions] # Project into the decoder's prompt space [arxiv:2311.03356 §3.1] prompts = projector(seg_states) # One decoder pass per prompt, on the SAM features [arxiv:2311.03356 §3.1] return [decoder(sam_features, p) for p in prompts] ``` The fourth sketch covers scoring candidate places. The prompt wording and the normalisation are the paper's [arxiv:2207.04429 §4.3]. ```pseudocode # Probability that each map node matches one landmark [arxiv:2207.04429 §4.3] def node_probs(landmark, node_images): # One fixed prompt, no ensembling [arxiv:2207.04429 app. A.2] text = clip_encode_text("This is a photo of " + landmark) # One CLIP logit per node image logits = [clip_logit(text, img) for img in node_images] # Softmax over all nodes for this landmark return softmax(logits) ``` The fifth sketch covers the point step. It follows the absolute and relative split in the paper's prediction module [arxiv:2407.14229 §3.3]. ```pseudocode # One pixel point from a heatmap or from box arithmetic # [arxiv:2407.14229 §3.3] def contact_point(instruction, analysis, image): # Named object: the heatmap's maximum is the point # [arxiv:2407.14229 §3.3] if analysis.position == "absolute": heatmap = segment_by_text(image, analysis.objects) row, col = argmax_2d(heatmap) return col, row # Relative place: detector boxes, then the LLM writes two formulas # [arxiv:2407.14229 §3.3] boxes = detect_by_text(image, analysis.objects) formula_x, formula_y = write_expressions(instruction, boxes) # The LLM writes the formulas; a separate parser computes the numbers return evaluate(formula_x), evaluate(formula_y) ``` Notes: - The paper's own example writes a space after each comma. Decide which form your targets and parser use, and keep them the same [arxiv:2306.15195 §5.2]. - Points use two numbers, not four [arxiv:2306.15195 §5.2]. - Inference is one forward pass, with no post-processing of the box [arxiv:2104.08541 §4.2]. - The released code uses "A photo of " and raw cosine similarity, not a softmax. Keep whichever you use consistent when reproducing the numbers [arxiv:2207.04429 §4.3] (see Contradictions). - GeoChat writes coordinates in [0, 100] and adds a rotation angle to boxes [arxiv:2311.15826 §3.1]. To reuse the first sketch, change the scale and parse five values. The axis used for normalisation is not stated, so the mapping cannot be checked against the paper. - ScanQA's localization head is a softmax over the 256 proposals, trained with cross-entropy against one target [arxiv:2112.10482 §4]. The code's target differs from the paper's; see the ScanQA variant. - Words2Contact does not state the image frame or the origin of its point coordinates [arxiv:2407.14229 §3.3]. A controller reading the point must agree with the image's axis direction, so check that before you use it. ## Contradictions Across the carded papers the designs report different tasks and metrics, so their numbers are not comparable. The RefCOCO val figures for Shikra (87.01, box text), TransVG (81.02, box accuracy at IoU 0.5) and GLaMM (79.5, mask metric not named) measure different outputs. ScanQA (EM@1 on 3D questions), GeoChat (acc@0.5 on its own aerial benchmark) and Words2Contact (median success on a contact task) measure different things again. SpatialRGPT (GPT-4-judged accuracy on its own benchmark) and RoboSpatial (accuracy on its held-out split, built by the same pipeline as its training data) add two more. Do not rank any of them against each other [arxiv:2306.15195 tab. 3] [arxiv:2104.08541 §4.3] [arxiv:2311.03356 tab. 4] [arxiv:2112.10482 tab. 3] [arxiv:2311.15826 tab. 7] [arxiv:2407.14229 tab. 1] [arxiv:2406.01584 tab. 1] [arxiv:2411.16537 tab. 3]. - **Coordinate scale in text boxes.** Shikra normalises coordinates by image size and keeps three decimals [arxiv:2306.15195 §5.2]. GeoChat writes values in [0, 100] and does not say which image axis each one is divided by [arxiv:2311.15826 §3.1]. A prompt or parser written for one format will misread the other. - **TransVG loss.** The paper trains with smooth-L1 plus a GIoU term [arxiv:2104.08541 §3.3] [arxiv:2104.08541 eq. 5]. The released code uses plain L1 plus 1 minus GIoU, and the paper does not say which loss the released weights used. See the [TransVG card](/papers/2104.08541), paper-vs-code section. - **TransVG released versus reported.** The paper reports 69.76 on ReferItGame test for ResNet-50 [arxiv:2104.08541 tab. 1]. The released README lists 69.3 for the same setting. The released weights give 76.94 on RefCOCO testB for ResNet-101, against 78.35 in the paper [arxiv:2104.08541 tab. 2]. Both are the authors' figures; this page quotes the paper's table values. - **Mask type in GLaMM.** The paper says masks are binary [arxiv:2311.03356 §3.1]. The released code trains on logits and thresholds at 0 at evaluation. See the [GLaMM card](/papers/2311.03356), paper-vs-code section. - **LM-Nav success rate.** The text says 85% of experiments followed the instruction without collisions or disengagements [arxiv:2207.04429 §5.3]. Table 1 reports 0.8 net success in both environments [arxiv:2207.04429 tab. 1]. Both are the authors' figures; this page quotes the table value. - **LM-Nav similarity scale.** The paper applies a softmax over nodes to CLIP outputs [arxiv:2207.04429 §4.3]. The released code uses raw cosine similarity and takes the best image per node [arxiv:2207.04429 §4.3]. These are two different implementations of the same step. - **Words2Contact best combination.** The text calls GPT-3.5 with GroundingDINO and CLIPSeg the best combination [arxiv:2407.14229 §4.1], and the table bolds it on the overall column. The same table gives GPT-3.5 with Florence-2 and CLIPSeg a higher absolute median, 0.74 against 0.71 [arxiv:2407.14229 tab. 1]. The paper does not state the sample size per cell. - **Words2Contact loop and code.** The paper describes a loop of module selection, correction and confirmation, with absolute and relative branches [arxiv:2407.14229 §3]. The released code's entry point calls only a relative path, and that path always runs the detector, so the branches are not reproduced as written. See the [Words2Contact card](/papers/2407.14229), paper-vs-code section. - **Shikra chessboard wording.** The text says LLaVA "should achieve" 25.96%, which reads as a likely typo for "only"; the number is the one reported [arxiv:2306.15195 §4]. This is a side result, not a grounding number. - **SpatialRGPT region baselines.** The text says SpatialRGPT beats the region baselines on classification [arxiv:2406.01584 §4.2]. In its table, RegionGPT-7B is higher on both mAP and accuracy, and only the VILA variants are higher [arxiv:2406.01584 tab. 3]. The paper's quantitative claim holds for relative error, not for every success rate [arxiv:2406.01584 tab. 1]. - **RoboSpatial training size.** The abstract and introduction describe about 3M relations [arxiv:2411.16537 §1]. The model is trained on 900,000 [arxiv:2411.16537 app. B.1], and the 3M row is an extra scaling point [arxiv:2411.16537 tab. 9]. Quote the size that matches the number you report. - **RoboSpatial transfer direction.** Table 5's column headers do not say which environment was used for training, so the direction of its cross-domain numbers is unclear [arxiv:2411.16537 tab. 5]. ## Papers - [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](/papers/2306.15195): an MLLM that reads and writes image coordinates as plain text, with a frozen CLIP encoder and one linear projection. - [TransVG: End-to-End Visual Grounding with Transformers](/papers/2104.08541): a transformer-only box regressor with a learned [REG] token and no proposals or anchors. - [GLaMM: Pixel Grounding Large Multimodal Model](/papers/2311.03356): a chat model that attaches a mask to each named phrase, trained on the GranD data and evaluated on the GCG task it introduces. - [LM-Nav](/papers/2207.04429): a frozen LLM extracts landmarks, frozen CLIP scores map nodes against them, and a graph search picks the route. - [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](/papers/2311.15826): a LLaVA-1.5-style chat model for aerial images that writes boxes as text, with three task tokens, trained on instruction data built from detection labels. - [ScanQA: 3D Question Answering for Spatial Scene Understanding](/papers/2112.10482): answers questions about a 3D indoor scan and points to the objects they refer to, using VoteNet proposals and transformer fusion. - [Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models](/papers/2407.14229): an LLM, a detector and a segmenter turn a spoken instruction into a contact point on a humanoid's camera image, which a person can correct before the robot acts. - [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](/papers/2406.01584): a LLaMA2-7B VLM that takes user-given boxes or masks as region input, with an optional depth plugin, trained on 3D scene graphs built from OpenImages. - [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](/papers/2411.16537): a generator of relation and free-space questions from 3D scans, and the fine-tuning results for 2D and 3D models trained on them.