GeoChat: Grounded Large Vision-Language Model for Remote Sensing

In plain words

General-purpose chatbots that can look at pictures tend to fail on satellite and aerial photos. Those images are large, show many tiny objects, and need answers both about the whole scene and about one marked region. Training data for that kind of conversation barely exists.

GeoChat tackles this in two parts. First, its authors built a conversation dataset from existing labelled aerial-image collections: they turned each image's object list (types, colours, sizes, positions, relations) into short text descriptions, and then had a text-only language model write questions and answers about them. Second, they took an existing chat model for ordinary photos and tuned it on that dataset, so it can also be asked about a region and can answer with the location of an object written as a box.

The point is one model for several jobs: describing a scene, answering questions, describing a boxed area, and pointing to the object someone names.

TL;DR

• GeoChat is a remote-sensing vision-language model built on LLaVA-1.5: CLIP ViT-L/14 at 504x504, a two-layer MLP adaptor, and Vicuna-v1.5 7B tuned with LoRA §3.1 §3.2.
• Input: an image, a text query, optional region boxes, and optional task tokens; output: text with boxes written as numbers §3.1.
• Training data: about 318k image-instruction pairs (306k train, 12k test) generated from RS detection, VQA and scene-classification datasets tab. 1 §4.
• Author-reported results include 84.43% (UCMerced) and 72.03% (AID) zero-shot scene classification tab. 5, and 90.70 average on RSVQA-LRBEN, against 92.29 for RSGPT, which is fine-tuned on that data tab. 6 (see Pitfalls on training overlap).
• Use it as a reference design for a grounded RS chat model. Do not trust the zero-shot labels blindly: see Pitfalls.

Problem

General-domain VLMs trained on web image-text pairs give inaccurate or invented answers on remote-sensing images §1. RS images are high-resolution, contain many small objects at widely varying scales, and need region-level as well as scene-level reasoning §1. Two gaps block a fix: there is little RS multimodal instruction-following data, and there are few strong RS backbone models §1. Earlier RS VQA work framed the task as classification over fixed answers, which does not support open-ended chat §1.

Method

GeoChat has three parts: a visual encoder, an MLP adaptor, and an LLM. It follows LLaVA-v1.5 §3.1 and adds three things: a task token that selects the kind of answer, spatial coordinates in both inputs and outputs, and a remote-sensing instruction set §3.1 §4. Figure 2 gives the overview fig. 2.

1. Visual encoding at higher resolution §3.1. Start from CLIP ViT-L/14 (336 px, 576 patches). Interpolate the position embeddings so the encoder takes 504x504 input, giving 1296 patch tokens of dimension 1024 per image §3.1. The paper says this is "almost double" the token count; 1296 / 576 = 2.25 §3.1.
2. Vision-to-language adaptor §3.1. A small MLP with one hidden layer and GELU projects each 1024-d vision token to the LLM width of 4096.
3. Spatial text representation §3.1. A box is written as text: {x_left, y_top, x_right, y_bottom | θ}. Coordinates are normalised to [0, 100]; θ is the rotation angle measured from the lower edge. Region boxes given in the input use the same form.
4. Task tokens §3.1 tab. 1. One of three tokens switches the task: [grounding] (describe the image and give boxes for the objects mentioned), [identify] (describe the object inside a given box), [refer] (locate the object named in a <p> ... </p> phrase). Plain VQA and scene-classification prompts carry no task token; they ask for a single word or phrase §3.1 tab. 1. The paper says no task token is used for vision-irrelevant commands §3.1.
5. LLM and LoRA §3.1 §3.2. Vicuna-v1.5 7B is the LLM. The CLIP encoder and the MLP adaptor stay frozen. LoRA adapters train on the LLM; after training they are merged into the pretrained weights for inference §3.1 §3.2.
6. Initialisation §3.2. The MLP adaptor comes from LLaVA-1.5 pre-training on a 558K subset of LAION-CC-SBU with BLIP captions. The CLIP encoder and Vicuna-v1.5 come from their own pre-trained weights §3.2.

The region and grounding tasks are described in §3.1 as three task variants: image-level conversation (no boxes), region-level conversation (boxes in the input), and grounded conversation (boxes in the output, selected by task tokens) §3.1.

# GeoChat inference for one request  [§3.1] [fig. 2]
# Input:  image (504, 504, 3); query text; task token; region boxes
# Output: text with boxes written as numbers

def answer(image, query, task_token, region_boxes):
    # Encode at 504 x 504 with interpolated position embeddings  [§3.1]
    patch_tokens = clip_vit_l14(image)           # (1296, 1024)
    # Project each vision token into the LLM embedding space  [§3.1]
    visual_embeds = mlp_adaptor(patch_tokens)    # (1296, 4096)
    # Boxes and task tokens are plain text in the prompt  [§3.1] [tab. 1]
    prompt = build_prompt(query, task_token, region_boxes)
    # Vicuna-v1.5 7B with LoRA weights merged in  [§3.1]
    return llm.generate(visual_embeds, prompt)

Instruction-set creation

The dataset is built in four steps. The paper gives a single pipeline; the details below are its own §4.

# Instruction-data generation from detection annotations  [§4] [tab. 2]
# Input:  per image: object boxes, categories, masks, colours
# Output: multi-turn QA, complex QA and detailed descriptions

def make_instruction_pairs(image, objects):
    # Per object: category, colour, relative size, 3x3 grid position  [§4]
    attributes = [describe_object(image, obj) for obj in objects]
    # Relations from box-distance groups and containment checks  [§4]
    relations = relate_objects(objects)
    # Short description from the attributes; exact wording not stated  [§1] [§4]
    description = short_description(attributes, relations)
    # Vicuna writes QA from the text alone; it never sees the image  [§1] [§4]
    return vicuna_qa(description)

• Source datasets §4 tab. 2: three object-detection sets (DOTA, DIOR, FAIR1M, which together make SAMRS), one scene-classification set (NWPU-RESISC-45), one VQA set (LRBEN), and one flood VQA set (Floodnet).
• Missing classes §4: buildings, roads and trees are absent from the detection sets. The authors ran a ViTAE-RVSA model pre-trained on LoveDA to produce pseudo-labels and kept only predictions for classes the ground truth lacks §4.
• Colour §4: K-Means on the object's pixels inside its ground-truth box; the centre of the largest cluster is the colour.
• Size §4: the 20th and 80th percentiles of each class's instance areas over the whole dataset set the small and large cut-offs; the rest is normal.
• Position §4: the image is split into a 3x3 grid; each cell gets a compass-style name, and the object's centre pixel decides which cell applies.
• Relations §4 tab. 4: objects are grouped by box distance; a relation such as "surrounded by" is set only if one box lies fully inside another. Relation labels are chosen from the pair of class labels; the paper gives examples only.
• Referring expressions §4: a fixed template fills the attributes, with colour and size in either order: "The/A in/on the ." Relations between two objects use a second template. The paper gives these templates without equation numbers.
• Grounding data §4: the short descriptions are reused as referring expressions for three QA types: grounding description, referring expression, and region captioning §4 tab. 1.
• Counts tab. 1 §4: 65k images for multi-round conversations, 10k for complex questions, 30k for detailed descriptions (all Vicuna-written from the description); 45k grounding description, 40k region captioning, 25k referring expression; 56k LRBEN, 31.5k NWPU-RESISC-45, 4k Floodnet. Total about 306k train, 12k test tab. 1.
• Figures of the dataset fig. 4 show the types of annotation used: attributes, relations, referring expressions and region captions over an RS image.

Hyperparameters

Training data mixture

Test set: 12k instruction pairs tab. 1. Source images tab. 2: DOTA 17,480 (1024 x 1024); DIOR 23,463 (800 x 800); FAIR1M 64,147 (600 x 600); LRBEN 600 (256 x 256); Floodnet 4,056 (3000 x 4000); NWPU-RESISC-45 31,500 (256 x 256).

Results

Author-reported, zero-shot unless noted. Scene classification prompts list every class and ask for one word or phrase §5.2. Grounding and region results come from the authors' own benchmark, built with the same pipeline as the training data §5.4.

Scene classification (zero-shot accuracy, %) §5.2 tab. 5

AID: a 20% split §5.2. UCMerced: the whole set, zero-shot §5.2.

RSVQA-LRBEN, test set (accuracy, %) §5.3 tab. 6

Table 6 caption: GeoChat is "a generic model not specifically finetuned on target dataset" tab. 6. See Pitfalls: RSVQA-LRBEN is in the training mix tab. 1.

RSVQA-HRBEN (zero-shot, accuracy, %) §5.3 tab. 8

GeoChat is lower than Qwen-VL and LLaVA-1.5 on Presence tab. 8. Its gain is on Comparison and the average §5.3.

Visual grounding, authors' benchmark (acc@0.5, %) §5.4 tab. 7

Benchmark size: 7,653 [refer] questions, 758 [grounding] questions, 555 grounding-description questions §5.4.

Grounding description §5.4 tab. 9

Region-level captioning §5.4 tab. 10

Qualitative examples are in fig. 1 (teaser), fig. 5 (grounding, referring detection, damage detection) and fig. 3 (instruction template).

Paper vs code

Code: https://github.com/mbzuai-oryx/geochat, commit 4850920e005a849bd224d0ce35aa9db031fa5155, a shallow clone. The paper's abstract links this repository. The repository has no LICENSE file; pyproject.toml classifies it as "Apache Software License" pyproject.toml. The code licence is therefore not stated.

Differences and gaps:

• LoRA targets. Paper: W_q and W_v only §5.1. Code: every torch.nn.Linear in the model except the multimodal modules (mm_projector, vision_tower, vision_resampler), via find_all_linear_names train.py train.py.
• Global batch size. Paper: 144 §5.1. The released LoRA script uses 32 per device with one accumulation step finetune_lora.sh. 144 needs 4.5 GPUs at that setting; the README's 3 GPUs give 96 README.md. No configuration in the repository gives 144.
• Training schedule. Paper: two stages, with stage 2 on grounding data for 1600 steps §5.1. Released LoRA script: one run of 1 epoch on GeoChat_Instruct.json finetune_lora.sh finetune_lora.sh. The full-schedule script trains on llava_instruct_158k.json, which is LLaVA's data, not GeoChat's, for 3 epochs at learning rate 2e-5 finetune_full_schedule.sh finetune_full_schedule.sh finetune_full_schedule.sh. No stage-2 script is in the repository.
• Undocumented hyperparameters. The paper gives no LoRA alpha, dropout, learning rate, warm-up, weight decay, max length or precision. The code sets alpha 16, dropout 0.05 train.py, learning rate 2e-4, warm-up ratio 0.03, weight decay 0, max length 2048, bf16 finetune_lora.sh finetune_lora.sh. These are the code's values, not the paper's.
• Resolution. Paper and code agree on 504 x 504 with interpolated position embeddings §5.1 clip_encoder.py.
• Pre-trained adaptor. Paper: LLaVA-1.5 adaptor §3.2. Code: llava-v1.5-mlp2x-336px-pretrain-vicuna-7b-v1.5, a 336 px adaptor used at 504 px finetune_lora.sh.
• Preprocessing mismatch. Training pads non-square images to a square before resizing to 504 when image_aspect_ratio == 'pad' train.py. The evaluation scripts resize and centre-crop to 504 with no padding batch_geochat_grounding.py. Evaluation therefore sees different image content from training for non-square inputs. The paper does not mention this.
• Evaluation scripts. batch_geochat_grounding.py has an unconditional pdb.set_trace() on line 33 batch_geochat_grounding.py. The [refer] and [grounding] branches use qs before it is assigned batch_geochat_grounding.py batch_geochat_grounding.py. As written, this looks like it raises UnboundLocalError on the first question. Not run; the repo does not show a fix.
• Missing pipeline. The released code has no code for building the dataset (pseudo-labelling, attributes, relations, expression templates). The dataset is a separate HuggingFace download Data.md. No box parser or box-normalisation code is in the release, so the [0, 100] convention cannot be checked from the code. The grounding metric's rotated-box IoU is also absent.
• Evaluation prompt. The eval scripts use the prompt forms [refer] Give me the location of <p> ... </p> batch_geochat_grounding.py and [identify] What is the object present at ... batch_geochat_referring.py. The prompt forms match tab. 1.

Agreements: task-token names, the MLP projector shape, the 504 px resolution and the LoRA rank (64).

Pitfalls

• Zero-shot labels overlap training data. RSVQA-LRBEN is listed in the training mix with 56k pairs tab. 1, yet the table caption calls GeoChat "not specifically finetuned on target dataset" tab. 6. The LRBEN result is therefore not a zero-shot number in the strict sense. The HRBEN table says "not trained on the target dataset" tab. 8, and AID and UCMerced do not appear in the training list tab. 1 tab. 2, so the scene-classification results are the cleanest zero-shot numbers.
• In-distribution grounding benchmark. The test benchmark uses "the same dataset creation pipeline" as training data §5.4, and the training descriptions are reused as referring expressions §4. Grounding and region-caption scores are therefore not out-of-distribution. The 83.9 METEOR for region captioning is against captions produced by the same pipeline §5.4 tab. 10.
• Baseline output format. The paper attributes MiniGPT-v2's low scene-classification accuracy partly to not following the one-word answer format §5.2. Those scores (4.76, 12.90) measure format following as well as recognition tab. 5.
• Text-only data generation. The QA pairs are written by Vicuna from text descriptions, with no image access §4. The paper reports no human check of label or answer quality. The system prompt asks for only definite answers system_message.txt, but nothing in the paper measures how often answers are wrong. Treat hallucination control as not stated.
• Ambiguous coordinate convention. Coordinates are "normalised within [0, 100]" §3.1. The paper does not say whether x is divided by width and y by height, and it gives no rotated-box IoU formula for acc@0.5 §5.4.
• Token cost. At 504 px the vision encoder yields 1296 tokens per image, 2.25 times the 576 at 336 px §3.1. The paper calls this "almost double". Budget for it when comparing to LLaVA-1.5.
• Adaptor resolution. The projector was pre-trained at 336 px §3.2 finetune_lora.sh and is then used at 504 px. The paper does not discuss this.
• Variance and seeds. Number of runs, seeds and error bars are not stated for any table.
• Grounding weakness. On large objects MiniGPT-v2 is slightly better (21.9 vs 21.7) tab. 7. The paper also says overall performance is low on small objects and multi-box predictions §5.4.
• Stage-2 recipe. Stage 2 on grounding only is described §5.1 but its data split and script are not in the release (see Paper vs code).

Lineage

• Builds on: LLaVA-1.5, which supplies the MLP adaptor, the training recipe and the pre-trained adaptor §3.1 §3.2. The paper also cites LLaVA as prior visual instruction tuning §1.
• Builds on, not in the wiki: LoRA, arXiv:2106.09685 (from the paper's bibliography) §3.1; Vicuna-v1.5 7B and CLIP ViT-L/14, both cited without an arXiv id §3.1.
• Compared against: MiniGPT-v2, arXiv:2310.09478 tab. 5 tab. 7; Qwen-VL, arXiv:2308.12966 tab. 5; RSGPT, arXiv:2307.15266 tab. 6 tab. 8; RSVQA, EasyToHard, Bi-Modal, SHRNet (remote-sensing VQA models) tab. 6.
• Superseded by: not stated in this wiki.