Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

In plain words

Chat models that can see images usually talk about them in general terms. They cannot be pointed at a spot in a picture, and they cannot say where something is. Shikra is built to do both. A user can mark a region with a box or a point, and the model can answer with boxes or points of its own.

It does this without special machinery. The image passes through a standard vision encoder and one linear layer into a language model. A location is just text: a bracketed list of numbers with three decimals. Training teaches the model to read and write those numbers like any other word. Its training data comes from existing caption and box datasets, plus question-and-answer pairs written by GPT-4 from captions that already carry boxes.

The results below are the authors' own. A reader should check the setup of each number before comparing it with other work.

TL;DR

• Shikra is a multimodal LLM whose inputs and outputs can include image coordinates written as plain text §1 §5.2.
• It adds no extra vocabulary, position encoder or detector: a frozen CLIP ViT-L/14, one linear layer and Vicuna-7B or 13B §5.1.
• It is trained in two stages on public VL data and GPT-4 generated referential-dialogue data §5.3 §5.4.
• Author-reported: REC 87.01 on RefCOCO val for Shikra-7B tab. 3, and the highest PointQA scores in tab. 4 tab. 5.
• The official code is CC BY-NC 4.0. Its paths and lines are cited here, but nothing is quoted (see Paper vs code).

Problem

Earlier MLLMs can perceive image content but cannot ground their language in image positions: users cannot indicate regions, and the models cannot give the exact location of what they describe §1. The authors test this with a 2×2 chessboard task on LVIS objects. LLaVA-13B reaches 25.96% accuracy, which the paper calls comparable to random choice §4.

Existing position-aware models add dedicated coordinate tokens (for example Pix2seq and OFA) or detection heads §2. The paper argues that coordinates should be ordinary text instead §2 §6.2.

Method

Architecture §5.1. A CLIP ViT-L/14 vision encoder is frozen. Its 16×16×1024 output is mapped by one fully connected layer to 256 visual tokens of width D, where D is 4096 for Vicuna-7B and 5120 for Vicuna-13B. The visual tokens can sit anywhere in the input sequence. The fully connected layer and the whole language model are trained.

Coordinates as text §5.2. Boxes are written as [x_min,y_min,x_max,y_max] and points as [x_center,y_center]. Coordinates are normalised by the image size and kept to three decimal places by default. They can appear anywhere in the input or output, and they are tokenised like other text. The paper's example reply contains bracketed coordinates inside a sentence §5.2.

Training data §5.3. Two sources:

1. Reorganised public data §5.3.1 app. A, tab. 8: captioning on LLaVA pre-training data, spotting captioning on Flickr30K Entities, grounded captioning on Visual Genome, REG on RefCOCO/RefCOCO+/RefCOCOg, REC on those sets plus Visual Genome, VQAv2, PointQA (Local, Twice, and Visual-7W's "which box" subset). LLaVA-Instruct-150K, VCR and Shikra-RD are used in stage 2 only app. A. Images that appear in test or validation sets were removed from training §5.3.1 app. A.
2. Generated referential-dialogue data, Shikra-RD §5.3.2: GPT-4 could not see images, so it received the five captions of each Flickr30K Entities image together with the boxes of the mentioned objects. It wrote question-and-answer pairs that can be answered from that information. This gave 5,922 pairs, where coordinates may appear in questions and answers.

Task prompts §5.3.3 app. B, tab. 9. Each task has a template, which GPT-4 expanded into hundreds of paraphrases. One is sampled at random during training. Three examples given in the text:

• spotting captioning: "Can you provide a description of the image and include the coordinates [x0,y0,x1,y1] for each mentioned object?"
• PointQA: "Referring to point in image , give a direct answer to ''"
• REC: "In , I need the bounding box coordinates of ."

Training §5.4. Stage 1 trains on the reorganised data. Stage 2 raises the sampling ratio of LLaVA-Instruct-150K and the generated referential-dialogue data to 50%. In both stages the visual encoder is frozen and all language-model parameters are tuned. The optimiser is AdamW with a cosine schedule, learning rate 2e-5 and global batch 64 on 8 A100 GPUs.

Grounding chain of thought §6.1. Shikra-7B is trained without pre-training on CLEVR in three settings: question to answer (Q→A), question to chain of thought to answer (Q→CA), and question to chain of thought with centre points to answer (Q→C^Point A). Results are in tab. 1.

Numerical versus vocabulary coordinates §6.2. Two toy Shikra-7B models are trained on REC data, one with numbers and one with extra coordinate tokens. Numbers do better on every split tab. 2. The paper notes that numbers need more tokens per box, which raises cost for dense scenes §6.2.

Pseudocode of one training sample, as the paper and the code describe it:

# Shikra training sample  [§5.1] [§5.2] [§5.3]
# Input:  image, pixel boxes, dialogue turns with "<image>" placeholders
# Output: causal-LM loss over the assistant tokens

def shikra_loss(image, boxes_px, dialogue):
    # Pad the image to a square with white, centred
    # [code: mllm/dataset/utils/transform.py#L52-L63]
    square = pad_to_square(image, fill=(255, 255, 255))
    # Shift each box by the padding offset (floor division)
    # [code: mllm/dataset/utils/transform.py#L66-L78]
    boxes_sq = shift_by_padding(boxes_px, image.size)
    # Divide by the square side, clamp to [0, 1], round to 3 decimals
    # [code: mllm/dataset/utils/transform.py#L30-L40]
    boxes_norm = normalise(boxes_sq, side=max(image.size))
    # Write "[x1,y1,x2,y2]" with 3 decimals; boxes joined by ";"
    # [code: mllm/dataset/process_function/box_process_function.py#L176-L183]
    text = write_boxes_into_text(dialogue, boxes_norm)
    # Frozen CLIP features, penultimate layer, CLS dropped: 256 patches
    # [§5.1] [code: mllm/models/shikra/shikra.py#L101-L109]
    patches = clip_vit_l14(square)              # (256, 1024)
    # One linear layer maps patches to the LLM width  [§5.1]
    image_tokens = linear_projection(patches)   # (256, D)
    # Wrap the 256 image tokens in <im_start> and <im_end>
    # [code: mllm/dataset/process_function/shikra_process_function.py#L53-L56]
    tokens = insert_image_tokens(text, image_tokens)
    # Loss only on assistant tokens; instruction tokens are masked
    # [code: mllm/dataset/process_function/shikra_process_function.py#L115-L131]
    labels = mask_user_tokens(tokens)
    return cross_entropy(llm(tokens), labels)

Hyperparameters

Values marked "code" are not in the paper's text. Each comes from the official repository at code_commit.

Results

Author-reported. Each table below repeats the paper's numbers. Setup is stated in each caption. The paper does not give variance or seeds for any table.

Table 1: CLEVR chain-of-thought forms (Shikra-7B, no pre-training) tab. 1

The gains over Q→CA and Q→A are 13.29 and 5.90 points; the text rounds these to 13 and 5.9 §6.1.

Table 2: Vocabulary versus numerical coordinates (two toy Shikra-7B models, trained on RefCOCO, RefCOCO+/g and Visual Genome only) tab. 2

Table 3: Standard REC tab. 3. The caption does not define the metric. The code reports the share of predictions with IoU above 0.5 rec.py. Generalist rows are without fine-tuning; specialist rows are fine-tuned models.

Table 4: Visual-7W "which box" PointQA tab. 4. Four baseline columns are the cited methods (keys zhu2016v7w, hu2017v7w_method1, lu202012in1, and its starred variant). Shikra's size is not stated.

Table 5: LookTwice-QA PointQA (Shikra-13B; accuracy %) tab. 5

Table 6: VQA and captioning, compared with generalist models (Shikra-13B) tab. 6

FM is Flamingo. The paper also quotes VQAv2 val 83.3 and OK-VQA 53.8 from a third-party toolbox tab. 6.

Table 7: POPE object hallucination tab. 7. Shikra-7B; the other columns are taken from the POPE paper, as stated in the caption.

The text says Shikra's POPE results are comparable to InstructBLIP and well ahead of the other MLLMs §6.3.

Figures. Figure 1 is a teaser demo and Figure 2 shows Shikra-7B dialogues with users fig. 1 fig. 2. Appendix figures 3 to 10 show more dialogues and tasks, including OCR, REG and Visual-7W points and boxes app. C. No images are copied into this card.

Paper vs code

The repository is shikras/shikra at ff79a4c, the code link in the abstract. Its LICENSE is CC BY-NC 4.0, so we read it for comparison and quote nothing.

• Stage 1 length. Paper: 100,000 steps, about 1.5 epochs §5.4. Code: the stage-1 config sets 2 epochs over the concat8 mix shikra_pretrain_concat8_stage1.py. The paper's step count and the config's epoch count cannot be reconciled from the code.
• Stage 2 sampling. Paper: 50% for LLaVA-Instruct-150K and the generated data §5.4. Code: the instruction-and-generated group is weighted 0.1 against 0.9 for the rest [code: config/base/dataset/mix_pretrain_final19.py#L5] [code: config/base/dataset/mix_pretrain_final55.py#L7]. Which of the two matches the released checkpoints cannot be checked from the code alone.
• Image padding and coordinates. Paper: coordinates are normalised by image size §5.2. Code: each image is padded to a square first, and boxes are shifted and normalised by the square side transform.py single_image_convsation.py. The paper does not mention this step.
• Feature layer. Paper: "ViT-L/14 of CLIP" §5.1. Code: the penultimate hidden layer is used, with the CLS token dropped shikra.py.
• Special tokens. Paper: no extra vocabulary for coordinates §5.1. Code: adds <im_start>, <im_end> and <im_patch> tokens around the image tokens, and initialises their embeddings to the mean of the existing ones shikra.py.
• Projector initialisation. The paper says nothing. The _o configs load a projector trained on LLaVA v1.1 [code: config/base/model/shikra_7bo.py#L5] [code: config/base/model/shikra_13bo.py#L5]. The stage-1 config leaves the model path unset shikra_pretrain_concat8_stage1.py, so the released initialisation cannot be verified from the code.
• Optimiser settings. Paper: AdamW, cosine, learning rate 2e-5, global batch 64 §5.4. Code: these match, and it adds 3% warmup and zero weight decay [code: config/base/train/shikra_fsdp.py#L13-L19].
• Truncation. The base model caps length at 2048 tokens [code: config/base/model/shikra.py#L13], while the stage configs truncate to 4096 shikra_pretrain_concat8_stage1.py. The paper gives no length limit.
• Demo padding. The demo server pads the image and shifts the boxes the same way as training server.py, so inference matches training.

Pitfalls

• Coordinates are relative to a white square, not the original image. Implementers who divide x by width and y by height, without padding, will produce inconsistent targets on non-square images. Pad to a centred square first; the shift is (max - min) // 2 on the short axis; divide by the square's side transform.py.
• Rounding is to 3 decimals with round, not truncation. Clamp to [0, 1] first transform.py.
• The output parser is strict. It accepts only bracketed numbers with no spaces and a leading digit, such as [0.268,0.372]. The paper's example writes [0.268, 0.372] with a space. A parser built from that example will miss the training format box_process_function.py §5.2.
• Negative coordinates are not matched. The pattern accepts only digits, so a box with a minus sign is dropped without an error box_process_function.py.
• A tokenisation mismatch turns off the loss for that sample. When token counts disagree, all labels are set to ignore and only a warning is printed shikra_process_function.py.
• Truncation can cut text but not image tokens. The code keeps image and boundary tokens when shortening a sequence, and asserts the final length shikra_process_function.py.
• Two training-length figures disagree. The paper's 100,000 steps and the code's 2 epochs cannot both be the schedule of the released weights §5.4 shikra_pretrain_concat8_stage1.py.
• Projector initialisation is not in the paper. A reimplementation that starts from random projector weights differs from the official pipeline in ways the paper does not record.
• The in-code REC metric is a training check. The code labels the IoU as "non-rigorous training progress checking" rec.py. The paper does not say how the reported REC numbers were computed.
• Decoding is greedy in the eval configs. The paper does not state a decoding method shikra_eval_multi_rec.py.
• Results use different model sizes. Table 6 is Shikra-13B, Table 7 is Shikra-7B and Table 3 gives both. Compare like with like tab. 3 tab. 6 tab. 7.
• Chessboard result wording. The text says LLaVA "should achieve" 25.96%. The sentence is probably a typo for "only"; the number is the one reported §4.
• Single runs. No variance, seed or confidence interval is reported for any result tab. 1.
• Grounded chain of thought is a preliminary test. The authors call it "a preliminary attempt" on CLEVR only §6.1.
• Some data is generated by GPT-4. Shikra-RD is 5,922 pairs written from captions and boxes, not from images §5.3.2.
• Unused source material. The source includes a LocalQA table and a commented-out vision table. Neither is in the paper, so they carry no results here app. A.
• Limitations stated by the authors. English only; unsuitable for dense detection and segmentation; may produce harmful or counterfactual replies §7.

Lineage

• Builds on: LLaVA-style projection and instruction tuning §2 §5.4; the CLIP ViT-L/14 encoder §5.1; Vicuna §5.1.
• Compared against: Pix2seq and OFA, which add coordinate vocabularies §2 §6.2; generalist and specialist models in tab. 3.
• Superseded by: not stated.