SIFThinker: Spatially-Aware Image Focus for Visual Reasoning

In plain words

Vision-language models often look at a whole picture at once. This paper's model is trained to look the way a person does: it names a small area that matters for the question, says what it sees there, then moves or tightens the area if it was wrong, and only then answers. Each area also carries a rough estimate of how far away its contents are, so the model is pushed to reason about where things sit in space, not only what they are.

The training data is built backwards. Starting from boxes that are already known to be right, the authors widen them step by step until they cover the whole image, record the chain in reverse, and have a strong model write the reasoning that goes with it. The model is first taught this style by example, then refined with a reward signal that checks the answer, the boxes, the depth values and the output format.

This matters for anyone who wants an assistant that answers questions about small or spatially arranged details, such as which object is behind another or how many of something there are, without a separate tool to crop the image.

TL;DR

SIFThinker fine-tunes Qwen2.5-VL-7B to reason in an interleaved text and bounding-box format, where each box is paired with a depth value. Its training data (SIF-50K) is produced by expanding ground-truth boxes into a chain of boxes, and then reversing that chain. Training is supervised fine-tuning followed by GRPO-SIF reinforcement learning with four reward terms. Results are author-reported and strongest on spatial and fine-grained benchmarks, with smaller gains or losses on general benchmarks. The official code is only partly released: the training code is present, but the dataset and the judge model are not.

Problem

Stepwise crop-and-answer methods (V*, VisCoT, VisRL) separate the box from the reasoning, which weakens the chain of thought and anchoring of regions § Introduction. Methods that add depth either use it as an external tool or as training data, so the model does not reason about depth inside its own reasoning § Introduction. Grounding is often trained apart from the answer, so result-level supervision cannot correct the boxes § Introduction. The paper assumes a reasoning chain that revisits regions, with depth attached, can fix these weaknesses within one model § Introduction.

Method

Overview. The pipeline has three parts fig. 1 § Methodology (Method overview). (1) Data generation: each training item is a question, an image, ground-truth boxes and an answer; a reverse expansion procedure turns these into a chain of interleaved boxes and text § Methodology (Data Generation) alg. 1. (2) Supervised fine-tuning (SFT) on the full SIF-50K set § Methodology (Method overview). (3) Reinforcement learning with GRPO-SIF on a 200-item subset § Methodology (Method overview). The inference prompt asks the model to alternate <area> (boxes with depth as JSON) and <text> blocks inside <think>, then give the answer in <answer> § Methodology (Reasoning-format reward) § Appendix (Prompt template).

Data generation. Source images come from Flickr30k, Visual7W, GQA, Open Images, VSR, Birds-200-2011 and TallyQA. All sources have ground-truth boxes § Methodology (Data Generation). The depth map for each image comes from Depth Anything V2 alg. 1. The chain is produced by a vision-language model (Doubao-1.5-vision-pro) given the question, the answer, the depth map and the reversed box sequence, with the boxes drawn on the image alg. 1 § Methodology (Data Generation).

Reverse expansion alg. 1. Start from the ground-truth boxes. Repeat for t = 1 to K, with K = 5 and the step count S = K − t + 1 each round: push every side of each box outward by its distance to the image border divided by S; merge any pair of boxes whose IoU is above zero into their union; if a single box remains, allow two more rounds (once) alg. 1. Then add zero, one or two extra boxes, each chosen at random from outside the current boxes, with an area difference of at most T = 0.2 from the previous one alg. 1. Finally reverse the order of the box layers, draw them, and call the depth-aware model.

# Reverse expansion for CoT completion  [alg. 1]
# Input:  boxes_gt, normalised [x1, y1, x2, y2] boxes
# Output: layers, box sets ordered from the ground truth outward

def reverse_expansion(boxes_gt):
    # Layer 0 holds the ground-truth boxes  [alg. 1]
    layers = [boxes_gt]
    steps = 5  # K  [alg. 1]
    merged_once = False  # ET flag  [alg. 1]
    t = 1
    while t <= steps:
        # Step count shrinks each round: S = K - t + 1  [alg. 1]
        s = steps - t + 1
        boxes = []
        for x1, y1, x2, y2 in layers[-1]:
            # Push each side outward by its border distance / S  [alg. 1]
            boxes.append((x1 - x1 / s, y1 - y1 / s,
                          x2 + (1 - x2) / s, y2 + (1 - y2) / s))
        # Merge every pair with IoU > 0 into the union  [alg. 1]
        boxes = merge_overlapping(boxes)
        layers.append(boxes)
        # One box left: allow two more rounds, once  [alg. 1]
        if len(boxes) == 1 and not merged_once:
            steps = t + 2
            merged_once = True
        t = t + 1
    # Add 0, 1 or 2 random extra boxes, area within T = 0.2  [alg. 1]
    for _ in range(random_count([0, 1, 2])):
        layers.append(sample_box_outside(layers, threshold=0.2))
    return layers

The algorithm then reverses layers, draws them on the image, and asks the model for the reasoning text alg. 1. The result is SIF-50K, stored as tuples of question, image, depth map, ground-truth boxes, answer and chain § Methodology (Data Generation).

SFT. The SFT target is the full SIF-50K chain. Hyperparameters are in the table below § Appendix (Hyper-parameters setup).

GRPO-SIF RL. The objective is the GRPO clipped surrogate with a KL term to a reference model. For each question, the policy samples N completions. Each completion gets one scalar reward, which is the sum of four components. The advantage is the reward minus the group mean, divided by the group standard deviation plus a small constant eq. 1. The policy-gradient loss is averaged over tokens within each completion and then over completions § Methodology (Reinforcement Learning with GRPO-SIF). The paper gives no weights for the four components.

# GRPO-SIF advantages for one question  [eq. 1]
# Input:  completions (N model outputs), gt (answer and boxes)
# Output: advantages (N floats)

def grpo_sif_advantages(completions, gt):
    rewards = []
    for completion in completions:
        # Sum of the four components, no weights stated  [§ Methodology]
        reward = (format_reward(completion)
                  + answer_reward(completion, gt)
                  + grounding_reward(completion, gt)
                  + depth_reward(completion, gt))
        rewards.append(reward)
    # Group mean and standard deviation  [eq. 1]
    mean = average(rewards)
    std = standard_deviation(rewards)
    # delta = 1e-8 in the paper  [eq. 1]
    return [(r - mean) / (std + delta) for r in rewards]

Format reward. 1.0 if the output strictly follows the template, else 0.0 § Methodology (Reasoning-format reward).

Hierarchical IoU (HIoU). Compare the predicted boxes with the ground truth in two ways and average them fig. 2 § Methodology (Hierarchical Intersection over Union). The global IoU compares the union of all predicted boxes with the union of all ground-truth boxes eq. 2. The pairwise IoU matches the boxes one-to-one with the Kuhn–Munkres algorithm, maximising total IoU eq. 3, and averages the IoU over the matched pairs eq. 4. HIoU is the mean of the two eq. 5.

# Hierarchical IoU of predicted vs ground-truth boxes  [eq. 2][eq. 5]
# Input:  pred_boxes, gt_boxes (lists of [x1, y1, x2, y2])
# Output: score in [0, 1]

def hiou(pred_boxes, gt_boxes):
    # Global IoU of the union of all boxes on each side  [eq. 2]
    global_iou = iou(union(pred_boxes), union(gt_boxes))
    # One-to-one matching that maximises total IoU  [eq. 3]
    pairs = hungarian_max_iou(pred_boxes, gt_boxes)
    # Mean IoU over the matched pairs  [eq. 4]
    pair_iou = mean([iou(p, g) for p, g in pairs])
    # Average of the two  [eq. 5]
    return (global_iou + pair_iou) / 2

Answer reward. A judge model scores each answer from 0 to 1 against the ground truth § Methodology (Progressive-answer-accuracy reward) § Appendix (Prompt template). The reward adds the change from the previous step to the judge score eq. 6. The paper says the previous step is the group mean at t−1, but the released code uses a different reference (see Paper vs code).

# Progressive answer reward  [eq. 6]
# Input:  score_now (judge score in [0, 1]), prev_scores (step t-1)
# Output: reward (float)

def answer_reward(score_now, prev_scores):
    # Judge score plus its gain over the previous group mean  [eq. 6]
    return score_now + (score_now - mean(prev_scores))

Grounding reward with correction. Take the last box in the reasoning and the first box that is not the whole image. Score both with HIoU. The reward is the last score plus the gain from the first to the last eq. 7. This rewards corrections, not only the final box § Methodology (Correction-enhanced grounding reward).

# Correction-enhanced grounding reward  [eq. 7]
# Input:  box_trace (every box in order), gt_boxes
# Output: reward (float)

def grounding_reward(box_trace, gt_boxes):
    # Last box in the reasoning  [§ Methodology]
    s_end = hiou(box_trace[-1], gt_boxes)
    # First box that is not the full image  [§ Methodology]
    s_init = hiou(first_partial_box(box_trace), gt_boxes)
    # Final accuracy plus the gain from correction  [eq. 7]
    return s_end + (s_end - s_init)

Depth-consistency reward. For each (box, depth) pair the model writes, read the ground-truth depth from the depth map inside that box. The reward is 1 only if every pair has a relative error of at most T = 0.1, else 0 eq. 8 § Methodology (Depth-consistency reward).

# Depth-consistency reward  [eq. 8]
# Input:  pairs of (box, depth_pred), depth_map (H, W) in [0, 1]
# Output: 1.0 or 0.0

def depth_reward(pairs, depth_map):
    for box, depth_pred in pairs:
        # Ground-truth depth inside the box; the code takes the mean
        # [code: open_r1/vlm_modules/qwen_module.py#L71-L110]
        depth_gt = mean_inside(depth_map, box)
        # Relative error must be at most T = 0.1  [eq. 8]
        if abs(depth_pred - depth_gt) / depth_gt > 0.1:
            return 0.0
    return 1.0

Overall RL setup. Completions are sampled from the SFT model, and the policy is a LoRA adapter § Appendix (Hyper-parameters setup). The vision encoder is frozen in the released run script [code: GRPO-SIF/run_scripts/train_grpo_sif.sh#L64]. The paper does not state this.

Hyperparameters

Figures are for Qwen2.5-VL-7B, the example the appendix gives § Appendix (Hyper-parameters setup). Where the paper is silent, the released configs are given as well.

Results

Author-reported. Setup: Qwen2.5-VL-7B base, trained on SIF-50K; baselines are the listed open-source and proprietary models, evaluated by the authors § Experiments. The judge prompt is also used for benchmark scoring § Appendix (Prompt template).

Spatial intelligence tab. 1. Scores on SpatialBench (position, existence, counting, size), SAT (static) and CV-Bench, 7B models:

The paper also reports 8B-scale results with Bunny-Llama3-8B as base, where the average over the four SpatialBench categories is 64.3, against 59.6 for SpatialBot-8B and 74.5 against 67.1 for SSR-7B, the latter on the 7B base § Experiments (Spatial Intelligence) tab. 1. Averages computed from the table are consistent with these numbers.

Visual perception tab. 2 tab. 3 fig. 3 fig. 4. VisCoT_s (average over five subsets) and V*Bench (attribute and spatial):

Grounding tab. 3. Top-1 accuracy at IoU 0.5 on RefCOCO, RefCOCO+ and RefCOCOg, and NMS-AP on OVDEval. Qwen2.5-VL-7B:

General benchmarks tab. 4. Mean across the three base-model groups; a few entries are lower than the baseline (see Pitfalls):

Ablation (Qwen2.5-VL-7B, VisCoT_s average) tab. 5:

The text also reports 0.723 when only the correction term is removed, and 0.735 for a step-wise non-decreasing IoU reward § Experiments (Ablation on Correction-enhanced Grounding Reward). These two are not in the table.

Paper vs code

Official repository used: bytedance/SIFThinker at code_commit. The abstract links zhangquanchen/SIFThinker, a repository of the first author that holds only a README. That README says the official code moved to bytedance/SIFThinker, which is the repository used here. Its license is MIT, copyright ByteDance LICENSE. Code paths are relative to the repository root; open_r1/... abbreviates GRPO-SIF/src/open-r1-multimodal/src/open_r1/....

• Depth reward is partial, not all-or-none. The paper gives 1 only if every pair passes eq. 8. The code adds 1/(number of boxes) for each passing box, so it returns fractions [code: open_r1/vlm_modules/qwen_module.py#L984-L1002] (the ground-truth depth is the mean inside the box qwen_module.py).
• Grounding progress is clipped at zero. The paper's correction term may be negative eq. 7. The code adds the gain only when the final IoU is at least the initial IoU, otherwise adds 0 qwen_module.py. The HIoU itself matches the paper as an average of the global and pairwise scores qwen_module.py.
• Answer progress uses a different reference. The paper compares with the group mean at step t−1 eq. 6. The code compares with one stored value per (question, image): the reward is raised only if it is not below the stored value, and the stored value is the batch mean of the rewards that were returned in the previous step qwen_module.py grpo_trainer.py. The trainer writes one entry per step, for the first sample of the batch grpo_trainer.py, so the progress term fires only for later samples with the same question and image. This is a code reading; it was not run.
• Judge is not released. The paper uses Doubao-1.5-vision-pro § Methodology (Progressive-answer-accuracy reward). The code's judge client has placeholder endpoints and model names qwen_module.py qwen_module.py. The code also sends the image to the judge; the paper's judge description lists the question, the predicted answer and the ground truth only § Methodology (Progressive-answer-accuracy reward).
• Advantage constant. Paper δ = 1e-8 eq. 1; code adds 1e-4 grpo_trainer.py.
• Reward sum. The code sums the reward functions without weights grpo_trainer.py, consistent with the paper, which gives no weights.
• RL duration and length. The paper says 20 epochs and 2028 tokens § Appendix (Hyper-parameters setup). The script runs 1 epoch train_grpo_sif.sh and 2048 tokens train_grpo_sif.sh.
• LoRA alpha and dropout are not in the paper. The script uses alpha 128 and dropout 0.05 train_grpo_sif.sh.
• SFT schedule. The paper gives rate and epochs only. The config uses cosine decay with 10% warm-up, effective batch 16, and 50,000 maximum samples with 10% held out for validation train_sft.yaml train_sft.yaml train_sft.yaml.
• Expansion procedure. The data-generation code uses a fixed step size from a helper, not the per-side margin / S of alg. 1 produce.py. It merges boxes on strict overlap produce.py, not IoU > 0. It also adds a final jump to the whole image produce.py, which the algorithm does not contain. I did not verify how the random extra boxes are chosen in the code.
• Extra box jitter. data_generation/bbox_adjust.py expands boxes by 0.2, scales them by 0.7 to 1.3 and translates them bbox_adjust.py. The paper does not describe this step, and its role in SIF-50K is not verified.
• Not in the repository. The SIF-50K data, the 200-sample RL set, the trained checkpoints, and the judge model are all absent. The SFT config points to a dataset called SIF-50K that must be supplied separately train_sft.yaml.
• Consistent. The clipped surrogate, the per-completion token mean, and the unbiased KL estimator match the paper's objective grpo_trainer.py.

Pitfalls

• Judge dependence. Every answer reward and every benchmark score in the paper uses the same judge prompt § Appendix (Prompt template). The judge is not released, so answer rewards and reported accuracies cannot be reproduced exactly. Expect to need your own judge and to report its identity.
• Judge cost. Each RL step calls the judge once per completion (N = 8 per question) qwen_module.py. Budget for those calls on top of the 8-GPU training run.
• Depth scale. The depth-consistency test uses a relative error. In the code the depth values are in [0, 1] and decoded from a pseudo-colour palette (Spectral_r) by nearest colour, mapped as 1 − level/255 qwen_module.py. A relative error is not invariant to shifts of the depth scale, so a different depth model or encoding can change which answers pass. The paper does not describe how depth values were decoded.
• Index slip in the pairwise IoU. The paper's pairwise IoU formula reuses the index i for both boxes in the intersection eq. 4; for matched pairs it should use the matched ground-truth index. The code's behaviour was not checked here.
• Empty box sets are undefined. HIoU, the matching, and the progress terms are not defined when the model outputs no boxes, nor when all boxes are the whole image. The code catches exceptions and returns 0 for the grounding reward qwen_module.py.
• Rewards are per completion, indexed by t. The paper writes r_{i,t} with a step index, but each completion receives one scalar. The paper does not say what the step index means in a single completion.
• Algorithm 1 is ambiguous. The loop bound changes inside the loop when early termination sets K. Whether the bound updates is not stated. The area threshold T = 0.2 is undefined as relative or absolute, and how extra boxes are drawn is only described in words.
• SIF-50K size. The dataset table sums to 49,681 items, while the paper calls it 50K § Appendix (Training Dataset). Birds-200-2011 and Flickr30k are both listed as 9,782, which may be a copy error.
• Table and text disagree. Table 5 labels the grounding-reward ablation as "w/o r_bbox" (0.695), but the text reports 0.723 for removing only the correction term tab. 5. The text also gives 0.674 for VisCoT with LLaVA, while Table 2 gives 0.672 § Experiments (Effectiveness with RGB-Depth Input) tab. 2.
• Not every general benchmark improves. With the LLaVA base, VisCoT scores higher on MME-C (308 vs 295) and MMB-D (66.6 vs 62.8) tab. 4. The paper describes general results as "not degraded"; that is accurate, but not uniform.
• Small spatial subsets. SpatialBench has 34, 40, 20 and 40 items per category § Appendix (Benchmarks). One item moves position accuracy by about 3 points and counting accuracy by 5 points, so single-category gaps are noisy.
• Progress reward needs memory. The progress terms depend on the previous step's value stored in a module-level list [code: open_r1/trainer/record.py#L1-L2]. Under multi-process training, each process holds its own copy (code reading only). Resumed runs will lose it unless it is restored.
• Progress reward gating. The released code requires the new judge score to be at least the stored score before adding the gain [code: open_r1/vlm_modules/qwen_module.py#L870-L871]. The paper has no such gate, so reward distributions will differ.

Lineage

• Builds on: GRPO (DeepSeekMath, arXiv:2402.03300) for the RL objective and the group-normalised advantage § Methodology (Reinforcement Learning with GRPO-SIF). Visual CoT (arXiv:2403.16999) supplies the source data and the benchmark subsets § Appendix (Benchmarks). Depth Anything V2 alg. 1 supplies depth maps. The KL term uses the unbiased estimator the paper cites to Schulman (2020) § Methodology (Reinforcement Learning with GRPO-SIF).
• Compared against: VisRL (arXiv:2503.07523), VisCoT (arXiv:2403.16999), V* (CVPR), SEAL, SpatialBot (arXiv:2406.13642), SSR (arXiv:2505.12448), VLM-R1, Grounding-DINO, UNINEXT, and ChatGPT-o3 (closed).
• Superseded by: not known from content already in this wiki.