wikillm

Paper · 2022-07-20 · cs.CV, cs.AI, cs.CL

GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features

Transformer-only captioner that decodes from two visual inputs at once: DETR-style region features and self-attended grid features, both from one Swin-Base backbone. A gated parallel cross-attention fuses them. Trained with cross-entropy, then CIDEr-D self-critical training. Author-reported 144.2 CIDEr on COCO Karpathy test, above methods without V&L pretraining and SimVLM-huge (143.3).

Image captioningCross-attentionDirect set predictionimage-captioningtransformerobject-detection

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Most image captioning systems describe a picture from a list of detected objects, each with a box around it. That is good at naming things, but the detector can miss objects, the boxes ignore the space between objects, and running the detector is slow. A second, cheaper view is a regular grid of features that covers the whole image, which keeps some of the context the boxes lose.

GRIT uses both views. One image backbone feeds two branches: one turns it into object-like features with a detector that does not need a separate duplicate-removal step, and the other refines the grid of features with self-attention. A small text decoder writes the caption word by word, and at each word it decides how much to look at the objects and how much at the grid. Every part is built from Transformer blocks, so the whole model can be trained in one go, detector included.

The authors report that this is both more accurate and faster than earlier object-based captioners. Those numbers are theirs, measured on their own setup.

TL;DR

  • Image captioner made only of Transformer blocks, with a Swin-Base backbone §3.1.1.
  • It uses two visual inputs at once: region features from a DETR-style detector §3.1.2 and grid features from the same backbone §3.1.3.
  • A gated "parallel" cross-attention in the decoder fuses them; this design beat concatenation and sequential variants in the authors' ablation tab. 1.
  • Author-reported: 144.2 CIDEr on the COCO Karpathy test split without vision-language pretraining, above SimVLM-huge (143.3) tab. 2. Feature extraction is reported at 31 ms per image app. B.
  • Official code exists, but it has no license file, and several details differ from the paper (see Paper vs code).

Problem

Region features from a CNN detector (Faster R-CNN in earlier work) are the standard input for captioning. They have three costs the authors name: they miss context between objects, a detector can miss or invent objects, and running the detector is expensive. Non-maximum suppression in the detector also blocks end-to-end training, so detector and captioner are usually trained separately §1. Grid features avoid these issues but carry no object-level information. Combining the two was open: the authors note that how to extract and fuse them was unclear §1, §2.

Method

Overview. A Swin-Base backbone produces four feature maps at strides 8 to 64 §3.1.1. Two modules read them: a DETR-style object detector (only its decoder half, Deformable DETR) that produces region features, and a self-attention network that produces grid features §3.1.2, §3.1.3. A caption decoder with three layers consumes both and generates the caption autoregressively §3.2.1. The detector can be pretrained on object detection (and optionally attribute prediction) before captioning training, and then trained jointly with the captioner §3.1.2, §4.3.1, §4.3.2.

Backbone §3.1.1. Swin Transformer's patch merging halves the spatial size after each stage. The four stage outputs are kept as multi-scale maps, and the last map is also reused as the input to the grid network.

Region features §3.1.2. Each map is linearly projected to d = 512 (the paper calls this a linear transformation; the code uses a 1×1 convolution plus group normalisation detector.py). N = 150 learnable object queries are updated by 6 deformable decoder layers. The last-layer output, N×d, is the region feature set R. The detector is pretrained first, then fine-tuned with the attribute term added to the loss §3.1.2.

Pseudocode
# Detection + attribute loss for one image  [eq. 1] [§3.1.2]
# sigma: Hungarian matching of queries to ground-truth objects  [§3.1.2]
# c_i: class, b_i: box, a_i: attribute of ground-truth object i
def detection_attribute_loss(probs, boxes, attrs, targets, sigma):
    total = 0
    for i in range(N):
        total = total - log(probs_class[sigma[i]][c_i])
        if c_i is not "no object":
            total = total + box_loss(b_i, boxes[sigma[i]])
        total = total - log(probs_attr[sigma[i]][a_i])
    return total

Grid features §3.1.3. The last Swin map (M = H/64 × W/64 tokens) is linearly projected to d and passed through 3 standard self-attention layers. The grid feature set G is the output of the last layer transformer.py.

Caption decoder §3.2.1. Word embeddings plus a frozen sinusoidal position embedding of the time step go through 3 layers. Each layer has masked self-attention over the words, then a cross-attention sub-layer over the visual features, then a feed-forward sub-layer. A final linear layer and log-softmax predict the next word.

Cross-attention designs §3.2.2, tab. 1. The paper compares three ways to use R and G:

  • Concatenated: one attention over the joined set [R; G].
  • Sequential: one attention over G then R (or R then G).
  • Parallel (chosen): two separate attentions, one over R and one over G, each from the same word queries. Each output is gated by a sigmoid computed from that output and the word state, and the two gated outputs are summed with the word state and normalised.
Pseudocode
# Parallel gated cross-attention for one decoder layer  [§3.2.2] [eq. 2-4]
# words: x' (B, T, d) after masked self-attention
# region_feats R, grid_feats G: (B, N, d) and (B, M, d)
def parallel_cross_attention(words, region_feats, grid_feats):
    # Two independent multi-head attentions, queries = words  [§3.2.2]
    attended_region = mha_region(words, region_feats, region_feats)
    attended_grid = mha_grid(words, grid_feats, grid_feats)
    # One sigmoid gate per branch, from its output and the word state  [eq. 2-3]
    gate_grid = sigmoid(W_g @ concat(attended_grid, words) + b_g)
    gate_region = sigmoid(W_r @ concat(attended_region, words) + b_r)
    # Gated sum, added to the word state, then layer norm  [eq. 4]
    fused = gate_grid * attended_grid + gate_region * attended_region
    return layer_norm(fused + words)

The gate equations, in the paper's notation:

c^g_i = \mathrm{sigmoid}(W^g[a^g_i; x'_i] + b^g), \qquad c^r_i = \mathrm{sigmoid}(W^r[a^r_i; x'_i] + b^r) eq. 2 eq. 3

a^{(l)}_i = \mathrm{LN}\left(c^g_i \otimes a^g_i + c^r_i \otimes a^r_i + x'_i\right) eq. 4

Training, stage 1 (detector) §3.1.2, §4.3.1. Pretrain with the detection loss on Visual Genome, or on a four-dataset union (COCO, Visual Genome, Open Images, Objects365) §4.1.1. Then fine-tune on detection plus attributes on Visual Genome. Box loss is a weighted L1 plus GIoU term app. A.

Training, stage 2 (captioner) §3.2.3, §4.3.2. Cross-entropy first:

\mathcal{L}_{XE}(\theta) = -\sum_{t=1}^{T} \log p_\theta\left(x^*_t \mid x^*_{0:t-1}\right) eq. 5

Then self-critical sequence training with CIDEr-D as reward, where the baseline is the mean reward of the k beam-search captions for the same image:

\mathcal{L}_{RL}(\theta) = -\frac{1}{k}\sum_{i=1}^{k} \left(r(\mathbf{w}^i) - b\right)\log p(\mathbf{w}^i) eq. 6

Pseudocode
# Cross-entropy pre-training step  [§3.2.3] [eq. 5]
# log_probs: (T, V) decoder outputs; gt_words: (T,) ground-truth ids
def xe_loss(log_probs, gt_words):
    total = 0
    for t in range(T):
        total = total - log_probs[t][gt_words[t]]
    return total
Pseudocode
# Self-critical update for one image  [§3.2.3] [eq. 6]
# seq_logprobs: (k,) log p(w^i) of the k beam captions  [§4.3.2]
# rewards: (k,) CIDEr-D of those captions
def scst_loss(seq_logprobs, rewards):
    # Baseline is the mean reward over the k captions  [§3.2.3]
    baseline = mean(rewards)
    total = 0
    for i in range(k):
        total = total - (rewards[i] - baseline) * seq_logprobs[i]
    return total / k

Inference §4.3.2. Beam search with beam size 5 and maximum length 20. The same beam settings are used for self-critical training.

Hyperparameters

NameValueWhereNotes
Model width d512§4.2.2
Attention heads8§4.2.2
Dropout0.2§4.2.2Paper: on the output of each attention and feed-forward sub-layer. Code also applies it to attention weights attention.py; the config key attn_dropout (also 0.2) is never read coco_config.yaml
Detector decoder layers L_r6§4.2.2Config: num_layers: 6 train_config.yaml
Grid network layers L_g3§4.2.2
Caption decoder layers L_c3§4.2.2
Object queries N150app. AAblation over 50, 100, 150 tab. 1
Sampling points per query4app. A
Detector feed-forward widthnot statedCode: 1024 train_config.yaml
BackboneSwin-Base 384, ImageNet-21K initialisationapp. AConfig sets frozen_stages: 2 coco_config.yaml
Vocabularywords seen fewer than 5 times excluded§4.2.2Code uses a shipped vocab of 10,201 entries coco_config.yaml; cut-off not checked
Text preprocessinglowercase, strip punctuation, spaCy tokenisation§4.2.2
Detector pretrain, VG setting150,000 iterations, batch 32, Adam, lr 1e-4, ÷10 at iteration 120,000, short side ≤ 800, long side ≤ 1333§4.3.1Detector training config in code uses epochs, not iterations train_config.yaml
Detector pretrain, four-dataset setting125,000 iterations, batch 256, lr 1e-4, ÷10 at iteration 100,000§4.3.1Code: batch 4 × world size 64 = 256 train_config.yaml
Attribute fine-tune5 epochs on Visual Genome, lr 1e-5§4.3.1Shipped detection config disables the attribute head train_config.yaml
Box loss weightsL1 5, GIoU 2app. ACode adds a class weight of 2 that eq. 1 does not show train_config.yaml
Captioning input sizeshort side 384, long side 640§4.3.2coco_config.yaml
XE epochs10§4.3.2
XE learning rate, grid network and decoderwarm up from 1e-5 to 1e-4 during epoch 1§4.3.2Code keeps 1e-4 afterwards (min = max) coco_config.yaml
XE learning rate, backbone and detectorfixed 1e-5§4.3.2coco_config.yaml
SCST epochs10§4.3.2
SCST learning rate, whole modelfixed 5e-6§4.3.2coco_config.yaml
SCST beam and lengthbeam 5, max length 20§4.3.2coco_config.yaml
OptimiserAdam§4.3.2Betas 0.9 and 0.99, weight decay 0.01 in code; not stated in paper coco_config.yaml
Batch size128§4.3.2Code: 16 per process × 8 processes coco_config.yaml
Data augmentationnot statedCode enables RandAugment coco_config.yaml
ArtEmis training5 XE epochs, then 5 CIDEr-D epochs§4.6Emotion head: linear map to 8 dimensions app. B

Results

Author-reported. Setup: COCO Karpathy split for offline evaluation (5,000 test images) §4.1.2; "GRIT" uses a detector pretrained on Visual Genome, "GRIT†" on the four-dataset union §4.5.1. Metrics are BLEU-4 (B4), METEOR (M), ROUGE-L (R), CIDEr (C) and SPICE (S).

Offline, COCO Karpathy test split tab. 2:

MethodVisual inputV&L pretrainingB4MRCS
SimVLM-hugegrid1.8B pairs40.633.7-143.325.4
DLCTregion + gridno39.829.559.1133.823.0
M² Transformerregionno39.129.258.6131.222.6
GRITregion + gridno41.930.560.5142.224.2
GRIT†region + gridno42.430.660.7144.224.3

Online, COCO evaluation server, 40K test images tab. 3. CIDEr with five references (c5) and forty references (c40):

MethodEnsembleC (c5)C (c40)
VinVL-largeno134.7138.7
DLCTsix models133.3135.4
GRIT†no138.3141.8
GRIT†six models141.3143.8

The authors state that their single model outperforms all published methods that use ensembles, and that their method has the best score on every metric §4.5.2.

Ablations on the COCO test split, all with XE then CIDEr tab. 1:

  • Detector pretraining: ImageNet-only grid features reach CIDEr 135.5; Visual Genome detector 142.3; four-dataset detector 144.2 tab. 1a.
  • Region count: 50 → 141.4, 100 → 141.8, 150 → 142.3 (Visual Genome detector) tab. 1a.
  • End-to-end training: 144.2 CIDEr with it, 139.6 without tab. 1a.
  • Cross-attention: concatenated 142.1 (grid), 142.9 (region), 143.1 (both); sequential 144.0 (grid then region) and 143.6 (region then grid); parallel with sigmoid gates 144.2, parallel with identity gates 143.9 tab. 1b.

Other datasets tab. 4: ArtEmis test split (GRIT†, no emotion grounding): B1 70.1, B4 11.3, M 16.8, R 33.3 tab. 4a. nocaps validation, zero-shot from COCO (GRIT†): overall CIDEr 90.2, SPICE 12.8, in-domain CIDEr 105.9 tab. 4b.

Efficiency §4.7: feature extraction of 31 ms per image for GRIT, against 304 ms (VinVL-large) and 736 ms (M² Transformer) in the authors' measurements. Caption generation of 138 ms against 542 ms (VinVL-large) and 174 ms (M²) app. B. Per-image time falls to about 32 ms with batch size 32 or more on one V100 §4.7.

Paper vs code

Official code: https://github.com/davidnvq/grit, cloned at 4664d69fc0b01a5459cfae73f1fd68045ce1a6ea. The link is given in the arXiv comments, which makes it an author-provided link. The repository has no LICENSE file. Its file headers say "Licensed under the Creative Commons Attribution." without a version, so the license is not stated. No code is quoted here.

  • Gate sharing. The paper uses separate learnable projections for the two gates §3.2.2, eq. 2-3. The code computes the grid-branch gate with the region branch's projection fc_alpha1; fc_alpha2 is defined and initialised but never used cap_generator.py.
  • Fusion arithmetic. The paper's fused output is a plain sum of gated outputs plus the word state, then layer norm eq. 4. The code divides the gated sum by √2 cap_generator.py, has no layer norm there, and feeds it to a feed-forward block that adds its own residual and layer norm pos_embed.py. The residual in each attention output is also applied inside the attention block, so the attended values are already layer-normalised attention.py.
  • Grid projection. The paper describes a linear map for the grid input §3.1.3. The code applies ReLU, dropout and layer norm after that linear map grid_net.py.
  • Grid memory slots. The config sets grid_net.n_memories: 1 coco_config.yaml. The Transformer does not pass this value to the grid network transformer.py, so the shipped model has no memory slots. The paper describes no memory slots.
  • SCST loss. The paper uses the sequence log-probability, i.e. the sum over tokens eq. 6. The code averages the per-token log-probabilities over the length axis before multiplying by the advantage caption_engine.py. Per-token values come from the beam search transformer.py; the comment above the stacking step says "sum" transformer.py, but the averaging is what the loss does.
  • Warm-up. The paper says warm-up goes from 1e-5 to 1e-4 during the first epoch §4.3.2. The code's warm-up starts at 1e-5 + 0.1 × (1e-4 − 1e-5) ≈ 1.9e-5 cap_scheduler.py.
  • Attribute loss. Eq. 1 includes the attribute term, but the shipped detector config disables the attribute head and sets its loss weight to 0 train_config.yaml.
  • Detector schedule. The paper gives 150k or 125k iterations with a drop at 120k or 100k §4.3.1. The shipped detector config uses 50 epochs, a learning rate of 1e-5 (backbone 2e-5), and a drop at epoch 40 train_config.yaml.
  • Detector input projection. The code uses a 1×1 convolution with group normalisation, not a plain linear map detector.py.
  • Evaluation metric. The SCST reward is CIDEr-D with a Gaussian length penalty (sigma 6) and clipping cider_scorer.py. Its document frequencies come from the training captions train_caption.py.
  • Training stages. The default config runs 10 XE epochs followed by 10 SCST epochs, with no frozen phase coco_config.yaml. At the start of each SCST epoch the code reloads checkpoint_best_valid.pth, not the last XE state train_caption.py.
  • Cost. The README gives about 16 hours on 8 A100 GPUs for training README.md. The paper reports V100 timings for inference only §4.7.

Pitfalls

  • Ablation text vs table. The text says end-to-end training moves CIDEr "from 139.6 to 144.3" §4.4, but Table 1 gives 144.2 tab. 1a. Use the table value.
  • Parallel gate. Implement two separate gate projections. The shipped code reuses one for both branches, so reproductions that copy the code will not match the paper.
  • Length normalisation in SCST. Averaging over tokens (code) and summing over tokens (paper eq. 6) give different gradient scales for captions of different length. Pick one and say which.
  • Default checkpoint path. The detector checkpoint in the shipped config is empty coco_config.yaml. A run without a pretrained detector trains from the Swin backbone only. The README links pretrained checkpoints on Google Drive; their licence was not checked.
  • Costs the abstract hides. Training needs the detector and backbone, and end-to-end training is what the authors credit for part of the gain tab. 1a. Inference also runs both branches, so the reported speed is for their hardware and input sizes (384×640 for GRIT against 800×1333 for the baselines) §4.7.
  • Freezing. The config supports caching detector features for a frozen XE phase, but its default is 0 epochs coco_config.yaml. The paper's end-to-end setting is the default.
  • Under-specified details. Not stated in the paper: Adam betas, weight decay, decoder feed-forward width, data augmentation, and the SCST batch size. The code sets values for all of these (see Hyperparameters).
  • Dropout. The code also applies dropout to the attention weights, and the attn_dropout config key is unused. Check which of these you want.

Lineage

  • Builds on: Deformable DETR for the region branch §3.1.2; Swin Transformer for the backbone §3.1.1; the standard Transformer decoder §3.2.1; self-critical sequence training §3.2.3. The code states it is modified from Meshed-Memory Transformer attention.py.
  • Compared against: grid-based methods (SAT, SCST, RSTNet), region-based methods (Up-Down, AoA, M² Transformer, X-LAN, DLCT, and others), and vision-language pretrained models (VinVL, Oscar, UVLP, SimVLM) tab. 2.
  • Superseded by: not stated in this card's sources.

Source and license

Adapted from “GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features” by Van-Quang Nguyen, Masanori Suganuma, Takayuki Okatani (arXiv:2207.09666), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/davidnvq/grit at commit 4664d69fc0b0 (not stated (no LICENSE file in repository; file headers say 'Licensed under the Creative Commons Attribution.')). Any quoted code is shown under that license.