Paper · 2022-07-20 · cs.CV, cs.AI, cs.CL
GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features
Transformer-only captioner that decodes from two visual inputs at once: DETR-style region features and self-attended grid features, both from one Swin-Base backbone. A gated parallel cross-attention fuses them. Trained with cross-entropy, then CIDEr-D self-critical training. Author-reported 144.2 CIDEr on COCO Karpathy test, above methods without V&L pretraining and SimVLM-huge (143.3).
Image captioningCross-attentionDirect set predictionimage-captioningtransformerobject-detection
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Most image captioning systems describe a picture from a list of detected objects, each with a box around it. That is good at naming things, but the detector can miss objects, the boxes ignore the space between objects, and running the detector is slow. A second, cheaper view is a regular grid of features that covers the whole image, which keeps some of the context the boxes lose.
GRIT uses both views. One image backbone feeds two branches: one turns it into object-like features with a detector that does not need a separate duplicate-removal step, and the other refines the grid of features with self-attention. A small text decoder writes the caption word by word, and at each word it decides how much to look at the objects and how much at the grid. Every part is built from Transformer blocks, so the whole model can be trained in one go, detector included.
The authors report that this is both more accurate and faster than earlier object-based captioners. Those numbers are theirs, measured on their own setup.
TL;DR
- Image captioner made only of Transformer blocks, with a Swin-Base backbone §3.1.1.
- It uses two visual inputs at once: region features from a DETR-style detector §3.1.2 and grid features from the same backbone §3.1.3.
- A gated "parallel" cross-attention in the decoder fuses them; this design beat concatenation and sequential variants in the authors' ablation tab. 1.
- Author-reported: 144.2 CIDEr on the COCO Karpathy test split without vision-language pretraining, above SimVLM-huge (143.3) tab. 2. Feature extraction is reported at 31 ms per image app. B.
- Official code exists, but it has no license file, and several details differ from the paper (see Paper vs code).
Problem
Region features from a CNN detector (Faster R-CNN in earlier work) are the standard input for captioning. They have three costs the authors name: they miss context between objects, a detector can miss or invent objects, and running the detector is expensive. Non-maximum suppression in the detector also blocks end-to-end training, so detector and captioner are usually trained separately §1. Grid features avoid these issues but carry no object-level information. Combining the two was open: the authors note that how to extract and fuse them was unclear §1, §2.
Method
Overview. A Swin-Base backbone produces four feature maps at strides 8 to 64 §3.1.1. Two modules read them: a DETR-style object detector (only its decoder half, Deformable DETR) that produces region features, and a self-attention network that produces grid features §3.1.2, §3.1.3. A caption decoder with three layers consumes both and generates the caption autoregressively §3.2.1. The detector can be pretrained on object detection (and optionally attribute prediction) before captioning training, and then trained jointly with the captioner §3.1.2, §4.3.1, §4.3.2.
Backbone §3.1.1. Swin Transformer's patch merging halves the spatial size after each stage. The four stage outputs are kept as multi-scale maps, and the last map is also reused as the input to the grid network.
Region features §3.1.2. Each map is linearly projected to d = 512 (the paper calls this a linear transformation; the code uses a 1×1 convolution plus group normalisation detector.py). N = 150 learnable object queries are updated by 6 deformable decoder layers. The last-layer output, N×d, is the region feature set R. The detector is pretrained first, then fine-tuned with the attribute term added to the loss §3.1.2.
# Detection + attribute loss for one image [eq. 1] [§3.1.2]
# sigma: Hungarian matching of queries to ground-truth objects [§3.1.2]
# c_i: class, b_i: box, a_i: attribute of ground-truth object i
def detection_attribute_loss(probs, boxes, attrs, targets, sigma):
total = 0
for i in range(N):
total = total - log(probs_class[sigma[i]][c_i])
if c_i is not "no object":
total = total + box_loss(b_i, boxes[sigma[i]])
total = total - log(probs_attr[sigma[i]][a_i])
return totalGrid features §3.1.3. The last Swin map (M = H/64 × W/64 tokens) is linearly projected to d and passed through 3 standard self-attention layers. The grid feature set G is the output of the last layer transformer.py.
Caption decoder §3.2.1. Word embeddings plus a frozen sinusoidal position embedding of the time step go through 3 layers. Each layer has masked self-attention over the words, then a cross-attention sub-layer over the visual features, then a feed-forward sub-layer. A final linear layer and log-softmax predict the next word.
Cross-attention designs §3.2.2, tab. 1. The paper compares three ways to use R and G:
- Concatenated: one attention over the joined set [R; G].
- Sequential: one attention over G then R (or R then G).
- Parallel (chosen): two separate attentions, one over R and one over G, each from the same word queries. Each output is gated by a sigmoid computed from that output and the word state, and the two gated outputs are summed with the word state and normalised.
# Parallel gated cross-attention for one decoder layer [§3.2.2] [eq. 2-4]
# words: x' (B, T, d) after masked self-attention
# region_feats R, grid_feats G: (B, N, d) and (B, M, d)
def parallel_cross_attention(words, region_feats, grid_feats):
# Two independent multi-head attentions, queries = words [§3.2.2]
attended_region = mha_region(words, region_feats, region_feats)
attended_grid = mha_grid(words, grid_feats, grid_feats)
# One sigmoid gate per branch, from its output and the word state [eq. 2-3]
gate_grid = sigmoid(W_g @ concat(attended_grid, words) + b_g)
gate_region = sigmoid(W_r @ concat(attended_region, words) + b_r)
# Gated sum, added to the word state, then layer norm [eq. 4]
fused = gate_grid * attended_grid + gate_region * attended_region
return layer_norm(fused + words)The gate equations, in the paper's notation:
c^g_i = \mathrm{sigmoid}(W^g[a^g_i; x'_i] + b^g), \qquad c^r_i = \mathrm{sigmoid}(W^r[a^r_i; x'_i] + b^r) eq. 2 eq. 3
a^{(l)}_i = \mathrm{LN}\left(c^g_i \otimes a^g_i + c^r_i \otimes a^r_i + x'_i\right) eq. 4
Training, stage 1 (detector) §3.1.2, §4.3.1. Pretrain with the detection loss on Visual Genome, or on a four-dataset union (COCO, Visual Genome, Open Images, Objects365) §4.1.1. Then fine-tune on detection plus attributes on Visual Genome. Box loss is a weighted L1 plus GIoU term app. A.
Training, stage 2 (captioner) §3.2.3, §4.3.2. Cross-entropy first:
\mathcal{L}_{XE}(\theta) = -\sum_{t=1}^{T} \log p_\theta\left(x^*_t \mid x^*_{0:t-1}\right) eq. 5
Then self-critical sequence training with CIDEr-D as reward, where the baseline is the mean reward of the k beam-search captions for the same image:
\mathcal{L}_{RL}(\theta) = -\frac{1}{k}\sum_{i=1}^{k} \left(r(\mathbf{w}^i) - b\right)\log p(\mathbf{w}^i) eq. 6
# Cross-entropy pre-training step [§3.2.3] [eq. 5]
# log_probs: (T, V) decoder outputs; gt_words: (T,) ground-truth ids
def xe_loss(log_probs, gt_words):
total = 0
for t in range(T):
total = total - log_probs[t][gt_words[t]]
return total# Self-critical update for one image [§3.2.3] [eq. 6]
# seq_logprobs: (k,) log p(w^i) of the k beam captions [§4.3.2]
# rewards: (k,) CIDEr-D of those captions
def scst_loss(seq_logprobs, rewards):
# Baseline is the mean reward over the k captions [§3.2.3]
baseline = mean(rewards)
total = 0
for i in range(k):
total = total - (rewards[i] - baseline) * seq_logprobs[i]
return total / kInference §4.3.2. Beam search with beam size 5 and maximum length 20. The same beam settings are used for self-critical training.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Model width d | 512 | §4.2.2 | |
| Attention heads | 8 | §4.2.2 | |
| Dropout | 0.2 | §4.2.2 | Paper: on the output of each attention and feed-forward sub-layer. Code also applies it to attention weights attention.py; the config key attn_dropout (also 0.2) is never read coco_config.yaml |
| Detector decoder layers L_r | 6 | §4.2.2 | Config: num_layers: 6 train_config.yaml |
| Grid network layers L_g | 3 | §4.2.2 | |
| Caption decoder layers L_c | 3 | §4.2.2 | |
| Object queries N | 150 | app. A | Ablation over 50, 100, 150 tab. 1 |
| Sampling points per query | 4 | app. A | |
| Detector feed-forward width | not stated | Code: 1024 train_config.yaml | |
| Backbone | Swin-Base 384, ImageNet-21K initialisation | app. A | Config sets frozen_stages: 2 coco_config.yaml |
| Vocabulary | words seen fewer than 5 times excluded | §4.2.2 | Code uses a shipped vocab of 10,201 entries coco_config.yaml; cut-off not checked |
| Text preprocessing | lowercase, strip punctuation, spaCy tokenisation | §4.2.2 | |
| Detector pretrain, VG setting | 150,000 iterations, batch 32, Adam, lr 1e-4, ÷10 at iteration 120,000, short side ≤ 800, long side ≤ 1333 | §4.3.1 | Detector training config in code uses epochs, not iterations train_config.yaml |
| Detector pretrain, four-dataset setting | 125,000 iterations, batch 256, lr 1e-4, ÷10 at iteration 100,000 | §4.3.1 | Code: batch 4 × world size 64 = 256 train_config.yaml |
| Attribute fine-tune | 5 epochs on Visual Genome, lr 1e-5 | §4.3.1 | Shipped detection config disables the attribute head train_config.yaml |
| Box loss weights | L1 5, GIoU 2 | app. A | Code adds a class weight of 2 that eq. 1 does not show train_config.yaml |
| Captioning input size | short side 384, long side 640 | §4.3.2 | coco_config.yaml |
| XE epochs | 10 | §4.3.2 | |
| XE learning rate, grid network and decoder | warm up from 1e-5 to 1e-4 during epoch 1 | §4.3.2 | Code keeps 1e-4 afterwards (min = max) coco_config.yaml |
| XE learning rate, backbone and detector | fixed 1e-5 | §4.3.2 | coco_config.yaml |
| SCST epochs | 10 | §4.3.2 | |
| SCST learning rate, whole model | fixed 5e-6 | §4.3.2 | coco_config.yaml |
| SCST beam and length | beam 5, max length 20 | §4.3.2 | coco_config.yaml |
| Optimiser | Adam | §4.3.2 | Betas 0.9 and 0.99, weight decay 0.01 in code; not stated in paper coco_config.yaml |
| Batch size | 128 | §4.3.2 | Code: 16 per process × 8 processes coco_config.yaml |
| Data augmentation | not stated | Code enables RandAugment coco_config.yaml | |
| ArtEmis training | 5 XE epochs, then 5 CIDEr-D epochs | §4.6 | Emotion head: linear map to 8 dimensions app. B |
Results
Author-reported. Setup: COCO Karpathy split for offline evaluation (5,000 test images) §4.1.2; "GRIT" uses a detector pretrained on Visual Genome, "GRIT†" on the four-dataset union §4.5.1. Metrics are BLEU-4 (B4), METEOR (M), ROUGE-L (R), CIDEr (C) and SPICE (S).
Offline, COCO Karpathy test split tab. 2:
| Method | Visual input | V&L pretraining | B4 | M | R | C | S |
|---|---|---|---|---|---|---|---|
| SimVLM-huge | grid | 1.8B pairs | 40.6 | 33.7 | - | 143.3 | 25.4 |
| DLCT | region + grid | no | 39.8 | 29.5 | 59.1 | 133.8 | 23.0 |
| M² Transformer | region | no | 39.1 | 29.2 | 58.6 | 131.2 | 22.6 |
| GRIT | region + grid | no | 41.9 | 30.5 | 60.5 | 142.2 | 24.2 |
| GRIT† | region + grid | no | 42.4 | 30.6 | 60.7 | 144.2 | 24.3 |
Online, COCO evaluation server, 40K test images tab. 3. CIDEr with five references (c5) and forty references (c40):
| Method | Ensemble | C (c5) | C (c40) |
|---|---|---|---|
| VinVL-large | no | 134.7 | 138.7 |
| DLCT | six models | 133.3 | 135.4 |
| GRIT† | no | 138.3 | 141.8 |
| GRIT† | six models | 141.3 | 143.8 |
The authors state that their single model outperforms all published methods that use ensembles, and that their method has the best score on every metric §4.5.2.
Ablations on the COCO test split, all with XE then CIDEr tab. 1:
- Detector pretraining: ImageNet-only grid features reach CIDEr 135.5; Visual Genome detector 142.3; four-dataset detector 144.2 tab. 1a.
- Region count: 50 → 141.4, 100 → 141.8, 150 → 142.3 (Visual Genome detector) tab. 1a.
- End-to-end training: 144.2 CIDEr with it, 139.6 without tab. 1a.
- Cross-attention: concatenated 142.1 (grid), 142.9 (region), 143.1 (both); sequential 144.0 (grid then region) and 143.6 (region then grid); parallel with sigmoid gates 144.2, parallel with identity gates 143.9 tab. 1b.
Other datasets tab. 4: ArtEmis test split (GRIT†, no emotion grounding): B1 70.1, B4 11.3, M 16.8, R 33.3 tab. 4a. nocaps validation, zero-shot from COCO (GRIT†): overall CIDEr 90.2, SPICE 12.8, in-domain CIDEr 105.9 tab. 4b.
Efficiency §4.7: feature extraction of 31 ms per image for GRIT, against 304 ms (VinVL-large) and 736 ms (M² Transformer) in the authors' measurements. Caption generation of 138 ms against 542 ms (VinVL-large) and 174 ms (M²) app. B. Per-image time falls to about 32 ms with batch size 32 or more on one V100 §4.7.
Paper vs code
Official code: https://github.com/davidnvq/grit, cloned at 4664d69fc0b01a5459cfae73f1fd68045ce1a6ea. The link is given in the arXiv comments, which makes it an author-provided link. The repository has no LICENSE file. Its file headers say "Licensed under the Creative Commons Attribution." without a version, so the license is not stated. No code is quoted here.
- Gate sharing. The paper uses separate learnable projections for the two gates §3.2.2, eq. 2-3. The code computes the grid-branch gate with the region branch's projection
fc_alpha1;fc_alpha2is defined and initialised but never used cap_generator.py. - Fusion arithmetic. The paper's fused output is a plain sum of gated outputs plus the word state, then layer norm eq. 4. The code divides the gated sum by √2 cap_generator.py, has no layer norm there, and feeds it to a feed-forward block that adds its own residual and layer norm pos_embed.py. The residual in each attention output is also applied inside the attention block, so the attended values are already layer-normalised attention.py.
- Grid projection. The paper describes a linear map for the grid input §3.1.3. The code applies ReLU, dropout and layer norm after that linear map grid_net.py.
- Grid memory slots. The config sets
grid_net.n_memories: 1coco_config.yaml. The Transformer does not pass this value to the grid network transformer.py, so the shipped model has no memory slots. The paper describes no memory slots. - SCST loss. The paper uses the sequence log-probability, i.e. the sum over tokens eq. 6. The code averages the per-token log-probabilities over the length axis before multiplying by the advantage caption_engine.py. Per-token values come from the beam search transformer.py; the comment above the stacking step says "sum" transformer.py, but the averaging is what the loss does.
- Warm-up. The paper says warm-up goes from 1e-5 to 1e-4 during the first epoch §4.3.2. The code's warm-up starts at 1e-5 + 0.1 × (1e-4 − 1e-5) ≈ 1.9e-5 cap_scheduler.py.
- Attribute loss. Eq. 1 includes the attribute term, but the shipped detector config disables the attribute head and sets its loss weight to 0 train_config.yaml.
- Detector schedule. The paper gives 150k or 125k iterations with a drop at 120k or 100k §4.3.1. The shipped detector config uses 50 epochs, a learning rate of 1e-5 (backbone 2e-5), and a drop at epoch 40 train_config.yaml.
- Detector input projection. The code uses a 1×1 convolution with group normalisation, not a plain linear map detector.py.
- Evaluation metric. The SCST reward is CIDEr-D with a Gaussian length penalty (sigma 6) and clipping cider_scorer.py. Its document frequencies come from the training captions train_caption.py.
- Training stages. The default config runs 10 XE epochs followed by 10 SCST epochs, with no frozen phase coco_config.yaml. At the start of each SCST epoch the code reloads
checkpoint_best_valid.pth, not the last XE state train_caption.py. - Cost. The README gives about 16 hours on 8 A100 GPUs for training README.md. The paper reports V100 timings for inference only §4.7.
Pitfalls
- Ablation text vs table. The text says end-to-end training moves CIDEr "from 139.6 to 144.3" §4.4, but Table 1 gives 144.2 tab. 1a. Use the table value.
- Parallel gate. Implement two separate gate projections. The shipped code reuses one for both branches, so reproductions that copy the code will not match the paper.
- Length normalisation in SCST. Averaging over tokens (code) and summing over tokens (paper eq. 6) give different gradient scales for captions of different length. Pick one and say which.
- Default checkpoint path. The detector checkpoint in the shipped config is empty coco_config.yaml. A run without a pretrained detector trains from the Swin backbone only. The README links pretrained checkpoints on Google Drive; their licence was not checked.
- Costs the abstract hides. Training needs the detector and backbone, and end-to-end training is what the authors credit for part of the gain tab. 1a. Inference also runs both branches, so the reported speed is for their hardware and input sizes (384×640 for GRIT against 800×1333 for the baselines) §4.7.
- Freezing. The config supports caching detector features for a frozen XE phase, but its default is 0 epochs coco_config.yaml. The paper's end-to-end setting is the default.
- Under-specified details. Not stated in the paper: Adam betas, weight decay, decoder feed-forward width, data augmentation, and the SCST batch size. The code sets values for all of these (see Hyperparameters).
- Dropout. The code also applies dropout to the attention weights, and the
attn_dropoutconfig key is unused. Check which of these you want.
Lineage
- Builds on: Deformable DETR for the region branch §3.1.2; Swin Transformer for the backbone §3.1.1; the standard Transformer decoder §3.2.1; self-critical sequence training §3.2.3. The code states it is modified from Meshed-Memory Transformer attention.py.
- Compared against: grid-based methods (SAT, SCST, RSTNet), region-based methods (Up-Down, AoA, M² Transformer, X-LAN, DLCT, and others), and vision-language pretrained models (VinVL, Oscar, UVLP, SimVLM) tab. 2.
- Superseded by: not stated in this card's sources.
Source and license
Adapted from “GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features” by Van-Quang Nguyen, Masanori Suganuma, Takayuki Okatani (arXiv:2207.09666), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/davidnvq/grit at commit 4664d69fc0b0 (not stated (no LICENSE file in repository; file headers say 'Licensed under the Creative Commons Attribution.')). Any quoted code is shown under that license.