TransVG: End-to-End Visual Grounding with Transformers

In plain words

Visual grounding means this: given a picture and a sentence that describes one object in it, output the box around that object. Earlier systems did this with many purpose-built parts. Some first proposed many candidate boxes and then picked the best one. Others added special modules that split the sentence into pieces or built a graph of objects. Each part encodes a guess about how language relates to pictures, and those guesses can suit one dataset's style and fail on another.

TransVG takes a simpler route. The picture becomes a sequence of visual pieces, the words become a sequence of language pieces, and one special learned token is added. All of these go into a single plain stack of transformer layers, where every piece can look at every other piece. The model therefore learns for itself which words go with which parts of the picture. The answer is read from the special token as four numbers for the box, so no candidate boxes are needed.

The point is that the design is much simpler, and the authors report state-of-the-art results on five standard benchmarks. Those numbers are theirs, and the card below gives the setups behind them.

TL;DR

TransVG is a transformer-only visual grounding model. Image tokens from a visual branch initialised from DETR, word tokens from BERT-base and a learnable [REG] token pass through a fusion stack of transformer encoder layers, and the [REG] output regresses one box §3.2. The paper trains with smooth-L1 plus GIoU §3.3, but the released code uses plain L1 loss_utils.py. Use this card when you want a direct box regressor without proposals or anchors, and expect the released weights to score below the paper's headline numbers on some splits tab. 2.

Problem

Visual grounding predicts the region of an image that a language expression refers to §1. Two-stage methods first generate region proposals and then pick the one that best matches the expression. One-stage methods predict over dense anchors inside a detector §1. Both families make the answer out of candidates, so results depend on the prior used to make proposals or anchors and on heuristics for assigning targets to candidates §1.

Multimodal fusion modules built for these pipelines use hand-designed structures such as query decomposition, scene graphs, modular attention networks, dynamic graphs or multi-modal trees §1. The authors argue that such structures overfit to particular query lengths and relationships and limit the interaction between visual and linguistic context §1. They also aim to learn this reasoning from a comparatively small amount of grounding data §2.2.

Method

The model has four parts: a visual branch, a linguistic branch, a visual-linguistic fusion module and a prediction head §3.2 fig. 2. Inference is only the forward pass; no post-processing is applied §4.2.

1. Visual branch §3.2. A ResNet backbone gives a 2048-channel feature map at 1/32 of the input size. A 1×1 convolution reduces it to 256 channels, and the map is flattened into N_v = H·W tokens. Six transformer encoder layers process these tokens, with sine position codes added to queries and keys in each layer. The encoder is initialised from DETR's backbone and encoder §4.2.
2. Linguistic branch §3.2. The expression is tokenised with [CLS] and [SEP] added at the ends, then passed through the first 12 layers of BERT-base, which give 768-dimensional tokens. Queries are padded to a fixed length N_l and padded words are masked §4.2.
3. Fusion input §3.2 eq. 4. Each modality is linearly projected to 256 channels. The visual tokens, the language tokens and one learnable [REG] token are concatenated in that order.
4. Fusion encoder §3.2 eq. 2 eq. 3. Six post-norm transformer encoder layers run over the joint sequence. Learnable position encodings are added to the input of each layer. Attention lets visual tokens attend to visual tokens, language tokens to visual tokens, and so on, without a predefined fusion rule §3.2.
5. Prediction head §3.2. The output state of [REG] goes into an MLP with two ReLU-activated hidden layers of 256 units and a linear output of 4 values, giving (x, y, w, h).
6. Loss §3.3 eq. 5. The predicted box is compared with the normalised ground-truth box using smooth-L1 and a GIoU loss, with weight λ = 1 §4.2.

Key equations, as typeset in the paper:

\text{Attn}(\bm{f}^q,\bm{f}^k,\bm{f}^v) = \text{softmax}\left(\frac{\bm{f}^q\bm{f}^k}{\sqrt{{d}^k}}\right)\cdot\bm{f}^v eq. 1 (the transpose on the key is missing as typeset; see Pitfalls)

\bm{x}'_n = \text{LN}(\bm{x}_n + \mathcal{F}_\text{MSA}(\bm{x}_n)) eq. 2

\bm{x}_{n+1} = \text{LN}(\bm{x}'_n + \mathcal{F}_\text{FFN}(\bm{x}'_n)) eq. 3

\bm{x}_0 = [\ p_v^1,\ \cdots,\ p_v^{N_v},\ p_l^1,\ \cdots,\ p_l^{N_l},\ p_r\ ] eq. 4

\mathcal{L} = \mathcal{L}_{\text{smooth-l1}}(\bm{b},\hat{\bm{b}}) + \lambda\cdot\mathcal{L}_{\text{giou}}(\bm{b},\hat{\bm{b}}) eq. 5

# TransVG forward pass  [§3.2] [eq. 4]
# Input:  image (3, 640, 640), padded; query_ids (N_l,) BERT token ids
# Output: box (x, y, w, h), normalised to the padded image
def transvg_forward(image, query_ids, image_mask, query_mask):
    # Visual branch: ResNet features, then 1x1 reduction to 256 channels
    # [§3.2]
    feature_map = resnet_backbone(image)              # (2048, 20, 20)
    reduced = conv_1x1(feature_map)                   # (256, 20, 20)
    # Flatten to N_v = 400 tokens; sine codes enter each layer [§3.2]
    visual_tokens = flatten(reduced)                  # (256, 400)
    visual_tokens = visual_encoder_6(visual_tokens, sine_codes, image_mask)
    # Linguistic branch: BERT-base, output of its 12th layer [§3.2]
    language_tokens = bert_layer_12(query_ids, query_mask)  # (768, N_l)
    # Project both modalities to 256 channels [§3.2]
    visual_tokens = linear_visual(visual_tokens)      # (256, 400)
    language_tokens = linear_language(language_tokens)  # (256, N_l)
    # Concatenate visual, language, then the learnable [REG] token [eq. 4]
    joint_tokens = concat(visual_tokens, language_tokens, reg_token)
    # Padded pixels and words are excluded from attention [§4.2]
    joint_mask = concat(image_mask, query_mask, [False])
    # Six fusion layers; positions are learned per layer [§3.2]
    for layer in fusion_encoder_layers:
        joint_tokens = encoder_layer(
            joint_tokens, learned_positions, joint_mask
        )
    # Regress the box from the output state of the [REG] token [§3.2]
    reg_state = joint_tokens[-1]
    box = regression_mlp(reg_state)                   # (4,)
    return box

# One post-norm fusion encoder layer  [eq. 2] [eq. 3]
def encoder_layer(tokens, positions, token_mask):
    # Positions are added to queries and keys only [§3.2]
    queries = tokens + positions
    keys = tokens + positions
    # Self-attention, dropout, residual, then LayerNorm  [eq. 2]
    attended = self_attention(queries, keys, tokens, token_mask)
    tokens = layer_norm(tokens + dropout(attended))
    # Feed-forward with ReLU (hidden width 2048 in the visual branch
    # [§3.2]), dropout, residual, then LayerNorm  [eq. 3]
    expanded = relu(linear_up(tokens))
    tokens = layer_norm(tokens + dropout(linear_down(expanded)))
    return tokens

# Training loss for one image, expression and target box  [§3.3]
# Both boxes are (x, y, w, h), normalised by the image size  [§3.3]
def training_loss(predicted_box, target_box):
    # Smooth L1 on the coordinates; scale-sensitive  [§3.3] [eq. 5]
    coordinate_term = smooth_l1(predicted_box, target_box)
    # GIoU loss, unaffected by box scale; 1 minus GIoU is the form the
    # code uses  [§3.3] [code: utils/loss_utils.py#L121-L124]
    overlap_term = 1 - generalized_iou(predicted_box, target_box)
    # Weight lambda = 1  [§4.2] [eq. 5]
    return coordinate_term + 1.0 * overlap_term

Hyperparameters

Results

Author-reported. Setup: top-1 accuracy at IoU 0.5 on the ReferItGame and Flickr30K test sets tab. 1, and on the RefCOCO, RefCOCO+ and RefCOCOg splits tab. 2. Backbones are ResNet-50 and ResNet-101. The authors compare with two-stage and one-stage methods §4.3. Hardware, runtime setup and precision are not stated; tab. 4 reports runtimes without hardware. The paper's headline improvements are +6.13 over ReSC-Large on ReferItGame, +5.80 over the previous best on Flickr30K and +6.05 on RefCOCO testB §4.3.

Table 1 tab. 1: ReferItGame and Flickr30K, test sets.

Table 2 tab. 2: RefCOCO, RefCOCO+ and RefCOCOg, validation and test splits. "(released)" rows are the authors' reproduction with the released code.

Table 3 tab. 3: RefCOCO validation accuracy for different initial states of the [REG] token, ResNet-50, 90 epochs.

Table 4 tab. 4: ReferItGame test accuracy with and without the two branch transformers, ResNet-50. Runtime is in milliseconds, hardware not stated.

Qualitative results: on RefCOCOg test examples, the [REG] token's attention over visual tokens in the top row of fig. 3 lands on the referred object, which the authors describe as interpretable and matching its shape and location §4.5. In fig. 4, the second layer spreads [REG] attention over several regions of the whole image, the fourth layer focuses on regions tied to the referred object such as a bus behind the man, and the sixth layer concentrates on the object itself §4.5.

Paper vs code

Code: official repository README.md, commit c8624277018fc9786a39181e229f155f933b2b59, owner djiajunustc, linked from the paper's abstract. Checked files: models/trans_vg.py, models/vl_transformer.py, models/visual_model/*.py, models/language_model/bert.py, datasets/*.py, utils/loss_utils.py, utils/eval_utils.py, engine.py, train.py, train.sh, test.sh, README.md, docs/GETTING_STARTED.md. Differences:

• Loss type. Paper: smooth-L1 plus λ times GIoU §3.3 eq. 5. Code: plain L1 (F.l1_loss) plus 1 minus GIoU, both summed with weight 1 loss_utils.py engine.py. The paper does not say which loss the released weights were trained with; the code is the only evidence.
• Max expression length. Paper: 40 for all datasets §4.2. Code: 20 for ReferItGame, RefCOCO and RefCOCO+, 40 for RefCOCOg train.sh train.sh train.sh train.sh train.sh. README agrees [README.md#L103].
• Epochs for RefCOCO+. Paper: 90 for all datasets except Flickr30K at 60 §4.2. Code: 180 epochs with the LR drop at 120 for RefCOCO+ train.sh [README.md#L105].
• Batch size. Paper: 64 §4.2. Code: 8 per process, launched on 8 processes train.sh train.sh. Equivalent only with 8 GPUs.
• Token order. Paper: visual, then linguistic, then [REG] §3.2 eq. 4. Code: [REG], then language, then visual trans_vg.py; the prediction reads index 0 trans_vg.py. Because the learned position table is indexed by position, this only relabels which learned vector each token receives; the paper does not discuss the difference.
• Sigmoid on the output. Paper: linear output layer §3.2. Code: a sigmoid is applied to the 4 outputs trans_vg.py.
• Target normalisation. Paper: normalised "by the scale of the image" §3.3. Code: divides by the 640×640 padded canvas after resizing and padding transforms.py. The two agree only if the reference size is the padded canvas.
• IoU comparison. Paper: IoU "above 0.5" counts as correct §4.3. Code: IoU ≥ 0.5 eval_utils.py.
• Flickr30K. Paper: Flickr30K results tab. 1 and a training schedule §4.2. Code: a Flickr dataset branch exists data_loader.py, but no training or test command covers it train.sh test.sh, and the README model zoo lists no Flickr30K weights [README.md#L31-L33]. Flickr30K numbers cannot be reproduced from this repository as shipped.
• DETR initialisation. Paper: visual branch initialised from DETR's backbone and encoder §4.2. Code: loads a dataset-specific file (detr-r50-referit.pth, detr-r50-unc.pth, detr-r50-gref.pth) into the visual model, ignoring unmatched keys train.sh train.sh train.sh train.py. The repository's download script fetches the original DETR files named detr-r50.pth and detr-r101.pth download_detr_model.sh, which these names do not match. The getting-started guide points to a Google Drive folder of pretrained checkpoints, and says that when these are pre-trained on MSCOCO, images overlapping the val and test sets are excluded GETTING_STARTED.md. It does not name the files.
• Released versus reported. The README lists ReferItGame test accuracy 69.3 for ResNet-50 [README.md#L63-L75], against 69.76 in the paper tab. 1. It lists no ResNet-101 ReferItGame weights [README.md#L79-L91]. The "(released)" rows in tab. 2 match the README, and they are below the paper's ResNet-101 RefCOCO testB, 76.94 against 78.35.
• Augmentation. Paper: "we follow" prior work, with no list §4.2. Code, in training: multi-scale resizing from 640 down to 448 in steps of 32 [code: datasets/init.py#L22-L24]; a crop path with probability 0.5 [code: datasets/init.py#L28-L41]; colour jitter of 0.4 with probability 0.8 transforms.py; translation of the padded image transforms.py. A horizontal flip with probability 0.5 is always on in training and swaps the words "left" and "right" in the query [code: datasets/init.py#L45] transforms.py.
• Frozen layers. Paper: silent. Code: the ResNet stem and layer1 are frozen, and BERT is trainable whenever lr_bert > 0 backbone.py bert.py bert.py.
• Checkpoint selection. Paper: silent. Code: validates on the val split each epoch and keeps the checkpoint with the best val accuracy train.py train.py.

No discrepancies were found for the core layer sizes, head depth, optimiser, weight decay or the 1e-4 and 1e-5 learning rates.

Pitfalls

• Eq. 1 is missing a transpose. As typeset, the attention score is written as q times k divided by the root of d_k. The standard scaled dot-product attention is q times k-transpose eq. 1. The code uses PyTorch's multi-head attention, so implement the standard form.
• Table 4 arrow. The row with 66.78 shows an arrow of 3.54, but 66.78 minus 64.24 is 2.54. The text calls the absolute accuracies "an improvement of 68.48% and 66.78%" §4.4. Use the table values, not the arrows.
• Table 3 labels. The bullet list in §4.4 labels the fourth option "Average pooled linguistic tokens" twice; the second describes max pooling, and tab. 3 lists it as max-pooled §4.4.
• Target normalisation is ambiguous. The paper says "by the scale of the image" §3.3. The code uses the 640×640 padded canvas, and predictions live in that same frame transforms.py. Implement the canvas version to match the code.
• Predictions are bounded and clamped. Sigmoid bounds the output to (0, 1) trans_vg.py. Evaluation clamps again eval_utils.py. The paper does not state either step.
• Query truncation drops the tail. Longer queries are cut at the end, so the final words are lost data_loader.py. The paper's 40 keeps 38 word pieces, but the scripts' 20 keeps 18 for ReferItGame and RefCOCO. The two settings are not interchangeable.
• Checkpoint shape depends on query length. The learned position table has size visual tokens plus max_query_len plus one trans_vg.py. A checkpoint trained at one max_query_len should not load into a model built with another; this follows from the code and was not tested.
• Compute per layer. At 640 px with stride 32 there are 400 visual tokens trans_vg.py. With 40 text positions and [REG], each fusion layer attends over 441 positions, so attention cost grows quadratically with the number of tokens. The paper gives runtimes in tab. 4 but no hardware.
• Model selection. The paper does not say which checkpoint it reports. The code keeps the epoch with the best val accuracy train.py, so test numbers depend on that rule.
• Dropout placement. The paper places dropout after attention and after the feed-forward §4.2. The code also drops inside the feed-forward after the activation and inside attention vl_transformer.py vl_transformer.py. Matching the code changes results by an unknown amount.
• BERT is fine-tuned. Paper: initialised from pre-trained BERT §4.2. Code: BERT trains at the linguistic learning rate of 1e-5 by default train.sh train.py. Freezing it is a change from the default.
• Batch drop-last. Training drops the final incomplete batch each epoch train.py. Not mentioned in the paper.
• Data indices are external. The split files and images are not in the repository and come from separate downloads GETTING_STARTED.md. Split sizes in §4.1 can only be checked against those downloads.
• Failure case. A query that is only a number collapses the BERT path to an MLP, according to the authors §4.3.
• Licence. The repository has no LICENSE file, so the code's licence is not stated. This card uses paraphrase and path-and-line citations only; no code is quoted.

Lineage

• Builds on: DETR, which provides the visual backbone and encoder initialisation and the sine position codes §3.2 §4.2 (/papers/2005.12872). BERT-base for the linguistic branch §3.2 §4.2. ResNet for the visual backbone §3.2. The transformer encoder layer follows Vaswani et al., cited in §2.2.
• Compared against: ReSC-Large, FAOA, RCCF, ZSGNet and SSG (one-stage); MAttNet, NMTree, DDPN, Similarity Net and CITE (two-stage) tab. 1 tab. 2. The authors single out ReSC for sharing a recursive idea with their fusion stack §4.3.
• Superseded by: not stated in this wiki.