Paper · 2021-12-16 · cs.CV, cs.CL
Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds
BUTD-DETR grounds a referring utterance in 3D scans or 2D images by decoding boxes with DETR-style queries, not by picking one of a detector's proposals. Language, visual and proposal streams feed the decoder, and detection prompts add supervision. Author-reported: SR3D 52.1 and ScanRefer 52.2 (Acc@0.25, detected boxes).
Visual groundingObject detectionDirect set predictionCross-attentionvisual-groundingdetrpoint-cloudsmultimodal
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Finding "the chair next to the window" in a room or a photo is hard when the detector that finds objects has missed the chair. Most earlier systems work like a multiple-choice quiz: a detector first proposes a fixed set of candidate boxes, and the system picks the right one. If the detector never proposed the object, the system cannot find it.
This paper's model, BUTD-DETR, does not only pick from the candidates. It uses the candidates as hints, next to the words and the raw scene, and then draws its own box for each object the sentence names. The name comes from "bottom-up" (what the detector saw) and "top-down" (what the sentence asks for). The model is also trained on lists of object names, some present and some absent, which teaches it to find objects and to say when one is missing. It works on 3D room scans and on 2D photos.
TL;DR
- BUTD-DETR localizes every object a referring utterance names, in 3D point clouds and 2D images, by decoding boxes with a DETR-style decoder instead of selecting from a fixed proposal pool §1 §3.2.
- Its inputs are three streams: the scene, the words and the box proposals from a frozen detector, which the decoder uses as context rather than as a candidate list §3.2.1 §3.2.3.
- Detection prompts (lists of category labels with some absent) add dense supervision, and a symmetric contrastive loss aligns queries with words §3.3 §3.4.
- Author-reported gains over earlier 3D methods are 12.6 points on SR3D, 11.6 on NR3D and 6.3 on ScanRefer §1. 2D results are close to MDETR's at lower training cost tab. 3 tab. 4.
- Only the 3D code is inspected here, and it is non-commercial (see Paper vs code).
Problem
Most referring-expression grounding models are box-bottlenecked: they choose the referred object from proposals given by a pre-trained detector. They cannot recover objects the detector misses, which the paper says happens for small, occluded or rare objects §1.
MDETR removes the bottleneck in 2D by decoding boxes directly, but it concatenates language and visual tokens and uses parametric queries §3.1 §3.2.2. The paper says that the original visual self-attention scales quadratically with the number of visual tokens §4.2. For 3D, the paper states that all existing models tested on SR3D and NR3D are box-bottlenecked §4.1. Its 3D detector is trained on ScanNet while the benchmarks are also ScanNet-based, so proposals are better at train time than at test time §3.2.1.
Method
The paper's convention for unnumbered supplementary material is cited as
[§ Supplementary] (Section name), because the supplement has no numbers in
the source.
- Encode the three streams §3.2.1.
- Visual tokens V (n_v by c_v). In 3D, a PointNet++ backbone plus learnable 3D positional encodings from an MLP over point coordinates. In 2D, a ResNet-101 with 2D Fourier positional encodings §3.2.1.
- Word tokens L from a pre-trained RoBERTa §3.2.1.
- Object proposal tokens O. Each proposal embeds its coordinates and its category; the category is encoded by a frozen RoBERTa §3.2.1.
- Proposal sources: Group-Free trained on 485 ScanNet categories in 3D, and Faster R-CNN trained on 1601 Visual Genome classes in 2D §3.2.1. The supplement cites Anderson et al. (2018) for the 2D proposal detector § Supplementary (Implementation details).
- Each stream is mapped to a vector of the same length by its own MLP §3.2.1.
- Corrupt 30% of 3D proposals at train time. Randomly replacing 30% of the detected boxes with random boxes leads to better generalization when the detector fails §3.2.1. The supplement reports about a 1-point gain § Supplementary (Effect of detection augmentation).
- Cross-modality encoder §3.2.2. Visual and word tokens cross-attend to each other with standard key-value attention. The language-conditioned visual tokens then attend to the proposal tokens. Unlike MDETR, the three streams are kept separate instead of concatenated. In 2D the visual stream uses deformable attention; in 3D standard attention is used, because bilinear sampling on point clouds is costly and unreliable §3.2.2 § Supplementary (Implementation details).
- Non-parametric object queries §3.2.3. A linear head gives each visual token a confidence score. The top-K tokens each produce one query vector, which predicts a box center and size relative to its token. K is 256 in 3D and 300 in 2D § Supplementary (Implementation details).
- Decoder with N_D = 6 layers §3.2.3 § Supplementary. In each layer a query self-attends to the other queries, attends to the word tokens, attends to the proposal tokens, and then attends to the visual tokens. A prediction head at the end of each layer outputs a center displacement, a height and width vector, and a token span for the query §3.2.3.
- Detection prompts §3.3 fig. 3. A prompt is a sequence of category labels: some present in the scene, some absent. The model must localize every present instance and align it with its label span; absent labels must match no box. The supplement sets a ratio r = 1 of positive to negative labels, with at most 10 positive labels per prompt § Supplementary (Negative training with detection prompts).
- Supervision of every decoder layer §3.4. Queries are assigned to
ground-truth boxes by Hungarian matching, and the losses are applied to the
output of each layer.
- Box loss: L1 plus generalized IoU, as in MDETR §3.1 §3.4.
- Soft token loss: a matched query predicts a uniform distribution over the tokens of its object's span; an unmatched query predicts "no object" §3.1.
- Symmetric contrastive loss: the span "not mentioned" is appended to every utterance and serves as the target span for unmatched queries. The object contrastive term then covers all queries, not only matched ones §3.4. The paper finds this sufficient alone, but co-optimizing the soft token loss converges faster §3.4.
- Query confidence. The confidence head is trained with a focal loss. In 3D, the 4 points closest to each ground-truth center are positives. In 2D, every token inside a ground-truth box is a positive § Supplementary (Implementation details).
The paper gives no numbered equations for these losses, and the loss weights are not stated §3.4.
# BUTD-DETR forward pass for one scene and one utterance [§3.2] [fig. 2]
# Inputs: scene (point cloud or image), utterance (string),
# proposals (boxes with labels from a frozen detector)
# Output: per query, one box and one span of utterance tokens
def butd_detr(scene, utterance, proposals):
# Encode each stream separately [§3.2.1]
visual = visual_encoder(scene) # (n_v, c_v)
words = text_encoder(utterance) # (n_l, c_l)
proposal_tokens = proposal_encoder(proposals) # (n_o, c_o)
# Cross-modality encoder, N_E layers [§3.2.2]
for layer in range(num_encoder_layers):
# Visual and word tokens cross-attend to each other
visual, words = cross_attend_pair(visual, words)
# Visual tokens then attend to the proposal tokens
visual = cross_attend(visual, proposal_tokens)
# Keep the K visual tokens with the highest confidence [§3.2.3]
confidence = confidence_head(visual) # one score per token
top = top_k_indices(confidence, num_queries)
queries = query_head(visual[top]) # (K, c_q)
# Decoder, N_D layers [§3.2.3]
for layer in range(num_decoder_layers):
# Queries refine one another first
queries = self_attend(queries)
# Then condition on the words, the proposals and the scene
queries = cross_attend(queries, words)
queries = cross_attend(queries, proposal_tokens)
queries = cross_attend(queries, visual)
# Predict a box and a span for each query [§3.2.3]
boxes, spans = prediction_head(queries)
return boxes, spans# Detection prompt for one scene [§3.3] [fig. 3]
# Inputs: labels present in the scene, the detector's label vocabulary
# Output: one utterance string, with present and absent labels mixed
def make_detection_prompt(present_labels, vocabulary):
# Sample a random number of present labels; the count is not stated [§3.3]
chosen = sample_present(present_labels)
# Add absent labels as negatives at ratio r = 1 [§ Supplementary]
negatives = sample_absent(vocabulary, len(chosen))
# Join the labels into one string; the order is not stated [§3.3]
return join_labels(chosen + negatives)# Supervision for one decoder layer [§3.4]
# Inputs: predicted boxes and spans, ground-truth boxes and spans,
# the utterance tokens (with "not mentioned" appended)
# Output: scalar loss for this layer
def layer_loss(pred_boxes, pred_spans, targets, tokens):
# Match queries to ground-truth boxes with Hungarian matching [§3.4]
matched, unmatched = hungarian_match(pred_boxes, targets)
# Box loss: L1 plus generalized IoU, as in MDETR [§3.1] [§3.4]
box_loss = l1_loss(matched) + giou_loss(matched)
# Soft token loss, as in MDETR [§3.1]
token_loss = soft_token_loss(matched, unmatched, targets)
# Symmetric contrastive loss over all queries and tokens [§3.4]
contrastive = symmetric_contrastive(pred_spans, tokens)
return box_loss + token_loss + contrastiveHyperparameters
Code paths are relative to the repository at code_commit. Values marked
"not stated" are not in the paper; where the official code sets a value,
the code is cited and the row says so.
| Name | Value | Where | Notes |
|---|---|---|---|
| 3D visual tokens | 1024 | § Supplementary (Implementation details) | last PointNet++ layer |
| 2D visual tokens | 17821, dimension 256 | § Supplementary (Implementation details) | flattened multi-scale maps |
| Object queries K, 3D | 256 | § Supplementary (Implementation details) | top-K tokens by confidence; code passes it as the query count train_dist_mod.py main_utils.py |
| Object queries K, 2D | 300 | § Supplementary (Implementation details) | |
| Cross-modality encoder layers N_E | 3D: 3; 2D: 6 | § Supplementary (Implementation details) | code default 3 main_utils.py |
| Decoder layers N_D | 6 (3D and 2D) | § Supplementary (Implementation details) | code default 6 main_utils.py |
| Positive points per 3D object center | 4 | § Supplementary (Implementation details) | code default 4 main_utils.py |
| Proposal replacement at train time | 30% of boxes, 3D | §3.2.1 | code uses the same rate and also randomizes labels joint_det_dataset.py |
| Detection prompt negatives | ratio r = 1, at most 10 positives | § Supplementary (Negative training with detection prompts) | code samples a fixed 10 negatives joint_det_dataset.py |
| Text encoder | pre-trained RoBERTa, frozen in 3D | §3.2.1 § Supplementary (Implementation details) | code loads roberta-base and freezes it bdetr.py |
| Learning rate, 3D visual encoder | 1e-3 | § Supplementary (Implementation details) | script sets the same value train_test_det.sh |
| Learning rate, 3D other layers | 1e-4 | § Supplementary (Implementation details) | script sets the same value train_test_det.sh |
| Batch size, 3D | 6 on a 12 GB GPU; 24 on an A100 | § Supplementary (Implementation details) | script uses 24 train_test_det.sh |
| Time per epoch, 3D | about 50 minutes on an A100 | § Supplementary (Implementation details) | |
| Learning rate, 2D | ResNet-101 1e-6; RoBERTa 5e-6; rest 1e-5 | § Supplementary (Implementation details) | 2D code not inspected |
| 2D pre-training | 64 V100 GPUs, batch size 1 | § Supplementary (Implementation details) | fine-tuning on RefCOCO and RefCOCO+: batch size 2 on 16 V100s, 5 epochs § Supplementary (Implementation details) §4.2 |
| Optimizer | AdamW | not stated in paper | main_utils.py main_utils.py |
| Weight decay | 5e-4 | not stated in paper | train_test_det.sh |
| Gradient clipping norm | 0.1 | not stated in paper | main_utils.py |
| Learning-rate decay, 3D | epochs 25 and 26 in the detection script; 30 and 35 in the classification script | not stated in paper | train_test_det.sh train_test_cls.sh |
| Maximum epochs | 400 | not stated in paper | code default main_utils.py |
| Contrastive temperature | 0.07 | not stated in paper | losses.py |
| Matcher costs (class, box, GIoU) | 1, 0, 2 as called | not stated in paper | the class default is 1, 5, 2 losses.py main_utils.py |
| Loss weights | see Paper vs code | not stated in paper | losses.py |
| Dropout in transformer layers | 0.1 | not stated in paper | bdetr.py |
| Model width and heads | 256 and 8 | not stated in paper | encoder_decoder_layers.py |
| Proposal confidence threshold | not stated in paper | §3.2.1 | the LaTeX source has commented-out values of 0.50 (2D) and 0.25 (3D), which the compiled paper does not show |
Results
Author-reported. The 3D setup trains one model per benchmark, with ScanNet detection prompts added as supervision, and reports top-1 accuracy at IoU 0.25 and 0.5 on the validation split §4.1. Boxes are either detected by the Group-Free detector (Det) or ground truth (GT). Previous methods are re-trained or evaluated with their own code and boxes, as the paper notes §4.1.
The 2D setup pre-trains on Flickr30k, RefCOCO-style referring data and Visual Genome, with COCO detection prompts added, then fine-tunes on RefCOCO and RefCOCO+ §4.2. Flickr30k uses Recall@k §4.2.
3D point clouds tab. 1
| Method | SR3D Acc@0.25 (Det) | SR3D Acc (GT) | NR3D Acc@0.25 (Det) | ScanRefer Acc@0.25 (Det) | ScanRefer Acc@0.5 (Det) |
|---|---|---|---|---|---|
| ReferIt3DNet | 27.7 | 39.8 | 24.0 | 26.4 | 16.9 |
| ScanRefer | - | - | - | 35.5 | 22.4 |
| TGNN | - | 45.0 | - | 37.4 | 29.7 |
| 3DRefTransformer | - | 47.0 | - | - | - |
| InstanceRefer | 31.5 | 48.0 | 29.9 | 40.2 | 32.9 |
| FFL-3DOG | - | - | - | 41.3 | 34.0 |
| LanguageRefer | 39.5 | 56.0 | 28.6 | - | - |
| 3DVG-Transformer | - | 51.4 | - | 45.9 | 34.5 |
| TransRefer3D | - | 57.4 | - | - | - |
| SAT-2D (uses 2D image features) | 35.4 | 57.9 | 31.7 | 44.5 | 30.1 |
| MDETR-3D (authors' implementation) | 45.4 | - | 31.5 | 47.2 | 31.9 |
| BUTD-DETR | 52.1 | 67.0 | 43.3 | 52.2 | 39.8 |
The Det rows of the SR3D and NR3D columns for prior methods are re-run by the authors with the same detector, per the paper's footnotes tab. 1. The supplement adds the GT overall score on NR3D, 54.6, and per-context breakdowns § Supplementary (Detailed results on SR3D/NR3D and ScanRefer).
Headline margins over prior 3D methods are 12.6 points on SR3D, 11.6 on NR3D and 6.3 on ScanRefer §1.
Ablation on SR3D tab. 2
| Model | Accuracy |
|---|---|
| BUTD-DETR | 52.1 |
| without visual tokens (object-bottlenecked) | 41.9 |
| without detection prompts | 47.9 |
| without box stream | 51.0 |
| with MDETR's contrastive loss | 49.6 |
| without detection prompts, box stream, with MDETR's loss (MDETR-3D) | 45.4 |
| MDETR-3D with learned parametric queries instead of non-parametric ones | 33.8 |
| streams joined along the sequence axis instead of kept separate | 51.3 |
The paper reads these as: box-decoding models beat object-bottlenecked variants by 10.2 points; removing the box stream costs 1.1 points; detection prompts add 4.2 points; the symmetric contrastive loss adds 2.5 points; and separate streams beat concatenation by 0.8 points §4.1.1.
2D images tab. 3 tab. 4
| Method | RefCOCO val | testA | testB | RefCOCO+ val | testA | testB | Epochs (pre+fine) | V100 GPU hours |
|---|---|---|---|---|---|---|---|---|
| UNITER_L | 81.4 | 87.0 | 74.2 | 75.9 | 81.5 | 66.7 | - | - |
| VILLA_L | 82.4 | 87.5 | 74.8 | 76.2 | 81.5 | 66.8 | - | - |
| MDETR | 86.8 | 89.6 | 81.4 | 79.5 | 84.1 | 70.6 | 40 + 5 | 5560 |
| BUTD-DETR | 85.9 | 88.5 | 81.5 | 78.2 | 82.8 | 70.0 | 12 + 5 | 2748 |
| Method | Flickr30k val R@1 | val R@5 | val R@10 | test R@1 | test R@5 | test R@10 | Epochs | V100 GPU hours |
|---|---|---|---|---|---|---|---|---|
| VisualBERT | 70.4 | 84.5 | 86.3 | 71.3 | 85.0 | 86.5 | - | - |
| MDETR | 82.5 | 92.9 | 94.9 | 83.4 | 93.5 | 95.3 | 40 | 5480 |
| BUTD-DETR | 81.2 | 90.9 | 92.8 | 81.0 | 91.6 | 93.2 | 12 | 2688 |
The paper calls the 2D results "comparable" to MDETR, with about half the training time, which it attributes to deformable attention on the visual stream §4.2. The 2D ablation on RefCOCO validation, without pre-training tab. 5: full model 79.4; without detection prompts 77.0; without box stream and detection prompts 76.3; with MDETR's contrastive loss as well, 74.2 §4.2.1.
Paper vs code
The official repository is github.com/nickgkan/butd_detr at the commit in
the frontmatter. The project page named in the abstract § Abstract links to
it, and its owner is Nikolaos Gkanatsios, an author of this paper [code:
README.md#L4]. It implements the 3D model; the README says the 2D model is on
a separate branch, bdetr2d README.md, which this compile did not
inspect. Code paths below are relative to the repository root.
Discrepancies found by reading the code, with both citations:
- Negatives per prompt. The paper sets a ratio r = 1 and up to 10 positive labels § Supplementary (Negative training with detection prompts). The code adds a fixed 10 negatives and caps positives at 10 joint_det_dataset.py joint_det_dataset.py. With fewer than 10 positives, the code's negative count is larger than the paper's ratio implies.
- How often the random prompt is used. The figure caption says half of the sampled labels are negative fig. 3. The code uses the random prompt for about half of the ScanNet samples and otherwise a fixed 18-class phrase joint_det_dataset.py joint_det_dataset.py. The paper describes neither the fixed phrase nor the 50% sample rate.
- Proposal corruption. The paper replaces 30% of boxes §3.2.1. The code also replaces their class ids for the same boxes joint_det_dataset.py.
- Loss weights. The paper gives no weights §3.4. The code uses 8 times the query-generation loss, plus one over (N_D + 1) times the sum of the cross-entropy, 5 times the L1 box loss, the GIoU loss and the contrastive loss, summed over decoder layers losses.py.
- Matcher costs. The code calls the matcher with class cost 1, box cost 0 and GIoU cost 2 main_utils.py. The class default is 1, 5, 2 losses.py. The paper gives no weights.
- Self-attention in the encoder. The main text describes attention without mentioning self-attention §3.2.2. The supplement says all attention layers are standard key-value attention § Supplementary (Implementation details). In the code, encoder self-attention is behind a flag that is off by default main_utils.py; the released 3D scripts turn it on train_test_det.sh.
- Frozen text encoder. The paper says the 3D text encoder is frozen § Supplementary (Implementation details). The code freezes it bdetr.py. The separate text-encoder learning rate is then unused main_utils.py.
- Decoder order. The paper's order (self-attention, words, proposals, visual tokens) matches the code's decoder layer encoder_decoder_layers.py §3.2.3.
- Learning rates. The paper's 3D learning rates match the released detection script train_test_det.sh § Supplementary (Implementation details).
Items not checked: the 2D model and its learning rates (on bdetr2d), the
span predictor used at test time, and the paper's ScanNet detection results.
Licence note: the LICENSE file is CC BY-NC-SA 4.0 and the README says CC-BY-NC. This page uses the code only to confirm values with citations; no code text or pseudocode is taken from it.
Pitfalls
- The detector still matters. The paper's failure case on SR3D is one where the box exists among the proposals but the model ranks another box higher; the authors call ranking the correct box "a hard problem" § Supplementary (More qualitative results). Budget for the detector and for ranking.
- Detector quality differs between train and test. The 3D detector was trained on ScanNet, so proposals are better at train time. The 30% box replacement is the paper's mitigation, and it helped by about 1 point §3.2.1 § Supplementary (Effect of detection augmentation).
- Loss weights and temperature are not in the paper. A re-implementation from the paper alone cannot reproduce the loss balance. Use the values in the table above, which come from the code, and say so.
- Training schedule is not in the paper. Epoch count and decay points are not stated, and the code's choices differ by task (25/26 for detection, 30/35 for classification). The README says NR3D and ScanRefer need many more epochs and a manual decay at epochs 80 and 90 (NR3D) or 65 (ScanRefer) README.md.
- Proposal category text. Proposal category labels pass through a frozen RoBERTa §3.2.1. The paper does not say how label wording is chosen.
- Detection prompts are not a detector. The paper reports ScanNet detection mAP for models trained with these prompts. The objective is contrastive alignment, not classification. The authors say a softmax objective suits a fixed label set better: BUTD-DETR trained on ScanNet reaches 59.3 mAP@0.25, and 61.0 with softmax § Supplementary (Detection results).
- Limitations §4.3: the model relies on language-image alignment. It does not handle left/right under a change of viewpoint, counting, or logical constraints such as "furthest from the door", and it can violate them on out-of-distribution input.
- Compute. The 2D pre-training used 64 V100s and 2748 V100 GPU hours in the authors' setup tab. 3. The 3D model fits on a 12 GB GPU at batch size 6 § Supplementary (Implementation details).
- Proposal threshold. The compiled paper gives no confidence threshold for proposals. Choose one and report it.
Lineage
- Builds on: MDETR (Kamath et al., 2021) for box decoding, soft token loss and contrastive alignment §3.1. No card for MDETR exists in this wiki yet.
- Builds on: DETR (Carion et al., 2020) for the set-prediction decoder §3.1 and Deformable DETR (Zhu et al., 2021) for deformable attention and non-parametric queries §3.2.2 §3.2.3.
- Uses: Group-Free 3D (Liu et al., 2021) as the 3D proposal source and as the source of the query scoring method §3.2.1 § Supplementary (Implementation details).
- Compared against: ReferIt3DNet, ScanRefer, TGNN, LanguageRefer, InstanceRefer, 3DVG-Transformer, SAT-2D and others tab. 1; MDETR in 2D tab. 3 tab. 4.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds” by Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, Katerina Fragkiadaki (arXiv:2112.08879), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/nickgkan/butd_detr at commit 10570e0b6826 (CC BY-NC-SA 4.0 (non-commercial)). Any quoted code is shown under that license.