ScanQA: 3D Question Answering for Spatial Scene Understanding

In plain words

Most question-answering models for images look at a flat picture. This paper asks a harder question: can a model answer questions about a whole room, given a 3D scan of it? A question like "what color is the chair next to the table?" needs the model to know where things are, not only what they look like.

The authors build a dataset of such questions for indoor rooms from a public scan collection. Each answer is written by people, and each question is tied to the objects in the room it talks about, so the answer can be checked against the right place in the scan.

The model, called ScanQA, works in two parts at once. It finds candidate objects in the 3D scan and reads the question. It then mixes the two so that it can both name the answer and point to the object that the question is about. Training it to do both tasks together is the main idea: the answer and the location help each other.

TL;DR

• Task: given an RGB-D indoor scan and a free-form question, output an answer and the 3D boxes of the objects the question refers to §3.1.
• Model: VoteNet object proposals plus a question encoder, fused by transformer layers, with three heads (localization, object class, answer) trained jointly §4.
• Data: 41,363 questions over 800 ScanNet scenes, with a test split that has object annotations and one that does not tab. 2 §3.3.
• Results are author-reported: EM@1 23.45 on the test split with objects, against 22.31 for the best 2D baseline tab. 3.

Problem

Image question answering models see one or several 2D frames. They struggle with direction, distance, occlusion, and with telling whether two views show the same object §1. Existing 3D datasets for language are small or template-based §1. The paper frames a task where a scan of the room already exists, for example from prior navigation, and the model answers from that scan §1.

Method

The full pipeline is built from these parts; the forward pass and loss are in the pseudocode below.

1. Question encoder. Each question word is embedded with GloVe (300-dimensional). A one-layer bidirectional LSTM with hidden size 256 models the sequence, and a GELU projection gives contextual word vectors of size d = 256 §4. The paper describes the LSTM as bidirectional; the code uses a unidirectional LSTM by default (see Paper vs code).
2. Scene encoder. Point features r have 135 dimensions: height, colour, normals and multiview image features §4. VoteNet with a PointNet++ backbone detects object proposals, n_v = 256 of them §4. A GELU projection maps each proposal to size d §4.
3. Fusion. Two transformer encoder layers process the question words and two decoder layers let the proposals attend to them §4. The paper cites the MCAN architecture as the source of these blocks §4. The two streams are then merged into one fused vector of size d by an attention MLP §4.
4. Localization head. A two-layer MLP scores each proposal against the question. Scores are normalised over proposals and trained with cross-entropy §4.
5. Object-class head. A two-layer MLP predicts one of the 18 ScanNet benchmark classes from the fused vector §4.
6. Answer head. A projection to one score per training answer is trained with binary cross-entropy, so several answers can be correct §4.
7. Loss. The paper uses L = L_ans + L_obj + L_loc + L_det, a plain sum of answer, object-class, localization and VoteNet detection losses §4.

Dataset collection, for context: questions were generated from ScanRefer captions with a T5 question-generation model, filtered by rules and by workers, rewritten by workers, and answered in free form with object IDs attached §3.2.

# ScanQA forward pass and training loss  [§4]
# Input:  cloud (N, 3 + f); question tokens; answer targets
# Output: scalar loss

def scanqa_loss(cloud, question, targets):
    # Embed each question word with GloVe, then run a one-layer LSTM
    # over the sequence  [§4]
    word_vectors = glove_embed(question)           # (n_q, 300)
    words = lstm(word_vectors)                     # (n_q, d), d = 256
    words = gelu(linear(words))                    # (n_q, d)

    # Detect object proposals with VoteNet on a PointNet++ backbone,
    # keeping n_v = 256 proposals  [§4]
    proposals = votenet(cloud)                     # (n_v, 128)
    proposals = gelu(linear(proposals))            # (n_v, d)

    # Let words and proposals attend to each other through
    # L = 2 transformer encoder and decoder layers  [§4]
    words, proposals = fusion_layers(words, proposals)

    # Merge both streams into one question-aware vector with an
    # attention MLP  [§4]
    fused = attention_fuse(words, proposals)       # (d,)

    # Localization: score each proposal against the question  [§4]
    # Target is the proposal that best matches the annotated box
    # [code: lib/dataset.py#L346-L350]
    box_scores = two_layer_mlp(proposals)          # (n_v,)
    loc_loss = cross_entropy(box_scores, gt_index)

    # Object class: one of the 18 ScanNet classes  [§4]
    class_logits = two_layer_mlp(fused)            # (18,)
    obj_loss = cross_entropy(class_logits, object_class)

    # Answer: one score per training answer, multi-label  [§4]
    answer_logits = two_layer_mlp(fused)           # (n_a,)
    ans_loss = bce_with_logits(answer_logits, targets)

    # Detection: VoteNet losses on the same scene  [§4]
    det_loss = votenet_loss(cloud, boxes)

    # Unit-weighted sum of the four terms in the paper  [§4]
    return ans_loss + obj_loss + loc_loss + det_loss

# Training schedule  [§5.1]
# Input:  training scenes with questions, answers and object boxes
# Output: trained weights

def train_scanqa(scenes):
    # Adam with initial learning rate 5e-4 and weight decay 1e-5  [§5.1]
    optimizer = adam(lr=5e-4, weight_decay=1e-5)
    # Train for 30 epochs, with the learning rate multiplied by 0.2
    # after epoch 15  [§5.1]
    for epoch in range(30):
        for batch in batches(scenes, size=16):  # batch size 16  [§5.1]
            # Rotate each axis by up to 5 degrees and shift each
            # axis by up to 0.5 m  [§5.1]
            cloud, boxes = augment(batch)
            loss = scanqa_loss(cloud, batch.questions, batch.targets)
            optimizer.step(loss)

Key equation (the loss is written inline in the paper; no equation numbers are used in the body): \mathcal{L} = \mathcal{L}_{ans} + \mathcal{L}_{obj} + \mathcal{L}_{loc} + \mathcal{L}_{det} §4

Hyperparameters

Values in the Where column are the paper's unless the row says code. "not stated" means the paper does not give the value.

Results

Author-reported. Setup: the ScanQA dataset, 800 ScanNet scenes, with two test splits, one with and one without object annotations §3.3. Baselines: 2D VQA with MCAN on three random or oracle frames, VoteNet+MCAN, and ScanRefer+MCAN in pipeline and end-to-end form §5.1. Metrics are as in §5.1. Source: tab. 3 unless noted.

Test split with objects

Test split without objects

The paper reports the BLEU-3 value 16.53 for ScanRefer+MCAN (pipeline) on this split as printed, which is higher than its BLEU-2 and BLEU-4; the table here copies it as printed tab. 3.

Component ablation tab. 4. ANS is the answer head, OBJ the object-class head, LOC the localization head.

Input-feature ablation tab. 5. Localization and QA with different point features. Only Acc@0.25 and EM@10 are reported here.

The paper's text says RGB did not help localization or QA, and multiview features helped localization but not QA on the single-object model §5.3. The main ScanQA row matches the xyz+multiview+normal setting on EM@10 tab. 5 tab. 3.

Other reported numbers. Human performance: EM@1 of 51.6 for the best MTurk worker on the sampled questions of the test split with objects §5.2. The appendix adds a multi-object variant, ScanQA (multiple), which uses sigmoid scores and BCE over boxes and labels app. A; its ablations are in app. B. In the appendix sweeps, hidden size d = 128 scores higher than d = 256 on EM@1 and EM@10 on the test split with objects (24.38 and 56.71 against 23.45 and 56.51), and L = 1 scores higher on EM@1 only app. B. The main text keeps d = 256 and L = 2 §4.

Paper vs code

Code: official repository ATR-DBI/ScanQA, linked from the paper's first page §1, commit 0a9f162112c80b439903062a4097d63015a70cad. Its LICENSE file is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 (CC BY-NC-SA 3.0) LICENSE. No code is quoted here. The pseudocode is ours; citations are by path and line only.

Discrepancies, each with both sources:

1. Training length and schedule. Paper: 30 epochs, learning rate multiplied by 0.2 after 15 epochs §5.1. Code defaults: 50 epochs and decay steps at 100 and 200 train.py train.py. A comment beside the decay default reads "# 15" train.py. The run actually used for the paper is not stated.
2. Loss weights. Paper: unit weights on all four terms §4. Code: vote 1.0, objectness 0.5, box 1.0, semantic class 0.1, reference (localization) 0.1, language (object class) 0.1, answer 0.1 train.py loss_helper.py. Inside the box loss, heading-class and size-class terms are weighted 0.1 loss_helper.py.
3. Question encoder direction. Paper: one-layer bidirectional LSTM §4. Code: LSTM, unidirectional unless --lang_use_bidir is passed lang_module.py train.py. The README training command passes only --use_color and --tag README.md, so it does not turn bidirectional on.
4. Fusion output. Paper: a fused feature of size d after a two-layer MLP with attention §4. Code: attention pooling of the question stream and of the proposal stream, summed, then LayerNorm, with a default size of 512 qa_module.py train.py.
5. Answer targets. Paper: binary cross-entropy over answer candidates §4. Code: BCE with soft targets. A training answer scores 0, 0.3, 0.6, 0.9 or 1.0 by its training count of 0, 1, 2, 3 or at least 4 dataset.py dataset.py. The loss is summed and divided by batch size loss_helper.py. The paper does not describe these soft targets. A cross-entropy variant (--answer_cls_loss ce) also exists in the code loss_helper.py.
6. Localization target. Paper: "which of the proposed object boxes corresponds to the question" with cross-entropy §4. Code: the target is the proposal whose box has the highest IoU with the annotated box of the first annotated object only dataset.py loss_helper.py. The score is trained with a softmax ranking loss against a one-hot target loss.py loss_helper.py.
7. Object-class label. Paper: predict the object classes associated with the question, 18 classes §4. Code: the label is the class of the first annotated object; any name missing from the mapping becomes class 17 dataset.py.
8. Augmentation. Paper: rotations of up to 5 degrees and translations of up to 0.5 m §5.1. Code: the same, plus a random flip of x and y with probability 0.5 each dataset.py dataset.py.
9. Feature set in the default run. The paper's main configuration uses geometry, multiview features and normals §5.3. The README command adds colour and no multiview or normals README.md. Height is on unless --no_height is passed train.py.
10. Multiview features. The paper cites a multiview feature method for point features §4. The code expects precomputed image features in a separate pipeline with its own scripts config.py project_multiview_features.py.

Pitfalls

• Only the first annotated object is a localization target in the official code dataset.py. Answers that depend on several objects (for example, "what color are the chairs around the table?" §3.1) are only handled by the appendix variant app. A. An implementer who copies the single-target version will not match the multi-object setting.
• Soft answer targets are not in the paper. Using plain 0/1 BCE will change training behaviour, because frequent answers get lower target weight than rare ones in the code dataset.py.
• Answer vocabulary is closed. Candidates are the training answers, with no cap and minimum frequency 1 train.py train.py. Answers outside this set cannot be produced.
• Free-form answers. Exact match is strict. The paper adds captioning metrics because answers to "where is" questions vary in wording §3.3 §5.1. Report both families.
• Unit weights are not what the code uses. Copying the paper's loss will not reproduce the code's balance (see Paper vs code, item 2).
• Localization is not measurable on the split without objects. That split has no object annotations, so Acc@K does not apply to it §3.3.
• Model selection and test splits. The appendix sweeps over d and L show test-split scores next to validation scores, and the text calls d = 128 and L = 1 suitable for the test splits app. B. The main setting is d = 256 and L = 2 §4. A reader who wants the best reported test numbers should not assume the main setting is the sweep's optimum.
• Multiview features and the 135-dim vector depend on a separate image-feature pipeline that the paper only cites §4 config.py. Budget for it before reproducing the feature-ablation rows.
• Dataset provenance. Scenes come from ScanNet and questions are generated from ScanRefer captions §3.2. The terms of use of ScanNet and ScanRefer were not checked for this card; check them before reusing the data.
• Pretrained VoteNet is not stated. The paper does not say whether the detector starts from pretrained weights §4. The code trains the detection losses in the same run loss_helper.py.
• Hardware and training time are not stated §5, so compute cost cannot be estimated from the paper.
• Human baseline is one best worker on a sample of questions §5.2; treat it as a rough ceiling, not a benchmark.

Lineage

• Builds on: ScanRefer, for the localization loss and the scene data §4 §5.1; VoteNet, for object proposals §4; MCAN, for the transformer fusion blocks §4. None of these is a page in this wiki yet.
• Compared against: 2D VQA with MCAN on sampled or oracle frames, VoteNet+MCAN, and ScanRefer+MCAN, pipeline and end-to-end §5.1. The appendix adds Oscar as a 2D model app. B.
• Superseded by: not known from content already in this wiki.