Multi-View Transformer for 3D Visual Grounding

In plain words

The task is to point at the right object in a 3D scan of a room when someone describes it, for example "the pillow to the left of the bed". The trouble is that words like "left" depend on where you stand. A pillow that is on the left from the speaker's position is on the right from the opposite side of the bed, so a model trained on one viewpoint can fail when the view changes.

The idea of this paper is to stop relying on one viewpoint. The model rotates the whole scene to several fixed angles around the vertical axis, so it sees where every object sits from each side. It matches the description against each of these views and then combines the answers into one score per object. Because the combination does not depend on which view came first, the result does not depend on where the scene started.

The authors report that this beats earlier methods on the standard benchmarks Nr3D and Sr3D, including methods that also use 2D images. The model is built on a transformer, the same family of network used in language models.

TL;DR

• Task: 3D visual grounding, choosing the object in a point-cloud scene that a sentence describes.
• Method: Multi-View Transformer. Each object's features are shared across N rotated copies of the scene, the object-position code differs per view, a transformer decoder fuses each view with BERT word features, and the N results are averaged before scoring.
• Headline (author-reported): 55.1% overall on Nr3D versus 43.9% for the best non-assisted baseline; 64.5% on Sr3D versus 57.4% tab. 1 tab. 2.
• Relevance: useful if you work on language-conditioned 3D object selection, or want a view-robust fusion of spatial information with text. Code is official and public, but its license is not stated (see Paper vs code).

Problem

Earlier 3D grounding methods mostly work from one fixed view of the scene. The authors argue that the view changes the coordinates that encode "left" or "right", so these methods fail when the viewpoint in the description differs from the one the model learned §1. Simple rotation augmentation during training did not fix this in their experiments §1 tab. 4. Asking the model to predict the speaker's view first adds cost and errors when the prediction is wrong §1.

The paper assumes candidate objects are given: ground-truth objects on Nr3D and Sr3D, and detector proposals on ScanRefer §3.2 §4.2.

Method

Overview. The scene S is rotated about the vertical Z axis into N views, the same network f runs on each view, and an order-independent function g combines the N outputs §3.1 eq. 1 eq. 4 fig. 3. The equal-angle set of views is θ_j = 2π(j−1)/N §3.1 eq. 3. The rotation matrix is defined as the transpose of the standard 2D rotation, so R(θ)×S applied to column vectors is the same as the code's row-vector product §3.1 eq. 2.

Steps, with the component that does each step:

1. Point-cloud features, computed once. Each candidate object is sampled to 1024 points with RGB and XYZ, and a PointNet++ encoder with a linear layer and layer norm gives x_i §3.2 eq. 5. These features do not depend on the view, so they are shared across all N views, which is the reason for the decoupling §3.2.

2. Multi-view positional encoding. For each view j, the object's box centre is rotated by θ_j §3.2 eq. 6. Its rotated centre and box size are mapped to a d-dimensional code with a linear layer and layer norm §3.2 eq. 7.

3. Object feature per view. The view-specific object feature is the shared feature plus the positional code, o_i^j = x_i + PE(b_i^j) §3.2 eq. 8.

4. Language encoding. The query is embedded with BERT, and the sentence-level feature l_s and the per-word features l_i are kept §3.3 eq. 9. The BERT encoder is fine-tuned during training §3.3.

5. Multi-modal fusion. A standard transformer decoder, shared by all views, takes the object features of view j as the sequence and the word features as memory. In each layer it applies self-attention among objects, then cross-attention to the words §3.3 eq. 10.

6. Multi-view aggregation. The per-view outputs are averaged for each object, g_i = Σ_j f_i^j / N §3.4 eq. 11. Two fully connected layers on g_i give the grounding score §3.4. The paper also tests aggregation at earlier stages and max or avg+max pooling §3.4 tab. 6.

7. Auxiliary object classification. Object features are scored against the BERT features of the category names with an inner product, supervised by cross-entropy. The category sets have 524 classes on Nr3D and 607 on Sr3D §3.5.

8. Auxiliary text classification. Two fully connected layers on the sentence feature l_s predict the target's category, supervised by cross-entropy §3.3 §3.6.

9. Loss. L_total = L_ref + α(L_text + L_obj), with α = 0.5 by default §3.6 eq. 12.

Training uses random rotation augmentation: during training, the whole scene is rotated by one of N angles around Z before the view generation in step 2 §4.4.1 referit3d_net.py.

# Forward pass of one training batch
# Input:  points (B, M, 1024, 6), boxes (B, M, 4: xyz + size), tokens
# Output: loss scalar

def forward_batch(points, boxes, tokens, targets, class_labels):
    # 1. Training only: rotate the scene by one random angle per scene
    #    from the N angles 2*pi*i/N  [§4.4.1] [eq. 3]
    points, boxes = rotate_random_angle(points, boxes, rotate_number)

    # 2. Shared point-cloud feature per object  [§3.2] [eq. 5]
    shape_feats = layer_norm(linear(point_net_pp(points)))  # (B, M, d)

    # 3. Box centre rotated into each of the N views  [§3.2] [eq. 6]
    view_boxes = rotate_centres_per_view(boxes, view_number)  # (B, N, M, 4)

    # 4. Positional code of each object in each view  [§3.2] [eq. 7]
    pos_feats = layer_norm(linear(view_boxes))           # (B, N, M, d)

    # 5. Object feature per view = shared feature + position  [§3.2] [eq. 8]
    object_feats = shape_feats[:, None] + pos_feats      # (B, N, M, d)

    # 6. Word features and sentence feature from BERT  [§3.3] [eq. 9]
    word_feats = bert_first_layers(tokens)               # (B, K, d)
    sentence_feat = word_feats[:, 0]                     # (B, d)

    # 7. Shared decoder fuses each view with the words  [§3.3] [eq. 10]
    fused = decoder(object_feats, word_feats)            # (B, N, M, d)

    # 8. Average over views, then score each object  [§3.4] [eq. 11]
    averaged = sum(fused / view_number, axis=view)       # (B, M, d)
    scores = two_linear_layers(averaged)                 # (B, M)

    # 9. Auxiliary losses  [§3.5] [§3.6]
    obj_logits = object_feats_to_category_text(shape_feats, word_text)
    txt_logits = two_linear_layers(sentence_feat)

    # 10. Total loss, alpha = 0.5  [§3.6] [eq. 12]
    ref_loss = cross_entropy(scores, targets)
    obj_loss = cross_entropy(obj_logits, class_labels)
    txt_loss = cross_entropy(txt_logits, target_class)
    return ref_loss + 0.5 * (txt_loss + obj_loss)

Key equations, as printed in the paper:

\text{VG}(\text{S}, Q) = g\big(f(\text{S}^1, Q), \dots, f(\text{S}^N, Q)\big) eq. 4

g_i = \sum_{j=1}^{N} \frac{f^{j}_{i}}{N} eq. 11

\mathcal{L}_{total} = \mathcal{L}_{ref} + \alpha(\mathcal{L}_{text} + \mathcal{L}_{obj}) eq. 12

Hyperparameters

Values in the paper are cited to the paper. Values marked "code only" are set in the released code and not stated in the paper. Nothing is listed here that neither source gives.

Results

Author-reported. Setup: Nr3D and Sr3D with ground-truth candidate objects, accuracy as the fraction of utterances whose chosen object is correct; ScanRefer with PointGroup proposals, Acc@0.25 and Acc@0.5 §4.1 §4.2. Baselines are the paper's own table rows, trained with the same setting. "Non-assisted" means no extra 2D input tab. 1 tab. 2.

For reference, the 2D-assisted SAT reports 49.2 on Nr3D, 53.9 with Sr3D, 56.5 with Sr3D+ and 57.9 on Sr3D; the paper compares against these tab. 1 tab. 2. The ScanRefer rows use the paper's own ScanRefer table, where the InstanceRefer row is typed "IntanceRefer".

Ablation on Nr3D (overall accuracy, %) tab. 4:

Number of views, Nr3D tab. 5:

Aggregation, Nr3D, 4 views tab. 6:

Cost: on a TITAN X (Pascal), the average inference time per scene-utterance pair is 264 ms with N=4 and 261 ms with N=1 §4.3.

Paper vs code

Code: https://github.com/sega-hsj/MVT-3DVG at commit f680e274488c46b2bbc53e562c19e5f054576300. The link is in the abstract, so it is treated as the authors' code. The repository has no LICENSE file; setup.py declares MIT setup.py. We therefore record the license as not stated and quote no code verbatim.

• Text-encoder initialisation. Paper: the text encoder is initialised from the first three layers of BERT-base §4.2. Code: after loading the pretrained BERT, the encoder layers are replaced by the layers of a newly built, randomly initialised BertModel(BertConfig()) truncated to three layers referit3d_net.py. As written, the three transformer layers start random; only the pretrained embedding layer is kept. We could not check the released weights, which are not in the repository.
• Which parameters get the 0.1 learning-rate multiplier. Paper: "the learning rate of the transformer layer" is multiplied by 0.1 §4.2. Code: both the BERT encoder and the fusion decoder get 0.1, and the object encoder and heads get the full rate train_referit3d.py.
• Rotation-augmentation count. Paper: θ_aug is sampled from the view angles with N = 4 §4.4.1. Code: a separate rotate_number (default 4) sets the augmentation angles arguments.py referit3d_net.py. They match at the default; they differ if the two arguments differ.
• Evaluation seeds. Paper reports mean ± std but does not say how many runs tab. 1. Code: the test analysis samples object points with five seeds (2020, 1, 10, 20, 100) and averages deepnet_predictions.py.
• Model selection. Paper does not describe how the reported model is chosen. Code: best_model.pth is saved whenever the accuracy on the test split improves train_referit3d.py.
• Unit-sphere normalisation. Not stated in the paper. Code default is off arguments.py; the README's training and test commands turn it on README.md.
• Dropout, feed-forward width, number of heads, utterance length, PointNet++ layout are not in the paper and appear only in code (see Hyperparameters).
• ScanRefer. The README points to a separate repository, MVT_ScanRefer, for ScanRefer README.md. We did not clone or check it, so the ScanRefer numbers are not verified against code.
• Object classification input. Paper: the category logits are an inner product of the category text features and the object feature x_i §3.5. Code matches this: the object feature passes through the linear and layer-norm mapping of eq. 5 before the product referit3d_net.py referit3d_net.py. No discrepancy found.
• View rotation convention. The code multiplies row vectors by the matrix referit3d_net.py. This equals the paper's R(θ) applied to column vectors §3.1 eq. 2. No discrepancy, but a column-vector implementation with the standard matrix gives the opposite sign of θ.

Pitfalls

• Random text encoder. If you use the released code literally, the three BERT layers are not pretrained (see Paper vs code). Check this first when reproducing the numbers, and either load the pretrained layers or report the change.
• Gains are in percentage points, not relative. "+11.2%" on Nr3D is 55.1 − 43.9 absolute points tab. 1 §4.3.
• Joint training uses more data. The Sr3D and Sr3D+ rows train on synthetic utterances as well §4.1 tab. 1. Compare like with like.
• 2D-assisted comparisons. SAT uses 2D semantics during training tab. 1. The paper's "outperforms even with extra 2D" claim is against that method only.
• Small ScanRefer margin. Over InstanceRefer, the 4-view model gains 0.57 points on Acc@0.25 and 0.33 on Acc@0.5 tab. 3. The abstract's "significantly outperforms all state-of-the-art methods" is broader than this margin.
• Cost grows with views. The fusion decoder runs once per view, so its compute and memory grow with N referit3d_net.py. The paper's timing (264 ms with N=4, 261 ms with N=1) does not say which stages it covers §4.3; re-measure before you plan a budget.
• Test-split model selection. The best checkpoint is chosen on the test split (see Paper vs code). Reported accuracies on Nr3D and Sr3D therefore carry a selection effect; it is not stated how large.
• Training with several views helps even at test time. Training with four views and testing with one gives 51.6%, above training with one view (46.2%) tab. 5. The paper credits this to multi-view training, not only to extra test views §4.4.2.
• Unstated details. The paper gives no warm-up, no decoder head count, no dropout, no seed, and no unit-sphere setting. The code uses values for these (see Hyperparameters); they are not confirmed by the paper.
• Object proposals are assumed. On Nr3D and Sr3D the proposals are the ground-truth objects, so the numbers exclude detection errors §4.2.

Lineage

• Builds on: the ReferIt3D benchmarks Nr3D and Sr3D/Sr3D+, which are the datasets used throughout §4.1; BERT for the language encoder §3.3; PointNet++ for point features §3.2; the transformer decoder for fusion §3.3.
• Compared against: LanguageRefer, InstanceRefer, 3DVG-Transformer, TransRefer3D, SAT (2D-assisted), and on ScanRefer also TGNN and ScanRefer tab. 1 tab. 2 tab. 3.
• Superseded by: not known from content already in this wiki.