Multi-view representation

Overview

A multi-view representation describes one scene from several viewpoints and combines the per-view outputs. The reason, in the one card so far: a spatial phrase such as "left of the bed" depends on where the viewer stands, so a model fixed to one view can fail when the view changes 2204.02174 §1. The combination is order-independent, so the score for each object does not depend on which view came first 2204.02174 §3.4.

Variants

Only one variant is covered so far. No other card lists this slug. For a different view-based scene representation, see Bird's-eye-view (BEV) representation.

Rotated views, shared object features, averaged fusion

The scene is rotated about the vertical Z axis into N equal-angle views, θ_j = 2π(j−1)/N 2204.02174 §3.1. Each candidate object's point-cloud feature is computed once and shared by all views. A per-view position code, made from the rotated box centre and size, is added to it 2204.02174 §3.2. One transformer decoder, shared across views, fuses each view's object features with the query's word features 2204.02174 §3.3. The N outputs are averaged for each object before a two-layer head scores it 2204.02174 §3.4. Training also rotates the whole scene by one of the N angles 2204.02174 §4.4.1. The main model uses N = 4 2204.02174 §4.2. Card: Multi-View Transformer for 3D Visual Grounding.

When to use which

The choices below depend on constraints. All numbers are the paper's own ablations on Nr3D, so they do not show what happens on other tasks.

Implementation sketch

The core of the variant. Positional encoding and aggregation are the parts a different method would change.

# Shared point feature per object, computed once  [§3.2] [eq. 5]
shape = encode_points(object_points)              # (M, d)

fused = []
for j in range(N):
    # Rotate the box centre into view j  [§3.1] [§3.2] [eq. 3] [eq. 6]
    centre_j = rotate_z(box_centre, 2 * pi * j / N)
    # Position code of each object in view j  [§3.2] [eq. 7]
    pos_j = encode_box(centre_j, box_size)        # (M, d)
    # View feature = shared feature + position code  [§3.2] [eq. 8]
    obj_j = shape + pos_j
    # Shared decoder fuses view j with the words  [§3.3] [eq. 10]
    fused.append(decoder(obj_j, word_feats))

# Order-independent aggregation: mean over views  [§3.4] [eq. 11]
agg = sum(fused) / N
scores = score_head(agg)                          # (M,)

Contradictions

None known. Only one card covers this concept, so no two papers disagree.

Reproduction notes

• The paper says the text encoder starts from the first three BERT-base layers 2204.02174 §4.2. The released code, as the card reads it, builds those layers randomly. Details are on the card under "Paper vs code".
• The paper does not say how the reported checkpoint is chosen. The card notes that the released code keeps the checkpoint with the best test-split accuracy, so the reported Nr3D and Sr3D numbers carry a selection effect of unknown size.
• The ScanRefer results rest on separate code that the card did not check.

Papers

• Multi-View Transformer for 3D Visual Grounding: fuses N rotated views of a point-cloud scene with a shared transformer decoder, then averages the views before scoring.