View selection

Overview

A multi-view classifier turns a 3D shape into several rendered images and classifies the set. The open question is where the cameras go. The common choice is a fixed layout, the same for every shape: a ring of azimuths or evenly spaced points on a sphere 2011.13244 §3.1. The shared idea of the learned variant is to make the viewpoints a function of the shape and to train them with the classification loss, so no viewpoint labels are needed 2011.13244 §3.2 2011.13244 §7. The invariant that all variants share is that the renderer must be differentiable, so that gradients from the loss reach the camera parameters 2011.13244 §3.2 2011.13244 §4.4.

Variants

Fixed layouts (baseline)

Circular layouts (MVCNN) and spherical layouts (ViewGCN, RotationNet) place the M cameras at the same angles for all shapes. The paper argues that such fixed views can mislead the classifier on some shapes 2011.13244 §3.1. On ModelNet40 with 12 views, the paper reports 90.4% overall accuracy for MVCNN with fixed views and 92.6% with learned views; for ViewGCN, 93.0% against 93.8% 2011.13244 tab. 7. Card: MVTN.

Learned offsets from a canonical layout (MVTN-circular, MVTN-spherical)

A point encoder reads a coarse summary of the shape, and a regressor outputs 2M offsets. Each camera is the canonical angle plus a bounded offset, passed through tanh so the angle stays in range 2011.13244 §3.2 2011.13244 eq. 2. The spherical version uses equally spaced points on a sphere for any M; the circular version spreads azimuths over 360° 2011.13244 app. A. The paper's main results use MVTN-spherical with ViewGCN 2011.13244 §4.4 2011.13244 tab. 7. Card: MVTN.

Learned direct angles (MVTN-direct)

The regressor predicts the angles without a canonical layout. The appendix reports that it overfits when M is greater than 4: the predicted views become similar to one another 2011.13244 app. C.

Reinforcement-learning and recurrent view selection

The paper names earlier learned view selection with reinforcement learning and recurrent networks, and reports that it worked only with limited success and an elaborate training process 2011.13244 §2. No card for those works exists in this wiki, so their methods are not described here.

When to use which

All numbers in this table are author-reported and come from a single paper.

Implementation sketch

The common core is a training step in which a regressor produces bounded viewpoint offsets, a differentiable renderer draws the views, and one loss updates both the classifier and the regressor 2011.13244 §3.2 2011.13244 eq. 2. Variants differ in the canonical layout, the offset bounds and the encoder.

# Learned view selection, one training step  [§3.2] [eq. 2]
# Input:  points (B, P, 3), shapes, labels (B,)
# Output: loss; classifier and view regressor updated together

def training_step(points, shapes, labels):
    # Summarise each shape with a point encoder  [§4.4]
    shape_feats = point_encoder(points)                 # (B, b)
    # Predict an offset for each of the M cameras, in azimuth and elevation  [§4.4]
    offsets = regressor(shape_feats, canonical_angles)  # (B, 2M)
    # Bound each offset with tanh and add it to the canonical view  [eq. 2]
    angles = canonical_angles + bound * tanh(offsets)   # (B, 2M)
    # Render M views per shape with a differentiable renderer  [§3.2]
    images = renderer(shapes, angles)                   # (B, M, 3, H, W)
    # Classify the set of views  [§3.1]
    loss = cross_entropy(multi_view_net(images), labels)
    # Gradients reach the regressor through the renderer  [§3.2]
    loss.backward()
    optimizer_step()
    return loss

Contradictions

• Mean versus best run. The main tables report the best of four runs. For MVTN with 12 views on ModelNet40, the table gives 93.8% overall accuracy, while the mean over four runs is 93.4 ± 0.3 2011.13244 tab. 1 2011.13244 app. B.
• Cost of the module. The main table gives 1.78 GFLOPs, 4.24 ms and 3.5M parameters. The appendix gives 1.78 GFLOPs, 3.50 ms and 4.24M parameters for the same module 2011.13244 tab. 8 2011.13244 app. C.
• Regressor depth. The main text says five layers; the appendix says four 2011.13244 §4.4 2011.13244 app. A.
• Azimuth bound. The main text gives 180°; the appendix gives 180°/M for circular and spherical layouts 2011.13244 §3.2 2011.13244 app. A.
• Feature size and learning rate. The paper gives a 1024-dimensional ResNet-18 feature and an MVTN learning rate of 0.001 2011.13244 §4.4 2011.13244 app. A. The shipped code differs in these and in several schedule values; see MVTN for the paper-versus-code list before reproducing numbers.
• Occlusion range. The main text says occlusion goes to 100%; the appendix and the occlusion table stop at 75% 2011.13244 §5.4 2011.13244 app. B 2011.13244 tab. 5.
• Encoder training. The paper does not say whether the point encoder is pretrained or trained together with the views 2011.13244 app. A.

Papers

• MVTN: Multi-View Transformation Network for 3D Shape Recognition: learns bounded camera offsets per shape through a differentiable renderer; reports gains over fixed views on ModelNet40, ScanObjectNN and ShapeNet Core55.