---
kind: concept
slug: view-selection
title: View selection
summary: >-
  Choosing the camera viewpoints that a multi-view network renders from a 3D shape. Fixed layouts (circular or spherical) are the default; a learned variant predicts bounded viewpoint offsets per shape and trains them through a differentiable renderer with the classifier. One card so far, with author-reported gains on ModelNet40 and ScanObjectNN.
tags: [multi-view, view-selection, differentiable-rendering, 3d-shape-recognition, point-cloud]
papers: ["2011.13244"]
related: [differentiable-rendering, multi-view-representation, multi-view-reconstruction, novel-view-synthesis]
status: draft
updated_at: 2026-11-03
---

## Overview

A multi-view classifier turns a 3D shape into several rendered images and
classifies the set. The open question is where the cameras go. The common
choice is a fixed layout, the same for every shape: a ring of azimuths or
evenly spaced points on a sphere [arxiv:2011.13244 §3.1]. The shared idea of
the learned variant is to make the viewpoints a function of the shape and to
train them with the classification loss, so no viewpoint labels are needed
[arxiv:2011.13244 §3.2] [arxiv:2011.13244 §7]. The invariant that all variants
share is that the renderer must be differentiable, so that gradients from the
loss reach the camera parameters [arxiv:2011.13244 §3.2] [arxiv:2011.13244 §4.4].

## Variants

### Fixed layouts (baseline)

Circular layouts (MVCNN) and spherical layouts (ViewGCN, RotationNet) place the
M cameras at the same angles for all shapes. The paper argues that such fixed
views can mislead the classifier on some shapes [arxiv:2011.13244 §3.1]. On
ModelNet40 with 12 views, the paper reports 90.4% overall accuracy for MVCNN
with fixed views and 92.6% with learned views; for ViewGCN, 93.0% against
93.8% [arxiv:2011.13244 tab. 7]. Card: [MVTN](/papers/2011.13244).

### Learned offsets from a canonical layout (MVTN-circular, MVTN-spherical)

A point encoder reads a coarse summary of the shape, and a regressor outputs
2M offsets. Each camera is the canonical angle plus a bounded offset, passed
through tanh so the angle stays in range [arxiv:2011.13244 §3.2] [arxiv:2011.13244 eq. 2].
The spherical version uses equally spaced points on a sphere for any M; the
circular version spreads azimuths over 360° [arxiv:2011.13244 app. A]. The
paper's main results use MVTN-spherical with ViewGCN [arxiv:2011.13244 §4.4]
[arxiv:2011.13244 tab. 7]. Card: [MVTN](/papers/2011.13244).

### Learned direct angles (MVTN-direct)

The regressor predicts the angles without a canonical layout. The appendix
reports that it overfits when M is greater than 4: the predicted views become
similar to one another [arxiv:2011.13244 app. C].

### Reinforcement-learning and recurrent view selection

The paper names earlier learned view selection with reinforcement learning and
recurrent networks, and reports that it worked only with limited success and
an elaborate training process [arxiv:2011.13244 §2]. No card for those works
exists in this wiki, so their methods are not described here.

## When to use which

| Situation | Use | Why |
|---|---|---|
| No differentiable renderer, or views rendered in advance with a non-differentiable tool | Fixed layout | The learned variant needs gradients through the renderer [arxiv:2011.13244 §3.2] [arxiv:2011.13244 §5.1] |
| Multi-view network is MVCNN-like and the paper's setting applies | Learned spherical or circular offsets | Largest reported gain over fixed views on MVCNN, 90.4 → 92.6 [arxiv:2011.13244 tab. 7] |
| Strong baseline such as ViewGCN | Learned views, expect a small gain | Reported gain is 0.8 points (93.0 → 93.8), and the best-of-four table value is above the mean (see Contradictions) [arxiv:2011.13244 tab. 7] [arxiv:2011.13244 app. B] |
| More than four views | Circular or spherical, not direct | Direct offsets overfit when M > 4 [arxiv:2011.13244 app. C] |
| Shapes with high-polygon meshes | Point-cloud rendering | Mesh rendering of high-polygon ShapeNet models was memory-heavy; the authors switched to point clouds [arxiv:2011.13244 §4.1] [arxiv:2011.13244 §4.4] |
| Rotated or partly hidden test inputs | Learned views may help | Author-reported robustness gains, e.g. at half occlusion MVTN keeps 67.1% against 53.5% for PointNet [arxiv:2011.13244 tab. 5] [arxiv:2011.13244 §5.4] |
| Tight compute budget | Measure before choosing | The module adds about 1.8 GFLOPs, but the paper's two cost tables disagree (see Contradictions) [arxiv:2011.13244 app. C] |

All numbers in this table are author-reported and come from a single paper.

## Implementation sketch

The common core is a training step in which a regressor produces bounded
viewpoint offsets, a differentiable renderer draws the views, and one loss
updates both the classifier and the regressor [arxiv:2011.13244 §3.2]
[arxiv:2011.13244 eq. 2]. Variants differ in the canonical layout, the offset
bounds and the encoder.

```pseudocode
# Learned view selection, one training step  [§3.2] [eq. 2]
# Input:  points (B, P, 3), shapes, labels (B,)
# Output: loss; classifier and view regressor updated together

def training_step(points, shapes, labels):
    # Summarise each shape with a point encoder  [§4.4]
    shape_feats = point_encoder(points)                 # (B, b)
    # Predict an offset for each of the M cameras, in azimuth and elevation  [§4.4]
    offsets = regressor(shape_feats, canonical_angles)  # (B, 2M)
    # Bound each offset with tanh and add it to the canonical view  [eq. 2]
    angles = canonical_angles + bound * tanh(offsets)   # (B, 2M)
    # Render M views per shape with a differentiable renderer  [§3.2]
    images = renderer(shapes, angles)                   # (B, M, 3, H, W)
    # Classify the set of views  [§3.1]
    loss = cross_entropy(multi_view_net(images), labels)
    # Gradients reach the regressor through the renderer  [§3.2]
    loss.backward()
    optimizer_step()
    return loss
```

## Contradictions

- **Mean versus best run.** The main tables report the best of four runs. For
  MVTN with 12 views on ModelNet40, the table gives 93.8% overall accuracy,
  while the mean over four runs is 93.4 ± 0.3 [arxiv:2011.13244 tab. 1]
  [arxiv:2011.13244 app. B].
- **Cost of the module.** The main table gives 1.78 GFLOPs, 4.24 ms and 3.5M
  parameters. The appendix gives 1.78 GFLOPs, 3.50 ms and 4.24M parameters for
  the same module [arxiv:2011.13244 tab. 8] [arxiv:2011.13244 app. C].
- **Regressor depth.** The main text says five layers; the appendix says four
  [arxiv:2011.13244 §4.4] [arxiv:2011.13244 app. A].
- **Azimuth bound.** The main text gives 180°; the appendix gives 180°/M for
  circular and spherical layouts [arxiv:2011.13244 §3.2] [arxiv:2011.13244 app. A].
- **Feature size and learning rate.** The paper gives a 1024-dimensional
  ResNet-18 feature and an MVTN learning rate of 0.001 [arxiv:2011.13244 §4.4]
  [arxiv:2011.13244 app. A]. The shipped code differs in these and in several
  schedule values; see [MVTN](/papers/2011.13244) for the paper-versus-code list
  before reproducing numbers.
- **Occlusion range.** The main text says occlusion goes to 100%; the appendix
  and the occlusion table stop at 75% [arxiv:2011.13244 §5.4]
  [arxiv:2011.13244 app. B] [arxiv:2011.13244 tab. 5].
- **Encoder training.** The paper does not say whether the point encoder is
  pretrained or trained together with the views [arxiv:2011.13244 app. A].

## Papers

- [MVTN: Multi-View Transformation Network for 3D Shape Recognition](/papers/2011.13244): learns bounded camera offsets per shape through a differentiable renderer; reports gains over fixed views on ModelNet40, ScanObjectNN and ShapeNet Core55.