Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

In plain words

Given one photo of an object, the network guesses the object's full 3D shape, as a grid of filled and empty cells. The usual way to train such a network needs the true 3D shape for every training photo, and those are hard to collect. This paper trains without them. It shows the network several views of the same object, and it projects its own 3D guess onto each camera's view. The network is scored on how well those flat outlines match the outlines seen in the photos.

A built-in camera model does the projection, and it is differentiable, so the error can flow back into the network. Over many objects, the network learns shapes that look right from every angle, even though its main training signal is 2D outlines. The authors report that its guesses are close to those of a network trained with real 3D shapes.

TL;DR

• Single-image 3D reconstruction that predicts a 32^3 occupancy grid §3.1 §4.
• A fixed perspective transformer projects the grid to silhouettes with a max over depth, so training needs only 2D silhouettes §3.2 eq. 4.
• Author-reported: on single-class chair training, the no-volume model (PTN-Proj) matches the volume-supervised models on the test set tab. 1.
• Author-reported: a single multi-class model keeps good IoU on 13 categories and generalises partly to 5 unseen categories tab. 2 tab. 3.
• Released Torch code (MIT) differs from the paper in a few ways that matter for reproduction, most importantly how the target silhouettes are made (see Paper vs code).

Problem

Learning 3D shape from images normally needs ground-truth volumes for supervision. The paper notes that such volumes "may not be available" for training, and that an agent typically only observes 2D silhouettes through its camera §3.1. The aim is to predict a volumetric shape from one image while needing only 2D observations, or with volumes used as optional extra supervision §3.1.

The paper assumes a clean white background and constant lighting, and ignores colour and texture. This makes the silhouette trivial to obtain §3. The shape is a binary voxel grid §3. Experiments use a subset of ShapeNetCore with 64x64 input images and 32^3 volumes §4.

Method

The model has three parts: a 2D convolutional encoder h, a 3D decoder g, and a perspective transformer P that projects a volume to a silhouette for a given viewpoint §3.1 §3.2 §4.

1. Encode the input image into a latent code h(I). The code should identify the object but not the viewpoint §3.1 §3.3.
2. Decode the code into a volume V̂ = g(h(I)) of 32^3 voxels §3.1 §4.
3. For each viewpoint, build a 4x4 perspective matrix from the camera calibration K and extrinsics (R, t) §3.2 eq. 3.
4. Sample the volume onto the camera grid with a bilinear kernel, then take the max along depth to get a silhouette. The max approximates ray tracing §3.2 eq. 4 app. A.
5. Compare predicted and observed silhouettes for n views with a squared error, averaged over views §3.1 eq. 1.
6. Optionally add a volume loss against ground-truth volumes §3.1.
7. Train in two stages: first pre-train the encoder with a rotation curriculum that makes the code view-invariant §3.3 §4. Then train the decoder with the projection loss, with neighbouring views in each mini-batch §3.3 §4.

Key equations, with the paper's own numbering:

\mathcal{L}_{proj}(I^{(k)}) = \sum_{j=1}^{n} \mathcal{L}_{proj}^{(j)}(I^{(k)}; S^{(j)}, \alpha^{(j)}) = \frac{1}{n}\sum_{j=1}^{n} \left\lVert P\big(f(I^{(k)}); \alpha^{(j)}\big) - S^{(j)} \right\rVert_2^2 eq. 1

\mathcal{L}_{comb}(I^{(k)}) = \lambda_{proj}\,\mathcal{L}_{proj}(I^{(k)}) + \lambda_{vol}\,\mathcal{L}_{vol}(I^{(k)}) eq. 2

The volume loss, not numbered in the paper, is \mathcal{L}_{vol}(I) = \lVert f(I) - V \rVert_2^2 §3.1.

\Theta_{4\times4} = \begin{bmatrix} K & 0 \\ 0^T & 1 \end{bmatrix} \begin{bmatrix} R & t \\ 0^T & 1 \end{bmatrix} eq. 3

U_i = \sum_{n}^{H} \sum_{m}^{W} \sum_{l}^{D} V_{nml}\, \max(0, 1-|x_i^s-m|)\, \max(0, 1-|y_i^s-n|)\, \max(0, 1-|z_i^s-l|)

S_{n'm'} = \max_{l'} U_{n'm'l'} eq. 4

The numbered training step below is reconstructed from the text. The paper has no algorithm box, so no alg. citation exists.

# Training step, reconstructed from the text (no algorithm box in the paper)
# Inputs:  images (B, 3, 64, 64) scaled to [-1, 1]; views (B, K) azimuths
#          silhouettes (B, K, 32, 32); volumes (B, 32, 32, 32) or None
# Output:  scalar loss

def training_step(images, views, silhouettes, volumes=None):
    # Encode each image into the identity code h(I)  [§3.3] [§4]
    codes = encoder(images)                          # (B, nz)

    # Decode a 32^3 occupancy grid per image  [§3.1] [§4]
    predicted_volumes = decoder(codes)               # (B, 32, 32, 32)

    # Project each predicted volume into every view  [§3.2] [eq. 3-4]
    predicted_silhouettes = project(predicted_volumes, views)

    # Silhouette loss: squared error per view, mean over views  [§3.1] [eq. 1]
    loss_proj = mean_over_views(
        sum_squared_error(predicted_silhouettes, silhouettes)
    )
    loss = lambda_proj * loss_proj

    # Volume loss, only when ground-truth volumes are used  [§3.1]
    if volumes is not None:
        loss_vol = sum_squared_error(predicted_volumes, volumes)
        loss = loss + lambda_vol * loss_vol

    # Combined objective; the lambda values are not stated in the paper  [eq. 2]
    return loss

# Perspective projection of volumes to silhouettes  [§3.2] [eq. 3-4]
# Inputs:  volumes (B, 32, 32, 32); views (K,) azimuths
# Output:  silhouettes (B, K, 32, 32)

def project(volumes, views):
    silhouettes = []
    for view in views:
        # 4x4 perspective matrix for this viewpoint  [§3.2] [eq. 3]
        matrix = perspective_matrix(view)              # (4, 4)
        # Bilinear resampling onto the camera grid  [§3.2] [eq. 4]
        resampled = bilinear_sample(volumes, matrix)   # (B, 32, 32, 32)
        # Max along depth gives the silhouette  [§3.2] [eq. 4]
        silhouettes.append(max_over_depth(resampled))  # (B, 32, 32)
    return stack(silhouettes, axis=1)                  # (B, K, 32, 32)

Two details the paper gives only in the appendix. The projection matrix is built from the inverse of Θ, and the camera coordinates are normalised by the homogeneous coordinate app. A eq. 7. The sampling kernel is the bilinear kernel max(0, 1 - |x|) app. A eq. 8.

Hyperparameters

Results

Author-reported. Setup: ShapeNetCore subset, 64x64 input, 32^3 volumes, voxel IoU with a 0.5 threshold, averaged over 24 views per test instance §4.1. The baseline is nearest-neighbour retrieval on VGG fc6 features, returning the ground-truth volume of the closest training image §4.1. Multi-class training uses 13 categories, with 20 percent of each category held out for testing §4.2. The source reports one number per cell with no variance.

Single class, chair tab. 1 (IoU; "chair-N" is a narrow azimuth range, see Pitfalls):

Multi-class, first seven categories tab. 2 (IoU):

Multi-class, last six categories tab. 2 (IoU):

Out of category tab. 3 (IoU, categories never in training):

Partial views, chair app. B tab. 4 (IoU):

Qualitative results are in fig. 3 (single class), fig. 5 (multi class) and fig. 6 (unseen categories). The paper says the volume-supervised model sometimes fails to form a solid chair, while the projection-trained models do not §4.1 fig. 3. View-dependent IoU is shown in fig. 4. Its caption says reconstruction from 0 degrees is harder than from 30 degrees, because of self-occlusion fig. 4.

Paper vs code

Compared against the official Torch code at commit bd17c6c5aa8c94e2ec193f82ddb5741d62242ed0. The code is owned by the first author, and its README names this paper README.md. Each point below has both citations.

• Target silhouettes come from ground-truth volumes, not from images. The paper says the 2D silhouettes come from image observations, which are trivial to segment on a white background §3 §3.1. The training script never loads masks. It projects the ground-truth voxel grid with the same projector and uses that as the target train_PTN.lua train_PTN.lua. The validation loop does the same train_PTN.lua. So PTN-Proj in this code is trained on silhouettes of ground-truth volumes. A reimplementation that uses only image masks would differ from the released code.
• Loss normalisation. The paper's eq. 1 averages over views with no extra factor eq. 1. The code uses a sum over all pixels, divided by 2 times the number of samples train_PTN.lua train_PTN.lua. This differs by a constant factor of 1/2, which the lambda values absorb. The lambda values themselves are not stated in the paper eq. 2.
• Encoder heads. The paper describes three fully connected layers of 1024, 1024 and 512 §4. The code has two layers of 1024, then a 512 identity head with ReLU, and a second 512 viewpoint head arch_PTN.lua. The fine-tuning script only uses the identity output train_PTN.lua. The viewpoint head is built but unused in this stage.
• Pre-training batch sizes. The paper gives 32, 32, 8, 4, 3, 2 for RNN-1, RNN-2, RNN-4, RNN-8, RNN-12, RNN-16 §4. The demo script runs kstep 2, 4, 8, 12, 16 with batch sizes 32, 32, 32, 16, 8 demo_pretrain_singleclass.sh. The paper does not say how kstep maps to RNN-k, so this card cannot resolve the mismatch.
• Encoder source. The paper does not say which pre-trained encoder fine-tuning uses §3.3. The code loads the RNN-16 checkpoint at epoch 20 train_PTN.lua. The encoder is then held fixed train_PTN.lua.
• Pre-training network. The paper only cites Yang et al. for the rotation-curriculum recurrent network §3.3. Its actor, mixer and decoders exist only in code arch_rotatorRNN.lua.
• Output activation. The paper does not state an output nonlinearity for the decoder. The code ends with a sigmoid arch_PTN.lua. The paper's max-projection formula assumes binary volumes §3.2 eq. 4.
• Camera geometry. The paper gives the 30 degree elevation and 15 degree azimuth steps §4. The focal length (sqrt(3)/2) and the camera distance (focal + 0.5) appear only in code train_PTN.lua train_PTN.lua.
• No discrepancy found in the IoU threshold (0.5) §4 eval_quant_test.lua.

Pitfalls

• Silhouette targets. The released PTN-Proj is not free of 3D data in its targets (see Paper vs code). An implementation that follows the paper's text needs real masks from the images. The paper says these are trivial on a white background, but it does not say how to extract them in other settings §3.
• Max projection and gradients. The projection takes the max along depth, and the paper says this approximates ray tracing §3.2. The code uses nn.Max arch_PTN.lua. In Torch this sends the gradient only to the largest value on each ray; the paper does not discuss this. The paper's formula assumes binary volumes, while the code's decoder output is a sigmoid in (0, 1) arch_PTN.lua.
• Cost per update. Each object contributes all 24 views §4. At batch size 6, a fine-tuning step builds 144 projected volumes of 32^3 train_PTN.lua train_PTN.lua. The projector runs in every step, so compute per step scales with the view count.
• Pre-training time. The README estimates pre-training at several days on one TITAN X card README.md. The paper gives no time budget §4.
• Partial views. Appendix B uses 8 of 24 views in a 105 degree range, and a second setting uses 8 views with 45 degree steps app. B. No script for these settings is in the repository (checked against the file list). Reproducing them means changing the view subset in the training loop. The loop assumes all 24 views train_PTN.lua.
• Camera at test time. Training uses the known camera matrix for the projection §3.2. Single-view prediction needs only the encoder and decoder §3.1. The camera is not needed at test time.
• Data split and rendering. The 80/20 split, the 24 renders per object and the voxel grids come from a download script (prepare_data.sh) that this card did not open. The paper's description §4 is the only documented source here.
• Single runs. Each cell is one number with no seed variance reported. The gap between PTN-Proj and PTN-Comb on chair test is 0.0040 IoU tab. 1. The paper's claim that the two "perform as good as" each other rests on that gap.
• Chair-N and chair-S. Chair-N and chair-S restrict training views to a range or a sparse set app. B. The paper says the evaluation still uses all views app. B. Compare test numbers only within the same column.
• Out-of-category. The paper says the projection-loss model generalises better on train, motorbike and cabinet §4.3. In tab. 3, PTN-Proj:single is highest on all five categories. PTN-Proj:multi is highest on bed, bookshelf, cabinet and motorbike, but PTN-Comb:multi is highest on train (0.3744 vs 0.3670). Check each cell before quoting a "best" claim.

Lineage

• Builds on: Spatial transformer networks (Jaderberg et al., cited as jaderberg2015spatial), for the differentiable sampling operator §1 §3.2. No card in this wiki.
• Builds on: the rotation curriculum from Yang et al. (2015), used to pre-train the view-invariant encoder §3.3 §4. No card in this wiki.
• Builds on: space carving and visual hulls (cited as kutulakos2000theory), which justify using silhouettes as the supervision §3.1.
• Compared against: CNN-Vol (the same network with volume loss only) and nearest-neighbour retrieval on VGG fc6 features §4.1 tab. 1 tab. 2 tab. 3. No other published method is compared numerically.
• Related but not compared: 3D-R2N2 (Choy et al.), the 3D generative model of Wu et al., Girdhar et al. and Tatarchenko et al. §2. Rezende et al. and Wu et al. (2016) are described as concurrent work §2.
• Superseded by: not known from content already in this wiki.