Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images

In plain words

Give the system one photo of a chair, or several photos from different sides, and it outputs a small 3D block of cubes (voxels) that is filled where the object is. Each photo is handled by the same network on its own, so the result does not depend on the order in which the photos arrive.

Each photo produces its own rough 3D guess. The guesses are then combined by a small network that decides, for every cube, which photo's guess to trust most. Parts that one photo could not see are filled in from the photos that could. Finally, a last network cleans up the combined shape.

The authors say the same design handles one image and many images, runs faster than an earlier recurrent method, and works on real photos after training on their own synthetic dataset.

TL;DR

• Feed-forward voxel reconstruction from 1 to 20 RGB views, with one shared network for every view §3.1.
• Per-view coarse volumes are fused by learned per-voxel softmax weights, then corrected by a refiner §3.2.3 §3.2.4.
• Author-reported ShapeNet single-view IoU is 0.670 (/A) and 0.645 (/F) at 32^3 tab. 1.
• Needs no camera poses, and each view costs one encoder-decoder pass §1.
• Ships its own dataset (Things3D) and a 128^3 variant for ShapeNet-Cars §4 tab. 4.

Problem

Recurrent fusion of multi-view features (for example 3D-R2N2) is order dependent and can forget early inputs; the authors also say it is slow because images are processed one after another §1. Max or mean pooling of features avoids order dependence but keeps only the maximum or average, and attention pooling before the decoder struggles with cluttered backgrounds §1 §2. Multi-view methods that use geometry need camera extrinsics, which are often unavailable §2. Synthetic training data with random backgrounds is also limited in realism §4.

Method

Pix2Vox++ has four modules: an encoder, a decoder that produces one coarse volume per input, a multi-scale context-aware fusion module, and a refiner §1 fig. 2. Volumes are binary (0 empty, 1 occupied) on a 32^3 grid §3.1. Two variants share the design: /F (fewer parameters, ResNet-18 encoder) and /A (ResNet-50 encoder) §3.2 §3.2.1.

Encoder §3.2.1. The first ResNet blocks yield a 512 x 28^2 map from a 224 x 224 x 3 image. Three conv blocks (3x3 kernels, padding 1, batch norm, ReLU) follow; max pooling (2x2) follows the second and third ReLU. For /A the channels are 512, 256 and 256, giving a 256 x 7^2 map; for /F the channels are 128, 64 and 64, giving 64 x 7^2 §3.2.1.

Decoder §3.2.2. The per-view map is reshaped and passed through five transposed 3D convolutions (first four 4^3 kernels, stride 2, padding 1; last one a 1^3 filter), each followed by batch norm and ReLU except the last, which ends in a sigmoid. For /A the channels are 512, 128, 32, 8, 1; this produces a 32^3 occupancy volume §3.2.2. The paper also describes deeper decoder variants for 64^3 and 128^3 outputs §3.2.2 tab. 15.

Multi-scale context-aware fusion §3.2.3. For each view r, the context c_r concatenates the outputs of the last two decoder layers §3.2.3. A context scoring network of five 3D conv sets (3^3 kernels, padding 1, batch norm, leaky ReLU; 9 channels except the last layer with 1 channel) produces a score volume m_r. The first four layers are concatenated and fed to the fifth §3.2.3. Scores are normalised across the n views with a softmax eq. 1, and the fused volume is the score-weighted sum of the coarse volumes eq. 2. Pooling over views therefore happens per voxel fig. 3 fig. 4.

Refiner §3.2.4. The refiner is described as a residual network with U-Net-style connections §3.2.4. At 32^3 its encoder has three 3D convs (4^3 kernels, padding 2, channels 32, 64, 128, each with batch norm, leaky ReLU and 2^3 max pooling), two fully connected layers (2048 and 8192), and a decoder of three transposed 3D convs (4^3 kernels, padding 2, stride 1, channels 64, 32,

1. §3.2.4. A 64^3 variant adds a conv before the first layer and a transposed conv before the last; a 128^3 variant uses five conv and five transposed-conv layers §3.2.4.

Loss §3.3. The loss is the mean voxel-wise binary cross entropy between the prediction and the ground truth eq. 3.

# Pix2Vox++ inference  [§3.1] [§3.2] [fig. 1]
# Input:  images (B, V, 3, 224, 224), V views per object
# Output: refined volume (B, 32, 32, 32)
def reconstruct(images):
    # Encode each view with the shared encoder  [§3.2.1]
    features = encoder(images)                  # (B, V, 256, 7, 7) for /A
    # Decode each view to a coarse volume and its 9-channel context [§3.2.2]
    context, coarse = decoder(features)
    # context: (B, V, 9, 32, 32, 32); coarse: (B, V, 32, 32, 32)
    # Score every voxel of every view  [§3.2.3] [eq. 1]
    scores = fusion_scorer(context)             # (B, V, 32, 32, 32)
    weights = softmax(scores, over_views)       # (B, V, 32, 32, 32)
    # Weighted sum over views  [§3.2.3] [eq. 2]
    fused = sum_over_views(weights * coarse)    # (B, 32, 32, 32)
    # Clamp to [0, 1]; the paper does not state this step [code: models/merger.py#L79]
    fused = clamp(fused, 0, 1)
    # Correct the fused volume  [§3.2.4]
    refined = refiner(fused)                    # (B, 32, 32, 32)
    return refined

Key equations:

s_r^{(i,j,k)} = \frac{\exp(m_r^{(i,j,k)})}{\sum_{p=1}^{n}\exp(m_p^{(i,j,k)})} eq. 1

v^f = \sum_{r=1}^{n} s_r \, v_r^c eq. 2

\ell = \frac{1}{N}\sum_{i=1}^{N}\left[gt_i\log(p_i) + (1-gt_i)\log(1-p_i)\right] eq. 3

Training (from §5.3): batch size 64, Adam with beta1 0.9 and beta2 0.999, initial learning rate 0.001 halved after 150 epochs, 250 epochs on single-view images without the fusion module, then 100 epochs on multi-view images §5.3.

Hyperparameters

Results

Author-reported. Setup: ShapeNet (44K models, 13 categories, 24 views per model) §5.1; Pix3D chairs §5.5; Things3D §4. Baselines are as listed in each table. Authors' own table reports /A as best overall.

Paper vs code

Code location. The paper links https://gitlab.com/hzxie/Pix2Vox §1. The GitLab project is described as the Pix2Vox++ implementation and is MIT licensed (read via WebFetch, not a local clone). The GitHub repository hzxie/Pix2Vox has a default branch whose README says it implements the earlier Pix2Vox paper (arXiv 1901.11153) with a VGG16 encoder, so it is not the Pix2Vox++ code. No commit hash could be obtained for the GitLab code (the project page returned HTTP 403), so code_commit is not set. The citations below are to GitLab paths; line numbers come from summaries of those files and may be off by a few lines.

Encoder. Paper: ResNet-18 for /F and ResNet-50 for /A §3.2.1. Code: a ResNet-50 backbone with pretrained weights, using the first six children (up to layer2), with no frozen parameters encoder.py. Only the /A variant was read; no ResNet-18 config was found. The layers after the backbone match the paper's /A channels (512, 256, 256, with pooling after the second and third blocks) and the 256 x 7^2 output encoder.py.

Decoder. Channels 512, 128, 32, 8, 1 with a final 1 x 1 x 1 sigmoid layer match the paper §3.2.2 decoder.py. The reshape to 2^3 uses 1568 channels, which equals 256 x 7^2 divided by 8 and so is consistent with the 256 x 7^2 feature map decoder.py decoder.py. The 9-channel context (8 channels from layer 4 plus 1 from layer 5) matches the "last two decoder layers" context decoder.py §3.2.3. The 64^3 and 128^3 decoder layers from the paper were not seen in the code read.

Fusion network. Paper: five 3D conv sets, with the first four concatenated into the fifth §3.2.3. Code: six Conv3d layers; the fifth takes a 36-channel concatenation of the first four outputs (4 x 9 channels) and the sixth outputs one score channel merger.py merger.py. Softmax over views matches eq. 1 merger.py. The code clamps the fused volume to [0, 1] merger.py; the paper does not state this.

Refiner. Paper: U-Net connections §3.2.4. Code: encoder and decoder features are added element-wise, not concatenated, and the output of the last skip is multiplied by 0.5 refiner.py. The convolution widths (32, 64, 128) and fully connected sizes (8192 to 2048 and back) match the paper refiner.py.

Loss. Paper: one BCE term on the output §3.3. Code: both the fused prediction and the refined prediction get a BCE term, each multiplied by 10 train.py. The fused term is computed on the merger output, and both gradients flow back through the merger train.py.

Schedule. Paper: 250 epochs without fusion, then 100 epochs with fusion §5.3. Code: one loop over NUM_EPOCHES = 250 config.py. The merger and refiner start at epoch 0 config.py, so no separate fusion-only stage was found. Paper: one learning rate of 0.001 §5.3. Code: merger 1e-4 config.py; all four modules decay by 0.5 at epoch 150 config.py config.py.

Batch and input. Batch size 64 and 224 x 224 input match §5.3 config.py. Adam betas match config.py.

Not found in the code read. The paper describes a random-background augmentation of training images §5.1. The earlier ICCV code in the GitHub repository has random crop, random background, colour jitter, noise and RGB permutation transforms [github: core/train.py#L34-L49], but the transforms in the GitLab Pix2Vox++ code were not read.

No discrepancies found in the view-fusion equation, the threshold, or the Adam settings.

Pitfalls

• Result text and tables disagree. The text says /F and /A beat every baseline on ShapeNet §5.4.1. Table 1 gives IM-Net 0.659 IoU against /F 0.645, and table 2 gives IM-Net 0.405 F-Score against /F 0.394 tab. 1 tab. 2. On Things3D the text says both beat all baselines §5.6.1, but /F ties IM-Net at 0.419 IoU and is below IM-Net on F-Score tab. 6 tab. 7. Use the tables.
• Two-stage training is only partly specified. The paper gives 250 + 100 epochs §5.3. The code runs one loop with fusion active from epoch 0, so the stage boundary, the stage-2 learning rate and the stage-2 loss are not visible. Implement the paper as written and state which one you chose.
• Ambiguous LR decay. "Decayed by 2" is read as halving (gamma 0.5) in the code config.py. The milestone is 150 epochs for every module.
• Loss scale is code-only. The factor 10 is not in the paper §3.3. Without it the effective learning rate and the balance between the fused and refined losses change.
• Context scoring width. The paper says 9 channels except the last layer §3.2.3. The code's fusion network starts at 9 channels and concatenates four 9-channel outputs, so the exact widths of the concatenated path are only stated through the code summary.
• Refiner skip type. The paper's "U-Net connections" are implemented as additive residual skips with a 0.5 factor refiner.py. Concatenation skips would change parameter count and memory.
• Clamp after fusion. The clamp to [0, 1] is in the code, not the paper merger.py. Gradients are zero where the sum leaves [0, 1].
• F-Score threshold is not numeric. "F-Score@1%" §5.2 does not define the distance threshold d in units of the volume or the object. Compare numbers only within the same protocol.
• IoU threshold. IoU uses a binarisation threshold of 0.3 §5.2; the code logs several thresholds config.py. Report which one you used.
• Memory grows with resolution. The paper notes that memory scales badly with resolution because it does not use sparse volumes §6.6; /A uses 3997 MB at 128^3 tab. 15.
• Parameter count. /A has 96.31 M parameters, most of them in the ResNet-50 encoder and fully connected layers tab. 14. /F has 4.83 M tab. 14.
• Timing is a single GPU. The speed claim (about seven times faster than 3D-R2N2) uses one GTX 1080 Ti and excludes I/O §6.6. The 9.93 ms and 10.64 ms figures are from that setup tab. 14.
• Multi-view test set. The Things3D multi-view test keeps only objects with at least eight renders §5.6.2, so the 1-to-8 view numbers are on a different subset from the single-view results.
• Object-centred versus viewer-centred. Table 12 shows the object-centred model is better on seen categories, and the viewer-centred model is better on unseen ones §6.5 tab. 12.

Lineage

• Builds on: Pix2Vox, the ICCV 2019 version of this work, arXiv 1901.11153 according to the code's README. Changes described by the authors: VGG replaced by ResNet, multi-scale fusion in place of single-scale fusion, added decoder layers for 64^3 and 128^3, and the Things3D dataset §1.
• Compared against: 3D-R2N2, LSM, DeepMVS, RayNet, AttSets (multi-view); OGN, Matryoshka Networks, AtlasNet, Pixel2Mesh, OccNet, IM-Net (single-view) §2 §5.4.
• Superseded by: not known from content already in this wiki.