SynSin: End-to-end View Synthesis from a Single Image

In plain words

Show the model one photo of a room and ask what the room looks like from a camera that has moved a little. SynSin answers by building a rough 3D model of the scene inside the network. Every visible pixel gets a guessed distance from the camera and a learned description of what is there. These pieces form a cloud of dots floating in 3D, and each dot carries a learned feature instead of a colour.

To draw the new view, the dots are moved to the new camera position and projected onto the image plane. Dots are drawn as soft blobs, so the picture stays smooth and the drawing can be trained by gradient descent. The result still has holes where nothing was visible in the original photo. A second network fills those holes and cleans up the image, and a discriminator pushes the output to look like a real photo.

Nobody gives the model true depth during training. It learns depth only because a wrong depth makes the final picture look wrong. So it needs no depth labels, which the earlier methods did need.

TL;DR

• Input: one RGB image and a relative camera pose. Output: an image from the new viewpoint.
• A spatial feature network and a depth network build a point cloud of 64-dimensional features §3.1.
• A differentiable neural point-cloud renderer splats the points into the target view §3.2.
• A refinement network with a GAN discriminator inpaints holes and produces the final image §3.3.
• Trained end-to-end with L1, perceptual and adversarial losses on image pairs only §3.4.

Problem

Single-image view synthesis needs both 3D structure and scene semantics. Earlier methods used several views at test time §1, trained on ground-truth depth §1, or worked only on synthetic single objects §1. Voxel-based latent 3D representations tie output resolution to voxel size §1. The authors aim to handle complex real-world scenes from one image, without 3D supervision §1.

Method

The system has four parts §3, shown in fig. 2:

1. A spatial feature network f predicts a 64-dimensional feature for each pixel, at the input resolution §3.1.
2. A depth network d predicts a depth map at the same resolution §3.1.
3. A neural point-cloud renderer lifts features and depths into a 3D point cloud, moves it by the relative pose, and splats it into the new view §3.2.
4. A refinement network g turns the rendered features into the output image. A discriminator is applied only during training §3.3.

Feature network §3.1 app. B: eight ResNet blocks (BigGAN-style) that keep the image resolution. The final block outputs 64 channels. The paper defers exact details to the supplement §3.1.

Depth network §3.1 app. B: a UNet with eight downsampling and eight upsampling layers, then a sigmoid and a renormalisation so that depths fall within per-dataset minimum and maximum values §3.1.

Point-cloud renderer §3.2:

• A point p_i is splatted onto a disk of radius r around its projected centre. Its influence on pixel l is \rho_i(l) = \mathcal{N}(p_i, l), which is zero outside the disk and falls off linearly inside it eq. 1.
• The renderer keeps the K nearest points per pixel, sorted by depth, and accumulates them front to back with exponent \gamma eq. 2.
• \gamma = 0 gives a hard z-buffer §3.2.
• Gradients reach all K points, not only the nearest one, and reach the feature vectors as well as the depths §3.2 fig. 3.
• Performance, from the paper: a batch of six clouds with 512^2 points, rendered to six 256^2 images, takes 36 ms forward and 5 ms backward on one V100 §3.2.

Refinement network and discriminator §3.3 app. B:

• Eight ResNet blocks. Two blocks downsample the features, then two upsample back to the original resolution.
• Batch normalisation with noise, as in BigGAN §3.3.
• Spectral normalisation after each convolution §3.3.
• Discriminator: two multi-layer discriminators at a lower and a higher resolution, with a feature-matching loss §3.3.
• The GAN objective follows the cited high-resolution pix2pix-style work §3.3.

Training objective §3.4: \mathcal{L} = \lambda_{GAN}\mathcal{L}_{GAN} + \lambda_{l1}\mathcal{L}_{l1} + \lambda_c\mathcal{L}_c with \lambda_{GAN}=1, \lambda_c=10, \lambda_{l1}=1 §3.4. The paper trains on image pairs and their relative poses, with no ground-truth 3D §1 §3.4.

# View synthesis forward pass  [§3]
# Input:  image (B, 3, H, W); relative pose T; intrinsics K, K_inv
# Output: generated image (B, 3, H, W)

def synthesize(image, T, K, K_inv):
    # Spatial feature network f: one 64-d feature per pixel  [§3.1]
    features = feature_net(image)                   # (B, 64, H, W)
    # Depth network d: UNet, sigmoid, then rescaled to [min_z, max_z]  [§3.1]
    depth = sigmoid(depth_net(image))               # (B, 1, H, W)
    depth = depth * (max_z - min_z) + min_z
    # Lift each pixel to a 3D point that carries its feature  [§3.2]
    points = back_project(depth, K_inv)             # (B, H*W, 3)
    # Splat, keep K nearest per pixel, composite  [§3.2] [eq. 1] [eq. 2]
    rendered = splat_and_composite(points, features, T, K)
    # Refinement network g inpaints holes and corrects noise  [§3.3]
    return refinement_net(rendered)                 # (B, 3, H, W)

# Soft splatting and front-to-back compositing  [§3.2]
# Input:  points (N, 3) in the target view; features (N, C)
# Output: rendered features (C, H, W)

def splat_and_composite(points, features, T, K):
    # Move the points into the target camera  [§3.2]
    points = transform(points, T)
    for pixel in pixel_grid:
        # Keep the K nearest points that touch this pixel, sorted by depth  [§3.2]
        nearest = k_nearest_sorted(points, pixel, K)
        feature_sum = zeros(C)
        transmittance = 1.0
        for i in nearest:
            # Influence: 0 outside radius r, linear falloff inside  [eq. 1]
            # M is not stated in the paper; the code behaves as M = r
            distance = norm(centre[i] - pixel)
            rho = 0.0 if distance > r else 1.0 - distance / M
            weight = rho ** gamma
            # Front-to-back over-compositing  [eq. 2]
            feature_sum += weight * transmittance * features[i]
            transmittance *= 1.0 - weight
        rendered[pixel] = feature_sum
    return rendered

# One training step  [§3.4]  [code: models/base_model.py#L95-L137]
# Loss: lambda_GAN = 1, lambda_l1 = 1, lambda_c = 10  [§3.4]

def train_step(source, target, T):
    # Generator pass: synthesize the target view from the source  [§3.4]
    fake = synthesize(source, T)
    # Generator loss: adversarial, feature matching, L1, perceptual  [§3.3]
    g_loss = (gan_loss(fake)
              + 1.0 * l1(fake, target)
              + 10.0 * perceptual(fake, target)
              + feature_matching(fake, target))
    # Update the generator first, then the discriminators on the same fakes
    # [code: models/base_model.py#L104-L127]
    return g_loss

Hyperparameters

Results

Author-reported. Setup §4.1: Matterport3D (61 train, 11 val, 18 test scenes; image pairs with random view changes of up to \pm 20^\circ and \pm 0.32 m), RealEstate10K (about 57K, 14K and 7K scenes; 2000 test pairs), and Replica (evaluation only, no fine-tuning). Metrics: PSNR, SSIM and perceptual similarity (lower is better), split into all pixels (Both), regions visible in the input view (Vis) and regions not visible (InVis) §4.1 tab. 1.

Matterport3D (trained and tested on Matterport3D) tab. 1:

RealEstate10K (trained and tested on RealEstate10K) and Replica (models trained on Matterport3D, no fine-tuning) tab. 1. The paper gives standard deviations as subscripts; they are written here as "sd":

"Not run" marks cells the paper shows as "--" tab. 1.

System comparison on RealEstate10K, PSNR / SSIM / Perc tab. 2:

• SynSin: 22.31 / 0.74 / 1.18 (single input view).
• 3DView (single-image depth plus mesh, trained on depth data): 21.88 / 0.66 / 1.52.
• StereoMag [Zhou et al.]: 25.34 / 0.82 / 1.19, with two input views at test time §4.2.

Higher resolution, trained at 256^2 and tested at 512^2 tab. 3:

• SynSin: PSNR 22.06, SSIM 0.72, Perc 1.00.
• Vox w/ ours: PSNR 18.82, SSIM 0.61, Perc 2.47.

User study on 100 animated RealEstate10K trajectories, five raters, majority vote tab. 4:

• Either-or: SynSin preferred 68.7% of the time over Vox w/ ours (31.3%).
• Either-or-neither: SynSin 55.6%, Vox w/ ours 27.3%, neither 17.2%.

KITTI, against ContView app. A: SynSin PSNR 16.70, SSIM 0.52, Perc 2.07. ContView is 16.90, 0.54, 2.21. The paper says SynSin is similar or better on all metrics, but ContView is higher on PSNR and SSIM.

Paper vs code

Source: official repository facebookresearch/synsin at commit 501ec49b11030a41207e7b923b949fab8fd6e1b5 (shallow clone of the default branch, read on 2026-10-24). Paper version: v2 (2020-04-18).

• Learning rates conflict in the paper. Text gives generator 0.0001 and discriminator 0.01 §3.4. The appendix says "ours" uses 0.001 app. C. Code defaults are generator lr_g = 5e-4 and discriminator lr_d = 2e-3 [code: options/train_options.py#L205-L206]. The released scripts pass --lr 0.0001 train.sh submit.sh, but --lr sets only opt.lr, which the optimisers do not read. They use opt.lr_g and opt.lr_d [code: models/base_model.py#L21-L31]. So unless --lr_g/--lr_d are also set, the released scripts use the defaults. This is inferred from the code, not from a run log. The evaluation script sets lr_d = 0.001 [code: evaluation/eval_realestate.py#L116].
• Accumulation function. Paper uses alpha over-compositing eq. 2. Code default is wsum (weighted sum) train_options.py. The released scripts pass alphacomposite train.sh [code: submit.sh#L60], which matches the paper. The appendix says a weighted sum with and without normalisation gave similar results app. E.
• Falloff M. Paper names M but never gives its value §3.2 §3.4. Code computes the influence as 1 - \sqrt{\text{dist}} after dividing by r^2 [code: models/layers/z_buffer_layers.py#L84-L96]. With \text{rad\_pow} = 2 this is 1 - d/r if the rasteriser returns squared distances. That is the pytorch3d convention, which this repository does not contain. So M = r in effect.
• Depth range for Matterport3D. Appendix gives 0.1 to 10 app. D. Code has no Matterport override, so the default 0.5 to 10 applies [code: options/train_options.py#L164-L165] options.py. RealEstate10K (1 to 100) and KITTI (1 to 50) match options.py options.py.
• Training length. Paper: 50K iterations §3.4. Code: epoch loop up to max_epoch = 500 train_options.py train.py. No iteration cap was found, so the stop point is not reproducible from the code.
• Released scripts use --normalize_image. The RealEstate script passes it train.sh. In the code it rescales the output images from [-1, 1] to [0, 1] for the returned results base_model.py base_model.py. The input-side effect was not checked.
• Refinement network default. The argument default is unet [code: options/train_options.py#L33], which gives a UNet encoder and decoder [code: models/networks/utilities.py#L14-L33]. The paper and the released scripts use the ResNet variant resnet_256W8UpDown64 train.sh [code: models/networks/configs.py#L54]. Running with default flags builds a different model.
• Feature-matching weight and perceptual layers. Neither is in the paper. Code uses weight 10 train_options.py and layer weights 1/32 to 1 across five VGG-19 layers synthesis.py.
• Output activation. The refinement network ends in tanh [code: models/networks/architectures.py#L127-L145]. The paper does not say this. The released scripts also pass --normalize_image train.sh.
• Noise. The paper mentions batch normalisation with noise §3.3. Code implements it as a BigGAN-style conditional gain and bias, with a 20-dimensional Gaussian per sample, applied in each ResNet block [code: models/layers/normalization.py#L21-L47] blocks.py.
• Empty pixels. A learnable default_feature is registered but not concatenated; the line is commented out z_buffer_layers.py z_buffer_layers.py. Pixels with no point get zero features, which matches the paper's "near zero" description §3.3.
• Inverse depth. Code has an inverse-depth option for long-tailed depth [code: models/z_buffermodel.py#L97-L101]. It is not in the paper and is enabled only by a flag (commented out for KITTI in train.sh).
• Test-time noise. forward_angle resets torch.manual_seed(0) before each target view so that the noise is the same across angles [code: models/z_buffermodel.py#L166-L170]. The paper does not say this.
• Import-time seed. z_buffer_layers.py calls torch.manual_seed(42) when imported z_buffer_layers.py. This changes global RNG state for any program that imports the module.
• RealEstate10K pair sampling. Paper: frames within 30 of the reference, with the angle and position changes "if possible" §4.1. Code draws candidates from 80 frames centred on the reference realestate10k.py, and keeps frames where the angle or translation exceeds its threshold [code: data/realestate10k.py#L166]. Units of the angle threshold are not checked here.
• Batch size and epochs. Paper states neither §3.4. Code uses 32 in the RealEstate script train.sh and 16 by default [code: options/train_options.py#L282].

No discrepancies found for: feature dimension 64 §3.1 vs layers_enc ending at 64 configs.py; r = 4, K = 128, \gamma = 1 §3.4 vs defaults train_options.py and the released scripts submit.sh; \lambda_{l1} = 1, \lambda_c = 10 §3.4 vs train_options.py; two discriminator scales §3.3 vs discriminators.py.

Pitfalls

• Three learning rates. Generator 0.0001 and discriminator 0.01 in the text §3.4, "ours" 0.001 in the appendix app. C, and code defaults 5e-4 and 2e-3 train_options.py. Pick one set and state it in your implementation. The text values are the only ones the paper states as the training setup.
• --lr is silently ignored by the optimisers base_model.py.
• Table 1 is not uniformly best. The caption says the full model beats the baselines on all metrics tab. 1. The "small ft" ablation has higher PSNR on Matterport3D (21.14 vs 20.91 for Both) and Replica (22.36 vs 21.94), and a lower Vis perceptual score (0.98 vs 0.99). Only on RealEstate10K is the full model clearly best on PSNR. The full model ties or leads on SSIM on all three datasets.
• Matterport3D with ground-truth depth looks worse on InVis. The w/ GT variant has lower InVis PSNR (19.64) than the full model (19.80) tab. 1. The paper says true depth does not necessarily help §4.3.
• Unstated falloff. M is never given §3.2. Read it as M = r to match the code, but state it as a choice.
• Unstated constants. Batch size, the feature-matching weight, the noise dimension, the perceptual-loss layers and any learning-rate schedule are not in the paper. The paper gives no warm-up and no decay. The code defines niter and niter_decay options train_options.py; I did not verify whether they change the learning rate.
• Renderer memory and time. Each pixel composites up to K = 128 points §3.4. Timing is for a V100 only §3.2. The paper does not report memory use.
• Stochastic generator at test time. Batch-normalisation noise is sampled each forward pass normalization.py. Output varies from run to run unless you fix the seed, as forward_angle does.
• Resolution transfer. The paper attributes the 512^2 result to the point cloud being sampleable at any resolution §4.4. The voxel baseline degrades at 512^2 tab. 3. Only one such comparison is reported tab. 3.
• Figure and table cross-references. Figure and table numbers here follow source order. Floats may move in the compiled PDF, so check them against the paper.
• Matterport3D pair sampling is stated only in the paper (view angle within \pm 20^\circ, position within \pm 0.32 m) §4.1. I did not find it in data/habitat_data.py. The RealEstate10K sampler differs from the paper, as listed above.

Lineage

• Builds on: BigGAN (Brock et al., ICLR 2019) for the ResNet blocks and noise-injected batch normalisation §3.1 §3.3. The GAN objective and discriminator follow the high-resolution pix2pix-style work of Wang et al. §3.3. Differentiable point-based splatting follows Zwicker et al. §3.2. Soft rasterisation, Liu et al., is cited for the smooth-boundary idea §3.2.
• Compared against: DeepVoxels voxel renderer (Sitzmann et al.), re-implemented as "Vox" §4.2; appearance flow (Zhou et al., 2016) as "Im2Im" §4.2; StereoMag (Zhou et al., 2018), which uses two input views §4.2; a 3D View baseline built on MegaDepth-trained depth §4.2; ContView (Chen et al., 2019) on KITTI app. A.
• Superseded by: not stated in the card's sources.