Novel View Synthesis with Diffusion Models

In plain words

Show a model one photo of an object and ask it to draw the same object from another angle. This paper trains an image model that does that. It learns from pairs of photos of the same object, each labelled with its camera position, and learns to fill in one photo given the other and both positions.

The hard part is making the new views agree with each other. If you generate each view on its own, the back of a car can come out different every time. The trick here is to generate each new view step by step, and at each step to look at a randomly chosen earlier view. Over the many small steps, the picture is pulled toward one consistent object. A special network layout lets the two photos exchange information, which the authors found made results much better.

The authors also point out that the usual image-quality scores can reward blurry outputs or miss inconsistent ones, so they add a test that trains a small 3D model on the generated views and checks how well it fits.

TL;DR

• 3DiM is a diffusion model that maps one posed view to a view at a new pose §2.1.
• Stochastic conditioning: each denoising step conditions on a random earlier view, so new views depend on all previous ones §2.2.
• Its X-UNet shares weights across the two frames and adds cross-attention between them §2.3.
• Author-reported: on SRN ShapeNet from one view, FID is far better than the baselines, while PSNR and SSIM are lower than some geometry-aware methods §3 tab. 1.
• The paper also proposes "3D consistency scoring", a neural-field-based metric §4.

Problem

Novel view synthesis from few images is ambiguous. Given one view, several completions are plausible, so the views need to depend on each other to stay 3D-consistent §2. Geometry-aware NeRF methods are consistent by construction but tend to need many views or per-scene training §1. Regression models give blurry outputs §1 §3. Naive image-to-image diffusion, which always conditions on the single input view, produces sharp frames that disagree with each other §3.1 tab. 2.

Method

2.1 Pose-conditional image-to-image diffusion

Training uses pairs of views of one scene with known poses p_1 and p_2 in SE(3) §2.1. A view is noised with a Gaussian process indexed by log signal-to-noise ratio λ eq. 1 eq. 2. The network predicts the noise added to one frame, given the other clean frame, both poses and λ eq. 3. The objective is the simple noise-prediction loss, not the ELBO §2.1.

The paper does not state how λ is sampled during training, how the target frame is chosen within a pair, or any loss weighting beyond the unweighted form in eq. 3 §2.1.

2.2 Stochastic conditioning sampler

Each new view is generated by a reverse diffusion chain of 256 steps §2.2 app. §7.3. At each step a conditioning view is drawn uniformly at random from the set of available clean views, which holds the input view and all previously generated views §2.2. The denoiser is called with the noisy target and that one clean view §2.2 fig. 3.

After the chain ends, the new view is added to the set, and the procedure repeats for the next view §2.2. The paper calls this a naive approximation to true autoregressive sampling; it does not train a multi-view model §2.2.

# Stochastic conditioning: generate one new view  [§2.2] [eq. 5] [eq. 6]
# Input:  views (clean views with poses), target pose, step schedule
# Output: one new view

def sample_new_view(views, target_pose):
    # Start from Gaussian noise at the highest-noise level  [§2.2]
    noisy = normal(shape=(H, W, 3))
    for lambda_t, lambda_next in steps_from_noisiest_to_cleanest():
        # Re-draw the conditioning view at every denoising step  [§2.2]
        index = uniform_choice(len(views))
        # Predict noise from the noisy target and that clean view  [§2.1]
        eps = denoiser(noisy, views[index], lambda_t,
                       views[index].pose, target_pose)
        # Estimate the clean target from the prediction  [eq. 5]
        clean_estimate = (noisy - sqrt(sigmoid(-lambda_t)) * eps) \
                         / sqrt(sigmoid(lambda_t))
        # Clip the estimate to the image range  [app. §7.3]
        clean_estimate = clip(clean_estimate, -1, 1)
        # Sample the next, less noisy state  [eq. 6] [app. §7.3]
        noisy = sample_posterior(noisy, clean_estimate,
                                 lambda_t, lambda_next)
    # The last step is noiseless  [§2.2]
    return clean_estimate

The posterior sampling in eq. 6 is written only as q(z | z_t, x̂); the variance choice is given in words in app. §7.3 and the explicit formula is not written out in the paper.

2.2 (cont.) Noise schedule

The log-SNR runs from λ_max = 20 down to λ_min = -20 along a cosine-shaped schedule app. §7.2. The paper gives this as

# Cosine log-SNR schedule over t in [0, 1]  [app. §7.2]
# Same map as the paper's listing, with min -20 and max 20.

def logsnr_schedule(t):
    # Angle endpoints for the chosen range  [app. §7.2]
    b = arctan(exp(-0.5 * logsnr_max))
    a = arctan(exp(-0.5 * logsnr_min)) - b
    # Map time to log-SNR  [app. §7.2]
    return -2 * log(tan(a * t + b))

2.3 X-UNet

The paper first tries a Concat-UNet, which stacks the clean view and the noisy view on the channel axis. It reports poor 3D consistency and weak alignment with the input, and uses this as the reason for a new architecture §2.3 §3.2. X-UNet differs in three ways §2.3 fig. 4:

1. Weights are shared between the two frames for all convolution and attention layers. Each frame keeps its own noise level: the clean conditioning frame gets the positional encoding of λ_max §2.3.
2. FiLM modulation uses the sum of a noise-level embedding and a pose embedding. The pose encoding is camera rays, the same as in SRT §2.3 app. §6.
3. A cross-attention layer lets each frame's feature maps query the other frame's feature maps. The paper's text says this is added at the coarse blocks §3.2; the listing applies it at the resolutions in attn_resolutions app. §6.

Model size is stated as about 471M parameters for the main model (ch = 256) and about 421M for the Concat-UNet with the same channels §3.2 app. §7.1. A 1.3B-parameter variant (ch = 448) is used in tab. 4 only.

Classifier-free guidance at sampling

Models are trained with the conditioning dropped 10% of the time app. §7.2. Guidance weight is 3.0 for cars and 2.0 for chairs, picked by eye for least visible 3D inconsistency app. §7.3. The guidance combination formula is not stated.

Hyperparameters

Architecture values come from the paper's JAX listing, which the authors state is the default of their XUNet module app. §7.1 app. §6. Training and sampling values are in the appendix text app. §7.2 app. §7.3.

Results

Author-reported. Setup: SRN ShapeNet cars and chairs at 128 x 128. Each held-out scene has 251 views; the model is conditioned on view #64 and generates the other 250 with one sampler call. PSNR and SSIM are averaged over views; FID compares all generated views against all ground-truth views §3. Geometry-aware baselines other than 3DiM were recomputed by the authors for PSNR, SSIM and FID, except the SSIM values marked (*), which are copied from prior work §3 tab. 1.

Main comparison tab. 1 (SRN ShapeNet, single view input):

Chairs in tab. 1 use the original split; see Pitfalls. The paper's table marks some SSIM values (*) as copied from prior work and possibly not computed with a Gaussian kernel tab. 1 caption.

Ablations on the re-split chairs* tab. 2 (Cars and Chairs*, PSNR / SSIM / FID):

Architecture comparison tab. 3 (Concat-UNet vs X-UNet, re-split chairs*):

3D consistency scoring tab. 4 (neural fields trained on each source; scores against held-out views from the same source):

The paper reports that the 1.3B model is comparable to the 471M model on cars, and better on chairs in this metric §4 app. §7.1.

Figures: fig. 1 shows samples from a model trained on all of ShapeNet without class conditioning; no metrics are given for it. fig. 5 shows SRN samples against baselines. fig. 6 shows ablation samples.

Paper vs code

No official code.

Searched, in order:

1. The paper's LaTeX source and abstract page on arXiv: no code link. The paper's appendix gives JAX code for the X-UNet module and the log-SNR schedule app. §6 app. §7.2; these are listings in the paper, not a repository.
2. The project page 3d-diffusion.github.io, which cites this paper (the source gives no URL for its "Supplementary Website"): its only code link is google-research/kubric, a renderer used to build data. It is not the method's code, so not used.
3. GitHub repository search for "3DiM novel view synthesis", "3dim xunet", and "3dim" within google-research: zero results.

Because there is no official repository, all implementation details come from the paper text and the paper's listings.

Pitfalls

• The appendix listing does not run as printed. Examples: the if self.use_ref_pose_emb line has no colon; a Conv call has an unclosed parenthesis; ResnetBlock reads h before it is assigned; GroupNorm uses F, which is undefined; AttnBlock unpacks h.shape where h_in is meant. Implement from the described architecture §2.3 and the values in app. §7.1, and treat the listing as a guide app. §6.
• Unconditional training wording is unclear. Training says the conditioning frame is set "to be at the maximum noise level" app. §7.2. In this paper λ_max is the cleanest level, since the clean frame gets λ_max positional encoding §2.3. The sampling section says the clean frame is replaced by standard Gaussian noise and the pose zeroed app. §7.3. The paper does not resolve this; pick one, and state it in your implementation.
• Unstated training choices. The distribution of λ during training, the rule for choosing the target frame within a pair, and any loss weighting are not stated §2.1 eq. 3.
• Guidance formula not stated. The paper gives weights, not how conditional and unconditional predictions are combined. Weights were picked by eye [app. §7.3].
• Compute grows with each view. One new view takes 256 denoising steps, and with guidance about 512 score evaluations fig. 1. Each step draws a view from a set that grows with every generated view, so the set has to be kept in memory §2.2.
• Variances. The sampler uses the ancestral variance choice named in app. §7.3. The paper says this choice made little difference to sample quality app. §7.3.
• Chairs split. The original SRN chairs test split has out-of-distribution poses; the authors re-split it, and the ablations use the re-split as chairs*. Numbers from tab. 1 are not comparable with tab. 2 to tab. 4 §3 tab. 2 caption.
• Metric caveats. Regression baselines score better on PSNR and SSIM but look blurry §3 tab. 2. FID can improve while consistency worsens §3. Use the consistency score §4 alongside the standard metrics. Its results depend on the neural-field details, so use the same ones for every model app. §7.4.
• SSIM and FID. Some baselines' SSIM may not use a Gaussian kernel, which the authors say can shift scores by up to 0.02 tab. 1 caption. FID uses the 250 generated views per scene and all ground-truth views §3, so FID depends on the set size.
• Neural-field details that are unclear. "Adam with weight decay 0.1" does not say whether decay is decoupled. The distortion and orientation loss terms are borrowed from other papers, and their weights are not stated app. §7.4.
• Scale-up unknowns. Training length and total examples are not stated. Only the 471M and 1.3B models are reported. The authors say a larger channel count (ch = 512) did not fit TPUv4 memory without model parallelism [app. §7.1].
• Section numbering. The source comments out \appendix, so the supplement is numbered 6 and 7 in the LaTeX source. The compiled PDF may label it differently; check the PDF before citing these numbers.
• Equation numbering is counted from the LaTeX source (the two-row eqnarray gives eq. 5 and eq. 6). Figure and table numbers are also counted from the source.

Lineage

• Builds on: denoising diffusion §2.1 (the simple noise-prediction objective from its citation); log-SNR parameterisation from progressive distillation §2.1; the UNet and its residual-block design §2.3 app. §7.1; weight sharing across frames from Video Diffusion Models, with the changes listed in §2.3; camera-ray pose encoding from Scene Representation Transformers §2.3; warmup and EMA settings from Karras et al. (2022) app. §7.2; classifier-free guidance app. §7.3.
• Compared against: SRN, PixelNeRF, VisionNeRF, CodeNeRF (geometry-aware); LFN, ENR (geometry-free) §3 tab. 1. Internal baselines: a Concat-UNet, a no-stochastic-conditioning sampler, and a one-step regression model §3 tab. 2 tab. 3.
• Superseded by: not stated in the paper. The paper does not claim a successor.