DiffRF: Rendering-Guided 3D Radiance Field Diffusion

In plain words

Image generators learn to start from random noise and clean it up step by step until a picture appears. DiffRF does the same, but its output is a 3D object stored as a grid of small cubes. Each cube holds how much material is there and what color it has. Because the result is a full 3D grid, the object can be viewed from any side.

Training 3D generators this way is hard, because the 3D grids used as training data are noisy and sometimes contain floating specks. DiffRF handles this by also checking its work with pictures. It makes a guess at the clean object, renders that guess from a few camera angles, and penalizes differences from the real photos. The model therefore learns to produce objects that look right from every side, not just objects that match a noisy 3D target.

Since the model works on the full grid, it can also fill in a hole. You mark the missing part of a chair, and the sampler regenerates that part while keeping the rest. The authors present this as a new task, and they report that it works without training a separate model for it.

TL;DR

DiffRF is a denoising diffusion model that generates explicit 32^3 voxel radiance fields. Its training loss adds a rendering loss on the denoised estimate to the standard noise-prediction loss, which the authors use to keep floating artifacts out of the learned prior. Results are author-reported on PhotoShape Chairs and ABO Tables, where the paper compares against π-GAN and EG3D. The paper also introduces masked completion by guided sampling. No official model code was found.

Problem

Radiance fields fitted to posed images are ambiguous and can contain artifacts, such as floating artifacts §1. Because of this, the per-object fits are poor targets for a diffusion model, which needs one clean ground-truth sample per noise vector §1. Earlier 3D-aware generators are GANs. They can produce photo-realistic views but do not naturally support conditional generation or completion §2 §4.1. The paper assumes that enough posed views exist for each training object §4.3.

Method

Overview. Each training object is first fitted as an explicit voxel radiance field §4 app. C. A 3D U-Net is trained to denoise these grids under a fixed noise schedule §3.2. Its loss has two parts: a noise-prediction loss, and a rendering loss computed on a clean-grid estimate, weighted by a factor that decays with the time step §3.3 fig. 2. At sampling time, the same denoiser generates a grid by ancestral DDPM sampling app. B.

1. Radiance field representation §3.1 eq. 1. A radiance field is a density field and an RGB field over a 3D domain, with the color independent of viewing direction (a footnote states the omission). Rendering along a ray integrates transmittance-weighted color eq. 1 with transmittance eq. 2. The fields are stored as explicit voxel grids, queried at continuous positions by bilinear interpolation of voxel vertices. The grid is a 4D tensor: three spatial axes and one channel axis for density and color §3.1.
2. Pre-activated fields §3.2 (footnote). The grids hold values before any activation. A proper activation is applied only at rendering, so that additive noise keeps the field valid. The paper does not say which activations it uses §3.2.
3. Forward (diffusion) process §3.2 eq. 5 eq. 6. Each step scales the previous state by sqrt(α_t) and adds Gaussian noise of variance β_t, with α_t = 1 − β_t. Here, α-bar_t is the cumulative product of α_1 to α_t §3.2. Its closed form is a Gaussian around sqrt(α-bar_t) f_0 with variance (1 − α-bar_t) eq. 6.
4. Reverse (generation) process §3.2 eq. 3 eq. 4. Sampling starts from a standard normal grid. Each step draws from a Gaussian with a learned mean and a fixed covariance Σ_t. The mean is parametrized by a noise-prediction network ε_θ through coefficients a_t and b_t eq. 4.
5. Coefficient choice §3.3. The paper sets a_t = 1/sqrt(α_t), b_t = β_t / sqrt(1 − α-bar_t), and Σ_t = β_t² / (2 α_t (1 − α-bar_t)) I §3.3. The first two match the standard DDPM mean. The variance is the paper's own choice, derived from its bound §3.3.
6. Noise-prediction loss §3.3 eq. 7 eq. 8. The objective starts from a variational bound on the negative log-likelihood eq. 7. With the choices in step 5 it reduces to a squared error between the true and predicted noise, summed over time steps eq. 8 §3.3. In training, a single time step is drawn uniformly, instead of summing over all steps §3.3.
7. Clean-grid estimate §3.3. From the predicted noise, the paper forms an estimate f̃_0 of the clean grid §3.3. The paper writes it as the ground-truth grid plus a scaled noise error (see Pitfalls).
8. Rendering loss §3.3 eq. 9 eq. 10 eq. 11. The estimate is rendered from a viewpoint with equation 1, and compared with the ground-truth image using the squared-norm distance ℓ_v eq. 10. The per-step term is weighted by ω_t = α-bar_t² eq. 11 app. B. The paper's text names the same weight w_t, with the same value §3.3.
9. Total loss §3.3 eq. 9. The total is L_RF + λ_RGB L_RGB, with a per-time-step version used in practice. The value of λ_RGB is not stated.
10. Architecture §3.3 app. B. The denoiser is a 3D U-Net. It adapts the 2D U-Net of [dhariwal2021diffusion] by replacing 2D convolutions and attention with 3D ones. It has four scaling blocks with two ResNet blocks each. The channel width grows linearly from 64 to 256. Skip attention is used at scale factors 2, 4 and 8, with 32 channels per head. Downsampling uses 3D average pooling.
11. Feature space bounding app. C. Grid features are clamped to [−1, 1]. The authors report that this stabilized sampling. How the bound is enforced (clipping, or another mapping) is not stated.
12. Unconditional sampling §4.1 app. B. DDPM sampling runs for 1000 steps. The paper reports 48.6 s per sample on an RTX 2080 Ti. A finished grid renders at 128×128 at over 380 FPS app. B.
13. Masked completion §4.2 fig. 6 (display unnumbered). Given a known grid f_in and a binary mask m, each step computes a fused estimate. Inside the mask it uses the model's estimate, and outside the mask it uses the input. The next state is then sampled around that fused estimate with variance (1 − α-bar_t). The method is adapted from RePaint §4.2. The authors state that it needs no task-specific training §1.
14. Image-to-volume synthesis §4.2 fig. 7 app. D. Classifier guidance [dhariwal2021diffusion] steers the sampler to reduce rendering error against a posed RGB image with a foreground mask. Each step updates the clean-grid estimate by a gradient step with a small guidance weight λ. The value of λ is not stated, and only qualitative results are reported §4.2.
15. CLIP conditioning app. E. Frozen CLIP ViT-B/32 embeddings condition the denoiser through cross-attention layers. The paper reports text-conditional and image-conditional examples, all qualitative app. E.

Pseudocode for one training step, following the paper's equations. The loop over views stands in for the expectation over viewpoints. The paper samples 8192 pixels per view, which this sketch omits:

# DiffRF training step for one radiance field f0  [§3.3]
# Input:  f0 pre-activation grid (32, 32, 32, 4)  [§4] [app. C]
#         views and ground-truth images (4 random views)  [app. B]
# Output: scalar loss L_RF + lam * L_RGB  (lam not stated)

def training_step(f0, views, images, lam):
    # Sample a time step uniformly from 1..T, T = 1000  [§3.3] [app. B]
    t = uniform_int(1, T)
    # Gaussian noise with the grid's shape  [§3.2] [eq. 6]
    eps = normal_like(f0)
    # Corrupt f0 in closed form  [§3.2] [eq. 6]
    f_t = sqrt(abar[t]) * f0 + sqrt(1 - abar[t]) * eps
    # Predict the noise with the 3D U-Net  [§3.3] [app. B]
    eps_pred = unet(f_t, t)
    # Noise-prediction loss  [§3.3] [eq. 8]
    loss_rf = squared_norm(eps - eps_pred)
    # Clean-grid estimate; uses f0, see Pitfalls  [§3.3]
    f0_tilde = f0 + sqrt((1 - abar[t]) / abar[t]) * (eps - eps_pred)
    loss_rgb = 0
    for view, image in zip(views, images):
        # Render the estimate; activation not stated  [§3.1] [eq. 1]
        rendered = render(f0_tilde, view)
        # Squared-norm distance to the image  [§3.3] [eq. 10]
        loss_rgb = loss_rgb + squared_norm(image - rendered)
    # Weight that decays with t  [§3.3] [eq. 11] [app. B]
    omega = abar[t] ** 2
    return loss_rf + lam * omega * loss_rgb

Constants and the sampler. The schedule is defined only by its endpoints, so the interpolation is one reading.

# Noise schedule and derived constants  [§3.3] [app. B]
# beta rises linearly from 0.0015 to 0.05 over T = 1000 steps
beta = linspace(0.0015, 0.05, T)  # interpolation rule not stated
alpha = 1 - beta
abar = cumprod(alpha)  # abar_t = product of alpha_1..alpha_t  [§3.2]
a = 1 / sqrt(alpha)  # mean coefficient a_t  [§3.3]
b = beta / sqrt(1 - abar)  # mean coefficient b_t  [§3.3]
sigma2 = beta ** 2 / (2 * alpha * (1 - abar))  # Sigma_t  [§3.3]

# Ancestral DDPM sampling of one radiance field  [§3.2] [app. B]
# Output: pre-activation grid f0 of shape (32, 32, 32, 4)

def sample(unet):
    # Start from standard normal noise over the grid  [§3.2]
    f = normal(shape)
    for t in reversed(range(1, T + 1)):
        # Predict the noise at step t  [§3.2]
        eps_pred = unet(f, t)
        # Mean from the reparametrization  [§3.2] [eq. 4]
        mean = a[t] * (f - b[t] * eps_pred)
        # Fixed variance Sigma_t; noise at t = 1 not stated  [§3.2]
        f = mean + sqrt(sigma2[t]) * normal(shape)
    return f

Masked completion, following the single-line update in §4.2:

# Masked completion by guided sampling  [§4.2]
# Input:  f_in known grid; m binary mask, 1 where the paper generates
# Output: completed grid

def complete(unet, f_in, m):
    # Start from noise  [§4.2]
    f = normal(shape)
    for t in reversed(range(1, T + 1)):
        eps_pred = unet(f, t)
        # Clean estimate from f_t; the paper's f0_tilde needs f0, so this
        # is an assumed substitute  [§3.3] [§4.2]
        x0_tilde = (f - sqrt(1 - abar[t]) * eps_pred) / sqrt(abar[t])
        # Model estimate inside the mask, input outside  [§4.2]
        prev_estimate = sqrt(abar[t]) * (m * x0_tilde + (1 - m) * f_in)
        # Next state sampled around the fused estimate  [§4.2]
        f = prev_estimate + sqrt(1 - abar[t]) * normal(shape)
    return f

Hyperparameters

Results

Author-reported. Setup: PhotoShape Chairs and ABO Tables, unconditional generation, compared against π-GAN and EG3D. Both GAN baselines train on the rendered images directly, whereas DiffRF trains on fitted radiance fields §4.1. KID and MMD are printed already multiplied by 10³ tab. 1 tab. 2.

Table 1, PhotoShape Chairs tab. 1:

Table 2, ABO Tables tab. 2:

Masked completion, Table 3 tab. 3. Higher is better for mPSNR; lower for FID. Masks cover 20 to 80% of the grid, with 200 samples per level. FID comes from renderings of 10 random views:

Headline reading (author-reported): on PhotoShape, DiffRF improves FID by 0.59 over EG3D and cuts MMD from 5.619 to 4.416 §1 tab. 1. On ABO it improves FID by 4.12 and MMD by 1.717 tab. 2. Removing the rendering loss raises FID by about 2.3 on PhotoShape and 8.8 on ABO §4.1. Both differences are in the authors' tables and text. No variance or number of seeds is reported.

Figures are described in words only. fig. 1 is the teaser. fig. 2 shows the training pipeline with the two losses on the denoised estimate. fig. 3 and fig. 4 are qualitative samples on PhotoShape and ABO, against π-GAN and EG3D. fig. 5 shows masked completions against EG3D, and fig. 6 shows the masking procedure. fig. 7 shows single-view reconstruction on ScanNet chairs. Appendix figures show more samples, the no-2D-loss comparison, and CLIP conditioning app. A app. E.

Paper vs code

No official code.

The paper gives no code link. The project page is at sirwyver.github.io/DiffRF. It redirects to normanm.de/DiffRF, which links the GitHub profile github.com/sirwyver and no repository. A GitHub search for "DiffRF" found SirWyver/DiffRF. Its only content is the authors' project-website template, a fork of the Nerfies page source with a README about that site. It has no LICENSE file and no model, training or rendering code. Another repository, evaapavel/diffrfe, is an unrelated diff service and was not used.

The paper's Method and Implementation sections are the only source for the model, so discrepancies cannot be checked against code.

Pitfalls

• The clean-grid estimate needs f_0. The paper writes f̃_0 as f_0 plus a scaled noise error §3.3. At training time f_0 is known. At sampling time (masked completion, guidance) it is not. Substituting f_t from equation 6 shows the paper's expression equals the usual estimate (f_t − sqrt(1 − α-bar_t) ε_θ) / sqrt(α-bar_t), when ε is the noise that made f_t. This equivalence is derived here, not stated by the paper. The masked completion pseudocode uses this form.
• Weight name and symbol. Equation 11 writes ω_t, while the text writes w_t §3.3. The value is α-bar_t² in both app. B.
• Unstated values. λ_RGB (loss weight), the viewpoint prior ψ beyond "random training views", the number of noise samples per step, the activations applied to density and color, the feature-bound method, and the λ of image guidance. Write "not stated" for each, and do not fill them from other papers.
• Noise schedule. The text gives only the endpoints 0.0015 and 0.05 for β. The interpolation rule is not stated. The covariance Σ_t is unusual and is implemented as written §3.3. The paper does not say whether noise is added at the final step t = 1 §3.2.
• Distance is squared. The text calls ℓ_v a Euclidean distance, but equation 10 is a squared norm eq. 10 §3.3. The paper does not say whether pixel errors are summed or averaged over the 8192 sampled pixels app. B.
• Masked completion update. The paper writes the update in one line §4.2. It uses f_0^{t−1} with index t − 1, so the t = 1 case is unclear. The paper does not say how the known region is noised, and the pseudocode treats it literally. The mask convention is inferred from the formula: m = 1 marks the region being generated, while the text describes the mask as applied to the input.
• Metrics mismatch. The main text names FID and KID §4. The appendix says FID and IS app. C. The tables report FID and KID tab. 1 tab. 2. Coverage and MMD use Chamfer distance on 2048 points per mesh, with generated sets twice the reference size app. C.
• Coverage drops without the 2D loss. Without the rendering loss, COV is higher on both datasets tab. 1 tab. 2. The authors attribute the difference to fewer spurious shapes with the loss §4.1. This is their interpretation.
• Cost. Each training step renders 4 views with 8192 pixels and 92 z-steps per ray app. B. The paper does not report training time or memory. Sampling takes 48.6 s per object on an RTX 2080 Ti app. B. The paper also notes that sampling is slower than GAN-based approaches §4.3.
• Resolution limit. The grid is capped at 32³ by training memory §4.3. The paper names adaptive and sparse grids as future work.
• Data pipeline. The radiance fields are fitted from renderings, and the GAN baselines train on the renderings directly §4.1 app. C. Comparisons therefore do not isolate the denoiser alone. The paper reports no variance across seeds.
• Dataset split. PhotoShape uses a 10% random hold-out by shape id, and ABO uses the official split app. C. Implementers should match these splits before comparing FID.

Lineage

• Builds on: denoising diffusion probabilistic models [ho2020denoising] for the training objective §3.2 §3.3; NeRF [mildenhall2020nerf] for the rendering equation §3.1; the 2D U-Net and classifier guidance of [dhariwal2021diffusion] for the architecture and image-to-volume guidance app. B §4.2; RePaint [Lugmayr_2022_CVPR] for masked sampling §4.2. None of these has a card on this site yet.
• Compared against: π-GAN and EG3D tab. 1 tab. 2 tab. 3.
• Related concepts: explicit voxel grids for radiance fields §3.1.