Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

In plain words

Image generators such as Stable Diffusion learn, by practising on noisy pictures, which direction an image should move in to look more like a real one. This paper reuses that learned direction to build 3D objects, and it needs no 3D training data. The object is stored as a grid of small cells that hold colour and density, and it is drawn from many camera angles. The image model judges each drawn view, and that judgement is passed back through the drawing step to adjust the grid, using the chain rule.

The hard part is that the image model only ever saw noisy pictures during training, while the drawn views are clean, so its answers on them are unreliable. The authors add noise to each view and average the answers to get a usable direction. A few extra rules keep the shape tidy, for example by penalising faint density in empty space. With a text prompt, the method produces 3D objects. The paper shows pictures rather than a benchmark score, and the work runs one optimisation per object.

TL;DR

• Score Jacobian Chaining (SJC) lifts a pretrained 2D diffusion model to 3D: it optimises a voxel radiance field so that its renders receive the denoiser's score, chained through a differentiable volume renderer §4.
• Clean renders are out of distribution for the denoiser, so the score on them is estimated with Perturb-and-Average Scoring (NEScore): add noise, denoise, average §4.1.
• An emptiness loss, a two-stage weight schedule and an optional centre-depth loss suppress floating density §4.2.
• Text-to-3D results use Stable Diffusion v1.5 and are shown qualitatively; the source reports no quantitative metric §5.2.
• Use it when you need text-driven 3D optimisation from a 2D model without 3D data and can afford about 25 minutes per object on one A6000 (author-reported) app. A3.

Problem

3D generative models are usually trained on 3D datasets, which are costly to acquire and come in incompatible formats (point clouds, meshes, radiance fields) §1. The paper wants to use only a pretrained 2D image diffusion model plus a differentiable renderer §1, §4.

The naive route fails. A denoiser is trained only on noisy inputs, while a rendered view is a clean image, so evaluating the denoiser directly on it is out of distribution. The resulting update produces sharp artefacts and moves the image away from the data manifold §4.1 fig. 2.

The setup assumes that a 3D asset is as likely as the expected density of its renders over camera poses §4, eq. 6. The score of this 3D distribution is the expectation over poses of the 2D score times the renderer Jacobian §4, eq. 11.

Method

The method has four parts, each cited to the paper.

1. 3D score by chain rule. Lower-bound the 3D log-density by Jensen's inequality over camera poses §4, eqs. 7–8. Its gradient is the expected 2D score at each render, multiplied by the renderer Jacobian §4, eq. 11.

2. 2D score on clean renders (NEScore). The standard score estimate from a denoiser is §3, eq. 3:

   \nabla_x \log p_\sigma(x) \approx \frac{D(x;\sigma) - x}{\sigma^2} eq. 3

   For a clean render, the paper adds Gaussian noise at level σ and averages the denoiser's scores over the noise. The term that averages the noise to zero is dropped §4.1, eqs. 13–16:

   \mathrm{NEScore}(x_\pi, \sqrt{2}\sigma) \approx \mathbb{E}_{n \sim \mathcal{N}(0, I)} \left[ \frac{D(x_\pi + \sigma n;\, \sigma) - x_\pi}{\sigma^2} \right] eq. 16

   The paper estimates this expectation by Monte Carlo §4.1. The √2 label comes from a lemma that relates the noised density at √2σ to the average over noisy copies at σ §4.1. The lemma and the claim that NEScore is a gradient of a lower bound are derived for an empirical mixture-of-Gaussians data distribution §4.1.

3. Renderer Jacobian (voxel radiance field). The 3D asset is a voxel grid of density and appearance features. Rays are cut into equal segments, values are trilinearly interpolated at segment starts, and they are alpha-composited with weights w_i = \alpha_i \prod_{j<i} (1-\alpha_j) and \alpha_i = 1 - \exp(-\tau_i d) §4.2. The 2D score is back-propagated through this renderer, which gives the Jacobian-vector product of the 3D score §4.2.

4. Regularisers. An emptiness loss on the ray weights, \mathcal{L}_{\text{emptiness}} = \frac{1}{N}\sum_i \log(1 + \beta w_i) with β = 10 §4.2. It is weighted by λ₁ and then λ₂ after K iterations §4.2. A centre-depth loss, used only for some objects, pushes the object to the image centre §4.2 app. A2.

   \mathcal{L}_{\text{center}}(D) = -\log\Big( \tfrac{1}{|B|} \sum_{p \in B} D(p) - \tfrac{1}{|B^c|} \sum_{q \notin B} D(q) \Big) §4.2

For text-to-3D with Stable Diffusion, the method renders a 4-channel latent feature field and uses a random σ schedule with a high language-guidance scale §5.2. The paper found annealing σ hard to tune in 3D and left it as future work §5.2.

The optimisation step below follows the paper's equations. The sign convention descends on the negative score so that the optimiser ascends log p.

# Score Jacobian Chaining, one optimisation step  [§4, app. A3]
# Input:  denoiser D, voxel field theta, camera sampler, prompt
# Output: theta updated in place

def sjc_step(denoiser, theta, camera, prompt, optimiser):
    # Render the clean feature image for this camera  [§4.2]
    render = volume_render(theta, camera)  # (1, C, H, W)
    # Draw one noise level from the schedule  [§5.2]
    sigma = sample_sigma_uniform(schedule)
    # Estimate the 2D score on the clean render  [§4.1, eq. 16]
    score_2d = nescore(denoiser, render, sigma, prompt)
    # Back-propagate the score through the renderer  [§4, eq. 11]
    # The minus sign makes gradient descent ascend log p  [§4]
    render.backward(-score_2d)
    # Add the emptiness regulariser on the ray weights  [§4.2]
    emptiness_loss(theta, step).backward()
    # Apply the Adamax update  [app. A3]
    optimiser.step()

# Perturb-and-Average Scoring on a clean render  [§4.1, eq. 16]
# Input:  clean render x (B, C, H, W); noise level sigma; prompt
# Output: 2D score estimate, same shape as x

def nescore(denoiser, clean, sigma, prompt):
    # Accumulate over Monte Carlo noise samples  [§4.1]
    total = zeros_like(clean)
    for noise in noise_samples:
        # Add noise at level sigma to the clean render  [§4.1]
        noisy = clean + sigma * noise
        # Denoise the noisy copy at the same level  [§4.1]
        denoised = denoiser(noisy, sigma, prompt)
        # Score term; the noise term averages to zero  [§4.1, eq. 16]
        total = total + (denoised - clean) / sigma**2
    # Average over the noise samples  [§4.1]
    return total / len(noise_samples)

# Emptiness regulariser on the ray weights  [§4.2]
# Input:  weights (rays, samples) from the volume renderer
# Output: scalar loss

def emptiness_loss(weights, beta=10.0):
    # Penalise small non-zero weights, which cause floating density  [§4.2]
    return mean(log(1 + beta * weights))

# Two-stage weight on the emptiness loss  [§4.2]
# K is not stated in the paper; see Pitfalls

def emptiness_weight(step, K):
    # Small weight first so geometry can form, larger afterwards  [§4.2]
    if step < K:
        return 1e4
    return 2e5

# Centre-depth loss on the depth image  [§4.2]
# Input:  depth (H, W); centre box mask (H, W), boolean
# Output: scalar loss

def center_depth_loss(depth, centre_box):
    # Mean depth inside the box minus mean depth outside it  [§4.2]
    inside = mean(depth[centre_box])
    outside = mean(depth[~centre_box])
    return -log(inside - outside)

Hyperparameters

Results

Author-reported. Setup: text-to-3D with Stable Diffusion v1.5 in a latent voxel field §5; comparison against Stable-DreamFusion, a third-party implementation on the same model §5.2. The source reports no quantitative metric, so all results are qualitative.

Comparison details: both methods run 10,000 iterations with no per-prompt tuning fig. 6. The authors note that both systems vary between trials §5.2. Video results are in the supplementary material and are not described further here app. A2.

Paper vs code

Official repository pals-ttic/sjc at commit f63c40d (the owner is the authors' lab; the README lists the five authors and this arXiv id). Discrepancies found:

1. Score scale. Paper: score is (D − x)/σ² eq. 3 and NEScore uses σ² eq. 16. Code: the same difference is divided by σ only run_sjc.py. D is x − σ·output adapt_sd.py. The code's gradient is therefore σ times the paper's score.
2. Monte Carlo count. Paper: average over sampled noises {n_i} §4.1. Code: one noise sample per step run_sjc.py. The appendix says the optimiser's momentum does the averaging app. A3, which matches the code.
3. Emptiness switch K. Paper: K is not stated §4.2. Code: the switch happens at half the steps, so at step 5,000 of 10,000 run_sjc.py. The ratio 2e5 / 1e4 = 20 matches the code's multiplier run_sjc.py.
4. Centre-depth loss. Paper: used only for a few objects, box size not stated §4.2. Code: default weight 0 run_sjc.py. Box is the central 50×50 of the 64×64 depth map run_sjc.py.
5. View-prompt threshold. Paper: overhead prompt when camera elevation is above 30° app. A3. Code: overhead when the sampled value vs is below 30°, and the polar angle is arccos(1 − 2·vs/π) pose.py. At vs = 30° that gives roughly 48° from the top, or about 42° elevation. This is our derivation from the code; the paper does not state it.
6. Camera range. Paper: cameras uniform on a hemisphere covering the voxel cube app. A3. Code: vs is sampled in [0°, 100°] pose.py, which reaches a polar angle of about 96°, slightly below the equator.
7. Noise schedule. Paper: random σ from the schedule §5.2; no range for SD. Code: uniform over ticks 30 to 990 of a 1000-step linear β schedule (β from 0.00085 to 0.012) run_sjc.py adapt_sd.py.
8. Denoiser input and guidance (code only). The noisy latent is divided by √(1+σ²) before the UNet, and classifier-free guidance uses an empty negative prompt adapt_sd.py. The paper states only the guidance scale §5.
9. Agreements checked. Adamax, lr 0.05, 10,000 steps app. A3 run_sjc.py. 100³ grid app. A3 run_sjc.py. 4×4 background app. A3 run_sjc.py. Radius 1.5 app. A3 run_sjc.py. Train FoV 40°–70° app. A3 pose.py. Guidance 100 app. A3 run_sjc.py. β = 10 §4.2 run_sjc.py. λ₁ = 1e4 and λ₂ = 2e5 §4.2 run_sjc.py. Emptiness formula §4.2 run_sjc.py.

Pitfalls

• Score scale matters. Following eqs. 3 and 16 literally gives a score that is 1/σ times the code's gradient. Because σ is drawn at random for each step, the two choices weight noise levels differently. Pick one and state it.
• Which σ is the schedule value. The paper labels the estimate NEScore(x, √2σ) §4.1 but the update evaluates the denoiser at σ eq. 16. The code uses the scheduled σ both as the noise scale and as the denoiser level run_sjc.py. Use σ inside the estimate, and treat the √2 only as a label.
• The justification is conditional. The lemma and claim assume an empirical mixture-of-Gaussians data distribution §4.1. The paper does not measure how far a real denoiser departs from this.
• Mode seeking. With random σ on the FFHQ and LSUN DDPMs, the 2D canvas converges to average faces or a blurry, empty canvas §5.1 fig. 4. The paper leaves the correct use of NEScore as an optimisation gradient open §5.1.
• Annealed σ is untested in 3D. The authors found it hard to tune and used random σ with high guidance §5.2. Do not assume an annealed schedule works for 3D.
• K is undefined. Choose it explicitly; the code uses half the steps run_sjc.py.
• Optimised space is latent features. The Stable Diffusion path optimises a 4-channel feature grid, not RGB §5.2. Images are decoded only for visualisation run_sjc.py. The decoder is not in the loss.
• Undocumented init and blending. The code initialises density with a −1.0 shift and blends a 4×4 background texture run_sjc.py. Neither is in the paper.
• Cost and reproducibility. About 25 minutes and 9 GB per object on one A6000 app. A3. The Stable-DreamFusion baseline is third-party code and its exact setup is not in the paper §5.2.
• Licence of the code. The repository is under CreativeML Open RAIL-M LICENSE. Its use-based restrictions (Attachment A) bind the model and its derivatives; the licence says they do not apply to the code LICENSE §4. Because the licence is not MIT, Apache-2.0 or BSD, no code is quoted in this card.
• Equation numbers were counted from the LaTeX source, not from the compiled PDF §4. Check them against the PDF before relying on them.

Lineage

• Builds on: denoising score matching and the score view of diffusion models §3; volume rendering and voxel radiance fields (Plenoxels, DVGO, TensoRF) §4.2, named in the paper but not linked here.
• Compared against: DreamFusion, arXiv:2209.14988, concurrent work with no card here §2, §4.3; Stable-DreamFusion, a third-party implementation §5.2; CLIP-guided methods such as DreamFields §2, not compared directly.
• Shares ideas: the randomised σ schedule and view-dependent prompts were influenced by DreamFusion §4.3 app. A3.
• Superseded by: not stated, and no later content in this wiki.