Score distillation sampling

Overview

SDS uses a pretrained diffusion model as a judge rather than as a sampler. A differentiable generator produces an image, the image is noised, and the frozen prior predicts the noise given the prompt. The difference between the predicted and the injected noise is the gradient, and it is back-propagated through the renderer into the generator's parameters 2211.10440 §3 2211.10440 eq. 1.

The invariant all variants share: the prior is frozen, and the signal comes from its noise prediction. Some variants use the residual directly; others use the clean estimate derived from it 2311.17042 §3.3 2303.14184 §3.2. Variants differ in what the parameters describe (one 3D scene per prompt, one 3D object from a photo, an explicit cloud of Gaussians, or the weights of a few-step generator), how the signal is weighted or switched over the timestep, and which prior sets the target: an image model at low resolution, a latent model, or a video model 2211.10440 §4 2311.17042 §3.3 2403.12008 §4.

Variants

Per-scene SDS on a neural field

The scene is a multi-resolution hash grid with two small output networks for colour, density and normals. Stage 1 of Magic3D uses this form with a 64x64 base diffusion prior and uniform timesteps with weight w(t) = 1 2211.10440 §4.1 2211.10440 §4.2 2211.10440 app. B. Card: Magic3D: High-Resolution Text-to-3D Content Creation. The paper reports about 15 minutes for this stage on 8 A100 GPUs 2211.10440 §5.

Coarse-to-fine SDS on a textured mesh

Magic3D's second stage converts the field into a deformable tetrahedral grid, extracts a mesh, and optimizes it with a 512x512 latent diffusion prior. The timestep range is narrowed to (0.02, 0.5) with weight σ_t², and the authors say a maximum in [0.5, 0.7] works well 2211.10440 §4.1 2211.10440 §4.3 2211.10440 app. B. Card: Magic3D. The high-resolution gradient through the encoder is the main cost of this stage 2211.10440 §4.1.

SDS on explicit Gaussians with a point-cloud term

GSGEN stores the asset as anisotropic 3D Gaussians, each with a centre, covariance, colour and opacity, and renders them by alpha compositing 2309.16585 §3.2 2309.16585 eq. 2. It starts from a Point-E point cloud for the prompt, with random colours rather than the cloud's 2309.16585 §4.3. The geometry stage adds a 3D SDS term on the Gaussian centres, with Point-E as the noise predictor. The 2D SDS term has weight 0.1 and the 3D term has weight 0.01 2309.16585 §4.1 2309.16585 §5.1 2309.16585 eq. 3. The appearance stage keeps only 2D SDS, acting on colours and opacities, with penalties on position and opacity and a compactness-based densification that fills gaps between Gaussians 2309.16585 §4.2 2309.16585 eq. 4. Card: GSGEN. The paper reports about 40 minutes per asset on one NVIDIA 3090 and gives no quantitative metrics; its comparisons are visual 2309.16585 §5.2. It also reports that a small densification threshold over-densifies under noisy SDS gradients, and a large one over-smooths 2309.16585 §4.2 2309.16585 §5.3.

Single-image SDS with a switch to a CLIP loss

Make-It-3D fits a NeRF to one reference photo, with no multi-view training data 2303.14184 §1. Its novel views get SDS with classifier-free guidance at weight 10, with timesteps drawn from 200 to 600 2303.14184 §4.1. For timesteps up to 400, the step uses a CLIP-based loss on a one-step denoised estimate instead, and SDS and that loss are never applied to the same step 2303.14184 §3.2 2303.14184 app. C.1. The paper states the SDS weight only as uniform over the timestep 2303.14184 §4.1. A masked reference-view loss and a depth correlation term sit alongside the SDS term 2303.14184 §3.2. Card: Make-It-3D.

The switch is a trade-off. In the coarse-stage ablation, SDS alone gives LPIPS 0.3045, and the CLIP loss alone gives the lowest LPIPS, 0.1260. The threshold of 400 gives LPIPS 0.1427, but better contextual (1.74 against 2.43) and CLIP (87.50% against 80.27%) scores 2303.14184 tab. 3.

SDS with a prompt token learned from the photo

RealFusion fits an Instant-NGP radiance field to one masked RGBA photo 2302.10663 §3.1. Its random prior views get SDS through Stable Diffusion at guidance scale 100 2302.10663 §3.1 2302.10663 app. A. The prompt is "an image of a ", where is a new token learned from random augmentations of the photo with the diffusion model frozen 2302.10663 §3.2 2302.10663 app. A. The paper's SDS omits the diffusion model's Jacobian, which it says improves quality and reduces compute and memory 2302.10663 §3.1. The photo itself supervises a fixed reconstruction camera, and normal smoothness applies on the prior views 2302.10663 §3.2 2302.10663 eq. 6. Card: RealFusion.

On 7 CO3D categories the paper reports a mean F-score of 9.58 against 8.24 for Shelf-Supervised Mesh Prediction, but it is lower on two categories 2302.10663 tab. 1. On a synthetic novel-view test that favours depth-supervised methods, it reports PSNR 16.5, below the depth-supervised methods in its table 2302.10663 app. D 2302.10663 tab. 2.

Masked SDS with a video prior

SV3D's 3D stage fits an Instant-NGP NeRF to its own generated orbit, converts it to a DMTet mesh, and uses photometric losses 2403.12008 §4. In the last 200 iterations of the fine stage, SDS with the video model is applied only where no reference view of the orbit sees the surface well. Each surface point takes its best-aligned reference camera, and a smoothstep over the dot-product range [0, 0.5] sets the mask 2403.12008 §4 2403.12008 app. F. The paper says plain SDS with this guidance oversaturates and blurs the result, which is why the mask is applied 2403.12008 §4. Card: SV3D.

Distillation into a few-step student

Adversarial Diffusion Distillation (ADD) replaces the scene with a student network initialised from the teacher's weights. The student sample is re-noised, the frozen teacher denoises it, and the distillation term is a squared L2 loss against that stop-gradient target, weighted by c(t) and combined with an adversarial term at λ = 2.5 2311.17042 §3.3 2311.17042 §4 2311.17042 eq. 4. The paper shows that squared L2 with c(t) = α_t / (2σ_t) · w(t) gives the SDS gradient 2311.17042 app. A. The final model uses a different weighting, NFSD, whose formula the paper does not give 2311.17042 §4.1. Card: Adversarial Diffusion Distillation.

When to use which

Treat the reported numbers as author-reported. The Magic3D speed comparison uses a DreamFusion runtime quoted from its paper, not a re-run 2211.10440 §1 2211.10440 §5.

Implementation sketch

Common core of the per-scene variant. The prior is used only to produce the residual, and the gradient reaches the scene through the renderer.

# One SDS step for a differentiable scene  [arxiv:2211.10440 §3] [eq. 1]
# Inputs: scene params theta, prompt y, frozen prior
# Output: updates theta in place

def sds_step(theta, prompt, prior):
    # Render from a random camera and light  [arxiv:2211.10440 app. B]
    x = render(theta, random_camera(), random_light())
    # Sample timestep and noise; stage 1 uses t ~ U(0, 1)  [app. B]
    t = sample_timestep()
    eps = normal_like(x)
    x_t = add_noise(x, eps, t)
    # Prior predicts the noise; the prior itself is not updated  [§3]
    eps_hat = prior.predict_noise(x_t, prompt, t)
    # Residual on the render, weighted by w(t)  [eq. 1] [app. B]
    grad_x = w(t) * (eps_hat - eps)
    # Back-propagate through the renderer into theta  [§3]
    x.backward(grad_x)

Distillation variant. The same residual becomes a loss on the student's sample, with the teacher's clean estimate as a fixed target.

# Distillation loss for one student sample  [arxiv:2311.17042 §3.3] [eq. 4]
# Inputs: student, real batch x0, prompt, frozen teacher
# Output: scalar loss for the student parameters

def distill_loss(student, x0, prompt, teacher):
    # One student evaluation at a sampled student timestep  [§3.1]
    x_hat = student(noise_real(x0), s)
    # Re-noise with the teacher's forward process  [§3.3]
    x_t = renoise(x_hat, t)
    # Teacher's clean estimate is a stop-gradient target  [§3.3]
    target = stop_gradient(teacher.denoise(x_t, prompt, t))
    # Weighted squared L2; c(t) = alpha_t / (2 sigma_t) * w(t) gives SDS  [app. A]
    return c(t) * sum_squares(x_hat - target)

Where the variants differ: the timestep range and weight w(t) are chosen by the paper for each stage 2211.10440 app. B; Make-It-3D switches to a CLIP-based loss at small timesteps 2303.14184 §3.2; RealFusion's prompt is a learned token 2302.10663 §3.2; GSGEN adds a point-cloud SDS term on Gaussian centres 2309.16585 §4.1; SV3D masks the SDS term per surface point 2403.12008 §4; the distillation weight c(t) is a free choice in the student case 2311.17042 §4.1; and the renderer is either a volume, a Gaussian cloud or a mesh rasteriser 2211.10440 §4.2 2309.16585 §3.2.

Contradictions

Most of the differences below are about scope, not disagreement. Each side is cited so the reader can check them.

• Magic3D optimizes each scene with SDS as its training signal 2211.10440 §3. ADD reports that distillation alone is ineffective at one step, with a COCO FID-5k of 315.6, and that the adversarial term is needed 2311.17042 tab. 1. That result concerns training a student, not optimizing one scene, so it does not contradict Magic3D.
• Weighting differs across the papers. Magic3D uses w(t) = 1 in stage 1 and σ_t² in stage 2 2211.10440 app. B. Make-It-3D leaves its weight as uniform over the timestep 2303.14184 §4.1, so its weight cannot be matched to ADD's equivalence result. RealFusion also states its weight only as w(t) 2302.10663 §3.1.
• Guidance scale differs. RealFusion says DreamFusion-style SDS needs a classifier-free guidance weight of 100 2302.10663 §3.1, and it reports that values above 30 gave good results 2302.10663 app. A. GSGEN also uses 100 2309.16585 §5.1. Make-It-3D uses weight 10 on its novel views 2303.14184 §4.1. The papers do not reconcile these values, so a reader should not assume one setting carries over.
• ADD's text and its table disagree on quality. The text says SDS-style and NFSD weighting give better quality and text alignment than the exponential default 2311.17042 §4.1. Table 1 agrees on the CLIP score (0.325 for SDS and 0.319 for the exponential default), but gives the SDS row a worse FID (22.3 against 20.6) 2311.17042 tab. 1. NFSD also has a worse FID than the exponential default (21.8) in that table.
• SV3D's text says masked SDS improves the result, and its 2D metrics agree. Its 3D table does not: the Chamfer distance is 0.024 with or without the soft mask, and the 3D IoU moves from 0.611 to 0.614 2403.12008 app. G.3 2403.12008 §4.

Papers

• Magic3D: High-Resolution Text-to-3D Content Creation: coarse-to-fine SDS, a hash-grid neural field and a textured mesh stage with a 512x512 latent prior.
• Make-It-3D: SDS with classifier-free guidance on novel views of a single-image NeRF, switched to a CLIP-based loss at timesteps up to 400.
• RealFusion: 360° Reconstruction of Any Object from a Single Image: SDS on random views of a radiance field fit to one photo, with a prompt from a token learned from that photo.
• Text-to-3D using Gaussian Splatting: SDS on explicit Gaussians, with a Point-E point-cloud SDS term in the geometry stage and 2D SDS alone in the appearance stage.
• SV3D: masked SDS with a video diffusion prior, applied to unseen surface in the last fine-stage iterations.
• Adversarial Diffusion Distillation: a distillation loss equivalent to SDS under one weighting, used to train one-to-four-step text-to-image students.